Heterogeneous unmanned cluster cooperative hunting method, device and equipment and storage medium

Abstract

The invention discloses a heterogeneous unmanned cluster collaborative hunting method, device and equipment and a storage medium, and relates to the technical field of agent automatic control, and the method comprises the steps: determining a heterogeneous unmanned cluster, environment information and constraint conditions of a multi-agent collaborative hunting system, and constructing a physical model of the multi-agent collaborative hunting system; at the beginning of each decision cycle, modeling a task allocation problem of multiple agents into a'maximum flow-minimum cost 'graph theory problem, constructing a task allocation directed graph, determining an individual capture target corresponding to each agent, and based on a PPO framework, taking the maximum return of the corresponding individual capture target completed by each agent as an individual target, and performing task allocation on each agent; and with the maximum return of all captured targets completed by the heterogeneous unmanned cluster as a global target, an intelligent agent is guided to learn an optimal hunting strategy. According to the method, the problem of obvious insufficiency in the aspect of processing heterogeneous capability constraints in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent agent automatic control, and in particular to a method, apparatus, equipment and storage medium for heterogeneous unmanned swarm collaborative capture. Background Technology

[0002] In recent years, cross-domain collaborative technology for intelligent unmanned swarm systems has become a global research hotspot. This technology enables information sharing, collaborative decision-making, and efficient execution among intelligent devices such as unmanned vehicles and drones, greatly improving the efficiency and safety of unmanned swarm collaborative encirclement. However, the heterogeneous characteristics of drones and unmanned vehicles drastically increase the difficulty of unmanned swarm collaborative control. For example, unmanned vehicles and drones have different action spaces, state spaces, and dynamic characteristics. These differences are not only reflected in the differences in mathematical models but also lead to inconsistencies in the dimensions of the agent's state space and action space, further increasing the design difficulty of unmanned swarm encirclement. Constrained by these difficulties, the current encirclement efficiency and reliability of heterogeneous unmanned swarms composed of drones and unmanned vehicles are low. Therefore, researching efficient joint formation control strategies for heterogeneous unmanned swarms of drones and unmanned vehicles is extremely challenging.

[0003] The core idea of ​​existing multi-agent reinforcement learning (MARL) algorithms, especially MARL schemes applied to heterogeneous unmanned swarms, is to design an Actor-Critic decision network for each agent or class of agents, learning through a paradigm of centralized training and decentralized execution. During training, a central Critic network can access global information from all agents, enabling joint actions for accurate value assessment; however, during execution, each agent's Actor network makes independent decisions based solely on its local observations. While this scheme effectively addresses non-stationarity issues in multi-agent environments, it still suffers from several profound shortcomings in solving the problem of cooperative trapping of heterogeneous unmanned swarms considering capability constraints. These shortcomings include coarse-grained handling of heterogeneous capability constraints and a lack of targeted optimization mechanisms.

[0004] Therefore, we need a heterogeneous unmanned swarm collaborative capture method to address the significant shortcomings of existing technologies in handling heterogeneous capability constraints. Summary of the Invention

[0005] This invention provides a method, apparatus, equipment, and storage medium for the collaborative capture of heterogeneous unmanned swarms, which solves the problem that existing technologies have significant shortcomings in handling heterogeneous capability constraints.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a heterogeneous unmanned swarm cooperative capture method, the method comprising:

[0008] The heterogeneous unmanned swarm, environmental information, and constraints of the multi-agent cooperative encirclement system are defined, and a physical model of the multi-agent cooperative encirclement system is constructed. The heterogeneous unmanned swarm includes multiple agents and multiple targets. The agents include drones and unmanned vehicles. The environmental information includes the encirclement space and obstacle information. The constraints include obstacle avoidance and collision avoidance constraints, encirclement success condition constraints, and speed and energy constraints.

[0009] At the beginning of each decision cycle, based on the physical model, the multi-agent task allocation problem is modeled as a graph theory problem of "maximum flow - minimum cost", a directed graph of task allocation is constructed, and the individual capture target corresponding to each agent is determined;

[0010] Based on the PPO framework, the individual objective is to maximize the reward for each agent to complete its corresponding individual capture target, while the global objective is to maximize the reward for the heterogeneous unmanned swarm to complete all capture targets, thus guiding the agents to learn the optimal capture strategy.

[0011] In one possible implementation, the process of defining the heterogeneous unmanned swarm, environmental information, and constraints of the multi-agent cooperative trapping system, and constructing a physical model of the multi-agent cooperative trapping system, specifically includes:

[0012] Based on the number of drones and their positions, the number of unmanned vehicles and their positions, and the number of targets and their positions in the multi-agent cooperative capture system, a heterogeneous unmanned cluster is determined.

[0013] Environmental information is determined based on a preset three-dimensional containment space and the position and size information of randomly generated obstacles within the three-dimensional containment space;

[0014] Based on the preset obstacle area and the preset safe distances between agents and between agents and obstacles, obstacle avoidance and collision avoidance constraints are determined; the condition for successful capture is that the distance between the agent and the target is less than the preset radius; the constraints for speed and energy are that the agent's speed is greater than the target's speed and that the agent's energy consumption is linearly related to the travel distance.

[0015] Based on the heterogeneous unmanned swarm, the environmental information, and the constraints of obstacle avoidance and collision avoidance, the constraints of successful capture conditions, and the constraints of speed and energy, a three-dimensional model of the multi-agent cooperative capture system is performed to obtain the physical model of the multi-agent cooperative capture system.

[0016] In one possible implementation, at the beginning of each decision cycle, based on the physical model, the multi-agent task allocation problem is modeled as a graph theory problem of "maximum flow - minimum cost," a directed graph for task allocation is constructed, and the individual capture target corresponding to each agent is determined, specifically including:

[0017] Within each decision cycle, based on the physical model, a directed graph for task allocation is constructed, with the start of the decision as the source node, each agent as an agent node, each target as a target node, and the end of the decision as the sink node. The edges from the source node to each agent node represent the agent's maximum capability; the edges from each agent node to each target node represent the comprehensive cost including the distance between the agent and the target, the estimated energy consumption, and the heterogeneous type matching degree; and the edges from each target node to the sink node represent the minimum number of agents required to capture the target. The maximum capability of an agent represents the maximum number of targets that agent can capture at one time.

[0018] With the goal of minimizing the overall cost while satisfying the constraints, the task allocation model corresponding to the directed graph of task allocation is solved to obtain the task allocation scheme; the task allocation scheme assigns a corresponding individual capture target to each agent.

[0019] In one possible implementation, the PPO framework includes Actor-Critic network pairs, a global experience pool, and a centralized Critic network.

[0020] There are multiple Actor-Critic network pairs, each corresponding to one of the agents. Each agent receives its state space and action space through the Actor network in the Actor-Critic network pair and predicts its next action. The Critic network in the Actor-Critic network pair evaluates the expected reward of the agent executing the predicted next action.

[0021] The global experience pool is a single entity used to receive and store data streams from all Actor-Critic network pairs of agents; the data streams include the agent's current state, current action, reward, and next state;

[0022] The centralized Critic network is a single network used to evaluate the expected return of all agents working together; the centralized Critic network is also periodically trained by sampling data from the global experience pool.

[0023] In one possible implementation, the PPO framework uses maximizing the reward for each agent to complete its corresponding individual capture objective as the individual objective, and maximizing the reward for the heterogeneous unmanned swarm to complete all capture objectives as the global objective, guiding the agents to learn the optimal encirclement strategy, specifically including:

[0024] Each agent's Actor network receives the agent's state space and action space, and predicts the agent's next action based on the probability distribution of the state output action through the Actor network's decision function.

[0025] After executing the predicted next action, the agent's state space is sent to the agent's Critic network, and the agent's individual objective function is evaluated through the Critic network; the individual objective function is used to reflect the expected reward of the agent in completing the corresponding individual capture objective;

[0026] The global objective function of the heterogeneous unmanned swarm is evaluated through the centralized Critic network; the global objective function reflects the expected reward for the heterogeneous unmanned swarm to complete all captured targets.

[0027] The agent is guided to learn the optimal trapping strategy by maximizing the individual objective function corresponding to each agent and maximizing the global objective function.

[0028] In one possible implementation, the step of guiding the agent to learn the optimal trapping strategy by maximizing the individual objective function corresponding to each agent as the individual objective and maximizing the global objective function as the global objective specifically includes:

[0029] The value function of the global capture task is determined based on the sum of the global task value corresponding to the global target and the individual task value corresponding to each individual target;

[0030] The parameters of the PPO framework are updated in the direction of maximizing the value function of the global capture task, guiding the agent to learn the optimal capture strategy.

[0031] In one possible implementation, when guiding the agent to learn the optimal trapping strategy, the decision-making and learning behavior of each agent are treated as a time series, and the parameter update method of the PPO framework adopts a serialized update paradigm.

[0032] Secondly, the present invention provides a heterogeneous unmanned swarm cooperative capture device, the device comprising:

[0033] The system modeling module clarifies the heterogeneous unmanned swarm, environmental information, and constraints of the multi-agent collaborative encirclement system, and constructs the physical model of the multi-agent collaborative encirclement system; the heterogeneous unmanned swarm includes multiple agents and multiple targets, the agents include drones and unmanned vehicles, the environmental information includes encirclement space and obstacle information, and the constraints include obstacle avoidance and collision avoidance constraints, encirclement success condition constraints, and speed and energy constraints.

[0034] The task allocation module is used to model the multi-agent task allocation problem as a graph theory problem of "maximum flow - minimum cost" based on the physical model at the beginning of each decision cycle, construct a directed graph for task allocation, and determine the individual capture target corresponding to each agent.

[0035] The reinforcement learning module is used to guide the agents to learn the optimal capture strategy based on the PPO framework, with the individual objective being the maximum reward for each agent to complete its corresponding individual capture target, and the global objective being the maximum reward for the heterogeneous unmanned swarm to complete all capture targets.

[0036] In one possible implementation, the system modeling module is specifically configured to execute:

[0037] Based on the number of drones and their positions, the number of unmanned vehicles and their positions, and the number of targets and their positions in the multi-agent cooperative capture system, a heterogeneous unmanned cluster is determined.

[0038] Environmental information is determined based on a preset three-dimensional containment space and the position and size information of randomly generated obstacles within the three-dimensional containment space;

[0039] Based on the preset obstacle area and the preset safe distances between agents and between agents and obstacles, obstacle avoidance and collision avoidance constraints are determined; the condition for successful capture is that the distance between the agent and the target is less than the preset radius; the constraints for speed and energy are that the agent's speed is greater than the target's speed and that the agent's energy consumption is linearly related to the travel distance.

[0040] Based on the heterogeneous unmanned swarm, the environmental information, and the constraints of obstacle avoidance and collision avoidance, the constraints of successful capture conditions, and the constraints of speed and energy, a three-dimensional model of the multi-agent cooperative capture system is performed to obtain the physical model of the multi-agent cooperative capture system.

[0041] In one possible implementation, the task allocation module is specifically configured to execute:

[0042] Within each decision cycle, based on the physical model, a directed graph for task allocation is constructed, with the start of the decision as the source node, each agent as an agent node, each target as a target node, and the end of the decision as the sink node. The edges from the source node to each agent node represent the agent's maximum capability; the edges from each agent node to each target node represent the comprehensive cost including the distance between the agent and the target, the estimated energy consumption, and the heterogeneous type matching degree; and the edges from each target node to the sink node represent the minimum number of agents required to capture the target. The maximum capability of an agent represents the maximum number of targets that agent can capture at one time.

[0043] With the goal of minimizing the overall cost while satisfying the constraints, the task allocation model corresponding to the directed graph of task allocation is solved to obtain the task allocation scheme; the task allocation scheme assigns a corresponding individual capture target to each agent.

[0044] In one possible implementation, the PPO framework in the reinforcement learning module is configured to include Actor-Critic network pairs, a global experience pool, and a centralized Critic network.

[0045] There are multiple Actor-Critic network pairs, each corresponding to one of the agents. Each agent receives its state space and action space through the Actor network in the Actor-Critic network pair and predicts its next action. The Critic network in the Actor-Critic network pair evaluates the expected reward of the agent executing the predicted next action.

[0046] The global experience pool is a single entity used to receive and store data streams from all Actor-Critic network pairs of agents; the data streams include the agent's current state, current action, reward, and next state;

[0047] The centralized Critic network is a single network used to evaluate the expected return of all agents working together; the centralized Critic network is also periodically trained by sampling data from the global experience pool.

[0048] In one possible implementation, the reinforcement learning module is specifically configured to perform:

[0049] Each agent's Actor network receives the agent's state space and action space, and predicts the agent's next action based on the probability distribution of the state output action through the Actor network's decision function.

[0050] After executing the predicted next action, the agent's state space is sent to the agent's Critic network, and the agent's individual objective function is evaluated through the Critic network; the individual objective function is used to reflect the expected reward of the agent in completing the corresponding individual capture objective;

[0051] The global objective function of the heterogeneous unmanned swarm is evaluated through the centralized Critic network; the global objective function reflects the expected reward for the heterogeneous unmanned swarm to complete all captured targets.

[0052] The agent is guided to learn the optimal trapping strategy by maximizing the individual objective function corresponding to each agent and maximizing the global objective function.

[0053] In one possible implementation, when guiding the agents to learn the optimal trapping strategy with the individual objective function for each agent as the individual objective and the global objective function as the global objective, the reinforcement learning module is specifically configured to execute:

[0054] The value function of the global capture task is determined based on the sum of the global task value corresponding to the global target and the individual task value corresponding to each individual target;

[0055] The parameters of the PPO framework are updated in the direction of maximizing the value function of the global capture task, guiding the agent to learn the optimal capture strategy.

[0056] In one possible implementation, when guiding the agent to learn the optimal trapping strategy, the reinforcement learning module is further configured to: treat the decisions and learning behaviors of each agent as a time series and update the parameters of the PPO framework using a serialized update paradigm.

[0057] Thirdly, the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the heterogeneous unmanned swarm cooperative encirclement method described above.

[0058] Fourthly, the present invention provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the heterogeneous unmanned swarm cooperative encirclement and capture method described above.

[0059] The heterogeneous unmanned swarm cooperative trapping method provided in this invention, in practical applications, firstly, accurately constructs a physical model of the multi-agent cooperative trapping system based on the determined heterogeneous unmanned swarm, environmental information, and constraints. Then, at the beginning of each decision cycle, high-level task allocation is performed based on the physical model, assigning a corresponding individual capture target to each agent. This step significantly reduces the exploration space of the agent's reinforcement learning process, allowing it to focus on the path planning problem of "how to efficiently reach and trap the specified target," significantly improving the agent's learning efficiency in learning the optimal trapping strategy. Finally, based on the PPO framework, the agent optimizes the trapping strategy through reinforcement learning. In reinforcement learning, a hierarchical consensus mechanism enables the agent to simultaneously pursue both "individual goals" and "global goals." That is, while pursuing "team victory," the agent also receives rewards for "excellent individual performance," thus directly and precisely solving the problem of heterogeneous capability constraints for agents and guiding agents to learn optimal decision-making behavior based on their own characteristics. This invention, by introducing a task allocation mechanism and a hierarchical consensus mechanism, systematically solves the significant shortcomings of existing technologies in handling heterogeneous capability constraints. Attached Figure Description

[0060] Figure 1 A flowchart illustrating the steps of a heterogeneous unmanned swarm cooperative encirclement method provided in an embodiment of the present invention;

[0061] Figure 2 This is an overall network architecture diagram of a heterogeneous unmanned swarm cooperative encirclement method provided in an embodiment of the present invention;

[0062] Figure 3 This is a directed graph for task allocation corresponding to the "maximum flow-minimum cost" graph theory problem in a heterogeneous unmanned swarm cooperative capture method provided in an embodiment of the present invention.

[0063] Figure 4 The diagram shows a comparison of the capture task completion effects of the existing SAC, MAPPO, and HADDPG methods with the PPO-HSL heterogeneous unmanned swarm cooperative capture method provided by this invention.

[0064] Figure 5 The graphs show a comparison of the task completion times of the existing SAC, MAPPO, and HADDPG methods with the PPO-HSL heterogeneous unmanned swarm cooperative encirclement method provided in this invention, after running 100 times in four test environments (a), (b), (c), and (d).

[0065] Figure 6 This is a structural block diagram of a heterogeneous unmanned swarm collaborative capture device provided in an embodiment of the present invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, the use of "based on" or "according to" implies openness and inclusiveness, because processes, steps, calculations, or other actions "based on" or "according to" one or more of the stated conditions or values ​​may in practice be based on additional conditions or beyond the stated values.

[0068] While various solutions exist for target tracking and containment in complex environments, such as rule-based methods and optimization algorithms, these approaches have limitations in handling high-dimensional, large-scale problems, struggling to perceive environmental information in real time and make informed decisions. Furthermore, most existing solutions focus on the independent operation of a single unmanned system, lacking exploration of agent-based collaborative work, especially in dynamically changing urban environments. For example, traditional formation control strategies based on behavioral methods and artificial potential fields perform poorly in complex environments, particularly those with capability constraints, often exhibiting slow response times and inefficient energy allocation.

[0069] To improve the efficiency of target capture, researchers have also proposed methods based on potential field theory; for example, an ensemble algorithm based on improved potential field (IPF) was proposed for the multi-agent cooperative target search problem in cross-domain environments; and the AAPF-MPCI algorithm, improved based on adaptive artificial potential field (AAPF). However, potential field-based methods are sensitive to initial conditions, which may lead to excessive avoidance behavior by agents, resulting in low capture efficiency.

[0070] In recent years, game theory and multi-agent reinforcement learning methods have been widely used for heterogeneous swarm control of drones and autonomous vehicles. For example, game theory-based swarm control methods apply game theory and multi-agent reinforcement learning to the swarm control of drones and autonomous vehicles. Game theory allows multiple agents to make independent decisions in a distributed environment without global control. This characteristic is particularly suitable for task allocation and dynamic cooperation in drone and autonomous vehicle swarms. However, game theory-based methods also have significant drawbacks. Some game theory methods assume that agents can fully understand the policies or payoff functions of other agents, an assumption that is often difficult to satisfy in practice.

[0071] With the development of machine learning technology, more and more researchers are introducing multi-agent reinforcement learning to solve target encirclement problems. Agents are trained through multi-agent reinforcement learning algorithms to gradually improve their encirclement efficiency and optimize their path planning. For example, a cooperative pursuit method based on game theory and Q-learning can effectively solve the pursuit problem of a single escapee with learning capabilities in an obstacle-filled environment, but it has not studied the case of multiple escapees. Furthermore, value-based methods have the advantages of relatively high sampling efficiency, small variance in value function estimation, and less susceptibility to local optima; however, they typically cannot handle problems with continuous action spaces.

[0072] As can be seen from the above, heterogeneous unmanned swarm cooperative trapping technology considering capability constraints is a highly promising research direction. Most existing multi-agent reinforcement learning (MARL) schemes are based on the MADDPG, MAPPO, and HAPPO algorithms. Among them, the MADDPG algorithm is an extension of the DDPG algorithm in the multi-agent domain, equipping each agent with an independent Actor network and Critic network, but utilizing information from other agents when training the Critic network; the MAPPO algorithm applies the PPO algorithm to multi-agent scenarios, typically assuming that agents are homogeneous and sharing policy network parameters to improve learning efficiency; the HAPPO algorithm is specifically designed for heterogeneous agents, allowing each agent to have an independent policy network without sharing parameters, and making no assumptions about the decomposability of the joint value function, making it theoretically very suitable for heterogeneous systems such as UAV-autonomous vehicles.

[0073] Existing multi-agent reinforcement learning (MARL) algorithms have the following drawbacks:

[0074] Defect 1: The granularity of handling heterogeneous capability constraints is coarse, lacking a targeted optimization mechanism;

[0075] Defect 2: The learning paradigm is flat and fails to utilize the inherent structure of the task;

[0076] Thirdly, the exploration mechanism is too simplistic and easily gets stuck in local optima in complex adversarial environments.

[0077] One of the shortcomings is that while existing MARL algorithms can support heterogeneity in network structure, they do not fully consider and utilize the specific ability differences between agents, including speed and energy, at the policy learning level. Existing MARL algorithms typically input constraints as part of the environment state into the algorithm network, allowing the network to learn "black-box" on its own. This lack of guidance leads to low learning efficiency and makes it difficult to guarantee that the final policy can effectively balance task performance under different constraints.

[0078] The first defect arises because the original design of the traditional MARL algorithm framework was to solve the problem of multi-agent coordination. Its core lies in dealing with the credit allocation and non-stationarity among multiple agents. However, it lacks an explicit mechanism to model and optimize sub-problems that are strongly related to physical capability constraints, such as how to allocate tasks in winter based on remaining energy and speed advantages.

[0079] The second defect is that existing MARL algorithms typically treat the encirclement task as an end-to-end, flat decision-making process. The agent directly outputs low-level control commands, such as acceleration and angular velocity, at each time step; this approach ignores the inherent hierarchical and sequential structure of the encirclement task.

[0080] The second defect arises because the encirclement task can be logically decomposed into two levels: "high-level task allocation" and "low-level path execution." Furthermore, the agent's decisions are strongly correlated over time; decisions made at the current moment profoundly influence future states and action choices. The flattened learning paradigm forces the agent to explore a vast, hybrid policy space, making learning extremely difficult, convergent slowly, and hindering the acquisition of logically interpretable macroscopic collaborative strategies.

[0081] The third drawback is that in adversarial tasks, where the target actively tries to escape, the agent needs to conduct efficient exploration to find a robust capture strategy. Existing MARL algorithms typically rely on adding simple Gaussian noise to actions or on the randomness of the strategy itself. This approach is inefficient when facing "intelligent" targets that adjust their strategies based on the agent's actions, and is easily "deceived" by certain fixed escape patterns, thus converging to a suboptimal capture strategy.

[0082] The third defect arises because standard exploration mechanisms are passive and undirected; they do not actively create richer and more diverse training scenarios for the agent.

[0083] To address the significant shortcomings of existing technologies in handling heterogeneous capability constraints, this invention provides a method, apparatus, device, and storage medium for heterogeneous unmanned swarm collaborative capture.

[0084] like Figure 1 As shown, this embodiment of the invention provides a heterogeneous unmanned swarm cooperative encirclement method, which includes:

[0085] Step 101: Define the heterogeneous unmanned cluster, environmental information, and constraints of the multi-agent cooperative hunting system, and construct the physical model of the multi-agent cooperative hunting system.

[0086] The heterogeneous unmanned swarm includes multiple intelligent agents and multiple targets. The intelligent agents include drones and unmanned vehicles. The environmental information includes the capture space and obstacle information. The constraints include obstacle avoidance and collision avoidance constraints, capture success conditions constraints, and speed and energy constraints.

[0087] Multi-agent cooperative trapping system refers to a distributed cooperative trapping system involving multiple agents and multiple objectives.

[0088] In this embodiment, the system consists of N heterogeneous intelligent agents and P targets. The N heterogeneous intelligent agents include N1 drones and N2 unmanned vehicles. To delve into the mechanism of drone-unmanned vehicle cooperative encirclement missions, it is first necessary to perform precise physical modeling of the system based on the heterogeneous drone swarm, environmental information, and constraints of the multi-agent cooperative encirclement system.

[0089] Step 102: At the beginning of each decision cycle, based on the physical model, the multi-agent task allocation problem is modeled as a graph theory problem of "maximum flow - minimum cost", a directed graph of task allocation is constructed, and the individual capture target corresponding to each agent is determined.

[0090] Specifically, at the beginning of each decision cycle, we need to address the high-level task allocation problem of the system, that is, to solve the problem of unclear learning objectives for each agent, and assign a clear local objective to each agent, rather than letting reinforcement learning explore blindly.

[0091] This invention models the task allocation problem as a "maximum flow - minimum cost" graph theory problem, that is, solving the model corresponding to the constructed task allocation directed graph to obtain a task allocation scheme with minimum total cost that satisfies all constraints.

[0092] The output of this step assigns a specific local target to each agent, which greatly narrows the exploration space of reinforcement learning, allowing it to focus on the path planning problem of "how to efficiently reach and capture the specified target", thereby significantly improving learning efficiency.

[0093] Step 103: Based on the PPO framework, take the maximum reward for each agent to complete its corresponding individual capture target as the individual objective, and take the maximum reward for the heterogeneous unmanned swarm to complete all capture targets as the global objective, and guide the agents to learn the optimal capture strategy.

[0094] Specifically, the PPO framework is a reinforcement learning algorithm based on the Actor-Critic architecture, which improves training stability by limiting the stride of policy updates. The Actor-Critic architecture is a framework in reinforcement learning that combines policy gradient methods and value function evaluation methods, with these two core components working together.

[0095] An Actor, or policy network, is responsible for selecting actions based on the current state and updating parameters through policy gradients to maximize long-term rewards. The core function of an Actor is action selection, which involves generating an action probability distribution based on the policy function and then choosing the appropriate action to execute.

[0096] The Critic, or value network, is responsible for evaluating the value of states or state-action pairs. It updates parameters using temporal difference errors to approximate the true value function. The core function of the Critic is value assessment, which involves calculating the value of states or actions to provide feedback to Actors and guide their optimization strategies.

[0097] The PPO framework introduces a "near-end policy" constraint to limit the magnitude of policy updates and prevent policy failure due to excessive updates.

[0098] In this step, the individual objective function of each agent is maximized as the individual objective function, while the global objective function is maximized as the global objective function. That is, through a hierarchical consensus mechanism, the total value function of the agents is decomposed into individual objective functions and main task value functions. In other words, in the experiment, it is expected that each agent will complete its individual task (individual objective) while also enabling the entire heterogeneous unmanned cluster to complete the overall capture task (global objective) as much as possible.

[0099] This invention, through a hierarchical design, enables intelligent agents to pursue "team victory" (the global objective) while also receiving rewards for "individual outstanding performance" (such as completing tasks with lower energy consumption). This directly and precisely solves the capability constraint problem of heterogeneous unmanned swarms, guiding drones and unmanned vehicles to learn optimal behavior based on their own characteristics (e.g., unmanned vehicles prioritize speed, while drones prioritize endurance).

[0100] In practical applications, the heterogeneous unmanned swarm cooperative trapping method provided in this invention first accurately constructs a physical model of the multi-agent cooperative trapping system based on the determined heterogeneous unmanned swarm, environmental information, and constraints. Then, at the beginning of each decision cycle, high-level task allocation is performed based on the physical model, assigning a corresponding individual capture target to each agent. This step significantly reduces the exploration space of the agent's reinforcement learning process, allowing it to focus on the path planning problem of "how to efficiently reach and trap the specified target," thus significantly improving the agent's learning efficiency in developing the optimal trapping strategy. Finally, based on the PPO framework, the agent optimizes the trapping strategy through reinforcement learning. In reinforcement learning, a hierarchical consensus mechanism enables the agent to simultaneously pursue both "individual goals" and "global goals." That is, while pursuing "team victory," the agent also receives rewards for "outstanding individual performance," thereby directly and precisely solving the problem of heterogeneous capability constraints for the agent and guiding the agent to learn optimal decision-making behavior based on its own characteristics.

[0101] This invention systematically solves the problems of existing technologies in handling heterogeneous capability constraints, utilizing task structures, designing efficient exploration, and accurately allocating credit by introducing a task allocation mechanism and a hierarchical consensus mechanism.

[0102] Optional, such as Figure 2 As shown, in step 101, the heterogeneous unmanned swarm, environmental information, and constraints of the multi-agent cooperative trapping system are defined, and a physical model of the multi-agent cooperative trapping system is constructed, specifically including:

[0103] Step 1011: Based on the number of drones and their positions in the multi-agent collaborative capture system, the number of unmanned vehicles and their positions, and the number and positions of targets, determine the heterogeneous unmanned cluster.

[0104] Specifically, assume a heterogeneous unmanned swarm consists of N1 drones, N2 unmanned vehicles, and P targets. Drones and unmanned vehicles are represented by UAVs and UGVs, respectively, and the UAVs and UGVs are responsible for capturing the P targets. The agents include both drones and unmanned vehicles. The i-th agent in the agent swarm can be represented as:

[0105] UCX={UXV1, UXV2, UXV3,…,UXV N} (1);

[0106] Where N = N1 + N2 represents the total number of agents UXV participating in the capture in the agent cluster.

[0107] Suppose the i-th objective can be represented as Tar iFurthermore, the target can be described by an unordered sequence, and the target generation location is random, as shown below:

[0108] Tar={Tar1, Tar2, Tar3,…,Tar P} (2);

[0109] The positions of agent UXV and target Tar can be represented as:

[0110]

[0111] Step 1012: Determine the environmental information based on the preset size of the three-dimensional trapping space and the position and size information of the obstacles randomly generated within the three-dimensional trapping space.

[0112] Specifically, in actual encirclement tasks, the environment is a three-dimensional space, often containing various obstacles. These obstacles significantly impact the agent's path planning and task execution. In this embodiment, a three-dimensional encirclement space is defined, and obstacles are randomly generated within this space. The positions of these obstacles are randomly generated, and their dimensions (including length, width, and height) closely resemble normal buildings, thus ensuring the complexity and realism of the environmental information. The obstacles are specifically represented as follows:

[0113]

[0114] in, Cub represents the set of obstacles. Q This represents the Q-th obstacle in the set of obstacles.

[0115] Step 1013: Based on the preset obstacle area, preset safe distances between agents and between agents and obstacles, determine obstacle avoidance and collision avoidance constraints; use the distance between the agent and the target being less than the preset radius as the condition for successful capture; use the agent speed being greater than the target speed and the energy consumption of the agent being linearly related to the travel distance as speed and energy constraints.

[0116] Specifically, within the three-dimensional containment space, the agent not only needs to avoid obstacles but also needs to avoid collisions with other agents. Therefore, the constructed physical model should not only include static obstacles but also be able to provide real-time feedback on collision information between the agent and obstacles, and between agents themselves. In this embodiment, the modeling of obstacle avoidance and collision prevention can be achieved by setting spatial obstacle areas, safe distances between agents, and relative positional relationships. These constraints can be described as follows:

[0117]

[0118] Where C1 represents the intersection between the safe distance of agent i and the spatial obstacle region of the obstacle; C2 represents the intersection between the safe distance of agent i and the safe distance of agent j.

[0119] The successful capture condition is defined as the distance between the agent and the target being less than a preset radius r. For multi-target capture tasks, the capture task of the multi-agent cooperative capture system is considered successful when the capture radii of all targets meet the following condition:

[0120]

[0121] The speed and energy constraints explicitly define that the agent's speed must be greater than the target speed, and that the agent's energy consumption is linearly related to the distance traveled.

[0122] Specifically, in this invention, in terms of speed, it is assumed that the speed of the agents participating in the capture must be greater than the escape speed of the target; otherwise, the capture will never be possible. In terms of energy, it is assumed that the target has unlimited energy, while the agents participating in the capture need to complete the capture of the target under limited endurance conditions.

[0123] Therefore, the speed constraint of the present invention is specifically as follows:

[0124] max(υ UAV )>max(υ target (8);

[0125] max(υ UGV )>max(υ target (9);

[0126] Formula (8) indicates that the maximum speed of the UAV is greater than the maximum speed of the target, and Formula (9) indicates that the maximum speed of the unmanned vehicle is greater than the maximum speed of the target.

[0127] In multi-agent cooperative capture missions, energy consumption is a significant factor affecting mission execution. Each agent consumes energy during mission execution due to factors such as speed changes, steering angle adjustments, and movement. Therefore, a simple energy consumption model was developed for both unmanned vehicles and drones. In this embodiment of the invention, the energy consumption of drones and unmanned vehicles is considered to have a simple linear relationship with the travel distance, as detailed below:

[0128] ΔE=τ UAV ΔS (10);

[0129] ΔE=τ UG VΔS(11).

[0130] Step 1014: Based on the heterogeneous unmanned swarm, environmental information, and constraints such as obstacle avoidance and collision avoidance, successful capture conditions, and speed and energy constraints, perform three-dimensional modeling of the multi-agent cooperative capture system to obtain the physical model of the multi-agent cooperative capture system.

[0131] In this embodiment of the invention, the multi-agent cooperative trapping system is modeled in three dimensions using formulas (1) to (11) to obtain the physical model of the multi-agent cooperative trapping system.

[0132] Optionally, in step 102, at the beginning of each decision cycle, based on the physical model, the multi-agent task allocation problem is modeled as a graph theory problem of "maximum flow - minimum cost," a directed graph for task allocation is constructed, and the individual capture target corresponding to each agent is determined, specifically including:

[0133] Step 1021: Within each decision cycle, based on the physical model, construct a directed graph for task allocation, with the start of the decision as the source node, each agent as an agent node, each target as a target node, and the end of the decision as the sink node. The edges from the source node to each agent node represent the agent's maximum capability. The edges from each agent node to each target node represent the comprehensive cost, including the distance between the agent and the target, the estimated energy consumption, and the heterogeneous type matching degree. The edges from each target node to the sink node represent the minimum number of agents required to capture the target. The maximum capability of an agent indicates the maximum number of targets that agent can capture in a single operation.

[0134] like Figure 3 As shown in the figure, this diagram visually illustrates the graphical model of task assignment, which helps to understand how to transform complex assignment problems into solvable mathematical models.

[0135] Specifically, the constructed task allocation directed graph includes source nodes, agent nodes, target nodes, and sink nodes.

[0136] The capacity C of the edge from the source node to the agent node si This indicates the upper limit of agent i's capabilities. For example, agent i can only capture one target at a time.

[0137] The cost f of the edge from the agent node to the target node ij This represents a comprehensive cost that takes into account the distance between agent i and target j, energy consumption prediction, and heterogeneous type matching degree.

[0138] The capacity W of the edge between the target node and the sink node j This represents the minimum number of agents required to capture target j.

[0139] Step 1022: With the goal of minimizing the overall cost while satisfying the constraints, solve the task allocation model corresponding to the directed graph of task allocation to obtain the task allocation scheme.

[0140] In this task allocation scheme, each agent is assigned a corresponding individual capture target.

[0141] Specifically, by solving the task allocation model corresponding to the above directed graph, we can obtain a task allocation scheme with the minimum total cost that satisfies all constraints.

[0142] Optional, such as Figure 2 As shown, in step 103, the PPO framework includes Actor-Critic network pairs, a global experience pool, and a centralized Critic network.

[0143] There are multiple Actor-Critic network pairs, each corresponding to a specific agent.

[0144] In this process, each agent receives its state space and action space through the Actor network in the Actor-Critic network pair and predicts the agent's next action; the Critic network in the Actor-Critic network pair evaluates the expected reward for the agent to execute the predicted next action.

[0145] There is a single global experience pool, which is used to receive and store data streams from Actor-Critic network pairs from all agents.

[0146] The data stream includes the agent's current state, current action, reward, and next state.

[0147] The centralized Critic network is a single entity used to evaluate the expected return of all agents working together.

[0148] The centralized Critic network also periodically samples data from the global experience pool for training.

[0149] The PPO framework is the overall network architecture of this invention. This framework is a decentralized learning strategy in which each agent has an independent Actor-Critic network pair, but they learn collaboratively through a global experience pool and a shared centralized Critic network.

[0150] Optionally, in step 103, based on the PPO framework, the individual objective is to maximize the reward for each agent to complete its corresponding individual capture target, while the global objective is to maximize the reward for the heterogeneous unmanned swarm to complete all capture targets. This guides the agents to learn the optimal encirclement strategy, specifically including:

[0151] Step 1031: The Actor network of each agent receives the state space and action space of the agent, and predicts the next action of the agent based on the probability distribution of the state output action through the decision function of the Actor network.

[0152] Specifically, in actual data processing, the state space and action space of each drone and unmanned vehicle are input into the corresponding Actor network for each agent. The Actor network guides the agent's next action based on the probability distribution of the output action from the state through a decision function. Specifically, the Actor network π for the unmanned vehicle... θ (a t，UGV |S t，UGV ) represents the probability of an autonomous vehicle choosing an action in a given state; the Actor network π of the drone θ (a tUAV |S t，UXV ) represents the probability of the drone's action choice under a given state.

[0153] Let UXV represent drones or unmanned vehicles. The calculations for each agent are as follows:

[0154] π θ (a t，UXV |S t，UXV )=P(a t，UXV |S t，UXV θ UXV (12);

[0155] Among them, S t，UXV Indicates state, a t，UXV Indicates an action, θ UXV Let P(|) represent the policy parameters, and let P(|) represent the conditional probability calculation.

[0156] Step 1032: After executing the next action of the prediction, the agent's state space is sent to the agent's Critic network, and the agent's individual objective function is evaluated through the Critic network.

[0157] The individual objective function reflects the expected reward for the agent to complete the corresponding individual capture objective.

[0158] Specifically, after executing the predicted next action, the agent's state space is sent to the agent's Critic network, which then estimates the state-value function V. φ (S t,UXV and action-value function Q t,UXV (S t,UXV a t,UXV ).

[0159] Among them, the state-value function V φ(S t,UXV ) indicates that the drone and unmanned vehicle are in a certain state S. t,UXV The expected return is calculated as follows:

[0160]

[0161] Among them, R t，UXV Let γ represent the reward value at time step t, where t represents the current time step, k represents the future time step, T represents the total number of time steps, and k ranges from t to T. kt represents the time difference between the current time step t and the future time step k. γ is a discount factor that controls the discount on future rewards, determining the model's emphasis on future rewards. k-t This makes the future reward value R k It will be adjusted according to a decay ratio, meaning that future rewards will have a reduced impact, and the decay rate is controlled by γ.

[0162] Q t,UXV (S t,UXV a t,UXV Q is used to evaluate the quality of an agent's action choices, calculate the expected reward corresponding to the current state and action, and thus help the agent optimize its long-term strategy. t,UXV (S t,UXV a t,UXV The calculation method for ) is as follows:

[0163]

[0164] Where E(|) represents the conditional expectation calculation.

[0165] The state-value function V is calculated. φ (S t,UXV and action-value function Q t,UXV (S t,UXV a t,UXV After that, the advantage function can be used. Calculate the advantage of a particular action by an autonomous vehicle or drone in a given state, relative to average behavior.

[0166] The dominance function is used to measure a certain action a. t，UXV Relative to a certain policy in state S t，UXV The quality of the average action.

[0167] If the advantage function value of an action is large, it means that it is better than other possible actions in the policy in the current state. If the advantage function value is negative, it means that the action is worse. By introducing the advantage function into the policy gradient, we can effectively reduce the update variance, improve training stability, and achieve efficient policy optimization in complex environments.

[0168] Advantage function The specific calculation is performed using the following formula:

[0169]

[0170] In this invention, an importance sampling ratio and a pruning mechanism are used to ensure the stability of policy updates. The pruning mechanism ensures that policy optimization is asymptotic and avoids over-reliance on noisy advantage function estimates. Therefore, the individual objective function, i.e., the objective function for the agent to complete its individual task, is calculated as follows:

[0171]

[0172] in, This represents the objective function for each agent.

[0173] The importance sampling ratio represents the importance sampling ratio for each agent. This ratio measures the consistency between the old and new policies on the same data during policy updates. A ratio close to 1 indicates that the old and new policies are very similar; a ratio deviating from 1 indicates a significant change in the policy. In reinforcement learning, policy updates typically rely on empirical data sampled from the old policy. Directly using old data to optimize the new policy may introduce bias, but the importance sampling ratio can correct this bias. It allows the use of sampled data from the old policy during the optimization of the new policy.

[0174] The importance sampling ratio is calculated as follows:

[0175]

[0176] in, Indicating a new strategy, This indicates the old strategy.

[0177] `clip` represents the clipping function, and `ε` is the clipping threshold. It controls the magnitude of each measurement update, preventing excessive changes in the strategy and thus maintaining training stability.

[0178] Step 1033: Evaluate the global objective function of the heterogeneous unmanned cluster through a centralized Critic network.

[0179] The global objective function is used to reflect the expected reward of a heterogeneous unmanned swarm completing all captured targets.

[0180] Specifically, for a multi-agent cooperative trapping system, using a synchronous heterogeneous policy update method, the policies of all agents are updated simultaneously. Therefore, the objective function of the i-th agent should be merged into:

[0181] L(θ)=L CLIP (θ i (18);

[0182] Using the above formula, agents in a multi-agent cooperative trapping system can undergo multiple iterative training sessions based on the initial state, enabling them to adaptively learn and optimize the trapping strategy. The system's goal is to trap the target. To achieve global consistency through cooperative optimization, each agent is trained based on a unified optimization objective. In this invention, a centralized Critic network is used to evaluate the expected reward of the heterogeneous unmanned swarm completing the capture of all targets. Therefore, the global objective function, i.e., the objective function for the heterogeneous unmanned swarm to achieve the global objective, is as follows:

[0183]

[0184] Step 1034: With maximizing the individual objective function corresponding to each agent as the individual objective, and maximizing the global objective function as the global objective, guide the agents to learn the optimal trapping strategy.

[0185] When optimizing the strategy, both the individual objective function and the global objective function are considered. That is, while expecting each agent to complete its individual task, the system should also be made to complete the overall capture task as much as possible.

[0186] Optionally, in step 1034, the individual objective function for each agent is maximized, while the global objective function is maximized, guiding the agent to learn the optimal trapping strategy. This specifically includes:

[0187] The value function of the global capture task is determined based on the sum of the global task value corresponding to the global objective and the individual task value corresponding to each individual objective.

[0188] The parameters of the PPO framework are updated in the direction of maximizing the value function of the global capture task, guiding the agent to learn the optimal capture strategy.

[0189] Specifically, the joint reward of all agents is represented as the global objective, i.e., policy updates are performed through collaborative optimization, and the policy is updated at the end of each stage to ultimately achieve global consistency. The global value function of the agents is decomposed into two parts: the main task value function and the individual task value function. That is, in the experiment, it is expected that each agent completes its individual task, while also ensuring that the entire system completes the overall capture task as much as possible.

[0190] The value function of the global capture task is specifically expressed as follows:

[0191]

[0192] Among them, V g(S t V represents the global task value corresponding to the global objective, used to reflect the global impact of all agents in a multi-agent cooperative trapping system on the trapping environment; l，i (S t，i ) represents the local value of agent i corresponding to the individual goal, which is used to reflect the impact of agent i's individual behavior on agent i itself and the hunting environment.

[0193] The value function V of this hierarchical global capture task φ (S t，UXV This can decouple the global task value from the individual task value, allowing the agent's local behavior to be optimized independently.

[0194] Optionally, when guiding agents to learn the optimal trapping strategy, each agent's decision and learning behavior are treated as a time series, and the parameter update method of the PPO framework adopts a serial update paradigm.

[0195] The core idea of ​​the sequential update paradigm is to progressively use time-series data for learning and updating. That is, the data at each time step not only directly contributes to the optimization objective but also influences the updates at subsequent time steps. It is a dynamic update process based on historical experience. The sequential update paradigm recursively captures short-term historical dependencies. This design not only avoids competition or conflict caused by simultaneous updates but also utilizes the action information of the previous agent to guide the learning process of subsequent agents, thereby improving training efficiency and effectiveness. In multi-agent systems, the decisions of each agent at each time step affect the learning process at subsequent time steps. Therefore, a sequential global optimization objective can be designed to explicitly quantify the dependencies between time series.

[0196] The global optimization objective J of serialization ssq Specifically, it is expressed as follows:

[0197]

[0198] Where t represents the current time step, t′ represents the next time step, and l represents the importance adjustment parameter for time step dependencies.

[0199] I(a t′ a t The action mutual information at time steps t and t′ can be represented by the conditional entropy H(a). t′ |a t ) and marginal entropy H(a t′ The formula is as follows:

[0200] I(a t′ a t )=H(a t′)-H(a t′ |a t ) (twenty two);

[0201]

[0202] Wherein, p(a t′ ) represents the marginal distribution, p(a t′ a t () represents the joint distribution. These formulas, when combined, can be written as:

[0203]

[0204] Optionally, in addition to optimizing the agent's reinforcement learning strategy, this invention also designs a dynamic behavioral strategy for the target.

[0205] Specifically: when no intelligent agent for encirclement appears within the target's preset safe distance, the target moves randomly according to the distribution u(-1,1); when one or more intelligent agents for encirclement appear within the target's preset safe distance, the target moves in a direction perpendicular to the intelligent agent or the target.

[0206] This escape design of the target artificially increases the randomness and adversarial nature of the capture environment, forcing the agent swarm to not only learn strategies for a certain fixed pattern, but also to master a general capture method that can deal with various escape behaviors, thereby greatly enhancing the algorithm's generalization ability and robustness.

[0207] Optionally, in this embodiment of the invention, in order to guide the agent to learn the desired behavior, the reward value is calculated using a set comprehensive reward function:

[0208] R = R t +R c +R e +R p (26);

[0209] Where R represents the comprehensive reward function, R t Distance rewards encourage agents to move closer to and toward the target; R c The shorter the completion time, the higher the reward; R e The reward is based on energy; the more energy the agent has remaining after completing the task, the higher the reward. p This is a penalty item used to punish collisions and mission failures.

[0210] The reward function is a core concept in reinforcement learning; it defines the reward value in a given state S. t，UXV Take a certain action a t，UXVSubsequently, the environment provides immediate feedback to the agent. The reward function is the primary basis for the agent's learning and decision-making. The reward function needs to have a clear objective, balance short-term and long-term rewards, avoid suboptimal behavior, and provide sufficient feedback. To achieve more efficient and accurate model training, it is necessary to design a reasonable set of reward functions for the agent to guide its learning process towards maximizing cumulative rewards.

[0211] In an embodiment of the present invention, R t This represents the distance reward. If the agent gets closer to the target or moves towards the target, it will be given a corresponding reward. This reward is designed to encourage the agent to get as close to the target as possible.

[0212]

[0213] Among them, D t，i D represents the distance between agent i and the target at time t. t-1，i Let represent the distance between agent i and the target at time t-1. Let represent the angle between the direction of motion of agent i and the target at time t. κ1 represents the angle between the direction of motion of agent i and the target at time t-1, and κ2 represents the preset distance weight coefficient and angle weight coefficient, respectively.

[0214] Meanwhile, the design of the reward function needs to consider the time span of the task to avoid the agent focusing only on short-term gains. Therefore, a reward R for completing the goal is set for the agent. c If an agent meets the capture conditions, it will receive this reward. For target capture tasks, the shorter the capture time, the greater the capture reward. Therefore, the capture time reward can be defined as:

[0215] R c =κ3(1000-t)+κ4 (28);

[0216] Wherein, κ3 and κ4 represent the preset linear weighting coefficients of the reward for completing the encirclement.

[0217] For intelligent agents, battery life is crucial. Considering this factor, it is expected that they can select relatively energy-efficient capture strategies when faced with equivalent tasks. Therefore, R is also set for the intelligent agent. e The reward, specifically the energy reward, is given based on the remaining energy after completing the task. R e The calculation method is as follows:

[0218]

[0219] Among them, E UXV，i Let ΔE represent the initial energy of agent i. UXV，iκ5 represents the energy consumption of agent i in the capture task, and κ5 represents the preset energy weighting coefficient. The more energy the agent has remaining after completing the task, the greater the reward it receives.

[0220] Rp is the penalty term, mainly targeting collision penalties and task failure penalties. This is to avoid collisions between agents or with obstacles. The collision penalty can be defined as:

[0221]

[0222] Wherein, under the condition that C1 is 1, that is, under the condition that the intersection between the safe distance of agent i and the spatial obstacle area of ​​the obstacle is 1, the preset first penalty coefficient is represented by κ6; under the condition that C2 is 1, that is, under the condition that the intersection between the safe distance of agent i and the safe distance of agent j is 1, the preset second penalty coefficient is represented by κ7.

[0223] At the end of each time step, collision detection is required. If a collision occurs, the agent is penalized. The setting of the reward function is a crucial part of the entire algorithm implementation and directly affects its performance.

[0224] The integrated reward function designed in this invention, combined with this invention, jointly guides the algorithm toward convergence towards an efficient, energy-saving, and robust cooperative trapping strategy.

[0225] The heterogeneous unmanned swarm cooperative trapping method proposed in this invention, denoted as the PPO-HSL method, systematically solves the various defects of existing technologies (such as SAC method, MAPPO method, and HADDPG method) in dealing with heterogeneous trapping problems considering capability constraints through mechanisms such as hierarchical consensus mechanism, serialized update, agent task allocation, target uncertainty exploration, and multi-agent advantage estimation. It achieves significant technical effects and economic benefits.

[0226] Table 1 is a comparison of the completion time of the encirclement task for heterogeneous unmanned swarm collaborative encirclement using the PPO-HSL method of the present invention, as well as the existing SAC method, MAPPO method, and HADDPG method.

[0227] Table 1 Comparison of Encirclement Task Completion Time for Heterogeneous Unmanned Swarm Collaborative Encirclement

[0228]

[0229]

[0230] As shown in Table 1, the average task completion time of PPO-HSL is 315.02 seconds, significantly lower than that of HADDPG (561.10 seconds), SAC (927.27 seconds), and MAPPO (963.25 seconds). The experimental data in Table 1 demonstrate that the PPO-HSL method of this invention performs exceptionally well in completing encirclement tasks, reducing the time by approximately 44% compared to its closest competitor, HADDPG, and by nearly two-thirds compared to MAPPO or SAC.

[0231] like Figure 4 As shown, within 1000 seconds, the task completion rate of PPO-HSL is close to 99%, while HADDPG is only about 60%, and MAPPO is only 21%. This indicates that the strategy of the present invention is not only fast but also extremely reliable. Therefore, the core advantage of the present invention lies in its extremely high task completion efficiency. Using the method of the present invention can significantly improve the efficiency and success rate of encirclement and capture, and greatly shorten the task completion time.

[0232] like Figure 5 As shown in the violin plot, the data distribution of PPO-HSL is more concentrated and stable. Due to the characteristics of the violin plot, there are predicted portions at both ends; images with predicted portions less than 0 are discarded here. In terms of time metrics, the PPO-HSL algorithm performs best in all four environments, requiring less time to complete the task, with most data concentrated between 200 and 300 seconds. Therefore, the method strategy of this invention exhibits extremely strong robustness and stability, adapting to complex dynamic environments.

[0233] Table 2 is a comparison of the acquisition completion time of ground or air targets using the PPO-HSL method of the present invention, as well as the existing SAC method, MAPPO method, and HADDPG method, where target1 and target2 are ground targets and target3 is an air target.

[0234] Table 2 Comparison of acquisition completion time for ground and air targets

[0235] MAPPO SAC HADDPG PPO-HSL target1 134.69 81.50 31.51 32.52 target2 146.96 86.17 31.26 33.32 target3 607.91 984.71 576.25 371.23 Average time 300.99 384.13 213.01 145.69

[0236] As shown in Table 2, for ground targets (target1, target2), PPO-HSL is almost as fast as the best-performing HADDPG (approximately 30 seconds). However, for the more challenging air targets (target3), PPO-HSL demonstrates its superiority, with an average time of 371.23 seconds, while HADDPG and MAPPO require nearly 600 seconds, and SAC approaches 1000 seconds. PPO-HSL is approximately 55% faster than the suboptimal algorithm. Therefore, this invention performs exceptionally well against both ground and air targets. It can intelligently schedule different types of intelligent agents (UAV / UGV) based on target characteristics to maximize their effectiveness. This invention fully utilizes heterogeneous advantages to achieve efficient encirclement and capture of different types of targets.

[0237] The PPO-HSL of this invention decomposes the global value function into global task value and individual task value through a hierarchical consensus mechanism. This allows the system to independently optimize and reward the behavior of each agent (such as energy saving and efficient movement) while pursuing global capture success, thereby handling heterogeneous capability constraints more finely and efficiently.

[0238] The PPO-HSL of this invention introduces a sequential update paradigm and a task allocation module. The task allocation module (based on the maximum flow minimum cost model) first solves the high-level decision-making problem, providing explicit local goals for path planning in reinforcement learning. The sequential update paradigm explicitly models the time-series dependencies of decisions, using action information from the previous time step to guide learning in the current time step. This not only avoids policy conflicts that may arise from simultaneous updates but also significantly improves training efficiency and policy coordination.

[0239] This invention designs an uncertainty-guided exploration mechanism. By introducing random movement patterns (when there are no pursuers nearby) and explicit "reverse escape" behavior (when approached), the randomness and adversarial nature of the environment are artificially increased. This design forces the agent to learn a general approach to deal with various escape strategies, rather than just a specific pattern, thereby greatly improving the robustness and generalization ability of the strategy and avoiding local optima.

[0240] The PPO-HSL of this invention overcomes these shortcomings by introducing a series of innovations such as hierarchical consensus, sequential updates, uncertainty exploration, and multi-agent advantage estimation, thereby achieving more efficient and robust heterogeneous unmanned swarm cooperative capture.

[0241] like Figure 6 As shown, this embodiment of the invention provides a heterogeneous unmanned swarm cooperative capture device, which includes:

[0242] System modeling module 21 clarifies the heterogeneous unmanned swarm, environmental information, and constraints of the multi-agent collaborative encirclement system, and constructs the physical model of the multi-agent collaborative encirclement system; the heterogeneous unmanned swarm includes multiple agents and multiple targets, the agents include drones and unmanned vehicles, the environmental information includes the encirclement space and obstacle information, and the constraints include obstacle avoidance and collision avoidance constraints, encirclement success condition constraints, and speed and energy constraints.

[0243] The task allocation module 22 is used to model the multi-agent task allocation problem as a graph theory problem of "maximum flow - minimum cost" based on the physical model at the beginning of each decision cycle, construct a directed graph for task allocation, and determine the individual capture target corresponding to each agent.

[0244] Reinforcement learning module 23 is used to guide agents to learn the optimal capture strategy based on the PPO framework, with the individual objective being the maximum reward for each agent to complete its corresponding individual capture target, and the global objective being the maximum reward for the heterogeneous unmanned swarm to complete all capture targets.

[0245] Optionally, the system modeling module 21 is specifically configured to execute:

[0246] Based on the number of drones and their positions, the number of unmanned vehicles and their positions, and the number and positions of targets in the multi-agent cooperative encirclement system, a heterogeneous unmanned cluster is determined.

[0247] Environmental information is determined based on a preset three-dimensional enclosure space and the position and size information of randomly generated obstacles within the three-dimensional enclosure space;

[0248] Based on the preset obstacle area and the preset safe distances between agents and between agents and obstacles, obstacle avoidance and collision avoidance constraints are determined; the condition for successful capture is that the distance between the agent and the target is less than the preset radius; the constraints for speed and energy are that the agent's speed is greater than the target's speed and that the agent's energy consumption is linearly related to the travel distance.

[0249] Based on the heterogeneous unmanned swarm, environmental information, and constraints such as obstacle avoidance and collision avoidance, successful capture conditions, and speed and energy constraints, a three-dimensional model of the multi-agent cooperative capture system is constructed, resulting in the physical model of the multi-agent cooperative capture system.

[0250] Optionally, the task allocation module 22 is specifically configured to execute:

[0251] Within each decision cycle, based on a physical model, a directed graph for task allocation is constructed, with the start of the decision as the source node, each agent as an agent node, each target as a target node, and the end of the decision as the sink node. The edges from the source node to each agent node represent the agent's capability limit; the edges from each agent node to each target node represent the comprehensive cost including the distance between the agent and the target, the estimated energy consumption, and the heterogeneous type matching degree; and the edges from each target node to the sink node represent the minimum number of agents required to capture the target. The capability limit of an agent represents the maximum number of targets that agent can capture in a single operation.

[0252] With the goal of minimizing overall cost while satisfying constraints, the task allocation model corresponding to the directed graph of task allocation is solved to obtain the task allocation scheme; the task allocation scheme assigns a corresponding individual capture target to each agent.

[0253] Optionally, in reinforcement learning module 23, the PPO framework is configured to include Actor-Critic network pairs, a global experience pool, and a centralized Critic network.

[0254] There are multiple Actor-Critic network pairs, each corresponding to an agent. Each agent receives its state space and action space through the Actor network in the Actor-Critic network pair and predicts the agent's next action. The Critic network in the Actor-Critic network pair evaluates the expected reward of the agent executing the predicted next action.

[0255] There is a single global experience pool, used to receive and store data streams from all Actor-Critic network pairs of agents; the data streams include the agent's current state, current action, reward, and next state;

[0256] The centralized Critic network is a single entity used to evaluate the expected return of all agents working together; the centralized Critic network is also trained periodically by sampling data from a global experience pool.

[0257] Optionally, reinforcement learning module 23 is specifically configured to perform:

[0258] Each agent's Actor network receives the agent's state space and action space, and predicts the agent's next action based on the probability distribution of the output action in the state space using the Actor network's decision function.

[0259] After executing the predicted next action, the agent's state space is sent to the agent's Critic network, which evaluates the agent's individual objective function. The individual objective function reflects the expected reward for the agent to complete the corresponding individual capture objective.

[0260] The global objective function of the heterogeneous unmanned swarm is evaluated through a centralized Critic network. The global objective function is used to reflect the expected reward of the heterogeneous unmanned swarm completing all capture objectives.

[0261] The individual objective function of each agent is maximized as the individual objective function, while the global objective function is maximized as the global objective function, guiding the agents to learn the optimal trapping strategy.

[0262] Optionally, when guiding agents to learn the optimal trapping strategy with the individual objective function for each agent as the individual objective and the global objective function as the global objective, reinforcement learning module 23 is specifically configured to execute:

[0263] The value function of the global capture task is determined based on the sum of the global task value corresponding to the global objective and the individual task value corresponding to each individual objective.

[0264] The parameters of the PPO framework are updated in the direction of maximizing the value function of the global capture task, guiding the agent to learn the optimal capture strategy.

[0265] Optionally, when guiding agents to learn the optimal trapping strategy, reinforcement learning module 23 is also configured to: treat each agent's decision and learning behavior as a time series and update the parameters of the PPO framework using a serialized update paradigm.

[0266] The heterogeneous unmanned swarm cooperative capture device provided in this embodiment of the invention is used to execute the above-mentioned heterogeneous unmanned swarm cooperative capture method, and thus can achieve the same effect as the above-mentioned heterogeneous unmanned swarm cooperative capture method.

[0267] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0268] Thirdly, embodiments of the present invention also provide an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the heterogeneous unmanned swarm cooperative encirclement method in the embodiments of the present invention.

[0269] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the heterogeneous unmanned swarm cooperative encirclement method in embodiments of the present invention.

[0270] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0271] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A heterogeneous unmanned swarm cooperative encirclement method, characterized in that, include: The heterogeneous unmanned swarm, environmental information, and constraints of the multi-agent cooperative encirclement system are defined, and a physical model of the multi-agent cooperative encirclement system is constructed. The heterogeneous unmanned swarm includes multiple agents and multiple targets. The agents include drones and unmanned vehicles. The environmental information includes the encirclement space and obstacle information. The constraints include obstacle avoidance and collision avoidance constraints, encirclement success condition constraints, and speed and energy constraints. At the beginning of each decision cycle, based on the physical model, the multi-agent task allocation problem is modeled as a graph theory problem of "maximum flow - minimum cost", a directed graph of task allocation is constructed, and the individual capture target corresponding to each agent is determined; Based on the PPO framework, the individual objective is to maximize the reward for each agent to complete its corresponding individual capture target, while the global objective is to maximize the reward for the heterogeneous unmanned swarm to complete all capture targets, thus guiding the agents to learn the optimal capture strategy.

2. The heterogeneous unmanned swarm cooperative encirclement method according to claim 1, characterized in that, The process involves clarifying the heterogeneous unmanned swarm, environmental information, and constraints of the multi-agent cooperative trapping system, and constructing a physical model of the multi-agent cooperative trapping system, specifically including: Based on the number of drones and their positions, the number of unmanned vehicles and their positions, and the number of targets and their positions in the multi-agent cooperative capture system, a heterogeneous unmanned cluster is determined. Environmental information is determined based on a preset three-dimensional containment space and the position and size information of randomly generated obstacles within the three-dimensional containment space; Based on the preset obstacle area and the preset safe distances between agents and between agents and obstacles, obstacle avoidance and collision avoidance constraints are determined; the condition for successful capture is that the distance between the agent and the target is less than the preset radius; the constraints for speed and energy are that the agent's speed is greater than the target's speed and that the agent's energy consumption is linearly related to the travel distance. Based on the heterogeneous unmanned swarm, the environmental information, and the constraints of obstacle avoidance and collision avoidance, the constraints of successful capture conditions, and the constraints of speed and energy, a three-dimensional model of the multi-agent cooperative capture system is performed to obtain the physical model of the multi-agent cooperative capture system.

3. The heterogeneous unmanned swarm cooperative encirclement method according to claim 1, characterized in that, At the beginning of each decision cycle, based on the physical model, the multi-agent task allocation problem is modeled as a graph theory problem of "maximum flow - minimum cost," a directed graph for task allocation is constructed, and the individual capture target for each agent is determined, specifically including: Within each decision cycle, based on the physical model, a directed graph for task allocation is constructed, with the start of the decision as the source node, each agent as an agent node, each target as a target node, and the end of the decision as the sink node. The edges from the source node to each agent node represent the agent's maximum capability; the edges from each agent node to each target node represent the comprehensive cost including the distance between the agent and the target, the estimated energy consumption, and the heterogeneous type matching degree; and the edges from each target node to the sink node represent the minimum number of agents required to capture the target. The maximum capability of an agent represents the maximum number of targets that agent can capture at one time. With the goal of minimizing the overall cost while satisfying the constraints, the task allocation model corresponding to the directed graph of task allocation is solved to obtain the task allocation scheme; the task allocation scheme assigns a corresponding individual capture target to each agent.

4. The heterogeneous unmanned swarm cooperative encirclement method according to claim 1, characterized in that, The PPO framework includes Actor-Critic network pairs, a global experience pool, and a centralized Critic network. There are multiple Actor-Critic network pairs, each corresponding to one of the agents. Each agent receives its state space and action space through the Actor network in the Actor-Critic network pair and predicts its next action. The Critic network in the Actor-Critic network pair evaluates the expected reward of the agent executing the predicted next action. The global experience pool is a single entity used to receive and store data streams from all Actor-Critic network pairs of agents; the data streams include the agent's current state, current action, reward, and next state; The centralized Critic network is a single network used to evaluate the expected return of all agents working together; the centralized Critic network is also periodically trained by sampling data from the global experience pool.

5. The heterogeneous unmanned swarm cooperative encirclement method according to claim 4, characterized in that, The PPO framework guides the agents to learn the optimal encirclement strategy by maximizing the reward for each agent to complete its corresponding individual capture target, and by maximizing the reward for the heterogeneous unmanned swarm to complete all capture targets, which specifically includes: Each agent's Actor network receives the agent's state space and action space, and predicts the agent's next action based on the probability distribution of the state output action through the Actor network's decision function. After executing the predicted next action, the agent's state space is sent to the agent's Critic network, and the agent's individual objective function is evaluated through the Critic network; the individual objective function is used to reflect the expected reward of the agent in completing the corresponding individual capture objective; The global objective function of the heterogeneous unmanned swarm is evaluated through the centralized Critic network; the global objective function reflects the expected reward for the heterogeneous unmanned swarm to complete all captured targets. The agent is guided to learn the optimal trapping strategy by maximizing the individual objective function corresponding to each agent and maximizing the global objective function.

6. The heterogeneous unmanned swarm cooperative encirclement method according to claim 5, characterized in that, The process of guiding the agents to learn the optimal trapping strategy by maximizing the individual objective function corresponding to each agent as the individual objective and maximizing the global objective function as the global objective specifically includes: The value function of the global capture task is determined based on the sum of the global task value corresponding to the global target and the individual task value corresponding to each individual target; The parameters of the PPO framework are updated in the direction of maximizing the value function of the global capture task, guiding the agent to learn the optimal capture strategy.

7. The heterogeneous unmanned swarm cooperative encirclement method according to claim 6, characterized in that, When guiding the agent to learn the optimal trapping strategy, the decision-making and learning behavior of each agent are treated as a time series, and the parameter update method of the PPO framework adopts a serialized update paradigm.

8. A heterogeneous unmanned swarm collaborative trapping device, characterized in that, include: The system modeling module clarifies the heterogeneous unmanned swarm, environmental information, and constraints of the multi-agent collaborative encirclement system, and constructs the physical model of the multi-agent collaborative encirclement system; the heterogeneous unmanned swarm includes multiple agents and multiple targets, the agents include drones and unmanned vehicles, the environmental information includes encirclement space and obstacle information, and the constraints include obstacle avoidance and collision avoidance constraints, encirclement success condition constraints, and speed and energy constraints. The task allocation module is used to model the multi-agent task allocation problem as a "maximum flow-minimum cost" graph theory problem based on the physical model at the beginning of each decision cycle, construct a directed graph for task allocation, and determine the individual capture target corresponding to each agent. The reinforcement learning module is used to guide the agents to learn the optimal capture strategy based on the PPO framework, with the individual objective being the maximum reward for each agent to complete its corresponding individual capture target, and the global objective being the maximum reward for the heterogeneous unmanned swarm to complete all capture targets.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the heterogeneous unmanned swarm cooperative encirclement method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the heterogeneous unmanned swarm cooperative encirclement method as described in any one of claims 1-7.

Images