Heterogeneous unmanned aerial vehicle cooperative pursuit planning method and system under local observation

Abstract

The invention discloses a heterogeneous unmanned aerial vehicle cooperative pursuit planning method and system under local observation, and aims to solve the problem of low exploration efficiency caused by incomplete local observation information and sparse rewards of a heterogeneous unmanned aerial vehicle group. According to the method, a heterogeneous decision-making environment containing reconnaissance and pursuit unmanned aerial vehicles is constructed, the reconnaissance and pursuit unmanned aerial vehicles search targets and transmit information depending on wide-perception and high-maneuverability characteristics, and the latter executes work division of encirclement interception according to the information; and designing an independent state and decision-level action space based on local observation for the two types of unmanned aerial vehicles. The core is to introduce a random network distillation mechanism, generate exploration rewards through prediction errors of a trainable prediction network and a fixed random target network, and overcome the problem of reward sparseness; and a multi-agent double-delay depth deterministic strategy gradient algorithm framework is adopted to optimize the collaborative strategy. Through fusion of exploration and external task rewards, the unmanned aerial vehicle group can realize efficient and robust cooperative pursuit decision only depending on local information, and the task success rate and the intelligent level in a complex environment are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of multi-agent cooperative control and artificial intelligence, and particularly relates to a planning method and system for cooperative pursuit of a heterogeneous UAV group by deep reinforcement learning under local observation. BACKGROUND

[0002] In the multi-UAV cooperative pursuit task, adopting a heterogeneous formation composed of reconnaissance UAVs and pursuit UAVs is an effective strategy to improve task efficiency. However, this functional heterogeneity also brings a unique challenge of "information and incentive asymmetry". On the one hand, the reconnaissance UAVs responsible for large-scale search have the primary responsibility of finding targets in a vast unknown area. They face the problem of extremely sparse rewards, that is, they can hardly obtain any effective task feedback before finding the target, resulting in blind and inefficient exploration behavior. On the other hand, the pursuit UAVs are usually equipped with fewer detection devices to reduce cost and power consumption, and they rely on the intelligence provided by reconnaissance UAVs to perform the encirclement. The core challenge for them is how to efficiently cooperate and form the best encirclement formation, rather than autonomous exploration. Existing multi-agent reinforcement learning methods often adopt homogeneous exploration and reward mechanisms, and fail to design differentiated incentive strategies for UAVs with different roles, resulting in "ineffective exploration" of reconnaissance UAVs and "inefficient cooperation" of pursuit UAVs, which seriously restricts the overall combat effectiveness of the heterogeneous formation. SUMMARY

[0003] To solve the "information and incentive asymmetry" problem of the heterogeneous UAV group caused by different job divisions, the application proposes a planning method and system for cooperative pursuit of heterogeneous UAVs under local observation, aiming to solve the problems of incomplete information caused by local observation and low exploration efficiency caused by sparse rewards when the heterogeneous UAV group executes the pursuit task. The core of the application is to design differentiated incentive mechanisms for UAVs with different roles, tightly couple the two roles through the reinforcement learning framework, and achieve efficient cooperative exploration and pursuit.

[0004] To achieve the above purpose, the application adopts the following technical solutions:

[0005] A planning method for cooperative pursuit of heterogeneous UAVs under local observation, comprising the following steps:

[0006] S101, a cooperative pursuit decision environment containing heterogeneous UAVs, dynamic targets and static obstacles is constructed, and the initialization of the properties of each entity is completed;

[0007] S102, according to the heterogeneous characteristics of UAVs, independent state spaces based on local observation information are constructed for them respectively, and the action space of the decision level is defined;

[0008] S103, generating an exploration reward by calculating the output mean square error of the state observation of the trainable prediction network and the fixed target network through a random network distillation mechanism;

[0009] S104, fusing the exploration reward and the external environment reward to form a total reward signal, and optimizing a cooperative pursuit strategy through a multi-agent reinforcement learning algorithm of centralized training and distributed execution;

[0010] S105, using the trained strategy network, each unmanned aerial vehicle autonomously generates a decision according to its local observation to drive the heterogeneous group to execute the cooperative pursuit task.

[0011] In the embodiment of the application, the heterogeneous unmanned aerial vehicles of S101 include reconnaissance unmanned aerial vehicles and pursuit unmanned aerial vehicles, wherein the reconnaissance unmanned aerial vehicles have a larger sensing range and a faster maximum flight speed than the pursuit unmanned aerial vehicles; during task execution, the reconnaissance unmanned aerial vehicles periodically broadcast the latest state information of the observed target, including position and speed, to all pursuit unmanned aerial vehicles. The dynamic target is the pursuit object, and its position, speed, etc. can be dynamically changed, while the static obstacle is a fixed object, which is used to simulate the obstacle in the actual environment. In this step, the application clearly defines the functional division of the two types of unmanned aerial vehicles: the reconnaissance unmanned aerial vehicles are responsible for searching, discovering targets and delivering key information by using their wide sensing and high mobility characteristics; the pursuit unmanned aerial vehicles execute the encirclement and interception according to the received information.

[0012] In the embodiment of the application, S102, the state space of the reconnaissance unmanned aerial vehicle includes: the position and speed of itself, the relative position and speed of the target within the sensing range, the obstacle information within the sensing range and the relative position of other agents; wherein the agents include all unmanned aerial vehicles and dynamic targets; the obstacle information mainly includes its position information.

[0013] The state space of the pursuit unmanned aerial vehicle includes: the position and speed of itself, the target information received from the reconnaissance unmanned aerial vehicle, the obstacle information within the sensing range and the relative position of other agents.

[0014] The action space of the reconnaissance unmanned aerial vehicle and the pursuit unmanned aerial vehicle is defined as the expected speed vector at the next moment, which is a continuous speed vector and is limited by the maximum speed.

[0015] In the embodiments of the present application, the S103 aims to design an independent state space and a decision level action space based on the local observation of different types of UAVs, and the core method is to introduce a random network distillation mechanism to generate an exploration reward through the prediction error between a trainable prediction network and a fixed random target network, thereby effectively driving the UAV to explore novel states and overcoming the problem of sparse external environment reward. Among them, the prediction network is trained by gradient descent method, and the optimization goal is to minimize the mean square error between the output feature vector of the target network and the output vector of the prediction network, and the network parameters of the target network remain fixed during the entire training process.

[0016] Specifically, the prediction network is trained by minimizing the loss function , wherein D is a set of historical observation data, is the output vector of the prediction network, f(o; φ) is the output feature vector of the target network, o is the state, θ is the parameter of the prediction network, which is updated by training, φ is the parameter of the target network, which is fixed and unchanged.

[0017] In the embodiments of the present application, the S104, for the reconnaissance UAV, the total reward is the fusion of the exploration reward and the external environment reward, and for the pursuit UAV, the total reward is the external environment reward; by fusing the exploration reward and the external task reward, the UAV group realizes efficient and robust cooperative pursuit decision under the condition of relying only on local information. The external environment reward includes:

[0018] (1) a distance reward guiding the pursuit UAV to move towards the target;

[0019] (2) a team cooperation reward encouraging the pursuit UAV to form an effective encirclement;

[0020] (3) a sparse positive reward given when the target is successfully captured;

[0021] (4) a negative reward given when collision or task timeout occurs.

[0022] In the embodiment of the application, the algorithm used in S104 is a multi-agent double-delay deep deterministic policy gradient (MATD3) algorithm, which optimizes the cooperative strategy in a centralized training and distributed execution manner. The centralized training process thereof includes: using multiple centralized critic networks to evaluate the value of joint actions, the critic network obtains the joint state and joint action information of all agents during training, and is updated by minimizing the time difference error; using a distributed actor network to generate decision actions for each agent, each actor network only depends on the local observation of the corresponding agent, and is updated by policy gradient to maximize the evaluation value of the critic network; after training converges, only the lightweight actor network of each UAV is retained and deployed; in the task execution stage, at time t, each UAV obtains its own local observation and inputs it into the actor network carried thereon to directly generate a decision action.

[0023] Further, in the training process, the historical data generated by the interaction of all agents and the environment are stored in a shared experience replay pool, and data are sampled from the pool to update the actor network and the critic network, and the historical data include state, action, reward and next state.

[0024] According to the above process, the application focuses on decision layer planning, and can significantly improve the task success rate and intelligent level of the heterogeneous UAV system in a complex environment.

[0025] The application also provides a heterogeneous UAV cooperative pursuit planning system under local observation, which comprises:

[0026] An environment initialization module configured to perform all preparations before system training. First, a simulation decision environment containing a reconnaissance UAV, a pursuit UAV, a dynamic target and static obstacles is constructed, and the physical properties (such as maximum speed, acceleration) of each entity in the environment, the perception parameters (such as detection radius) and the initial distribution area are set. Secondly, this module is responsible for initializing all neural networks required for multi-agent reinforcement learning, specifically including: a random target network and a prediction network for generating exploration rewards, and an independent actor network for each UAV and a centralized critic network shared by the system for cooperative strategy optimization. The network parameters are set by using a standard initialization method (such as Xavier initialization) to ensure the stability at the beginning of training.

[0027] Learning training module: configured to implement the core of the cooperative strategy optimization process offline, is the source of system intelligence. This module contains an exploration reward generation unit and a strategy optimization unit, whose internal logic is tightly coupled: first, the exploration reward generation unit converts the novel state observations collected by the reconnaissance unmanned aerial vehicle into a numerical exploration reward in real time according to the random network distillation (RND) mechanism. Subsequently, the strategy optimization unit fuses this exploration reward with the external environmental reward based on the task goal (such as approaching the target or forming an encirclement) to form the final reward signal. This unit uses the multi-agent double-delay deep deterministic policy gradient (MATD3) algorithm based on centralized training and distributed execution to train the cooperative pursuit strategy offline. Specifically, data is sampled from a shared experience replay pool, and the critic network and the actor network of all unmanned aerial vehicles are updated centrally. The critic network evaluates the value of the joint action using global information to guide the optimization of each actor network until the overall cooperative strategy converges.

[0028] Task execution module: configured to load and deploy the trained strategy model in actual applications, execute the trained strategy, receive real-time local observation information from each unmanned aerial vehicle, and output decision instructions to drive the heterogeneous unmanned aerial vehicle group to perform the cooperative pursuit task, achieving efficient online decision-making. This module deploys the trained, lightweight actor network to each unmanned aerial vehicle. During the task execution phase, the system operates in a completely decentralized manner. Each unmanned aerial vehicle only relies on its own local observation data (including sensor information and intelligence received from friendly aircraft) to directly output control actions through its onboard actor network. This process does not involve complex online learning or network parameter updates, ensuring real-time decision-making. Reconnaissance unmanned aerial vehicles autonomously perform exploration and broadcast target information to pursuit unmanned aerial vehicles, which then perform cooperative encirclement based on the received intelligence and their own strategy, achieving efficient distributed cooperative pursuit.

[0029] The data interaction between the learning training module and the task execution module of the present application follows the principle of centralized training and distributed execution: during training, the strategy optimization unit has access to global information to guide learning; during execution, each decision unit in the task execution module of the unmanned aerial vehicle only relies on its local observation information.

[0030] Compared with the prior art, the present application has the following advantages:

[0031] The present application designs an exploration reward based on random network distillation for reconnaissance unmanned aerial vehicles that undertake exploration tasks. By converting the "novelty" of the state into a dense reward signal, it effectively addresses the problem of low exploration efficiency caused by sparse external environmental rewards, providing effective target guidance for subsequent cooperative pursuit.

[0032] The application adopts a differentiated reward mechanism, so that the reconnaissance and pursuit unmanned aerial vehicles perform their respective functions.

[0033] The application provides dense reward signals for reconnaissance unmanned aerial vehicles responsible for exploration, avoids the training stagnation of a reinforcement learning algorithm in a sparse reward environment due to lack of effective feedback, and thus accelerates the convergence process of an effective cooperative strategy.

[0034] The application adopts a centralized training and distributed execution architecture, allows a critic network to learn a more optimal cooperative strategy by using global information in the training stage, and allows each agent to make an independent decision in the execution stage, so that the dependence on high-bandwidth real-time communication is reduced, and the robustness and practical deployment feasibility of the system are enhanced.

[0035] In order to make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are used for detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which form a part of the present application, are used to provide a further understanding of the application, and the schematic embodiments of the application and the description thereof are used to explain the application, and do not constitute an improper limitation on the application. In the drawings:

[0037] Figure 1 Flow chart of the heterogeneous multi-unmanned aerial vehicle cooperative pursuit method.

[0038] Figure 2 Schematic view of the multi-unmanned aerial vehicle cooperative pursuit simulation environment.

[0039] Figure 3 Schematic view of the random distillation network structure.

[0040] Figure 4 Reward convergence curve of the heterogeneous unmanned aerial vehicle cooperative pursuit. DETAILED DESCRIPTION

[0041] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] In order to make the person skilled in the art better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0043] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0044] Figure 1 A flow chart of a local observation heterogeneous unmanned aerial vehicle cooperative pursuit planning method according to the present application. The flow chart clearly shows the complete process from environment initialization to final task execution, as shown in Figure 1 The method comprises the following steps:

[0045] S101, a cooperative pursuit decision environment containing heterogeneous unmanned aerial vehicles, dynamic targets and obstacles is constructed, and the initialization of the properties of each entity is completed.

[0046] Specifically, a heterogeneous unmanned aerial vehicle flight continuous decision environment is constructed, which contains a heterogeneous formation composed of one Reconnaissance unmanned aerial vehicle and Two pursuit unmanned aerial vehicles, one dynamic target and several static obstacles. The physical properties of the reconnaissance unmanned aerial vehicle are set to have a larger sensing range R r And a higher maximum flight speed v max，r , to adapt to a wide range of search tasks; while the sensing range R p And the maximum speed v max，p of the pursuit unmanned aerial vehicle are relatively small, that is, R r >R p And v max，r >v max，pThis setting simulates a scenario with less detection equipment to reduce cost, which mainly relies on intelligence for pursuit. Dynamic targets have independent motion strategies, and random obstacles are distributed in the environment to test the obstacle avoidance ability of the algorithm. The motion of all agents (including UAVs and dynamic targets) follows the kinematic model under discrete time. At time t, the position p i,t of agent i is updated according to its speed v i,t , and the kinematic equation can be simplified as: p i,t+1 = p i,t + v i,t ·Δt, where Δt is the simulation step. The speed v i,t of the UAV is directly determined by the action instruction output by its policy network. At the beginning of each round of training, all entities are randomly placed in the effective initial position in the environment.

[0047] S102, according to the heterogeneous characteristics of the UAV, an independent state space based on local observation information is constructed for each UAV, and the action space of the decision level is defined.

[0048] Specifically, according to the heterogeneous characteristics of the UAV, the observation state and action of the two types of UAVs are defined respectively. For any UAV i, its local observation o i is an instance of its state space S i . The state s j of the reconnaissance UAV j is composed of its own motion information (position p j , speed v j ) and relative information within the perception range, which can be expressed as: s j = {p j , v j , p E，rel， v E，rel , {p k，rel} k≠j , {p o，rel} o}, where E represents the target, k represents other UAVs, o represents obstacles, p E,rel represents the relative distance to the target, v E，rel represents the relative speed to the target, p k,rel represents the relative distance to the UAV k, and p o，rel represents the relative distance to the obstacle o. The state s k of the pursuit UAV k additionally contains the target intelligence I E = {p E，info , v E，info} received from the reconnaissance UAV, which is expressed as: s k = {p k , v k , I E , {pj，rel} j≠k , {p o，rel} o}, wherein p k , v k represents the position and velocity of the pursuit drone k, p E，info represents the absolute position information of the target received from the scout drone, p v，info represents the absolute velocity information of the target received from the scout drone, p j，rel represents the relative distance to the drone j, p o，rel represents the relative distance to the obstacle o.

[0049] The action space A i of all drones is defined as a continuous velocity vector a i , whose norm is limited by the maximum speed, i.e.

[0050] S103, using the random network distillation mechanism, the exploration reward is generated by calculating the mean square error of the output of the state observation of the trainable prediction network and the fixed target network.

[0051] Specifically, this step is mainly aimed at the scout drone to solve the problem of sparse rewards when searching for targets in a wide area. The system establishes two neural networks with the same network structure: a target network with parameters φ and fixed and a prediction network with parameters θ and can be updated by training At each time step t, for the state observation o t of the scout drone, the exploration reward is defined as the mean square error (MSE) between the output feature vectors of the two networks, and the calculation formula is as follows: The more novel a state is, the less accurate the prediction of the prediction network is, and the greater the generated, thereby encouraging the scout drone to actively explore unknown areas. The prediction network is trained by minimizing the loss function , where D is a set of historical observation data.

[0052] S104, the total reward signal is composed of the exploration reward and the external environment reward, and the multi-agent reinforcement learning algorithm with centralized training and distributed execution is used to optimize the cooperative pursuit strategy.

[0053] Specifically, the system uses the multi-agent double-delay deep deterministic policy gradient (MATD3) algorithm in the framework of centralized training and distributed execution. First, the exploration reward is fused with the external environment reward r ext . For the scout drone j, the total reward is For the chaser drone k, since it does not perform the active exploration task, its total reward is During training, the centralized critic network Q(s, a) (where s and a are the joint state and action of all agents) is updated by minimizing the temporal difference error. Its loss function is:

[0054]

[0055] where the target value y = R + γQ'(s', a'). The individual actor network μ i of each drone i is then updated by policy gradient to maximize the critic network's evaluation. i

[0056] where the extrinsic environmental reward is oriented towards the core goal of the pursuit task: a positive reward r capture is given as environmental feedback when the chaser drone successfully achieves the interception capture event of the target; a phased extrinsic reward r discover is given when the scout drone first discovers the target and uploads accurate position information; a negative reward r collision is given when a drone collides with another drone or an obstacle. If the above task nodes of interception capture or target discovery upload are not achieved, the extrinsic environmental reward is zero. The overall extrinsic environmental reward is r ext = r capture + r discover + r collision .

[0057] S105, using the trained policy network, each drone autonomously generates decisions based on its own local observations to drive the heterogeneous group to perform the cooperative pursuit task.

[0058] Specifically, after the policy network training converges, only the lightweight actor network μ i of each drone is retained and deployed. In the task execution phase, the system switches to a completely distributed mode. At time t, each drone i obtains its own local observation o i，t and inputs it into the actor network carried, directly generating the decision action a i，t , a i，t = μ i (o i，t ; φ i ), where φ i is the trained actor network parameter. The whole process is efficient and robust, without the need for a central server to perform calculations, and no longer needs to calculate exploration rewards, ensuring the feasibility and real-time performance of the method in real-world scenarios.

[0059] Figure 2 ​This is a schematic diagram of the heterogeneous UAV collaborative pursuit simulation environment described in this invention. Figure 2 As shown, this example exists in a three-dimensional space of size L×L×L, containing three reconnaissance drones (red spheres), three capture drones (blue spheres), a dynamic target (green sphere), and multiple cylindrical obstacles (gray). To reflect heterogeneous division of labor, the key parameters of the two types of drones are set differently: the reconnaissance drones have a larger perception radius R. scout =0.34, for performing wide-area exploration tasks; while the detection radius of the capture drone is smaller, R. pursuit =0.16. Regarding speed, the maximum speed for the pursuit drone is set to v. max，p =0.05, while the maximum speed v of the reconnaissance drone max，s If the value is 0.3, it will be configured to be greater than this value, i.e., v max，s >v max，p This is to ensure its exploration advantage. The target's capture determination radius is R. cap =0.05, the target is considered successfully captured when at least two pursuit drones simultaneously enter this radius. All entities are treated as ideal rigid bodies, their motion follows a discrete-time kinematic model, and collisions with environmental boundaries and obstacles are considered. This environment accurately simulates a complex scenario where drones of different roles cooperate to perform exploration and pursuit tasks under local perception conditions.

[0060] Figure 3An Actor-Critic network structure diagram used in the MATD3 algorithm of the present application. The algorithm uses a "centralized training, distributed execution" framework, providing an efficient solution for the cooperative strategy learning of heterogeneous UAV groups. Specifically, in the distributed execution phase, each UAV, whether reconnaissance or pursuit, carries an independent execution (Actor) network. The network only receives the local observation information of the UAV itself as input and directly outputs its decision action. This design ensures that in actual tasks, the UAV formation can run in a completely decentralized manner, with high response speed and strong robustness. In the centralized training phase, the system introduces a critic (Critic) network for global value evaluation. In order to improve the stability and efficiency of training, MATD3 uses a double critic mechanism, that is, two independent critic networks are used. When training is updated, these two critic networks can access the joint state and joint action information of all UAVs, so they can accurately evaluate the long-term value of the entire formation taking joint actions in a particular state. The global perspective of the critic network enables the algorithm to learn the complex coordination relationship between UAVs, such as when reconnaissance UAVs explore and when they share intelligence to maximize the overall benefits of the team. The gradient signal generated after the critic network evaluation will be used to guide the optimization of each independent execution network. In addition, the algorithm also introduces a target network (Target Networks) and a delayed update (Delayed Updates) mechanism, that is, a parameter smoothing update target network copy is created for each execution and critic network, and the update frequency of the execution network is reduced, which effectively alleviates the problem of overestimation and instability in the training process.

[0061] Figure 4The figure shows the convergence curve of the team average reward in the heterogeneous UAV cooperative pursuit training process. The single curve obtained by adding and averaging the rewards of all reconnaissance and pursuit UAVs is shown, with the horizontal axis representing the number of rounds of training and the vertical axis representing the team average cumulative reward value obtained in that round. The trend of the curve clearly reflects the complete learning process of the heterogeneous formation from disorder to efficient cooperation. In the early stage of training, since the reconnaissance UAV has not learned an efficient exploration strategy, the target is difficult to find, resulting in very sparse external rewards oriented to the pursuit task, so the team average reward remains at a low level, and its value is mainly composed of the initial exploration reward of the reconnaissance UAV. With the progress of training, under the effective driving of exploration reward, the reconnaissance UAV gradually masters the method of efficient search for the target. When the target is stably discovered and reported to the pursuit UAV, the entire team's cooperative link is opened, and the pursuit UAV can execute effective pursuit and encirclement according to the intelligence, thereby obtaining a large amount of external environmental reward. This stage corresponds to the rapid climbing period of the reward curve in the figure, marking the initial formation of the "exploration-pursuit" cooperative strategy. In the later stage of training, the strategies of reconnaissance and pursuit UAVs tend to mature, and they can stably and efficiently complete the cooperative pursuit task. At this time, the team average reward curve tends to be stable and converges to a higher value interval, indicating that the differentiated incentive and cooperative training method proposed by the present application is effective, and can enable the heterogeneous UAV group to learn a robust cooperative strategy. According to the planning method and system provided by the present application, the advantages of heterogeneous UAVs can be effectively utilized to solve the exploration and cooperation problems in cooperative pursuit, and to provide strong technical support for the application of multi-agent systems in complex environments.

[0062] The above detailed description further explains the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included within the scope of protection of the present application.

Claims

1. A method for cooperative pursuit planning of heterogeneous unmanned aerial vehicles (UAVs) under partial observation, characterized in that, The method comprises the following steps: S101, constructing a cooperative pursuit decision environment containing heterogeneous unmanned aerial vehicles, dynamic targets and static obstacles, and completing initialization of properties of each entity; S102, constructing an independent state space based on local observation information for each unmanned aerial vehicle according to the heterogeneous characteristics of the unmanned aerial vehicle, and defining an action space of the decision level; S103, using a random network distillation mechanism to generate an exploration reward by calculating the output mean square error of a trainable prediction network and a fixed target network on state observation; S104, fusing the exploration reward and an external environment reward to form a total reward signal, and optimizing a cooperative pursuit strategy through a multi-agent reinforcement learning algorithm of centralized training and distributed execution; S105, using the trained strategy network, each unmanned aerial vehicle generates a decision according to its own local observation to drive the heterogeneous group to execute the cooperative pursuit task.

2. The method of claim 1, wherein, In the S101, the heterogeneous unmanned aerial vehicles include a reconnaissance unmanned aerial vehicle and a pursuit unmanned aerial vehicle, wherein the reconnaissance unmanned aerial vehicle has a larger sensing range and a faster maximum flight speed than the pursuit unmanned aerial vehicle; during task execution, the reconnaissance unmanned aerial vehicle periodically broadcasts the latest state information of the observed target to all pursuit unmanned aerial vehicles, including position and speed.

3. The method of claim 1, wherein, In the S102, the state space of the reconnaissance unmanned aerial vehicle includes the position and speed of the reconnaissance unmanned aerial vehicle, the relative position and speed of the target within the sensing range, the obstacle information within the sensing range and the relative position of other agents; the agents include all unmanned aerial vehicles and dynamic targets; The state space of the pursuit unmanned aerial vehicle includes the position and speed of the pursuit unmanned aerial vehicle, the target information received from the reconnaissance unmanned aerial vehicle, the obstacle information within the sensing range and the relative position of other agents; The action space of the reconnaissance unmanned aerial vehicle and the pursuit unmanned aerial vehicle is defined as the expected speed vector at the next moment. In the S103, the prediction network is trained by gradient descent method, and the optimization target is to minimize the mean square error between the output feature vector of the target network, and the network parameters of the target network remain fixed during the entire training process.

4. The method of claim 1, wherein, In the S104, for the reconnaissance unmanned aerial vehicle, the total reward is the fusion of the exploration reward and the external environment reward, and for the pursuit unmanned aerial vehicle, the total reward is the external environment reward; 5. The method of claim 1 or 4, wherein, The prediction network is trained by minimizing a loss function where D is a set of historical observation data, is an output vector of the prediction network, f(o; φ) is an output feature vector of the target network, o is a state, θ is a parameter of the prediction network, updated by training, and φ is a parameter of the target network, fixed.

6. The method of claim 1, wherein, The external environment reward includes: (1) a distance reward guiding the pursuit unmanned aerial vehicle to move towards the target; (2) a team cooperation reward encouraging the pursuit unmanned aerial vehicle to form an effective encirclement; (3) a sparse positive reward given when the target is successfully captured; (4) a negative reward given when a collision occurs or the task is timed out. ​ 7. The method of claim 1, wherein, The algorithm used in S104 is a multi-agent double-delay deep deterministic policy gradient algorithm, and its centralized training process includes: using multiple centralized critic networks to evaluate the value of joint actions, the critic network obtains the joint state and joint action information of all agents during training, and is updated by minimizing the time difference error; using a distributed executor network to generate decision actions for each agent, each executor network only depends on the local observation of the corresponding agent, and is updated by policy gradient to maximize the evaluation value of the critic network; after training converges, only the lightweight executor network of each UAV is retained and deployed; in the task execution stage, at time t, each UAV obtains its own local observation and inputs it into the installed executor network to directly generate a decision action.

8. The method of claim 7, wherein, In the training process, the historical data generated by the interaction of all agents and the environment is stored in a shared experience replay pool, and data is sampled from the pool to update the executor network and the critic network, the historical data includes state, action, reward and next state.

9. A heterogeneous unmanned aerial vehicle cooperative pursuit planning system under partial observation, characterized in that, Comprise: An environment initialization module for building a decision-making environment containing reconnaissance UAVs, pursuit UAVs, dynamic targets and obstacles, and setting the physical and perception parameters of various entities; A learning and training module, which includes an exploration reward generation unit and a policy optimization unit; the exploration reward generation unit is used to generate exploration rewards in real time according to a random network distillation mechanism; The policy optimization unit is used to fuse exploration rewards and external environment rewards, and uses a multi-agent reinforcement learning algorithm MATD3 based on centralized training and distributed execution to train the cooperative pursuit strategy offline; A task execution module for loading and deploying the trained strategy model, receiving real-time local observation information of each UAV, and outputting decision instructions to drive the heterogeneous UAV group to perform cooperative pursuit tasks.

10. The heterogeneous UAV cooperative pursuit planning system under partial observation of claim 9, wherein, The data interaction between the learning and training module and the task execution module follows the principle of centralized training and distributed execution: during training, the policy optimization unit can access global information to guide learning; during execution, each UAV decision unit in the task execution module only relies on its local observation information.

Images