UAV capture resource allocation method based on adversarial generative imitation learning in pursuit scenarios

Abstract

The drone roundup resource allocation method based on adversarial generative imitation learning in pursuit scenarios aims to determine the pursuit targets of each pursuer drone in a many-to-many pursuit scenario, thereby solving the roundup resource allocation problem. The method first establishes a system model for the many-to-many drone pursuit scenario and performs symbolic declaration. Then, the concept of Apollonius circle is introduced to establish a potential game model for drone roundup resource allocation. Then, a linear programming-based solution algorithm for the optimal roundup resource allocation scheme is proposed. Finally, a drone roundup resource allocation algorithm based on adversarial generative imitation learning is proposed. The algorithm designed by the present invention is not affected by the number and initial state of drones on both sides of the pursuit. It can use a linear programming algorithm to find the Nash equilibrium under the potential game, and adaptively learn the optimal roundup resource allocation scheme based on adversarial generative imitation learning, ultimately realizing real-time dynamic allocation and scheduling of roundup resources in a many-to-many drone pursuit scenario.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent decision-making of multi-agent systems, and specifically provides a method for allocating drone capture resources based on adversarial generative imitation learning in pursuit scenarios. Background Art

[0002] The multi-to-multi drone pursuit scenario is a classic multi-agent system decision-making scenario. Drones in this scenario are divided into two categories: escaper drones and pursuer drones. The pursuer drones' goal is to capture all escaper drones as quickly as possible through cooperative means, based on a reasonable resource allocation scheme. Meanwhile, the escaper drones need to escape the pursuer drones and prolong their survival as much as possible. Multi-to-multi drone pursuit has attracted widespread attention due to its broad application scenarios. However, current research has largely focused on the decision-making and action generation between the pursuer and the fugitive in a competitive relationship, while the resource allocation problem among multiple pursuer drones has been gradually neglected. The few algorithms or schemes that have been explored for drone capture resource allocation often rely on the straight-line distance between the pursuers. These approaches lack a weak theoretical foundation, a single starting point, and a lack of universal applicability. In recent years, with the continuous vigorous development of deep learning and reinforcement learning, solving problems in multi-to-multi drone pursuit scenarios has ushered in new opportunities. However, the emergent strategies directly fitted by neural network models also lack theoretical basis and interpretability, which poses new challenges to solving the problem of resource allocation in multi-chasing and multi-escape scenarios.

[0003] Compare with patent CN116796843A_A drone multi-to-multi pursuit and escape game method based on PSO-M3DDPG and patent CN118276438A_A multi-agent pursuit and escape optimal strategy method based on threat degree reinforcement learning algorithm

[0004] 1. Algorithm used in CN116796843A

[0005] Patent CN116796843A proposes a PSO-M3DDPG-based optimization solution for multi-to-multi drone pursuit and escape games. The method combines the particle swarm optimization algorithm (PSO) with the minimax deep deterministic policy gradient algorithm (M3DDPG) to improve the adversarial learning performance of multi-agent systems. The specific steps include:

[0006] 1. Use the PSO algorithm to optimize the initial weights of the deep neural network to improve the rationality of initialization.

[0007] 2. Based on the M3DDPG algorithm, a multi-agent pursuit-escape game model is constructed to optimize the strategic choices of both the pursuit and escape parties.

[0008] 3. Conduct simulation training for different escape strategies, such as linear motion, simple curve motion, and intelligent escape, to optimize the pursuit path of the drone swarm.

[0009] 4. Utilize the design of reward functions to perform real-time strategy optimization in dimensions such as distance, direction, and mission success.

[0010] This method combines multi-agent learning with PSO to optimize drone pursuit paths. However, when allocating resources to capture drones, it only considers the impact of a single pursuit drone on the escaping drone, failing to account for the collaborative relationships between multiple pursuit drones within the pursuit team. Furthermore, in terms of strategy design for escaping drones, this method only provides three fixed escape strategies, making the pursuit relatively easy.

[0011] 2. Algorithm used in CN118276438A

[0012] Patent CN118276438A proposes a multi-agent pursuit and escape optimal strategy method based on a threat level reinforcement learning algorithm. This method uses a particle swarm algorithm to pre-assign threat levels and builds a multi-agent deep reinforcement learning model to achieve an optimal capture strategy for drone swarms. Specific steps include:

[0013] 1. Build a threat pre-allocation model based on the particle swarm algorithm to optimize the UAV cluster mission and trajectory.

[0014] 2. Construct a threat level objective function and optimize the solution by comprehensively considering the threat level, track distance and mission cost.

[0015] 3. Use a multi-agent deep reinforcement learning model to conduct centralized training to achieve distributed capture missions for drone swarms.

[0016] 4. Through the threat matching algorithm, the capture strategy of the drone cluster is output in real time.

[0017] This method optimizes the drone capture path and strategy by combining particle swarm optimization with multi-agent deep reinforcement learning. However, the particle swarm algorithm used in this method cannot guarantee the global optimality of the results, and the various hyperparameter settings also lack theoretical explanations. In addition, given the

[0018] The capture target is static in the environment, so in a large-scale dynamic environment, the model convergence efficiency still faces certain challenges.

[0019] 3. Method of the present invention

[0020] This paper proposes a method for allocating drone capture resources based on adversarial generative imitation learning. This method uses the area of ​​the Apollonius circle to measure the competitive relationship between the pursuer and the fugitive, employs a linear programming algorithm to generate the optimal capture resource allocation plan, and proposes a multi-drone capture resource allocation method based on adversarial generative imitation learning, achieving a reasonable allocation of capture resources in a multi-drone capture scenario. The core innovations of this method include:

[0021] 1. A potential game for allocating drone capture resources based on Apollonius circles. This paper uses the intersection area of ​​multiple Apollonius circles to measure the effectiveness of centralized pursuit drones on escaping drones. Based on this, a potential game for allocating drone capture resources is constructed, providing a theoretical basis for the existence of optimal capture resource allocation schemes.

[0022] 2. Solving the optimal resource allocation method for drone capture based on linear programming. This method transforms the resource allocation problem in a multi-drone pursuit scenario into a linear programming problem, solving the optimal solution while providing expert experience for subsequent adversarial generative imitation learning.

[0023] 3. A drone capture resource allocation method based on adversarial generative imitation learning. By using a generative adversarial network model framework, the neural network output strategy is guided to align with the expert strategy, achieving dynamic, real-time allocation of capture resources in a multi-drone pursuit scenario.

[0024] 4. Advantages Comparison

[0025] 1. Theoretical Interpretability: This paper constructs a potential game for allocating resources to capture drones and seeks the global optimal solution for their pursuit. Compared with the threat pre-allocation model based on the particle swarm algorithm proposed in CN118276438A, this paper has a more solid theoretical foundation and is more interpretable.

[0026] 2. Reasonableness of the Solution: This invention no longer uses the straight-line distance between the two drones as a metric to measure the competitive relationship between the pursuing and fleeing drones. Instead, it proposes describing the generation space of the fleeing drone using the area of ​​a convex polygon formed by the Apollonius circle. Compared to CN116796843A, this solution fully considers the mutual influence of multiple pursuing drones within the pursuit team when solving for the global optimal solution, resulting in a more reasonable and complete solution.

[0027] 3. Innovation: This solution proposes a drone capture resource allocation method based on adversarial generative imitation learning. This method learns the optimal capture resource allocation method by imitating expert strategies. Compared with the particle swarm algorithm-based pre-training or pre-allocation methods in CN118276438A and CN116796843A, this solution is more innovative.

[0028] In summary, the drone capture resource allocation method based on adversarial generative imitation learning proposed in the present invention is superior to CN118276438A and CN116796843A in terms of theoretical interpretability, scheme rationality, and scheme innovation, and can effectively solve the capture resource allocation problem in multi-drone pursuit scenarios. Summary of the Invention

[0029] To solve the above technical problems, the present invention proposes a drone capture resource allocation method based on adversarial generative imitation learning in a pursuit scenario. During the drone pursuit process, the algorithm can dynamically adjust the pursuit targets of each pursuer drone in real time according to the learning results of GAIL and the global state, thereby helping the pursuer drone to complete the pursuit mission faster and better.

[0030] To achieve the above object, the technical solution adopted by the present invention is:

[0031] The drone capture resource allocation method based on adversarial generative imitation learning in a pursuit scenario includes the following steps:

[0032] Step 1: Based on the many-to-many drone pursuit scenario, a system model is constructed and symbolic declarations are made. By defining an absolute rectangular coordinate system and a relative polar coordinate system, the positional relationship between the pursuing and fleeing drones is described, and the action space of both parties is defined, thereby introducing the resource allocation matrix and the concept of pursuit team set.

[0033] Step 2: Introduce the concept of Apollonius circle, construct a potential game of drone roundup resource allocation based on Apollonius circle, and prove that the roundup resource allocation has a Nash equilibrium solution, that is, the existence of the optimal roundup resource allocation scheme;

[0034] Step 3: Based on the theoretical foundation of step 2, a linear programming algorithm is proposed to solve the optimal roundup resource allocation solution. Based on the theory of step 2, this algorithm transforms the UAV roundup resource allocation problem into a multi-constrained optimization problem and finds the optimal solution for the roundup resource allocation through linear programming;

[0035] Step 4: Based on the optimal resource allocation solution algorithm designed in step 3, the optimal solution for drone capture resource allocation is used as the expert solution of the discriminator network, and a loss function is constructed to train the discriminator network.

[0036] Step 5: To determine the unique pursuit target for each pursuer drone based on the output value of the generator network, a gumbel-softmax reparameterization operation is used. This operation can sample the pursuit target value that conforms to the discrete probability distribution while ensuring differentiability.

[0037] Step 6: Based on the data generated in Steps 4 and 5, use a deep reinforcement learning algorithm to train the generator network assigned to each pursuer drone, so that the output value of each generator network gradually fits the optimal capture resource allocation plan.

[0038] As a further improvement of the present invention, in step 1, in a multi-to-multi drone pursuit scenario, drones are divided into two categories: pursuer drones and escaper drones. Suppose the number of escaper drones is n. e , the number of pursuer drones is n p , n p ≥n e , the escaped drone set E={1,2,...,n e}, the set of pursuer drones is P = {1,2,...,n p}, in the absolute rectangular coordinate system of space, the coordinates of the i-th pursuer drone are p i =(x i ,y i ), i∈P, the coordinates of the j-th escaper drone are p′ j =(x′ j ,y′ j ), j∈E, the coordinates of any pursuer drone are used as the origin to construct a relative polar coordinate system. The positions of each escaper drone in the relative polar coordinate system can be expressed as d ij ,ω ij express:

[0039]

[0040] To describe the relative position relationship between the escaper drone and the pursuer drone in each polar coordinate, the relative distance matrix and relative deflection angle matrix can be constructed:

[0041]

[0042] For the jth escaper drone, its action space contains the velocity v′ j and heading angle θ j ′, where v′ j ∈[0,v′ max ],θ j ′∈[0,2π],v′ max is the maximum speed of the escaper drone, and then:

[0043]

[0044] For the pursuer drone, similarly, its action space includes the velocity v i and heading angle θ i , v i ∈[0,vmax ],θ i ∈[0,2π], in addition, since the number of escaper drones n e >1, so for the pursuer drone, there are multiple options for pursuit targets, so add pursuit target t i To its action space, to describe the pursuit target t i , construct the identity matrix I and decompose its row vectors:

[0045]

[0046] Then construct the pursuit target set T={I1,I2,...,I ne},for I j is a row vector in the form of a one-hot code. For any pursuer drone, if it pursues the target t i =I j ∈T, it is considered that the i-th pursuer drone takes the j-th escaper drone as the pursuit target, and for n p When each pursuer drone determines its own pursuit target, a new matrix can be reconstructed based on its pursuit target, named the capture resource allocation matrix:

[0047]

[0048] For the roundup resource allocation matrix, given n p ≥n e Therefore, it is stipulated that each escaper drone must be pursued by at least one pursuer drone, so when say The resource allocation matrix is ​​a feasible resource allocation matrix. p and n e After determination, all feasible resource allocation matrices constitute the set

[0049] For each escaped drone, there are several pursuer drones that regard it as a pursuit target, so we can build a pursuit team set for the j-th escaped drone, which is defined as follows:

[0050] G j ={i|t i =I j},i∈Pj∈E.

[0051] As a further improvement of the present invention, step 2 is based on the potential game of drone capture resource allocation based on the Apollonius circle, thereby proving that the capture resource allocation for the pursuer drone has a Nash equilibrium solution, as follows;

[0052] Step 2-1. Introduce the concept of the Apollonius circle and use the area of ​​the Apollonius convex polygon to describe the competitive relationship between the pursuer and the fugitive. Specifically:

[0053] Suppose there are two points A and B on a plane. If there exists a moving point C, then:

[0054]

[0055] Then the locus of point C in the plane is a circle, which is called the Apollonian circle, where CA and CB are line segments connecting point C and points A and B, and λ is a positive constant less than 1, that is, 0<λ<1;

[0056] In the pursuit scenario, the coordinates of the i-th pursuer drone are p i =(x i ,y i ), with a velocity of v i , the coordinates of the jth escaper drone are p′ j =(x′ j ,y′ j ), with a velocity of v′ j To ensure the feasibility of pursuit, the speed between the pursuers and the fugitives satisfies v i >v′ j Based on the position and velocity ratio between the two, an Apollonius circle based on pursuit and escape is constructed. The points on the Apollonius circle are the future meeting points under the condition that the pursuers and the escapees maintain uniform linear motion. Any point o on the circle satisfies:

[0057]

[0058] The center of the constructed Apollonius circle (x o ,y o ) Radius r o satisfy:

[0059]

[0060] When there are n pursuer drones chasing a fleeing drone, the pursuit team G is composed of n pursuer drones. j When chasing the j-th escaped drone, each chaser drone in the pursuit team can construct an Apollonius circle with the escaped drone. The overlapping part of each Apollonius circle is called an Apollonius convex polygon.

[0061] When there are multiple escaped drones, similarly, there are multiple pursuit teams G j To chase the corresponding escaped drone, for each pursuit team, an Apollonius convex polygon can be formed with the pursuit target of the team, and the pursuit team G is recorded. jThe area of ​​the Apollonian convex polygon formed for the j-th escaper drone is S j , according to S j The size of the hunting group G j The roundup effect, S j The smaller it is, the smaller the potential survival space of the escaped drone is, that is, the better the capture effect is. On the contrary, S j The larger it is, the larger the potential survival space of the escaped drone is, and the less effective the roundup is;

[0062] Step 2-2. Construct the drone capture resource allocation potential game based on the Apollonius circle, specifically:

[0063] Definition: G P =(P,U p ,A p ) is a potential game. If satisfy:

[0064]

[0065] Where sgn is the sign function, U i is the utility function of the ith pursuer drone, φ is the potential function, A is a collection of actions for each escaper drone. ej =(v′ j ,θ′ j ),j∈E. A collection of actions for all the pursuer drones. pi =(t i ,v i ,θ i ),i∈P. To remove A pi The action set of other pursuer drones is then determined, and the actions between each agent satisfy:

[0066]

[0067] The pursuit targets t of each pursuer drone together form the resource allocation matrix When the action of the i-th pursuer drone is determined by a pi becomes a′ pi When the resource allocation matrix changes from c to D′, in order to ensure the rationality and feasibility of the resource allocation scheme, both D and D′ need to satisfy This condition ensures that G j =φ,j∈E will not appear, that is, every escaper drone is pursued by at least one pursuer drone;

[0068] Proof: Assume that the utility function and potential function of the agent are as follows:

[0069]

[0070] Assume that the actions of all escape drones are determined by a e Convert to a e ′, the action of the i-th pursuer drone is determined by a pi becomes a′ pi When satisfying Under the condition of , the area of ​​the Apollonius convex polygon corresponding to each escaped drone changes. Assume that the area change of the Apollonius circle corresponding to the jth escaped drone is ΔS j ,but:

[0071]

[0072] Similarly for the potential function:

[0073]

[0074] Therefore, it satisfies:

[0075]

[0076] Proof completed

[0077] According to the theorem, any finite potential game has a pure strategy Nash equilibrium. Therefore, for the drone capture resource allocation potential game based on the Apollonius circle, there is a pure strategy Nash equilibrium, that is, the optimal capture resource allocation plan exists.

[0078] As a further improvement of the present invention, in step 3, the optimal roundup resource allocation solution algorithm is used to transform the drone roundup resource allocation problem into a multi-constrained optimization problem, and the optimal solution for drone roundup resource allocation is solved by linear programming:

[0079] For the set of pursuer drones P = {1,2,...,n p},P has a total of Non-empty subsets, non-empty subsets are all possible forms of the pursuit team G, and then consider the escaped drone set E = {1,2,...,n e}, since there are n e There are escaping drones and each escaping drone is pursued by at least one pursuing drone. Therefore, in practice, there are only m possible forms of the pursuit team G, among which:

[0080]

[0081] C is the binomial coefficient. Since there are m possible capture teams, ne Runaway drones, so set up m×n e unknown decision variables, expressed in matrix form:

[0082]

[0083] For m feasible capture teams, an Apollonius convex polygon can be formed for each escaped drone, and then the area can be calculated. Similarly, we use a matrix to represent the area of ​​each convex polygon:

[0084]

[0085] For the matrix The included decision variables need to satisfy the following constraints:

[0086] Constraint 1: Each pursuit team can pursue at most one escaped drone;

[0087] Constraint 2: Each escapee drone is pursued by only one pursuit team;

[0088] Constraint 3: Each pursuer drone can only join one pursuit team;

[0089] Therefore, the constraints are as follows:

[0090]

[0091] Where l represents the case where two capture teams have the same subset. The resource allocation problem of drone capture in the many-to-many scenario is transformed into an optimization problem, where the optimization objectives and constraints are listed as follows:

[0092]

[0093] in It means to find the Hadamard product of two matrices, that is, multiply the elements in corresponding positions to get matrices of the same dimension, and ||||1 is the matrix norm of the matrix.

[0094] As a further improvement of the present invention, in step 4, the discriminator network training algorithm uses the optimal roundup resource allocation solution as the label value of the discriminator network and uses supervised learning to train the discriminator network, specifically:

[0095] Assume that under a certain state s, the optimal resource allocation solution for drones is That is, drones capture resources according to d * When allocating, the sum of the advantageous areas surrounded by each drone capture team is the smallest, and the minimum area is set as S * , calculate the area and S for each resource allocation scheme, when S i ∈[S *,S * +ΔS], it is considered as an expert solution, where ΔS is a hyperparameter threshold. On the contrary, when S i ∈[S * +ΔS,+∞], is regarded as a non-expert solution. The ultimate training goal of the discriminator network is to be able to independently judge whether the solution is an imitation action generated by the generator network based on the input information;

[0096] During the training process, random sampling is performed from the experience pool and the state information is and decision-making information Send it to the discriminator network, and the output is recorded as It reflects that the discriminator network thinks the state Make a plan for allocating resources for roundup is the probability of expert decision, and the experience pool records Is the label value of whether the generator network imitates the action, combining the above two to form the loss function as follows

[0097]

[0098] As a further improvement of the present invention, step 5 uses gumbel-softmax to perform a reparameterization operation to determine the output value of the generator network: specifically:

[0099] For the original discrete probability output by the generator network, we first perform a logarithmic operation and then add gumbel noise. The discrete variable corresponding to the maximum value of the output can be equivalent to a random sampling value. In actual use, gumbel-softmax is used to select the sampling value. The formula is as follows:

[0100]

[0101] Where π is the original probability distribution of the agent's actions, g is a standard gumbel noise, and τ is called the temperature coefficient. The smaller τ is, the closer the output z is to the one-hot code, and the larger τ is, the closer the output z is to the uniform distribution.

[0102] As a further improvement of the present invention, step 6: deep reinforcement learning algorithm is used to train the generator network so that the output value of the generator network gradually fits the optimal resource allocation scheme for drone capture, specifically:

[0103] Each pursuer drone is assigned a critic network, which is used to fit the policy gradient of the actor network to guide the gradient descent of the actor. For the training of the critic network, the temporal difference algorithm is usually used, and its formula is as follows:

[0104] Loss(ζ)=E[(Q(s,a;ζ)-r-γQ(s′,a′;ζ- )) 2 ]

[0105] Where ζ is the critic network parameter, - is the target network parameter for delayed update, s is the state information, s′ is the state at the future moment. a is the action information, a′ is the action generated by the actor network at the future moment, and γ∈(0,1) is the reinforcement learning discount rate;

[0106] For each actor network, this design is the generator in adversarial generative imitation learning, which is updated using a deterministic gradient strategy. The formula is as follows:

[0107]

[0108] In summary, for the resource allocation problem in multiple pursuit scenarios, a multi-agent deterministic gradient strategy algorithm can be used to fit the optimal solution. The specific reinforcement learning parameters are as follows:

[0109] State information and local observation: Let the global state be {P,P′}, where For the local observations of each pursuer UAV, considering that the optimal resource allocation solution is closely related to the global state, the algorithm design framework is based on the paradigm of centralized training and centralized execution. The input of the generator network of each pursuer UAV is the global state, i.e., {P, P′};

[0110] Action: For each pursuer drone’s generation network, after the output is processed by gumbel-softmax and other operations, the output is t in the form of a one-hot code. i , each pursuit target t i , forming a global decision

[0111] Reward: For the multi-chasing and multi-scenarios drone capture resource allocation problem, a discriminator is used to generate a global reward r = log(p(s, d)), where p(s, d)∈(0,1) is the discriminator output, i.e., the probability value that the discriminator considers to be the expert solution.

[0112] Compared with the prior art, the present invention has the following beneficial effects:

[0113] 1. Compared to existing drone pursuit schemes based on straight-line distance or angle, this invention proposes a coordinated drone capture scheme based on the Apollonius circle. This scheme, informed by game theory, can be effectively applied to multi-to-multi drone pursuit scenarios.

[0114] 2. The linear programming-based optimal capture resource allocation strategy search algorithm designed in this invention transforms the capture resource allocation problem of the pursuer drone into a multi-constrained optimization problem, and solves the optimal capture resource allocation strategy by setting reasonable constraints.

[0115] 3. The proposed adversarial generative imitation learning-based drone capture resource allocation algorithm has a solid theoretical foundation and a novel implementation. By training the discriminator and generator networks, it can effectively fit the optimal capture resource allocation strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] Figure 1 This is a schematic diagram of the design steps of the resource allocation algorithm for drone capture in the present invention;

[0117] Figure 2 This is a system model diagram established by the present invention based on a multi-to-multi drone pursuit scenario;

[0118] Figure 3 Schematic diagram of the drone capture effect based on the Apollonius circle of the present invention;

[0119] Figure 4 It is a pseudo code diagram of the algorithm for finding the optimal roundup resource allocation strategy based on linear programming of the present invention;

[0120] Figure 5 This is a framework diagram of the algorithm for solving the roundup resource allocation problem based on adversarial generative imitation learning of the present invention;

[0121] Figure 6 This is a training flow chart of the discriminator network in adversarial generative imitation learning of the present invention;

[0122] Figure 7 It is the overall pseudo code diagram of the roundup resource allocation scheme based on adversarial generative imitation learning of the present invention;

[0123] Figure 8 This is a schematic diagram of the pursuit simulation based on adversarial generative imitation learning in a 3-on-2 pursuit scenario of the present invention;

[0124] Figure 9 This is a comparison chart of the average capture rounds of various experimental schemes in the many-to-many pursuit scenario of the present invention;

[0125] Figure 10 This is a graph showing the output strategy proportions of the GAIL-based roundup resource allocation algorithm during the pursuit process. DETAILED DESCRIPTION

[0126] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0127] like Figure 1The drone capture resource allocation method based on adversarial generative imitation learning in the pursuit scenario shown below:

[0128] Step 1: Based on the many-to-many drone pursuit scenario, a system model is constructed and symbolic declarations are made. By defining the absolute rectangular coordinate system and the relative polar coordinate system, the positional relationship between the drones of the pursuit and escape parties is described, and the action space of the pursuit and escape parties is defined, thereby introducing concepts such as the resource allocation matrix and the pursuit team set.

[0129] Step 2: Figure 2 and Figure 3 As shown in the figure, the concept of Apollonius circle is introduced, and the potential game of drone roundup resource allocation based on Apollonius circle is constructed. It is proved that the roundup resource allocation has a Nash equilibrium solution, that is, the existence of the optimal roundup resource allocation plan.

[0130] Step 3: Based on the theoretical basis of step 2, a linear programming algorithm is proposed to solve the optimal roundup resource allocation solution. Based on the theory of step 2, this algorithm transforms the drone roundup resource allocation problem into a multi-constraint optimization problem and finds the optimal solution for roundup resource allocation through linear programming. The pseudo code of step 3 is as follows: Figure 4 shown.

[0131] Step 4: Figure 6 As shown in the flowchart, according to the optimal resource allocation solution algorithm designed in step 3, the optimal solution for drone capture resource allocation is used as the expert solution of the discriminator network, and a loss function is constructed to train the discriminator network.

[0132] Step 5: In order to determine the unique pursuit target of each pursuer drone based on the output value of the generator network, a gumbel-softmax reparameterization operation is used. This operation can sample the pursuit target value that conforms to the discrete probability distribution while ensuring differentiability.

[0133] Step 6: Based on the data generated in Steps 4 and 5, use the deep reinforcement learning algorithm to train the generator network assigned to each pursuer drone, so that the output value of each generator network gradually fits the optimal capture resource allocation plan. The flowchart and pseudo code of Steps 4 to 6 are as follows: Figure 5 and Figure 7 shown.

[0134] The following is a simulation example. This example uses Python programming language and the PyTorch framework to train various neural networks. The specific steps are as follows:

[0135] Step A: Initialize the multi-to-multi drone pursuit environment. In this case, we simulate a 3-to-2 drone pursuit scenario. First, we create a 2D canvas as the environment map based on matplotlib. Set n e =2,n p =3,v e =0.5,v p ∈[0,0.625]. At the initial moment, the coordinates of the escaper drone e1 are set to p1′=(5,5), the coordinates of the escaper drone e2 are set to p′2=(4,3), and the positions of the three pursuer drones are randomly generated in the environment map.

[0136] Step B: Create or train the driving motions for each agent. This step can be implemented in two ways. One is to simulate motions based on a fixed strategy, for example, the pursuer drone moves based on distance minimization, while the escaper drone moves based on an artificial potential field. The other approach is to use reinforcement learning algorithms to train the agent's motions. For example, the pursuer drone's reward signal can be based on the area of ​​the Apollonius circle, while the escaper drone's reward signal can be based on the shortest distance to the pursuer drone. In this step, since the pursuit target (i.e., the allocation of capture resources) has not yet been trained, the pursuit target for each agent can be randomly generated.

[0137] Step C: Arrange a generator network for each pursuer drone, and the generator network output is of dimension n e A one-dimensional tensor whose value is the probability of pursuing each escaped drone. Gumbel noise is added to the original output tensor and gumbel-softmax is used to sample the discrete probabilities to select the pursuit target for each agent.

[0138] Step D: Train the discriminator of the adversarial generative network. The training process generates the collection state information s based on steps 1 and 2, and then calculates the optimal capture allocation strategy D according to the optimal capture resource allocation algorithm based on linear programming. * , and D * The label of is set to 1. Based on the current generator network, D′ is obtained and the label of D′ is set to 0. Then (s,D * ,1) and (s,D′,l′) are sent to the experience pool. Then the experience pool is randomly sampled to obtain the sampling value Will Input into the discriminator network as the neural network input, and get the output value Will and label values Perform the comparison, construct the loss function, and perform gradient descent to train the discriminator network.

[0139] Step E: According to Step C and Step D, the state s and decision matrix D in the environment are obtained. The state s and decision matrix D are used as the input of the discriminator and input into the discriminator network to generate a reward signal r = log(p(s,D)). Then, the agent joint action A is performed in the environment to achieve the state transition from s to s′. (s,A,r,s′) is stored in the experience pool. Then, the maddpg algorithm is used to train the generator network of each pursuer drone. The experimental simulation results of the generator network output after training are shown as follows: Figure 8 shown.

[0140] Step F: Design comparative experiments and verification experiments. For the resource allocation problem in the many-to-many pursuit scenario, design two sets of comparative experiments. The first one uses the DQN algorithm to select the resource allocation scheme. When implementing it, each output of DQN is used as a feasible resource allocation scheme. The second one uses the imitation learning behavior cloning algorithm to calculate the optimal resource allocation strategy D in step D. * , directly based on D * The behavioral cloning algorithm is used to perform gradient descent on each generator network, and finally these two experimental schemes are compared with the experimental scheme based on adversarial generative imitation learning. The comparison results are as follows Figure 9 At the same time, in order to verify the universality of the overall experimental scheme, a test experiment was designed to record the proportion of the output strategy of the generator network of the pursuer drone during the pursuit process. The test results are shown as follows: Figure 10 shown.

[0141] In summary, this paper proposes a drone capture resource allocation algorithm based on adversarial generative imitation learning. This algorithm, based on the Apollonius circle-based potential game for drone capture resource allocation in many-to-many scenarios, uses a generative adversarial network framework to guide a generator network to adaptively learn the optimal capture resource allocation solution. This algorithm, unconstrained by the number and location of the pursuing and fugitive agents, successfully solves capture in many-to-many scenarios through a utility function derived from the potential game.

[0142] The above description is merely a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent variation based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A drone capture resource allocation method based on adversarial generative imitation learning in pursuit scenarios, characterized by: The steps include: Step 1: Based on the many-to-many drone pursuit scenario, a system model is constructed and symbolic declarations are made. By defining an absolute rectangular coordinate system and a relative polar coordinate system, the positional relationship between the pursuing and fleeing drones is described, and the action space of both parties is defined, thereby introducing the resource allocation matrix and the concept of pursuit team set. Step 2: Introduce the concept of Apollonius circle, construct a potential game for drone roundup resource allocation based on the Apollonius circle, and prove that the roundup resource allocation has a Nash equilibrium solution, that is, the existence of an optimal roundup resource allocation scheme, as follows; Step 2-1. Introduce the concept of the Apollonius circle and use the area of ​​the Apollonius convex polygon to describe the competitive relationship between the pursuer and the fugitive. Specifically: Suppose there are two points A and B on a plane. If there exists a moving point C, then: ； Then the locus of point C in the plane is a circle, which is called the Apollonian circle, where , is the line segment connecting point C, point A and point B. is a positive constant less than 1, that is, ; In the pursuit scenario, The coordinates of the pursuer drone are , the speed is , No. The coordinates of the escapee drone are , the speed is To ensure the feasibility of pursuit, the speed between the two parties is set to meet Based on the position and speed ratio between the two, an Apollonius circle based on pursuit is constructed. The points on the Apollonius circle are the future meeting points under the condition that the pursuit and the escape parties maintain uniform linear motion. Any point on the circle satisfy: ； The center of the constructed Apollonius circle radius satisfy: ； When there is When a pursuer drone chases a fleeing drone, A pursuit team consisting of 30 hunter drones Chasing the When there is a runaway drone, each pursuer drone in the pursuit team can construct an Apollonius circle with the runaway drone. The overlapping part of each Apollonius circle is called an Apollonius convex polygon. When there are multiple escaped drones, similarly, there are multiple pursuit teams Chasing the corresponding escaped drone, for each pursuit team, it can form an Apollonius convex polygon with the pursuit target of the team, record the pursuit team The area of ​​the Apollonian convex polygon formed for the j-th escaper drone is ,according to The size of the hunting team The capture effect, The smaller the size, the smaller the potential survival space of the escaped drone, which means the capture effect is better. The larger it is, the larger the potential survival space of the escaped drone is, and the less effective the roundup is; Step 2-2. Construct the drone capture resource allocation potential game based on the Apollonius circle, specifically: definition: It is a potential game. If ,satisfy: ； in is a symbolic function, , is the utility function of the ith pursuer drone, is the potential function, A collection of actions for each escaper drone. , A collection of actions for all the pursuer drones. , To remove The action set of other pursuer drones is then determined, and the actions between each agent satisfy: ； The targets of each pursuer drone Together they form the resource allocation matrix , when the The actions of the pursuer drone are determined by becomes When , let the resource allocation matrix be becomes To ensure the rationality and feasibility of the resource allocation plan, it is required that still , all need to meet , which ensures that It will not happen that every escaper drone is pursued by at least one pursuer drone; Proof: Assume that the utility function and potential function of the agent are as follows: ； Assume that the actions of all escaper drones are Convert to , No. The actions of the pursuer drone are determined by becomes When satisfying Under the condition of , the area of ​​the Apollonius convex polygon corresponding to each escape drone changes. Assuming that The change in the area of ​​the Apollonius circle corresponding to the escaped drone is: ,but: ； Similarly for the potential function: ； Therefore, it satisfies: ； Proof completed; According to the theorem, any finite potential game has a pure strategy Nash equilibrium. Therefore, for the drone capture resource allocation potential game based on the Apollonius circle, there is a pure strategy Nash equilibrium, that is, the optimal capture resource allocation plan exists. Step 3: Based on the theoretical basis of step 2, an optimal hunting resource allocation solution algorithm based on linear programming algorithm is proposed. Based on the theory of step 2, this algorithm transforms the UAV capture resource allocation problem into a multi-constraint optimization problem and finds the optimal solution for the capture resource allocation through linear programming. Step 4: Based on the optimal resource allocation solution algorithm designed in step 3, the optimal solution for drone capture resource allocation is used as the expert solution of the discriminator network, and a loss function is constructed to train the discriminator network. Step 5: To determine the unique pursuit target for each pursuer drone based on the output value of the generator network, a gumbel-softmax reparameterization operation is used. This operation can sample the pursuit target value that conforms to the discrete probability distribution while ensuring differentiability. Step 6: Based on the data generated in Steps 4 and 5, use a deep reinforcement learning algorithm to train the generator networks assigned to each pursuer drone, so that the output values ​​of each generator network gradually fit the optimal capture resource allocation plan.

2. The method for allocating drone resources using adversarial generative imitation learning in pursuit scenarios according to claim 1 is characterized by: Step 1: In the multi-to-multi drone pursuit scenario, drones are divided into two categories: pursuer drones and escaper drones. Let the number of escaper drones be , the number of pursuer drones is , , Runaway Drone Collection , the hunter drone collection is , in the absolute rectangular coordinate system of space, The coordinates of the pursuer drone are , No. The coordinates of the escapee drone are , the coordinates of any pursuer drone are used as the origin to construct a relative polar coordinate system. The positions of each escaper drone in the relative polar coordinate system can be expressed as express: ； To describe the relative position relationship between the escaper drone and the pursuer drone in each polar coordinate, the relative distance matrix and relative deflection angle matrix can be constructed: ； For the A runner drone whose action space includes velocity and heading angle ,in , , is the maximum speed of the escaper drone, and then: ； For the pursuer drone, similarly, its action space includes speed and heading angle , , ,In addition, due to the number of escapee drones Therefore, for the pursuer drone, there are multiple options for pursuit targets, so add pursuit targets To its action space, to describe the pursuit target , construct the identity matrix And decompose its row vector: ； Then build a collection of pursuit targets ,for , is a row vector in the form of a one-hot code. For any pursuer drone, if its pursuit target , regarded as the The first hunter drone will A runaway drone is the target of pursuit. When each pursuer drone determines its own pursuit target, a new matrix can be reconstructed based on its pursuit target, named the capture resource allocation matrix: ； For the roundup resource allocation matrix, given that Therefore, it is stipulated that each escaper drone must be pursued by at least one pursuer drone, so when ,say The resource allocation matrix formed is a feasible resource allocation matrix. and After determination, all feasible resource allocation matrices constitute the set ; For each escaper drone, there are several pursuer drones that regard it as a pursuit target, so a plan can be constructed for the first The set of pursuit teams for the escaped drones is defined as follows: 。 3. The method for allocating drone resources using adversarial generative imitation learning in pursuit scenarios according to claim 1 is characterized by: Step 3: The optimal resource allocation solution algorithm is used to convert the resource allocation problem of drone capture into a multi-constrained optimization problem, and the optimal solution of drone capture resource allocation is solved through linear programming: For the Hunter Drone Collection , A total of A non-empty subset is the pursuit team All possible forms, then consider the escaper drone collection , because there are a total of There are escaper drones and each escaper drone is chased by at least one pursuer drone, so in reality, the pursuit team The only possible forms are Species, of which: ； is the binomial coefficient, since there are A feasible arrest team, A runaway drone, so set up unknown decision variables, expressed in matrix form: ； for A feasible capture team can form an Apollonius convex polygon for each escaped drone, and then the area can be calculated. Similarly, we use a matrix to represent the area of ​​each convex polygon: ； For the matrix , the decision variables included need to satisfy the following constraints: Constraint 1: Each pursuit team can pursue at most one escaped drone; Constraint 2: Each escapee drone is pursued by only one pursuit team; Constraint 3: Each pursuer drone can only join one pursuit team; Therefore, the constraints are as follows: ； in Indicates the situation where two capture teams have the same subset. The resource allocation problem of drone capture in the many-to-many scenario is transformed into an optimization problem, where the optimization objectives and constraints are listed as follows: ； in It means to find the Hadamard product of two matrices, that is, to multiply the elements in corresponding positions to obtain a matrix of the same dimension. It is the matrix norm of the matrix.

4. The method for allocating drone resources using adversarial generative imitation learning in pursuit scenarios according to claim 1 is characterized by: Step 4: Discriminator network training algorithm. The optimal roundup resource allocation solution is used as the label value of the discriminator network. The discriminator network is trained using supervised learning. Specifically: Assume a state The optimal resource allocation solution for drones is , that is, drones capture resources according to When allocating, the sum of the advantageous areas surrounded by each drone capture team is the smallest, and the minimum area is set as , calculate the area and ,when When , it is considered as an expert solution, where Is a hyperparameter threshold, otherwise when , is regarded as a non-expert solution. The ultimate training goal of the discriminator network is to be able to independently judge whether the solution is an imitation action generated by the generator network based on the input information; During the training process, random sampling is performed from the experience pool and the state information is and decision-making information Send it to the discriminator network, and the output is recorded as , It reflects that the discriminator network thinks the state Make a plan for allocating resources for roundup is the probability of expert decision, and the experience pool records Is the label value of whether the generator network imitates the action, combining the above two to form the loss function as follows 。 5. The method for allocating drone resources for pursuit and capture based on adversarial generative imitation learning in pursuit and capture scenarios according to claim 1 is characterized by: Step 5 uses gumbel-softmax to perform a reparameterization operation to determine the output value of the generator network: specifically: For the original discrete probability output by the generator network, first perform logarithmic operation and then add Noise, the discrete variable corresponding to the maximum value of the output can be equivalent to a random sampling value. In actual use, use The sampling value is selected using the following formula: ； in is the original probability distribution of the agent’s actions, Is a standard noise, is called the temperature coefficient, The smaller the output The closer it is to a one-hot code, The larger the output The closer to a uniform distribution.

6. The method for allocating drone resources using adversarial generative imitation learning in pursuit scenarios according to claim 1 is characterized by: Step 6: The deep reinforcement learning algorithm trains the generator network so that the output value of the generator network gradually fits the optimal resource allocation plan for drone capture. Specifically: Assign a network, which is used to fit The network's policy gradient guides Gradient descent, for The network is usually trained using the temporal difference algorithm, which has the following formula: ； in for Network parameters, is the target network parameter for delayed update, For status information, It is the future state. is the action information, For the future Network-generated actions, is the reinforcement learning discount rate; For each The network, in this design, is the generator in adversarial generative imitation learning and is updated using a deterministic gradient strategy. The formula is as follows: ； In summary, for the resource allocation problem in multiple pursuit scenarios, a multi-agent deterministic gradient strategy algorithm can be used to fit the optimal solution. The specific reinforcement learning parameters are as follows: State information and local observation: Assume that the global state is ,in , ; For the local observations of each pursuer drone, considering that the optimal resource allocation solution is closely related to the global state, the algorithm design framework is based on the paradigm of centralized training and centralized execution. The input of the generator network of each pursuer drone is the global state, that is, ; Action: For each hunter drone’s generation network, after output, it passes After the operation, the output is in the form of a one-hot code , each pursuit target , forming a global decision ; Reward: For the resource allocation problem of drone capture in multiple pursuit scenarios, a discriminator is used to generate a global reward. ,in The discriminator outputs the result, that is, the probability value that the discriminator considers to be the expert solution.

Images