Unmanned aerial vehicle hunting resource allocation method based on confrontation generation imitation learning in pursuit scene

Abstract

The invention discloses an unmanned aerial vehicle hunting resource allocation method based on confrontation generation imitation learning in a pursuit scene, and aims to determine a pursuit target of each pursuit unmanned aerial vehicle in a many-to-many pursuit scene so as to solve the problem of hunting resource allocation. The method comprises the following steps: firstly, establishing a system model for a many-to-many-chasing unmanned aerial vehicle escape scene and carrying out symbol declaration; and then introducing the concept of an Apollonius circle, and establishing a potential game model for unmanned aerial vehicle hunting resource allocation. And then, providing an optimal surrounding resource allocation scheme solving algorithm based on linear programming. And finally, providing an unmanned aerial vehicle hunting resource allocation algorithm based on confrontation generation imitation learning. The algorithm designed by the invention is not influenced by the number and the initial state of the unmanned aerial vehicles of both pursuit and escape parties, Nash equilibrium under a potential game can be searched by adopting a linear programming algorithm, and an optimal hunting resource allocation scheme is generated based on adversarial generation imitation learning and adaptive learning; and finally, real-time dynamic allocation and scheduling of surrounding resources in a many-to-many unmanned aerial vehicle pursuit scene are realized.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent decision-making of multi-agent systems, and specifically is a method for allocating resources for drone capture based on adversarial generative imitation learning in pursuit scenarios. Background Art

[0002] The multi-to-multi drone pursuit scenario is a classic multi-agent system decision-making scenario. In this scenario, drones are divided into two categories: escaper drones and pursuer drones. The goal of the pursuer drone is to capture all escaper drones as quickly as possible through cooperation based on a reasonable capture resource allocation plan, while the escaper drone needs to get rid of the pursuit of the pursuer drone and extend its survival time as much as possible. Due to its wide application scenarios, multi-to-multi drone pursuit has attracted widespread attention. However, most of the current research focuses on the decision-making and action generation of the two parties in the competitive relationship, but gradually ignores the capture resource allocation problem between multiple pursuer drones. The few algorithms or solutions involving drone capture resource allocation are often based on the straight-line distance between the two parties. Their theoretical foundation is weak, the starting point of the idea is single, and the universality of the solution cannot be taken into account. In recent years, with the continuous vigorous development of deep learning and reinforcement learning, solving problems in multi-to-multi drone pursuit scenarios has ushered in new opportunities. However, the emergent strategies directly fitted by neural network models also lack theoretical basis and explainability, which poses new challenges to the solution of the resource allocation problem in multi-pursuit and multi-pursuit scenarios.

[0003] Compare with patent CN116796843A_A drone multi-to-multi pursuit game method based on PSO-M3DDPG and patent CN118276438A_A multi-agent pursuit optimal strategy method based on threat degree reinforcement learning algorithm

[0004] 1. Algorithm used in CN116796843A

[0005] Patent CN116796843A proposes a PSO-M3DDPG-based optimization solution for drone multi-to-multi pursuit and escape games. The method combines the particle swarm optimization algorithm (PSO) and the deep deterministic policy gradient algorithm (M3DDPG) of the maximum and minimum method to improve the adversarial learning performance of multi-agents. The specific steps include:

[0006] 1. Use the PSO algorithm to optimize the initial weights of the deep neural network to improve the rationality of initialization.

[0007] 2. Based on the M3DDPG algorithm, a multi-agent pursuit and escape game model is constructed to optimize the strategy selection of both parties.

[0008] 3. Conduct simulation training for different escape strategies, such as linear motion, simple curve motion, and intelligent escape, to optimize the pursuit path of the drone cluster.

[0009] 4. Utilize the design of reward functions to perform real-time strategy optimization in terms of distance, direction, and mission success.

[0010] This method optimizes the pursuit path of drones by combining multi-agent learning with PSO. However, when allocating resources to capture drones, this method only considers the impact of a single pursuit drone on the escaping drone, and fails to consider the collaborative relationship between multiple pursuit drones in the pursuit team. In addition, in terms of the strategy design for escaping drones, this method only provides three fixed escape strategies, and the difficulty of pursuit is relatively low.

[0011] 2. Algorithm used in CN118276438A

[0012] Patent CN118276438A proposes a multi-agent pursuit optimal strategy method based on threat degree reinforcement learning algorithm. The method uses particle swarm algorithm to pre-allocate threat degree and builds a multi-agent deep reinforcement learning model to achieve the optimal capture strategy for drone clusters. The specific steps include:

[0013] 1. Construct a threat pre-allocation model based on the particle swarm algorithm to optimize the UAV cluster mission and trajectory.

[0014] 2. Construct a threat level objective function and optimize the solution by comprehensively considering the threat level, track distance and mission cost.

[0015] 3. Use the multi-agent deep reinforcement learning model to conduct centralized training to achieve distributed capture tasks for drone swarms.

[0016] 4. Through the threat matching algorithm, the capture strategy of the drone cluster is output in real time.

[0017] This method optimizes the drone capture path and strategy by combining particle swarm optimization with multi-agent deep reinforcement learning. However, the particle swarm algorithm used in this method cannot guarantee the global optimality of the results, and the various hyperparameter settings also lack theoretical explanations.

[0018] The capture target is static in the environment, so in a large-scale dynamic environment, the model convergence efficiency still faces certain challenges.

[0019] 3. Method of the present invention

[0020] The present invention proposes a method for allocating drone capture resources based on adversarial generative imitation learning. The method uses the area of ​​the Apollonius circle to measure the competitive relationship between the two parties in pursuit and escape, uses a linear programming algorithm to generate the optimal capture resource allocation plan, and proposes a multi-drone capture resource allocation method based on adversarial generative imitation learning, which realizes the reasonable allocation of capture resources in the multi-drone pursuit scenario. The core innovations of the method include:

[0021] 1. Potential game of drone capture resource allocation based on Apollonius circles. The intersection area of ​​multiple Apollonius circles is used to measure the capture effect of the pursuit drone concentration on the escaped drone. Based on this, the potential game of drone capture resource allocation is constructed to provide a theoretical basis for the existence of the optimal capture resource allocation plan.

[0022] 2. Solution to the optimal resource allocation method for drone capture based on linear programming. The resource allocation problem for multiple drone capture scenarios is transformed into a linear programming problem, which not only solves the optimal solution but also provides expert experience for the subsequent adversarial generative imitation learning.

[0023] 3. A method for allocating drone capture resources based on adversarial generative imitation learning. Through the adversarial generative network model framework, the neural network output strategy is guided to be closer to the expert strategy, realizing the dynamic real-time allocation of capture resources in multi-drone pursuit scenarios.

[0024] 4. Advantages comparison

[0025] 1. Theoretical interpretability: This paper constructs a potential game of resource allocation to solve the global optimal solution for hunting drone targets. Compared with the threat pre-allocation model based on particle swarm algorithm proposed in CN118276438A, the theoretical basis of this paper is more solid and the method is more interpretable.

[0026] 2. Reasonableness of the scheme: When measuring the competitive relationship between the two UAVs, the present invention no longer uses the straight-line distance between the two UAVs as an indicator, but proposes to describe the generation space of the escaper UAV based on the convex polygon area formed by the Apollonius circle. Compared with CN116796843A, the present invention fully considers the mutual influence of multiple pursuit UAVs in the pursuit team in the process of solving the global optimal solution, and the scheme is more reasonable and complete.

[0027] 3. Innovation of the solution: This solution proposes a drone capture resource allocation method based on adversarial generative imitation learning, which learns the optimal capture resource allocation method by imitating expert strategies. Compared with the pre-training or pre-allocation based on the particle swarm algorithm in CN118276438A and CN116796843A, the solution is more innovative.

[0028] In summary, the drone capture resource allocation method based on adversarial generative imitation learning proposed in the present invention is superior to CN118276438A and CN116796843A in terms of theoretical explainability, scheme rationality, and scheme innovation, and can effectively solve the capture resource allocation problem in multi-drone pursuit scenarios. Summary of the invention

[0029] In order to solve the above technical problems, the present invention proposes a drone encirclement resource allocation method based on adversarial generative imitation learning in a pursuit scenario. During the drone pursuit process, the algorithm can dynamically adjust the pursuit targets of each pursuer drone in real time according to the learning results of GAIL and the global state, thereby helping the pursuer drone to complete the pursuit mission faster and better.

[0030] To achieve the above object, the technical solution adopted by the present invention is:

[0031] The method of allocating resources for drone capture based on adversarial generative imitation learning in the pursuit scenario includes the following steps:

[0032] Step 1: Based on the many-to-many UAV pursuit scenario, a system model is constructed and symbolic declaration is made. By defining the absolute rectangular coordinate system and the relative polar coordinate system, the positional relationship between the UAVs of the pursuit and escape parties is described, and the action space of the pursuit and escape parties is defined, thereby introducing the resource allocation matrix and the concept of the pursuit team set;

[0033] Step 2: Introduce the concept of Apollonius circle, construct the potential game of drone capture resource allocation based on Apollonius circle, and prove that the capture resource allocation has a Nash equilibrium solution, that is, the existence of the optimal capture resource allocation plan;

[0034] Step 3: Based on the theoretical basis of step 2, an optimal capture resource allocation solution algorithm based on a linear programming algorithm is proposed. Based on the theory of step 2, the algorithm transforms the drone capture resource allocation problem into a multi-constrained optimization problem, and obtains the optimal solution for capture resource allocation through linear programming;

[0035] Step 4: According to the optimal capture resource allocation solution algorithm designed in step 3, the optimal solution of drone capture resource allocation is used as the expert solution of the discriminator network, and a loss function is constructed to train the discriminator network;

[0036] Step 5: In order to determine the unique pursuit target of each pursuer drone according to the output value of the generator network, gumbel-softmax is used for reparameterization operation, which can sample the pursuit target value that conforms to the discrete probability distribution while ensuring the differentiability;

[0037] Step 6: Based on the data generated in Steps 4 and 5, use a deep reinforcement learning algorithm to train the generator network assigned to each pursuer drone, so that the output value of each generator network gradually fits the optimal capture resource allocation plan.

[0038] As a further improvement of the present invention, in step 1, in a multi-to-multi drone pursuit scenario, drones are divided into two categories: pursuer drones and escaper drones. Suppose the number of escaper drones is n. e , the number of pursuer drones is n p , n p ≥n e , the escaped drone set E = {1,2,...,n e}, the set of pursuer drones is P = {1,2,...,n p}, in the absolute rectangular coordinate system of space, the coordinates of the i-th pursuer drone are p i =(x i ,y i ),i∈P, the coordinates of the jth escaper drone are p′ j =(x′ j ,y′ j ), j∈E, the coordinates of any pursuer drone are used as the origin to construct a relative polar coordinate system. The positions of each escaper drone in the relative polar coordinate system can be expressed as d ij ,ω ij express:

[0039]

[0040] In order to describe the relative position relationship between the escaper drone and the pursuer drone in each polar coordinate, the relative distance matrix and relative deflection angle matrix can be constructed:

[0041]

[0042] For the jth escaper drone, its action space contains the velocity v′ j and heading angle θ j ′, where v′ j ∈[0,v′ max ],θ j ′∈[0,2π],v′ max is the maximum speed of the escaper drone, and then:

[0043]

[0044] For the pursuer drone, similarly, its action space includes the velocity v i and heading angle θ i , v i ∈[0,vmax ],θ i ∈[0,2π], in addition, due to the number of escapee drones n e >1, so for the pursuer drone, there are multiple choices of pursuit targets, so add pursuit target t i To its action space, to describe the pursuit target t i , construct the identity matrix I and decompose its row vectors:

[0045]

[0046] Then construct the pursuit target set T = {I1, I2, ..., I ne},for I j is a row vector in the form of a one-hot code. For any pursuer drone, if its pursuit target t i =I j ∈T, it is considered that the i-th pursuer drone takes the j-th escaper drone as the pursuit target, and for n p There are three chaser drones. When each chaser determines its own pursuit target, a new matrix can be reconstructed based on their pursuit targets, named the capture resource allocation matrix:

[0047]

[0048] For the roundup resource allocation matrix, given that n p ≥n e Therefore, it is stipulated that each escaper drone must be pursued by at least one pursuer drone, so when say The resource allocation matrix formed is a feasible resource allocation matrix. p and n e After determination, all feasible resource allocation matrices constitute the set

[0049] For each escaper drone, there are several pursuer drones that regard it as a pursuit target, so a set of pursuit teams for the j-th escaper drone can be constructed, which is defined as follows:

[0050] G j ={i|t i =I j},i∈Pj∈E.

[0051] As a further improvement of the present invention, step 2 is based on the potential game of the UAV capture resource allocation of the Apollonius circle, thereby proving that the capture resource allocation for the pursuer UAV has a Nash equilibrium solution, as follows;

[0052] Step 2-1. Introduce the concept of Apollonius circle and use the area of ​​Apollonius convex polygon to describe the competitive relationship between the two parties in pursuit, specifically:

[0053] Suppose there are two points A and B on the plane. If there is a moving point C, it satisfies:

[0054]

[0055] Then the locus of point C in the plane is a circle, which is called the Apollonian circle, where CA and CB are line segments connecting point C and point A and point B, and λ is a positive constant less than 1, that is, 0<λ<1;

[0056] In the pursuit scenario, the coordinates of the i-th pursuer drone are p i =(x i ,y i ), with a velocity of v i , the coordinates of the jth escaper drone are p′ j =(x′ j ,y′ j ), with a velocity of v′ j To ensure the feasibility of pursuit, the speed between the two parties is assumed to satisfy v i >v′ j Based on the position and speed ratio between the two, an Apollonius circle based on pursuit is constructed. The points on the Apollonius circle are the future meeting points under the condition that the pursuit and escape parties maintain uniform linear motion. Any point o on the circle satisfies:

[0057]

[0058] The center of the constructed Apollonius circle (x o ,y o ) Radius r o satisfy:

[0059]

[0060] When there are n pursuer drones chasing a fleeing drone, the pursuit team G is composed of n pursuer drones. j When chasing the jth escaper drone, each chaser drone in the pursuit team can construct an Apollonius circle with the escaper drone. The overlapping parts of each Apollonius circle are called Apollonius convex polygons.

[0061] When there are multiple escapee drones, similarly, there are multiple pursuit teams G j To hunt down the corresponding escaped drone, for each hunting group, an Apollonian convex polygon can be formed with the group's pursuit target, and the hunting group G is recorded as jThe area of ​​the Apollonian convex polygon formed for the jth escapee drone is S j , according to S j The size of the hunting group G j The round-up effect, S j The smaller it is, the smaller the potential survival space of the escaped drone is, that is, the better the capture effect is. On the contrary, S j The larger it is, the greater the potential survival space of the escaped drone, and the less effective the roundup will be;

[0062] Step 2-2. Construct the potential game of drone capture resource allocation based on the Apollonius circle, specifically:

[0063] Definition: G P =(P,U p ,A p ) is a potential game. If satisfy:

[0064]

[0065] Where sgn is the sign function, U i is the utility function of the ith pursuer drone, φ is the potential function, A is a collection of actions for each escaper drone. ej =(v′ j ,θ′ j ),j∈E. A collection of actions for all the hunter drones. pi =(t i ,v i ,θ i ),i∈P. To remove A pi The action set of other pursuer drones is then followed, and the actions between each agent satisfy:

[0066]

[0067] The pursuit targets t of each pursuer drone together form the resource allocation matrix When the action of the i-th pursuer drone is changed by a pi becomes a′ pi When the resource allocation matrix changes from c to D′, to ensure the rationality and feasibility of the resource allocation scheme, it is required that both D and D′ must satisfy This condition ensures that G j =φ,j∈E will not appear, that is, every escaper drone is chased by at least one pursuer drone;

[0068] Proof: Assume that the utility function and potential function of the agent are as follows:

[0069]

[0070] Assume that when the actions of all escaper drones are e Convert to a e ′, the action of the i-th pursuer drone is determined by a pi becomes a′ pi When satisfying Under the condition of , the area of ​​the Apollonius convex polygon corresponding to each escaper drone changes. Assume that the area change of the Apollonius circle corresponding to the jth escaper drone is ΔS j ,but:

[0071]

[0072] Similarly for the potential function:

[0073]

[0074] Therefore, it satisfies:

[0075]

[0076] Proof completed

[0077] According to the theorem, any finite potential game has a pure strategy Nash equilibrium. Therefore, for the drone capture resource allocation potential game based on the Apollonius circle, there is a pure strategy Nash equilibrium, that is, the optimal capture resource allocation plan exists.

[0078] As a further improvement of the present invention, in step 3, the optimal capture resource allocation solution algorithm converts the drone capture resource allocation problem into a multi-constrained optimization problem, and solves the optimal solution of drone capture resource allocation through linear programming:

[0079] For the set of pursuer drones P = {1,2,...,n p},P has a total of non-empty subsets, which are all possible forms of the pursuit team G. Consider the escaped drone set E = {1,2,...,n e}, since there are n e There are only m possible forms of the pursuit team G, among which:

[0080]

[0081] C is the binomial coefficient. Since there are m possible capture teams, ne There are 1 escapee drones, so we set up m×n e unknown decision variables, expressed in matrix form:

[0082]

[0083] For m feasible capture teams, an Apollonius convex polygon can be formed for each escaped drone, and then the area can be calculated. Similarly, we use a matrix to represent the area of ​​each convex polygon:

[0084]

[0085] For the matrix The included decision variables need to satisfy the following constraints:

[0086] Constraint 1: Each pursuit team can pursue at most one escaped drone;

[0087] Constraint 2: Each escapee drone is pursued by only one pursuit team;

[0088] Constraint 3: Each pursuer drone can only join one pursuit team;

[0089] Therefore, the constraints are as follows:

[0090]

[0091] Where l represents the situation where two capture teams have the same subset. The resource allocation problem of drone capture in many-to-many scenarios is transformed into an optimization problem, where the optimization objectives and constraints are listed as follows:

[0092]

[0093] in It means to find the Hadamard product of two matrices, that is, multiply the elements in corresponding positions to get matrices of the same dimension, and ||||1 is the matrix norm of the matrix.

[0094] As a further improvement of the present invention, in step 4, the discriminator network training algorithm uses the optimal roundup resource allocation solution as the label value of the discriminator network and uses supervised learning to train the discriminator network, specifically:

[0095] Assume that under a certain state s, the optimal resource allocation solution for drones is That is, drones capture resources according to d * When allocating, the sum of the advantageous areas surrounded by each drone capture team is the smallest, and the minimum area is S * , calculate the area and S for each resource allocation scheme, when S i ∈[S *,S * +ΔS], it is considered as an expert solution, where ΔS is a hyperparameter threshold. i ∈[S * +ΔS,+∞], is regarded as a non-expert solution. The ultimate training goal of the discriminator network is to be able to autonomously judge whether the solution is an imitation action generated by the generator network based on the input information;

[0096] During the training process, random sampling is performed from the experience pool and the state information is and decision-making information Sent to the discriminator network, the output is recorded as It reflects that the discriminator network thinks that the state Make a plan for allocating resources for the roundup is the probability of expert decision, and the experience pool records Is the label value of whether the generator network imitates the action, combining the above two to form the loss function as follows

[0097]

[0098] As a further improvement of the present invention, step 5 uses gumbel-softmax to perform a reparameterization operation to determine the output value of the generator network: specifically:

[0099] For the original discrete probability output by the generator network, a logarithmic operation is performed first and then gumbel noise is added. The discrete variable corresponding to the maximum value of the output can be equivalent to a random sampling value. In actual use, gumbel-softmax is used to select the sampling value. The formula is as follows:

[0100]

[0101] Where π is the original probability distribution of the agent's actions, g is a standard gumbel noise, and τ is called the temperature coefficient. The smaller τ is, the closer the output z is to the one-hot code, and the larger τ is, the closer the output z is to the uniform distribution.

[0102] As a further improvement of the present invention, step 6: the deep reinforcement learning algorithm trains the generator network so that the output value of the generator network gradually fits the optimal resource allocation scheme for drone capture, specifically:

[0103] Each pursuer drone is assigned a critic network, which is used to fit the policy gradient of the actor network to guide the gradient descent of the actor. For the training of the critic network, the temporal difference algorithm is usually used, and its formula is as follows:

[0104] Loss(ζ)=E[(Q(s,a;ζ)-r-γQ(s′,a′;ζ- )) 2 ]

[0105] Where ζ is the critic network parameter, - is the target network parameter for delayed update, s is the state information, s′ is the state at the future moment, a is the action information, a′ is the action generated by the actor network at the future moment, γ∈(0,1) is the discount rate for reinforcement learning;

[0106] For each actor network, this design is the generator in adversarial generative imitation learning, which is updated using a deterministic gradient strategy. The formula is as follows:

[0107]

[0108] In summary, for the problem of allocating resources in multiple pursuit scenarios, the multi-agent deterministic gradient strategy algorithm can be used to fit the optimal solution. The specific reinforcement learning parameters are as follows:

[0109] State information and local observation: Let the global state be {P,P′}, where For the local observations of each pursuer drone, considering that the optimal resource allocation solution is closely related to the global state, the algorithm design framework is based on the paradigm of centralized training and centralized execution. The input of the generator network of each pursuer drone is the global state, that is, {P, P′};

[0110] Action: For each of the generation networks of the pursuer drones, after the output is processed by gumbel-softmax and other operations, the output is t in the form of a one-hot code i , each pursuit target t i , forming a global decision

[0111] Reward: For the resource allocation problem of drone capture in multi-chasing and multi-scenario, a discriminator is used to generate a global reward r = log(p(s, d)), where p(s, d) ∈ (0, 1) is the output of the discriminator, that is, the probability value that the discriminator considers to be an expert solution.

[0112] Compared with the prior art, the present invention has the following beneficial effects:

[0113] 1. Compared with the existing drone pursuit scheme based on straight-line distance or angle, the present invention proposes a drone cooperative capture scheme based on Apollonius circle. From the perspective of game theory, this scheme can be effectively applied to multi-to-multi drone pursuit scenarios.

[0114] 2. The linear programming-based optimal capture resource allocation strategy search algorithm designed by the present invention transforms the capture resource allocation problem of the pursuer UAV into a multi-constrained optimization problem, and realizes the solution of the optimal capture resource allocation strategy by setting reasonable constraints.

[0115] 3. The drone capture resource allocation algorithm designed by the present invention based on adversarial generative imitation learning has a solid theoretical basis and a novel implementation method. By training the discriminator network and the generator network, the optimal capture resource allocation strategy can be effectively fitted. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] Figure 1 It is a schematic diagram of the design steps of the resource allocation algorithm for drone roundup in the present invention;

[0117] Figure 2 It is a system model diagram established by the present invention based on a many-to-many UAV pursuit scenario;

[0118] Figure 3 It is a schematic diagram of the drone capture effect based on the Apollonius circle of the present invention;

[0119] Figure 4 It is a pseudo code diagram of the algorithm for finding the optimal roundup resource allocation strategy based on linear programming of the present invention;

[0120] Figure 5 It is a framework diagram of the algorithm for solving the roundup resource allocation based on adversarial generative imitation learning of the present invention;

[0121] Figure 6 It is a training flow chart of the discriminator network in adversarial generative imitation learning of the present invention;

[0122] Figure 7 It is the overall pseudo code diagram of the roundup resource allocation scheme based on adversarial generative imitation learning of the present invention;

[0123] Figure 8 It is a schematic diagram of the pursuit simulation in a 3-to-2 pursuit scenario based on adversarial generative imitation learning of the present invention;

[0124] Fig. 9 This is a comparison chart of the average capture rounds of each experimental scheme in the many-to-many pursuit scenario of the present invention;

[0125] Fig.10 It is a graph showing the output strategy proportions of the GAIL-based roundup resource allocation algorithm in the pursuit process of the present invention. DETAILED DESCRIPTION

[0126] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0127] like Figure 1The drone capture resource allocation method based on adversarial generative imitation learning in the pursuit scenario shown:

[0128] Step 1: Based on the many-to-many UAV pursuit scenario, a system model is constructed and symbolic declaration is made. By defining the absolute rectangular coordinate system and the relative polar coordinate system, the positional relationship between the UAVs of the pursuit and escape parties is described, and the action space of the pursuit and escape parties is defined, thereby introducing concepts such as resource allocation matrix and pursuit team set.

[0129] Step 2: Figure 2 and Figure 3 As shown in the figure, the concept of Apollonius circle is introduced, and the potential game of drone roundup resource allocation based on Apollonius circle is constructed. It is proved that the roundup resource allocation has a Nash equilibrium solution, that is, the existence of the optimal roundup resource allocation plan.

[0130] Step 3: Based on the theoretical basis of step 2, an optimal capture resource allocation solution algorithm based on linear programming algorithm is proposed. Based on the theory of step 2, this algorithm transforms the drone capture resource allocation problem into a multi-constrained optimization problem, and finds the optimal solution for capture resource allocation through linear programming. The pseudo code of step 3 is as follows: Figure 4 shown.

[0131] Step 4: Figure 6 As shown in the flow chart, according to the optimal capture resource allocation solution algorithm designed in step 3, the optimal solution for drone capture resource allocation is used as the expert solution of the discriminator network, and a loss function is constructed to train the discriminator network.

[0132] Step 5: In order to determine the unique pursuit target of each pursuer drone according to the output value of the generator network, gumbel-softmax is used for reparameterization operation, which can sample the pursuit target value that conforms to the discrete probability distribution while ensuring differentiability.

[0133] Step 6: Based on the data generated in Steps 4 and 5, use the deep reinforcement learning algorithm to train the generator network assigned to each pursuer drone, so that the output value of each generator network gradually fits the optimal capture resource allocation plan. The flowchart and pseudo code of Steps 4 to 6 are as follows: Figure 5 and Figure 7 shown.

[0134] The following is a specific implementation case using the simulation method. This case uses the Python programming language to train various neural networks based on the PyTorch framework. The specific steps are as follows:

[0135] Step A: Initialize the many-to-many drone pursuit environment. In this case, the experimental simulation is based on a 3-to-2 drone pursuit scenario. First, create a two-dimensional canvas as the environment map based on matplotlib. Set n e =2,n p =3, v e =0.5,v p ∈[0,0.625]. At the initial moment, the coordinates of the escaper drone e1 are set to p1′=(5,5), the coordinates of the escaper drone e2 are set to p′2=(4,3), and the positions of the three pursuer drones are randomly generated in the environment map.

[0136] Step B: Create or train the driving actions of each agent. There are two ways to implement this step. One is to execute the simulation action of a fixed strategy, for example, the pursuer drone moves based on the minimization of distance, and the escaper drone moves based on the artificial potential field. The second is to use the reinforcement learning algorithm to train the agent action. For example, the reward signal of the pursuer drone can be based on the area of ​​the Apollonius circle, and the reward signal of the escaper drone can be based on the shortest distance to the pursuer drone. In this step, since the pursuit target, that is, the allocation of the encirclement resources, has not yet been trained, the pursuit target of each agent can be temporarily generated randomly.

[0137] Step C: Arrange a generator network for each pursuer drone, and the generator network output is of dimension n e A one-dimensional tensor with the size of the tensor being the probability of chasing each escaped drone. Gumbel noise is added to the original output tensor, and gumbel-softmax is used to sample the discrete probabilities to select the pursuit targets of each agent.

[0138] Step D: Train the discriminator of the adversarial generative network. The training process generates the collection state information s based on steps 1 and 2, and then calculates the optimal capture allocation strategy D according to the optimal capture resource allocation algorithm based on linear programming. * , and D * The label of is set to 1. Based on the current generator network, D′ is obtained and the label of D′ is set to 0. Then (s,D * ,1) and (s,D′,l′) are sent to the experience pool. Then the experience pool is randomly sampled to obtain the sampling value Will As the neural network input to the discriminator network, get the output value Will and label value Perform an alignment, construct a loss function, and perform gradient descent to train the discriminator network.

[0139] Step E: According to step C and step D, the state s and decision matrix D in the environment are obtained, and the state s and decision matrix D are used as the input of the discriminator and input into the discriminator network to generate a reward signal r = log(p(s,D)). Then, the agent joint action A is performed in the environment to realize the state transfer from s to s′, and (s,A,r,s′) is stored in the experience pool. Then, the maddpg algorithm is used to train the generator network of each pursuer drone. The experimental simulation results of the generator network output after training are shown in the figure below. Figure 8 shown.

[0140] Step F: Design comparative experiments and verification experiments. For the resource allocation problem of the roundup in the many-to-many pursuit scenario, two sets of comparative experiments are designed. The first one uses the DQN algorithm to select the resource allocation scheme. When implementing it, each output of DQN is used as a feasible resource allocation scheme. The second one uses the imitation learning behavior cloning algorithm to calculate the optimal roundup resource allocation strategy D in step D. * , directly according to D * The behavior cloning algorithm is used to perform gradient descent on each generator network, and finally these two experimental schemes are compared with the experimental scheme based on adversarial generative imitation learning. The comparison results are shown in Fig. 9 At the same time, in order to verify the universality of the overall experimental scheme, a test experiment was designed to record the proportion of the output strategy of the generator network of the pursuer drone during the pursuit process. The test results are shown in Fig.10 shown.

[0141] In summary, the present invention proposes a drone capture resource allocation algorithm based on adversarial generative imitation learning. The algorithm takes the potential game of drone capture resource allocation based on Apollonius circle in many-to-many scenarios as its theoretical basis, and guides the generator network to adaptively learn the optimal capture resource allocation plan through the adversarial generative network framework. The algorithm designed in this scheme is not constrained by the number and position of the intelligent agents on both sides of the pursuit and escape, and successfully solves the capture in many-to-many scenarios through the utility function derived from the potential game.

[0142] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A drone capture resource allocation method based on adversarial generative imitation learning in pursuit scenarios, characterized by: The steps include: Step 1: Based on the many-to-many UAV pursuit scenario, a system model is constructed and symbolic declaration is made. By defining the absolute rectangular coordinate system and the relative polar coordinate system, the positional relationship between the UAVs of the pursuit and escape parties is described, and the action space of the pursuit and escape parties is defined, thereby introducing the resource allocation matrix and the concept of the pursuit team set; Step 2: Introduce the concept of Apollonius circle, construct the potential game of drone capture resource allocation based on Apollonius circle, and prove that the capture resource allocation has a Nash equilibrium solution, that is, the existence of the optimal capture resource allocation plan; Step 3: Based on the theoretical basis of step 2, an optimal hunting resource allocation solution algorithm based on linear programming algorithm is proposed. Based on the theory of step 2, this algorithm transforms the UAV capture resource allocation problem into a multi-constrained optimization problem, and finds the optimal solution for capture resource allocation through linear programming; Step 4: According to the optimal capture resource allocation solution algorithm designed in step 3, the optimal solution of drone capture resource allocation is used as the expert solution of the discriminator network, and a loss function is constructed to train the discriminator network; Step 5: In order to determine the unique pursuit target of each pursuer drone according to the output value of the generator network, gumbel-softmax is used for reparameterization operation, which can sample the pursuit target value that conforms to the discrete probability distribution while ensuring the differentiability; Step 6: Based on the data generated in Steps 4 and 5, use a deep reinforcement learning algorithm to train the generator network assigned to each pursuer drone, so that the output value of each generator network gradually fits the optimal capture resource allocation plan.

2. The method for allocating resources for drone capture based on adversarial generative imitation learning in a pursuit scenario according to claim 1 is characterized in that: Step 1: In the many-to-many drone pursuit scenario, drones are divided into two categories: pursuer drones and escaper drones. Let the number of escaper drones be n. e , the number of pursuer drones is n p , n p ≥n e , the escaped drone set E = {1,2,...,n e }, the set of pursuer drones is P = {1,2,...,n p }, in the absolute rectangular coordinate system of space, the coordinates of the i-th pursuer drone are p i =(x i ,y i ),i∈P, the coordinates of the jth escaper drone are p′ j =(x′ j ,y′ j ), j∈E, the coordinates of any pursuer drone are used as the origin to construct a relative polar coordinate system. The positions of each escaper drone in the relative polar coordinate system can be expressed as d ij ,ω ij express: In order to describe the relative position relationship between the escaper drone and the pursuer drone in each polar coordinate, the relative distance matrix and relative deflection angle matrix can be constructed: For the jth escaper drone, its action space contains the velocity v′ j and heading angle θ j ′, where v′ j ∈[0,v′ max ],θ j ′∈[0,2π],v′ max is the maximum speed of the escaper drone, and then: For the pursuer drone, similarly, its action space includes the velocity v i and heading angle θ i , v i ∈[0,v max ],θ i ∈[0,2π], in addition, due to the number of escapee drones n e >1, so for the pursuer drone, there are multiple choices of pursuit targets, so add pursuit target t i To its action space, to describe the pursuit target t i , construct the identity matrix I and decompose its row vectors: Then build a collection of pursuit targets for I j is a row vector in the form of a one-hot code. For any pursuer drone, if its pursuit target t i =I j ∈T, it is considered that the i-th pursuer drone takes the j-th escaper drone as the pursuit target, and for n p There are three chaser drones. When each chaser determines its own pursuit target, a new matrix can be reconstructed based on their pursuit targets, named the capture resource allocation matrix: For the roundup resource allocation matrix, given that n p ≥n e Therefore, it is stipulated that each escaper drone must be pursued by at least one pursuer drone, so when say The resource allocation matrix formed is a feasible resource allocation matrix. p and n e After determination, all feasible resource allocation matrices constitute the set For each escaper drone, there are several pursuer drones that regard it as a pursuit target, so a set of pursuit teams for the j-th escaper drone can be constructed, which is defined as follows: G j ={i|t i =I j },i∈P.j∈E。 3. The method for allocating resources for drone capture based on adversarial generative imitation learning in a pursuit scenario according to claim 1 is characterized in that: Step 2 is based on the potential game of drone capture resource allocation based on the Apollonius circle, thus proving that the capture resource allocation for the pursuer drone has a Nash equilibrium solution, as follows; Step 2-1. Introduce the concept of Apollonius circle and use the area of ​​Apollonius convex polygon to describe the competitive relationship between the two parties in pursuit and escape, specifically: Suppose there are two points A and B on the plane. If there is a moving point C, it satisfies: Then the locus of point C in the plane is a circle, which is called the Apollonian circle, where CA and CB are line segments connecting point C and point A and point B, and λ is a positive constant less than 1, that is, 0<λ<1; In the pursuit scenario, the coordinates of the i-th pursuer drone are p i =(x i ,y i ), with a velocity of v i , the coordinates of the jth escaper drone are p′ j =(x′ j ,y′ j ), with a velocity of v′ j To ensure the feasibility of pursuit, the speed between the two parties is assumed to satisfy v i >v′ j Based on the position and speed ratio between the two, an Apollonius circle based on pursuit is constructed. The points on the Apollonius circle are the future meeting points under the condition that the pursuit and escape parties maintain uniform linear motion. Any point o on the circle satisfies: The center of the constructed Apollonius circle (x o ,y o ) Radius r o satisfy: When there are n pursuer drones chasing a fleeing drone, the pursuit team G is composed of n pursuer drones. j When chasing the jth escaper drone, each chaser drone in the pursuit team can construct an Apollonius circle with the escaper drone. The overlapping parts of each Apollonius circle are called Apollonius convex polygons. When there are multiple escapee drones, similarly, there are multiple pursuit teams G j To hunt down the corresponding escaped drone, for each hunting group, an Apollonian convex polygon can be formed with the group's pursuit target, and the hunting group G is recorded as j The area of ​​the Apollonian convex polygon formed for the jth escapee drone is S j , according to S j The size of the hunting group G j The round-up effect, S j The smaller it is, the smaller the potential survival space of the escaped drone is, that is, the better the capture effect is. On the contrary, S j The larger it is, the greater the potential survival space of the escaped drone, and the less effective the roundup will be; Step 2-2. Construct the potential game of drone capture resource allocation based on the Apollonius circle, specifically: Definition: G P =(P,U p ,A p ) is a potential game. If satisfy: Where sgn is the sign function, U i is the utility function of the ith pursuer drone, φ is the potential function, A is a collection of actions for each escaper drone. ej =(v′ j ,θ j ′),j∈E. A collection of actions for all the hunter drones. pi =(t i ,v i ,θ i ),i∈P. To remove A pi The action set of other pursuer drones is then followed, and the actions between each agent satisfy: The pursuit targets t of each pursuer drone together form the resource allocation matrix When the action of the i-th pursuer drone is changed by a pi becomes a′ pi When the resource allocation matrix changes from c to D′, to ensure the rationality and feasibility of the resource allocation scheme, it is required that both D and D′ must satisfy This condition ensures that G j =φ,j∈E will not appear, that is, every escaper drone is chased by at least one pursuer drone; Proof: Assume that the utility function and potential function of the agent are as follows: Assume that when the actions of all escaper drones are e Convert to a e ′, the action of the i-th pursuer drone is determined by a pi becomes a′ pi When satisfying Under the condition of , the area of ​​the Apollonius convex polygon corresponding to each escaper drone changes. Assume that the area change of the Apollonius circle corresponding to the jth escaper drone is ΔS j ,but: Similarly for the potential function: Therefore, it satisfies: Proof completed According to the theorem, any finite potential game has a pure strategy Nash equilibrium. Therefore, for the drone capture resource allocation potential game based on the Apollonius circle, there is a pure strategy Nash equilibrium, that is, the optimal capture resource allocation plan exists.

4. The method for allocating resources for drone capture based on adversarial generative imitation learning in a pursuit scenario according to claim 1 is characterized in that: Step 3: The optimal resource allocation solution algorithm for rounding up drones is used to transform the resource allocation problem for rounding up drones into a multi-constrained optimization problem. The optimal solution for resource allocation for rounding up drones is solved through linear programming: For the set of pursuer drones P = {1,2,...,n p },P has a total of non-empty subsets, which are all possible forms of the pursuit team G. Consider the escaped drone set E = {1,2,...,n e }, since there are n e There are only m possible forms of the pursuit team G, among which: C is the binomial coefficient. Since there are m possible capture teams, n e There are 1 escapee drones, so we set up m×n e unknown decision variables, expressed in matrix form: For m feasible capture teams, an Apollonius convex polygon can be formed for each escaped drone, and then the area can be calculated. Similarly, we use a matrix to represent the area of ​​each convex polygon: For the matrix The included decision variables need to satisfy the following constraints: Constraint 1: Each pursuit team can pursue at most one escaped drone; Constraint 2: Each escapee drone is pursued by only one pursuit team; Constraint 3: Each pursuer drone can only join one pursuit team; Therefore, the constraints are as follows: Where l represents the situation where two capture teams have the same subset. The resource allocation problem of drone capture in the many-to-many scenario is transformed into an optimization problem, where the optimization objectives and constraints are listed as follows: in It means to find the Hadamard product of two matrices, that is, multiply the elements in corresponding positions to get matrices of the same dimension, and || ||1 is the matrix norm of the matrix.

5. The method for allocating resources for drone capture based on adversarial generative imitation learning in a pursuit scenario according to claim 1 is characterized in that: Step 4: Discriminator network training algorithm: take the optimal roundup resource allocation solution as the label value of the discriminator network, and use supervised learning to train the discriminator network. Specifically: Assume that under a certain state s, the optimal resource allocation solution for drones is That is, drones capture resources according to d * When allocating, the sum of the advantageous areas surrounded by each drone capture team is the smallest, and the minimum area is S * , calculate the area and S for each resource allocation scheme, when S i ∈[S * ,S * +ΔS], it is considered as an expert solution, where ΔS is a hyperparameter threshold. i ∈[S * +ΔS,+∞], is regarded as a non-expert solution. The ultimate training goal of the discriminator network is to be able to autonomously judge whether the solution is an imitation action generated by the generator network based on the input information; During the training process, random sampling is performed from the experience pool and the state information is and decision-making information Sent to the discriminator network, the output is recorded as It reflects that the discriminator network thinks that the state Make a plan for allocating resources for the roundup is the probability of expert decision, and the experience pool records Is the label value of whether the generator network imitates the action, combining the above two to form the loss function as follows 6. The method for allocating resources for drone capture based on adversarial generative imitation learning in a pursuit scenario according to claim 1 is characterized in that: Step 5 uses gumbel-softmax to perform a reparameterization operation to determine the output value of the generator network: specifically: For the original discrete probability output by the generator network, a logarithmic operation is performed first and then gumbel noise is added. The discrete variable corresponding to the maximum value of the output can be equivalent to a random sampling value. In actual use, gumbel-softmax is used to select the sampling value. The formula is as follows: Where π is the original probability distribution of the agent's actions, g is a standard gumbel noise, and τ is called the temperature coefficient. The smaller τ is, the closer the output z is to the one-hot code, and the larger τ is, the closer the output z is to the uniform distribution.

7. The method for allocating resources for drone capture based on adversarial generative imitation learning in a pursuit scenario according to claim 1 is characterized by: Step 6: The deep reinforcement learning algorithm trains the generator network so that the output value of the generator network gradually fits the optimal resource allocation plan for drone capture, specifically: A critic network is assigned to each pursuer drone. The critic network is used to fit the policy gradient of the actor network to guide the gradient descent of the actor. For the training of the critic network, the temporal difference algorithm is usually used, and its formula is as follows: Loss(ζ)=E[(Q(s,a;ζ)-r-γQ(s′,a′;ζ) - )) 2 ] Where ζ is the critic network parameter, - is the target network parameter for delayed update, s is the state information, s′ is the state at the future moment, a is the action information, a′ is the action generated by the actor network at the future moment, γ∈(0,1) is the discount rate for reinforcement learning; For each actor network, this design is the generator in adversarial generative imitation learning, which is updated using a deterministic gradient strategy. The formula is as follows: In summary, for the problem of allocating resources in multiple pursuit scenarios, the multi-agent deterministic gradient strategy algorithm can be used to fit the optimal solution. The specific reinforcement learning parameters are as follows: State information and local observation: Let the global state be {P,P′}, where For the local observations of each pursuer drone, considering that the optimal resource allocation solution is closely related to the global state, the algorithm design framework is based on the paradigm of centralized training and centralized execution. The input of the generator network of each pursuer drone is the global state, that is, {P, P′}; Action: For each of the generation networks of the pursuer drones, after the output is processed by gumbel-softmax and other operations, the output is t in the form of a one-hot code i , each pursuit target t i , forming a global decision Reward: For the resource allocation problem of drone capture in multi-chasing and multi-scenario, a discriminator is used to generate a global reward r = log(p(s, d)), where p(s, d) ∈ (0, 1) is the output of the discriminator, that is, the probability value that the discriminator considers to be an expert solution.

Images