Distributed multi-unmanned aerial vehicle intelligent dash decision-making method for three-dimensional urban environment

Abstract

The invention belongs to the technical field of multi-unmanned aerial vehicle breakout, and particularly relates to a distributed multi-unmanned aerial vehicle intelligent breakout decision-making method oriented to a three-dimensional city environment, which comprises the steps of building a digital twin city simulation environment, simulating a multi-unmanned aerial vehicle breakout and unmanned aerial vehicle hunting process through the simulation environment, obtaining real-time simulation data, and determining a three-dimensional city environment according to the real-time simulation data. A global-individual-based multi-unmanned-aerial-vehicle trajectory prediction network is designed, future trajectory prediction of a target is realized, a basis is provided for subsequent decision making, a multi-unmanned-aerial-vehicle random game model facing a suddenness task is established, a multi-unmanned-aerial-vehicle suddenness decision making algorithm based on maximum entropy is further designed, offline training and online decision making of the algorithm are completed, and the optimal suddenness task is established. And realizing real-time decision of the multiple unmanned aerial vehicles in a three-dimensional urban environment by using the trained multi-unmanned aerial vehicle breakthrough decision network. According to the invention, through obtaining the future trajectory of the unmanned aerial vehicle trapped by the opposite side, the distributed multi-unmanned aerial vehicle suddenly cooperative control method based on the maximum entropy is researched, and rapid and effective suddenly defense in a three-dimensional urban environment is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-UAV breakout, and in particular relates to a distributed multi-UAV intelligent breakout decision-making method for a three-dimensional urban environment. Background Art

[0002] Quad-rotor UAVs (Unmanned Aerial Vehicle, UAV) have excellent maneuverability, and due to their small size and low cost, they have been widely used in forest fire prevention, urban security, and power grid maintenance. However, a single quad-rotor UAV has limited effectiveness in handling complex tasks. Therefore, the concept of "drone swarm" has emerged as a more effective application model. With the development of drone swarm collaborative control technology, multi-drone pursuit and escape game has become an important research field in the context of urban security.

[0003] Deep reinforcement learning (DRL) has been fully developed in the past decade. Since defeating humans in Atari video games, Deepmind's AlphaGo and AlphaSTA have successively defeated top players in complex games such as Go and StarCraft. It can be seen that deep reinforcement learning has strong learning and decision-making capabilities and has great potential in solving the defense penetration problem of drone swarm systems;

[0004] At present, many excellent single-agent deep reinforcement learning algorithms, such as deep Q network (DQN), deep deterministic policy gradient (DDPG), double-delayed deep deterministic policy gradient (TD3) and soft actor-critic (SAC), have been widely used in continuous control. Furthermore, with the increase in the number of agents, in the field of multi-agent reinforcement learning, centralized training and distributed execution frameworks solve the non-stationarity problem caused by the joint training of multiple learning agents. For the problem of multi-UAV penetration in a three-dimensional urban environment, the existing research methods have the following difficulties:

[0005] (1) Existing multi-UAV breakout control methods rarely consider the design of virtual simulation interactive platforms in three-dimensional urban environments, and lack a digital twin city simulation platform that can provide real-time training data;

[0006] (2) Most of the existing traditional multi-UAV cooperative control methods use methods such as space discretization, particle swarm optimization and artificial potential field to process the state point set to achieve obstacle avoidance and controller design. At the same time, due to the complexity of the multi-to-multi UAV pursuit and escape game in three-dimensional space, it is difficult to quickly obtain a feasible analytical solution based on the traditional method of Hamilton-Jacobi-Isaacs (HJI) differential equation;

[0007] (3) Existing multi-agent reinforcement learning algorithms rarely consider the impact of the future trajectory of the adversary on their own control strategy when solving multi-agent pursuit and escape game problems;

[0008] Therefore, it is necessary to propose a new distributed multi-UAV intelligent breakthrough decision-making method for three-dimensional urban environments. Summary of the invention

[0009] The purpose of the present invention is to provide a distributed multi-UAV intelligent breakthrough decision-making method for a three-dimensional urban environment. First, the future trajectory of the opponent's encirclement UAV is obtained by designing a target trajectory prediction network, and then a distributed multi-UAV breakthrough collaborative control method based on maximum entropy is studied by considering factors such as the complexity, collision avoidance, and obstacle avoidance of the three-dimensional urban environment, so as to achieve rapid and effective breakthrough in a three-dimensional urban environment.

[0010] The technical solution adopted by the present invention is as follows:

[0011] The distributed multi-UAV intelligent breakout decision-making method for a three-dimensional urban environment includes the following steps:

[0012] S1: Build a digital twin city simulation environment, simulate the multi-UAV breakout and UAV roundup process through the simulation environment, and obtain real-time simulation data;

[0013] S2: Design a global-individual multi-UAV trajectory prediction network to achieve the future trajectory prediction of the target and provide a basis for subsequent decision-making;

[0014] S3: Establish a multi-UAV random game model for breakout tasks, then design a multi-UAV breakout decision algorithm based on maximum entropy, and complete offline training of the algorithm;

[0015] S4: Online decision making, using the trained multi-UAV breakout decision network to achieve real-time decision making of multiple UAVs in a three-dimensional urban environment.

[0016] The technical effects achieved by the present invention are:

[0017] (1) The present invention aims at the decision-making problem of multiple drones breaking out in a three-dimensional urban environment. The existing simulation environment is difficult to simulate the real urban environment and lacks drone breakout simulation scenes. This patent designs a digital twin city simulation environment based on Unity3d, and uses graphics processing software such as RenderDoc, Blender and Unity3d and game engine software to build a digital twin three-dimensional city, simulate the process of multiple drones breaking out and drone encirclement, and obtain real-time simulation data based on real-time interactive technology.

[0018] (2) The present invention combines the group trajectory prediction method with the multi-agent reinforcement learning algorithm, and proposes a multi-UAV maximum entropy breakout intelligent decision-making algorithm based on group trajectory prediction. Based on the multi-UAV random game model for breakout tasks, real-time expression of the breakout situation is achieved. By considering the future trajectory of the target movement, collision avoidance, and other factors, the multi-UAV breakout intelligent decision-making algorithm based on maximum entropy makes real-time decisions on the multi-UAV breakout strategy. Simulation results show that compared with a single multi-agent reinforcement learning algorithm, the algorithm proposed in this patent has higher breakout benefits during the training process and has a faster convergence speed.

[0019] (3) The present invention first obtains the future trajectory of the opponent's encirclement drone by designing a target trajectory prediction network, and then studies the distributed multi-UAV breakthrough collaborative control method based on maximum entropy by considering factors such as the complexity, collision avoidance, and obstacle avoidance of the three-dimensional urban environment, so as to achieve rapid and effective breakthrough in the three-dimensional urban environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is the operation interface of RenderDoc in the present invention;

[0021] Figure 2 This is the effect diagram of importing the RDC model into Blender in the present invention;

[0022] Figure 3 It is a real urban training scenario of multiple UAVs in the present invention;

[0023] Figure 4 It is a schematic diagram of the real-time interaction framework in the present invention;

[0024] Figure 5 It is a schematic diagram of a multi-UAV trajectory prediction network in the present invention;

[0025] Figure 6 This is a schematic diagram of the maximum entropy multi-UAV breakout decision network in the present invention;

[0026] Figure 7 is a schematic diagram of a digital twin city environment for a breakout task in the present invention;

[0027] Figure 8 It is a curve diagram of the success rate of breaking out in the present invention;

[0028] Fig. 9 It is a curve diagram of the number of steps in the breakout round in the present invention;

[0029] Fig.10 It is a curve diagram of average reward of breakout rounds in the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose and advantages of the present invention more clearly understood, the present invention is specifically described below in conjunction with embodiments. It should be understood that the following text is only used to describe one or several specific embodiments of the present invention, and does not strictly limit the scope of protection of the specific claims of the present invention.

[0031] like Figure 1-10 As shown, a distributed multi-UAV intelligent breakout decision method for a three-dimensional urban environment, specifically, the present invention first considers the real-time interaction requirements of algorithm training, designs a digital twin city simulation platform with a city platform with multiple obstacles, then designs a target trajectory prediction network to realize trajectory prediction of multiple targets, and finally designs a multi-agent reinforcement learning algorithm based on maximum entropy to realize the rapid breakout of multiple UAVs in a three-dimensional urban environment. To this end. The technical solution adopted by the present invention is to simulate the scene of 5 breakout UAVs being captured by 10 capture UAVs based on a digital twin simulation platform, and the breakout UAVs reach the target breakout area, obtain real-time simulation data, and provide support for subsequent algorithm training, and then build a target prediction network, and design a multi-UAV random game model for the breakout task and a multi-UAV breakout intelligent decision algorithm based on maximum entropy. Finally, the target prediction network trained offline and the multi-UAV breakout decision network model based on maximum entropy are used to obtain the future trajectory of the target in real time and make online decisions on the breakout strategy of the breakout UAV, including the following steps:

[0032] S1: Build a digital twin city simulation environment, simulate the multi-UAV breakout and UAV roundup process through the simulation environment, and obtain real-time simulation data. First, in response to the design requirements of high-fidelity three-dimensional urban environment, use graphics processing software and game engine software such as RenderDoc, Blender and Unity3d to build a digital twin three-dimensional city; then design real-time interactive technology for three-dimensional urban breakthrough tasks to achieve real-time interaction between the digital twin city environment and the algorithm end. The high-quality city model ensures the restoration of the visual simulation and the real city; at the same time, the rigid body properties and obstacle properties of the buildings in the high-quality city model simulate the physical model in the real world, making the multi-UAV breakout algorithm more realistic.

[0033] S11: Construction of digital twin 3D city;

[0034] RenderDoc is a tool based on graphics API reading and analysis, which can call the interface of Google Earth to obtain rendering frame data and has data analysis and debugging functions. This patent uses RenderDoc to extract local 3D city models in Google Maps.

[0035] First, call the Google Earth API through RenderDoc and perform function calls and data transfer with it. During the graphics rendering process, use RenderDoc to read all the graphics data related to rendering during the loading process of the Google Earth 3D model. This data includes model rendering, state settings, shader data, and resource operations related to graphics rendering;

[0036] Through graphics API reading and analysis tools such as RenderDoc, the city graphics are read, and a complete rendering frame data structure is constructed based on the read graphics API calls and related data. This data structure includes the complete 3D city map information, rendering command sequence, map resource status, and shader information of the rendering frame;

[0037] Then, by Figure 1 The user interface shown in the figure analyzes and debugs the read rendering frame data, checks the order and parameters of rendering commands, views changes in resource status, debugs shader code, and views and analyzes the content of textures and buffer objects. By adjusting the details and problems in the rendering process, the performance and quality of the graphics application can be improved. After debugging is completed, select the required layers and models to render the frame and export the 3D city model in RDC format;

[0038] In addition, since Unity3D does not support 3D models in RDC format, they are first imported into Blender for optimized rendering and format conversion. Figure 2 As shown. Blender is an open source 3D computer graphics software used to create, render and animate 3D images, animations, visual effects and interactive applications. The incompatibility of the RDC format will lead to the loss of material maps. After re-matching, the color and material properties are optimized and adjusted and exported into a commonly used FBX format 3D model. After importing into the Unity3D project, a multi-UAV breakout mission scenario based on digital twins is realized, such as Figure 3 As shown;

[0039] S12: Real-time interactive framework of virtual simulation platform for breakout tasks. The real-time interactive framework of virtual simulation platform for breakout tasks designed in this application is mainly developed based on the Transmission Control Protocol (TCP), such as Figure 4 As shown, the terrain is simulated in real time, and the real-time simulation process includes the following steps:

[0040] S121: Breakthrough environment collects information about the drone's surroundings and sends it to the algorithm side, waiting for instructions from the algorithm side;

[0041] S122: The algorithm end performs calculations and decisions based on the received environmental information, and sends the obtained control amount to the breakout environment, waiting for the next step of information;

[0042] S123: The breakout environment is controlled according to the instructions of the algorithm end to obtain new environmental information.

[0043] S2: Design a global-individual multi-UAV trajectory prediction network to achieve the target's future trajectory prediction and provide a basis for subsequent decision-making. First, based on the motion characteristics of multiple capture UAVs in a three-dimensional urban environment, the impact of the UAV's historical trajectory and the interaction between the two parties are considered, and a global-individual multi-UAV trajectory prediction network model is designed. Then, a multi-UAV trajectory prediction loss function is designed. Based on the multi-UAV historical motion data provided by the environment, offline training of the multi-UAV target prediction network is achieved.

[0044] S21: Design of multi-UAV trajectory prediction network based on global-individual network, which consists of global network and individual network, such as Figure 5 As shown;

[0045] Among them, the global network is responsible for extracting global situation information, and the individual network is responsible for real-time prediction of the trajectories of multiple enemy encirclement drones;

[0046] The global network is responsible for extracting the global representation information of the interaction between the breakout drone and the capture drone. The global representation information is used as part of the individual network input to improve the accuracy of multi-drone trajectory prediction. t The calculation method is:

[0047]

[0048] Among them, E1, E2, W1, and W3 represent the embedding network and its weights in the global network, and W2 represents the weight of the Long Short-Term Memory (LSTM) network. and Respectively represent the output state and hidden state of the long short-term memory (LSTM) network, represents the hidden state of the global network at the initial moment, e t It represents the feature value extracted by the global network through the first Embedding layer. represents the target position of all captured drones in the first n steps, r t It shows the complex interactive information between the capture drone and the breakout drone;

[0049] The individual network corresponds to each opponent's capture drone. Based on the historical trajectory, speed information and global interaction representation of the target, it obtains the trajectory information of the next shot. The calculation formula is:

[0050]

[0051] Among them, E 1,i ,E 2,i ,W 1,i ,W 3,i Represent the embedded network and its weight of the i-th individual network, p i ,v i represent the i-th target, Respectively represent the hidden states of the individual network at the initial time and time t, It represents the feature value extracted by the individual network through the first Embedding layer, the target position and speed of all the captured drones in the first n steps, represents the trajectory prediction result of the next shot of target i at time step t;

[0052] S22: Offline training of multi-UAV trajectory prediction network based on global-individual network, which adopts the paradigm to train the multi-UAV trajectory prediction network

[0053] The loss function of each individual network is:

[0054]

[0055] Where ζ represents the individual network P ζ,i Parameters, Represents the predicted label of the target.

[0056] S3: Establish a multi-UAV random game model for breakout tasks, and then design a multi-UAV breakout decision algorithm based on maximum entropy, and complete the offline training of the algorithm. First, for the multi-UAV breakout decision problem in a three-dimensional urban environment, establish a multi-UAV random game model for breakout tasks, determine the state set, action set, and reward function in the breakout decision process, and realize the real-time expression of the breakout situation; then, consider the future trajectory of the target movement, collision avoidance, and other factors, design a multi-UAV breakout decision algorithm based on maximum entropy, make real-time decisions on multi-UAV breakout strategies, and complete the offline training of the algorithm;

[0057] The following steps are involved:

[0058] S31: Establish a multi-UAV random game model for breakout missions. The multi-UAV random game model includes the following contents:

[0059] The state set includes the following:

[0060] The positions and speeds of all drones (including breakout drones and capture drones), as well as the information of the breakout drone’s most recent collision point, are taken as the breakout drone’s state information s, and the state set S is:

[0061] s={p1,...,p i ,v1,...,v i ,p o}∈S (4)

[0062] Among them, p i represents the position of the drone, v i They represent the speed of the drone, p o Indicates the latest collision point information of the breakout drone;

[0063] Action set, including the following:

[0064] Taking acceleration as the action a of the breakout drone, the action set A is:

[0065] a={a x ,a y ,a z}∈A (5)

[0066] Among them, a x ,a y ,a z Indicates the acceleration in three directions;

[0067] The immediate benefit value R includes the following:

[0068] First, we design a reward function. Considering that using only global rewards to train drone agents will ignore the importance of individual contributions in teamwork, the reward function we design includes global rewards and local rewards. Each agent should have a common goal and its own goals that are consistent with the common goal to promote effective teamwork. The global-local rewards of the breakout drone are:

[0069] r i,t =r co,i +r self,i ∈R (6)

[0070]

[0071] Among them, r co,i ,r self,i They represent the global reward and the local reward respectively, dis(j,tar) represents the distance from the breakout drone to the target destination, and dis(j,obs) represents the shortest distance from the breakout drone to the obstacle;

[0072] will dis safeDefined as the safety buffer distance between the breakout drone and the nearest obstacle;

[0073] neib j ={neib j,1 ,neib j,2 ,neib j,3} represents the distance from the breakout drone to the three nearest enemy drones, ρ1, ρ2 are normalization factors used to balance the reward, and m is a static parameter set to 0.001;

[0074] Discount factors include the following:

[0075] The discount factor γ indicates the importance of future earnings relative to current earnings, and 0<γ<1;

[0076] When γ = 0, the breakout drone only considers the current benefits and ignores the future benefits. When γ = 1, it means that the future benefits and current benefits of the breakout drone are equally important.

[0077] During the breakout process, the target J(θ i ) is expressed as:

[0078]

[0079] Among them, θ i Represents the action network π i (a i |o i ) parameter, r i,t represents the immediate benefit of the breakout drone i at time t, represents the prediction result of the target, represents the current strategy entropy, α represents the weight of entropy. By increasing the breakout strategy entropy, while ensuring the benefits of a higher strategy, the strategy is made as random as possible, thereby improving the robustness of the breakout strategy.

[0080] S32: Establish a multi-UAV breakout decision algorithm design based on maximum entropy, such as Figure 6 As shown;

[0081] Based on the group trajectory prediction results, the multi-UAV breakout decision network consists of n action networks Judging Network Target Action Network Target Judgement Network The parameters are θ, φ, θ′, φ′. The action network inputs the state information and prediction information of the UAV and outputs the acceleration of the breakthrough UAV. The judgment network inputs the global state information, prediction information and action strategies of all UAVs, calculates the respective Q values, and approximates the cumulative benefits. The target network does not directly participate in the training, but the action network and the target network copy the parameters in a soft update manner.

[0082] Action network for each drone Updated to:

[0083]

[0084] Among them, p tar_1,...,k Represents the prediction results of k targets;

[0085] The judging network for each drone Updated to:

[0086]

[0087] Among them, φ, φ′ represent the parameters of the evaluation network and the target evaluation network of UAV i, y(r, s′, d) is the approximate target value in the temporal difference algorithm, and d represents whether the breakout mission is successful;

[0088] Target Action Network Target Judgement Network The update is:

[0089] θ i ′←τθ i +(1-τ)θ i ′,φ i ′←τφ i +(1-τ)φ i ′ (12)

[0090] Among them, τ is the soft update coefficient.

[0091] S4: Online decision-making, using the trained multi-UAV breakout decision network to realize real-time decision-making of multiple UAVs in a three-dimensional urban environment. Through the offline training process of S2 and S3, the parameters of the prediction network and the breakout decision network are determined. The decision target network, the judgment network and the judgment target network do not participate in the online decision-making process. At this time, the prediction network is responsible for realizing the trajectory prediction of multiple targets, and the breakout decision network is responsible for real-time calculation of the breakout strategies of our multiple UAVs.

[0092] In order to verify the effectiveness of the present invention, the present invention is designed as follows Figure 7The digital twin city environment for the breakout mission shown in the figure, in which 5 yellow drones represent breakout drones, 10 red drones represent encirclement drones, and the blue area represents the breakout area. When 3 drones arrive in this area, it means the mission is successful;

[0093] The invention adopts the maximum entropy multi-UAV breakout decision algorithm based on group prediction to effectively improve the efficiency of multiple UAVs in the breakout process. Figure 8-Figure 10 As shown in the figure, the group trajectory prediction method has greater advantages in success rate, round steps, and round average reward than the method without group trajectory prediction. The following is a specific analysis:

[0094] The success rate of the breakout is an indicator to measure the effectiveness of the breakout decision. Figure 8 As shown, MASAC_TP is the maximum entropy multi-UAV breakthrough decision algorithm for group prediction proposed in this patent, and MASAC is the traditional maximum entropy multi-UAV breakthrough decision algorithm. We can conclude that with the assistance of trajectory prediction, the algorithm proposed in this patent can converge faster, which means that compared with the general MASAC, the penetration UAV can learn the penetration strategy faster;

[0095] like Fig. 9 As shown, the average round reward is used to represent the average reward of the penetration drone in each step. The higher the average round reward curve is, the more effective the learned penetration strategy is.

[0096] The number of rounds is an efficiency indicator in the penetration mission. A lower number of rounds indicates higher efficiency, such as Fig.10 As shown, it can be concluded that the method based on trajectory prediction can complete the penetration task in fewer time steps.

[0097] This patent aims at the problem of multi-UAV penetration in a three-dimensional urban environment. First, the target trajectory prediction network is designed to obtain the future trajectory of the opponent's encirclement UAV. Then, by considering the complexity, collision avoidance, obstacle avoidance and other factors of the three-dimensional urban environment, a distributed multi-UAV breakthrough collaborative control method based on maximum entropy is studied to achieve rapid and effective penetration in a three-dimensional urban environment.

[0098] The Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm is a centralized training distributed execution multi-agent reinforcement learning framework first proposed by OpenAI in the United States. It uses an action network to calculate the strategy of each agent, and the evaluation network approximates the cumulative discounted reward of each agent, which better solves the non-stationary problem of the environment. Based on this method, this patent first developed a digital twin city simulation platform with multiple obstacles, which can build a digital city with high restoration, editability and real-time data interaction characteristics. It can realistically simulate multi-UAV penetration tasks and provide general training results. Then, this patent designs a target trajectory prediction network to obtain the future trajectories of multiple encirclement drones. Finally, this patent considers the complexity, collision avoidance, obstacle avoidance and other factors of the three-dimensional urban environment, and designs a distributed multi-UAV breakthrough collaborative control method based on maximum entropy, which improves the efficiency of multi-UAV rapid penetration in a three-dimensional urban environment.

[0099] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. Distributed multi-UAV intelligent breakout decision-making method for three-dimensional urban environment, Features: The following steps are involved: S1: Build a digital twin city simulation environment, simulate the multi-UAV breakout and UAV roundup process through the simulation environment, and obtain real-time simulation data; S2: Design a global-individual multi-UAV trajectory prediction network to achieve the future trajectory prediction of the target and provide a basis for subsequent decision-making; S3: Establish a multi-UAV random game model for breakout tasks, then design a multi-UAV breakout decision algorithm based on maximum entropy, and complete offline training of the algorithm; S4: Online decision making, using the trained multi-UAV breakout decision network to achieve real-time decision making of multiple UAVs in a three-dimensional urban environment.

2. The method according to claim 1, Features: The S1 comprises the following steps: S11: Construction of digital twin 3D city; First, the city graphics are read through the graphics API reading and analysis tool, and a complete rendering frame data structure is constructed based on the read graphics API calls and related data; Then, the read rendering frame data is analyzed and debugged to improve the performance and quality of the graphics application. After debugging is completed, the required layers and model rendering frames are selected and the 3D city model is exported in RDC format. S12: A real-time interactive framework of a virtual simulation platform for breakout missions, which simulates the terrain in real time.

3. The method according to claim 2, Features: The real-time simulation process in S12 comprises the following steps: S121: Breakthrough environment collects information about the drone's surroundings and sends it to the algorithm side, waiting for instructions from the algorithm side; S122: The algorithm end performs calculations and decisions based on the received environmental information, and sends the obtained control amount to the breakout environment, waiting for the next step of information; S123: The breakout environment is controlled according to the instructions of the algorithm end to obtain new environmental information.

4. The method according to claim 3, Features: The S2 comprises the following steps: S21: Design of multi-UAV trajectory prediction network based on global-individual network, which consists of global network and individual network; Among them, the global network is responsible for extracting global situation information, and the individual network is responsible for real-time prediction of the trajectories of multiple enemy encirclement drones; S22: Offline training of multi-UAV trajectory prediction network based on global-individual network, which adopts the paradigm to train the multi-UAV trajectory prediction network.

5. The method according to claim 4, Features: The global network in S21 is responsible for extracting the global representation information of the interaction between the breakout drone and the capture drone, where the global representation information r t The calculation method is: Among them, E 1 ,E 2 ,W 1 ,W 3 represents the embedded network and its weights in the global network, W 2 represents the weight of the long short-term memory network, and Represent the output state and hidden state of the long short-term memory network respectively, represents the hidden state of the global network at the initial moment, e t It represents the feature value extracted by the global network through the first Embedding layer. represents the target position of all captured drones in the first n steps, r t It shows the complex interactive information between the capture drone and the breakout drone; The individual networks in S21 correspond to each opponent's capture drone respectively. Based on the historical trajectory, speed information and global interaction representation of the target, the trajectory information of the next shot is obtained. The calculation formula is: Among them, E 1,i ,E 2,i ,W 1,i ,W 3,i Represent the embedded network and its weight of the i-th individual network, p i ,v i represent the i-th target, Respectively represent the hidden states of the individual network at the initial time and time t, It represents the feature value extracted by the individual network through the first Embedding layer, the target position and speed of all the captured drones in the first n steps, represents the trajectory prediction result of the next shot of target i at time step t; In S22, the loss function of each individual network is: Where ζ represents the individual network P ζ,i Parameters, Represents the predicted label of the target.

6. The method according to claim 5, Features: The S3 comprises the following steps: S31: Establish a multi-UAV random game model for breakout missions. The multi-UAV random game model includes the following contents: state set; Action Set; Immediate benefit value; Discount factor; S32: Establish a multi-UAV breakout decision algorithm design based on maximum entropy; Based on the group trajectory prediction results, the multi-UAV breakout decision network consists of n action networks Judging Network Target Action Network Target Judgement Network The parameters are θ, φ, θ′, φ′ respectively.

7. The method according to claim 6, Features: The state set in S31 includes the following contents: The positions, speeds, and the information of the most recent collision point of all the UAVs are taken as the state information s of the breakout UAVs, and the state set S is: s={p 1 ,...,p i ,v 1 ,...,v i ,p o }∈S (4) Among them, p i represents the position of the drone, v i They represent the speed of the drone, p o Indicates the latest collision point information of the breakout drone; The action set in S31 includes the following: Taking acceleration as the action a of the breakout drone, the action set A is: a={a x ,a y ,a z }∈A (5) Among them, a x ,a y ,a z Indicates the acceleration in three directions; The immediate benefit value R in S31 includes the following contents: First, we design a reward function, which includes global rewards and local rewards. The global-local rewards of the breakout drone are: r i,t ＝r co,i +r self,i ∈R (6) Among them, r co,i ,r self,i They represent the global reward and the local reward respectively, dis(j,tar) represents the distance from the breakout drone to the target destination, and dis(j,obs) represents the shortest distance from the breakout drone to the obstacle; will dis safe Defined as the safety buffer distance between the breakout drone and the nearest obstacle; neib j ={neib j,1 ,neib j,2 ,neib j,3 } represents the distance from the breakout drone to the three nearest enemy drones, ρ 1 ,ρ 2 Normalization factor used to balance rewards, m is a static parameter set to 0.001; The discount factors in S31 include the following: The discount factor γ indicates the importance of future earnings relative to current earnings, and 0<γ<1; When γ = 0, the breakout drone only considers the current benefits and ignores the future benefits. When γ = 1, it means that the future benefits and current benefits of the breakout drone are equally important. During the breakout process, the target J(θ i ) is expressed as: Among them, θ i Represents the action network π i (a i |o i ) parameter, r i,t represents the immediate benefit of the breakout drone i at time t, represents the prediction result of the target, represents the current policy entropy, and α represents the entropy weight; The action network of each drone in S32 Updated to: Among them, p tar_1,...,k Represents the prediction results of k targets; The judging network for each drone Updated to: Among them, φ, φ′ represent the parameters of the evaluation network and the target evaluation network of UAV i, y(r, s′, d) is the approximate target value in the temporal difference algorithm, and d represents whether the breakout mission is successful; Target Action Network Target Judgement Network The update is: i i ′←τθ i +(1-τ)θ i ′,φ i ′←tφ i +(1-τ)φ i ′ (12) Among them, τ is the soft update coefficient.

8. The method according to claim 7, Features: Through the offline training process of S2 and S3, the parameters of the prediction network and the breakout decision network are determined. At this time, the prediction network is responsible for realizing the trajectory prediction of multiple targets, and the breakout decision network is responsible for real-time calculation of the breakout strategies of multiple UAVs of our side.

Images