A multi-UAV roundup tactical method based on PER-IDQN

Abstract

The present invention provides a multi-UAV roundup tactical method based on PER‑IDQN, models a grid digital map and a UAV motion model, deploys a multi-UAV neural network model by using a deep Q network algorithm through the interaction between each UAV and the environment, and optimizes the algorithm model by using a priority experience playback strategy, and then constructs a state space, an action space, and a reward function to carry out a targeted design of a multi-UAV roundup tactical model. Finally, the multi-UAV roundup tactical model constructed can formulate effective roundup tactics in a complex obstacle environment to achieve the encirclement and capture of mobile targets. The present invention can achieve the roundup of mobile targets, effectively improve the sampling efficiency of experience samples, improve the problem of slow training rate of UAV decision models in complex mission scenarios, and finally construct a multi-UAV roundup tactical model with stronger stability, which can be applied to multi-UAV roundup and autonomous obstacle avoidance tasks in complex dynamic environments.

Description

Technical Field

[0001] The present invention relates to the field of multi-agent system and unmanned aerial vehicle (UAV) intelligent decision-making, and in particular to a multi-UAV roundup tactical method. Background Art

[0002] With its strong concealment and high security, drones provide a new model to meet the requirements of multi-machine coordination and low casualty rate required by modern information-based defense tactics. In the scenario where the enemy invades our airspace for illegal intelligence reconnaissance, it is of great significance to use multiple defense drones to form a multi-drone formation, so that the multi-drone formation can automatically surround and expel the target or accompany and monitor it according to the situation and environment.

[0003] There are few studies on multi-UAV capture tactics, which mainly use artificial intelligence methods to solve the position of the target in real time, and then plan the corresponding tracking path to approach and capture the target. Public patent CN112241173A proposes an intelligent planning method for multi-agent assembly points based on artificial potential fields. By converting the target into a virtual assembly point, and then using the artificial potential field model to calculate the repulsive force between agents and the repulsive force between agents and obstacles, the position and path information of the virtual assembly point of the agent is solved. However, this method does not take into account the large amount of calculation caused by model solving in a dynamic environment, and cannot guarantee the real-time performance of multi-agent decision-making. In recent years, the development of deep reinforcement learning technology has provided new ideas for real-time online intelligent decision-making of unmanned systems. Public patent CN113625775A proposes a multi-UAV capture method combining state prediction and DDPG. The drone state is predicted based on the least squares method, and then the drone model is trained using the deep reinforcement learning DDPG algorithm, and finally deployed in a multi-UAV system to achieve multi-UAV capture decision-making. However, when this method is used to train the drone decision-making model, the amount of training sample data is large and the types of variables are complex, resulting in low training efficiency. The final multi-drone capture model has poor stability and certain limitations.

[0004] The priority experience replay strategy is a deep reinforcement learning optimization method that calculates and prioritizes the importance of each experience sample, improves the utilization rate of high-priority experience samples, and ultimately improves the training speed of the agent. Therefore, how to introduce the priority experience replay strategy into the multi-agent deep reinforcement learning method and combine it with the complex multi-UAV capture tactical model to improve the autonomous behavior of each UAV, and finally achieve the capture of the target through collaborative decision-making, has become a difficult problem in the application of deep reinforcement learning in the field of multi-UAV intelligent decision-making. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a multi-UAV roundup tactical method based on PER-IDQN. The present invention is a multi-UAV roundup tactical method based on a prioritized experience replay strategy independent deep Q learning network (Prioritized Experience Replay Independent Deep Q-learning Network, PER-IDQN). Specifically, a grid digital map and a UAV motion model are modeled, and through the interaction between each UAV and the environment, a deep Q network (Deep Q Network, DQN) algorithm is used to deploy the multi-UAV neural network model, and the prioritized experience replay strategy (Prioritized Experience Replay, PER) is used to optimize the algorithm model, and then the state space, action space, and reward function are constructed to carry out targeted design of the multi-UAV roundup tactical model. Finally, the multi-UAV roundup tactical model constructed can formulate effective roundup tactics in a complex obstacle environment to achieve the encirclement and capture of mobile targets.

[0006] The technical solution adopted by the present invention to solve the technical problem comprises the following steps:

[0007] Step 1: Construct raster digital map model and drone model;

[0008] Step 2: Construct a multi-UAV roundup decision model based on the PER-IDQN algorithm;

[0009] Step 3: Based on the PER-IDQN algorithm, a multi-UAV capture decision model is constructed and trained; each UAV inputs its state information into the neural network, which is then fitted through the trained PER-IDQN neural network to output the UAV flight actions. Each capture UAV achieves the capture of the target through collaborative decision-making.

[0010] The steps of constructing the grid digital map model and the drone model are:

[0011] Step 1-1: To facilitate the quantification of the specific location of the drone, the entire airspace is evenly divided into a number of grids. Each grid is set to be a square with a length of l km. The mission scene is a*l km×b*l km. The total width of the mission scene is l width =a*l km, the total length is l length = b*l km;

[0012] Step 1-2: Set the speed of the capture drone to l km / time step, and the speed of the target drone to n*l km / time step;

[0013] Step 1-3: Set the size of the drone's action space to 4, that is, the drone can only move in four directions: up, down, left, and right at each step;

[0014] Step 1-4: Set the detection range of each drone to a circular area with a radius of 1 km, which is approximately the surrounding nine-square grid area centered on the drone in the grid scene.

[0015] The steps of constructing a multi-UAV roundup decision model based on the PER-IDQN algorithm in step 2 are as follows:

[0016] Step 2-1: Set the action space A for capturing drones as:

[0017] A=[(0,-l),(0,l),(-l,0),(0,l)]

[0018] (0,-l),(0,l),(-l,0),(0,l) represent the four actions of the drone moving down, up, left, and right, and l represents the length of each grid;

[0019] Step 2-2: Set the state space S of the drone capture to:

[0020] S=[S uav ,S teamer ,S obser ,S target ,S finish ]

[0021] Where S uav ,S teamer ,S obser ,S target ,S finish Respectively represent the drone's own status information, other friendly drone information, drone detection information, target information and mission status information;

[0022] Specifically, for the i-th drone in the multi-capture drone system, its own state information is set as:

[0023]

[0024] x i and i Represents the coordinate information of the i-th drone;

[0025] For the i-th drone, set the available friendly drone status information for:

[0026]

[0027] Among them, x i and iThey represent the horizontal and vertical coordinate values ​​of the i-th captured drone, and n represents the number of drones;

[0028] Set the observation information of the capture drone i to in:

[0029]

[0030] o m It is the detection indication, which indicates the exploration information of the surrounding nine-square grid positions by the capture drone;

[0031] In addition, combined with the relative distance and orientation information of the target relative to our drone i, for the i-th capture drone, the target information that can be obtained is set for:

[0032]

[0033] Among them, d i and θ i They represent the distance and relative azimuth between our capture drone and the target, x e and e Indicates the horizontal and vertical coordinate values ​​of the escape target;

[0034] In addition, in order to help the capture drones to effectively complete the capture of the target, for the i-th capture drone, the sub-state quantity is set

[0035]

[0036] Indicates whether the capture of the target is completed;

[0037] Step 2-3: Considering the three decision-making processes of maneuvering approach to the target, cooperative capture, and autonomous obstacle avoidance in the multi-UAV capture tactics, for each individual capture UAV, set its reward function R as:

[0038] R=σ1r pos +σ2r safe +σ3r effi +σ4r task

[0039] Among them, r pos ,r safe ,r effi ,r task They represent position reward, safe flight reward, efficient flight reward and task completion reward respectively, and σ1~σ4 are the corresponding weight values ​​of each reward;

[0040] Specifically, the position sub-reward is set as:

[0041] r pos=(|x e -x i |+|y e -y i |)-(|x e -x i |+|y e -y i |)′

[0042] Among them, (|x e -x i |+|y e -y i |) and (|x e -x i |+|y e -y i |)′ represents the distance between the capture drone and the target at the current moment and the next moment respectively;

[0043] Set the sub-rewards for safe flight of the roundup drone to:

[0044]

[0045] Set the sub-reward for efficient flight of the roundup drone to:

[0046] r effi =-n stay

[0047] n stay Indicates the number of times the capture drone stays at the current grid position;

[0048] Set the sub-reward items for completing the drone capture mission to:

[0049]

[0050] Step 2-4: Set the multi-drone capture judgment condition as follows: when the target is one unit grid distance away from each capture drone, it is considered that the target cannot escape and the capture mission is completed.

[0051] Step 3: Based on the PER-IDQN algorithm, a multi-UAV roundup decision model is constructed and trained;

[0052] Step 3-1: For each captured drone, construct the main BP neural network hidden layer θ in the PER-IDQN algorithm i and the state-action value function The value function input parameter are the state and behavior of UAV i at time t, and the main network parameter θ i Copy to target networkθ i′ θ i →θi′ , where i represents the drone serial number;

[0053] Step 3-2: Set the experience replay queue size to M, the discount factor to γ, the maximum number of rounds to E, the maximum number of steps per round to T, and the experience extraction size to N batch , set the number of rounds e = 0;

[0054] Step 3-3: Initialize the states s1,…,s of n capture drones n , update the current time to t=0;

[0055] Step 3-4: Generate a random number z, and for each drone i, perform the following actions:

[0056]

[0057] Among them, ε greedy is the greed coefficient, The action corresponding to the maximum Q value output of the main network;

[0058] Step 3-5: Execute the action set a1,…,a n , calculate the reward value r1,…,r n , update the state to s′1,…,s′ n , and calculate the priorities p1,…,p respectively n , stored together in the experience replay queue;

[0059] Step 3-6: Based on Collection N batch Samples are based on, where j represents the number of the empirical sample drawn, p j Indicates its priority, and parameter α is a parameter used to adjust the degree of sample priority sampling;

[0060] Calculate the weight coefficient w for importance sampling j :

[0061] w j =(M·P(i)) -β / max i w i

[0062] β is a hyperparameter used to adjust the impact of importance sampling on the PER algorithm and model convergence rate;

[0063] Calculate the time difference error at the current moment:

[0064]

[0065] in, represents the reward obtained by drone i at time t+1;

[0066] Calculate the target value to obtain the target value

[0067]

[0068] Among them, γ is the reward discount factor, j is the sample serial number, and θ i′ represents the target network of the i-th agent;

[0069] Combine the importance weight w j , and update the parameters of the current network L(θ i ) based on minimizing the loss function:

[0070]

[0071] Step 3-7: Update the target network parameters of each UAV agent respectively:

[0072] θ i′ ← τθ i +(1 - τ)θ i′

[0073] τ represents the update ratio coefficient;

[0074] Step 3-8: Increment the update step t by 1 and perform the determination: When t < T and the multi-UAV encirclement determination condition is not met, go to Step 3-4; otherwise, go to Step 3-9;

[0075] Step 3-9: Increment the number of rounds e by 1 and perform the determination: If e < E, update to Step 3-3; otherwise, the training is complete and go to Step 3-10;

[0076] Step 3-10: Terminate the training process of the PER-IDQN network and save the current network parameters; load the saved parameters into the multi-UAV encirclement system. At each moment, each UAV inputs the state information into the neural network respectively, and through the PER-IDQN neural network obtained by training for fitting, outputs the UAV flight actions, and each encircling UAV realizes the encirclement of the target through collaborative decision-making.

[0077] The beneficial effect of the present invention lies in the proposed multi-UAV encirclement tactical method based on PER-IDQN, and the advantages are specifically reflected in:

[0078] (1) The constructed multi-UAV encirclement decision-making system does not require manual individual specification of the tactics for each UAV. The UAVs can complete tactical and task coordination through environmental perception and information sharing by themselves, and the finally formulated multi-UAV encirclement tactics can achieve the encirclement of mobile targets.

[0079] (2) This paper introduces the Prioritized Experience Replay (PER) strategy into the deep reinforcement learning IDQN algorithm, which effectively improves the sampling efficiency of experience samples and improves the problem of slow training rate of drone decision-making models in complex mission scenarios. The multi-drone capture tactical model finally constructed is more stable and can be applied to multi-drone capture and autonomous obstacle avoidance tasks in complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 Schematic diagram of drone detection.

[0081] Figure 2 Schematic diagram of the positional relationship between the capture drone and the target.

[0082] Figure 3 Schematic diagram of multi-UAV roundup model training based on PER-IDQN.

[0083] Figure 4 Schematic diagram of the encirclement and capture of multiple drones. DETAILED DESCRIPTION

[0084] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0085] The present invention proposes a multi-UAV roundup tactical method based on PER-IDQN. The overall process is as follows: Figure 3 The following is a further clear and complete description of the technical solution in conjunction with the accompanying drawings and specific embodiments:

[0086] Step 1: Construct raster digital map model and drone model;

[0087] Step 1-1: To facilitate the quantification of the specific location of the drone, the entire airspace is divided into a number of grids. Each grid is set to be a square with a length of 0.1 km. The total width and total length of the mission scene are set to be l width = 8 km and l length = 4 km;

[0088] Step 1-2: Set the speed of the capture drone to 0.1 km / time step and the speed of the target drone to 0.2 km / time step;

[0089] Step 1-3: Set the size of the drone's action space to 4, that is, the drone can only move in four directions: up, down, left, and right at each step;

[0090] Step 1-4: Set the detection range of each drone to a circular area with a radius of 0.1 km, which is approximately the surrounding nine-square grid area centered on the drone in the grid scene;

[0091] Step 2: Construct a multi-UAV roundup decision model based on the PER-IDQN algorithm;

[0092] Step 2-1: Set the action space A for capturing drones as:

[0093] A=[(0,-l),(0,l),(-l,0),(0,l)]

[0094] (0,-l),(0,l),(-l,0),(0,l) represent the four actions of the drone moving down, up, left, and right, and l represents the length of each grid;

[0095] Step 2-2: Set the state space S of the drone capture to:

[0096] S=[S uav ,S teamer ,S obser ,S target ,S finish ]

[0097] Where S uav ,S teamer ,S obser ,S target ,S finish Respectively represent the drone's own status information, other friendly drone information, drone detection information, target information, and mission status information;

[0098] Specifically, for the i-th drone in the multi-capture drone system, its own state information is set as:

[0099]

[0100] x i and i Represents the coordinate information of the i-th drone;

[0101] For the i-th drone, set the friendly drone status information that it can obtain for:

[0102]

[0103] Among them, x i and i They represent the horizontal and vertical coordinate values ​​of the i-th captured drone, and n represents the number of drones;

[0104] Set the observation information of the capture drone i to in:

[0105]

[0106] om is the detection indication, which indicates the exploration information of the surrounding nine-square grid positions by the capture drone; the drone detection information is as follows Figure 1 As shown;

[0107] In addition, combined with the relative distance and orientation information of the target relative to our drone i, for the i-th capture drone, set the target information that can be obtained for:

[0108]

[0109] Among them, d i and θ i They represent the distance and relative azimuth between our capture drone and the target, x e and e Indicates the horizontal and vertical coordinate values ​​of the escaping target; the position relationship between the capture drone and the target is as follows Figure 2 As shown;

[0110] In addition, in order to help the capture drones to effectively complete the capture of the target, for the i-th capture drone, the sub-state quantity is set

[0111]

[0112] Indicates whether the capture of the target is completed;

[0113] Step 2-3: Considering the decision-making process of maneuvering approach to the target, cooperative capture, autonomous obstacle avoidance, etc. in the multi-UAV capture tactics, for each individual capture UAV, set its reward function R as:

[0114] R=σ1r pos +σ2r safe +σ3r effi +σ4r task

[0115] Among them, r pos ,r safe ,r effi ,r task They represent location reward, safe flight reward, efficient flight reward, and task completion reward, respectively. 1～4 The corresponding weight value of each reward;

[0116] Specifically, the position sub-reward is set as:

[0117] r pos =(|x e -x i |+|y e -y i |)-(|x e -xi |+|y e -y i |)′

[0118] Among them, (|x e -x i |+|y e -y i |) and (|x e -x i |+|y e -y i |)′ represents the distance between the capture drone and the target at the current moment and the next moment respectively;

[0119] Set the sub-rewards for safe flight of the roundup drone to:

[0120]

[0121] Set the sub-reward for efficient flight of the roundup drone to:

[0122] r effi =-n stay

[0123] n stay Indicates the number of times the capture drone stays at the current grid position;

[0124] Set the sub-reward items for completing the drone capture mission to:

[0125]

[0126] Step 2-4: Set the multi-drone capture judgment conditions: when the target is one unit grid away from each capture drone, it is considered that the target cannot escape and the capture mission is completed;

[0127] Step 3: Based on the deep reinforcement learning PER-IDQN algorithm, build a multi-UAV roundup decision model and train it;

[0128] Step 3-1: For each captured drone, construct the main BP neural network hidden layer θ in the PER-IDQN algorithm i and the state-action value function The value function input parameter are the state and behavior of UAV i at time t, and the main network parameter θ i Copy to target networkθ i′ θ i →θ i′ , where i represents the drone serial number;

[0129] Step 3-2: Set the experience replay queue size M, discount factor γ, maximum number of rounds E, maximum number of steps per round T, and experience extraction size N batch , set the number of rounds e = 0;

[0130] Step 3-3: Initialize the states s1,…,s of n capture drones n , update the current time to t=0;

[0131] Step 3-4: Generate a random number z, and for each drone i, perform the following actions:

[0132]

[0133] Among them, ε greedy is the greed coefficient, The action corresponding to the maximum Q value output of the main network;

[0134] Step 3-5: Execute the action set a1,…,a n , calculate the reward value r1,…,r n , update the state to s′1,…,s′ n , and calculate the priorities p1,…,p respectively n , stored together in the experience replay queue;

[0135] Step 3-6: Based on Collection N batch Samples are based on, where j represents the number of the empirical sample drawn, p j Indicates its priority, and parameter α is a parameter used to adjust the degree of sample priority sampling;

[0136] Calculate the weight coefficient w for importance sampling j :

[0137] w j =(M·P(i)) -β / max i w i

[0138] M is the size of the experience replay queue, and β is a hyperparameter used to adjust the impact of importance sampling on the PER algorithm and model convergence rate;

[0139] Calculate the time difference error at the current moment:

[0140]

[0141] in, represents the reward obtained by drone i at time t+1;

[0142] Calculate the target value and get the target value Yt i :

[0143]

[0144] Among them, γ is the reward discount factor, j is the sample serial number, and θ i′ represents the target network of the i-th agent;

[0145] Combined with the importance weight w j , update the parameters of the current network L(θ i ):

[0146]

[0147] Step 3-7: Update the target network parameters of each UAV agent respectively:

[0148] θ i′ ← τθ i +(1 - τ)θ i′

[0149] τ represents the update ratio coefficient;

[0150] Step 3-8: Update the step size t = t + 1, and perform the determination: When t < T and the multi-UAV encirclement determination condition shown in Step 2-4 is not satisfied, go to Step 3-4; otherwise, go to Step 3-9;

[0151] Step 3-9: Update the number of rounds e = e + 1, and perform the determination: If e < E, then update to Step 3-3; otherwise, the training is completed and go to Step 3-10;

[0152] Step 3-10: Terminate the PER-IDQN network training process and save the current network parameters; load the saved parameters into the multi-UAV encirclement system. At each moment, each UAV inputs the state information into the neural network respectively, and through the trained PER-IDQN neural network for fitting, outputs the UAV flight actions, and each encircling UAV realizes the encirclement of the target through collaborative decision-making.

[0153] To better illustrate the superiority of the method of the present invention, this embodiment case was tested in different scenarios. Specifically, in the task scenario with a grid number of 80×40, while keeping the obstacle movement rate of 10% unchanged, different obstacle coverage rates were set and tested respectively, and the test results are shown in Table 1.

[0154] Table 1 Multi-UAV encirclement performance under different environmental obstacle coverage rates

[0155]

[0156]

[0157] As can be seen from the table above, as the coverage rate of environmental obstacles increases, the time it takes to capture multiple drones increases; when the obstacle coverage rate increases to 0.10 or above, compared with the traditional IDQN algorithm, the average simulation step length of the multi-drone capture tactics based on the PER-IDQN algorithm is smaller, which means that the capture tactics formulated by the PER-IDQN algorithm for multiple drones facing complex obstacle environments are more effective and can capture the target in a shorter time. Figure 4 shown.

[0158] In summary, the present invention proposes a multi-UAV capture tactical method based on PER-IDQN, which uses offline learning to train the neural network, stores the data generated during training in the experience pool, provides learning samples for the optimization of the neural network, and combines the multi-UAV maneuver control and collaborative capture mission requirements to design the UAV actions and states, thereby realizing intelligent decision-making control of multiple UAVs.

[0159] Based on the multi-UAV roundup tactical method proposed in the present invention, the model training efficiency is high, and the constructed multi-UAV roundup tactical model can be applied to and in complex dynamic scenes to improve the execution efficiency of the multi-UAV roundup tactic.

[0160] The above is only a preferred embodiment of the present invention. It should be pointed out that the embodiments of the present invention are not limited to the above-mentioned implementation methods. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made without departing from the principles of the present invention. It should be noted that any equivalent replacement, modification and improvement made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A multi-UAV roundup tactical method based on PER-IDQN, characterized in that It includes the following steps: Step 1: Construct a grid digital map model and a UAV model; Step 2: Construct a multi-UAV encirclement decision-making model based on the PER-IDQN algorithm; The steps of constructing a multi-UAV encirclement decision-making model based on the PER-IDQN algorithm in Step 2 are as follows: Step 2-1: Set the action space A of the encircling UAVs as: A＝[(0,-l),(0,l),(-l,0),(0,l)] (0, -l), (0, l), (-l, 0), (0, l) represent the four actions of the UAV moving downward, upward, left, and right, and l represents the length of each grid; Step 2-2: Set the state space S of the encircling UAVs as: S=[S uav ,S teamer ,S obser ,S target ,S finish ] Where S uav ,S teamer ,S obser ,S target ,S finish Respectively represent the drone's own status information, other friendly drone information, drone detection information, target information and mission status information; Specifically, for the i-th UAV in the multi-encircling UAV system, set its own state information as: x i and i Represents the coordinate information of the i-th drone; For the i-th drone, set the available friendly drone status information for: Among them, x i and i They represent the horizontal and vertical coordinate values ​​of the i-th captured drone, and n represents the number of drones; Set the observation information of the capture drone i to in: o m It is the detection indication, which indicates the exploration information of the surrounding nine-square grid positions by the capture drone; In addition, combined with the relative distance and orientation information of the target relative to our drone i, for the i-th capture drone, the target information that can be obtained is set for: Among them, d i and θ i They represent the distance and relative azimuth between our capture drone and the target, x e and e Indicates the horizontal and vertical coordinate values ​​of the escape target; In addition, in order to help the capture drones to effectively complete the capture of the target, for the i-th capture drone, the sub-state quantity is set Indicates whether the encirclement of the target is completed; Step 2-3: Considering the three decision-making processes of maneuvering close to the target, collaborative encirclement, and autonomous obstacle avoidance in the multi-UAV encirclement tactic, for each individual encircling UAV, set up its reward function R as: R=σ1r pos +σ2r safe +σ3r effi +σ4r task Among them, r pos ,r safe ,r effi ,r task They represent position reward, safe flight reward, efficient flight reward and task completion reward respectively, and σ1~σ4 are the corresponding weight values ​​of each reward; Specifically, set the position sub-reward as: r pos =(|x e -x i |+|and e -and i |)-(|x e -x i |+|and e -and i |)′ Among them, (|x e -x i |+|y e -y i |) and (|x e -x i |+|y e -y i |)′ represents the distance between the capture drone and the target at the current moment and the next moment respectively; Set the safe flight sub-reward of the encircling UAV as: Set the efficient flight sub-reward of the encircling UAV as: r effi ＝-n stay n stay Indicates the number of times the capture drone stays at the current grid position; Set the task completion sub-reward item of the encircling UAV as: Step 2-4: Set the multi-UAV encirclement determination condition as: when the distance between the target and each encircling UAV is one unit grid distance, it is regarded that the target cannot escape and the encirclement task is completed; Step 3: Based on the PER-IDQN algorithm, construct a multi-UAV encirclement decision-making model and train it; each UAV respectively inputs the state information into the neural network, fits through the trained PER-IDQN neural network, outputs the UAV flight actions, and each encircling UAV realizes the encirclement of the target through collaborative decision-making.

2. The multi-UAV encirclement tactic method based on PER-IDQN according to claim 1, characterized in that: The steps of constructing the grid digital map model and the UAV model are as follows: Step 1-1: To facilitate the quantification of the specific location of the drone, the entire airspace is evenly divided into a number of grids. Each grid is set to be a square with a length of l km. The mission scene is a*l km×b*l km. The total width of the mission scene is l width =a*l km, the total length is l length = b*l km; Step 1-2: Set the speed of the encircling UAV as l km / time step, and the speed of the target UAV as n*l km / time step; Step 1-3: Set the size of the UAV action space as 4, that is, the UAV can only move in four directions: up, down, left, and right in each step; Step 1-4: Set the detectable range of each UAV as a circular area with a radius of l km, that is, approximately the surrounding nine-grid area centered on the UAV in the grid scene.

3. The multi-UAV encirclement tactic method based on PER-IDQN according to claim 1, characterized in that: Step 3: Based on the PER-IDQN algorithm, construct a multi-UAV encirclement decision-making model and train it; Step 3-1: For each captured drone, construct the main BP neural network hidden layer θ in the PER-IDQN algorithm i and the state-action value function The value function input parameter are the state and behavior of UAV i at time t, and the main network parameter θ i Copy to target networkθ i′ θ i →θ i′ , where i represents the drone serial number; Step 3-2: Set the experience replay queue size to M, the discount factor to γ, the maximum number of rounds to E, the maximum number of steps per round to T, and the experience extraction size to N batch , set the number of rounds e = 0; Step 3-3: Initialize the states s1,…,s of n capture drones n , update the current time to t=0; Step 3-4: Generate a random number z, and for each UAV i, execute the action: Among them, ε greedy is the greed coefficient, The action corresponding to the maximum Q value output of the main network; Step 3-5: Execute the action set a1,…,a n , calculate the reward value r1,...,r n , update the state to s′1,...,s′ n , and calculate the priorities p1,...,p respectively n , stored together in the experience replay queue; Step 3-6: Based on Collection N batch Samples are based on, where j represents the number of the empirical sample drawn, p j Indicates its priority, and parameter α is a parameter used to adjust the degree of sample priority sampling; Calculate the weight coefficient w for importance sampling j : w j =(M·P(i)) -β / max i w i β is a hyperparameter used to adjust the influence calculation of importance sampling on the PER algorithm and the model convergence rate; Calculate the temporal difference error at the current moment: in, represents the reward obtained by drone i at time t+1; Calculate the target value and get the target value Y t i : Among them, γ is the reward discount factor, j is the sample number, and θ i′ represents the target network of the i-th agent; Combined with the importance weight w j , based on minimizing the loss function, update the parameters L(θ of the current network i ): Step 3-7: Update the target network parameters of each UAV agent respectively: i i′ ←tth i +(1-τ)θ i′ τ represents the update ratio coefficient; Step 3-8: Increment the update step t by 1, and execute the determination: when t < T and the multi-UAV encirclement determination condition is not satisfied, enter Step 3-4; otherwise, enter Step 3-9; Step 3-9: Increment the round number e by 1, and perform the judgment: If e < E, update to Step 3-3; otherwise, the training is completed and proceed to Step 3-10; Step 3-10: Terminate the training process of the PER-IDQN network and save the current network parameters; load the saved parameters into the multi-UAV pursuit system. At each moment, each UAV inputs the state information into the neural network respectively, and performs fitting through the trained PER-IDQN neural network to output the UAV flight actions. Each pursuit UAV realizes the pursuit of the target through collaborative decision-making.

Images