A multi-unmanned aerial vehicle cooperative tracking method based on deep reinforcement learning

Abstract

The application discloses a kind of multi-unmanned aerial vehicle cooperative tracking methods based on deep reinforcement learning, belong to multi-agent control field.The method is first modeled on unmanned aerial vehicle war simulation platform to multi-unmanned aerial vehicle combat system;Subsequently, set the fixed rule action of multi-unmanned aerial vehicle cooperative combat and the initial position and tracking condition of multi-unmanned aerial vehicle;Then, set the state, action and reward function of unmanned aerial vehicle agent reinforcement learning;Finally, in the case of facing different opponents, unmanned aerial vehicle agent is trained using QMIX algorithm, and according to the training effect adjustment hyperparameter, the purpose of unmanned aerial vehicle cooperative tracking is realized.

Description

Technical Field

[0001] This invention relates to a multi-UAV cooperative tracking method based on deep reinforcement learning. This method achieves cooperative tracking of multiple UAVs by combining control methods and reinforcement learning, and belongs to the field of multi-agent control. Background Technology

[0002] Unmanned aerial vehicles (UAVs) have garnered widespread attention in both civilian and military fields due to their lightweight, small size, high maneuverability, good stealth, strong adaptability, and ease of operation. In recent years, UAVs have occupied a key position in various missions such as intelligence reconnaissance, target search and tracking, and target attack, achieving considerable success.

[0003] However, as the battlefield environment faced by drones becomes increasingly complex, in the complex and ever-changing information-based battlefield environment, individual drones face many limitations when performing reconnaissance or attack missions, such as reconnaissance angle and range, kill radius and destruction capability, which restricts the realization of combat effectiveness and makes it increasingly difficult for individual drones to complete missions.

[0004] Because multi-UAV systems can form coordinated and orderly collective movement patterns and respond quickly and consistently to external stimuli, they possess advantages such as strong self-organization, high coordination, and strong stability, and also have a strong adaptability to the environment. Therefore, more and more people have begun to study the cooperative combat of multi-UAVs.

[0005] However, the cooperative combat of multiple UAVs is a complex problem, and it is difficult to achieve good adaptability of multiple UAVs in complex environments by relying solely on the experience and knowledge of the designers. Therefore, reinforcement learning algorithms, as a feasible technical route to achieve good adaptability of multiple UAVs in complex environments, have become a research hotspot in the field of cooperative multi-UAV combat.

[0006] In the field of multi-agent deep reinforcement learning (MARL), common methods in fully cooperative environments include the COMA algorithm, VDN algorithm, and QMIX algorithm. Based on the QMIX algorithm, by setting appropriate rules and corresponding rewards, a strategy for multi-UAV cooperative tracking can be implemented, which has certain application prospects and significance. Summary of the Invention

[0007] Technical issues:

[0008] The multi-UAV cooperative combat strategy based on deep reinforcement learning mainly studies how multiple UAVs continuously improve their own and the system's strategies through reinforcement learning training in complex environments, and how they conduct cooperative combat through certain strategies. This invention patent realizes cooperative combat of a multi-UAV system under certain conditions, mainly using the QMIX algorithm in deep reinforcement learning. By setting the action space, state space, and reward function, cooperative tracking of multiple UAVs is achieved.

[0009] Technical solution:

[0010] A multi-UAV cooperative tracking method based on deep reinforcement learning, characterized by the following steps:

[0011] Step 1: Establish a multi-drone combat model on the drone combat simulation platform;

[0012] Step 2: Set fixed rules and actions for multi-drone collaborative operations;

[0013] Step 3: Set the initial positions and tracking conditions for multiple drones;

[0014] Step 4: Set the state, action, and reward functions for the drone agent's reinforcement learning;

[0015] Step 5: Train the UAV agent using the QMIX algorithm and adjust the hyperparameters based on the training results;

[0016] Step 6: Repeat step 5 when facing different opponents to achieve the goal of multi-drone collaborative tracking.

[0017] Furthermore, the multi-UAV combat model established on the UAV combat simulation platform in step 1 is as follows:

[0018] Step 1.1 Establish a two-dimensional absolute coordinate system

[0019] A two-dimensional absolute coordinate system is established with the bottom left corner of the currently selected battle map as the origin, the horizontal direction as the x-axis, and the vertical direction as the y-axis.

[0020] Step 1.2 Establish a multi-UAV combat system

[0021] To establish a multi-UAV combat system containing n UAVs, let A represent the intelligent agent of each UAV. Then, this multi-UAV combat system can be represented by the set D = {A1, A2, ..., A...}. n}express.

[0022] Step 1.3 Establish the UAV intelligent agent model

[0023] For each UAV A in the multi-UAV system D, the action selected at time step t consists of four modules: movement, detection, jamming, and attack.

[0024] For the movement module, the UAV selects a direction within the 0-359° range in a two-dimensional plane coordinate system and moves a unit step in this direction per unit time. The direction selected by the UAV at time step t can be denoted as F. 1t (A i );

[0025] For the detection module, assuming the total number of radar frequencies is m, after the UAV selects any radar frequency from the radar frequency list, it will detect whether there are enemy UAVs within a fixed sector area with its orientation as the axis, 60° to the left and right, and a distance of d1, and record the detected enemy UAVs in the detection list L. i In this context, the radar frequency selected by the UAV at time step t can be denoted as F. 2t (A i );

[0026] For the jamming module, after the UAV selects any frequency from the radar frequency list, it jams the enemy UAV's radar at the selected frequency within a fixed sector radius of 15° to the left and right of its direction, with a distance of d1, thus disabling the enemy UAV's detection capability. Alternatively, the UAV can choose to activate jamming, jamming all enemy UAV radars at all frequencies within a fixed sector radius of d1 to the left and right of its direction. Let F be the jamming frequency selected by the UAV at time step t. 3t (A i Then we have:

[0027]

[0028] For the attack module, the UAV launches a short-range missile (distance d2) or a long-range missile (distance d3) at an enemy UAV in the enemy UAV detection list. Let L be the enemy UAV selected by the UAV for attack in the detection list. i (1≤i≤n), the type of missile selected by the UAV to launch is p, (when p=0, the UAV does not launch a missile; when p=1, the UAV launches a short-range missile; when p=2, the UAV launches a long-range missile), whether the UAV has any other missiles of this type is w(p), (when w(p)=0, the UAV does not have any missiles of this type; when w(p)=1, the UAV has missiles of this type; the default is w(0)=0), let the attack action selected by the UAV at time step t be F. 4t (A i Then we have:

[0029] F 4t (A i)=w(p)×((p-1)×n+Li)

[0030] Therefore, the action space of the unmanned aerial vehicle agent at time step t can be defined by the set {F1(A)} i ), F2(A i ), F3(A i ), F4(A i )}express.

[0031] For each UAV A in the multi-UAV system D, its state at time step t is represented as follows:

[0032] Drone A i Position information at the current time step t (x) it y it ); Drone A i List of enemy drones detected at the current time step t L it The enemy drone detection list L is obtained by aggregating the detection lists of each drone from multiple drone systems. Dt (L Dt =L 1t ∪L 2t ∪...∪L nt ); Drone A i The remaining short-range missiles w at the current time step t it1 and the number of long-range missiles w it2 .

[0033] Therefore, the state space of the unmanned aerial vehicle agent at time step t can be represented by the set {x}. it y it L it w it1 w it2}express.

[0034] This led to the development of a multi-drone combat model.

[0035] Furthermore, the specific steps for setting fixed rules for multi-UAV collaborative operations described in step 2 are as follows:

[0036] Step 2.1 Set the fixed rule actions of the detection module

[0037] For the detection module, since the probability of the enemy drone interfering with the randomly changing radar frequency of our drone within a continuous step is low, and the effectiveness of the detection module depends more on the direction selection of the drone in the movement module, the radar frequency is generated using random numbers at each time step.

[0038] Let the total number of radar frequencies of the UAV be m, then the radar frequency selected by the UAV at time step t can be expressed as:

[0039] F 2t (A i ) = random(1, m)

[0040] Step 2.2 Set the fixed rule actions of the interference module

[0041] For the jamming module, a conditional judgment is introduced. If our UAV continuously detects the same enemy UAV within τ time steps, then full-channel jamming is used to prevent it from detecting our UAV. If our UAV does not detect the enemy UAV or does not continuously detect the enemy UAV, then random numbers are used to generate jamming frequencies.

[0042] Therefore, the interference frequency selected by the UAV at time step t can be expressed as:

[0043]

[0044] Step 2.3 Set the fixed rule actions of the attack module

[0045] For the attack module, a conditional judgment is introduced: if our UAV continuously detects the same enemy UAV within τ time steps, then a missile is launched. Let the current positions of our UAV and the enemy UAV be (x, y, y) and (x, y, y) respectively. 1t y 1t ) and (x 2t y 2t If ), then the distance between our drone and the enemy drone is

[0046]

[0047] If the distance d is within the range of a short-range missile and there are remaining short-range missiles, then launch a short-range missile; if the distance d is within the range of a long-range missile and there are remaining long-range missiles, then launch a long-range missile; otherwise, do not launch a missile.

[0048] Let the range of the short-range missile be d2 and the range of the long-range missile be d3. Then the attack action selected by the UAV at time step t can be expressed as:

[0049]

[0050] This led to the establishment of fixed rules and actions for multi-drone collaborative operations.

[0051] Furthermore, the specific steps for setting the initial positions and tracking conditions of the multiple drones in step 3 are as follows:

[0052] Step 3.1 Adjusting the initial position at the start of the game

[0053] At the start of the game, assume that all our drones are located on the left side of the map, and all enemy drones are located on the right side of the map, with both sides' drones lined up in a row, and the distance between the drones is constant, i.e.:

[0054] x 10 =x 20 =...=x n0

[0055] y 20 -y 10 =y 30 -y 20 =...=y n0 -y (n-1)0

[0056] In order to give the drone system an early advantage in the game, the initial position of each drone in the drone system needs to be adjusted at the beginning of the game. Within a time step t1, the formation of the multi-drone system is adjusted to a goose formation, so as to gain a certain advantage when encountering enemy drones.

[0057] If the number of drones n in a multi-drone system is odd, then the drone in the middle position... To become a team leader, the initial position is And after the start of the game, it always moves to the right (0° direction), that is...

[0058]

[0059] Assuming all drones in a multi-drone system have the same speed, v, then the distance the leader moves to the right is:

[0060]

[0061] Its position at time t1 is:

[0062]

[0063] Let θ be the angle between the two drones and the x-axis when the drone system is adjusted into a goose-shaped configuration, and d4 be the distance between any two adjacent drones. Then, at this time, drone A i The location is:

[0064]

[0065] Similarly, if the number of drones n in a multi-drone system is even, then the two drones in the middle position... and As a team leader, the initial positions are as follows: and After moving t1 steps to the right at the start of the game, the position is:

[0066]

[0067] At this time, drone A i The location is:

[0068]

[0069] Therefore, the formation of the multi-drone system was adjusted to a goose-shaped configuration.

[0070] Step 3.2 Collaborative Tracking Condition Judgment

[0071] After the multi-UAV system completes its formation adjustment, it begins to explore the map and uses the detection information at the current time step to make collaborative tracking condition judgments. That is, the multi-UAV system performs condition judgments at each time step t after t1; if a certain UAV A exists... i Enemy drone detected, and drone A i If the ratio of the number of friendly drones (including itself) within a circle of radius r to the number of enemy drones detected by these friendly drones is greater than or equal to 2, then the cooperative tracking condition is met; otherwise, it is not met.

[0072] Let the condition for collaborative tracking be f(t,i), then we have

[0073]

[0074] Among them, ⊙ represents drone A i The current position is a unit circle with center r and radius L. K List of enemy drones for detection L Dt A list of all enemy drones k that satisfy the condition k∈⊙ at time t.

[0075] This completes the setting of cooperative tracking conditions, allowing the drone to make judgments based on these conditions and select the appropriate actions for its mobile module.

[0076] Furthermore, the specific steps for setting the state, actions, and reward functions for the UAV agent's reinforcement learning in step 4 are as follows:

[0077] For each drone agent A in a multi-drone system i It uses reinforcement learning to find the optimal action a in state S and seeks to maximize the reward R by interacting with the environment.

[0078] Among them, state S includes drone intelligent agent A i Position information at the current time step t (x) it ,y it Unmanned Aerial Vehicle (UAV) Intelligent Agent A iList of enemy drones detected at the current time step t L it Unmanned Aerial Vehicle (UAV) Agent A i The remaining short-range missiles w at the current time step t it1 and the number of long-range missiles w it2 And the current moment of the drone intelligent agent A i The tracking condition f(t,i).

[0079] Action 'a' represents the drone intelligent agent A. i The direction F selected at time step t 1t (A i ), due to F 1t (A i The value of F can be any direction between 0 and 359°. However, the training dimension is too large, making convergence difficult. Therefore, F is set to... 1t (A i The value can be an integer multiple of 15°, resulting in 24 possible values. This greatly reduces the training dimensionality without hindering the drone's turning.

[0080] Reward R includes the following components:

[0081] (1) Reward r1 for a drone detecting an enemy drone

[0082] (2) The reward r2 for a drone launching a missile and hitting an enemy drone, and the penalty r2' for a drone being hit by a missile from an enemy drone.

[0083] (3) Rewards r3 and r4 for the correct use of cooperative tracking conditions by the drone

[0084] If drone agent A i When f(t, i) = 1, cooperative tracking is performed on the enemy drones detected in the detection list, i.e., drone A i and in the drone A i If friendly drones within a circle of radius r simultaneously move towards the location of an enemy drone, a reward of r3 is awarded; if drone agent A... i When f(t, i) = 0, the drone A i If you randomly choose to rendezvous with the nearest friendly drone or explore in a direction where the angle between yourself and the enemy drone is greater than or equal to 90°, you will receive a reward of r4.

[0085] (4) Avoid the penalty of drones remaining stationary (r5)

[0086] Since the drone still has a relatively high probability of receiving a reward r4 when it chooses to stay in place, it may get stuck in a local optimum during training. Therefore, a penalty measure must be introduced to calculate the drone's displacement over τ consecutive time steps and impose a corresponding penalty on the drone agent based on the magnitude of the displacement.

[0087] Let A be a drone in the unmanned aerial vehicle system D. i Their positions in consecutive time steps τ are {(x1, y1), (x2, y2), ..., (x...} τ y τ If )}, then its displacement over τ time steps is:

[0088]

[0089] If the drone remains stationary in a circle for τ time steps, its displacement Δ is relatively small; if the drone is normally tracking an enemy drone or searching a map, its displacement Δ must be greater than a certain specific value. Therefore, the drone A... i The displacement Δ and penalty r5 over τ time steps can be approximately described by an inverse proportional function, and with appropriate modifications, it can satisfy all cases.

[0090] The penalty function is set as follows:

[0091]

[0092] Among them, C t δ is a scaling factor, applicable to modify training in different scenarios; δ is a negligible quantity compared to displacement, preventing the function value from becoming infinite when Δ is exactly 0; ε is the reward compensation, ensuring that when the drone's displacement Δ over τ time steps exceeds a certain value, the distance reward value will not decay; the max function is the activation function, ensuring that when Δ exceeds a certain value, i.e. When the time is right, the value of the penalty function r5 is 0.

[0093] This completes the setup of the state, actions, and reward functions for reinforcement learning of the drone agent.

[0094] Beneficial effects:

[0095] 1. This invention patent proposes a multi-UAV cooperative tracking method based on the deep reinforcement learning QMIX algorithm. By modeling a multi-UAV combat system and training and testing it on the open-source multi-UAV combat platform MaCA, cooperative tracking of multiple UAVs is achieved.

[0096] 2. Based on the deep reinforcement learning QMIX algorithm, this invention patent addresses the problem of sparse rewards caused by excessive dimensionality in reinforcement learning by designing the reward function. Furthermore, by designing the tracking conditions and incorporating them as part of the state space for training with the reinforcement learning algorithm, the win rate can be effectively improved.

[0097] 3. This invention patent adjusts the initial position of the drones at the start of the battle, adopting a goose formation. This formation is a combat formation that spreads out horizontally and has the left and right wings arranged in a staggered manner backward. It has good stability and efficiency, and helps to improve the coordination and win rate of multi-drone operations.

[0098] 4. The multi-UAV system collaborative combat model established by this invention can not only be used for collaborative combat of multiple UAVs, but also for other scenarios, such as maritime combat and ground combat units. Attached Figure Description

[0099] Figure 1 This is a general framework diagram of the present invention patent;

[0100] Figure 2 This is the overall program flowchart of the present invention patent;

[0101] Figure 3 For multi-UAV systems, a two-dimensional absolute coordinate system is used.

[0102] Figure 4 This is a schematic diagram showing the range of the UAV detection module;

[0103] Figure 5 This is a schematic diagram showing the range of the UAV jamming module;

[0104] Figure 6 This is a schematic diagram illustrating the range of the drone attack module.

[0105] Figure 7 A schematic diagram illustrating the initial position adjustment for a multi-UAV system;

[0106] Figure 8 A schematic diagram illustrating the tracking conditions for a drone;

[0107] Figure 9 An illustration showing the effect of collaborative tracking of multiple drones after training; Detailed Implementation

[0108] The method and system of the present invention will be described in detail below.

[0109] This invention patent is a multi-UAV cooperative tracking method based on deep reinforcement learning, characterized by the following steps:

[0110] Step 1: Establish a multi-UAV combat model on a UAV combat simulation platform. This includes establishing the multi-UAV combat system, absolute coordinate system, and mathematical models of each UAV agent within the system.

[0111] (1) Establish a two-dimensional absolute coordinate system

[0112] Establish a two-dimensional absolute coordinate system with the bottom left corner of the currently selected battle map as the origin, the horizontal direction as the x-axis, and the vertical direction as the y-axis, as follows: Figure 3 As shown.

[0113] (2) Establish a multi-UAV combat system

[0114] To establish a multi-UAV combat system containing n UAVs, let A represent the intelligent agent of each UAV. Then, this multi-UAV combat system can be represented by the set D = {A1, A2, ..., A...}. n}express.

[0115] (3) Establishing a UAV intelligent agent model

[0116] For each UAV A in the multi-UAV system D, the action selected at time step t consists of four modules: movement, detection, jamming, and attack.

[0117] For the movement module, the UAV selects a direction within the 0-359° range in a two-dimensional plane coordinate system and moves a unit step in this direction per unit time. The direction selected by the UAV at time step t can be denoted as F. 1t (A i );

[0118] For the detection module, assuming the total number of radar frequencies is m, after the UAV selects any radar frequency from the radar frequency list, it will detect whether there are enemy UAVs within a fixed sector area with its orientation as the axis, 60° to the left and right, and a distance of d1, and record the detected enemy UAVs in the detection list L. i In this context, the radar frequency selected by the UAV at time step t can be denoted as F. 2t (A i The specific range of the detection module is as follows: Figure 4 As shown;

[0119] For the jamming module, after the UAV selects any frequency from the radar frequency list, it will jam the enemy UAV's radar at the selected frequency within a fixed fan-shaped area extending 15° to the left and right of its direction, with a distance of d1, thus disabling its detection capability. Alternatively, the UAV can choose to activate jamming, jamming all enemy UAV radar frequencies within a fixed fan-shaped area extending 1° to the left and right of its direction, with a distance of d1. The specific range of the jamming module is as follows: Figure 5As shown, let F be the interference frequency selected by the UAV at time step t. 3t (A i Then we have:

[0120]

[0121] For the attack module, the UAV launches a short-range missile (range d2) or a long-range missile (range d3) at an enemy UAV in the enemy UAV detection list. The specific range of the attack module is as follows: Figure 6 As shown, suppose the enemy drone selected by the drone to attack is L in the detection list. i (1≤i≤n), the type of missile selected by the UAV to launch is p, (when p=0, the UAV does not launch a missile; when p=1, the UAV launches a short-range missile; when p=2, the UAV launches a long-range missile), whether the UAV has any other missiles of this type is w(p), (when w(p)=0, the UAV does not have any missiles of this type; when w(p)=1, the UAV has missiles of this type; the default is w(0)=0), let the attack action selected by the UAV at time step t be F. 4t (A i Then we have:

[0122] F 4t (A i )=w(p)×((p-1)×n+Li)

[0123] Therefore, the action space of the unmanned aerial vehicle agent at time step t can be defined by the set {F1(A)} i ), F2(A i ), F3(A i ), F4(A i )}express.

[0124] For each UAV A in the multi-UAV system D, its state at time step t is represented as follows:

[0125] Drone A i Position information at the current time step t (x) it y it ); Drone A i List of enemy drones detected at the current time step t L it The enemy drone detection list L is obtained by aggregating the detection lists of each drone from multiple drone systems. Dt (L Dt =L 1t ∪L 2t ∪...∪L nt ); Drone A i The remaining short-range missiles w at the current time step tit1 and the number of long-range missiles w it2 .

[0126] Therefore, the state space of the unmanned aerial vehicle agent at time step t can be represented by the set {x}. it y it L it w it1 w it2}express.

[0127] This led to the development of a multi-drone combat model.

[0128] Step 2: Set fixed rules and actions for multi-drone collaborative operations

[0129] (1) Set fixed rule actions for the detection module

[0130] For the detection module, since the probability of the enemy drone interfering with the randomly changing radar frequency of our drone within a continuous step is low, and the effectiveness of the detection module depends more on the direction selection of the drone in the movement module, the radar frequency is generated using random numbers at each time step.

[0131] Let the total number of radar frequencies of the UAV be m, then the radar frequency selected by the UAV at time step t can be expressed as:

[0132] F 2t (A i ) = random(1,m)

[0133] (2) Set fixed rule actions for the interference module

[0134] For the jamming module, a conditional judgment is introduced. If our UAV continuously detects the same enemy UAV within τ time steps, then full-channel jamming is used to prevent it from detecting our UAV. If our UAV does not detect the enemy UAV or does not continuously detect the enemy UAV, then random numbers are used to generate jamming frequencies.

[0135] Therefore, the interference frequency selected by the UAV at time step t can be expressed as:

[0136]

[0137] (3) Set fixed rule actions for the attack module

[0138] For the attack module, a conditional judgment is introduced: if our UAV continuously detects the same enemy UAV within τ time steps, then a missile is launched. Let the current positions of our UAV and the enemy UAV be (x, y, y) and (x, y, y) respectively. 1t ,y 1t ) and (x2t ,y 2t If ), then the distance between our drone and the enemy drone is

[0139]

[0140] If the distance d is within the range of a short-range missile and there are remaining short-range missiles, then launch a short-range missile; if the distance d is within the range of a long-range missile and there are remaining long-range missiles, then launch a long-range missile; otherwise, do not launch a missile.

[0141] Let the range of the short-range missile be d2 and the range of the long-range missile be d3. Then the attack action selected by the UAV at time step t can be expressed as:

[0142]

[0143] This led to the establishment of fixed rules and actions for multi-drone collaborative operations.

[0144] Step 3: Set the initial positions and tracking conditions for multiple drones

[0145] (1) Initial position adjustment

[0146] At the start of the game, assume that all our drones are located on the left side of the map, and all enemy drones are located on the right side of the map, with both sides' drones lined up in a row, and the distance between the drones is constant, i.e.:

[0147] x 10 =x 20 =...=x n0

[0148] y 20 -y 10 =y 30 -y 20 =...=y n0 -y (n-1)0

[0149] In order to give the drone system an early advantage in the game, the initial position of each drone in the drone system needs to be adjusted at the beginning of the game. Within a time step t1, the formation of the multi-drone system is adjusted to a goose formation, so as to gain a certain advantage when encountering enemy drones.

[0150] If the number of drones n in a multi-drone system is odd, then the drone in the middle position... To become a team leader, the initial position is And after the start of the game, it always moves to the right (0° direction), that is...

[0151]

[0152] Assuming all drones in a multi-drone system have the same speed, v, then the distance the leader moves to the right is:

[0153]

[0154] Its position at time t1 is:

[0155]

[0156] Let θ be the angle between the two drones and the x-axis when the drone system is adjusted into a goose-shaped configuration, and d4 be the distance between any two adjacent drones. Then, at this time, drone A i The location is:

[0157]

[0158] Similarly, if the number of drones n in a multi-drone system is even, then the two drones in the middle position... and As a team leader, the initial positions are as follows: and After moving t1 steps to the right at the start of the game, the position is:

[0159]

[0160] At this time, drone A i The location is:

[0161]

[0162] Therefore, the formation of the multi-drone system was adjusted to a goose formation, the specific process of which is as follows: Figure 7 As shown.

[0163] (2) Collaborative tracking condition judgment

[0164] After the multi-UAV system completes its formation adjustment, it begins to explore the map and uses the detection information at the current time step to make collaborative tracking condition judgments. That is, the multi-UAV system performs condition judgments at each time step t after t1; if a certain UAV A exists... i Enemy drone detected, and drone A i If the ratio of the number of friendly drones (including itself) within a circle of radius r to the number of enemy drones detected by these friendly drones is greater than or equal to 2, then the cooperative tracking condition is met; otherwise, it is not met.

[0165] Let the condition for collaborative tracking be f(t, i), then we have

[0166]

[0167] Among them, ⊙ represents drone A i The current location is a unit circle with radius r as the center and LK as the enemy drone detection list. Dt The list consists of all enemy drones k that satisfy the condition k∈⊙ at time t. The specific judgment is as follows: Figure 8 As shown.

[0168] This completes the setting of cooperative tracking conditions, allowing the drone to make judgments based on these conditions and select the appropriate actions for its mobile module.

[0169] Step 4: Set up the state, action, and reward functions for the drone agent's reinforcement learning.

[0170] For each drone agent A in a multi-drone system i It uses reinforcement learning to find the optimal action a in state S and seeks to maximize the reward R by interacting with the environment.

[0171] Among them, state S includes drone intelligent agent A i Position information at the current time step t (x) it y it Unmanned Aerial Vehicle (UAV) Intelligent Agent A i List of enemy drones detected at the current time step t L it Unmanned Aerial Vehicle (UAV) Agent A i The remaining short-range missiles w at the current time step t it1 and the number of long-range missiles w it2 And the current moment of the drone intelligent agent A i The tracking condition f(t, i).

[0172] Action 'a' represents the drone intelligent agent A. i The direction F selected at time step t 1t (A i ), due to F 1t (A i The value of F can be any direction between 0 and 359°. However, the training dimension is too large, making convergence difficult. Therefore, F is set to... 1t (A i The value can be an integer multiple of 15°, resulting in 24 possible values. This greatly reduces the training dimensionality without hindering the drone's turning.

[0173] Reward R includes the following components:

[0174] (1) Reward r1 for a drone detecting an enemy drone

[0175] (2) The reward r2 for a drone launching a missile and hitting an enemy drone, and the penalty r2' for a drone being hit by a missile from an enemy drone.

[0176] (3) Rewards r3 and r4 for the correct use of cooperative tracking conditions by the drone

[0177] If drone agent A i When f(t, i) = 1, cooperative tracking is performed on the enemy drones detected in the detection list, i.e., drone A i and in the drone A i If friendly drones within a circle of radius r simultaneously move towards the location of an enemy drone, a reward of r3 is awarded; if drone agent A... i When f(t, i) = 0, the drone A i If you randomly choose to rendezvous with the nearest friendly drone or explore in a direction where the angle between yourself and the enemy drone is greater than or equal to 90°, you will receive a reward of r4.

[0178] (4) Avoid the penalty of drones remaining stationary (r5)

[0179] Since the drone still has a relatively high probability of receiving a reward r4 when it chooses to stay in place, it may get stuck in a local optimum during training. Therefore, a penalty measure must be introduced to calculate the drone's displacement over τ consecutive time steps and impose a corresponding penalty on the drone agent based on the magnitude of the displacement.

[0180] Let A be a drone in drone system D. i Its positions over τ consecutive time steps are {(x1,y1),(x2,y2),...,(x...} τ ,y τ If )}, then its displacement over τ time steps is:

[0181]

[0182] If the drone remains stationary in a circle for τ time steps, its displacement Δ is relatively small; if the drone is normally tracking an enemy drone or searching a map, its displacement Δ must be greater than a certain specific value. Therefore, the drone A... i The displacement Δ and penalty r5 over τ time steps can be approximately described by an inverse proportional function, and with appropriate modifications, it can satisfy all cases.

[0183] The penalty function is set as follows:

[0184]

[0185] Among them, C tδ is a scaling factor, applicable to modify training in different scenarios; δ is a negligible quantity compared to displacement, preventing the function value from becoming infinite when Δ is exactly 0; ε is the reward compensation, ensuring that when the drone's displacement Δ over τ time steps exceeds a certain value, the distance reward value will not decay; the max function is the activation function, ensuring that when Δ exceeds a certain value, i.e. When the time is right, the value of the penalty function r5 is 0.

[0186] This completes the setup of the state, actions, and reward functions for reinforcement learning of the drone agent.

[0187] Step 5: Use the QMIX algorithm to train the UAV agents. Train n UAV agents to fit the best action a in state S and one source controller to fit the best reward distribution among the multiple agents. Adjust the hyperparameters according to the training results. Some of the hyperparameters set are shown in the table below.

[0188] Buffer size 320 Batch size 32 Learning rate 0.01 Epsilon 0.9 Gamma 0.9 Target update frequency 100

[0189] The trained multi-UAV system can perform cooperative tracking tasks relatively well to a certain extent, with specific results as follows: Figure 9 As shown.

[0190] Step 6: Repeat step 5 when facing different opponents to achieve the goal of multi-drone collaborative tracking.

Claims

1. A multi-UAV cooperative tracking method based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Establish a multi-drone combat model on the drone combat simulation platform; Step 2: Set fixed rules and actions for multi-drone collaborative operations; Step 3: Set the initial positions and tracking conditions for multiple drones; Step 4: Set the state, action, and reward functions for the drone agent's reinforcement learning, as follows: For each drone agent in a multi-drone system It uses reinforcement learning to find the optimal action a in state S and seeks to maximize the reward R by interacting with the environment. Among them, state S includes the drone intelligent agent Position information at the current time step t Unmanned aerial vehicle (UAV) intelligent agents List of enemy drones detected at the current time step t Unmanned aerial vehicle (UAV) intelligent agents The remaining short-range missiles at the current time step t and the number of long-range missiles And the current drone intelligent agent Tracking conditions ; Action a is the drone intelligent agent The direction selected at time step t ,because The value can be any direction between 0 and 359°. If the training dimension is too large, convergence becomes difficult; therefore, it is set to... There are 24 possible values ​​that are multiples of 15°, which greatly reduces the training dimensionality without hindering the drone's turning. Reward R includes the following components: (1) Reward for a drone detecting an enemy drone ; (2) Reward for a drone launching a missile and hitting an enemy drone Penalty for our drone being hit by an enemy drone missile ; (3) Rewards for the correct use of collaborative tracking conditions by drones , ; If the drone intelligent agent exist At the same time, coordinated tracking of enemy drones detected in the detection list is carried out, i.e., drones and in the drone If friendly drones within a circle of radius r simultaneously move towards the location of an enemy drone, a reward will be awarded. If the drone intelligent agent exist At that time, drones You will receive a reward if you randomly choose to rendezvous with the nearest friendly drone or explore in a direction where the angle between yourself and the enemy drone is greater than or equal to 90°. ; (4) Avoid penalties for drones remaining stationary ; Even when a drone chooses to remain stationary in a circle, it still has a relatively high probability of receiving a reward. This can lead to a local optimum where the drone remains stationary during training. Therefore, it is necessary to introduce a penalty measure, calculate the drone's displacement over τ consecutive time steps, and impose a corresponding penalty on the drone agent based on the magnitude of the displacement. Let a certain drone in the unmanned aerial vehicle system D be... Its positions in consecutive time steps τ are respectively Then its displacement over τ time steps is: ； If the drone remains stationary in a circle for τ time steps, then its displacement... Smaller; if the drone is normally tracking an enemy drone or searching a map, its displacement... It must be greater than a certain specific value; therefore, this drone Displacement over τ time steps With punishment It is approximately described by an inverse proportional function, and with appropriate modifications, it can satisfy all cases; The penalty function is set as follows: ； in, This is a scaling factor, applicable to modifying training in different scenarios; For a small quantity that is negligible compared to the displacement, to prevent when When the value is exactly 0, the function value is infinity; As a reward and compensation, the displacement of the drone over τ time steps is... When the value exceeds a certain threshold, no distance reward value decay will occur. The function is an activation function, such that when Greater than a certain specific value, that is At that time, the penalty function The value is 0; Thus, the state, action, and reward functions for reinforcement learning of the drone agent were set up. Step 5: Train the UAV agent using the QMIX algorithm and adjust the hyperparameters based on the training results; Step 6: Repeat step 5 when facing different opponents to achieve the goal of multi-drone collaborative tracking.

2. The multi-UAV cooperative tracking method based on deep reinforcement learning according to claim 1, characterized in that, The multi-UAV combat model established on the UAV combat simulation platform in step 1 is as follows: Step 1.1 Establish a two-dimensional absolute coordinate system A two-dimensional absolute coordinate system is established with the bottom left corner of the currently selected battle map as the origin, the horizontal direction as the x-axis, and the vertical direction as the y-axis. Step 1.2 Establish a multi-UAV combat system To establish a multi-UAV combat system containing n UAVs, let A represent the intelligent agent of each UAV. Then, the multi-UAV combat system uses a set... express; Step 1.3 Establish the UAV intelligent agent model For each drone A in the multi-UAV system D, the action selected at time step t consists of four modules: movement, detection, jamming, and attack. For the movement module, the UAV selects a direction within the 0-359° range in a two-dimensional plane coordinate system and moves a unit step in this direction per unit time. The direction selected by the UAV at time step t is denoted as... ; For the detection module, assuming the total number of radar frequencies is m, after the UAV selects any radar frequency from the radar frequency list, it fixes itself in a direction with its orientation as the axis, extending 60° to the left and right, and at a distance of m. The system detects enemy drones within a fan-shaped area and records any detected enemy drones in the detection list. In the diagram, the radar frequency selected by the UAV at time step t is denoted as . ; For the jamming module, after the UAV selects any frequency from the radar frequency list, it fixes itself in a direction with its orientation as the axis, 15° to the left and right, and at a distance of [missing information]. Within a fan-shaped area, jam the enemy drone's radar at the corresponding frequency, rendering it unable to detect; or, if the drone chooses to activate jamming, fix the direction with its orientation as the axis, 1° to the left and right, and at a distance of... The radar that interferes with all frequencies of the enemy drone within a fan-shaped range, assuming the interference frequency selected by the drone at time step t is... Then we have: ； For the attack module, the drone launches a short-range missile at one of the enemy drones in the enemy drone detection list, at a distance of [distance missing]. Or long-range missiles, with a range of Suppose that the enemy drone selected by the drone to attack is one of the drones in the detection list. Let p be the type of missile the UAV chooses to launch, where 1 ≤ i ≤ n. When p = 0, the UAV does not launch any missiles; when p = 1, the UAV launches short-range missiles; when p = 2, the UAV launches long-range missiles. Whether the UAV has any other missiles of that type is w(p). When w(p) = 0, the UAV does not have any missiles of that type; when w(p) = 1, the UAV has missiles of that type. By default, w(0) = 0. Let the attack action selected by the UAV at time step t be... Then we have: ； Therefore, the action space of the unmanned aerial vehicle agent at time step t is composed of the set { }express; For each UAV A in the multi-UAV system D, its state at time step t is represented as follows: drones Position information at the current time step t drones List of enemy drones detected at the current time step t The enemy drone detection list is obtained by aggregating the detection lists of each drone from multiple drone systems. ,in drones The remaining short-range missiles at the current time step t and the number of long-range missiles ; Therefore, the state space of the unmanned aerial vehicle agent at time step t is composed of the set { , }express; This led to the development of a multi-drone combat model.

3. The multi-UAV cooperative tracking method based on deep reinforcement learning according to claim 1, characterized in that, The specific actions for setting fixed rules for multi-drone cooperative operations described in step 2 are as follows: Step 2.1 Set the fixed rule actions of the detection module For the detection module, since the probability of the enemy drone interfering with the randomly changing radar frequency of our drone within a continuous step is low, and the effectiveness of the detection module depends more on the direction selection of the drone in the movement module, the radar frequency is generated using random numbers at each time step. Let the total number of radar frequencies of the UAV be m, then the radar frequency selected by the UAV at time step t is expressed as: ； Step 2.2 Set the fixed rule actions of the interference module For the jamming module, a condition judgment is introduced. If our UAV detects the same enemy UAV continuously within τ time steps, then full-channel jamming is used to prevent it from detecting our UAV. If our drones fail to detect enemy drones or fail to detect enemy drones continuously, then random numbers will be used to generate jamming frequencies. Therefore, the interference frequency point selected by the UAV at time step t is represented as: ； Step 2.3 Set the fixed rule actions of the attack module For the attack module, a conditional judgment is introduced: if our UAV continuously detects the same enemy UAV within τ time steps, then the missile is launched. Let the current positions of our UAV and the enemy UAV be respectively... and Then the distance between our drone and the enemy drone is ； If the distance d is within the range of a short-range missile and there are remaining short-range missiles, then launch a short-range missile; if the distance d is within the range of a long-range missile and there are remaining long-range missiles, then launch a long-range missile; otherwise, do not launch a missile. Let the range of the short-range missile be d2 and the range of the long-range missile be d3. Then the attack action selected by the UAV at time step t is expressed as: ； This led to the establishment of fixed rules and actions for multi-drone collaborative operations.

4. The multi-UAV cooperative tracking method based on deep reinforcement learning according to claim 1, characterized in that, Step 3, which involves setting the initial positions and tracking conditions for multiple drones, is as follows: Step 3.1 Adjusting the initial position at the start of the game At the start of the game, assume that all our drones are located on the left side of the map, and all enemy drones are located on the right side of the map, with both sides' drones lined up in a row, and the distance between the drones is constant, i.e.: ； ； To give the drone system an early advantage in the game, the initial positions of each drone in the system need to be adjusted at the start. Within a time step, the formation of multiple unmanned aerial vehicle systems is adjusted to a goose-shaped formation, thereby gaining a certain advantage when encountering enemy unmanned aerial vehicles. If the number of drones n in a multi-drone system is odd, then the drone in the middle position... To become a team leader, the initial position is And after the start of the game, it always moves to the right at 0°, that is... ； Assuming all drones in a multi-drone system have the same speed, v, then the distance the leader moves to the right is: ； Its in The position at that time is: ； Let θ be the angle between the two drones and the x-axis when the drone system is adjusted into a goose-shaped configuration, and let θ be the distance between any two adjacent drones. Then the drone The location is: ； Similarly, if the number of drones n in a multi-drone system is even, then the two drones in the middle position... and As a team leader, the initial positions are as follows: and Move to the right after the start of the game. The position after each step is: ； At this time, the drone The location is: ； Therefore, the formation of the multi-drone system was adjusted to a goose-shaped configuration; Step 3.2 Collaborative Tracking Condition Judgment After the multi-UAV system completes its formation adjustment, it begins to explore the map and uses the detection information at the current time step to make collaborative tracking condition judgments. That is, the multi-UAV system... At each subsequent time step t, a conditional check is performed; if a certain drone exists... Enemy drone detected, and the drone If the ratio of the number of friendly drones within a circle of radius r to the number of enemy drones detected by these friendly drones is ≥2, then the cooperative tracking condition is met; otherwise, it is not met. Let the collaborative tracking conditions be: Then there is ； in, For drones The current position is a unit circle with center r and radius L. K List of enemy drones to be detected All satisfying at time t A list consisting of the enemy drones k under the given conditions; This completes the setting of the cooperative tracking conditions, and the drone uses these conditions to make judgments and select the appropriate actions for its mobile module.

Images