A method and system for multi-target cooperative tracking guidance of a UAV cluster based on reinforcement learning

Abstract

The application discloses a kind of unmanned aerial vehicle cluster multi-target cooperative tracking guidance method and system based on reinforcement learning, belongs to unmanned aerial vehicle cluster cooperative control technical field, solve the problems such as dispersed decision, partial observability, variable scale and cooperation difficulty when unmanned aerial vehicle group executes multi-target tracking task.The method of the application comprises: defining reciprocity reward for each unmanned aerial vehicle, the reward is the dot product of the environmental reward vector of all adjacent unmanned aerial vehicles and the corresponding dependence vector.Capture interdependence between unmanned aerial vehicles through inter-point mutual information neural network.Original environmental reward of unmanned aerial vehicle is regularized and reshaped using reciprocity reward, by maximizing the reshaped reward, unmanned aerial vehicle is driven to maximize its own income while also maximizing the income of its adjacent unmanned aerial vehicles.Combined with experience replay sharing mechanism, experience sharing reciprocity reward multi-agent evaluation-action network algorithm is proposed.The application is suitable for civil unmanned aerial vehicle tracking technology, and is mainly used in aerial photography, logistics, agriculture and other civil scenarios.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) swarm control technology, and in particular to distributed cooperative tracking and control of multiple UAVs on multiple targets. Background Technology

[0002] Through close collaboration, drone swarms can demonstrate superior coordination, intelligence, and autonomy compared to traditional multi-drone systems. One important application area for drone swarms is multi-target tracking and guidance in unknown environments. In this mission, a large number of small quadcopter drones are deployed in the mission area, needing to collaboratively track detected targets and search for unknown targets.

[0003] However, achieving coordination among drone swarms is no easy task. Due to dimensionality, the computational complexity of joint decision-making by all drones increases exponentially with the number of drones. Simultaneously, drones are in a partially observable state, only able to communicate with neighboring drones and locally perceive targets, unable to directly acquire global information for global coordination. Therefore, centralized coordination requiring global information and a central node to coordinate all drone actions is not feasible; achieving decentralized coordination of large-scale drone swarms remains a significant challenge.

[0004] Multi-agent deep reinforcement learning techniques have proven effective in endowing agents with the ability to learn collaborative behaviors due to their groundbreaking advancements in game theory. However, the collaborative processes of most existing methods are centralized or require access to global information, which is incompatible with the distributed nature of swarm systems. Furthermore, many methods oversimplify the problem, such as assuming known or easily achievable complex environmental models, global communication capabilities of UAVs, or adherence to predetermined rules, making it difficult to meet the inherent characteristics of multi-target tracking tasks in UAV swarms. Summary of the Invention

[0005] The purpose of this invention is to solve the technical problems of decentralized decision-making, partial observability, variable scale, and difficulty in coordination when unmanned aerial vehicle (UAV) swarms perform multi-target tracking tasks in unknown environments. It provides a UAV swarm multi-target cooperative tracking guidance method and system based on reinforcement learning.

[0006] The technical solution adopted by this invention to solve the above problems is: a multi-target cooperative tracking and guidance method for UAV swarms based on reinforcement learning, the method comprising:

[0007] Step 1: Model and formally describe the problem of multi-target tracking task of UAV swarm;

[0008] Step 2: Design the drone's reward function, including target tracking reward, repeated tracking penalty, and boundary penalty, and obtain the drone's environmental reward at each step;

[0009] Step 3: Propose a method for maximizing reciprocal rewards, defining and calculating reciprocal rewards; specifically including:

[0010] To improve inter-drone collaboration, the instant reward for each drone has been modified as follows:

[0011]

[0012] in, For reciprocal rewards, For balance coefficient, For drones Environmental rewards at every step;

[0013] Reciprocal rewards are defined as:

[0014]

[0015] in, Indicates drone A collection of adjacent drones, Measuring drones and The degree of dependence between them These are the normalization coefficients;

[0016] Step 4: Construct a PMI neural network to estimate the dependency index among drones;

[0017] Step 5: Based on the experience replay sharing mechanism, construct an experience-sharing reciprocal reward multi-agent evaluation-action network algorithm that includes a shared action network, a shared evaluation network, a target evaluation network, and a PMI neural network; train the shared action network and the shared evaluation network using the reshaped instantaneous reward; after training, deploy the shared action network strategy to all UAVs; during the execution phase, each UAV makes action decisions in a decentralized manner based only on its own local information.

[0018] Further, step 1 specifically includes: assuming there are n isomorphic small quadcopter drones and m isomorphic targets in the task scenario; the drones fly at a constant speed in a two-dimensional plane, and their actions are bounded heading angular rates; the targets wander randomly in the environment; the drones perceive the targets, and can perceive and track the targets when they are within the perception range, but cannot distinguish the specific identity of the targets; at the same time, the drones can receive local communication information from neighboring drones and obtain local information relative to the boundary of the task area; the decision-making process is modeled as a distributed partially observable Markov decision process Dec-POMDP, which includes a joint state space, a joint action space, state transition probabilities, a joint reward function, a joint observation space, and an individual observation model.

[0019] Further, in step 2, the target tracking reward is defined as follows:

[0020]

[0021] in, For drones For the target Tracking rewards, For drones With the goal The relative distance, For maximum sensing distance; drone The reward for tracking multiple targets is .

[0022] Further, in step 2, the definition of the repeated tracking penalty is:

[0023]

[0024] in, The penalty for repeated tracking between drones i and j is given by the total penalty for repeated tracking of drone i. .

[0025] Further, in step 2, the boundary penalty is defined as follows:

[0026]

[0027] in, This represents the minimum distance from the drone to the boundary.

[0028] Furthermore, step 4 specifically includes:

[0029] Using point-to-point mutual information to capture dependencies between drones, for drones and Its dependency index is defined as:

[0030]

[0031] in, Representing drones Local information, For communication information, For observation information, For boundary information, Status information; For each drone Action decision-making; Representing local information-action pairs and The joint probability distribution; through a parameter of neural networks The inter-point mutual information is estimated, and the network is trained by maximizing the Jensen-Shannon mutual information. The variational lower bound is achieved, and the output of the PMI neural network satisfies: .

[0032] Furthermore, step 5 specifically includes:

[0033] Step 5.1: Initialize shared action network parameters Shared evaluation network parameters Target evaluation network parameters PMI network parameters and action-judgment experience replay buffer PMI Experience Replay Buffer Among them, the shared action network is used to output action strategies based on local information of the UAV. Shared evaluation networks are used to estimate action value functions. The target evaluation network is used to construct target values ​​when updating the shared evaluation network, and the PMI network is used to estimate the drone dependency index. ;

[0034] Step 5.2: In each training cycle:

[0035] Step 5.2.1: Reset the environment and obtain joint local information of all UAVs. ;

[0036] Step 5.2.2: For each step :

[0037] Each drone According to the sharing policy and exploration strategy selection action decision ;

[0038] Execute joint operations Receive instant rewards from the United Environment and new joint local information ;

[0039] For each drone Obtain the set of its neighboring drones. And use the PMI neural network to calculate the relationship with each neighboring drone. Dependence index ;Pair the data ( Store in PMI buffer ;

[0040] Based on step 3, reshape the drone using the dependency index and neighbor's environmental rewards. Instant rewards ; transfer tuple Save Action - Evaluation Experience Replay Buffer , For drones New local information;

[0041] Step 5.2.3: Replay Buffer of Action-Judgment Experience sampling Sample , For the first Current local information of each sample For the first The current action of each sample. For the first Immediate rewards for reshaping individual samples For the first Local information of each sample at the next moment;

[0042] Construct the target value using the target network: ; For discount rate, For the shared action network based on the next time-of-motion local information The next action to output, To evaluate the network's local information for the next time step With the next action The loss function of the shared evaluation network is defined based on the estimated value of the corresponding action. By minimizing the loss function Update shared evaluation network parameters Define the objective function of the shared action network. ,in, This indicates that the shared action network is in local information Down Output Action The probability is obtained by maximizing the objective function. Update shared action network parameters This allows for the maximization of learning of the reshaped reward;

[0043] After completing the shared evaluation network parameters After the update, the parameters of the target network are... Perform a soft update. ,in, This is the soft update coefficient, with a value ranging from 0 to 1;

[0044] Step 5.2.4: Replaying the PMI Experience Buffer Medium sampling Positive samples and construct One mismatched sample Among them, positive samples This represents local information about the actual matching between adjacent drones at the same time—action pairs; mismatched samples. This indicates that the first half should be retained. The original text remains unchanged, but the latter half is replaced with a random sample obtained from the PMI experience playback buffer. The resulting local information—action mismatch;

[0045] Define the loss function of the PMI network as follows:

[0046]

[0047] in, This represents the PMI network's estimated output for positive sample dependencies. This represents the estimated output of the PMI network for the dependency of mismatched samples; by minimizing the loss function... Update PMI network parameters ;

[0048] Step 5.3: Repeat step 5.2 until training is complete, obtaining the trained shared action network. After the strategy training is completed, the shared action network strategy is deployed to all drones; during the execution phase, each drone will share its local information. Input the same shared action network and output the corresponding action decision. This allows for multi-target collaborative tracking and guidance in a decentralized manner.

[0049] Secondly, the present invention provides a system for a multi-target cooperative tracking and guidance method for UAV swarms based on reinforcement learning. The system has program modules corresponding to the steps of the multi-target cooperative tracking and guidance method for UAV swarms based on reinforcement learning as described above, and executes the steps in the multi-target cooperative tracking and guidance method for UAV swarms based on reinforcement learning when running.

[0050] Thirdly, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, it executes the steps of a reinforcement learning-based multi-target cooperative tracking and guidance method for UAV swarms as described above.

[0051] Fourthly, the present invention provides a computer-readable storage medium for storing a computer program that executes a reinforcement learning-based multi-target cooperative tracking and guidance method for unmanned aerial vehicle swarms as described above.

[0052] The beneficial effects of this invention are:

[0053] The maximum reciprocal reward method and MAAC-R algorithm proposed in this invention enable UAV swarms to effectively learn cooperative tracking strategies under completely decentralized and partially observable conditions. Through reciprocal reward regularization, UAVs are incentivized to consider the impact on their neighbors' gains when making decisions, resulting in various flexible cooperative behaviors such as formation, many-to-many, one-to-many, many-to-one, and target exchange. This method avoids the poor scalability of centralized methods, utilizing only local interaction information, allowing the learned strategy to be well generalized to scenarios with more UAVs, targets, and larger task scales, significantly improving the overall tracking efficiency and cooperation level of UAV swarms in multi-target tracking tasks.

[0054] This invention provides a distributed multi-agent deep reinforcement learning method based on maximum reciprocal reward. This method combines maximum reciprocal reward with multi-agent deep reinforcement learning, which can incentivize drone swarms to exhibit rich cooperative tracking behaviors, effectively improving tracking efficiency and cooperation levels. Furthermore, the learned strategy can be well extended to complex scenarios with more drones and targets.

[0055] This invention is applicable to civilian drone tracking technology, such as visual tracking, GPS tracking, and obstacle avoidance. The tracking object is a person, vehicle, or scenery, and it is mostly used in civilian scenarios such as aerial photography, logistics, and agriculture. Attached Figure Description

[0056] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a schematic diagram of the MAAC-R algorithm training and execution process;

[0058] Figure 2 This is a schematic diagram of a local communication and observation model for unmanned aerial vehicles (UAVs).

[0059] Figure 3 A schematic diagram illustrating the emergence of collaborative behavior in a multi-target tracking task;

[0060] Figure 4 This is a comparison chart of the total number of targets tracked and the average number of targets tracked between MAAC-R and the baseline algorithms (MAAC, MAAC-G). Detailed Implementation

[0061] The specific implementation method of this embodiment, a multi-target cooperative tracking and guidance method for UAV swarms based on reinforcement learning, includes:

[0062] This paper models the multi-target tracking task of UAV swarms as a distributed, partially observable Markov decision process. Inspired by the reciprocal altruistic mechanism, a reciprocal reward is defined for each UAV, which is the dot product of the environmental reward vectors of all its neighboring UAVs and their corresponding dependency vectors. Instantaneous dependencies between UAVs are directly captured by a point-to-point mutual information neural network without complex aggregation statistics. The original environmental rewards of the UAVs are regularized and reshaped using the reciprocal reward. By maximizing the reshaped reward, each UAV is driven to maximize its own gains while also maximizing the gains of its neighboring UAVs, thus achieving distributed cooperation. Combined with an experience replay sharing mechanism, an experience-sharing reciprocal reward multi-agent evaluation-action network algorithm is proposed to learn a shared cooperative tracking strategy for all homogeneous UAVs.

[0063] The specific steps are as follows:

[0064] Step 1: Model and formally describe the multi-target tracking task of UAV swarms;

[0065] Step 2: Design the reward function for the UAV, including target tracking reward, repeated tracking penalty, and boundary penalty;

[0066] Step 3: Propose a method for maximizing reciprocal rewards, define and calculate reciprocal rewards;

[0067] Step 4: Construct a PMI neural network to estimate the dependency index among drones;

[0068] Step 5: Design an experience-sharing, reciprocal reward, multi-agent evaluation-action network algorithm;

[0069] Below is a detailed explanation of steps one through five:

[0070] Step 1: Task Modeling and Formal Description. Assume the task scenario contains n isomorphic small quadcopter UAVs and m isomorphic targets. The UAVs fly at a constant speed in a two-dimensional plane, with bounded angular velocities. The targets wander randomly in the environment. The UAVs perceive targets through their downward-looking onboard sensors; they can detect and track targets when they are within their perception range, but cannot distinguish the specific identities of the targets. Simultaneously, the UAVs can receive local communication information from neighboring UAVs and acquire local information relative to the task area boundaries. The local communication and observation model of the UAVs is as follows: Figure 2 As shown. Based on this, the decision-making process is modeled as a decentralized partially observable Markov decision process (Dec-POMDP), which includes a joint state space, a joint action space, state transition probabilities, a joint reward function, a joint observation space, and an individual observation model.

[0071] Step 2: Reward Function Design. To encourage drones to learn cooperative strategies, a reasonable reward system needs to be designed.

[0072] S1, Target Tracking Reward. The tracking reward for drone i on target k is defined as:

[0073]

[0074] in, Let be the relative distance between drone i and target k. The maximum sensing distance. The reward for drone i tracking multiple targets is... .

[0075] S2. Duplicate Tracking Penalty. To avoid resource waste and collision risks, duplicate tracking is penalized. The duplicate tracking penalty between drones i and j is:

[0076]

[0077] The total repeated tracking penalty for drone i is .

[0078] S3. Boundary Penalty. To encourage drones to fly within the mission area, a boundary penalty is set:

[0079]

[0080] in, This represents the minimum distance from the drone to the boundary.

[0081] In summary, the environmental reward for drone i at each step is: .

[0082] Step 3, Maximum Reciprocal Reward Method. To improve inter-drone collaboration, the instant reward for each drone is modified as follows:

[0083]

[0084] in, For reciprocal rewards, This is the balancing factor. The reciprocal reward is defined as:

[0085]

[0086] in, Indicates drone A collection of adjacent drones, Measuring drones and The degree of dependence between them This is the normalization coefficient.

[0087] Step 4: PMI Dependency Estimation. Inter-point mutual information is used to capture dependencies between drones. For drones i and j, the dependency index is defined as:

[0088]

[0089] in, Representing drones Local information (communication, observation, boundary, state). This is achieved through a parameter... neural networks To estimate the PMI. The network is trained by maximizing The variational lower bound is realized when the optimal condition is reached. .

[0090] In practice, inter-point mutual information is used: .

[0091] Step 5: MAAC-R Algorithm. Based on the experience playback and sharing mechanism, an experience-sharing, mutually beneficial reward multi-agent evaluation-action network algorithm is proposed. All UAVs share a single evaluation-action network and a PMI network. The training and execution process of the MAAC-R algorithm is as follows: Figure 1 As shown, the details are as follows:

[0092] S1. Initialize Action Network Parameters Evaluate network parameters Target network parameters PMI network parameters and action-judgment experience replay buffer PMI Experience Replay Buffer .

[0093] S2. In each training cycle:

[0094] a. Reset the environment and obtain joint local information for all drones. .

[0095] b. For each step :

[0096] i. Each drone i follows the strategy output by the shared action network. And explore strategy selection actions .

[0097] ii. Perform joint actions Receive instant rewards from the United Environment and new joint local information .

[0098] iii. For each drone i, obtain its set of neighboring drones. The dependency index with each neighboring UAV j is calculated using the PMI network. Pair the data ( Store in the PMI buffer.

[0099] iv. Based on the formula in step three, reshape the reward of drone i using the dependency index and the environmental reward from its neighbors. Transfer tuples ( Store in the AC buffer.

[0100] c. Sample from the action-judgment experience replay buffer and update the judge-action network parameters.

[0101] d. Sample from the PMI experience replay buffer and update the PMI network parameters.

[0102] S3. After training, the shared action network policy is deployed to all drones. During the execution phase, each drone operates based solely on its own local information. Make action decisions in a decentralized manner No need to calculate reciprocal rewards.

[0103] Example:

[0104] First, a multi-target tracking simulation platform for UAV swarms was built based on Python and reinforcement learning frameworks (such as PyTorch). The environmental parameters were set as follows: the task area was a 2km × 2km square; the number of UAVs was 10, their speed was 20m / s, their maximum heading angular rate was 30° / s, their communication range was 500m, and their perception range was 200m; the number of targets was 10, and their speed was 5m / s. The MAAC-R algorithm hyperparameters were set as follows: discount factor γ = 0.95, reciprocal reward weight α = 0.3, action network learning rate 1 × 10⁻⁴, judge network learning rate 5 × 10⁻⁴, target network soft update rate τ = 0.01, experience replay buffer size 1 × 10⁵, and batch size 128. The PMI network was a multilayer perceptron.

[0105] The MAAC-R algorithm was trained using the aforementioned parameters, with a total training epoch of 1000 and a maximum step size of 200 per epoch. During training, the exploration rate β was linearly annealed from 1 to 0. The training results are as follows: Figure 4 As shown, MAAC-R outperforms the benchmark algorithms MAAC and MAAC-G in both average number of tracked targets and total number of tracked targets in the cluster, demonstrating the effectiveness of reciprocal rewards in improving collaboration. After training, the trained action network policy is loaded and tested in scenarios of the same and different scales. In a test scenario with 10 drones and 10 targets, 100 random runs were performed, and MAAC-R achieved the highest average environmental reward and average number of tracked targets. Figure 3As shown, the UAV exhibits rich cooperative behaviors, and the learned strategy can maintain an excellent tracking success rate (tracking ratio), demonstrating the good scalability and generalization ability of the method of the present invention.

Claims

1. A multi-target cooperative tracking and guidance method for UAV swarms based on reinforcement learning, characterized in that, The method includes: Step 1: Model and formally describe the problem of multi-target tracking task of UAV swarm; Step 2: Design the drone's reward function, including target tracking reward, repeated tracking penalty, and boundary penalty, and obtain the drone's environmental reward at each step; Step 3: Propose a method for maximizing reciprocal rewards, defining and calculating reciprocal rewards; specifically including: To improve inter-drone collaboration, the instant reward for each drone has been modified as follows: in, For reciprocal rewards, For balance coefficient, For drones Environmental rewards at every step; Reciprocal rewards are defined as: in, Indicates drone A collection of adjacent drones, Measuring drones and The degree of dependence between them These are the normalization coefficients; Step 4: Construct a PMI neural network to estimate the dependency index among drones; Step 5: Based on the experience replay sharing mechanism, construct an experience-sharing reciprocal reward multi-agent evaluation-action network algorithm that includes a shared action network, a shared evaluation network, a target evaluation network, and a PMI neural network; train the shared action network and the shared evaluation network using the reshaped instantaneous reward; after training, deploy the shared action network strategy to all UAVs; during the execution phase, each UAV makes action decisions in a decentralized manner based only on its own local information.

2. The method for multi-target cooperative tracking and guidance of UAV swarms based on reinforcement learning according to claim 1, characterized in that: Step 1 specifically includes: assuming there are n isomorphic small quadcopter drones and m isomorphic targets in the task scenario; the drones fly at a constant speed in a two-dimensional plane, and their actions are bounded heading angular rates; the targets wander randomly in the environment; the drones perceive the targets, and can perceive and track the targets when they are within the perception range, but cannot distinguish the specific identities of the targets; at the same time, the drones can receive local communication information from neighboring drones and obtain local information relative to the boundary of the task area; the decision-making process is modeled as a distributed partially observable Markov decision process Dec-POMDP, which includes a joint state space, a joint action space, state transition probabilities, a joint reward function, a joint observation space, and an individual observation model.

3. The method for multi-target cooperative tracking and guidance of UAV swarms based on reinforcement learning according to claim 1, characterized in that: In step 2, the target tracking reward is defined as follows: in, For drones For the target Tracking rewards, For drones With the goal The relative distance, For maximum sensing distance; drone The reward for tracking multiple targets is .

4. The multi-target cooperative tracking and guidance method for UAV swarms based on reinforcement learning according to claim 1, characterized in that: In step 2, the repeated tracking penalty is defined as follows: in, The penalty for repeated tracking between drones i and j is given by the total penalty for repeated tracking of drone i. .

5. The method for multi-target cooperative tracking and guidance of UAV swarms based on reinforcement learning according to claim 1, characterized in that: In step 2, the boundary penalty is defined as follows: in, This represents the minimum distance from the drone to the boundary.

6. The method for multi-target cooperative tracking and guidance of UAV swarms based on reinforcement learning according to claim 1, characterized in that: Step 4 specifically includes: Using point-to-point mutual information to capture dependencies between drones, for drones and Its dependency index is defined as: in, Representing drones Local information, For communication information, For observation information, For boundary information, Status information; For each drone Action decision-making; Representing local information-action pairs and The joint probability distribution; through a parameter of neural networks The inter-point mutual information is estimated, and the network is trained by maximizing the Jensen-Shannon mutual information. The variational lower bound is achieved, and the output of the PMI neural network satisfies: .

7. The multi-target cooperative tracking and guidance method for UAV swarms based on reinforcement learning according to claim 1, characterized in that: Step 5 specifically includes: Step 5.1: Initialize shared action network parameters Shared evaluation network parameters Target evaluation network parameters PMI network parameters and action-judgment experience replay buffer PMI Experience Replay Buffer Among them, the shared action network is used to output action strategies based on local information of the UAV. Shared evaluation networks are used to estimate action value functions. The target evaluation network is used to construct target values ​​when updating the shared evaluation network, and the PMI network is used to estimate the drone dependency index. ; Step 5.2: In each training cycle: Step 5.2.1: Reset the environment and obtain joint local information of all UAVs. ; Step 5.2.2: For each step : Each drone According to the sharing policy and exploration strategy selection action decision ; Execute joint operations Receive instant rewards from the United Environment and new joint local information ; For each drone Obtain the set of its neighboring drones. And use the PMI neural network to calculate the relationship with each neighboring drone. Dependence index ;Pair the data ( Store in PMI buffer ; Based on step 3, reshape the drone using the dependency index and neighbor's environmental rewards. Instant rewards ; transfer tuple Save Action - Evaluation Experience Replay Buffer , For drones New local information; Step 5.2.3: Replay Buffer of Action-Judgment Experience sampling Sample , For the first Current local information of each sample For the first The current action of each sample. For the first Immediate rewards for reshaping individual samples For the first Local information of each sample at the next moment; Construct the target value using the target network: ; For discount rate, For the shared action network based on the next time-of-motion local information The next action to output, To evaluate the network's local information for the next time step With the next action Estimated value of corresponding action; definition of loss function for shared evaluation network. By minimizing the loss function Update shared evaluation network parameters Define the objective function of the shared action network. ,in, This indicates that the shared action network is in local information Down Output Action The probability is obtained by maximizing the objective function. Update shared action network parameters This allows for the maximization of learning of the reshaped reward; After completing the shared evaluation network parameters After the update, the parameters of the target network are... Perform a soft update. ,in, This is the soft update coefficient, with a value ranging from 0 to 1; Step 5.2.4: Replaying the PMI Experience Buffer Medium sampling Positive samples and construct One mismatched sample Among them, positive samples This represents local information about the actual matching between adjacent drones at the same time—action pairs; mismatched samples. This indicates that the first half should be retained. The original text remains unchanged, but the latter half is replaced with a random sample obtained from the PMI experience playback buffer. The resulting local information—action mismatch; Define the loss function of the PMI network as follows: in, This represents the PMI network's estimated output for positive sample dependencies. This represents the estimated output of the PMI network for the dependency of mismatched samples; by minimizing the loss function... Update PMI network parameters ; Step 5.3: Repeat step 5.2 until training is complete, obtaining the trained shared action network. After the strategy training is completed, the shared action network strategy is deployed to all drones; during the execution phase, each drone will share its local information. Input the same shared action network and output the corresponding action decision. This allows for multi-target collaborative tracking and guidance in a decentralized manner.

8. A multi-target cooperative tracking and guidance method system for UAV swarms based on reinforcement learning, characterized in that: The system has a program module corresponding to the steps of any one of claims 1-7, and executes the steps in the reinforcement learning-based UAV swarm multi-target cooperative tracking and guidance method when running.

9. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, it performs the steps of the multi-target cooperative tracking and guidance method for UAV swarm based on reinforcement learning as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium is used to store a computer program that executes a multi-target cooperative tracking and guidance method for UAV swarms based on reinforcement learning, as described in any one of claims 1-7.

Images