Multi-agent-based unmanned aerial vehicle cluster two-stage cooperative multi-target detection method

Abstract

The invention discloses an unmanned aerial vehicle cluster two-stage cooperative multi-target detection method based on multiple agents, and belongs to the technical field of unmanned aerial vehicle communication, and the method comprises the following steps: S1, building a TDGN module which outputs the prediction of all possible actions of an unmanned aerial vehicle according to the original observation data of the unmanned aerial vehicle, training a TDGN module in a virtual simulation environment to obtain a first model and a second model; s2, in a detection stage, outputting initial actions of the unmanned aerial vehicles by adopting a position dispersion strategy according to the first model, and dispersing the unmanned aerial vehicles in the unmanned aerial vehicle cluster; and S3, in the aggregation stage, outputting the initial action of the unmanned aerial vehicle by adopting a cluster action consistency strategy according to the second model, and ensuring that the actions of the unmanned aerial vehicles in the unmanned aerial vehicle cluster are consistent. The invention discloses an unmanned aerial vehicle cluster two-stage cooperative multi-target detection method for establishing an STDGNet framework, which is used for realizing multi-point monitoring in a wide-area environment and improving the target searching and tracking efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle (UAV) communication technology, and in particular relates to a two-stage collaborative multi-target detection method for UAV clusters based on multi-agents. Background Art

[0002] In recent years, with the advancement of the Internet of Things (IoT) technology, unmanned aerial vehicle (UAV) swarms have become a key component in both civilian and military applications, playing a vital role in smart cities and the development of the low-altitude economy. UAV swarms communicate through inter-UAV data links, enabling collaborative interaction and demonstrating exceptional performance in executing complex tasks in unknown environments. Among them, multi-target detection by UAVs stands out as a key example of collaborative tasks involving multiple UAVs, playing a particularly important role in real-world applications such as forest fire monitoring, military operations, and post-disaster rescue missions.

[0003] The effectiveness of completing the UAV multi-target detection mission depends on the swarm's ability to detect and collect information on multiple targets within a given area in an unknown and dynamic environment. This mission can be divided into two distinct phases, each with unique characteristics and objectives. The first phase is the detection phase, in which the UAV's goal is to quickly search and cover the unknown scene within a limited time to detect as many targets as possible. The second phase is the aggregation phase, in which the UAV re-establishes communication links and aggregates all information collected from the entire swarm, laying the foundation for subsequent online processing of the swarm data. Given the unknown and dynamic nature of the environment, the swarm's collaborative capabilities are crucial to effectively completing the UAV multi-target detection mission. The two-phase multi-target detection task can be viewed as two collaborative decision-making problems with different optimization objectives.

[0004] Multi-agent deep reinforcement learning (MADRL) has made significant progress in addressing the challenges of collaborative decision-making in drone swarms. MADRL enables individual agents to develop cooperative strategies aimed at improving their collective reward potential and ultimately achieving specified goals. In this framework, decentralized partially observable Markov decision processes (Dec-POMDPs) serve as the key model for multi-agent systems. MADRL methods, such as MADDPG, QMIX, VDN, and MAAC, adopt the CTDE (Centralized Training with Decentralized Execution) paradigm, in which the training of agents is performed independently and the policy network generates actions based on the local observation data of each agent. Summary of the Invention

[0005] In response to the above-mentioned deficiencies in the existing technology, the two-stage collaborative multi-target detection method of drone clusters based on multi-agents provided by the present invention solves the problem that the performance of existing drone cluster collaborative decision-making in the detection and aggregation process in different scenarios needs to be improved.

[0006] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a two-stage collaborative multi-target detection method of a multi-agent UAV cluster, comprising the following steps:

[0007] S1. Establish a TDGN module. The TDGN module outputs predictions of all possible actions of the UAV based on the original observation data of the UAV. The TDGN module is trained in a virtual simulation environment to obtain a first model and a second model.

[0008] S2. In the detection phase, the position dispersion strategy is used according to the first model to output the initial action of the drone and disperse the drones in the drone cluster;

[0009] S3. In the aggregation stage, the cluster action consistency strategy is adopted according to the second model to output the initial action of the drone to ensure the consistency of the drone actions in the drone cluster.

[0010] Further: in said S1, the TDGN module includes a feature extractor, a relation encoder and a strategy decoder connected in sequence;

[0011] The feature extractor is equipped with a two-layer Transformer block with a self-attention mechanism to extract the features of the drone's local observation of the environment;

[0012] The relation encoder uses a multi-head dot-product attention mechanism to aggregate information from neighboring drones;

[0013] The policy decoder is used to calculate Q values for different actions in different states to learn to select actions.

[0014] The beneficial effects of this further approach are as follows: the feature extractor effectively extracts contextual information from drone observations, guiding neighboring drones in the same cluster to focus on similar target features, thereby enhancing collaboration. The relation encoder simulates interactions with information from other drones in the same cluster, alleviating the limitations of a single observation domain and enriching the decision input data. Finally, the policy decoder learns to select actions by scoring different actions in different states.

[0015] Furthermore, in S1, the TDGN module outputs a prediction method of all possible actions of the drone based on the original observation data of the drone, including the following steps:

[0016] S11, inputting the original observation data of the UAV into the feature extractor, using the first loss function to supervise the feature extractor to extract features from the original observation data of the UAV to obtain a feature vector;

[0017] S12. Input the feature vector into the relational encoder, use the second loss function to supervise the relational encoder to calculate the relationship between the drones through the multi-head dot product attention mechanism, merge the feature vectors of adjacent drones according to the relationship, and obtain the latent feature;

[0018] S13. Input the latent features into the policy decoder, calculate the Q value of all possible actions of the drone based on the latent features, and update the Q value of the drone through the third loss function as the prediction of all possible actions of the drone.

[0019] Further: In said S11, the characteristic vector h of the i-th UAV is obtained i The specific expression is:

[0020] h i =Norm(Atten i +o i +MLP(Atten i +o i ))

[0021] In the formula, o i is the original observation data of the i-th UAV, MLP(·) is the multi-layer perceptron function, Norm(·) is the layer normalization operation, Atten i is the attention parameter of the i-th drone, and its specific expression is:

[0022]

[0023] Where, is the first learnable self-attention weight, is the second learnable self-attention weight, is the third learnable self-attention weight, softmax(·) is the normalized exponential function, T is the transposed symbol, d k is a vector dimension.

[0024] Further: In S11, the first loss function L α The specific expression is:

[0025]

[0026] Where, is the feature vector of the i-th UAV in the same cluster at time step t, is the feature vector of the jth UAV in the same cluster at time step t, θ is the learning parameter of the feature extraction process, and N is the number of UAVs.

[0027] The beneficial effect of the above further solution is: in order to enhance the coordination ability between adjacent drones in the same cluster and improve the overall efficiency of the model in completing collaborative tasks, the TDGN module is designed to guide them to pay more attention to the same or similar targets. The present invention uses Euclidean distance as the loss metric and designs a new loss function, which is expressed as the first loss function L α .

[0028] Furthermore, in S12, the method for merging the feature vectors of adjacent drones is specifically as follows:

[0029] The multi-head dot product attention mechanism is used to calculate the relationship between drones. Based on the relationship, the feature vectors of all adjacent drones are weighted and summed to obtain the potential feature, the potential feature h′ of the i-th drone i The specific expression is:

[0030]

[0031] Where σ(·) is a single-layer MLP with ReLU nonlinearity, concatenate(·) is the fully connected function, The mth attention head represents the relationship between the i-th UAV and the j-th UAV, h j is the feature vector of the j-th drone, M is the number of attention heads, is the weight of the value vector;

[0032]

[0033] Where τ is the proportional factor, h i is the feature vector of the i-th UAV, is the weight of the query vector, is the weight of the key vector, h k is the feature vector of the kth UAV, where k is the serial number of the UAV’s adjacent nodes, k∈B i , B i are the neighboring nodes of drone node i.

[0034] The beneficial effect of the above further solution is that the present invention adopts a multi-head dot product attention mechanism, which can effectively aggregate information from neighboring drones, thereby optimizing the communication and collaborative decision-making process within the cluster, and solving the potential redundancy and inefficiency problems associated with directly overlapping or connecting communication data between neighboring drones.

[0035] Further: In said S12, the second loss function L βThe specific expression is:

[0036]

[0037] Where, is the attention weight distribution of the relationship representation of the i-th drone at the convolutional layer k, θ is the learning parameter of the feature extraction process, is the observation of the i-th UAV at time step t, is the observation of the i-th UAV at time step t+1, Adj is the adjacency matrix of the UAV cluster communication connection topology graph, D KL is the KL divergence.

[0038] Further: In said S13, the third loss function L γ The specific expression is:

[0039]

[0040] Where S is the size of the data taken from the Replay Buffer during training, s is one set of data, and y i is the relevant information of the i-th UAV, Q(·) is the Q value of the UAV, O i,Adj is the set of observations within the receptive field of the i-th UAV, a i is the action of the i-th drone;

[0041] y i =r i +γmax a′ Q(O i,Adj ,a′ i ;θ′)

[0042] Where r i is the reward of the i-th drone, γ is the discount factor, a′ i is the action with the largest Q value, θ′ is the weight of the target network, max a′ The action to calculate the maximum Q value in the target network;

[0043] The expression of the total loss L(θ) of the TDGN module is specifically:

[0044] L(θ)=αL α +βL β +γL γ

[0045] Wherein, α is the first proportional coefficient, β is the second proportional coefficient, and γ is the third proportional coefficient.

[0046] The beneficial effect of the above further scheme is: in order to ensure the stability of the learned attention weight distribution over time, thereby improving the stability of feature encoding information, the present invention designs a regularization method based on time relationship, by minimizing the KL divergence of the encoding information between consecutive time steps as a regularization term. This method encourages drones to form consistent relationship representations over time and promotes more stable cooperation. The time relationship regularization loss is corrected with the second loss function L β To express.

[0047] Furthermore, in S2, the initial action of the drone outputted using the position dispersion strategy according to the first model is specifically an action of moving the drone to a position farthest from all other drones in the drone cluster.

[0048] Furthermore, in S3, the initial action of the drone outputted using the cluster action consistency strategy according to the second model is specifically: the most frequent action of the drones in the drone cluster.

[0049] The beneficial effects of the present invention are:

[0050] (1) The present invention provides a two-stage collaborative multi-target detection method for drone clusters based on multi-agents. The STDGNet framework, which includes a TDGN module, a position dispersion strategy, and a cluster motion consistency strategy, is used to effectively deal with various multi-target detection problems in complex scenarios. The TDGN module is designed to extract features from the observation and interaction dynamics between drones to generate collaborative strategies. The integration of the position dispersion strategy and the cluster motion consistency strategy enables the STDGNet framework to adapt well to multi-stage tasks. In the detection phase, the position dispersion strategy keeps the drones dispersed to speed up target discovery; in the aggregation phase, the cluster motion consistency strategy ensures that drones in the same group move in the same direction, promoting the formation of an interconnected communication network. Comprehensive experiments conducted in a designed environment show that the STDGNet framework exhibits excellent performance, outperforming existing methods. Moreover, the strategies learned from the STDGNet framework consistently show excellent results in the detection and aggregation processes in different scenarios, demonstrating its robustness in different types of tasks.

[0051] (2) The present invention establishes a two-stage collaborative multi-target detection method for drone clusters based on the STDGNet framework, which is used to achieve multi-point monitoring in a wide-area environment and improve the efficiency of target search and tracking; to achieve accurate target positioning in complex terrain and enhance the ability of emergency response and rescue operations; and to improve the real-time performance of data processing and analysis in fields such as military reconnaissance and civilian monitoring, and optimize decision support systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1This is a flow chart of the two-stage collaborative multi-target detection method of multi-agent-based drone clusters of the present invention.

[0053] Figure 2 It is the STDGNet framework structure diagram of the present invention.

[0054] Figure 3 This is a diagram of the UAV multi-target detection simulation environment of the present invention.

[0055] Figure 4 This is the structural diagram of the TDGN module of the present invention.

[0056] Figure 5 This is a diagram of the experimental results of the present invention. DETAILED DESCRIPTION

[0057] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0058] like Figure 1 As shown, in one embodiment of the present invention, a two-stage collaborative multi-target detection method of a multi-agent UAV cluster includes the following steps:

[0059] S1. Establish a TDGN module (a Transformer-based deep graph network). The TDGN module outputs predictions of all possible UAV actions based on the original observation data of the UAV. The TDGN module is trained in a virtual simulation environment to obtain the first model and the second model.

[0060] S2. In the detection phase, the position dispersion strategy is used according to the first model to output the initial action of the drone and disperse the drones in the drone cluster;

[0061] S3. In the aggregation stage, the cluster action consistency strategy is adopted according to the second model to output the initial action of the drone to ensure the consistency of the drone actions in the drone cluster.

[0062] In this embodiment, the present invention proposes a new integrated DRL framework, STDGNet, based on the classic deep reinforcement learning algorithm Deep Graph Network (DGN). Figure 2 According to the characteristics of multi-UAV cluster system, a multi-UAV target detection simulation environment is designed, as shown in Figure 3The core goal of the DGN algorithm is to promote communication between agents through graph convolution, thereby achieving cooperation between agents.

[0063] In S1, the TDGN module includes a feature extractor, a relation encoder, and a strategy decoder connected in sequence;

[0064] The feature extractor is equipped with a two-layer Transformer block with a self-attention mechanism to extract the features of the drone's local observation of the environment;

[0065] The relation encoder uses a multi-head dot-product attention mechanism to aggregate information from neighboring drones;

[0066] The policy decoder is used to calculate Q values for different actions in different states to learn to select actions.

[0067] like Figure 4 As shown in Figure 1, in this embodiment, the TDGN module consists of three core submodules: a feature extractor, a relation encoder, and a policy decoder. The feature extractor effectively extracts contextual information from drone observations, guiding neighboring drones in the same cluster to focus on similar target features, thereby enhancing collaboration. The relation encoder simulates interactions with information from other drones in the same cluster, alleviating the limitations of a single observation domain and enriching decision input data. Finally, the policy decoder learns to select actions by scoring different actions in different states.

[0068] In S1, the TDGN module outputs a prediction method for all possible actions of the drone based on the original observation data of the drone, which includes the following steps:

[0069] S11, inputting the original observation data of the UAV into the feature extractor, using the first loss function to supervise the feature extractor to extract features from the original observation data of the UAV to obtain a feature vector;

[0070] S12. Input the feature vector into the relational encoder, use the second loss function to supervise the relational encoder to calculate the relationship between the drones through the multi-head dot product attention mechanism, merge the feature vectors of adjacent drones according to the relationship, and obtain the latent feature;

[0071] S13. Input the latent features into the policy decoder, calculate the Q value of all possible actions of the drone based on the latent features, and update the Q value of the drone through the third loss function as the prediction of all possible actions of the drone.

[0072] In this embodiment, the values corresponding to all possible actions of the drone are calculated, and the actions that the drone may choose are determined by selecting the action with the highest Q value.

[0073] In S11, the characteristic vector h of the i-th drone is obtained. i The specific expression is:

[0074] h i =Norm(Atten i +o i +MLP(Atten i +o i ))

[0075] In the formula, o i is the original observation data of the i-th UAV, MLP(·) is the multi-layer perceptron function, Norm(·) is the layer normalization operation, Atten i is the attention parameter of the i-th drone, and its specific expression is:

[0076]

[0077] Where, is the first learnable self-attention weight, is the second learnable self-attention weight, is the third learnable self-attention weight, softmax(·) is the normalized exponential function, T is the transposed symbol, d k is a vector dimension.

[0078] In this embodiment, given the limited observation range of a single drone, inter-drone communication is crucial for sharing key information such as observation data, location, and velocity. Directly using raw observation data in message exchange can lead to increased communication losses and include irrelevant or harmful information, thus affecting mission efficiency. To address this issue, the present invention employs a two-layer Transformer block with a self-attention mechanism to efficiently extract features from the drone's raw observation data and generate feature vectors.

[0079] In order to enhance the coordination ability between adjacent drones in the same cluster and improve the overall efficiency of the model in completing collaborative tasks, the TDGN module aims to guide them to pay more attention to the same or similar targets. This paper uses Euclidean distance as the loss metric and designs a new loss function, which is expressed as the first loss function L α .

[0080] In S11, the first loss function L α The specific expression is:

[0081]

[0082] Where, is the feature vector of the i-th UAV in the same cluster at time step t, is the feature vector of the jth UAV in the same cluster at time step t, θ is the learning parameter of the feature extraction process, and N is the number of UAVs.

[0083] In this embodiment, and The feature vectors of drones belonging to the same cluster at time step t are encoded by minimizing the feature vectors of drones in the same cluster {h1,h2,…,h N}, the model can better identify and extract valuable features from the original observations of the UAV, thereby accelerating the convergence efficiency of the collaborative task training process.

[0084] In S12, the method for merging the feature vectors of adjacent drones is specifically as follows:

[0085] The multi-head dot product attention mechanism is used to calculate the relationship between drones. Based on the relationship, the feature vectors of all adjacent drones are weighted and summed to obtain the potential feature, the potential feature h′ of the i-th drone i The specific expression is:

[0086]

[0087] Where σ(·) is a single-layer MLP with ReLU nonlinearity, concatenate(·) is the fully connected function, The mth attention head represents the relationship between the i-th UAV and the j-th UAV, h j is the feature vector of the j-th drone, M is the number of attention heads, is the weight of the value vector;

[0088] In this embodiment, the relation encoder is a key component designed to facilitate information exchange between drones within the same cluster and compute their interdependencies across different states, thereby improving the performance of collaborative tasks. To address the potential redundancy and inefficiency associated with directly overlapping or connecting communication data between neighboring drones, the present invention employs a multi-head dot-product attention mechanism, which effectively aggregates information from neighboring drones to optimize communication and collaborative decision-making within the cluster. For each observed feature of a drone, it is represented as a query, key, and value for each independent attention head.

[0089] For the mth attention head, the mth attention head represents the relationship between the i-th drone and the j-th drone. The calculation is as follows:

[0090]

[0091] Where τ is the proportional factor, h iis the feature vector of the i-th UAV, is the weight of the query vector, is the weight of the key vector, h k is the feature vector of the kth UAV, where k is the serial number of the UAV’s adjacent nodes, k∈B i , B i are the neighboring nodes of drone node i.

[0092] In this embodiment, due to the movement of drones within the cluster at each time step, the number of adjacent drones may change, which may cause large fluctuations in the neighbor interaction information of the input module between adjacent time steps. In order to ensure the stability of the learned attention weight distribution over time, thereby improving the stability of feature encoding information, the present invention designs a regularization method based on temporal relations. By minimizing the KL divergence of the encoded information between consecutive time steps as a regularization term, this method encourages drones to form consistent relationship representations over time, promoting more stable cooperation. The temporal relationship regularization loss is corrected with the second loss function L β To express.

[0093] In S12, the second loss function L β The specific expression is:

[0094]

[0095] Where, is the attention weight distribution of the relationship representation of the i-th drone at the convolutional layer k, θ is the learning parameter of the feature extraction process, is the observation of the i-th UAV at time step t, is the observation of the i-th UAV at time step t+1, Adj is the adjacency matrix of the UAV cluster communication connection topology graph, D KL is the KL divergence.

[0096] In this embodiment, the function of the policy decoder is to decode the latent features into individual drone actions, and use a single fully connected layer to calculate the Q value corresponding to each drone action. The ultimate goal is to learn a policy that maximizes the Q value of the optimal action. At each time step, after the drone's action and interaction with the environment, it obtains an observation value, reward, adjacency matrix representing the topology of the communication connection between drones, and other related information. These data elements are systematically stored in the experience replay buffer D at each time step. During the training phase, experience is randomly sampled from the buffer D and passed through the third loss function L γ Update the Q value of the drone. γ The specific expression is:

[0097]

[0098] Where S is the size of the data taken from the Replay Buffer during training, s is one set of data, and y i is the relevant information of the i-th UAV, Q(·) is the Q value of the UAV, O i,Adj is the set of observations within the receptive field of the i-th UAV, a i is the action of the i-th drone;

[0099] y i =r i +γmax a′ Q(O i,Adj ,a′ i ;θ′)

[0100] Where r i is the reward of the i-th drone, γ is the discount factor, a′ i is the action with the largest Q value, θ′ is the weight of the target network, max a′ The action to calculate the maximum Q value in the target network;

[0101] The expression of the total loss L(θ) of the TDGN module is specifically:

[0102] L(θ)=αL α +βL β +γL γ

[0103] Wherein, α is the first proportional coefficient, β is the second proportional coefficient, and γ is the third proportional coefficient.

[0104] In S2, the initial action of the drone outputted using the position dispersion strategy according to the first model is specifically an action of moving the drone to a position farthest from all other drones in the drone cluster.

[0105] In this embodiment, in order to improve the performance of TDGN in the target detection process, the present invention adopts a position dispersion strategy. This strategy is conducive to the dispersion of drones, thereby expanding the search area to identify more targets. When no mission-related targets are observed, the position dispersion strategy guides the drone to disperse throughout the environment. Specifically, the action output by the position dispersion strategy is to move the drone i to a distance The farthest position among all other drones. Because when there is no target related to the mission in the drone's observation, the observation result is just useless information. Therefore, it is difficult for the model used by the drone to output the best action to complete the mission. When only one UAV or UAV cluster can observe the target related to the mission, the location dispersion strategy uses the first model Π D The output initial action of the UAV is used as the initial action of the detection phase.

[0106] In S3, the initial action of the drone outputted using the cluster action consistency strategy according to the second model is specifically: the most frequent action of the drones in the drone cluster.

[0107] In this embodiment, in order to improve the performance of TDGN in the aggregation process, a cluster motion consistency strategy is introduced. This strategy aims to ensure the consistency of motion between drones in the same cluster, thereby promoting the rapid reconstruction of communication connectivity within the drone swarm network. Specifically, if If there are multiple drones, the cluster action consistency strategy will be in the second model Π A Find the most frequent action in the output and use it as the initial action of the drone. When there is only one drone in the cluster, the cluster action consistency strategy uses the second model Π A The output initial action of the drone is used as the initial action of the aggregation stage.

[0108] like Figure 5 As shown in the figure, the comprehensive experiment of this paper is carried out in the designed environment, where Map size is the width of the environment (the environment is square), N U is the number of drones, N F is the fixed target number, N M is the number of moving targets, N T is the number of traps, Meanreward is the average reward, Meanfixedtarg. is the average number of fixed targets detected, Meanmobiletarg. is the average number of mobile targets detected, Meantraps is the average number of trap entries, and MeanfinalCt is the final average number of drone clusters. Comprehensive experiments show that the STDGNet framework exhibits excellent performance, outperforming existing methods.

[0109] The beneficial effects of the present invention are as follows: the present invention provides a two-stage collaborative multi-target detection method for drone clusters based on multi-agents, and adopts the STDGNet framework including the TDGN module, the position dispersion strategy and the cluster action consistency strategy to effectively deal with various multi-target detection problems in complex scenarios. The TDGN module is designed to extract features from the observation and interaction dynamics between drones to generate collaborative strategies. The integration of the position dispersion strategy and the cluster action consistency strategy enables the STDGNet framework to adapt well to multi-stage tasks. In the detection phase, the position dispersion strategy keeps the drones dispersed to speed up target discovery; in the aggregation phase, the cluster action consistency strategy ensures that drones in the same group move in the same direction, promoting the formation of an interconnected communication network. Comprehensive experiments conducted in a designed environment show that the STDGNet framework exhibits excellent performance, outperforming existing methods. And the strategies learned from the STDGNet framework consistently show excellent results in the detection and aggregation processes in different scenarios, proving its robustness in different types of tasks.

[0110] The present invention establishes a two-stage collaborative multi-target detection method for UAV swarms based on the STDGNet framework, which is used to achieve multi-point monitoring in wide-area environments and improve the efficiency of target search and tracking; to achieve precise target positioning in complex terrain and enhance the capabilities of emergency response and rescue operations; and to improve the real-time performance of data processing and analysis and optimize decision support systems in fields such as military reconnaissance and civilian surveillance.

[0111] In the description of the present invention, it should be understood that the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying the relative importance or the number of technical features implicitly specified. Therefore, the features defined by "first", "second", and "third" may explicitly or implicitly include one or more of such features.

Claims

1. A two-stage collaborative multi-target detection method for UAV swarms based on multi-agents, characterized by: The following steps are involved: S1. Establish a TDGN module. The TDGN module outputs predictions of all possible actions of the UAV based on the original observation data of the UAV. The TDGN module is trained in a virtual simulation environment to obtain a first model and a second model. S2. In the detection phase, the position dispersion strategy is used according to the first model to output the initial action of the drone and disperse the drones in the drone cluster; S3. In the aggregation stage, the cluster action consistency strategy is adopted according to the second model to output the initial action of the drone to ensure the consistency of the drone actions in the drone cluster.

2. The two-stage collaborative multi-target detection method of multi-agent UAV swarm based on claim 1 is characterized in that: In S1, the TDGN module includes a feature extractor, a relation encoder, and a strategy decoder connected in sequence; The feature extractor is equipped with a two-layer Transformer block with a self-attention mechanism to extract the features of the drone's local observation of the environment; The relation encoder uses a multi-head dot-product attention mechanism to aggregate information from neighboring drones; The policy decoder is used to calculate Q values for different actions in different states to learn to select actions.

3. The two-stage collaborative multi-target detection method of multi-agent UAV swarm based on claim 2 is characterized in that: In S1, the TDGN module outputs a prediction method for all possible actions of the drone based on the original observation data of the drone, which includes the following steps: S11, inputting the original observation data of the UAV into the feature extractor, using the first loss function to supervise the feature extractor to extract features from the original observation data of the UAV to obtain a feature vector; S12. Input the feature vector into the relational encoder, use the second loss function to supervise the relational encoder to calculate the relationship between the drones through the multi-head dot product attention mechanism, merge the feature vectors of adjacent drones according to the relationship, and obtain the latent feature; S13. Input the latent features into the policy decoder, calculate the Q value of all possible actions of the drone based on the latent features, and update the Q value of the drone through the third loss function as the prediction of all possible actions of the drone.

4. The two-stage collaborative multi-target detection method of multi-agent UAV swarm based on claim 3 is characterized in that: In S11, the characteristic vector h of the i-th drone is obtained. i The specific expression is: h i =Norm(Atten i +o i +MLP(Atten i +o i )) In the formula, o i is the original observation data of the i-th UAV, MLP(·) is the multi-layer perceptron function, Norm(·) is the layer normalization operation, Atten i is the attention parameter of the i-th drone, and its specific expression is: Where, is the first learnable self-attention weight, is the second learnable self-attention weight, is the third learnable self-attention weight, softmax(·) is the normalized exponential function, T is the transposed symbol, d k is a vector dimension.

5. The two-stage collaborative multi-target detection method of multi-agent UAV swarm based on claim 4 is characterized in that: In S11, the first loss function L α The specific expression is: Where, is the feature vector of the i-th UAV in the same cluster at time step t, is the feature vector of the jth UAV in the same cluster at time step t, θ is the learning parameter of the feature extraction process, and N is the number of UAVs.

6. The two-stage collaborative multi-target detection method of multi-agent UAV swarm based on claim 5 is characterized in that: In S12, the method for merging the feature vectors of adjacent drones is specifically as follows: The multi-head dot product attention mechanism is used to calculate the relationship between drones. Based on the relationship, the feature vectors of all adjacent drones are weighted and summed to obtain the potential feature, the potential feature h′ of the i-th drone i The specific expression is: Where σ(·) is a single-layer MLP with ReLU nonlinearity, concatenate(·) is the fully connected function, The mth attention head represents the relationship between the i-th UAV and the j-th UAV, h j is the feature vector of the j-th drone, M is the number of attention heads, is the weight of the value vector; Where τ is the proportional factor, h i is the feature vector of the i-th UAV, is the weight of the query vector, is the weight of the key vector, h k is the feature vector of the kth UAV, where k is the serial number of the UAV’s adjacent nodes, k∈B i , B i are the neighboring nodes of drone node i.

7. The two-stage collaborative multi-target detection method of multi-agent UAV swarm based on claim 6 is characterized in that: In S12, the second loss function L β The specific expression is: Where, is the attention weight distribution of the relationship representation of the i-th drone at the convolutional layer k, θ is the learning parameter of the feature extraction process, is the observation of the i-th UAV at time step t, is the observation of the i-th UAV at time step t+1, Adj is the adjacency matrix of the UAV cluster communication connection topology graph, D KL is the KL divergence.

8. The two-stage collaborative multi-target detection method of multi-agent UAV swarm according to claim 7 is characterized in that: In S13, the third loss function L γ The specific expression is: Where S is the size of the data taken from the Replay Buffer during training, s is one set of data, and y i is the relevant information of the i-th UAV, Q(·) is the Q value of the UAV, O i,Adj is the set of observations within the receptive field of the i-th UAV, a i is the action of the i-th drone; y i =r i +γmax a′ Q(O i,Adj ,a′ i ;θ′) Where r i is the reward of the i-th drone, γ is the discount factor, a′ i is the action with the largest Q value, θ′ is the weight of the target network, max a′ The action to calculate the maximum Q value in the target network; The expression of the total loss L(θ) of the TDGN module is specifically: L(θ)=αL α +βL β +γL γ Wherein, α is the first proportional coefficient, β is the second proportional coefficient, and γ is the third proportional coefficient.

9. The two-stage collaborative multi-target detection method of multi-agent UAV swarm according to claim 1 is characterized in that: In S2, the initial action of the drone outputted using the position dispersion strategy according to the first model is specifically an action of moving the drone to a position farthest from all other drones in the drone cluster.

10. The two-stage collaborative multi-target detection method of multi-agent UAV swarm based on claim 1 is characterized in that: In S3, the initial action of the drone outputted using the cluster action consistency strategy according to the second model is specifically: the most frequent action of the drones in the drone cluster.

Images