Multi-unmanned aerial vehicle target searching method based on self-attention and reinforcement learning

Abstract

A multi-unmanned aerial vehicle target searching method based on self-attention and reinforcement learning is characterized in that target searching tasks of unmanned aerial vehicles are converted into a multi-agent cooperation problem, each unmanned aerial vehicle is used as an independent agent, and target searching is carried out by adopting a strategy network SA-MADDPG; meanwhile, a search region is reasonably divided by adopting a Voronoi diagram, and a target probability graph TPM is used for helping to evaluate possible distribution of targets, so that the unmanned aerial vehicle preferentially searches a high-target probability region; finally, in the execution process of the search task, the search strategy is adjusted according to real-time feedback so as to adapt to the dynamic change of the target.

Description

Technical Field

[0001] The invention relates to a multi-UAV target search method based on self-attention and reinforcement learning, which belongs to the field of artificial intelligence and UAV swarm intelligent control. Background Art

[0002] In recent years, the use of drones has increased significantly in military, search and rescue, and environmental monitoring applications, particularly in search and reconnaissance missions. Drones, with their high speed, robust communications capabilities, and unrestricted terrain, are particularly well-suited for these missions in complex and dangerous environments. With the continuous advancement of technology, drones are demonstrating tremendous potential for exploring uncharted territory.

[0003] Multi-UAV collaborative target search refers to the process of UAVs detecting the target area through onboard sensors and sharing information through communication networks to collaboratively perform tasks, thereby significantly shortening the search time. Deep reinforcement learning (DRL) has been widely used in multi-agent collaborative tasks due to its self-learning and self-adaptation capabilities. In single-agent systems, methods based on shared experience (such as deep Q-network DQN) have been successfully applied to tasks such as target search and path planning. However, with the increase in task complexity, single-agent methods are difficult to meet the needs of multi-UAV collaboration. Therefore, multi-agent reinforcement learning (MARL) methods have emerged, especially frameworks with centralized training and decentralized execution (such as the "Multi-Agent Deep Deterministic Policy Gradient Algorithm" Multi-Agent Deep Deterministic Policy Gradient, MADDPG) have been widely used in collaborative decision-making and target search in multi-UAV systems.

[0004] Although multi-agent reinforcement learning methods have made significant progress in target search in drone swarms, dynamic target search in complex environments still faces some challenges: First, the algorithm performs poorly in environments with complex and dense obstacles. As the obstacle density and the size of the drone swarm increase, it becomes more difficult to capture global and local environmental features. Current MARL algorithms mainly rely on neural networks, such as multilayer perceptrons and convolutional neural networks, which have difficulty finding a balance between scalability and global information utilization. Second, when dealing with dynamic targets, the search efficiency is still insufficient, especially when the initial position of the target is unknown. Existing methods usually transform target search into a target tracking problem, but these methods perform poorly when the target moves quickly, and the search strategy that relies on target probability is inefficient when the target deviates from the predicted path. Summary of the Invention

[0005] In view of the above problems existing in the prior art, the present invention has made the following improvements:

[0006] First, the self-attention mechanism is introduced into the MADDPG algorithm to improve the drones' perception and collaborative decision-making capabilities. This mechanism enables each drone to prioritize relevant spatial features, such as obstacles and teammate positions, thereby enhancing its adaptability in complex environments.

[0007] Secondly, the target probability map TPM is combined with the region partition strategy based on the Voronoi diagram, and an exploration incentive mechanism is proposed to ensure the efficient distribution of drones, reduce redundant coverage, and promote the priority search of high-probability target areas, ultimately improving the search efficiency in dynamic target environments.

[0008] Specifically: The present invention is a multi-UAV collaborative dynamic target search method based on a multi-agent deep reinforcement learning method. This method converts the target search task of multiple UAVs into a multi-agent collaboration problem, and optimizes the UAV search strategy under the framework of centralized training and decentralized execution.

[0009] To address the shortcomings of existing methods in complex environments, this paper introduces a self-attention mechanism based on the traditional MADDPG algorithm. This allows each drone to autonomously identify and prioritize key spatial features relevant to target search, improving its adaptability and search efficiency in dynamic, obstacle-filled environments. Furthermore, this paper combines the search region partitioning strategy of the target probability map (TPM) with the Voronoi diagram to effectively avoid redundant coverage during the search process and ensure the collaborative search capabilities of the drone swarm under diverse environmental conditions.

[0010] The implementation steps of the method of the present invention include:

[0011] Step S1: discretize the search area into a grid, construct a target probability map (TPM) to represent the distribution probability of the target in the environment, and dynamically update the TPM using the Bayesian inference method;

[0012] Step S2: training the multi-UAV search strategy based on the improved MADDPG algorithm and making decentralized decisions during the mission execution phase;

[0013] In step S2, the improved MADDPG is to use the self-attention mechanism to optimize the observation information on the basis of MADDPG to obtain SA-MADDPG (Self-Attention MADDPG, a multi-agent deep deterministic policy gradient algorithm with self-attention);

[0014] In step S2, the UAV search area is divided based on the Voronoi diagram to reduce redundant searches, and the TPM is combined to guide the UAV to search the high-probability target area first;

[0015] In step S3, the UAV performs the search mission according to the optimized strategy and updates the environmental information in real time.

[0016] Specifically, in step S2:

[0017] Each UAV is regarded as an independent intelligent agent. Each agent makes independent decisions according to the different environments and shares collaborative mission target information with other agents.

[0018] Define the drone's observation space t , action space a t and search rewards t .

[0019] The observation space o t , input the policy network combined with the self-attention mechanism to obtain action a t

[0020] The observation space o t and action a t Input the evaluation function and obtain the evaluation score qi for the observation space and action.

[0021] Minimizing the variance is used as the loss function, and the parameters of the policy network and the evaluation network are updated according to the gradient descent method.

[0022] Assume that the observation space of the UAV is o t After three layers of convolutional network g i The feature map obtained is x=g i (o t ), the calculation method of the policy network combined with the self-attention mechanism includes:

[0023] Step 1. Let the input feature map x be of dimension C×H×W, where C represents the number of channels and H×W is the spatial dimension.

[0024] Step 2. Convert the feature map x into three feature spaces Q = W Q x, K = W K x and V = W V x; where W Q , W K , W V is a learnable weight matrix, Q, K, and V represent the query, key, and value vectors in the attention mechanism, respectively.

[0025] Step 3. Use the dot product calculation between the query Q and the key K representation, and then perform a soft max operation to obtain the attention map A:

[0026] A=soft max(Q T K)

[0027] Step 4. Apply the attention map A to the value V representation to calculate the attention weight O:

[0028] O=VA

[0029] Step 5. Reshape the self-attention feature map to match the input dimension:

[0030] c i =reshape(O, C, H, W).

[0031] Combining the Voronoi diagram to divide the drone search area and the TPM to guide the drone to prioritize the search for high-probability target areas is achieved by designing a dedicated reward function;

[0032] The total reward function r is calculated as:

[0033] r=r cov +r vor +r obs +r step ,

[0034] in,

[0035] Coverage Rewards cov The reward for the drone search is calculated based on the newly explored area. The formula is:

[0036] Here, D sensor，t represents the set of grid cells perceived by the UAV at time t, represents the complement of the coverage area at the previous time step.

[0037] Voronoi diagram reward r vor The reward for drone distribution is calculated by the maximum Voronoi diagram area difference between drones. The formula is:

[0038]

[0039] in, It is the Voronoi unit area of ​​the UAV at time t, calculated based on the UAV position; is the maximum Voronoi unit area at time t-1.

[0040] Obstacle penalty r obs The penalty for drone collision is calculated by the number of collisions. The formula is:

[0041]

[0042] Constant penalty r step It is a constant negative number with a value of -0.03.

[0043] Beneficial effects of the present invention:

[0044] (1) The present invention introduces the self-attention mechanism (i.e., SA-MADDPG) into the MADDPG algorithm, enabling the UAV to prioritize key environmental information (such as obstacles, teammate distribution, etc.), thereby improving the perception ability and collaborative decision-making ability, thereby improving the target search efficiency in complex environments.

[0045] (2) The present invention combines the target probability map TPM with the region partitioning strategy based on the Voronoi diagram to optimize the search path and coverage mode of the UAV, reduce redundant searches, and improve the response capability in dynamic target environments, making it suitable for search tasks of high-speed moving targets.

[0046] (3) SA-MADDPG is used to enable drone swarms to operate stably in complex environments with dense obstacles and unpredictable target motion. At the same time, the scalability of the algorithm is optimized to ensure its applicability to drone swarms of different sizes, thereby improving the system's deployability and application scope. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0048] Figure 1 This is a schematic diagram of the distributed decision-making and centralized training process of the SA-MADDPG algorithm of the present invention;

[0049] Figure 2 Schematic diagram of the strategy network and evaluation network integrated with the self-attention mechanism of the present invention;

[0050] Figure 3 This is a comparison chart of the training return curves of the SA-MADDPG algorithm that integrates the self-attention mechanism and the MADDPG and DQN algorithms. DETAILED DESCRIPTION

[0051] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0052] 1. Overview

[0053] The specific implementation steps of the present invention include:

[0054] First, the problem is modeled by transforming the drone target search task into a multi-agent collaboration problem. Each drone acts as an independent agent and uses MADDPG with self-attention mechanism (i.e. SA-MADDPG) to perform target search.

[0055] Next, the Voronoi diagram is used to rationally divide the search area, ensuring efficient collaborative search within the drone swarm within different areas. The target probability map (TPM) helps assess the likely distribution of targets, enabling drones to prioritize search areas with high probability, significantly improving search efficiency and accuracy.

[0056] During execution, the search strategy is adjusted based on real-time feedback to adapt to the dynamic changes of the target.

[0057] 2. Modeling of multi-UAV collaborative search problem:

[0058] First, the environment modeling, UAV modeling and target probability map updating of the UAV collaborative search problem are clarified.

[0059] The environment is represented by a three-channel binary grid that represents the discretized target area. Each grid cell represents a part of the environment. The value of each grid cell (x, y) consists of three parts: the probability of the target existing Obstacle presence mark and drone presence markers These values ​​meet the following constraints:

[0060] Let d represent the sensing range of each drone’s sensor.

[0061] D = {0, 1, 2, 3} represents the set of optional movement directions of the drone, which respectively represent moving one grid up, down, left, and right.

[0062] The target probability map is used to represent the target distribution, where each grid cell P xy (t)∈[0,1] represents the probability that the target appears in the (x,y) cell at time t. Initially, the target probability P of each grid cell is xy (0) = 0.5 means the target point is unknown. The probability update rule for target movement is: The Bayesian model is used to update the target probability map based on the detection information of the sensor. When the drone scans the target, the probability of the target appearing is updated as follows:

[0063] where p a Indicates the accuracy of the drone sensor. If the drone does not detect the target, it will a Replaced by 1-p a .

[0064] 3. Multi-UAV collaborative search method based on SA-MADDPG

[0065] The following describes how to use SA-MADDPG to solve the UAV collaborative search problem.

[0066] 1. First, define the observation space O of the drone at time t t , action space a t and search rewards t .

[0067] Drone observation space in, It is the latest update of TPM. represents the positions of all other drones, represents the distribution of obstacles in the environment, Represents the position of the drone at that moment, serving as a reference point within the location. Each of these components is represented as two-dimensional data, whose dimensions match the size of the environment.

[0068] The action space is represented by D = {0, 1, 2, 3} to represent the optional movement directions of the drone.

[0069] The search reward r is defined as the sum of the coverage reward, Voronoi map reward, obstacle penalty, and constant penalty:

[0070] r=r cov +r vor +r obs +r step

[0071] Among them, the coverage reward r cov The reward for the drone search is calculated by the newly explored area. The formula is: Among them D sensor，t represents the set of grid cells perceived by the UAV at time t, represents the complement of the coverage area at the previous time step.

[0072] Voronoi diagram reward r vor The reward for drone distribution is calculated by the maximum Voronoi diagram area difference between drones. The formula is: for i=1, 2, ..., N. where It is the Voronoi unit area of ​​the UAV at time t, calculated based on the UAV position; is the maximum Voronoi unit area at time t-1. ovs The penalty for drone collision is calculated by the number of collisions. The formula is: The constant penalty is a constant negative number with a value of -0.03.

[0073] 2. Then observe the space o t Input the policy network SA-MADDPG to get action a tIn the SA-MADDPG network structure, the data processing steps include:

[0074] a t =h i (g i (o t ), c i (o t ))

[0075] Among them, h i It is a two-layer fully connected network, g i It is a three-layer convolutional neural network, c i It is a weighting factor that measures the importance of these environmental features to the drone’s decision-making process.

[0076] c i is a feature extraction network based on self-attention, c i (o t )=f i (g i (o t ));f i It is a self-attention network.

[0077] Among them, c i The calculation steps are:

[0078] Step 1. Set the drone's observation space o t Enter g i The feature map obtained is x=g i (o t ), the dimension of the input feature map x is set to C×H×W, where C represents the number of channels and H×W is the spatial dimension.

[0079] Step 2. Convert the feature map x into three feature spaces Q = W Q x, K = W K x and V = W V xW Q , W K , W V is a learnable weight matrix.

[0080] Step 3. Use the dot product calculation between the query and the key representation, and then perform a soft max operation to obtain the attention map:

[0081] A=soft max(Q T K)

[0082] Step 4. Apply the attention map to the value representation to calculate the attention weights:

[0083] O=VA

[0084] Step 5. Reshape the self-attention feature map to match the input dimension:

[0085] c i =reshape(O, C, H, W)

[0086] 3. Then combine the observation space and action of all agents to obtain the joint observation space O t and joint action A t Input the evaluation function and get the evaluation score q for the joint observation space and joint action i ,

[0087] q i =h i (g i (O t ), c i (O t ), A t )

[0088] Figure 1 The diagram of the distributed decision-making and centralized training process of the SA-MADDPG algorithm is shown. Figure 2 A schematic diagram showing the policy network and evaluation network integrated with the self-attention mechanism.

[0089] Fourth, use minimization of variance as the loss function and update the parameters of the policy network and evaluation network according to the gradient descent method.

[0090] 4. Performance Analysis

[0091] In order to verify the effectiveness of the proposed multi-UAV target search method based on self-attention mechanism and reinforcement learning, we evaluated the algorithm performance by designing experiments to analyze the reward curves in the training of the algorithm and the benchmark algorithm. A two-dimensional UAV search environment was constructed based on a custom framework similar to OpenAIGym, and it was discretized into cells to represent the state. At the beginning of each simulation scene, the initial positions of the UAV and the moving target are randomly generated. To ensure the continuity of the search task, there are always a fixed number of targets in the environment: once a target is detected and removed, a new target is generated at a random position, and obstacles are randomly distributed throughout the environment. Set the number of UAVs to 4, the number of targets to 2, and the number of obstacles to 0-200, which are randomly selected at the beginning of each round. Set the maximum number of training rounds to 10,000 and the maximum time step for each round to 150. We trained the algorithm according to the above experimental settings and compared this method with the multi-agent deep deterministic policy gradient (MADDPG) and deep Q network (DQN) methods. The corresponding training reward curves are shown in the figure. Figure 3As shown in Figure 2, it can be seen that as the number of training iterations increases, the average reward values ​​of all algorithms increase and eventually converge. From the final convergence results, the average reward curve of the SA-MADDPG algorithm shows a clear advantage over the other two methods.

[0092] 5. Summary

[0093] The present invention relates to a method for multi-UAV collaborative dynamic target search, belonging to the field of artificial intelligence and UAV group intelligent control.

[0094] Aiming at the target search problem of UAVs in complex environments, this paper proposes a multi-UAV collaborative search strategy that combines the SA-MADDPG search method with the self-attention mechanism and the Voronoi region partitioning exploration incentive.

[0095] The present invention discretizes the search area into a grid map, constructs a target probability map (TPM) to represent the target distribution, and dynamically updates the TPM using Bayesian reasoning. Each drone perceives the environment based on its onboard sensors and uses a self-attention mechanism to optimize the observed information.

[0096] Search strategy optimization: The SA-MADDPG algorithm is used to train multi-UAV search strategies, using global information to optimize strategies during the centralized training phase and distributed decision-making during the execution phase.

[0097] Explore incentive mechanisms: Divide the drone search area based on the Voronoi diagram to reduce redundant searches, and combine TPM to guide drones to prioritize searching high-probability target areas;

[0098] Mission execution: The drone performs the search mission according to the trained strategy and updates the environmental information in real time.

[0099] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A multi-UAV target search method based on self-attention and reinforcement learning, characterized in that: The proposed method transforms the target search task of UAVs into a multi-agent collaborative problem, treating each UAV as an independent agent and using the SA-MADDPG strategy network to perform target search. Furthermore, the Voronoi diagram is used to rationally divide the search area, and the target probability map (TPM) is used to help evaluate the possible distribution of targets, so that UAVs prioritize searching areas with high target probability. Finally, during the search task, the search strategy is adjusted based on real-time feedback to adapt to the dynamic changes of the target. SA-MADDPG is a multi-agent deep deterministic policy gradient MADDPG algorithm that combines the self-attention mechanism. Each drone uses onboard sensors to observe the environment, and the self-attention mechanism is used to optimize the observed information. The target search method comprises the following steps: S1, discretize the search area into a grid map, construct a TPM to represent the distribution probability of the target in the environment, and dynamically update the TPM using the Bayesian inference method; S2, using the SA-MADDPG algorithm network structure as the strategy network to obtain the search strategy; S3, the UAV performs the search mission according to the search strategy and updates the environmental information in real time; In step S2, each drone, i.e., an intelligent agent, makes independent decisions based on its environment and shares target information of the collaborative search task with other intelligent agents. For the drone's observation space, action space, and search reward, the observation space is input into the policy network SA-MADDPG to obtain the action, and the observation space and action are input into the evaluation network to obtain the evaluation score for the observation space and action. The optimal search strategy is obtained based on the evaluation score. The parameters of the policy network and the evaluation network are iteratively updated in combination with the loss function. The observation space includes the latest update of the TPM, the positions of other drones, the distribution of obstacles in the environment, and the current position of the drone as a reference point within the position; the action space includes the directions in which the drone can move; the search reward is the sum of the coverage reward, the Voronoi map reward, the obstacle penalty, and the constant penalty; The coverage reward refers to the reward for drone search calculated by the newly explored area. The Voronoi map reward refers to the reward for drone distribution calculated by the difference in the maximum Voronoi map area divided by the drones. The obstacle penalty refers to the penalty for drone collision calculated by the number of collisions. The constant penalty is a constant negative number.

2. The multi-UAV target search method based on self-attention and reinforcement learning according to claim 1 is characterized in that: In step S1, the TPM is: The discretized task area is represented by a three-channel binary grid, where each grid cell represents a part of the environment, defined as follows: The value of any grid cell (x, y) includes: the probability of the target existing Obstacle presence mark and drone presence markers These values ​​satisfy the constraints: Let d represent the sensing range of each drone’s sensor; Let D = {0, 1, 2, 3} represent the set of movement directions of the drone, which respectively represent moving one grid up, down, left, and right; Let the probability of the target appearing in the grid cell (x, y) in TPM be P x，y (t), P x，y (t)∈[0,1] represents the probability that the target appears in the grid cell (x,y) at time t; Initially, the P of the grid cell (x,y) is x，y (0) = 0.5 means the target's position is unknown, i.e. the probability is 50%. Let the probability update rule of the target appearing be Among them, P x，y (t+1) represents the probability that the target appears in the grid unit (x, y) at time t+1; p m，n (t) represents the probability that the target appears in the grid cell (m,n) at time t; Use the Bayesian method to update the TPM based on the sensor detection information, specifically: When the drone scans a target, the probability of the target appearing is updated as follows: where p a Indicates the accuracy of the sensor. If the drone does not detect the target, it will a Replaced by 1-p a .

3. The multi-UAV target search method based on self-attention and reinforcement learning according to claim 2 is characterized in that: In step S2, the data processing method in the SA-MADDPG network structure is: the observation space o of UAV i is t Input its policy network to get action a t , the formula is a t =h i (g i (o t ), c i (o t )), where h i It is a two-layer fully connected network, g i It is a three-layer convolutional neural network, c i is a feature extraction network based on self-attention, c i (o t )=f i (g i (o t ))f i It is a self-attention network.

4. The multi-UAV target search method based on self-attention and reinforcement learning according to claim 3 is characterized in that Assume that the observation space of the UAV is o t Enter g i The feature map obtained is x=g i (o t ), which is fed into the self-attention network f i The subsequent calculation steps are: 1) Assume that the dimension of the input feature map x is C×H×W, where C represents the number of channels and H×W is the spatial dimension; 2) Convert the feature map x into three feature spaces Q = W Q x, K = W K x and V = W V x; where W Q , W K , W V is a learnable weight matrix, Q, K, and V represent the query, key, and value vectors in the attention mechanism, respectively; 3) Use the dot product calculation between the query Q and the key K representation, and then perform a soft max operation to obtain the attention map A: A=soft max(Q T K), 4) Apply the attention map A to the value V representation to calculate the attention weight O: O=VA, 5) Use the reshape function to reshape the self-attention feature map to match the input dimension: c i =reshape(O,C,H,W)。 5. The multi-UAV target search method based on self-attention and reinforcement learning according to claim 2 is characterized in that: In step S2, observe the space o t and action a t Input the evaluation network to obtain the evaluation score for the observation space and action, the formula is q i =h i (g i (o t ), c i (o t ), a t ).

6. The multi-UAV target search method based on self-attention and reinforcement learning according to claim 1 is characterized in that: In step S2, the variance minimization is used as the loss function, and the parameters of the policy network and the evaluation network are updated according to the gradient descent method.

7. The multi-UAV target search method based on self-attention and reinforcement learning according to claim 1 is characterized in that: In step S2, the search reward r is defined as the sum of the coverage reward, Voronoi map reward, obstacle penalty, and constant penalty: r＝r cov +r vor +r obs +r step ， Coverage Rewards cov The reward for the drone search is calculated by the newly explored area. The formula is: Among them D sensor，t represents the set of grid cells perceived by the UAV at time t, represents the complement of the coverage area at the previous time step. Voronoi diagram reward r vor The reward for drone distribution is calculated by the maximum Voronoi diagram area difference between drones. The formula is: for i=1, 2, ..., N. where It is the Voronoi unit area of ​​the drone at time t, calculated based on the position of drone i; is the maximum Voronoi unit area at time t-1; Obstacle penalty r obs The penalty for drone collision is calculated by the number of collisions. The formula is: Constant penalty r step It is a constant negative number with a value of -0.03.

Images