Highly adaptive unmanned aerial vehicle cluster collaborative target search method based on deep reinforcement learning

Abstract

The invention discloses a highly adaptive unmanned aerial vehicle cluster collaborative target search method based on deep reinforcement learning. According to the method, the problem of large-scale moving target search in a complex environment is solved through a revisit time compensation mechanism, wherein the problem is caused by target movement and the environmental cognitive information graph is unreliable. The method adopts a distributed partially observable Markov decision process to model layered collaborative search of an unmanned aerial vehicle group, and learns a joint strategy based on a deep reinforcement learning algorithm of a value decomposition network. Therefore, the unmanned aerial vehicle group can make full use of sensor performance characteristics at different heights, the risk of collision with adjacent unmanned aerial vehicles and obstacles is reduced, and the capability of cooperatively searching dynamic targets is improved. Simulation results show that compared with a baseline algorithm and an existing traditional algorithm, the algorithm has superiority and expandability.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle collaborative search, and in particular relates to multiple dynamic target search and highly hierarchical adaptive search of unmanned aerial vehicles. Background Art

[0002] The problem of multi-UAV collaborative search is a hot topic in the field of UAV research. UAV swarms can make up for the problems of target omission, loss, and long search time that may occur when a single UAV performs tasks in complex environments. However, in the face of the ever-expanding scope of applications and diverse mission requirements, intelligent collaborative search of UAV swarms faces many challenges: First, due to the curse of dimensionality, as the number of UAVs and targets increases, the problem solution space expands exponentially, and finding the optimal solution requires a lot of computing resources and time, which affects the real-time performance of the task; second, the actual search problem is complex and changeable, and the mission objectives are also varied. The dynamic changes of the target trajectory in the search environment and the existence of obstacles will further increase the complexity of the search problem. Therefore, how to achieve intelligent collaborative search of mobile multi-targets by UAV swarms in a dynamic environment is still a challenging problem.

[0003] According to the motion characteristics of the target, the target search problem can be divided into two categories: static target search and dynamic target search. For static target search, if the target position is known, it can be equivalent to a coverage search of the task area; if the target position is unknown or the target is movable, it is generally constructed as an optimization problem, and a search information graph describing environmental information is designed, such as a target probability graph, a pheromone graph, etc.; then heuristic algorithms, bionic cluster optimization algorithms, artificial potential field algorithms, graph theory and other methods are used to solve it. Although these algorithms have powerful non-convex optimization performance and robustness, they require environmental prior information in most cases, and have problems such as being prone to local optimization and low real-time performance.

[0004] In contrast, reinforcement learning, as a machine learning technology, allows agents to complete tasks through trial and error mechanisms. It has been widely used in various fields in the past few years. When reinforcement learning is applied to the collaborative search of multiple targets by drone clusters, drones are constructed as agents, and the problem is transformed into a multi-agent reinforcement learning (MADRL) problem. MADRL is more complex than the single-agent problem: each agent interacts with the environment while also interacting directly or indirectly with other agents. From the perspective of a single agent, the environment is non-stationary, which increases the complexity of training and evaluation. Centralized Training with Decentralized Execution (CTDE) is a classic paradigm for solving multi-agent reinforcement learning. CTDE can effectively use global information during training to achieve better and more stable training results, and can only use local information when inferring the policy model, which makes the MADRL algorithm have good scalability and is very suitable for multi-target search problems in complex environments. Summary of the invention

[0005] The purpose of the present invention is to propose a highly adaptive UAV cluster collaborative target search method based on deep reinforcement learning. The strategy model trained by this method can make full use of the performance changes of sensors at different heights, while reducing collisions with neighboring UAVs and obstacles, and improving the ability of UAV clusters to collaboratively search for dynamic targets. In order to achieve this purpose, the steps adopted by the present invention are:

[0006] Step 1: Construct a discretized search mission scenario. After the dynamic positions of the target and obstacles are updated, the drone uses detection updates, communication updates, and compensation to update the target probability map to represent the perception information of the mission environment.

[0007] Step 2: The problem of UAV swarm collaborative search for moving targets is constructed as a distributed partially observable Markov decision process, and the action space and observation space are designed based on a highly hierarchical design;

[0008] Step 3: Adopt the strategy of centralized training and distributed execution. During the training phase, a deep reinforcement learning algorithm based on value decomposition network is used to learn the joint strategy of maximizing the cumulative discounted reward.

[0009] Step 4: In the execution phase, each drone makes independent decisions based on the target network obtained in the training phase to achieve distributed search. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1It is a training flow chart of a highly adaptive UAV cluster collaborative target search method based on deep reinforcement learning of the present invention;

[0011] Figure 2 is a schematic diagram of the search environment of the present invention;

[0012] Figure 3 It is a schematic diagram of the system framework of the present invention. DETAILED DESCRIPTION

[0013] The method assumes that multiple homogeneous drones search for mobile targets in the mission area, and use airborne sensors to look down to perceive environmental information; the drones have sufficient memory and processing power to share information with neighboring drones within the communication range; after the drones search for the mobile target, the drones move along with the movement of the target until the search mission is completed. The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0014] Step 1: Initialize the mission environment information, update the positions of the target and obstacles, and the drone updates the environmental cognition information based on the current position coordinates and sensor parameters:

[0015] (1) Mission scenario Uniformly discretized into cells G x，y , there exists N u homogeneous drones with coordinates Each drone has two actions at each time step. There are four candidate actions in the horizontal direction, namely forward, backward, left, and right, and three candidate actions in the vertical direction, namely rise, lower, and hold. The detection probability P of the airborne sensor d , false alarm probability P f , Detection radius R i，z Varies with the z height of the drone. t targets and N s obstacles, the motion of the two conforms to the random motion model, and the positions are updated;

[0016] (2) The drone uses the target probability map to represent the perception information of the mission environment. First, the detection information is updated. The target probability map of drone i at time step t is expressed as:

[0017]

[0018] in, Represents the grid G x，y The probability of the target existing at time t. Use the Bayesian criterion to update the target probability value in the detected grid:

[0019]

[0020] To reduce the complexity, a linear update method is used. Updated as follows:

[0021]

[0022]

[0023] (3) The UAV performs communication information fusion and update. The neighbor node set of UAV i at time t is expressed as d c is the communication distance. Each drone records the number of positive detections and negative detections for each grid. After the drone detects the mission area at each time step, it updates the positive detection number graph. and negative detection times And use it as neighbor communication interaction information. The information fusion of drone i at time t is expressed as:

[0024]

[0025] Grid G x，y The deformation target probability map is updated as:

[0026]

[0027] (4) Update of revisit time compensation mechanism. When the time difference between the last detection of the grid and the present is greater than the revisit time threshold T 0 When , the number of positive detections of clearing drones and negative detection times To deal with the unreliable detection information caused by target movement, a compensation mechanism is used to update the probability of target existence in the undetected grid:

[0028]

[0029] Step 2: The UAV swarm collaborative search for moving targets problem is constructed as a distributed partially observable Markov decision process, represented by a tuple (N, S, A, T, R, O, Z, γ):

[0030] (1) N represents the set of agents, S represents the global state space, T is the transition probability between states, represented by P(s′|s, a)→[0, 1], and Z: (i) = Z(s, i) is the observation function of each UAV in a specific state, γ∈[0, 1] represents the discount factor;

[0031] (2) A: (1) , A (2) , ..., A (n)} represents the action space of all agents, where a (i) ∈A (i)Represents the action of the i-th agent. The action space of the agent has two dimensions: horizontal and vertical. The multi-dimensional discrete action space is converted into a one-dimensional discrete action space by permutation and combination.

[0032] (3) R(s, a): {r (1) , r (2) , ..., r (n)} is the joint reward obtained after performing a joint action in a certain state s. In order to guide the drone swarm to efficiently search for unknown targets and avoid overlapping perception and collision, the real-time reward obtained after the drone interacts with the environment is designed to include six parts: target discovery reward, energy consumption penalty, collision penalty, uncertainty reduction reward, revisit pheromone reward, and coverage reward;

[0033] (4)o:{O (1) , O (2) , ..., O (n)} represents the joint observation space of all agents. At each time step, the local observation information of drone i is reduced to a space centered on the current unit of the drone with a side length of L 0 The square observation area is extracted to extract the local cognitive information of the observation area as the input of the policy network:

[0034] o t =(p i , ph i , f t ,pos t ) (8)

[0035] Among them, p t is the extracted target probability distribution information, ph t is the extracted digital pheromone graph information, f t To search for time graph information, pos i Indicates neighbor location information and obstacle information.

[0036] Step 3: Adopt the strategy of centralized training and distributed execution. During the training phase, a deep reinforcement learning algorithm based on value decomposition network is used to learn the joint strategy of maximizing the cumulative discounted reward:

[0037] (1) Set the maximum number of training rounds E, and the maximum time step that the drone can move in each round is S. Use the random parameter θ (1) , ..., θ (n) Initialize the training network of each agent using Initialize the corresponding target network and the shared experience replay pool D;

[0038] (2) In each training round, collect observations from all drones For each agent i, we randomly select an action with probability ∈ Otherwise select:

[0039]

[0040] (3) Agent i performs actions and interacts with the environment to obtain the observation at the next moment and reward r i Using historical local observation information h i =a 1 o 1 r 1 , ..., a i-1 o i-1 r i-1 Replace global information;

[0041] (5) i , a t , r i ,h t+1 ) is stored in the experience replay pool D;

[0042] (6) The local action value function of agent i performing a joint action is expressed as:

[0043]

[0044] (6) Using the target network to calculate:

[0045]

[0046] (8) Calculate the mean square error loss function:

[0047]

[0048] (9) By minimizing the loss function Update the parameter training network parameters θ, and update the target network parameters of each agent through the reverse transfer of the gradient at every set time interval C

[0049] Step 4: In the execution phase, each UAV makes independent decisions based on the target network obtained in the training phase to implement distributed collaborative search tasks. When all targets are found or the set maximum working time is reached, the current training round ends.

Claims

1. A highly adaptive UAV cluster collaborative target search method based on deep reinforcement learning, characterized in that The UAV updates the information of the mission environment for the mobile target perception, based on the highly hierarchical action space and observation space design of the UAV, centralized training and distributed execution. The steps adopted are: Step 1: In the discretized mission scenario, the moving targets and obstacles are updated in position. Each drone uses a target probability map to represent the perceived information of the mission environment. The height of the drone is discretized in layers, and the coordinates are expressed as Each drone has two actions at each time step. There are four candidate actions in the horizontal direction, namely forward, backward, left, and right, and three candidate actions in the vertical direction, namely raising, lowering, and maintaining. The detection probability P of the sensor on the drone is d , false alarm probability P f , Detection radius R i，z As the z height of the drone changes, the target probability map of drone i at time step t is expressed as: in, Represents the grid G x，y The probability of the target existing at time t. The update of the target probability map is divided into detection update, communication update and compensation update. First, UAV i is based on its own position And sensor performance detection update, use the Bayesian criterion to update the target probability value in the detected grid, and use the linear update method to reduce the complexity. Updated as follows: The communication distance of the drone is d c , the neighbor node set of drone i at time t is expressed as Each drone records the number of positive and negative detections for each grid. After the drone detects the mission area at each time step, it updates the positive detection count map. and negative detection times And take it as the neighbor communication interaction information, the information fusion of UAV i at time t is expressed as: Grid G x，y The deformation target probability map is updated to: When the time difference between the last detection of the grid and the current detection time is greater than the revisit time threshold T0, the positive detection times of the drone are cleared. and negative detection times To address the problem of unreliable detection information caused by target movement, a compensation mechanism is used to update the probability of target existence in undetected grids: where p Δ is the probability increment unit, t x，y The grid G x,y The last detection time of Step 2: The problem of UAV swarm collaborative search for moving targets is constructed as a distributed partially observable Markov decision process, represented by a tuple (N, S, A, T, R, O), where N represents the set of agents, S represents the global state space, and A: {A (1) , A (2) , ..., A (n) } represents the action space of all agents, where a (i) ∈A (i) represents the action of the ith agent. The action space of the agent has two dimensions, horizontal and vertical. The multidimensional discrete action space is converted into a one-dimensional discrete action space by permutation and combination. P(s′|s, a)→[0, 1] represents the transition probability of transferring to another global state after executing a joint action in the current global state. R(s, a): {r (1) , r (2) , ..., r (n) } is the joint reward obtained after performing a joint action in a certain state s. In order to guide the drone swarm to efficiently search for unknown targets and avoid overlapping perception and collision, the real-time reward obtained after the drone interacts with the environment is designed to include six parts: target discovery reward, energy consumption penalty, collision penalty, uncertainty reduction reward, revisit pheromone reward, and coverage reward. (1) , O (2) , ..., O (n) } represents the joint observation space of all agents. At each time step, the local observation information of the drone is reduced to a square observation area with a side length of L0 centered on the current unit of the drone. The local cognitive information of the observation area is extracted as the input of the training network: O i，t =(p i,t， ph i,t ,f i，t ,pos i，t ) (7) Among them, p i，t is the extracted target probability distribution information, ph i，t is the extracted digital pheromone graph information, f i，t To search for time graph information, pos i,t Indicates neighbor location information and obstacle information; Step 3: Adopt the strategy of centralized training and distributed execution. In the training phase, a deep reinforcement learning algorithm based on value decomposition network is used to learn the joint strategy of maximizing the cumulative discounted reward to complete the task of cooperative search for moving targets by drone clusters. Each agent makes its own behavior decision based on its local observation information. The local action value function of agent i is expressed as: Centralized training requires access to the global action-value function Since the environment is partially observable and the global state information is unknown, the idea of ​​value decomposition is adopted and the historical local observation information h is used. t =a1o1r1,...,a t-1 o t-1 r t-1 Replace global information t , the specific decomposition formula is as follows: Among them, θ (i) Represents the deep neural network parameters of agent i, which is approximately obtained Then, use the time difference method to perform incremental updates: The loss function is constructed in the form of mean square error: By minimizing the loss function Update the parameters θ, and after a set time interval C, update the target network parameters of each agent through the reverse transfer of the gradient Step 4: In the execution phase, each drone makes independent decisions based on the target network obtained in the training phase to achieve distributed collaborative search. The current round ends when all targets are found or the set maximum working time is reached.

Images