Unmanned aerial vehicle cluster collaborative search method and system based on deep multi-agent reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle cluster collaborative search method and system based on deep multi-agent reinforcement learning. The method comprises the following steps: defining a tracker unmanned aerial vehicle, a reconnaissance unmanned aerial vehicle and an intelligent target unmanned aerial vehicle; initializing an actor network, a commentator network and an experience playback buffer area for each unmanned aerial vehicle agent; each unmanned aerial vehicle agent selects an action and executes the action through an actor network according to the current state, and after the environment returns a new state and a corresponding reward, the interaction experience is stored in an experience playback buffer area; randomly extracting a batch of experience samples from the experience playback buffer area, and updating a commentator network; calculating gradients using the commentator network, and updating the actor network of each agent according to the gradients; and continuously performing interaction, data collection, network training and target network updating of the intelligent agent until a certain training period or performance standard is reached. According to the invention, the efficiency and safety of the unmanned aerial vehicle cluster for executing the collaborative search task in a complex environment can be improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control technology of drone clusters, and in particular to a drone cluster collaborative search method and system based on deep multi-agent reinforcement learning. Background Art

[0002] In the UAV swarm collaborative search mission, how to effectively coordinate the behavior of each UAV to achieve rapid response and accurate tracking of the target while ensuring the ability to explore the unknown environment is a very challenging problem. Traditional centralized control methods, due to their inherent computing and communication bottlenecks, are difficult to adapt to complex and changing environments and the real-time control requirements of large-scale UAV swarms. Although model-based distributed control methods have alleviated these problems to a certain extent, they are often limited by the accuracy and generalization ability of the model.

[0003] In recent years, breakthroughs in deep learning technology have provided a new way to solve the above problems. With its powerful feature extraction and pattern recognition capabilities, deep neural networks have achieved remarkable results in visual tasks such as image classification, target detection, and semantic segmentation. However, in actual drone cluster control applications, there are a series of challenges such as insufficient data samples, dynamic environmental changes, and limited computing resources. In particular, for specific tasks such as height-restricted scenes, traditional deep learning methods are difficult to achieve ideal performance due to the sparsity and diversity of samples.

[0004] As an emerging research direction, multi-agent reinforcement learning provides a new approach to solving the problem of coordinated control of drone swarms by introducing competition and cooperation mechanisms among multiple agents. In the framework of multi-agent reinforcement learning, each drone can be regarded as an agent, which learns the optimal strategy through interaction with the environment to achieve coordination between individuals and completion of tasks.

[0005] For example, Chinese patent document with publication number CN113110582A discloses a control method for an intelligent system of a drone swarm, and Chinese patent document with publication number CN115755949A discloses an autonomous aggregation method for a multi-drone formation cluster based on multi-agent deep reinforcement learning.

[0006] However, most existing multi-agent reinforcement learning algorithms focus on the coordination of strategies, while ignoring the interpretability and generalization ability of strategies, which may lead to unpredictable behaviors and risks in actual drone swarm control. Summary of the invention

[0007] The present invention provides a method and system for collaborative search of drone clusters based on deep multi-agent reinforcement learning. Through the organic combination of deep learning and reinforcement learning algorithms, intelligent control of drone clusters is realized. It is particularly suitable for multi-target tracking and exploration tasks, and can improve the efficiency and safety of drone clusters in performing collaborative search tasks in complex environments.

[0008] A drone swarm collaborative search method based on deep multi-agent reinforcement learning includes the following steps:

[0009] (1) Constructing UAV intelligent agents to form a cluster, including a tracker UAV for tracking and identifying specific targets, a scout UAV for exploring unknown areas in the environment, and an intelligent target UAV for simulating actual tracking targets;

[0010] (2) Based on the deep multi-agent reinforcement learning algorithm, the actor network and critic network are initialized for each drone agent in the cluster, and the experience replay buffer is initialized to store the interaction experience of the drone agents in the cluster;

[0011] (3) Each drone agent in the cluster selects and executes an action based on the current state through its actor network. When the environment returns a new state and corresponding reward, the interaction experience is stored in the experience playback buffer; the interaction experience includes the current state, action, reward, and new state.

[0012] (4) Randomly draw a batch of experience samples from the experience replay buffer, use these samples and the critic network to calculate the temporal difference error, and update the critic network accordingly; where the temporal difference error is the difference between the actual return and the return predicted by the critic network;

[0013] (5) Use the critic network to calculate gradients, which represent the expected returns of the drone agent's actions under the current policy, and update the actor network of each drone agent based on these gradients;

[0014] (6) The interaction, data collection, network training, and target network updating of the UAV agents are continuously performed. After reaching the set training cycle or performance standard, the images captured by the UAV cluster are input, and the optimal action output of each UAV agent in the cluster is calculated to achieve collaborative search for the target.

[0015] In step (1), the duties and functions of the tracker drone, scout drone, and smart target drone are to track and identify specific targets, explore unknown areas in the environment, and simulate actual tracking targets.

[0016] In step (2), a deep neural network architecture is designed, including an input layer for receiving states and actions, multiple fully connected layers, and an output layer for outputting values ​​(critic network) or actions (actor network); an experience replay buffer is implemented to store the experience sequence obtained by each drone agent in the cluster interacting with the environment; a copy of the target network is created to stabilize the training process.

[0017] In step (2), the deep multi-agent reinforcement learning algorithm adopts an improved multi-agent deep deterministic policy gradient algorithm. Each drone agent in the cluster has its own actor network and critic network, but the critic network is centralized and can access the observations and actions of all agents, including a centralized critic mechanism and a decentralized actor mechanism.

[0018] The centralized critic mechanism specifically processes the observations and actions of all agents, that is, uses the observations and actions of all drone agents in the cluster to update the critic network. The centralized critic mechanism can take into account the information of all agents, which can help agents learn how to work together in a team to achieve common goals.

[0019] The decentralized actor mechanism allows each agent to learn independently, that is, the actor network is updated for each agent according to its corresponding critic network output. The decentralized actor mechanism enables each drone agent in the cluster to flexibly adapt to changes in the environment because each agent can quickly respond to new observations.

[0020] The deep multi-agent reinforcement learning algorithm also includes the following three mechanisms:

[0021] Reward strategy mechanism based on Voronoi diagram: The geometric property of Voronoi diagram is to divide the plane into different regions based on a set of specific points, and each region contains only a unique specific point, and the distance from any position in the region to the specific point is closer than to other specific points. For each scout drone agent in the environment, its goal is to minimize the area of ​​its Voronoi diagram region, so the agents need to spread out to cover a larger space;

[0022] Collision avoidance mechanism: Use sensors (such as radar and lidar) to detect the position of neighboring drones and predict the flight trajectory of each drone agent in the cluster. When the agents are too close, they receive negative rewards to prevent collisions between drones.

[0023] Boundary constraint mechanism: clearly define the flight boundary of the agent and continuously detect its position; when the agent's position exceeds this boundary, it receives a negative reward; thereby encouraging the agent to stay within the task-related area and prevent it from getting lost in the environment or entering dangerous areas.

[0024] In steps (4) and (5), a centralized critic network is designed, which has access to the observations and actions of all agents. Each drone agent in the cluster has its own actor network and makes decisions based on its own observations of the local environment. The evaluation of the critic network is used to coordinate the strategies of different drone agents in the cluster to achieve overall optimization.

[0025] In step (6), the interaction of the UAV agent, data collection, network training and target network update are continued. The process is as follows:

[0026] (6-1) Initialize an actor network and a critic network for each drone agent in the cluster; the actor network outputs the optimal action in a given state, while the critic network evaluates the value of the action under the current strategy;

[0027] (6-2) Each drone agent in the cluster performs actions in the environment and collects data on interactions with the environment, including state, action, reward, and new state; this data is stored in the experience replay buffer for use during training;

[0028] (6-3) Update critic network;

[0029] (6-4) Update the actor network;

[0030] (6-5) Update the target network;

[0031] (6-6) Asynchronous update: In practical applications, agents can update their network parameters asynchronously, allowing them to learn independently at their own time steps;

[0032] (6-7) Exploration mechanism: The ε-greedy strategy is integrated into the actor network to ensure that the agent can explore sufficient environmental information in the early stages.

[0033] In step (6-3), the critic network is updated. The specific process is:

[0034] A batch of experience samples are randomly drawn from the experience replay buffer and used to compute the temporal difference error loss function of the critic network:

[0035] L critic =E (s，a，r，s') [r+γQ′ i (s′,μ′ i (s′))-Q i (s, a)] 2

[0036] Where E represents the mathematical expectation, s is the current state, a is the selected action, r is the immediate reward, s′ is the next state, γ is the discount factor used to measure the current value of future rewards, i represents the drone number, and Q′ i (s′,μ′ i (s′)) is the value estimate of the next state, μ′ i (s′) is the value function parameter, Q i (s,a) is the value estimate of the current state;

[0037] Update the parameters of the critic network using gradient descent:

[0038]

[0039] Where ω is the parameter of the critic network, α ω is the learning rate of the critic network, is the gradient of the loss function L(ω) with respect to the parameter ω.

[0040] In step (6-4), the actor network is updated. The specific process is:

[0041] Use the critic network to compute the gradients of the actor network, which represent the expected returns of the agent's actions under the current policy:

[0042]

[0043] Among them, i represents the drone serial number, θ i is the actor network parameter of UAV i, J(θ i ) is the expected return of drone i, Q i (s,a i ) is a pair of state-action pairs (s,a i ), is the loss function μ i (s) about parameter θ i The gradient of

[0044] Update the parameters of the actor network using gradient ascent:

[0045]

[0046] in, is the learning rate of the actor network, θ i are the parameters of the actor network, is the loss function J with respect to the parameter θ i gradient.

[0047] In step (6-5), the target network is updated. The specific process is as follows:

[0048] In order to stabilize the training process, the parameters of the target network are gradually approached to the parameters of the current network through soft updates:

[0049] ω'←τωω+(1-τ)ω'θ′ i ←τθ i +(1-τ)θ′ i

[0050] Where ω is the parameter of the critic network, θ i is the parameter of the actor network, τ is the soft update parameter, ranging from 0.0001 to 0.001.

[0051] A drone cluster collaborative search system based on deep multi-agent reinforcement learning includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned drone cluster collaborative search method.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. This invention significantly improves the tracking capability of drone swarms for moving targets, especially in complex and dynamic environments, through a deep multi-agent reinforcement learning algorithm.

[0054] 2. The present invention realizes effective coordination among the tracker UAV, the scout UAV and the intelligent target UAV, and optimizes the resource allocation and role allocation in the multi-agent system.

[0055] 3. Through the introduction of experience replay and target network, the present invention reduces the dependence on real-time data and improves the stability and efficiency of the learning process.

[0056] 4. The drone swarm can adapt to environmental changes and flexibly adjust strategies to cope with different tracking and exploration tasks, improving the robustness of the system.

[0057] 5. The method and system of the present invention are designed with scalability in mind and can be easily integrated into existing drone swarms or multi-agent systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of a drone cluster collaborative search method based on deep multi-agent reinforcement learning according to an embodiment of the present invention.

[0059] Figure 2 It is the training framework of the UAV agent in the embodiment of the present invention. DETAILED DESCRIPTION

[0060] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0061] like Figure 1 As shown, a drone cluster collaborative search method based on deep multi-agent reinforcement learning includes the following steps:

[0062] S01, defines three roles in the system: tracker drone, scout drone and smart target drone. Each drone is equipped with corresponding sensors and computing modules according to its function.

[0063] In the embodiment of the present invention, the tracker UAV is designed to track and identify UAVs of specific targets, and is equipped with high-speed processors and advanced sensors, such as cameras, infrared sensors, etc., to achieve precise tracking; the scout UAV is responsible for exploring unknown areas, collecting environmental information, and providing possible target locations for the tracker, and usually has a longer flight time and a larger detection range; the smart target UAV simulates enemy targets, has the ability to avoid tracking, and is equipped with obstacle avoidance technology and possible countermeasures to simulate avoidance behavior in complex environments.

[0064] S02, designs a deep neural network architecture for each agent, defines the state space of each agent, implements an experience replay buffer to store the agent's interaction experience, and creates a target network for each agent, which is a copy of its actor network to stabilize the training process.

[0065] In the embodiment of the present invention, the deep neural network architecture designed for each agent specifically includes the following 5-layer structure:

[0066] A. Input layer: It first receives data from the drone’s sensors, including position, velocity, acceleration, gyroscope and compass readings, etc. These data are fused to form global and local representations of the environment.

[0067] B. Convolutional layers: Multiple convolutional layers are used to extract spatial features from images or sensor data. Each convolutional layer is followed by an activation function ReLU to introduce nonlinearity and enhance the expressiveness of the network.

[0068] C. Pooling layer: After the convolutional layer, the pooling layer uses max pooling to reduce the dimensionality of the feature map while retaining the most important features.

[0069] D. Hidden layer: The fully connected layer further combines and processes the features extracted by the previous layer. There can be multiple hidden layers, and each layer increases the nonlinear expression ability of the network through activation functions.

[0070] E. Output layer: The last fully connected layer maps the feature vector to the action space. For a continuous action space, the output of this layer can be a parameterized policy, such as the mean and variance of a normal distribution, which is used to decide the next action of the drone.

[0071] In the embodiment of the present invention, the state information of the intelligent entities corresponding to the tracker drone, the scout drone and the intelligent target drone includes the following aspects:

[0072] A. Own position and speed: Where (x, y, z) represents the position coordinates of the drone in three-dimensional space. Represents the velocity vector of the drone.

[0073] B. Environmental information: (V, E), where V is the set of obstacles in the environment and E is the environmental boundary information.

[0074] C. Other drone locations: (x i ,y i ,z i |i∈teammates), which represents the positions of other UAVs of the same type in the same mission.

[0075] D. Task-related parameters: (T, R), where T represents the task type and R represents the task-related rules and restrictions.

[0076] E. The state space of the pursuer drone should also have the position and velocity of the target drone: Indicates the position and speed of the target drone that the chaser drone is tracking.

[0077] F. The state space of the scout drone should also contain information about unexplored areas: x ,U y ,U z ), indicating unexplored areas or locations of potential targets.

[0078] In the embodiment of the present invention, a deep neural network is used to approximate the strategy function π and the value function Q. For each agent i, the actor network μ i and value network Q i is defined as:

[0079] μ i (s i ) = Neural network parameterization strategy

[0080] Q i (s i ,a i ) = Neural network parameterized value function

[0081] In the embodiment of the present invention, an experience playback mechanism is designed and implemented to store the interactive experience of the intelligent agent. t (s t ,a t ,r t ,s t+1 ).

[0082] In the embodiment of the present invention, the target network μ is introduced. i ′ and Q i ′, to stabilize the training process:

[0083] μ i ′←τμ i +(1-τ)μ i '

[0084] Q i ′←τQ i +(1-τ)Q i '

[0085] Among them, μ i and Q i are the original actor network and the critic network, and τ is the soft update parameter.

[0086] S03, each drone agent in the cluster selects and executes actions through its actor network according to the current state. When the environment returns a new state and corresponding reward, the interaction experience is stored in the experience playback buffer; the interaction experience includes the current state, action, reward, and new state.

[0087] S04, use the observations and actions of all agents in the experience replay buffer to update the critic network, and update the actor network for each agent according to its corresponding critic network output, the formula is as follows:

[0088] L critic =E( s，a，r，s′ )[r+yQ′ i (s', μ' i (s'))-Q i (s, a)] 2

[0089] Where E represents the mathematical expectation, s is the current state, a is the selected action, r is the immediate reward, s′ is the next state, γ is the discount factor used to measure the current value of future rewards, i represents the drone number, and Q′ i (s′,μ′ i (s′)) is the value estimate of the next state, μ′ i (s′) is the value function parameter, Q i (s,a) is the value estimate of the current state.

[0090]

[0091] Among them, θ i is the actor network parameter of UAV i, J(θ i ) is the expected return of drone i, Q i (s,a i ) is a pair of state-action pairs (s,a i ), is the loss function μ i (s) about parameter θ i gradient.

[0092] S05, use a specific loss function to train the model and optimize the network parameters. The specific method is as follows:

[0093] Update the parameters of the critic network using gradient descent:

[0094]

[0095] Where ω is the parameter of the critic network, α ω is the learning rate of the critic network, is the gradient of the loss function L(ω) with respect to the parameter ω.

[0096] Update the parameters of the actor network using gradient ascent:

[0097]

[0098] in, is the learning rate of the actor network, θ i are the parameters of the actor network, is the loss function J with respect to the parameter θ i gradient.

[0099] S06, continues the interaction of UAV agents, data collection, network training and target network update, and tests the effectiveness of the algorithm in a simulation environment, evaluates system performance, including tracking accuracy, exploration efficiency, etc., and optimizes algorithm parameters based on feedback.

[0100] like Figure 2 As shown in Figure 1, the training framework is that actor network i receives observation i, makes action i, feeds back to critic network i, and stores this set of information in the experience replay buffer, combining these actions with observations to further update the critic network.

[0101] The present invention realizes effective control of drone swarms by introducing advanced deep learning technology and multi-agent reinforcement learning algorithm. The core of the system includes an improved multi-agent deep deterministic policy gradient algorithm, which realizes the coordinated optimization of the individual behavior of each drone in the drone swarm through a centralized critic mechanism and a decentralized actor mechanism. At the same time, the system also integrates a series of key technologies such as reward strategy based on Voronoi diagram, collision avoidance mechanism, boundary constraint mechanism, etc., to enhance the environmental adaptability and task execution capability of drone swarms.

[0102] The embodiments of the present invention have been tested in both simulated environments and real-world scenarios, demonstrating their effectiveness and robustness in the task of cooperative target search for drone swarms. By combining deep neural networks with multi-agent reinforcement learning technology, the present invention not only improves the collaborative working ability of drone swarms, but also enhances their adaptability to complex environments, and has important practical value and broad application prospects.

[0103] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A drone swarm collaborative search method based on deep multi-agent reinforcement learning, characterized in that: The following steps are involved: (1) Constructing UAV intelligent agents to form a cluster, including a tracker UAV for tracking and identifying specific targets, a scout UAV for exploring unknown areas in the environment, and an intelligent target UAV for simulating actual tracking targets; (2) Based on the deep multi-agent reinforcement learning algorithm, the actor network and critic network are initialized for each drone agent in the cluster, and the experience replay buffer is initialized to store the interaction experience of the drone agents in the cluster; (3) Each drone agent in the cluster selects and executes an action based on the current state through its actor network. When the environment returns a new state and corresponding reward, the interaction experience is stored in the experience playback buffer; the interaction experience includes the current state, action, reward, and new state. (4) Randomly draw a batch of experience samples from the experience replay buffer, use these samples and the critic network to calculate the temporal difference error, and update the critic network accordingly; where the temporal difference error is the difference between the actual return and the return predicted by the critic network; (5) Use the critic network to calculate gradients, which represent the expected returns of the drone agent's actions under the current policy, and update the actor network of each drone agent based on these gradients; (6) The interaction, data collection, network training, and target network updating of the UAV agents are continuously performed. After reaching the set training cycle or performance standard, the images captured by the UAV cluster are input, and the optimal action output of each UAV agent in the cluster is calculated to achieve collaborative search for the target.

2. The drone cluster collaborative search method based on deep multi-agent reinforcement learning according to claim 1 is characterized in that: In step (2), the deep multi-agent reinforcement learning algorithm adopts an improved multi-agent deep deterministic policy gradient algorithm. Each drone agent in the cluster has its own actor network and critic network, but the critic network is centralized and can access the observations and actions of all agents, including a centralized critic mechanism and a decentralized actor mechanism.

3. The drone cluster collaborative search method based on deep multi-agent reinforcement learning according to claim 2 is characterized in that: The centralized critic mechanism specifically processes the observations and actions of all agents, i.e., uses the observations and actions of all drone agents in the cluster to update the critic network; The decentralized actor mechanism specifically allows each agent to learn independently, that is, the actor network is updated for each agent according to its corresponding critic network output.

4. The drone cluster collaborative search method based on deep multi-agent reinforcement learning according to claim 1 is characterized in that: The deep multi-agent reinforcement learning algorithm also includes the following three mechanisms: Voronoi diagram-based reward strategy mechanism: For each scout drone agent in the environment, its goal is to minimize the area of ​​its Voronoi diagram region, so the agents need to spread out to cover a larger space; Collision avoidance mechanism: Use sensors to detect the location of neighboring drones and predict the flight trajectory of each drone agent in the cluster. When the agents are too close, they receive negative rewards to prevent collisions between drones. Boundary constraint mechanism: clearly define the flight boundary of the agent and continuously detect its position; when the agent's position exceeds this boundary, it receives a negative reward; thereby encouraging the agent to stay within the task-related area and prevent it from getting lost in the environment or entering dangerous areas.

5. The drone cluster collaborative search method based on deep multi-agent reinforcement learning according to claim 1 is characterized in that: Continue to interact with the drone agent, collect data, train the network, and update the target network. The process is as follows: (6-1) Initialize an actor network and a critic network for each drone agent in the cluster; the actor network outputs the optimal action in a given state, while the critic network evaluates the value of the action under the current strategy; (6-2) Each drone agent in the cluster performs actions in the environment and collects data on interactions with the environment, including state, action, reward, and new state; this data is stored in the experience replay buffer for use during training; (6-3) Update critic network; (6-4) Update the actor network; (6-5) Update the target network; (6-6) Asynchronous update: In practical applications, agents can update their network parameters asynchronously, allowing them to learn independently at their own time steps; (6-7) Exploration mechanism: The ε-greedy strategy is integrated into the actor network to ensure that the agent can explore sufficient environmental information in the early stages.

6. The drone cluster collaborative search method based on deep multi-agent reinforcement learning according to claim 5 is characterized in that: In step (6-3), the critic network is updated. The specific process is: A batch of experience samples are randomly drawn from the experience replay buffer and used to compute the temporal difference error loss function of the critic network: L critic =E (s,a,r,s′) [r+γQ′ i (s′,μ′ i (s′))-Q i (s,a)] 2 Where E represents the mathematical expectation, s is the current state, a is the selected action, r is the immediate reward, s′ is the next state, γ is the discount factor used to measure the current value of future rewards, i represents the drone number, and Q′ i (s′,μ′ i (s′)) is the value estimate of the next state, μ′ i (s′) is the value function parameter, Q i (s,a) is the value estimate of the current state; Update the parameters of the critic network using gradient descent: Where ω is the parameter of the critic network, α ω is the learning rate of the critic network, is the gradient of the loss function L(ω) with respect to the parameter ω.

7. The drone cluster collaborative search method based on deep multi-agent reinforcement learning according to claim 5 is characterized in that: In step (6-4), the actor network is updated. The specific process is: Use the critic network to compute the gradients of the actor network, which represent the expected returns of the agent's actions under the current policy: Among them, i represents the drone serial number, θ i is the actor network parameter of UAV i, J(θ i ) is the expected return of drone i, Q i (s,a i ) is a pair of state-action pairs (s,a i ), is the loss function μ i (s) about parameter θ i The gradient of Update the parameters of the actor network using gradient ascent: in, is the learning rate of the actor network, θ i are the parameters of the actor network, is the loss function J with respect to the parameter θ i gradient.

8. The drone cluster collaborative search method based on deep multi-agent reinforcement learning according to claim 5 is characterized in that: In step (6-5), the target network is updated. The specific process is as follows: In order to stabilize the training process, the parameters of the target network are gradually approached to the parameters of the current network through soft updates: ω′←τω+(1-τ)ω′θ′ i ←tth i +(1-τ)θ′ i Where ω is the parameter of the critic network, θ i is the parameter of the actor network, τ is the soft update parameter, ranging from 0.0001 to 0.

001.

9. A drone swarm collaborative search system based on deep multi-agent reinforcement learning, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the drone cluster collaborative search method described in any one of claims 1-8.

Images