A Method for Cooperative Encirclement Control of Firefighting by Unmanned Aerial Vehicle Groups Based on MADDPG Algorithm

Abstract

This invention provides a method for collaborative encirclement control of unmanned aerial vehicles (UAVs) in fire suppression based on the MADDPG algorithm, belonging to the field of multi-UAV collaborative control technology. The method includes the following steps: initializing the UAV group and the fire suppression area, and constructing a generated forest topology network; setting the state space and reward function of the UAV group; constructing and training a UAV group encirclement task decision network model based on the MADDPG algorithm; and using the trained UAV group encirclement task decision network model based on the MADDPG algorithm to implement the encirclement task of the UAV group in a fire suppression scenario. This invention constructs a generated forest topology network by establishing a communication matrix LR between fire suppression UAVs and a communication matrix LT between fire suppression UAVs and a navigator UAV. A reward function is designed for the generated forest topology, effectively improving the stability and fault tolerance of UAV communication in a fire environment, while avoiding the model getting trapped in local optima or failing to converge.

Description

Technical Field

[0001] This invention relates to the field of multi-UAV cooperative control technology, and in particular to a UAV group fire cooperative encirclement control method based on the MADDPG algorithm. Background Technology

[0002] Fire disasters pose a serious threat to the natural environment and human society, making rapid and effective firefighting crucial. Multi-drone collaborative operations demonstrate great potential in fire emergency response due to their flexibility and efficiency. Through the coordinated actions of drone swarms, rapid reconnaissance of the fire scene, multi-angle fire monitoring, and coordinated firefighting and containment operations can be achieved.

[0003] In the field of multi-UAV cooperative control, multi-agent reinforcement learning (MARL) methods, especially multi-agent deep deterministic policy gradient algorithms (MADDPG), have attracted much attention due to their ability to handle continuous action spaces and achieve complex cooperative behaviors. Traditional MADDPG algorithms employ centralized training and distributed execution, requiring stable global communication capabilities between agents to exchange state, action, and other information for collaborative decision-making. However, in practical applications, UAV communication is often affected by factors such as terrain obstruction, electromagnetic interference, and node failures, leading to unstable communication links, information transmission delays, or interruptions. Especially in large-scale clusters or complex environments, the technical cost and energy consumption of maintaining full connectivity increase significantly, drastically reducing the cooperative efficiency and robustness of traditional algorithms.

[0004] In existing technologies, multi-UAV communication topologies often employ fully connected or fixed-hierarchical structures, lacking adaptability to communication-constrained conditions. For example, fully connected topologies assume all UAVs can communicate in real time, but in communication-constrained scenarios (such as partial link failures), the algorithm fails to obtain complete state information, leading to the failure of the collaborative strategy. Fixed-hierarchical topologies, on the other hand, lack flexibility and struggle to dynamically respond to changes in communication links. Furthermore, the reward function design of the traditional MADDPG algorithm does not explicitly consider communication topology constraints, making it difficult for agents to effectively utilize local information for collaboration under communication constraints, easily leading to local optima or failure to converge.

[0005] How to solve the above-mentioned technical problems is the challenge facing this invention. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method and system for collaborative fire encirclement control of UAV groups based on the MADDPG algorithm, which effectively improves the stability and fault tolerance of UAV communication in fire environments while avoiding the model getting trapped in local optima or failing to converge.

[0007] The technical solution adopted by this invention to solve its technical problem is:

[0008] This invention provides a method for cooperative fire encirclement control of unmanned aerial vehicle (UAV) groups based on the MADDPG algorithm, comprising the following steps:

[0009] Initialize the drone group and firefighting area, and construct and generate a forest topology network;

[0010] Configure the state space and reward function for the drone group;

[0011] Construct and train a decision network model for unmanned aerial vehicle (UAV) group encirclement missions based on the MADDPG algorithm;

[0012] The trained decision network model for drone swarm encirclement missions based on the MADDPG algorithm is used to realize the drone swarm encirclement mission in firefighting scenarios.

[0013] Preferably, the drone group includes at least one navigator drone and several firefighting drones, each of which is equipped with 12 distance sensors.

[0014] Preferably, the initialization of the drone group and firefighting area, and the establishment of the generated forest topology, includes:

[0015] Initialize the number m and position of the Navigator drones Number n of firefighting drones, location ,speed acceleration Maximum speed Maximum acceleration Distance sensor parameters The maximum detection range L of the distance sensor and the length and width of the fire extinguishing area are set. Obstacles are set to be circular with a fixed radius Radius. The positions of obstacles are randomly initialized.

[0016] The location set of firefighting drones is as follows: ,in, , represents the position of the i-th firefighting drone at time t.

[0017] The motion model of the firefighting drone is as follows: ,in, Let be the velocity of the i-th firefighting drone at time t. Let be the velocity of the i-th firefighting drone in the x-direction at time t-1. Let be the velocity of the i-th firefighting drone in the y-direction at time t-1. Let x be the acceleration of the i-th firefighting drone in the x-direction at time t. Let be the acceleration in the y-direction of the i-th firefighting drone at time t.

[0018] The location set of the Navigator drone is as follows: ,in, , , representing the position of the i-th Navigator UAV at time t.

[0019] The firefighting drone is used as a child node of the generated forest topology network, and the navigator drone is used as the root node of the generated forest topology network. The generated forest topology network is constructed by establishing a communication matrix LR between firefighting drones and a communication matrix LT between firefighting drones and navigator drones.

[0020] It should be noted that LR is an n×n matrix (n is the number of firefighting drones), if This indicates that the i-th firefighting drone can receive the message from the j-th firefighting drone; if This indicates that the i-th firefighting drone cannot receive messages from the j-th firefighting drone.

[0021] LT is an m×n matrix (where m is the number of navigators), if This indicates that the i-th firefighting drone can receive messages from the k-th navigator drone; if This indicates that the i-th firefighting drone cannot receive messages from the k-th firefighting drone.

[0022] The navigator drone acts as the root of the forest topology network, responsible for global decision-making (such as surrounding the fire-fighting area); the fire-fighting drone acts as a child node of the forest topology network, responsible for distance and environmental perception, and communication with the navigator and other drones.

[0023] Preferably, the distance between any two firefighting drones, the distance between any firefighting drone and any navigator drone, and the distance between any two navigator drones are all greater than 0.

[0024] Preferably, the state space S of the unmanned aerial vehicle group includes several states. The state This includes the individual status of several firefighting drones, represented as... The individual state of the i-th firefighting drone is ,in, Let i represent the motion state of the i-th firefighting drone. Let i represent the distance between the i-th firefighting drone and the obstacle. For the communication and location status of the i-th firefighting drone, , , , This indicates whether the i-th firefighting drone can receive the message from the j-th firefighting drone. This indicates whether the i-th firefighting drone can receive the message from the k-th navigator drone.

[0025] Preferably, the reward function for the i-th firefighting drone is defined as follows: , ;

[0026] in, The representative tracking reward function is calculated based on the distance between the firefighting drone and the target: ,

[0027] In the formula, Represents the tracking reward coefficient. This represents the distance between the i-th firefighting drone and the target at time t;

[0028]

[0029] It should be noted that the communication matrix is ​​used (Communication between firefighting drones) and (Communication between firefighting drones and navigators), only aggregating the positional differences of communicable nodes ( or (This is only included in the summation). Matching the local communication characteristics of the generated forest topology avoids using invalid information from non-communicable nodes, improving computational efficiency and decision rationality.

[0030] The reward for close proximity encourages drones to remain within effective range of the target, thereby improving firefighting efficiency. ,

[0031] In the formula, This represents the reward coefficient for maintaining proximity.

[0032] The reward represents directional alignment, where the drone aligns itself with the target's direction to enhance the encirclement effect.

[0033] ,

[0034] In the formula, Represents the direction alignment reward coefficient;

[0035] This represents an energy efficiency penalty / reward system, which penalizes strenuous movements that exceed a certain amplitude threshold, thereby reducing energy consumption.

[0036] ,

[0037] In the formula, Let be the motion vector of the i-th firefighting drone at time t. This represents the energy efficiency penalty / reward coefficient. Represents the threshold of motion amplitude;

[0038] This represents the reward for a single successful firefighting drone. When a single firefighting drone meets the success criteria, it receives a positive reward: ,

[0039] In the formula, This represents the reward value for a single successful firefighting drone operation. This represents the success distance threshold;

[0040] This represents the reward for all successful firefighting drones. When all drones meet the success criteria, an additional global positive reward is given: ,

[0041] In the formula, This represents the reward value for all successful firefighting drone operations.

[0042] Collision penalty reward: ,

[0043] In the formula, Represents the collision penalty reward value;

[0044] This represents a safety distance penalty, where a negative reward is given when the distance between drones or between drones and obstacles is less than a safety threshold. ,

[0045] In the formula, Represents the safety distance penalty coefficient. This represents the percentage of the safety threshold.

[0046] It should be noted that, To determine whether the i-th firefighting drone collided,

[0047]

[0048] In the formula, This is the safe distance for a collision.

[0049] To determine whether the i-th firefighting drone has completed its mission,

[0050]

[0051] In the formula, The current round's step size. This represents the maximum step size for a round.

[0052] It should be noted that the threshold and coefficient values ​​involved in the formula were optimized in experiments based on specific tasks, combining trial and error with experience to obtain the best training results. The value range is [2, 4]. The value range is [1, 3]. The value range is [1, 3]. The value range is [0.1, 0.3]. The value range is [5, 15]. The value range is [300, 600]. The value range is [5, 15]. The value range is [5, 15]. The value range is [0.6, 0.8]. The value range is [0.3, 0.8]. The value range is [0.001, 0.006]. The value range is [0.01, 0.06]. Value range: [0.9, 0.99] The value range is [0.001, 0.01].

[0053] Preferably, the construction and training of the UAV group encirclement task decision network model based on the MADDPG algorithm includes:

[0054] For each firefighting drone, an Actor network and a Critic network are constructed, and the parameters of the Actor network and the Critic network are randomly initialized. Both the Actor network and the Critic network include a main network and a target network. The parameters of the main network and the target network of the Actor network are the same, and the parameters of the main network and the target network of the Critic network are the same.

[0055] It should be noted that the Actor network is responsible for generating actions. It consists of a main network and a target network. The main network directly outputs the current action, while the target network is used to generate future actions. Through soft parameter updates, the parameters of the target network are gradually updated to follow those of the main network, thus preventing training divergence.

[0056] The Critic network evaluates the value of actions generated by the Actor network (e.g., whether the action brings the drone closer to the target or whether it is safe). It also includes a main network and a target network. The main network evaluates the value of the current action, while the target network assists in calculating future long-term rewards, thus stabilizing the training process.

[0057] "Centralized training" optimizes the network using global information, improving policy learning efficiency; "distributed decision-making" empowers the drone with autonomous execution capabilities, adapting to complex scenarios; and "master-target network architecture" (including both Actor and Critic) stabilizes the training process, preventing learning failures due to drastic parameter fluctuations.

[0058] For the i-th firefighting drone, input the individual state into the Actor main network. Generate actions According to the action Earn reward points by interacting with the environment The state at the next moment and whether the task has been completed Loop through each firefighting drone and check the individual status of all firefighting drones. ,action Reward Value Next moment state and whether the task has been completed As an empirical sample The experience samples are placed into the experience storage pool and continuously interacted with the environment to obtain more experience samples, which are then stored in the experience storage pool.

[0059] A batchSize number of experience samples are sampled from the experience storage pool to form an experience set. The experience set is the set of states of the i-th firefighting drone at the next moment. Input the Actor target network of the i-th firefighting drone to obtain the target action set. The set of target actions for all firefighting drones is obtained by looping through the data. ,in, , ;

[0060] Set the target actions of all firefighting drones and the next-moment state set of all firefighting drones in the experience set [ ,in, , Given the Critic target network of the i-th firefighting drone, obtain the Q-values ​​of the target network for batchSize empirical samples of the i-th firefighting drone. ,in, j=1,2,…,batchSize , These are the parameters of the Critic target network.

[0061] Calculate the target Q-value based on the target network Q-value and reward function:

[0062] ,

[0063] In the formula, These are the parameters of the Critic target network. As a discount factor, To determine whether the i-th firefighting drone has completed its mission in the j-th sample. Let be the reward value for the i-th firefighting drone.

[0064] It should be noted that when (When the task is not completed) ,in, It is the immediate reward gained from the current action. It represents the future discounted reward corresponding to the state and action at the next moment (estimated through the Q-value of the target network), reflecting a consideration of long-term returns.

[0065] when (When the task has been completed) At this point, we only focus on the current immediate reward and no longer consider future rewards.

[0066] Calculate the Critic network loss based on the sample main network Q-value and the target Q-value:

[0067] ;

[0068] In the formula, The Q-value of the sample main network.

[0069] The Critic network gradient is calculated using backpropagation based on the Critic network loss, and the Critic master network parameters of the firefighting drone are updated; the Critic master network parameter update for all firefighting drones is completed iteratively.

[0070] The set of individual states of the i-th firefighting drone in the experience set [ Input the Actor main network to obtain the action set [ The action set of all firefighting drones is obtained by looping through them. ,in, ;

[0071] Update the action set of the i-th firefighting drone as follows The set of individual states of all firefighting drones in the joint experience set [ ,in, Input the Critic master network of the i-th firefighting drone to obtain the Q-value of the policy master network. ,in, j=1, 2,…,batchSize , These are the parameters for the Critic main network.

[0072] It should be noted that the Q-value of the sample main network It is obtained by sampling the state set and action set of firefighting drones from the input experience set, and the Q value of the main network of the policy is obtained. It is obtained by sampling the state set of firefighting drones from the input experience set and the updated action set of the i-th firefighting drone.

[0073] Calculate the gradient of the Actor network based on the Q-value of the main strategy network:

[0074]

[0075] The Actor main network is updated using the gradient ascent algorithm.

[0076] Perform soft updates to the Actor and Critic target networks at fixed intervals.

[0077] ;

[0078] In the formula, and These are the parameters of the Critic main network and the target network, respectively. and These are the parameters of the Actor's main network and target network, respectively. It is a relatively small coefficient ( <1, typical value is 0.01).

[0079] During each soft update, the target network parameters partially absorb the main network parameters while retaining most of their original values. This allows the target network parameters to slowly and smoothly converge towards the main network, avoiding drastic fluctuations in target values ​​and ensuring training stability and convergence.

[0080] Repeat the above steps until the maximum number of training iterations is reached and the model that obtains the maximum average reward per step during training is saved, resulting in a fully trained UAV group encirclement task decision network model based on the MADDPG algorithm.

[0081] Preferably, when inputting the state into the Actor main network and generating actions, noise is added to the actions and restrictions are imposed on them:

[0082] ,

[0083]

[0084] In the formula, It is a random vector uniformly distributed in the interval [0, 1); This is a constant value, representing the minimum noise level. This is a constant value, representing the maximum value of the noise. The attenuation rate, This is the global training step size.

[0085] The beneficial effects of this invention are as follows: Using firefighting drones as child nodes and a navigator drone as the root node, a generative forest topology network is constructed by establishing a communication matrix LR between firefighting drones and a communication matrix LT between firefighting drones and the navigator drone. When some communication links or nodes fail, other child nodes can still maintain information propagation, effectively addressing the problem of easy communication interruption in fully connected topologies under complex environments, maintaining the consistency of collaborative decision-making and the robustness of the system, and avoiding collaborative failure due to communication interruptions. Combined with the generative forest topology design reward function, the model can utilize local information to achieve collaboration under communication constraints, avoiding the model getting trapped in local optima or failing to converge. Attached Figure Description

[0086] Figure 1 This is a diagram illustrating the method steps of the present invention.

[0087] Figure 2 This is a diagram of the Actor and Critic network structure of the present invention.

[0088] Figure 3 This is a schematic diagram of the fire-fighting coordination and encirclement of unmanned aerial vehicles (UAVs) in Embodiment 2 of the present invention. Detailed Implementation

[0089] To clearly illustrate the technical features of this solution, the following detailed implementation method will be used to explain the solution.

[0090] Example 1:

[0091] See Figure 1 , Figure 2 As shown, this embodiment is a method for cooperative encirclement control of unmanned aerial vehicle (UAV) group fires based on the MADDPG algorithm, including the following steps:

[0092] S1. Initialize the drone group and firefighting area, and construct the generated forest topology network.

[0093] The drone group consists of at least one navigator drone and several firefighting drones, each equipped with several distance sensors.

[0094] Initialize the number m and position of the Navigator drones Number n of firefighting drones, location ,speed acceleration Maximum speed Maximum acceleration Distance sensor parameters The maximum detection range L of the distance sensor and the length and width of the fire extinguishing area are set. Obstacles are set to be circular with a fixed radius Radius. The positions of obstacles are randomly initialized.

[0095] The location set of firefighting drones is as follows: ,in, , represents the position of the i-th firefighting drone at time t.

[0096] The motion model of the firefighting drone is as follows: ,in, Let be the velocity of the i-th firefighting drone at time t. Let be the velocity of the i-th firefighting drone in the x-direction at time t-1. Let be the velocity of the i-th firefighting drone in the y-direction at time t-1. Let x be the acceleration of the i-th firefighting drone in the x-direction at time t. Let be the acceleration in the y-direction of the i-th firefighting drone at time t.

[0097] The location set of the Navigator drone is as follows: ,in, , , representing the position of the i-th Navigator UAV at time t.

[0098] The distance between any two firefighting drones, the distance between any firefighting drone and any navigator drone, and the distance between any two navigator drones are all greater than 0.

[0099] The firefighting drone is used as a child node of the generated forest topology network, and the navigator drone is used as the root node of the generated forest topology network. The generated forest topology network is constructed by establishing a communication matrix LR between firefighting drones and a communication matrix LT between firefighting drones and navigator drones.

[0100] It should be noted that LR is an n×n matrix (n is the number of firefighting drones), if This indicates that the i-th firefighting drone can receive the message from the j-th firefighting drone; if This indicates that the i-th firefighting drone cannot receive messages from the j-th firefighting drone.

[0101] LT is an m×n matrix (where m is the number of navigators), if This indicates that the i-th firefighting drone can receive messages from the k-th navigator drone; if This indicates that the i-th firefighting drone cannot receive messages from the k-th firefighting drone.

[0102] The navigator drone acts as the root of the forest topology network, responsible for global decision-making (such as surrounding the fire-fighting area); the fire-fighting drone acts as a child node of the forest topology network, responsible for distance and environmental perception, and communication with the navigator and other drones.

[0103] S2. Set the state space and reward function for the drone group.

[0104] The state space S of the unmanned aerial vehicle group includes several states. The state This includes the individual status of several firefighting drones, represented as... The individual state of the i-th firefighting drone is ,in, Let i represent the motion state of the i-th firefighting drone. Let i represent the distance between the i-th firefighting drone and the obstacle. For the communication and location status of the i-th firefighting drone, , , , This indicates whether the i-th firefighting drone can receive the message from the j-th firefighting drone. This indicates whether the i-th firefighting drone can receive the message from the k-th navigator drone.

[0105] The reward function for the i-th firefighting drone is defined as follows: , ;

[0106] in, The representative tracking reward function is calculated based on the distance between the firefighting drone and the target: ,

[0107] In the formula, Represents the tracking reward coefficient. This represents the distance between the i-th firefighting drone and the target at time t;

[0108]

[0109] It should be noted that the communication matrix is ​​used (Communication between firefighting drones) and (Communication between firefighting drones and navigators), only aggregating the positional differences of communicable nodes ( or (This is only included in the summation). Matching the local communication characteristics of the generated forest topology avoids using invalid information from non-communicable nodes, improving computational efficiency and decision rationality.

[0110] The reward for close proximity encourages drones to remain within effective range of the target, thereby improving firefighting efficiency. ,

[0111] In the formula, This represents the reward coefficient for maintaining proximity.

[0112] The reward represents directional alignment, where the drone aligns itself with the target's direction to enhance the encirclement effect.

[0113] ,

[0114] In the formula, Represents the direction alignment reward coefficient;

[0115] This represents an energy efficiency penalty / reward system, which penalizes strenuous movements that exceed a certain amplitude threshold, thereby reducing energy consumption.

[0116] ,

[0117] In the formula, Let be the motion vector of the i-th firefighting drone at time t. This represents the energy efficiency penalty / reward coefficient. Represents the threshold of motion amplitude;

[0118] This represents the reward for a single successful firefighting drone. When a single firefighting drone meets the success criteria, it receives a positive reward: ,

[0119] In the formula, This represents the reward value for a single successful firefighting drone operation. This represents the success distance threshold;

[0120] This represents the reward for all successful firefighting drones. When all drones meet the success criteria, an additional global positive reward is given: ,

[0121] In the formula, This represents the reward value for all successful firefighting drone operations.

[0122] Collision penalty reward: ,

[0123] In the formula, Represents the collision penalty reward value;

[0124] This represents a safety distance penalty, where a negative reward is given when the distance between drones or between drones and obstacles is less than a safety threshold. .

[0125] It should be noted that, To determine whether the i-th firefighting drone collided,

[0126]

[0127] In the formula, This is the safe distance for a collision.

[0128] To determine whether the i-th firefighting drone has completed its mission,

[0129]

[0130] In the formula, The current round's step size. This represents the maximum step size for a round.

[0131] S3. Construct and train a UAV group encirclement task decision network model based on the MADDPG algorithm.

[0132] For each firefighting drone, an Actor network and a Critic network are constructed, and the parameters of the Actor network and the Critic network are randomly initialized. Both the Actor network and the Critic network include a main network and a target network. The parameters of the main network and the target network of the Actor network are the same, and the parameters of the main network and the target network of the Critic network are the same.

[0133] It should be noted that the Actor network is responsible for generating actions. It consists of a main network and a target network. The main network directly outputs the current action, while the target network is used to generate future actions. Through soft parameter updates, the parameters of the target network are gradually updated to follow those of the main network, thus preventing training divergence.

[0134] The Critic network evaluates the value of actions generated by the Actor network (e.g., whether the action brings the drone closer to the target or whether it is safe). It also includes a main network and a target network. The main network evaluates the value of the current action, while the target network assists in calculating future long-term rewards, thus stabilizing the training process.

[0135] "Centralized training" optimizes the network using global information, improving policy learning efficiency; "distributed decision-making" empowers the drone with autonomous execution capabilities, adapting to complex scenarios; and "master-target network architecture" (including both Actor and Critic) stabilizes the training process, preventing learning failures due to drastic parameter fluctuations.

[0136] For the i-th firefighting drone, input the state into the Actor main network. Generate actions .

[0137] When inputting state into the Actor main network and generating actions, noise is added to the actions and restrictions are imposed on them:

[0138] ,

[0139]

[0140] In the formula, It is a random vector uniformly distributed in the interval [0, 1); This is a constant value, representing the minimum noise level. This is a constant value, representing the maximum value of the noise. The attenuation rate, This is the global training step size.

[0141] According to the action Earn reward points by interacting with the environment The state at the next moment and whether the task has been completed Loop through each firefighting drone and record the status of all firefighting drones. ,action Reward Value Next moment state and whether the task has been completed As an empirical sample The experience samples are placed into the experience storage pool and continuously interacted with the environment to obtain more experience samples, which are then stored in the experience storage pool.

[0142] A batchSize set of experience samples is formed by sampling experience samples from the experience storage pool. The next state set of the i-th firefighting drone in the experience set is then set. Input the Actor target network of the i-th firefighting drone to obtain the target action set. The set of target actions for all firefighting drones is obtained by looping through the data. ,That , ;

[0143] Set the target actions of all firefighting drones and the next-moment state set of all firefighting drones in the experience set [ ,in, , Given the Critic target network of the i-th firefighting drone, obtain the Q-values ​​of the target network for batchSize empirical samples of the i-th firefighting drone. .

[0144] Calculate the target value based on the target network's Q-value and reward function:

[0145] ,

[0146] In the formula, These are the parameters of the Critic target network. As a discount factor, To determine whether the i-th firefighting drone has completed its mission in the j-th sample. Let be the reward value for the i-th firefighting drone.

[0147] It should be noted that when (When the task is not completed) ,in, It is the immediate reward gained from the current action. It represents the future discounted reward corresponding to the state and action at the next moment (estimated through the Q-value of the target network), reflecting a consideration of long-term returns.

[0148] when (When the task has been completed) At this point, we only focus on the current immediate reward and no longer consider future rewards.

[0149] Calculate the Critic network loss based on the sample main network Q-value and the target value:

[0150] ;

[0151] In the formula, The Q-value of the sample main network.

[0152] The Critic network gradient is calculated by backpropagation based on the Critic network loss, and the Critic master network parameters of the firefighting drone are updated; the Critic master network parameter update for all firefighting drones is completed in a loop.

[0153] The set of individual states of the i-th firefighting drone in the experience set [ Input the Actor main network to obtain the action set [ The action set of all firefighting drones is obtained by looping through them. ,in, ;

[0154] Update the action set of the i-th firefighting drone as follows The set of individual states of all firefighting drones in the joint experience set [ ,in, Input the Critic master network of the i-th firefighting drone to obtain the Q-value of the policy master network. ,in, j=1, 2,…,batchSize , These are the parameters for the Critic main network.

[0155] It should be noted that the Q-value of the sample main network It is obtained by sampling the state set and action set of firefighting drones from the input experience set, and the Q value of the main network of the policy is obtained. It is obtained by sampling the state set of firefighting drones from the input experience set and the updated action set of the i-th firefighting drone.

[0156] Calculate the gradient of the Actor network based on the Q-value of the main network:

[0157]

[0158] The Actor main network is updated using the gradient ascent algorithm.

[0159] Perform soft updates to the Actor and Critic target networks at fixed intervals.

[0160] ;

[0161] In the formula, and These are the parameters of the Critic main network and the target network, respectively. and These are the parameters of the Actor's main network and target network, respectively. It is a relatively small coefficient ( <1, typical value is 0.01).

[0162] During each soft update, the target network parameters partially absorb the main network parameters while retaining most of their original values. This allows the target network parameters to slowly and smoothly converge towards the main network, avoiding drastic fluctuations in target values ​​and ensuring training stability and convergence.

[0163] Repeat the above steps until the maximum number of training iterations is reached and the model that obtains the maximum average reward per step during training is saved, resulting in a fully trained UAV group encirclement task decision network model based on the MADDPG algorithm.

[0164] S4. Use the trained decision network model for drone group encirclement task based on the MADDPG algorithm to realize the drone group encirclement task in the firefighting scenario.

[0165] Example 2:

[0166] To demonstrate the effectiveness of the proposed MADDPG algorithm-based UAV fire collaborative encirclement control method, this embodiment uses 6 firefighting UAVs and 4 navigator UAVs to construct and generate a forest topology for fire encirclement simulation.

[0167] In this embodiment, the Actor network and Critic network for the firefighting drone are specifically defined as follows:

[0168] In the Actor network, the input state dimension is 18, with two fully connected hidden layers (128 dimensions) and LeakyReLU activation. The output layer dimension is 2 and Softsign activation. The output action is processed by L2 norm judgment. When the magnitude of the action vector is greater than 1, it is normalized to a unit vector (divided by the magnitude). When the magnitude is less than or equal to 1, the original value remains unchanged (multiplied by 1). Finally, the processed action vector is scaled to the range of [-a_max, a_max]. In the Critic network, the input dimension is 120, with two fully connected hidden layers (128 dimensions) and ReLU activation. The output layer dimension is 1.

[0169] The model training parameters, drone group parameters, and fire area parameters are as follows:

[0170] The Actor network uses the Adam optimizer with a learning rate alpha = 0.0001, and the Critic network also uses the Adam optimizer with a learning rate beta = 0.001. The total number of training epochs is 10,000, the maximum step size per epoch is 200, the learning interval is 10 steps, and the retention condition is an increase in average reward per step. =0.9, =0.01, experience pool size is 100,000, batch size is 1024, round termination condition is: reaching the maximum number of steps or all firefighting drones complete the mission, during training, scene length=3, width=3, navigator position is randomly generated in the 0.5-2.5 area, firefighting drone positions are... The obstacle locations are randomly generated with a radius of 0.15, while ensuring that they do not cover the firefighting drone and the navigator; , Maximum speed Maximum acceleration Update duration is Maximum detection range of the sensor , , .

[0171] When using the trained Actor network model to perform a multi-drone encirclement task in a firefighting scenario, the positions of the firefighting drones are as follows: The navigator's position is The location of the obstacle is Ultimately, the coordinated fire encirclement effect of the drone group was as follows: Figure 3 As shown,

[0172] In the diagram, the larger circular areas represent obstacles, the solid lines represent the flight paths of the firefighting drones, the arrows on the solid lines indicate the endpoints of the firefighting drones' flights, the smaller circular areas represent the positions of the navigator drones, and the area enclosed by the dashed lines represents the fire zone. This embodiment uses the navigator drone as the root node and the firefighting drones as child nodes, constructing a topology connection through communication matrices LR (communication between firefighting drones) and LT (communication between firefighting drones and the navigator). Even if some communication links or nodes fail ( =0 or (=0), other child nodes can still maintain information transmission, ensuring that the drone group avoids obstacles, adjusts its flight path, and effectively surrounds the fire area, improving the robustness of the model and avoiding collaborative failure caused by communication interruption in complex environments.

[0173] The technical features of this invention not described can be implemented by or using existing technology, and will not be repeated here. Of course, the above description is not a limitation of this invention, and this invention is not limited to the examples above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of this invention should also be within the protection scope of this invention.

Claims

1. A method for fire cooperative surrounding control of a UAV group based on a MADDPG algorithm, characterized in that, The method comprises the following steps: initializing a UAV team and a fire extinguishing area, and constructing a generated forest topology network; setting a state space and a reward function of the UAV team; constructing and training a UAV team surrounding task decision network model based on a MADDPG algorithm; implementing the surrounding task of the UAV team in a fire extinguishing scene by using the trained UAV team surrounding task decision network model based on the MADDPG algorithm; the initialization of the UAV team and the fire extinguishing area, and the establishment of the generated forest topology comprise: Initialize the number of leader drones and positions , the number of fire extinguishing drones , positions , velocities , accelerations , maximum velocities , maximum accelerations , distance sensor parameters , the maximum detection range L of the distance sensor and the length Length and width Width of the fire extinguishing area, set the obstacles to be circular, the radius Radius is a fixed value, and the positions Obs of the obstacles are randomly initialized The state space S of the unmanned aerial vehicle group includes several states. The state This includes the individual status of several firefighting drones, represented as... ;No. The individual status of the firefighting drone is ,in, For the first The movement status of the firefighting drone itself. For the first The distance between the firefighting drone and the obstacle. For the first Communication and location status of the firefighting drone. , , , , , Representing the Can a firefighting drone receive the first... The news of the firefighting drone Representing the Can the firefighting drone receive the first... News about a leading drone manufacturer; The reward function of the fire extinguishing unmanned aerial vehicle is defined as , , ; wherein representative pursuit reward function , In the formula, represents a tracking reward coefficient, represents the first The fire extinguishing unmanned aerial vehicle is in The distance between the time and the target; Representing close proximity to maintain rewards: , In the formula, represents the near distance maintenance reward coefficient; Represents the direction alignment reward: ， In the formula, represents the direction alignment reward coefficient; Represents energy efficiency penalty and reward: ， In the formula, is the first The fire extinguishing unmanned plane in The action vector at the moment, Represent the energy efficiency penalty reward coefficient, Represent the action amplitude threshold; Representing a single successful fire extinguishing drone reward: , In the formula, represents a successful reward value of a single fire-extinguishing drone, represents a successful distance threshold value; Represent all fire extinguishing drones successfully rewarded: , In the formula, represents the success reward value of all fire extinguishing drones; Representative collision penalty reward: , In the formula, represents the collision penalty reward value; representative safety distance penalty: , In the formula, Represents the safety distance penalty coefficient. Represents the proportion of the safety threshold. For the first Firefighting drones and the first The distance between firefighting drones.

2. The method of claim 1, wherein the MADDPG algorithm-based UAV team fire cooperative encirclement control method is characterized in that, the UAV team comprises at least one leader UAV and a plurality of fire extinguishing UAVs, and each of the fire extinguishing UAVs is equipped with a plurality of distance sensors.

3. The method of claim 2, wherein the MADDPG algorithm-based UAV team fire cooperative encirclement control method is characterized in that, The distance between any two fire extinguishing UAVs, the distance between any one fire extinguishing UAV and any one leader UAV, and the distance between any two leader UAVs are all greater than 0.

4. The method of claim 3, wherein the MADDPG algorithm-based UAV team fire cooperative encirclement control method is characterized in that, The construction and training of the UAV team surrounding task decision network model based on the MADDPG algorithm comprise: an Actor network and a Critic network are constructed for each fire extinguishing UAV, and the Actor network parameters and the Critic network parameters are randomly initialized, the Actor network and the Critic network each comprise a main network and a target network, the main network and the target network of the Actor network have the same parameters, and the main network and the target network of the Critic network have the same parameters; To the first The fire extinguishing unmanned aerial vehicle inputs the individual state to the Actor master network , generates the action , interacts with the environment according to the action , obtains the reward value , the state of the next moment And whether the task is completed state , loop each fire extinguishing unmanned aerial vehicle, put all the individual space , action , reward value , next moment state And whether the task is completed state As an experience sample Put into the experience storage pool, and continue to obtain experience samples from the environment interaction and store them in the experience storage pool; Sampling batchSize experience sample groups from the experience storage pool to form an experience set , wherein , the first , the next time state set of the fire extinguishing UAV , inputting the Actor target network of the fire extinguishing UAV , obtaining a target action set , obtaining the target action set of all fire extinguishing UAVs through a loop , wherein , ; collecting all target action sets of the fire extinguishing unmanned aerial vehicles and experience all next moment state sets of the fire extinguishing unmanned aerial vehicles , wherein, , , inputting the Critic target network of the first fire extinguishing unmanned aerial vehicle, obtaining the target network Q value of the first fire extinguishing unmanned aerial vehicle batchSize experience samples; a target Q value is calculated based on a target network Q value and a reward function, a Critic network loss is calculated according to a sample main network Q value and the target Q value, a Critic network gradient is calculated by reverse propagation according to the Critic network loss, and the Critic main network parameters of the fire extinguishing UAV are updated; the Critic main network parameter updating of all the fire extinguishing UAVs is completed in cycles; Concentrate the experience on the Individual state set of the fire extinguishing unmanned plane Input the Actor master network to obtain the action set , and loop to obtain the action set of all fire extinguishing unmanned planes , wherein , ; updating the first action set of the fire extinguishing drone is , the individual state set of all fire extinguishing drones in the joint experience set , wherein , , input the Critic master network of the first fire extinguishing drone, obtain the Q value of the policy master network, calculate the Actor network gradient according to the Q value of the policy master network, and update the Actor master network through the gradient ascent algorithm; loop to complete the Actor master network parameter update of all fire extinguishing drones; the Actor target network and the Critic target network are updated every fixed time ； wherein, and are Critic main network and target network parameters, respectively, and are Actor main network and target network parameters, respectively, is a small coefficient; the above steps are repeated until the maximum training times are reached, and the model with the maximum average reward in each step is saved in the training, thereby obtaining the trained UAV team surrounding task decision network model based on the MADDPG algorithm.

5. The method of claim 4, wherein the MADDPG algorithm-based UAV team fire cooperative encirclement control method is characterized in that, when the state is input into the Actor main network to generate an action, noise is added to the action and the action is limited: wherein, is a random vector uniformly distributed in the interval [0, 1); is a constant representing the minimum value of the noise; is a constant representing the maximum value of the noise; is the decay rate, is the global training step size.

Images