Unmanned aerial vehicle group fire cooperative surrounding control method based on MADDPG algorithm

Abstract

The invention provides an unmanned aerial vehicle group fire cooperative surrounding control method based on an MADDPG algorithm, and relates to the technical field of multi-unmanned aerial vehicle cooperative control, and the method comprises the following steps: initializing an unmanned aerial vehicle group and a fire extinguishing region, and constructing a generative forest topology network; setting a state space and a reward function of the unmanned aerial vehicle group; constructing and training an unmanned aerial vehicle group surrounding task decision network model based on an MADDPG algorithm; and using the trained MADDPG algorithm-based unmanned aerial vehicle group surrounding task decision network model to realize a surrounding task of the unmanned aerial vehicle group in a fire extinguishing scene. According to the method, the communication matrix LR between the fire extinguishing unmanned aerial vehicles and the communication matrix LT between the fire extinguishing unmanned aerial vehicles and the pilot unmanned aerial vehicle are established to construct the generated forest topology network, and the reward function is designed for the generated forest topology, so that the communication stability and fault tolerance of the unmanned aerial vehicles in the fire environment are effectively improved; and meanwhile, the condition that the model is caught in local optimum or cannot converge is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-UAV cooperative control, and in particular to a UAV group fire cooperative encirclement control method based on the MADDPG algorithm. Background Art

[0002] Fire disasters pose a serious threat to the natural environment and human society, making rapid and effective firefighting crucial. Multi-drone collaborative operations, due to their flexibility and efficiency, show great potential in fire emergency response. The coordinated actions of drone swarms enable rapid fire reconnaissance, multi-angle fire monitoring, and coordinated firefighting and containment operations.

[0003] In the field of multi-UAV collaborative control, Multi-Agent Reinforcement Learning (MARL) methods, particularly the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, have attracted considerable attention due to their ability to handle continuous action spaces and implement complex collaborative behaviors. Traditional MADDPG algorithms employ centralized training and distributed execution, requiring stable global communication between agents to exchange information such as state and action for collaborative decision-making. However, in practical applications, UAV communications are often affected by factors such as terrain obstruction, electromagnetic interference, and node failures, resulting in unstable communication links, delayed or interrupted information transmission, and especially in large-scale clusters or complex environments. The technical cost and energy consumption of maintaining fully connected communications increase significantly, significantly reducing the collaborative efficiency and robustness of traditional algorithms.

[0004] Existing multi-UAV communication topologies often employ fully connected or fixed hierarchical structures, lacking adaptability to communication-constrained scenarios. For example, fully connected topologies assume that all UAVs can communicate with each other in real time. However, in communication-constrained scenarios (e.g., partial link failure), the algorithm cannot obtain complete state information, resulting in the failure of collaborative strategies. Fixed hierarchical topologies lack flexibility and struggle to dynamically adapt to changes in communication links. Furthermore, the reward function design of the traditional MADDPG algorithm fails to explicitly consider communication topology constraints, making it difficult for agents to effectively utilize local information to achieve collaboration under communication-constrained conditions, leading to the risk of becoming trapped in local optima or failing to converge.

[0005] How to solve the above technical problems is the subject faced by the present invention. Summary of the Invention

[0006] In order to address the shortcomings of the existing technology, the present invention provides a drone group fire collaborative surround control method and system based on the MADDPG algorithm, which effectively improves the stability and fault tolerance of drone communication in a fire environment, while avoiding the model falling into local optimality or non-convergence.

[0007] The technical solution adopted by the present invention to solve its technical problem is: The present invention provides a method for coordinated fire control of a UAV group based on the MADDPG algorithm, comprising the following steps: Initialize the drone group and fire extinguishing area, and build a forest topology network; Set up the state space and reward function for the drone group; Build and train a drone group encirclement task decision network model based on the MADDPG algorithm; The trained drone group encirclement task decision network model based on the MADDPG algorithm is used to realize the encirclement task of the drone group in the fire-fighting scenario.

[0008] Preferably, the drone group includes at least one navigator drone and several fire-fighting drones, and each of the fire-fighting drones is equipped with 12 distance sensors.

[0009] Preferably, the initializing of the drone group and the fire extinguishing area and establishing the generated forest topology includes: Initialize the number m and position of the leader drone , the number of fire-fighting drones n, location ,speed , acceleration , maximum speed , maximum acceleration , distance sensor parameters , the maximum detection range L of the distance sensor and the length Length and width Width of the fire extinguishing area, set the obstacle to a circle with a fixed radius Radius, and randomly initialize the position Obs of the obstacle.

[0010] The firefighting drone location set is: ,in, , represents the position of the i-th firefighting drone at time t.

[0011] The motion model of the firefighting drone is: ,in, is the speed of the i-th firefighting drone at time t, is the speed of the i-th firefighting drone in the x direction at time t-1, is the speed of the i-th firefighting drone in the y direction at time t-1, is the acceleration of the i-th firefighting drone in the x direction at time t, is the acceleration of the i-th firefighting drone in the y direction at time t.

[0012] The set of pilot drone positions is: ,in, , , represents the position of the i-th leader UAV at time t.

[0013] The firefighting drone is used as a child node of the generated forest topology network, and the navigator drone is used as the root node of the generated forest topology network. The generated forest topology network is constructed by establishing a communication matrix LR between the firefighting drones and a communication matrix LT between the firefighting drones and the navigator drone.

[0014] It should be noted that LR is an n×n matrix (n is the number of firefighting drones). , indicating that the i-th firefighting drone can receive the message from the j-th firefighting drone; if , indicating that the i-th firefighting drone cannot receive the message from the j-th firefighting drone.

[0015] LT is an m×n matrix (m is the number of pilots), if , indicating that the i-th firefighting drone can receive the message from the k-th navigator drone; if , indicating that the i-th firefighting drone cannot receive the message from the k-th firefighting drone.

[0016] The navigator drone serves as the root of the forest topology network and is responsible for global decision-making (such as surrounding the fire-fighting area). The fire-fighting drone serves as a child node of the forest topology network and is responsible for distance environment perception and communication with the navigator and other drones.

[0017] Preferably, the distance between any two of the fire-fighting drones, the distance between any one of the fire-fighting drones and any one of the pilot drones, and the distance between any two pilot drones are all greater than 0.

[0018] Preferably, the state space S of the drone group includes several states , the state Including the individual states of several fire-fighting drones, represented as ;The individual state of the i-th firefighting drone is ,in, is the motion state of the i-th firefighting drone, is the distance between the i-th firefighting drone and the obstacle, is the communication and position status of the i-th firefighting drone, , , , Indicates whether the i-th firefighting drone can receive the message from the j-th firefighting drone. Indicates whether the i-th firefighting drone can receive the message from the k-th leader drone.

[0019] Preferably, the reward function of the i-th firefighting drone is defined as , ; in, Represents the tracking reward function, which is calculated based on the distance between the firefighting drone and the target: , Where, Represents the tracking reward coefficient, represents the distance between the i-th firefighting drone and the target at time t;

[0020] It should be noted that the communication matrix (communication between firefighting drones) and (Firefighting drone communicates with the navigator), only the position difference of the communicable nodes is aggregated ( or The local communication characteristics of the forest topology are matched to avoid using invalid information of non-communicable nodes, thus improving computational efficiency and decision rationality.

[0021] Close Range Rewards encourages drones to stay within the effective range near the target, improving firefighting efficiency. , Where, Represents the close range maintenance bonus coefficient; Represents the direction alignment bonus, which rewards the drone for keeping the target direction consistent, enhancing the encirclement effect: , Where, Represents the direction alignment reward coefficient; Represents the energy efficiency penalty reward, which penalizes violent movements that exceed the action amplitude threshold to reduce energy consumption: , Where, is the action vector of the i-th firefighting drone at time t, represents the energy efficiency penalty reward coefficient, represents the action amplitude threshold; Represents the success reward for a single firefighting drone. When a single firefighting drone meets the success conditions, the firefighting drone receives a positive reward: , Where, Represents the success reward value of a single firefighting drone, represents the success distance threshold; Represents the success reward for all firefighting drones. When all drones meet the success conditions, additional global positive rewards will be given: , Where, Represents the success reward value of all fire-fighting drones; Represents collision penalty reward: , Where, Represents the collision penalty reward value; Represents the safety distance penalty, which gives negative rewards when the distance between drones or obstacles is less than the safety threshold: , Where, Represents the safety distance penalty coefficient, Represents the safety threshold ratio.

[0022] It should be noted that Is the collision of the i-th firefighting drone?

[0023] Where, The collision safety distance.

[0024] Whether the i-th firefighting drone has completed its mission,

[0025] Where, is the current round step length, is the maximum step length of the round.

[0026] It should be noted that the thresholds and coefficients involved in the formula are optimized in the experiment according to the specific task combined with trial and error and experience to obtain the best training effect. The value range is [2, 4], The value range is [1, 3], The value range is [1, 3], The value range is [0.1, 0.3], The value range is [5, 15], The value range is [300, 600]. The value range is [5, 15], The value range is [5, 15], The value range is [0.6, 0.8], The value range is [0.3, 0.8], The value range is [0.001, 0.006], The value range is [0.01, 0.06], Value range: [0.9, 0.99] The value range is [0.001, 0.01].

[0027] Preferably, the construction and training of the drone group encirclement task decision network model based on the MADDPG algorithm includes: For each firefighting drone, an actor network and a critic network are constructed, and the actor network parameters and the critic network parameters are randomly initialized. The actor network and the critic network both include a main network and a target network. The main network and the target network parameters of the actor network are the same, and the main network and the target network parameters of the critic network are the same. It should be noted that the Actor network is responsible for generating actions. It consists of a main network and a target network. The main network directly outputs the current action, while the target network is used to generate future actions. Through soft parameter updates, the target network parameters slowly follow the main network to avoid training divergence.

[0028] The Critic network evaluates the value of actions generated by the Actor network (e.g., whether the action brings the drone closer to its target and whether it is safe). It also includes a main network and a target network. The main network evaluates the value of the current action, while the target network assists in calculating future long-term rewards and stabilizing the training process.

[0029] Through "centralized training", global information is used to optimize the network and improve the efficiency of strategy learning; through "distributed decision-making", drones are given autonomous execution capabilities and can adapt to complex scenarios; through the "master-target network architecture" (including both actors and critics), the training process is stabilized to avoid learning failures caused by drastic fluctuations in parameters.

[0030] For the i-th firefighting drone, input the individual state to the Actor main network , generate actions , according to the action Interact with the environment to gain rewards , the state at the next moment and whether the task status has been completed , loop each fire-fighting drone and set the individual status of all fire-fighting drones ,action , reward value , next moment status and whether the task status has been completed As an empirical sample Put it into the experience storage pool, and continue to interact with the environment to continue to obtain experience samples and store them in the experience storage pool; Sample batchSize experience samples from the experience storage pool to form an experience set , the next moment state set of the i-th firefighting drone in the experience concentration [ Input the Actor target network of the i-th firefighting drone and obtain the target action set [ , loop to get the target action set of all fire-fighting drones ,in, , ; Set the target actions of all firefighting drones And the next moment state set of all firefighting drones in the experience set [ ,in, , , input the Critic target network of the i-th fire-fighting drone, and obtain the target network Q value of batchSize experience samples of the i-th fire-fighting drone ,in, j=1,2,…,batchSize , are the parameters of the Critic target network.

[0031] Calculate the target Q value based on the target network Q value and the reward function: , Where, are the parameters of the Critic target network, is the discount factor, Whether the i-th firefighting drone has completed the task at the j-th sample, is the reward value of the i-th firefighting drone.

[0032] It should be noted that when (When the task is not completed), ,in, is the immediate reward obtained by the current action, It is the future discounted reward corresponding to the state and action at the next moment (estimated by the target network Q value), reflecting the consideration of long-term returns.

[0033] when (When the task is completed), , at this time, only focus on the current immediate reward and no longer consider future rewards.

[0034] Calculate the Critic network loss based on the sample main network Q value and the target Q value: ; Where, is the Q value of the sample main network.

[0035] Backpropagation is performed based on the critic network loss to calculate the critic network gradient and update the critic main network parameters of the fire-fighting drone; the critic main network parameter update of all fire-fighting drones is completed in a cycle.

[0036] The individual state set of the i-th firefighting drone in the experience set [ Input Actor main network and get action set [ , loop to get the action set of all fire-fighting drones ,in, ; Update the action set of the i-th firefighting drone to , the individual state set of all firefighting drones in the combined experience set [ ,in, , input the Critic main network of the i-th firefighting drone, and get the Q value of the strategy main network ,in, j=1, 2,…,batchSize , are the parameters of the Critic main network.

[0037] It should be noted that the Q value of the sample main network The Q value of the main strategy network is obtained by inputting the state set and action set of the fire-fighting drone sampled from the experience set. It is obtained by inputting the state set of the fire-fighting drones sampled in the experience set and the updated action set of the i-th fire-fighting drone.

[0038] Calculate the Actor network gradient based on the Q value of the strategy main network:

[0039] Update the Actor main network through the gradient ascent algorithm.

[0040] Perform soft updates on the Actor target network and Critic target network at fixed intervals ; Where, and They are the parameters of the Critic main network and the target network, and They are the Actor main network and target network parameters, is a smaller coefficient ( <1, typical value is 0.01).

[0041] During each soft update, the target network parameters partially absorb the main network parameters and retain most of the original values, so that the target network parameters slowly and smoothly approach the main network, avoiding drastic fluctuations in the target values and ensuring training stability and convergence.

[0042] Repeat the above steps until the maximum number of training times is reached and save the model with the largest average reward per step during training to obtain the trained drone group encirclement task decision network model based on the MADDPG algorithm.

[0043] Preferably, when inputting the state into the Actor main network and generating actions, add noise to the actions and restrict the actions: ,

[0044] Where, is a random vector uniformly distributed in the interval [0, 1); is a constant value, representing the minimum value of noise; is a constant value, representing the maximum value of the noise; is the attenuation rate, is the global training step size.

[0045] The beneficial effects of the present invention are as follows: using firefighting drones as child nodes and a navigator drone as the root node, a generative forest topology network is constructed by establishing a communication matrix LR between the firefighting drones and a communication matrix LT between the firefighting drones and the navigator drone. When some communication links or nodes fail, other child nodes can still maintain information transmission, effectively addressing the problem of communication interruption easily encountered by fully connected topologies in complex environments, maintaining the coherence of collaborative decision-making and the robustness of the system, and avoiding collaborative failures caused by communication interruptions. By combining the generative forest topology with a reward function, the model can leverage local information to achieve collaboration when communication is limited, preventing the model from falling into a local optimum or failing to converge. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A diagram showing the steps of the method of the present invention.

[0047] Figure 2 This is the Actor and Critic network structure diagram of the present invention.

[0048] Figure 3 This is a schematic diagram of the coordinated fire encirclement of the drone group according to Example 2 of the present invention. DETAILED DESCRIPTION

[0049] In order to clearly illustrate the technical features of this solution, the solution is described below through specific implementation methods.

[0050] Example 1: See also Figure 1 、 Figure 2 As shown, this embodiment is a UAV group fire cooperative encirclement control method based on the MADDPG algorithm, comprising the following steps: S1. Initialize the drone group and fire extinguishing area, and build a forest topology network.

[0051] The drone group includes at least one navigator drone and several fire-fighting drones, and each fire-fighting drone is equipped with several distance sensors.

[0052] Initialize the number m and position of the leader drone , the number of fire-fighting drones n, location ,speed , acceleration , maximum speed , maximum acceleration , distance sensor parameters , the maximum detection range L of the distance sensor and the length Length and width Width of the fire extinguishing area, set the obstacle to a circle with a fixed radius Radius, and randomly initialize the position Obs of the obstacle.

[0053] The firefighting drone location set is: ,in, , represents the position of the i-th firefighting drone at time t.

[0054] The motion model of the firefighting drone is: ,in, is the speed of the i-th firefighting drone at time t, is the speed of the i-th firefighting drone in the x direction at time t-1, is the speed of the i-th firefighting drone in the y direction at time t-1, is the acceleration of the i-th firefighting drone in the x direction at time t, is the acceleration of the i-th firefighting drone in the y direction at time t.

[0055] The set of pilot drone positions is: ,in, , , represents the position of the i-th leader UAV at time t.

[0056] The distance between any two firefighting drones, the distance between any firefighting drone and any pilot drone, and the distance between any two pilot drones are all greater than 0.

[0057] The firefighting drone is used as a child node of the generated forest topology network, and the navigator drone is used as the root node of the generated forest topology network. The generated forest topology network is constructed by establishing a communication matrix LR between the firefighting drones and a communication matrix LT between the firefighting drones and the navigator drone.

[0058] It should be noted that LR is an n×n matrix (n is the number of firefighting drones). , indicating that the i-th firefighting drone can receive the message from the j-th firefighting drone; if , indicating that the i-th firefighting drone cannot receive the message from the j-th firefighting drone.

[0059] LT is an m×n matrix (m is the number of pilots), if , indicating that the i-th firefighting drone can receive the message from the k-th navigator drone; if , indicating that the i-th firefighting drone cannot receive the message from the k-th firefighting drone.

[0060] The navigator drone serves as the root of the forest topology network and is responsible for global decision-making (such as surrounding the fire-fighting area). The fire-fighting drone serves as a child node of the forest topology network and is responsible for distance environment perception and communication with the navigator and other drones.

[0061] S2. Set the state space and reward function of the drone group.

[0062] The state space S of the drone group includes several states , the state Including the individual states of several fire-fighting drones, represented as ;The individual state of the i-th firefighting drone is ,in, is the motion state of the i-th firefighting drone, is the distance between the i-th firefighting drone and the obstacle, is the communication and position status of the i-th firefighting drone, , , , Indicates whether the i-th firefighting drone can receive the message from the j-th firefighting drone. Indicates whether the i-th firefighting drone can receive the message from the k-th leader drone.

[0063] The reward function of the i-th firefighting drone is defined as , ; in, Represents the tracking reward function, which is calculated based on the distance between the firefighting drone and the target: , Where, Represents the tracking reward coefficient, represents the distance between the i-th firefighting drone and the target at time t;

[0064] It should be noted that the communication matrix (communication between firefighting drones) and (Firefighting drone communicates with the navigator), only the position difference of the communicable nodes is aggregated ( or The local communication characteristics of the forest topology are matched to avoid using invalid information of non-communicable nodes, thus improving computational efficiency and decision rationality.

[0065] Close Range Rewards encourages drones to stay within the effective range near the target, improving firefighting efficiency. , Where, Represents the close range maintenance bonus coefficient; Represents the direction alignment bonus, which rewards the drone for keeping the target direction consistent, enhancing the encirclement effect: , Where, Represents the direction alignment reward coefficient; Represents the energy efficiency penalty reward, which penalizes violent movements that exceed the action amplitude threshold to reduce energy consumption: , Where, is the action vector of the i-th firefighting drone at time t, represents the energy efficiency penalty reward coefficient, represents the action amplitude threshold; Represents the success reward for a single firefighting drone. When a single firefighting drone meets the success conditions, the firefighting drone receives a positive reward: , Where, Represents the success reward value of a single firefighting drone, represents the success distance threshold; Represents the success reward for all firefighting drones. When all drones meet the success conditions, additional global positive rewards will be given: , Where, Represents the success reward value of all fire-fighting drones; Represents collision penalty reward: , Where, Represents the collision penalty reward value; Represents the safety distance penalty, which gives negative rewards when the distance between drones or obstacles is less than the safety threshold: .

[0066] It should be noted that Is the collision of the i-th firefighting drone?

[0067] Where, The collision safety distance.

[0068] Whether the i-th firefighting drone has completed its mission,

[0069] Where, is the current round step length, is the maximum step length of the round.

[0070] S3. Build and train a drone group encirclement task decision network model based on the MADDPG algorithm.

[0071] For each firefighting drone, an actor network and a critic network are constructed, and the actor network parameters and the critic network parameters are randomly initialized. The actor network and the critic network both include a main network and a target network. The main network and the target network parameters of the actor network are the same, and the main network and the target network parameters of the critic network are the same. It should be noted that the Actor network is responsible for generating actions. It consists of a main network and a target network. The main network directly outputs the current action, while the target network is used to generate future actions. Through soft parameter updates, the target network parameters slowly follow the main network to avoid training divergence.

[0072] The Critic network evaluates the value of actions generated by the Actor network (e.g., whether the action brings the drone closer to its target and whether it is safe). It also includes a main network and a target network. The main network evaluates the value of the current action, while the target network assists in calculating future long-term rewards and stabilizing the training process.

[0073] Through "centralized training", global information is used to optimize the network and improve the efficiency of strategy learning; through "distributed decision-making", drones are given autonomous execution capabilities and can adapt to complex scenarios; through the "master-target network architecture" (including both actors and critics), the training process is stabilized to avoid learning failures caused by drastic fluctuations in parameters.

[0074] For the i-th firefighting drone, input the status to the Actor main network , generate actions .

[0075] Input the state to the Actor main network, and when generating actions, add noise to the actions and restrict the actions: ,

[0076] Where, is a random vector uniformly distributed in the interval [0, 1); is a constant value, representing the minimum value of noise; is a constant value, representing the maximum value of the noise; is the attenuation rate, is the global training step size.

[0077] According to the action Interact with the environment to gain rewards , the state at the next moment and whether the task status has been completed , loop each fire-fighting drone and set the status of all fire-fighting drones ,action , reward value , next moment status and whether the task status has been completed As an empirical sample Put it into the experience storage pool, and continue to interact with the environment to continue to obtain experience samples and store them in the experience storage pool; Sampling batchSize experience samples from the experience storage pool to form an experience set, and the next moment state set of the i-th firefighting drone in the experience set [ Input the Actor target network of the i-th firefighting drone and obtain the target action set [ , loop to get the target action set of all fire-fighting drones ,That , ; Set the target actions of all firefighting drones And the next moment state set of all firefighting drones in the experience set [ ,in, , , input the Critic target network of the i-th fire-fighting drone, and obtain the target network Q value of batchSize experience samples of the i-th fire-fighting drone .

[0078] Calculate the target value based on the target network Q value and reward function: , Where, are the parameters of the Critic target network, is the discount factor, Whether the i-th firefighting drone has completed the task at the j-th sample, is the reward value of the i-th firefighting drone.

[0079] It should be noted that when (When the task is not completed), ,in, is the immediate reward obtained by the current action, It is the future discounted reward corresponding to the state and action at the next moment (estimated by the target network Q value), reflecting the consideration of long-term returns.

[0080] when (When the task is completed), , at this time, only focus on the current immediate reward and no longer consider future rewards.

[0081] Calculate the Critic network loss based on the sample main network Q value and target value: ; Where, is the Q value of the sample main network.

[0082] Backpropagation is performed based on the critic network loss to calculate the critic network gradient and update the critic main network parameters of the fire-fighting drone; the critic main network parameter updates of all fire-fighting drones are completed in a loop; The individual state set of the i-th firefighting drone in the experience set [ Input Actor main network and get action set [ , loop to get the action set of all fire-fighting drones ,in, ; Update the action set of the i-th firefighting drone to , the individual state set of all firefighting drones in the joint experience set [ ,in, , input the Critic main network of the i-th firefighting drone, and get the Q value of the strategy main network ,in, j=1, 2,…,batchSize , are the parameters of the Critic main network.

[0083] It should be noted that the Q value of the sample main network The Q value of the main strategy network is obtained by inputting the state set and action set of the fire-fighting drone sampled from the experience set. It is obtained by inputting the state set of the fire-fighting drones sampled in the experience set and the updated action set of the i-th fire-fighting drone.

[0084] Calculate the Actor network gradient based on the main network Q value:

[0085] Update the Actor main network through the gradient ascent algorithm.

[0086] Perform soft updates on the Actor target network and Critic target network at fixed intervals ; Where, and They are the parameters of the Critic main network and the target network, and They are the Actor main network and target network parameters, is a smaller coefficient ( <1, typical value is 0.01).

[0087] During each soft update, the target network parameters partially absorb the main network parameters and retain most of the original values, so that the target network parameters slowly and smoothly approach the main network, avoiding drastic fluctuations in the target values and ensuring training stability and convergence.

[0088] Repeat the above steps until the maximum number of training times is reached and save the model with the largest average reward per step during training to obtain the trained drone group encirclement task decision network model based on the MADDPG algorithm.

[0089] S4. Use the trained drone group encirclement task decision network model based on the MADDPG algorithm to realize the drone group's encirclement task in the fire-fighting scenario.

[0090] Example 2: To illustrate the effectiveness of the proposed drone group fire collaborative encirclement control method based on the MADDPG algorithm, this embodiment uses six firefighting drones and four navigator drones to construct a forest topology for fire encirclement simulation.

[0091] In this embodiment, the firefighting drone actor network and critic network are specifically defined as follows: In the Actor network, the input state dimension is 18, there are two fully connected hidden layers (128 dimensions), LeakyReLU activation, the output layer dimension is 2, Softsign activation, and the output action is processed by L2 norm judgment. When the modulus of the action vector is greater than 1, it is normalized to a unit vector (divided by the modulus). When the modulus is less than or equal to 1, the original value is kept unchanged (multiplied by 1). Finally, the processed action vector is scaled to the range of [-a_max, a_max]. In the Critic network, the input dimension is 120, there are two fully connected hidden layers (128 dimensions), ReLU activation, and the output layer dimension is 1.

[0092] The model training parameters, drone group parameters, and fire area parameters are as follows: The Actor network uses the Adam optimizer with a learning rate of alpha = 0.0001, and the Critic network uses the Adam optimizer with a learning rate of beta = 0.001. The total number of training rounds is 10,000, the maximum step size of the round is 200, the learning interval is 10 steps, and the preservation condition is that the average reward per step is improved. =0.9, =0.01, the experience pool size is 100000, the batch size is 1024, and the round termination condition is: reaching the maximum number of steps or all firefighting drones complete the task. During training, the length of the scene is Length=3, the width is Width=3, the position of the navigator is randomly generated in the range of 0.5-2.5, and the position of the firefighting drone is ,The obstacle position is randomly generated with a radius of 0.15, and it must be ensured that the firefighting drone and the navigator are not covered; , , maximum speed , maximum acceleration , update duration is , Maximum detection range of the sensor , , .

[0093] When using the trained Actor network model to perform the multi-UAV encirclement mission in a firefighting scenario, the position of the firefighting UAV is , the position of the navigator is , the obstacle position is Finally, the drone group fire collaborative encirclement effect is as follows Figure 3 As shown, In the figure, the larger circular area is an obstacle, the solid line is the flight path of the fire-fighting drone, the solid line arrow is the flight endpoint of the fire-fighting drone, the smaller circular area is the location of the pilot drone, and the area surrounded by the dotted line is the fire area. This embodiment uses the pilot drone as the root node and the fire-fighting drones as child nodes to establish a topological connection through the communication matrix LR (communication between fire-fighting drones) and LT (communication between fire-fighting drones and pilots). Even if some communication links or nodes fail ( =0 or =0), other child nodes can still maintain information transmission, ensuring that the drone group avoids obstacles, adjusts the flight path, and effectively surrounds the fire area, which improves the robustness of the model and avoids coordination failure caused by communication interruption in complex environments.

[0094] Technical features not described in the present invention can be achieved through or by adopting existing technologies and will not be described in detail here. Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. A UAV group fire cooperative encirclement control method based on MADDPG algorithm, characterized by: The following steps are involved: Initialize the drone group and fire extinguishing area, and build a forest topology network; Set up the state space and reward function for the drone group; Build and train a drone group encirclement task decision network model based on the MADDPG algorithm; The trained drone group encirclement task decision network model based on the MADDPG algorithm is used to realize the encirclement task of the drone group in the fire-fighting scenario.

2. The UAV group fire cooperative surrounding control method based on the MADDPG algorithm according to claim 1 is characterized in that: The drone group includes at least one navigator drone and several fire-fighting drones, and each of the fire-fighting drones is equipped with several distance sensors.

3. The UAV group fire cooperative surrounding control method based on the MADDPG algorithm according to claim 2 is characterized in that: Initializing the drone group and the fire extinguishing area and establishing the generated forest topology includes: Initialize the number of pilot drones and location , the number of firefighting drones ,Location ,speed , acceleration , maximum speed , maximum acceleration , distance sensor parameters , Maximum detection range of distance sensor and the length of the fire extinguishing area ,width , set the obstacle to be circular, with a radius is a fixed value, and the position of the obstacle is randomly initialized ; The firefighting drone is used as a child node of the generated forest topology network, and the navigator drone is used as the root node of the generated forest topology network. The generated forest topology network is constructed by establishing a communication matrix LR between the firefighting drones and a communication matrix LT between the firefighting drones and the navigator drone.

4. The UAV group fire cooperative surrounding control method based on the MADDPG algorithm according to claim 3 is characterized in that: The distance between any two of the fire-fighting drones, the distance between any one of the fire-fighting drones and any one of the pilot drones, and the distance between any two of the pilot drones are all greater than 0.

5. The UAV group fire cooperative surrounding control method based on the MADDPG algorithm according to claim 4 is characterized in that: The state space S of the drone group includes several states , the state Including the individual states of several fire-fighting drones, represented as ;The individual state of the i-th firefighting drone is ,in, is the motion state of the i-th firefighting drone, is the distance between the i-th firefighting drone and the obstacle, is the communication and position status of the i-th firefighting drone, , , , Indicates whether the i-th firefighting drone can receive the message from the j-th firefighting drone. Indicates whether the i-th firefighting drone can receive the message from the k-th leader drone.

6. The UAV group fire cooperative surrounding control method based on the MADDPG algorithm according to claim 5 is characterized in that: The reward function of the i-th firefighting drone is defined as , ; in, Represents the tracking reward function: , Where, Represents the tracking reward coefficient, represents the distance between the i-th firefighting drone and the target at time t; Represents close range maintenance bonus: , Where, Represents the close range maintenance bonus coefficient; Represents direction alignment reward: ， Where, Represents the direction alignment reward coefficient; Represents energy efficiency penalty reward: ， Where, is the action vector of the i-th firefighting drone at time t, represents the energy efficiency penalty reward coefficient, represents the action amplitude threshold; Rewards for a single firefighting drone: , Where, Represents the success reward value of a single firefighting drone, represents the success distance threshold; Rewards for all successful firefighting drones: , Where, Represents the success reward value of all fire-fighting drones; Represents collision penalty reward: , Where, Represents the collision penalty reward value; Representative safety distance penalty: , Where, Represents the safety distance penalty coefficient, represents the safety threshold ratio, is the distance between the i-th firefighting drone and the j-th firefighting drone.

7. The UAV group fire cooperative surrounding control method based on the MADDPG algorithm according to claim 6 is characterized in that: The construction and training of the drone group encirclement task decision network model based on the MADDPG algorithm includes: For each firefighting drone, an actor network and a critic network are constructed, and the actor network parameters and the critic network parameters are randomly initialized. The actor network and the critic network both include a main network and a target network. The main network and the target network parameters of the actor network are the same, and the main network and the target network parameters of the critic network are the same. For the i-th firefighting drone, input the individual state to the Actor main network , generate actions , according to the action Interact with the environment to gain rewards , the state at the next moment and whether the task status has been completed , loop each fire-fighting drone and set the individual space of all fire-fighting drones ,action , reward value , next moment status and whether the task status has been completed As an empirical sample Put it into the experience storage pool, and continue to interact with the environment to continue to obtain experience samples and store them in the experience storage pool; Sample batchSize experience samples from the experience storage pool to form an experience set ,in, j=1, 2,…,batchSize , the next moment state set of the i-th firefighting drone in the experience concentration [ Input the Actor target network of the i-th firefighting drone and obtain the target action set [ , loop to get the target action set of all fire-fighting drones ,in, , ; Set the target actions of all firefighting drones And the next moment state set of all firefighting drones in the experience set [ ,in, , , input the Critic target network of the i-th fire-fighting drone, and obtain the target network Q value of batchSize experience samples of the i-th fire-fighting drone; The target Q value is calculated based on the target network Q value and the reward function. The critic network loss is calculated based on the sample main network Q value and the target Q value. The critic network gradient is calculated based on the critic network loss through backpropagation, and the critic main network parameters of the fire-fighting drone are updated. The update of the critic main network parameters of all fire-fighting drones is completed in a loop. The individual state set of the i-th firefighting drone in the experience set [ Input Actor main network and get action set [ , loop to get the action set of all fire-fighting drones ,in, ; Update the action set of the i-th firefighting drone to , the individual state set of all firefighting drones in the joint experience set [ ,in, , input the Critic main network of the i-th firefighting drone, obtain the Q value of the policy main network, calculate the Actor network gradient based on the Q value of the policy main network, and update the Actor main network through the gradient ascent algorithm; the cycle completes the update of the Actor main network parameters of all firefighting drones; Perform soft updates on the Actor target network and Critic target network at fixed intervals ； Where, and They are the parameters of the Critic main network and the target network, and They are the Actor main network and target network parameters, is a smaller coefficient; Repeat the above steps until the maximum number of training times is reached and save the model with the largest average reward per step during training to obtain the trained drone group encirclement task decision network model based on the MADDPG algorithm.

8. The UAV group fire cooperative surrounding control method based on the MADDPG algorithm according to claim 7 is characterized in that: Input the state to the Actor main network, and when generating actions, add noise to the actions and restrict the actions: ， Where, is a random vector uniformly distributed in the interval [0, 1); is a constant value, representing the minimum value of noise; is a constant value, representing the maximum value of the noise; is the attenuation rate, is the global training step size.

Images