Unmanned aerial vehicle cluster dynamic area coverage method based on deep reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle cluster dynamic area coverage method based on deep reinforcement learning. The method comprises the following steps: constructing a multi-physical field coupling simulation environment comprising node motion, a directional antenna, a space channel and a radar detection model; based on the environment, converting a dynamic coverage problem form into a partially observable Markov decision process, a defined state, observation, action and reward mechanism, performing multi-agent training by adopting a centralized training decentralized execution framework, evaluating a joint action by using a global Critic network during training, and storing interactive experience by using an experience playback buffer area; based on a priority experience playback mechanism, according to a time sequence difference error and a reward value, an experience sampling probability is calculated, samples are extracted, Critic and Actor networks are updated in sequence, a decision model is deployed to an unmanned aerial vehicle cluster after training is completed, and each unmanned aerial vehicle executes an area coverage task in a distributed manner, so that efficient and low-risk communication coverage can be realized in a complex threat environment. The communication quality is improved; and the detected probability is reduced.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) swarm and ad hoc network technology, and more specifically, to a dynamic region coverage method for UAV swarms based on deep reinforcement learning. Background Technology

[0002] As a flexible aerial platform, unmanned aerial vehicles (UAVs) possess significant application potential in numerous fields requiring wide-area monitoring and communication support due to their high mobility, ease of deployment, and low operating costs. Especially in scenarios with complex geographical environments or damaged or disrupted ground infrastructure, UAVs can quickly form networks, acting as aerial communication relay nodes to provide temporary and reliable network coverage for dispersed remote users or devices, thereby effectively restoring and maintaining communication connectivity in critical areas. Therefore, researching how UAV swarms can achieve efficient and stable regional communication coverage has become a valuable technological direction.

[0003] To achieve regional coverage, the core lies in optimizing the deployment location and trajectory of the drone swarm. Traditional methods often employ heuristic algorithms such as ant colony optimization and particle swarm optimization for single-drone trajectory planning. However, these methods typically struggle to fully consider the mutual influence and collaborative constraints between individual drone states, thus limiting their ability to optimize the global coverage performance of the entire drone network and maintain its connectivity. In recent years, deep reinforcement learning methods have been introduced into this field, providing a new approach to dynamic coverage problems through continuous interaction between agents and the environment to learn optimal strategies. However, existing research largely focuses on coverage and network connectivity metrics in static environments, neglecting the complex and dynamic external environmental factors in practical applications.

[0004] In complex and dynamic real-world application scenarios, the operating environment of UAV networks is neither ideal nor constant. For example, in areas with various external interferences or physical obstacles, UAVs need to possess the ability to perceive and respond dynamically in real time. If only a fixed coverage strategy is adopted, the network may experience a sharp decline in performance or even failure due to its inability to adapt to environmental changes. Therefore, how to design a collaborative trajectory optimization strategy for UAV swarms that can adapt to dynamic environmental changes, ensure overall communication coverage quality, and maintain overall network connectivity and long-term stable operation is a key technical problem that urgently needs to be solved. Summary of the Invention

[0005] The purpose of this application is to overcome the shortcomings of existing technologies and provide a dynamic area coverage method for UAV swarms based on deep reinforcement learning, which can improve the information transmission rate during the communication coverage process and reduce the probability of joint detection by enemy radar.

[0006] The objective of this application is achieved through the following technical solution: Firstly, this application proposes a dynamic region coverage method for UAV swarms based on deep reinforcement learning, the method comprising: Step S1: Construct a simulation environment model with multi-physics coupling. The simulation environment model includes a node motion model, a directional antenna model, a space channel model, and an enemy radar detection model. Step S2: Based on the simulation environment model, the dynamic area coverage problem of UAV swarm is transformed into a partially observable Markov decision process, and the joint state space, joint observation space, joint action space and reward function of the agents are defined. Step S3: For partially observable Markov decision processes, a centralized training-decentralized execution framework is adopted for multi-agent training. During the training phase, the Actor network configured for each UAV agent is used for policy execution, the global Critic network is used to evaluate the joint actions of all agents, and the experience data generated by the interaction with the environment is stored through the experience replay buffer. Step S4: During the training process, when the number of samples in the experience replay buffer reaches the preset threshold, samples are extracted for network updates based on the priority experience replay mechanism. The priority experience replay mechanism calculates the probability of being sampled based on the temporal difference error of each experience and the reward value obtained by the agent, and uses the extracted samples to update the parameters of the Critic network and the Actor network in turn. Step S5: After training is completed, the trained decision model is deployed to the physical entities of the drone cluster, and each drone performs dynamic area coverage tasks in a distributed manner according to the decision results of the model.

[0007] In one possible implementation, the node motion model is used to model the movement process of ground mobile nodes, wherein the ground mobile nodes are randomly distributed in the target area at the start of the task and update their positions according to preset motion rules at each time step. The directional antenna model includes the antenna gain model for communication between UAVs or between a UAV and a ground mobile node. The antenna gain model is modeled as a nonlinear function of the beam incidence angle and the directivity factor. The space channel model includes air-to-ground channels between UAVs and ground mobile nodes, and air-to-air channels between UAVs. The air-to-ground channel modeling includes small-scale channel fading following a Rician distribution. The enemy radar detection model uses a probabilistic model to simulate the process of enemy radar detecting electromagnetic signals emitted by UAVs. The probabilistic model is constructed based on the radar's false alarm probability, carrier frequency, distance between the UAV and the radar, and the deflection angle parameters of the directional antenna.

[0008] In one possible implementation, the joint state space contains global information on the positions of all UAVs, antenna deflection angles, and the positions of all ground vehicles. The joint observation space includes information from each agent's own observation range and the observation information from neighboring nodes; The joint action space includes the action components in the x-axis direction, the action components in the y-axis direction, and the horizontal and vertical deflection angles of the directional antenna.

[0009] In one possible implementation, the reward function is a multi-objective composite reward function, expressed as: ; in , and It is a constant greater than 0. The reward is the distance between the agent and the ground mobile node. The penalty for the interception of electromagnetic signals emitted by an intelligent agent. Rewards for connectivity in drone networks, The status of establishing a communication / data link between the drone and ground targets. The signal power received by the drone from the target node. The preset power reception threshold, The probability of being detected by enemy radar. To establish a safe detection probability threshold. For key node indicators.

[0010] In one possible implementation, the centralized training-distributed execution framework randomly initializes the Critic network, the Actor network for each agent, the corresponding target network, and the experience replay buffer before training begins. At each time step, the agent selects an action based on local observations using a greedy strategy or the Actor network output, and stores the empirical data in the experience replay buffer.

[0011] In one possible implementation, the formula for calculating the probability of each experience being sampled by the priority experience replay mechanism is as follows: , and This is the proportionality coefficient. This represents the total number of samples in the experience replay buffer at the current moment. This indicates the degree to which the priority experience replay mechanism is used to control the frequency of use. and These represent the combined calculation of experience using TD error and reward value, respectively. value, and These represent the normalization of TD error value and reward value, respectively.

[0012] In one possible implementation, the step of sequentially updating the parameters of the Critic network and the Actor network using the extracted samples includes: The parameters are optimized by minimizing the loss function, and the parameters of the Actor network and the target network are updated after the Critic network is updated a preset number of times. The training process is repeated at each time step until the entire scene or all training is completed.

[0013] In one possible implementation, the steps following the deployment of the trained decision model to the physical entities of the drone swarm include: The drone swarm performs area coverage tasks in a distributed manner on physical entities, dynamically adjusting its position and antenna deflection angle to optimize communication coverage.

[0014] The main solution and its various further alternatives described above can be freely combined to form multiple solutions, all of which are solutions that can be adopted and are claimed in this application; furthermore, the (non-conflicting alternatives) can also be freely combined with each other and with other alternatives. Those skilled in the art, after understanding the solution of this application, will realize from the prior art and common general knowledge that there are many combinations, all of which are technical solutions to be protected in this application, and will not be exhaustively listed here.

[0015] This application discloses a dynamic region coverage method for UAV swarms based on deep reinforcement learning. It constructs a multi-physics coupled simulation environment that includes node motion, directional antennas, spatial channels, and radar detection models. Based on this environment, the dynamic coverage problem is transformed into a partially observable Markov decision process, defining state, observation, action, and reward mechanisms. Centralized training is employed. A distributed execution framework is used for multi-agent training. During training, a global Critic network is used to evaluate joint actions, and an experience replay buffer is used to store interaction experiences. Based on a priority experience replay mechanism, the experience sampling probability is calculated according to the temporal difference error and reward value. Samples are extracted and the Critic and Actor networks are updated sequentially. After training, the decision model is deployed to a drone swarm. Each drone performs distributed regional coverage tasks, which can achieve efficient and low-risk communication coverage in complex threat environments, improve communication quality, and reduce the probability of being detected. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 The diagram shows a flowchart of a dynamic region coverage method for UAV swarms based on deep reinforcement learning proposed in an embodiment of this application.

[0018] Figure 2 A schematic diagram of a scenario illustrating the dynamic area coverage method for drone swarms proposed in this application is shown.

[0019] Figure 3 A block diagram of a multi-agent deep reinforcement learning system based on a "centralized training-distributed execution" architecture, as proposed in an embodiment of this application, is shown.

[0020] Figure 4 This diagram illustrates a comparison of the information transmission rates between the proposed UAV swarm dynamic region coverage method and other UAV swarm dynamic region coverage methods based on deep reinforcement learning.

[0021] Figure 5 This is a schematic diagram comparing the joint interception probability of the UAV swarm dynamic region coverage method proposed in this application with other UAV swarm dynamic region coverage methods based on deep reinforcement learning. Detailed Implementation

[0022] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0023] Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] To address the communication coverage challenges and detection risks posed by enemy radar threats, this application proposes a dynamic area coverage method for UAV swarms based on deep reinforcement learning. By constructing a multi-physics coupled simulation environment, the dynamic coverage problem of UAV swarms is formalized as a partially observable Markov decision process. A multi-agent deep reinforcement learning technique based on a centralized training-distributed execution framework is employed, combined with a priority experience replay mechanism that integrates temporal differential errors and reward values, and a multi-target composite reward function that guides cooperation and concealment. Ultimately, this enables the UAV swarm to dynamically adjust its position and antenna deflection angle in a distributed manner. This significantly improves the communication coverage quality and information transmission rate for ground mobile nodes under complex enemy radar threats, while minimizing the joint probability of detection by enemy radar. This effectively solves the problem of secure and efficient area coverage while ensuring network connectivity.

[0025] Please refer to Figure 1 , Figure 1 The diagram illustrates a flowchart of a dynamic region coverage method for UAV swarms based on deep reinforcement learning, as proposed in an embodiment of this application. Figure 2 This illustration shows a scenario diagram of the UAV swarm dynamic area coverage method proposed in this application. In a complex battlefield environment with enemy radar threats, the UAV swarm provides continuous, stable, and secure communication coverage for a mobile ground convoy. The core objective is to minimize the risk of detection and destruction by enemy radar while ensuring communication network connectivity, thereby improving the survivability and collaborative efficiency of the entire system. Multiple UAVs fly over the target area, forming a dynamic aerial communication network through their communication links, jointly providing communication relay services to ground mobile nodes below. The UAV swarm intelligently and dynamically adjusts its position and flight trajectory to ensure that its communication network effectively covers friendly ground units, while the entire swarm maintains a safe distance from the danger zone of enemy radar, achieving effective avoidance. The method includes: Step S1: Construct a simulation environment model with multi-physics coupling. The simulation environment model includes a node motion model, a directional antenna model, a space channel model, and an enemy radar detection model. The node motion model is used to model the movement process of ground mobile nodes, where ground mobile nodes are randomly distributed in the target area at the start of the task and update their positions according to preset motion rules at each time step; The directional antenna model includes the antenna gain model for communication between UAVs or between a UAV and a ground mobile node. The antenna gain model is modeled as a nonlinear function of the beam incidence angle and the directivity factor. The space channel model includes air-to-ground channels between UAVs and ground mobile nodes, and air-to-air channels between UAVs. The air-to-ground channel modeling includes small-scale channel fading following a Rician distribution. The enemy radar detection model uses a probabilistic model to simulate the process of enemy radar detecting electromagnetic signals emitted by UAVs. The probabilistic model is constructed based on the radar's false alarm probability, carrier frequency, distance between the UAV and the radar, and the deflection angle parameters of the directional antenna.

[0026] First, a highly realistic multiphysics coupled simulation environment model was constructed, which integrates the node motion model, directional antenna model, space channel model, and enemy radar detection model: The nodal motion model is responsible for modeling the maneuvering process of ground mobile nodes, which can be, for example, ground vehicles, mobile robots, or other mobile platforms. At the start of the mission, all ground mobile nodes are randomly distributed within the target area. At each simulation time step, these nodes update their positions according to preset motion rules, thereby simulating the dynamics of the ground target. The directional antenna model characterizes the properties of directional antennas used for communication between UAVs and between UAVs and ground mobile nodes. This model expresses the antenna gain as a nonlinear function of the beam incidence angle and the antenna's directivity factor. This simulates the main characteristics of a real directional antenna: the gain is highest when the antenna beam is precisely aligned with the target, and drops sharply as the pointing deviation increases, directly affecting the establishment and quality of the communication link. The spatial channel model is used to simulate the propagation characteristics of electromagnetic signals in complex spatial environments. It specifically includes two types of channels: one is the air-to-ground channel between UAVs and ground mobile nodes, and the other is the air-to-ground channel between UAVs. This model specifically considers that ground mobile nodes may receive signals superimposed from direct paths and multiple scattering paths. Therefore, it introduces small-scale channel fading following a Rician distribution to more realistically reflect the rapid fluctuations in signal strength in urban or complex terrain environments. The enemy radar detection model aims to assess the operational risks of drone swarms. This model uses a probabilistic approach to simulate the process by which enemy radar intercepts and identifies communication or detection signals emitted by drones. The detection probability is calculated based on several key parameters, including the radar's false alarm probability, the carrier frequency of the communication signal, the distance between the drone and the enemy radar, and the deflection angle of the drone's directional antenna. By integrating the above four sub-models, the constructed simulation environment can simultaneously characterize the target's motion, the constraints of directional communication, the random fading of the channel, and the enemy's detection threat, forming a complex dynamic system with multidimensional coupling.

[0027] Step S2: Based on the simulation environment model, the dynamic area coverage problem of UAV swarm is transformed into a partially observable Markov decision process, and the joint state space, joint observation space, joint action space and reward function of the agents are defined.

[0028] To transform the complex dynamic coverage problem of drone swarms into a mathematical problem that can be optimized using reinforcement learning, the entire task is formalized as a partially observable Markov decision process. Although the entire system possesses a real, constantly evolving global state, individual drones cannot directly know all of this; they can only make decisions based on their own limited, local observations. This closely aligns with the actual working mode of distributed drones.

[0029] The joint state space contains global information on the positions of all UAVs, antenna deflection angles, and the positions of all ground vehicles; The joint observation space includes information from each agent's own observation range and the observation information from neighboring nodes; The joint action space includes the action components in the x-axis direction, the action components in the y-axis direction, and the horizontal and vertical deflection angles of the directional antenna.

[0030] The POMDP framework defines four core elements: The joint state space defines the global truth of the problem. This space contains the positions and antenna deflection angles of all UAVs, as well as the positions of all ground vehicles. This is a complete set of information for evaluating the overall progress of the mission, but in actual distributed execution, no single agent can directly access it.

[0031] The joint observation space defines the local world that each individual UAV can perceive. Each agent's observation information consists of two parts: information within the observation range of its own sensors; and observation information shared from neighboring nodes via communication links. This simulates the actual situation where UAVs extend their perception capabilities through inter-Agent communication.

[0032] The joint action space defines the decisions each UAV can make at each step. Each agent's action is modeled as a four-dimensional vector, including: an action component along the x-axis and y-axis in a two-dimensional plane, and the horizontal and vertical deflection angles of the directional antenna used to adjust the communication beam direction. By controlling these four variables, the UAV can simultaneously achieve optimized alignment of maneuvers and communication links.

[0033] The reward function is a multi-objective composite reward function, expressed as: ; in , and It is a constant greater than 0. The reward is the distance between the agent and the ground mobile node. The penalty for the interception of electromagnetic signals emitted by an intelligent agent. Rewards for connectivity in drone networks, The status of establishing a communication / data link between the drone and ground targets. The signal power received by the drone from the target node. The preset power reception threshold, The probability of being detected by enemy radar. To establish a safe detection probability threshold. For key node indicators.

[0034] This function intelligently coordinates multiple competing objectives—coverage target, survivability and concealment, critical node maintenance, signal security, and network health—through four branches, driving the drone swarm to learn the optimal cooperative strategy under complex constraints.

[0035] Step S3: For partially observable Markov decision processes, a centralized training-distributed execution framework is adopted for multi-agent training. During the training phase, the Actor network configured for each UAV agent is used for policy execution, the global Critic network is used to evaluate the joint actions of all agents, and the experience data generated by the interaction with the environment is stored through the experience replay buffer.

[0036] Figure 3 This diagram illustrates a block diagram of a multi-agent deep reinforcement learning system based on a "centralized training, decentralized execution" architecture proposed in this application. Step S3 employs a Centralized Training, Decentralized Execution (CTDE) framework. Each agent has an independent Actor network for policy execution, while a global Critic network is used during the training phase to evaluate the joint actions of all agents. Before the training process begins, two Critic networks, each agent's Actor network, its corresponding target network, and an experience replay buffer are randomly initialized, and the algorithm is executed according to the set total number of episodes. At the beginning of each episode, the environment is initialized. In each time step, the UAV agent selects actions using a greedy policy based on the set probabilities, or executes actions based on the action values ​​output by its Actor network. After an agent executes an action, it records the action, the state information of the next time step, and the obtained reward value as experience. The Temporal Difference (TD) error corresponding to this experience is calculated, and the experience itself, along with its TD error, is stored in the experience replay buffer.

[0037] The centralized training-distributed execution framework randomly initializes the Critic network, the Actor network for each agent, the corresponding target network, and the experience replay buffer before training begins. At each time step, the agent selects an action based on local observations using a greedy strategy or the Actor network output, and stores the empirical data in the experience replay buffer.

[0038] Before training begins, two global Critic networks, each agent's Actor network, its corresponding target network, and experience replay buffer are randomly initialized, and training is performed according to the set total number of episodes. At the beginning of each episode, the environment is initialized, and at the beginning of each time step, agent i initializes its local observations. Make the corresponding actions and receive environmental feedback reward values. And the observation information at the next time step Then, agent i stores all this information into an experience replay buffer shared by all agents. The Actor network and the two Critic networks all use three fully connected layers. During the interaction, each agent's Actor network randomly selects an action using a greedy strategy based on the local observations at the current time step and a preset probability, or executes an action based on the action value output by its Actor network.

[0039] Step S4: During the training process, when the number of samples in the experience replay buffer reaches the preset threshold, samples are extracted for network updates based on the priority experience replay mechanism. The priority experience replay mechanism calculates the probability of being sampled based on the temporal difference error of each experience and the reward value obtained by the agent, and uses the extracted samples to update the parameters of the Critic network and the Actor network in turn. During training, once the number of samples accumulated in the shared experience replay buffer reaches a preset threshold, the system begins to extract small batches of sample data from the buffer for updating the neural network parameters. An innovative priority experience replay mechanism is employed to determine the probability of each experience being sampled. This mechanism comprehensively considers the learning value of experience, thereby accelerating training and improving the stability of the strategy.

[0040] The formula for calculating the probability of each experience being sampled using the priority experience replay mechanism is as follows: , and Let be the proportionality coefficient, satisfying , This represents the total number of samples in the experience replay buffer at the current moment. This indicates the degree to which the priority experience replay mechanism is used to control the frequency of use. and These represent the combined calculation of experience using TD error and reward value, respectively. value, and These represent the normalization of TD error value and reward value, respectively. and These represent the combined calculation of experience using TD error and reward value, respectively. The value of is expressed as: ,in This represents the reward value obtained by agent i at the current time step. It is a constant greater than 0, ensuring that the denominator is not zero when calculating the sampling probability. For experience TD error.

[0041] The steps for updating the parameters of the Critic network and the Actor network sequentially using the extracted samples include: The parameters are optimized by minimizing the loss function, and the parameters of the Actor network and the target network are updated after the Critic network is updated a preset number of times. The training process is repeated at each time step until the entire scene or all training is completed.

[0042] After sampling, the network update phase begins. The two global Critic networks are updated first. The system calculates the target Q-value for each sample and then updates the parameters of these two Critic networks separately using a method that minimizes the mean squared error loss function. To stabilize training, this scheme adopts advanced algorithms, selecting the smaller value from the outputs of the two Critic networks to calculate the target Q-value, which helps avoid overestimating the value. After the Critic networks are updated a preset number of times, the Actor networks for each agent are updated. The update objective of the Actor networks is to maximize the expected reward evaluated by the Critic networks. Finally, soft updates or periodic hard updates are used to synchronously update the parameters of all target networks, allowing them to slowly track changes in the online network and further improving training stability.

[0043] In steps S3 and S4, the environmental interaction and network update processes are repeated continuously at each time step until the entire mission or training is completed. Through this carefully designed priority experience replay and dual network update mechanism, the collaborative strategy of the UAV swarm can be optimized efficiently and stably, ultimately achieving the predetermined coverage and concealment objectives.

[0044] Step S5: After training is completed, the trained decision model is deployed to the physical entities of the drone cluster, and each drone performs dynamic area coverage tasks in a distributed manner according to the decision results of the model.

[0045] The central server stores and solidifies the parameters of the decision model, which has been fully trained and converged in the simulation environment. This model is then deployed to each physical entity in the UAV swarm. During actual dynamic area coverage missions, each UAV acts as an independent agent, relying solely on local real-time observations acquired by its own sensors and utilizing its onboard computing unit to run the deployed Actor network, instantly calculating the optimal action command. Based on this result, each UAV performs maneuvering and communication tasks completely distributed and autonomously, without relying on real-time commands from a central node or global information sharing. This achieves highly autonomous, highly collaborative, and highly resistant dynamic area coverage capabilities in adversarial and uncertain real-world battlefield environments.

[0046] The steps following the deployment of the trained decision model to the physical entities of the drone swarm include: The drone swarm performs area coverage tasks in a distributed manner on physical entities, dynamically adjusting its position and antenna deflection angle to optimize communication coverage.

[0047] After the trained decision-making model is deployed to the physical entities of the UAV swarm, the system enters the distributed autonomous execution phase: each UAV, based on the local real-time observation information obtained by its own sensors, instantly calculates and autonomously executes the optimal action through the onboard lightweight Actor network, dynamically adjusting its spatial position and antenna beam direction, thereby achieving coordinated communication coverage optimization of the swarm to moving ground targets without the need for real-time command from a central node.

[0048] Figure 4 This diagram illustrates a comparison of the information transmission rates between the proposed UAV swarm dynamic region coverage method and other UAV swarm dynamic region coverage methods based on deep reinforcement learning. Figure 5This diagram illustrates a comparison of the joint interception probability between the proposed UAV swarm dynamic area coverage method and other deep reinforcement learning-based UAV swarm dynamic area coverage methods. Under different ground mobile node scales, the information transmission rate and joint interception probability outperform traditional deep reinforcement learning-based area coverage methods using Multi-Agent Twin Delayed Deep Deterministic Policy Gradients (MATD3) and Multi-Agent Deep Deterministic Policy Gradient (MADDPG). This further verifies the effectiveness of the proposed area coverage method in improving the information transmission rate and reducing the joint detection probability by enemy radars during communication coverage. The information transmission rate is defined as the ratio of the amount of information successfully transmitted from ground mobile nodes to the UAV to the total amount of information from ground mobile nodes at the start of the mission. The joint interception probability is the sum of the probabilities that the electromagnetic signals emitted by any UAV at its current position and under the directional antenna deflection angle will be intercepted by each enemy radar.

[0049] Compared with the prior art, the embodiments of this application have the following beneficial effects: First, by formalizing the dynamic coverage problem into a POMDP and designing a multi-objective reward function, the UAV swarm can autonomously and collaboratively optimize its position and antenna pointing, thereby significantly improving the communication coverage and signal quality for dynamic ground targets.

[0050] Second, the reward function explicitly integrates a penalty mechanism for radar detection risk and signal interception, guiding UAVs to actively avoid threats and reduce their own exposure probability while completing their missions, thus achieving an effective balance between coverage performance and battlefield survivability.

[0051] Third, by using the connectivity reward term in the reward function, the drones are incentivized to maintain necessary network connections during maneuvers, ensuring the stability of information sharing and collaborative decision-making within the cluster and preventing overall mission failure due to individual disconnection.

[0052] Fourth, a centralized training-distributed execution framework is adopted, and a priority experience replay mechanism that integrates TD error and reward value is innovatively introduced, which enables the agent to learn from historical experience more efficiently, accelerates the policy convergence process, and improves the robustness and adaptability of the final decision model.

[0053] Fifth, after training, each UAV can make decisions based on local observations without the need for real-time command from a central node. The system has high scalability and resilience, and can adapt to complex and ever-changing real battlefield environments.

[0054] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A dynamic region coverage method for unmanned aerial vehicle (UAV) swarms based on deep reinforcement learning, characterized in that, The method includes: Step S1: Construct a simulation environment model with multi-physics coupling. The simulation environment model includes a node motion model, a directional antenna model, a space channel model, and an enemy radar detection model. Step S2: Based on the simulation environment model, the dynamic area coverage problem of UAV swarm is transformed into a partially observable Markov decision process, and the joint state space, joint observation space, joint action space and reward function of the agents are defined. Step S3: For partially observable Markov decision processes, a centralized training-decentralized execution framework is adopted for multi-agent training. During the training phase, the Actor network configured for each UAV agent is used for policy execution, the global Critic network is used to evaluate the joint actions of all agents, and the experience data generated by the interaction with the environment is stored through the experience replay buffer. Step S4: During the training process, when the number of samples in the experience replay buffer reaches the preset threshold, samples are extracted for network updates based on the priority experience replay mechanism. The priority experience replay mechanism calculates the probability of being sampled based on the temporal difference error of each experience and the reward value obtained by the agent, and uses the extracted samples to update the parameters of the Critic network and the Actor network in turn. Step S5: After training is completed, the trained decision model is deployed to the physical entities of the drone cluster, and each drone performs dynamic area coverage tasks in a distributed manner according to the decision results of the model.

2. The method for dynamic area coverage of unmanned aerial vehicle (UAV) swarms as described in claim 1, characterized in that, The node motion model is used to model the movement process of ground mobile nodes, where ground mobile nodes are randomly distributed in the target area at the start of the task and update their positions according to preset motion rules at each time step; The directional antenna model includes the antenna gain model for communication between UAVs or between a UAV and a ground mobile node. The antenna gain model is modeled as a nonlinear function of the beam incidence angle and the directivity factor. The space channel model includes air-to-ground channels between UAVs and ground mobile nodes, and air-to-air channels between UAVs. The air-to-ground channel modeling includes small-scale channel fading following a Rician distribution. The enemy radar detection model uses a probabilistic model to simulate the process of enemy radar detecting electromagnetic signals emitted by UAVs. The probabilistic model is constructed based on the radar's false alarm probability, carrier frequency, distance between the UAV and the radar, and the deflection angle parameters of the directional antenna.

3. The method for dynamic area coverage of unmanned aerial vehicle (UAV) swarms as described in claim 1, characterized in that, The joint state space contains global information on the positions of all UAVs, antenna deflection angles, and the positions of all ground vehicles; The joint observation space includes information from each agent's own observation range and observation information from neighboring nodes; The joint action space includes the action components in the x-axis direction, the action components in the y-axis direction, and the horizontal and vertical deflection angles of the directional antenna.

4. The method for dynamic area coverage of unmanned aerial vehicle (UAV) swarms as described in claim 1, characterized in that, The reward function is a multi-objective composite reward function, expressed as: ； in , and It is a constant greater than 0. The reward is the distance between the agent and the ground mobile node. The penalty for the interception of electromagnetic signals emitted by an intelligent agent. Rewards for connectivity in drone networks, The status of establishing a communication / data link between the drone and ground targets. The signal power received by the drone from the target node. The preset power reception threshold, The probability of being detected by enemy radar. To establish a probability threshold for safe detection. For key node indicators.

5. The method for dynamic area coverage of unmanned aerial vehicle (UAV) swarms as described in claim 1, characterized in that, The centralized training-distributed execution framework randomly initializes the Critic network, the Actor network for each agent, the corresponding target network, and the experience replay buffer before training begins. At each time step, the agent selects an action based on local observations using a greedy strategy or the Actor network output, and stores the empirical data in the experience replay buffer.

6. The method for dynamic area coverage of unmanned aerial vehicle (UAV) swarms as described in claim 1, characterized in that, The formula for calculating the probability of each experience being sampled using the priority experience replay mechanism is as follows: , and This is the proportionality coefficient. This represents the total number of samples in the experience replay buffer at the current moment. This indicates the degree to which the priority experience replay mechanism is used to control the frequency of use. and These represent the combined calculation of experience using TD error and reward value, respectively. value, and These represent the normalization of TD error value and reward value, respectively.

7. The method for dynamic area coverage of unmanned aerial vehicle (UAV) swarms as described in claim 1, characterized in that, The steps for updating the parameters of the Critic network and the Actor network sequentially using the extracted samples include: The parameters are optimized by minimizing the loss function, and the parameters of the Actor network and the target network are updated after the Critic network is updated a preset number of times. The training process is repeated at each time step until the entire scene or all training is completed.

8. The method for dynamic area coverage of unmanned aerial vehicle (UAV) swarms as described in claim 1, characterized in that, The steps following the deployment of the trained decision model to the physical entities of the drone swarm include: The drone swarm performs area coverage tasks in a distributed manner on physical entities, dynamically adjusting its position and antenna deflection angle to optimize communication coverage.

Images