Communication coverage optimization-oriented multi-unmanned aerial vehicle cooperative path planning method

Abstract

The invention discloses a communication coverage optimization-oriented multi-unmanned aerial vehicle cooperative path planning method. The method comprises the following steps of establishing a simulation environment comprising a plurality of ground base stations, obstacles, a target task point and a plurality of unmanned aerial vehicles; modeling a multi-unmanned aerial vehicle communication coverage cooperative path planning task into a Markov game process to construct a state space, an action space and a reward function of the unmanned aerial vehicle at each moment; an MHASD-MATD3 model based on an Actor-Critic network architecture is designed to ensure the flight path of the unmanned aerial vehicle to maximize the communication quality and avoid obstacle collision; a multi-head attention mechanism and a state difference mechanism are fused in the MHASD-MATD3 model. According to the method, the path planning efficiency and the communication reliability of the multi-unmanned aerial vehicle system in a complex urban communication coverage environment can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for unmanned aerial vehicles (UAVs), and in particular to a multi-UAV cooperative path planning method for communication coverage optimization. Background Technology

[0002] With the rapid development of unmanned aerial vehicle (UAV) technology, multi-UAV systems have been widely used in various fields such as disaster relief, environmental monitoring, agricultural inspection, urban communication, and logistics transportation due to their advantages of high mobility, flexible deployment, and low cost. Especially in complex urban environments, multiple UAVs can coordinate mission execution through communication connections with ground base stations (GBS). However, due to urban building obstructions, signal interference, and the dynamic environment, UAVs are prone to communication link interruptions, signal attenuation, and path conflicts during flight, thus affecting the safety and reliability of mission execution.

[0003] In multi-UAV cooperative missions, path planning is one of the core problems. The goal of path planning is to generate the optimal flight trajectory for each UAV to complete the arrival or cruising of specific mission points while ensuring communication quality and obstacle avoidance safety. Current research can be mainly divided into two categories: one is based on traditional optimization algorithms, such as improved genetic algorithms, particle swarm optimization algorithms, and fast expanding random tree algorithms. These methods achieve path generation through cost function optimization, but they have high computational complexity, slow convergence speed, and lack real-time adaptability in high-dimensional dynamic environments. The other category is based on deep reinforcement learning (DRL) methods, which learn the optimal policy through interaction with the environment and have better generalization and adaptability. However, existing reinforcement learning algorithms are prone to non-stationarity problems in multi-agent environments. The mutual influence between multiple agents makes the training process unstable, leading to difficulties in policy convergence.

[0004] To address the communication constraints of multi-UAV systems, some studies have considered incorporating communication connectivity into path planning models. For example, graph theory or distributed optimization frameworks are used to achieve communication-preserving path generation, or joint communication-navigation strategies are based on deep reinforcement learning. However, these methods suffer from the following shortcomings in complex dynamic environments: 1. The interaction relationships between multiple UAVs are not adequately modeled, easily leading to policy interference and reduced cooperation efficiency; 2. Fluctuations in communication link quality are not effectively perceived, making it difficult to achieve coordinated optimization of communication and navigation performance; 3. Traditional neural network structures lack the ability to model dynamic environmental changes, easily causing decision lag and instability; 4. Critic networks fail to distinguish the importance of different UAVs during evaluation, resulting in redundant training and slow convergence.

[0005] Therefore, existing technologies still suffer from problems such as unstable communication quality, slow policy convergence, and low collaboration efficiency when dealing with multi-UAV collaborative path planning tasks in complex urban communication environments. There is an urgent need for a reinforcement learning method that can adaptively model the interaction relationship between UAVs and sensitively perceive dynamic changes in the environment in order to achieve a comprehensive improvement in communication reliability and path planning performance. Summary of the Invention

[0006] To address the aforementioned technical issues, this invention proposes a multi-UAV cooperative path planning method for communication coverage optimization. This method can effectively improve the path planning efficiency and communication reliability of multi-UAV systems in complex urban communication coverage environments, and exhibits high robustness and convergence performance.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] A multi-UAV cooperative path planning method for communication coverage optimization includes the following steps:

[0009] Establish a simulation environment that includes several ground base stations, obstacles, target mission points, and multiple UAVs;

[0010] The multi-UAV communication coverage collaborative path planning task is modeled as a Markov game process to construct the state space, action space, and reward function of each UAV at each time step. The state space includes the position, velocity, signal-to-interference-plus-noise ratio, distance to the target task point, distance to obstacles, and relative position to other UAVs of each UAV. The action space is the two-dimensional velocity component of the UAV. The reward function consists of communication quality reward, target distance reward, collision penalty, and out-of-bounds penalty. The MHASD-MATD3 model based on the Actor-Critic network architecture is designed to ensure that the UAV's flight path maximizes communication quality and avoids obstacle collisions.

[0011] In training the MHASD-MATD3 model, a dual-commenter structure and a delayed update strategy are used to calculate the common Q-value of the two commenter networks. The loss function is calculated based on the Q-value and the parameters of the commenter network are updated by minimizing the loss function. The parameters of the actor network are updated by maximizing the Q-value. A soft update strategy is used to update the commenter network and the actor network until the maximum number of training rounds is reached, and the trained MHASD-MATD3 model is obtained.

[0012] The task of collaborative path planning for multi-UAV communication coverage was completed based on the trained MHASD-MATD3 model.

[0013] Preferably, in the simulation environment,

[0014] The ground base station adopts a three-dimensional linear array structure and is divided into three sectors;

[0015] By combining path loss model, shadow fading and small-scale Rayleigh / Rice fading effects, the signal power received by the UAV in each sector is calculated, and the sector with the highest signal power is selected for communication association.

[0016] Preferably, the actor network introduces a state difference mechanism, including the following:

[0017] Obtain the difference between the drone's current state and its previous state to obtain the state difference;

[0018] The current state of the drone and its state difference are concatenated to form a new input feature. This new input feature is then input into a multi-layer fully connected neural network to extract time-dynamic features.

[0019] Preferably, the critic network incorporates a multi-head attention mechanism, including the following:

[0020] Map the state-action pair of the j-th UAV and the state of the i-th UAV to a low-dimensional embedding space respectively to obtain the j-th UAV's state-action pair. The state and actions of the drone are related to its features and the Status characteristics of a drone The encoding process is represented as follows:

[0021]

[0022] in, and This is a two-layer neural network encoding function used to extract node features.

[0023] Multi-head attention mechanism It consists of several parallel attention heads, and the output of each head is:

[0024]

[0025] in, Let be the attention weight of the i-th drone towards the j-th drone. It is a value vector;

[0026] The multiple outputs are concatenated and linearly transformed to obtain the final aggregated feature, as shown in the following formula:

[0027]

[0028] The first Status characteristics of a drone e i With aggregated features z i After concatenation, the data is input into the critic network to obtain the action value function of drone i, as shown in the following formula:

[0029] .

[0030] Preferably, the attention weight of each head in the multi-head attention mechanism is calculated by scaling dot product attention calculation.

[0031] Preferably, during the training phase, the critic network shares global state, action, and reward information; during the execution phase, the actor network makes independent decisions based solely on its own local observation information.

[0032] Preferably, target distance reward Represented as:

[0033]

[0034] in, This is the distance penalty coefficient. Location of the drone. The coordinates of the target task point, The distance between the UAV and the target mission point is expressed in Euclidean form. Distance to the target threshold A fixed reward for reaching the target mission point.

[0035] Communication quality awards Represented as:

[0036]

[0037] in, These are the weighting coefficients. The signal-to-interference-plus-noise ratio (SIR) of the UAV at time t is given by the following formula:

[0038]

[0039] in, Indicates thermal noise power. For noise spectral density, For system bandwidth,

[0040] Out of bounds penalty Represented as:

[0041]

[0042] Where Ω represents the effective task area.

[0043] Based on the above technical solution, the beneficial effects of the present invention are:

[0044] 1) Enhanced global information aggregation capability: The multi-head attention mechanism can adaptively allocate neighbor influence weights, improving the modeling accuracy of the Critic network for the interaction relationships of multiple UAVs;

[0045] 2) Enhanced dynamic response capability: The state difference module enables the Actor network to capture state change trends, thereby generating smooth and stable control strategies;

[0046] (3) Faster convergence speed and higher stability: The dual critic and delayed update mechanism effectively suppress Q-value overestimation and improve the numerical stability of the training process.

[0047] (4) Communication and navigation performance optimization: In dynamic urban environments, multiple UAVs can achieve optimal path planning while maintaining stable communication links, significantly improving the overall mission success rate. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of a multi-UAV cooperative path planning method for communication coverage optimization in one embodiment;

[0049] Figure 2 This is a schematic diagram of multi-drone path planning under communication coverage in an urban environment, as shown in one embodiment.

[0050] Figure 3 This is a schematic diagram of a three-dimensional linear array model of a ground base station in one embodiment;

[0051] Figure 4 This is a heatmap of the average SINR in one embodiment;

[0052] Figure 5 This is a schematic diagram of the structure of the MHASD-MATD3 model in one embodiment;

[0053] Figure 6 This is the average reward of the MHASD-MATD3 model and the MADDPG, MAPPO, and MATD3 models in a simulation experiment, respectively, in one embodiment.

[0054] Figure 7 This is the average reward distribution of the MHASD-MATD3 model and the MADDPG, MAPPO, and MATD3 models in a simulation experiment, respectively, in one embodiment.

[0055] Figure 8 This is a schematic diagram comparing the average rewards of three drones under different models in one embodiment;

[0056] Figure 9 This is a schematic diagram comparing the communication reports of three drones under different models in one embodiment. Detailed Implementation

[0057] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0058] like Figures 1 to 5 As shown, this embodiment provides a multi-UAV cooperative path planning method for communication coverage optimization, including the following steps:

[0059] Step 100: Establish a simulation environment that includes several ground base stations, obstacles, target mission points, and multiple UAVs.

[0060] In this embodiment, to achieve communication quality modeling and navigation strategy optimization for a multi-UAV system in a complex 3D urban scene, a simulation environment for urban communication coverage is first established. This simulation environment consists of multiple ground base stations (GBS), building obstacles, target mission points, and multiple UAVs. The simulation area is 2x2km, where buildings occupy a certain proportion of the space. =0.3, the average number of buildings per square kilometer is =300, the building height follows a Rayleigh distribution, and the scale parameter is... =50, with a maximum height not exceeding 90m. There are a total of 7 ground base stations, each divided into three sectors, for a total of K=21 sectors. The transmit power of each sector is... =20dBm, carrier frequency is =8GHz. The UAV performs its mission at a fixed flight altitude h, and its three-dimensional position vector is represented as... Urban buildings are modeled as a set of three-dimensional polyhedra. When a straight line connecting the UAV to the base station intersects any building, it is defined as a non-line-of-sight (NLoS) link; otherwise, it is defined as a line-of-sight (LoS) link, thus achieving accurate modeling of complex occlusion environments.

[0061] To accurately describe the directional gain of a terrestrial base station antenna in space, a three-dimensional linear array model conforming to 3GPP standards is used, such as... Figure 3 As shown, the left figure depicts the vertical antenna gain pattern at an azimuth angle of 0°, while the right figure shows the horizontal antenna gain pattern at an elevation angle of 90°. Assume the azimuth angle of the antenna array is... The pitch angle is Then the wave vector can be expressed as:

[0062]

[0063] in, The wavelength is denoted as . The position vector of the nth element of the linear array is denoted as . Its guiding vector can be expressed as:

[0064]

[0065] The array factor describes the radiation pattern characteristics of an antenna array, determined by the steering vector. The conjugate transpose of the array weight vector Together, the calculation formula can be expressed as:

[0066]

[0067] By adjusting the amplitude and phase weighting of the antenna, a directional beam can be formed, achieving effective coverage of the target area.

[0068] The radiation pattern of a cell is determined by the gain in both the horizontal and vertical directions. According to the 3GPP standard model, the cell radiation pattern can be represented as follows:

[0069]

[0070] in, and These represent the half-power beamwidth in the horizontal and vertical directions, respectively. This refers to the front-to-back ratio (gain difference in the front and back directions). This represents the sidelobe level. Based on the radiation pattern, the element directional gain can be expressed using existing formulas (in dB):

[0071]

[0072] in, It is the maximum directional gain. Converted to linear gain, it is expressed as:

[0073]

[0074] The total antenna array gain (in dB), which measures the antenna's ability to radiate signals in all directions, can be expressed as:

[0075]

[0076] In the path propagation model, this embodiment considers both line-of-sight (LoS) and non-line-of-sight (NLoS) channel scenarios. When it's a LoS link, the path loss... It can be represented as:

[0077]

[0078] When it is an NLoS link, the path loss is... It can be represented as:

[0079]

[0080] The corresponding large-scale gain (linear scale) can be expressed as:

[0081]

[0082] Furthermore, small-scale fading is modeled using Rayleigh (NLoS) or Rice (LoS) distributions. Based on the above modeling, the baseband equivalent channel between the UAV and the k-th sector can be expressed as:

[0083]

[0084] in, Indicates antenna gain. Let represent the complex fading coefficient. Therefore, the signal power received by the UAV can be expressed as:

[0085]

[0086] Among all sectors, the drone selects the sector with the highest signal power for communication association, which can be represented as:

[0087]

[0088] And this sector is taken as the current serving sector. Then the signal-to-interference-plus-noise ratio (SINR) of the UAV at time t can be defined as:

[0089]

[0090] in, 'm' represents the GBS sector associated with the drone, and 'm' represents the non-associated GBS sector. This represents the signal power provided by the GBS sector associated with the UAV at time t. This represents the signal power provided by the GBS sector that is not associated with the UAV at time t. Indicates thermal noise power. For noise spectral density, Let be the system bandwidth. To meet the minimum SINR threshold, At that time, it is determined that the drone is within the communication coverage area; if At that time, it is in a coverage blind spot.

[0091] Finally, to visualize the communication performance distribution, Monte Carlo sampling was used to calculate the average SINR value on the plane at flight altitude h. The average SINR of each grid point under M random fading and blocking events can be expressed as: This allows us to obtain the average SINR level at this location:

[0092]

[0093] This generates, as follows Figure 4 The heatmap shown represents the average SINR, where shades of green indicate communication coverage strength, with darker green signifying better signal coverage. This heatmap provides a visual input for calculating communication rewards in subsequent reinforcement learning models.

[0094] Step 200: The multi-UAV communication coverage collaborative path planning task is modeled as a Markov game process to construct the state space, action space, and reward function of each UAV at each time step. The state space includes the position, velocity, signal-to-interference-plus-noise ratio, distance to the target task point, distance to obstacles, and relative position to other UAVs of each UAV. The action space is the two-dimensional velocity component of the UAV. The reward function consists of communication quality reward, target distance reward, collision penalty, and out-of-bounds penalty. The MHASD-MATD3 model based on the Actor-Critic network architecture is designed to ensure that the UAV's flight path maximizes communication quality and avoids obstacle collisions.

[0095] In this embodiment, see Figure 5 To achieve autonomous collaborative decision-making and path optimization among multiple UAVs in a communication coverage environment, this invention models the multi-UAV collaborative path planning task as a multi-agent Markov game process. This describes the process by which multiple UAVs make independent decisions based on local observations and jointly optimize the global objective in a shared environment. Each UAV is regarded as an agent, interacting with the environment and learning the optimal strategy.

[0096] 1. Definition of State Space

[0097] At time t, the state vector of the i-th UAV can be defined by the original formula as follows:

[0098]

[0099] in, Indicates the three-dimensional position of the drone; The speed of the drone on the horizontal plane; The Euclidean distance between the UAV and the target mission point; The minimum distance between the drone and the obstacle; This represents the minimum relative distance between a drone and other drones.

[0100] Each state variable is acquired in real time from airborne sensors, environmental simulation, or communication feedback. Through this state space definition, the model can simultaneously describe the UAV's motion state, environmental constraints, and communication perception results, achieving a comprehensive characterization of the global situation.

[0101] 2. Definition of Action Space

[0102] The action space of a drone is defined as a two-dimensional continuous action, which can be represented as:

[0103]

[0104] in, and These represent the velocity components in the x and y directions, respectively. Each actor network outputs the optimal action instruction based on the current state and the state difference, achieving smooth and continuous path planning.

[0105] 3. Definition of reward function

[0106] Within the deep reinforcement learning framework, the reward function quantifies the immediate gains obtained by a UAV taking a specific action in a given state, serving as the core basis for policy optimization. The reward function designed in this invention simultaneously considers factors such as mission distance, communication quality, obstacle avoidance, and flight constraints, and its formal definition can be expressed as:

[0107]

[0108] Target distance reward This system guides the drone to approach the target mission point. A negative reward is given when the drone is far from the target point, and the reward increases as it approaches. A fixed positive reward is given upon reaching the target point. The calculation formula can be expressed as follows:

[0109]

[0110] in, Location of the drone. The coordinates of the target task point, Let be the Euclidean distance between the two. Distance to the target threshold This is the distance penalty coefficient. A fixed reward for reaching the target point.

[0111] Communication quality awards To ensure good communication quality during navigation, this invention incorporates the signal-to-interference-plus-noise ratio (SINR) as a communication performance indicator into the reward function. The communication quality reward can be defined as follows:

[0112]

[0113] in, These are the weighting coefficients. The signal-to-interference-plus-noise ratio (SIR) of the UAV at time t is specifically calculated using the communication coverage model:

[0114]

[0115] A higher SINR value represents better communication quality, therefore this reward is related to... This is proportional to the level of drones, and is used to encourage drones to stay in high-coverage areas.

[0116] Collision Penalty To avoid collisions between drones or with obstacles, a minimum safe distance L is set. This is the closest distance between a drone and other entities. If a collision occurs, a negative reward is applied:

[0117]

[0118] in, This is a collision penalty constant. This mechanism ensures that multiple drones maintain a safe distance when performing cooperative tasks.

[0119] Out of bounds penalty When a drone flies out of the mission boundary or communication area, its status is deemed invalid, and a fixed penalty is immediately imposed. To limit the flight range of the drone, it can be expressed as:

[0120]

[0121] Where Ω represents the effective task area.

[0122] In summary, the reward function designed in this embodiment achieves multi-objective optimization for multiple UAVs in complex communication coverage environments by establishing a comprehensive balance between task completion, communication stability, and flight safety. Through cumulative maximization training, UAVs can adaptively plan the optimal path in dynamic environments that maintains communication connectivity while quickly and safely reaching the target, providing a unified mathematical foundation for subsequent reinforcement learning network training and policy evaluation.

[0123] In step 300, during the training of the MHASD-MATD3 model, a dual-commentator structure and a delayed update strategy are used to calculate the common Q-value of the two commentator networks. The loss function is calculated based on the Q-value, and the parameters of the commentator network are updated by minimizing the loss function. The parameters of the actor network are updated by maximizing the Q-value. A soft update strategy is used to update the commentator network and the actor network until the maximum number of training rounds is reached, resulting in the trained MHASD-MATD3 model.

[0124] In this embodiment, to achieve autonomous collaborative path planning and policy optimization among multiple UAVs in a communication coverage environment, a structural innovation was proposed based on the traditional Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. A multi-agent reinforcement learning algorithm, MHASD-MATD3 (Multi-Head Attention and State Differential Multi-Agent TD3), was developed, integrating multi-head attention mechanisms and state difference feature modeling. In multi-agent collaborative scenarios, this algorithm can simultaneously capture the interaction relationships between UAVs and the dynamic characteristics of environmental changes, thereby achieving a more accurate and stable policy optimization process.

[0125] like Figure 5 As shown, the MHASD-MATD3 algorithm extends the MATD3 framework with two core modules: a multi-head attention mechanism in the Critic network to model the interaction dependencies between different UAVs; and a state difference module in the Actor network to characterize the dynamic changes of the UAVs and the environment. The algorithm follows a Centralized Training with Decentralized Execution (CTDE) framework. During the training phase, each UAV shares global state, action, and reward information to achieve joint optimization; during the execution phase, each UAV makes independent decisions based on its own local observations, thus balancing global coordination and local autonomy.

[0126] 1. Design of State Differential Module

[0127] To enhance the responsiveness of actor networks to environmental changes, a state difference module is introduced into the input layer to describe the dynamic differences in states between consecutive time steps. The calculation formula for this module can be expressed as:

[0128]

[0129] in, This represents the current state vector of the drone. This represents the state vector at the previous moment. The state difference reflects the changing trends of the UAV's motion and environmental characteristics, including position movement, signal-to-interference-plus-noise ratio fluctuations, and relative velocities of neighboring UAVs.

[0130] During the input phase of the Actor network, the current state and the state difference are concatenated to form a new input feature, which can be represented as:

[0131]

[0132] The data is then fed into a multi-layer fully connected neural network to extract temporal dynamic features. This design enables the policy network to capture "trends of change" rather than relying solely on static observations, thereby significantly improving policy stability and generalization performance in dynamic scenarios.

[0133] 2. Design of multi-head attention mechanism

[0134] In multi-agent environments, complex cooperative and interference relationships exist among drones. Traditional MATD3 algorithms typically perform simple concatenation of information from other drones, failing to effectively distinguish the importance of different drones and resulting in significant errors in value function estimation. To address this, a multi-head attention module is introduced into the critic network, adaptively assigning weights to highlight neighboring information that has the greatest impact on the current drone's decision-making.

[0135] 1) Feature coding layer

[0136] First, the first The status of the drone – actions relative to the first The states of the drones are mapped to a low-dimensional embedding space to obtain the first... The state and actions of the drone are related to the characteristics and the first The state characteristics of a drone can be encoded as follows:

[0137]

[0138] in, and This is a two-layer neural network encoding function used to extract node features.

[0139] 2) Attention Computation Layer

[0140] Building upon embedded features, query, key, and value vectors are introduced. , , The transformation is performed using the existing formula and linear mapping matrix:

[0141]

[0142] in, , and These are shared projections used to generate query, key, and value vectors, respectively.

[0143] Then, the Scaled Dot-Product Attention mechanism is used to compute the first... The drone to the first The attention weight of a drone can be expressed by the existing formula as follows:

[0144]

[0145] in, The dimension of the key vector. This represents the number of drones. The weight indicates the degree of influence of other drones on the current drone's value assessment.

[0146] Multi-head attention mechanism It consists of several parallel attention heads, each responsible for capturing different levels of interaction features. The output of each head can be expressed by the existing formula as follows:

[0147]

[0148] The multi-head outputs are concatenated and linearly transformed to obtain the final aggregated feature, which can be expressed by the existing formula as follows:

[0149]

[0150] 3) Q-value estimation layer

[0151] The first Status characteristics of a drone e i With aggregated features z i After concatenation, the data is input into the critic network to obtain the action value function of drone i, which can be expressed as:

[0152]

[0153] This structure allows critics to adaptively evaluate the overall value of multi-UAV systems based on dynamic attention weights, thereby improving the accuracy of policy gradient estimation and training stability.

[0154] 3. Target value and network update mechanism

[0155] To avoid the problem of Q-value overestimation, this invention employs a dual-commentator structure and a delayed update strategy during training.

[0156] The target Q value can be calculated using the existing formula as follows:

[0157]

[0158] in, For immediate rewards, γ is the discount factor. For the first The evaluation function of the target commentator network represents the joint state at the next time step.

[0159] The loss function of the critic network can be expressed by the existing formula as follows:

[0160]

[0161] in, Indicates the first An evaluation function for the commentator network. The commentator parameters are updated in a manner that minimizes this loss. .

[0162] The goal of the actor network is to maximize the Q-value, for the th Parameters of a drone The gradient update rule can be expressed as follows based on the existing formula:

[0163]

[0164] in, Indicates the first The current strategy of a drone in its current state. This indicates that gradient calculation is being performed.

[0165] The target network parameters adopt a soft update strategy to ensure the smoothness and stability of the training process, which can be expressed by the existing formula as follows:

[0166]

[0167] in, This is the soft update coefficient. and These are parameters for online and target actor networks. and These are parameters for online and target commentator networks.

[0168] Step 400: Complete the multi-UAV communication coverage collaborative path planning task based on the trained MHASD-MATD3 model.

[0169] In this embodiment, the converged MHASD-MATD3 model is deployed in a multi-UAV system to achieve real-time path planning and communication coverage navigation. During operation, the UAVs dynamically adjust their flight paths based on the learned optimal strategy to maximize communication quality and avoid obstacle collisions.

[0170] See Figure 6 The average reward of the MHASD-MATD3 model compared to the MADDPG, MAPPO, and MATD3 models in simulation experiments; see [link to relevant documentation]. Figure 7 The average reward distribution of the MHASD-MATD3 model, MADDPG, MAPPO, and MATD3 models in the simulation experiments of this application are shown in [reference]. Figure 8The average reward distributions of the three drones in the MHASD-MATD3 model and the MADDPG, MAPPO, and MATD3 models of this application are shown in the attached table. Figure 9 The communication rewards of the three UAVs were compared with those of the MHASD-MATD3 model in this application and the MADDPG, MAPPO, and MATD3 models, respectively. The results show that the MHASD-MATD3 model significantly improves the average reward, average reward distribution, communication reward, and policy convergence speed. Experimental results demonstrate that this method maintains stable communication links and smooth flight paths in complex urban environments, exhibiting excellent robustness and real-time performance.

[0171] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0172] The above are merely preferred embodiments of the present application and are not intended to limit the scope of the present application. Various modifications and variations can be made to the embodiments of the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of the present application should be included within the protection scope of the embodiments of the present application.

Claims

1. A multi-UAV cooperative path planning method for communication coverage optimization, characterized in that, Includes the following steps: Establish a simulation environment that includes several ground base stations, obstacles, target mission points, and multiple UAVs; The multi-UAV communication coverage collaborative path planning task is modeled as a Markov game process to construct the state space, action space, and reward function of each UAV at each time step. The state space includes the position, velocity, signal-to-interference-plus-noise ratio, distance to the target task point, distance to obstacles, and relative position to other UAVs of each UAV. The action space is the two-dimensional velocity component of the UAV. The reward function consists of communication quality reward, target distance reward, collision penalty, and out-of-bounds penalty. The MHASD-MATD3 model based on the Actor-Critic network architecture is designed to ensure that the UAV's flight path maximizes communication quality and avoids obstacle collisions. In training the MHASD-MATD3 model, a dual-commenter structure and a delayed update strategy are used to calculate the common Q-value of the two commenter networks. The loss function is calculated based on the Q-value and the parameters of the commenter network are updated by minimizing the loss function. The parameters of the actor network are updated by maximizing the Q-value. A soft update strategy is used to update the commenter network and the actor network until the maximum number of training rounds is reached, and the trained MHASD-MATD3 model is obtained. The task of collaborative path planning for multi-UAV communication coverage was completed based on the trained MHASD-MATD3 model.

2. The multi-UAV cooperative path planning method for communication coverage optimization according to claim 1, characterized in that, In the simulation environment The ground base station adopts a three-dimensional linear array structure and is divided into three sectors; By combining path loss model, shadow fading and small-scale Rayleigh / Rice fading effects, the signal power received by the UAV in each sector is calculated, and the sector with the highest signal power is selected for communication association.

3. The multi-UAV cooperative path planning method for communication coverage optimization according to claim 1, characterized in that, The actor network introduces a state difference mechanism, including the following: Obtain the difference between the drone's current state and its previous state to obtain the state difference; The current state of the drone and its state difference are concatenated to form a new input feature. This new input feature is then input into a multi-layer fully connected neural network to extract time-dynamic features.

4. The multi-UAV cooperative path planning method for communication coverage optimization according to claim 1, characterized in that, The critic network introduces a multi-head attention mechanism, including the following: Map the state-action pair of the j-th UAV and the state of the i-th UAV to a low-dimensional embedding space respectively to obtain the j-th UAV's state-action pair. The state and actions of the drone are related to its features and the Status characteristics of a drone The encoding process is represented as follows: ； in, and This is a two-layer neural network encoding function used to extract node features. Multi-head attention mechanism It consists of several parallel attention heads, and the output of each head is: ； in, Let be the attention weight of the i-th drone towards the j-th drone. It is a value vector; The multiple outputs are concatenated and linearly transformed to obtain the final aggregated feature, as shown in the following formula: ； The first Status characteristics of a drone e i With aggregated features z i After concatenation, the data is input into the critic network to obtain the action value function of drone i, as shown in the following formula: 。 5. A multi-UAV cooperative path planning method for communication coverage optimization according to claim 4, characterized in that, In a multi-head attention mechanism, the attention weight of each head is calculated by scaling the dot product attention.

6. The multi-UAV cooperative path planning method for communication coverage optimization according to claim 1, characterized in that, During the training phase, the critic network shares global state, action, and reward information; during the execution phase, the actor network makes independent decisions based solely on its own local observation information.

7. The multi-UAV cooperative path planning method for communication coverage optimization according to claim 1, characterized in that, Target distance reward Represented as: ； in, This is the distance penalty coefficient. Location of the drone. The coordinates of the target task point, The distance between the UAV and the target mission point is expressed in Euclidean form. Distance to the target threshold A fixed reward for reaching the target mission point. Communication quality awards Represented as: ； in, These are the weighting coefficients. The signal-to-interference-plus-noise ratio (SIR) of the UAV at time t is given by the following formula: ； in, Indicates thermal noise power. For noise spectral density, For system bandwidth, Out of bounds penalty Represented as: ； Where Ω represents the effective task area.

Images