A Double-Layer UAV Communication Coverage Method Based on Deep Reinforcement Learning Network

Abstract

The present invention discloses a two - layer UAV communication coverage method based on a deep reinforcement learning network. The UAVs are divided into control UAVs and communication UAVs. First, the control UAVs cluster the users according to the geographical locations of the users, and based on the clustering results, match the clusters with the communication UAVs one by one, and obtain the flight trajectories of the communication UAVs under the constraints of collision and speed through the position change of the cluster centers. Then, taking the position information of the communication UAVs as the state input, the scheduling information of the communication UAVs, power allocation, and the flight trajectories of the control UAVs are obtained based on a pre - trained deep reinforcement learning network, so as to achieve the maximum fair energy efficiency of UAV communication coverage in the area. The present invention realizes the accurate planning of the UAV flight trajectories for ground user communication coverage by jointly optimizing the energy efficiency within the maximum period of UAV flight trajectories, user connection, communication UAV scheduling, and power allocation.

Description

Technical Field

[0001] The present invention belongs to the technical field of UAV trajectory planning, and specifically relates to a two-layer UAV communication coverage method based on a deep reinforcement learning network. Background Art

[0002] With the increasing access of a huge number of users, the communication infrastructure is significantly insufficient, and wireless communication networks require higher energy efficiency and more reliable service quality. Due to the advantages of good channel conditions, high mobility, strong flexibility, easy deployment, low cost, and having the necessary on-board computing capabilities, UAVs can be regarded as a supplement to ground communication facilities. With the development of various derivative solutions for assisting communication, the role of UAVs in assisting communication in communication systems is becoming more and more obvious. Whether in the current cellular network or in the future 6G wireless communication network, UAVs will play an indispensable role.

[0003] In order to give full play to the advantages of UAVs as aerial base stations, the network coverage performance can be significantly improved by jointly optimizing UAV trajectories and resource allocation. However, considering the limited on-board energy and computing capabilities, energy efficiency is particularly important for UAV-assisted communication networks. For a single-UAV-assisted communication network, although it is easy to deploy, it is difficult to adapt to increasingly complex communication scenarios due to limited capabilities. While multi-UAV-assisted communication networks can provide higher data rates, the cooperation and competition among UAVs generate many necessary consumptions and challenges.

[0004] In existing research, many studies and attempts have been made on UAV-assisted communication networks. Most of the work decomposes the original optimization problem into multiple convex optimization sub-problems for iterative solution. However, traditional algorithms based on convex optimization need to perform offline calculations through global information to obtain solutions, resulting in inability to be deployed in dynamic environments. At the same time, the computational complexity of the algorithm also increases sharply with the increase in the scale of the optimization problem. Algorithms based on deep reinforcement learning can transform the optimization problem of UAV-assisted communication into a Markov decision process and learn the optimal strategy by maximizing the cumulative reward. In algorithms based on deep reinforcement learning, the neural network structure of the algorithm is often associated with the number of ground users, resulting in limitations in the generality and robustness of the algorithm; in optimization problems in continuous spaces, it is also difficult to solve the huge exploration space. Summary of the Invention

[0005] Object of the Invention: The present invention provides a two-layer UAV communication coverage method based on a deep reinforcement learning network, which realizes UAV trajectory planning for ground user communication coverage by jointly optimizing UAV trajectories, user connections, communication UAV scheduling, and power allocation.

[0006] Technical solution: A two-layer UAV communication coverage method based on a deep reinforcement learning network according to the present invention includes the following steps:

[0007] (1) Divide the UAVs into control UAVs and communication UAVs, and cluster the users according to the position information of the users at each moment;

[0008] (2) Based on the clustering result, match each cluster group with a communication UAV one by one to obtain the user connection information at each moment;

[0009] (3) Obtain the position of the communication UAV at the current moment under the collision and speed constraints through the position change of the cluster center;

[0010] (4) Construct a deep reinforcement learning network and train it; input the position information of the communication UAV into the trained deep reinforcement learning network to obtain the scheduling information of the communication UAV, power allocation, and the position of the control UAV at the current moment;

[0011] (5) Repeat steps (1) to (4) at each moment to maximize the fair energy efficiency of the UAV communication coverage in the area under the conditions of meeting the UAV speed constraint, UAV energy constraint, and user quality of service constraint.

[0012] Further, the implementation process of clustering the users according to the position information of the users at each moment in step (1) is as follows:

[0013] Divide the service period T into N sub-slots, and each sub-slot is represented as δ t = T / N; Initialize the centroid C n at the current time slot; n represents the nth time slot. When n = 0, adopt the k-means++ centroid initialization scheme, randomly initialize the first centroid, and select the remaining centroids according to the selection probability ; M represents the set of ground users, and D(m) represents the shortest distance from user m to the nearest selected centroid; when n ≠ 0, C n = {u k,n-1 , k ∈ K}, K represents the set of communication UAVs, and u k,n-1 represents the three-dimensional Cartesian coordinates of communication UAV k at the nth time slot;

[0014] Calculate the distance from the three-dimensional Cartesian coordinates w m,n of ground user m at the nth time slot to C n , compare the distances to each centroid, and classify the user into the same category as the centroid with the shortest distance to itself; for each cluster L k , update its centroid according to the coordinates of the kth centroid at the nth time slot;

[0015] Calculate the distance between the new and old centroids, and determine whether it is less than the preset threshold. If it is less than the threshold, the calculation ends; otherwise, repeat the above clustering steps until the distance is less than the threshold.

[0016] Furthermore, the implementation process of step (2) is as follows:

[0017] Express the communication UAV and clustering matching problem as a weighted complete bipartite graph G=(K;L k ;E;W), where K represents the set of communication UAVs, L k represents the kth cluster after user clustering, E represents the set of edges between the communication UAV and the cluster, and W represents the edge weight;

[0018] Each time, select an unmatched vertex and search for a perfect matching of the current subgraph according to the depth-first search method. When all the edges in the subgraph satisfy f(i)+f(j)=w(i,j), the current matching result is the maximum weight matching of the current subgraph; f(i) and f(j) are the vertex labels of vertices i and j:

[0019]

[0020] f(j)=0,j∈L k

[0021] When a perfect matching of the current subgraph cannot be obtained, by modifying:

[0022] f(i)=f(i)+Δ,i∈X

[0023] f(j)=f(j)-Δ,j∈Y

[0024] where, and represent the vertices that have been matched, and Δ represents the modification difference:

[0025]

[0026] Loop the above steps until the maximum weight perfect matching between the communication UAV and the cluster is obtained.

[0027] Furthermore, the deep reinforcement learning network in step (4) includes two Actor networks and eight Critic networks; two Actor networks are used to online-decide the flight actions and resource allocation actions at each moment, and obtain the scheduling information of the communication UAV, power allocation, and control the position of the UAV at the current moment.

[0028] Furthermore, the implementation process of constructing the deep reinforcement learning network in step (4) is as follows:

[0029] Convert the problem of UAV trajectory planning into a Markov decision process, where the state space is expressed as:

[0030]

[0031] Among them, q(n) is the position of the UAV at time n; represents the total service time of communication UAV k before time n; v J (n) is the speed of the UAV at time n;

[0032] The action space is:

[0033]

[0034] Among them, λ v , and respectively represent the normalized speed of the UAV, the angle between the flight direction and the Z-axis, and the angle between the projection of the flight direction on the XY plane and the X-axis. {λ k (n)} k∈K is the power allocation set;

[0035] The reward function is:

[0036] r(n) = r fee (n) + r vel (n) + r alt (n)

[0037] Among them, r fee (n) is the fair energy efficiency reward, r vel (n) is the penalty for violating the speed constraint, and r alt (n) is the penalty for violating the altitude constraint;

[0038] Learn the optimal policy based on the maximum entropy framework to maximize the objective function:

[0039]

[0040] Among them, π* represents the optimal policy, α represents the temperature coefficient of the entropy value, and H(π(·∣s t )) represents the entropy of taking the policy π in the s t state.

[0041] Furthermore, the implementation process of training the deep reinforcement learning network in step (4) is as follows:

[0042] Train two Actor networks to output the trajectory action and resource allocation action of the UAV respectively, and the loss functions are:

[0043]

[0044] Among them, φ v and φ p represent the track actor network parameters and the resource allocation actor network parameters; represents the policy of the track actor network under φ v ; represents the Q value obtained by the critic network under the parameter θ i ;

[0045] The loss function of the Critic network is defined as:

[0046]

[0047] Among them, θ i represents the critic network parameters; is the advantage function, expressed as:

[0048]

[0049] The loss function of the temperature coefficient of the entropy value is expressed as:

[0050]

[0051] Among them, is the target entropy constant;

[0052] The deep reinforcement learning network parameters are trained and updated according to the above loss function, and the target Critic network is slowly updated with the Critic network. The update strategy is:

[0053]

[0054] Among them, represents the parameters of the target critic network; ε represents the parameters for updating the Critic network.

[0055] Furthermore, the weight W of the edge is calculated by w(i, j) = -||u i - C j ||, i ∈ K, j ∈ L k where w(i, j) represents the weight value of the edge between i and j; u i represents the coordinate of the i-th communication UAV; C j represents the centroid coordinate of the j-th cluster.

[0056] Furthermore, the fair energy efficiency reward r fee (n), the violation of speed constraint penalty r vel (n) and the violation of altitude constraint penalty r alt (n) are respectively:

[0057]

[0058] r vel = κ vel (n)l vel

[0059] r alt = κ alt (n)l alt

[0060] where l fee is the fair energy efficiency reward coefficient, is the service fairness index, α k is the communication UAV scheduling information, R k (n) is the reachable transmission rate of the communication UAV, and p(vJ(n)) represents the energy consumption of the control UAV; κ vel represents the violation of speed constraint indicator, l vel represents the violation of speed constraint penalty coefficient; κ alt is the violation of altitude constraint indicator, l alt is the violation of altitude constraint penalty coefficient.

[0061] Advantageous effects: Compared with the prior art, the advantageous effects of the present invention are as follows: The present invention first solves the problem that traditional algorithms such as convex optimization cannot be deployed online in a dynamic environment, such as considering the mobility of users; through the method of clustering and matching users, the neural network is decoupled from the number of users, so that the proposed algorithm can be deployed and implemented under different numbers of users without retraining after training, improving the generality of the algorithm; based on the framework of maximum entropy, the UAV environment planning and power allocation are trained and learned through different Actor networks, reducing the exploration space and enhancing the exploration ability. Compared with the DDPG algorithm, the cumulative reward is increased by about 100%. Description of the Drawings

[0062] Figure 1 is the application scenario diagram of the present invention;

[0063] Figure 2 is the simulation result diagram of the energy consumption of the communication UAV;

[0064] Figure 3 is the simulation result diagram of the cumulative reward during the training of the algorithm;

[0065] Figure 4 is the simulation result diagram of the cumulative fair energy efficiency during evaluation under different numbers of users. Detailed Embodiment

[0066] The present invention will be further described in detail below with reference to the drawings.

[0067] The present invention proposes a two - layer UAV communication coverage method based on a deep reinforcement learning network. The UAVs are divided into control UAVs and communication UAVs, and the fair energy efficiency within the maximum cycle is jointly optimized by optimizing the UAV trajectories, user connections, communication UAV scheduling, and power allocation. The specific implementation process is as follows:

[0068] Step 1: Cluster the users according to their location information at each moment.

[0069] The service period T is divided into N sub - time slots, and each sub - time slot is denoted as δ t = T / N.

[0070] Initialize the centroid C n at the current time slot, where n represents the nth time slot. When n = 0, a k - means++ centroid initialization scheme is adopted, and the first centroid is randomly initialized, and then the remaining centroids are selected according to the probability formula where g cods represents the selection probability, M represents the set of ground users, and D(m) represents the shortest distance from user m to the nearest selected centroid; when n≠0, C n = {u k,n-1 , k∈K}, where K represents the set of communication UAVs, and u k,n-1 represents the three - dimensional Cartesian coordinates of communication UAV k at the nth time slot.

[0071] Calculate the distance from w m,n to C n , compare the distances to each centroid, and classify the user into the category of the centroid closest to itself; where w m,n represents the three - dimensional Cartesian coordinates of ground user m at the nth time slot. For each cluster L k , update its centroid according to , where is the coordinate of the kth centroid at the nth time slot.

[0072] Calculate the distance between the new and old centroids, and judge whether it is less than the threshold. If it is less than the threshold, the calculation ends; otherwise, repeat the above clustering steps until the distance is less than the threshold.

[0073] Step 2: Based on the clustering results, match each cluster with a communication UAV one - by - one to obtain the scheduling information of the users.

[0074] Formulate the problem of matching communication UAVs with clusters as a weighted complete bipartite graph G=(K; L k ; E; W), where K represents the set of communication UAVs, L k represents the kth cluster after user clustering, E represents the set of edges between communication UAVs and clusters, W represents the edge weights, and w(i,j)= - ||u i-C j ||, where i ∈ K and j ∈ L k It is calculated that w(i,j) represents the weight of the edge between i and j; u i represents the coordinate of the i-th communication UAV; C j represents the centroid coordinate of the j-th cluster.

[0075] Each time an unmatched vertex is selected to search for a perfect matching of the current subgraph according to the depth-first search method. When all the edges in the subgraph satisfy f(i) + f(j) = w(i,j), the current matching result is the maximum-weight matching of the current subgraph. Here, f(i) and f(j) are defined as the vertex labels of vertices i and j:

[0076]

[0077] f(j) = 0, j ∈ L k

[0078] When a perfect matching of the current subgraph cannot be obtained, by modifying:

[0079] f(i) = f(i) + Δ, i ∈ X,

[0080] f(j) = f(j) - Δ, j ∈ Y

[0081] where, and represent the vertices that have been matched, and Δ represents the modification difference;

[0082]

[0083] Repeat the above steps until a perfect matching with the maximum weight between the communication UAVs and the clusters is obtained.

[0084] Step 3: According to the obtained cluster centroids, under the constraints of UAV speed and collision, obtain the position of the communication UAVs at the current moment.

[0085] Step 4: Use the position information of the communication UAVs as the state input to a pre-trained deep reinforcement learning network, including 2 Actor networks and 8 Critic networks. Use the 2 Actor networks to online-decide the flight actions and resource allocation actions at each moment, and obtain the scheduling information, power allocation, and control the position of the UAVs at the current moment.

[0086] The training process of the deep reinforcement learning network includes: converting the problem of controlling the UAV trajectory planning into a Markov decision process, designing the state space, action space, and reward function of the model; decomposing the agent into two sub-agents, which respectively output the trajectory action and resource allocation action for controlling the UAV; training the reinforcement learning network based on the maximum entropy framework to obtain a trained network. The specific process is as follows:

[0087] Convert the problem of controlling the UAV trajectory planning into a Markov decision process, where the state space s(n) is expressed as:

[0088]

[0089] where q(n) is the position of the control UAV at time n; represents the total service time of communication UAV k before time n; v J (n) is the speed of the control UAV at time n.

[0090] The action space a(n) is expressed as:

[0091]

[0092] where λ v 、 and respectively represent the normalized speed of the control UAV, the angle between the flight direction and the Z-axis, and the angle between the projection of the flight direction on the XY plane and the X-axis, {λ k (n)} k∈K is the power allocation set.

[0093] The reward function r(n) is:

[0094] r(n) = r fee (n) + r vel (n) + r alt (n)

[0095] where r fee (n) is the fair energy efficiency reward, r vel (n) is the penalty for violating the speed constraint, r alt (n) is the penalty for violating the altitude constraint, where:

[0096]

[0097] r vel = κ vel (n)l vel

[0098] r alt = κ alt (n)l alt

[0099] Among them, l fee is the fair energy efficiency reward coefficient, is the service fairness index, α k is the communication UAV scheduling information, R k (n) is the reachable transmission rate of the communication UAV, p(v J (n)) represents the energy consumption of the control UAV; κ vel represents the violation of speed constraint indicator, l vel represents the violation of speed constraint penalty coefficient; κ alt is the violation of altitude constraint indicator, l alt is the violation of altitude constraint penalty coefficient.

[0100] Learn the optimal policy based on the maximum entropy framework to maximize the objective function:

[0101]

[0102] Among them, π* represents the optimal policy, α represents the temperature coefficient of the entropy value, H(π(·∣s t )) represents the entropy of taking the policy π in the s t state.

[0103] Train two Actor networks to output the trajectory action and resource allocation action of the control UAV respectively, and the loss functions are respectively:

[0104]

[0105] Among them, φv v and φ p represent the trajectory actor network parameters and resource allocation actor network parameters; represents the policy of the trajectory actor network under φ v ; represents the Q value obtained by the critic network under the parameter θ i .

[0106] The loss function of the Critic network is defined as:

[0107]

[0108] Among them, θ i represents the critic network parameters; is the advantage function, which is expressed as:

[0109]

[0110] The loss function of the temperature coefficient α of the entropy value is expressed as:

[0111]

[0112] Among them, is the target entropy constant.

[0113] The parameters of the deep reinforcement learning network are trained and updated according to the above loss function. Among them, the target Critic network is slowly updated with the Critic network, and the update strategy is:

[0114]

[0115] Among them, represents the parameters of the target critic network; ε represents the parameter of soft update.

[0116] Step 5: Repeat the above steps at each moment, so as to maximize the energy efficiency of the communication coverage in the area by the UAV under the conditions of meeting the UAV speed constraint, UAV energy constraint, and user quality of service constraint.

[0117] Figure 1 This is the scenario of maximizing the energy efficiency of the communication coverage in the area by the two-layer UAV-assisted communication network under the conditions of meeting the UAV speed constraint, UAV energy constraint, and user minimum rate requirement constraint provided by the present invention. Figure 2 This is the simulation result diagram of the energy consumption of the communication UAV adopting the communication UAV flight path planning method obtained by the present invention and based on K-means clustering. Compared with the k-means algorithm, the present invention takes into account the time-varying movement of ground users and optimizes the selection of the initial centroid, while the k-means algorithm only considers the internal distance. Based on this, the total flight distance of the communication UAV adopting the present invention is reduced by 85%. Due to the reduction of the total flight distance, the total energy consumption of the communication UAV adopting the present invention is 60% of that of the traditional k-means algorithm in the figure. At the same time, it shows that the hovering energy consumption of the communication UAV accounts for a large part of the total energy consumption.

[0118] Figure 3This is a simulation result graph of the cumulative rewards during training for the present invention and three existing baseline methods. From top to bottom, they are the method proposed in the present invention (Multi UAVs Energy Efficiency Fair Communications, MUEEFC), the trajectory planning method under fixed power (MUEEFC-FP), the fixed trajectory method under power optimization (MUEEFC-FT), and the optimization method using the DDPG-based deep reinforcement learning algorithm. Through sufficient training, the MUEEFC method proposed in the present invention is superior to the other three baseline methods. In the initial stage of training, the cumulative fair energy efficiency of the optimization method based on the DDPG algorithm increases the fastest. This is because DDPG only maximizes the expected reward without expected entropy, which leads to poor exploration ability and convergence to a local optimum. In the first 200 epochs, the cumulative fair energy efficiency of the trajectory planning method under fixed power increases significantly compared to the MUEEFC method. There are two reasons to explain this phenomenon. First, compared with the trajectory planning method under fixed power, the MUEEFC method needs to consider power allocation in addition to trajectory design, which increases the difficulty of formulating a reasonable trajectory. Second, the flexible power allocation of the MUEEFC method increases fairness. In 800 training epochs, the MUEEFC method can gradually make appropriate actions for speed and power allocation, and its performance is significantly improved. At the same time, the cumulative fair energy efficiency of the fixed trajectory method under power optimization and the DDPG-based optimization method does not increase significantly, indicating that the MUEEFC method attaches importance to trajectory design and has strong exploration ability. Due to the differences between deep reinforcement learning and supervised learning, and there is no clear label information in deep reinforcement learning, obvious oscillations often occur during the training of all schemes. After 1200 epochs of training, the cumulative fair energy efficiency of the MUEEFC method converges without obvious improvement, but the DDPG-based optimization method still oscillates and does not converge due to the high-dimensional action space.

[0119] Figure 4 This is a simulation result graph of the cumulative fair energy efficiency when evaluating with the present invention, MUEEFC-FP, and MUEEFC-FT under different numbers of users. Each data point is composed of the average values of ten evaluations under five random seeds. When the number of ground users increases, the MUEEFC method proposed in this patent can be used without retraining, greatly improving the generality of the deep reinforcement learning method. As the number of ground users increases, the MUEEFC method always shows excellent performance. Compared with other benchmark algorithms, its total energy efficiency is increased by about 33%.

[0120] It is obvious to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, in any respect, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention. Any reference signs in the claims should not be construed as limiting the claims concerned.

Claims

1. A two-layer UAV communication coverage method based on a deep reinforcement learning network, characterized in that It includes the following steps: (1) Divide the unmanned aerial vehicles (UAVs) into control UAVs and communication UAVs, and cluster the users according to the position information of the users at each moment; (2) Based on the clustering results, match the clusters with the communication UAVs one by one to obtain the user connection information at each moment; (3) Obtain the position of the communication UAV at the current moment under the constraints of collision and speed through the position change of the cluster center; (4) Construct a deep reinforcement learning network and train it; input the position information of the communication UAV as the state into the trained deep reinforcement learning network to obtain the scheduling information of the communication UAV, power allocation, and the position of the control UAV at the current moment; (5) Repeat steps (1) to (4) at each moment to maximize the fair energy efficiency of UAV communication coverage in the area under the conditions of meeting the UAV speed constraint, UAV energy constraint, and user quality of service constraint; The implementation process of constructing the deep reinforcement learning network in step (4) is as follows: Convert the control UAV trajectory planning problem into a Markov decision process, where the state space is expressed as: Among them, q(n) is the position of the UAV at time n; represents the total service time of communication UAV k before time n; v J (n) is the speed of the UAV at time n; The action space is: Among them, λ v , and respectively represent the angle between the speed of the controlled drone, the flight direction and the Z-axis, and the angle between the projection of the flight direction on the XY plane and the X-axis after normalization. {λ k (n)} k∈K is the power distribution set; The reward function is: r(n) = r fee (n) + r vel (n) + r alt (n) where r fee (n) is the fair energy efficiency reward, and r vel (n) is the speed constraint violation penalty, and r alt (n) is the altitude constraint violation penalty; Learn the optimal policy based on the maximum entropy framework to maximize the objective function: Among them, π* represents the optimal policy, α represents the temperature coefficient of the entropy value, and H(π(·∣s t )) represents the entropy of adopting the policy π in the state s t .

2. The double-layer UAV communication coverage method based on a deep reinforcement learning network according to claim 1, wherein The implementation process of clustering the users according to the position information of the users at each moment in step (1) is as follows: Divide the service period T into N sub - time slots, and each sub - time slot is denoted as δ t = T / N; Initialize the centroid C at the current time slot n ; n represents the nth time slot. When n = 0, adopt the k - means++ centroid initialization scheme, randomly initialize the first centroid, and select the remaining centroids according to the selection probability ; M represents the set of ground users, and D(m) represents the shortest distance from user m to the nearest selected centroid; When n≠0, C n = {u k,n-1 ,k∈K}, K represents the set of communication UAVs, and u k,n-1 represents the three - dimensional Cartesian coordinates of communication UAV k at the (n - 1) time slot; Calculate the three-dimensional Cartesian coordinates w of the ground user m at the nth time slot m,n to C n Calculate the distances to each centroid, compare the distances to each centroid, and classify the user into the same class as the centroid with the closest distance to itself; for each cluster L k , update its centroid according to the coordinates of the kth centroid at the nth time slot ​ Calculate the distance between the new and old centroids, and judge whether it is less than the preset threshold. If it is less than the threshold, the calculation ends; Otherwise, repeat the above clustering steps until the distance is less than the threshold.

3. A two - layer UAV communication coverage method based on a deep reinforcement learning network according to claim 1, wherein, The implementation process of step (2) is as follows: The communication UAV and clustering matching problem is formulated as a weighted complete bipartite graph G=(K; L k ; E; W), where K represents the set of communication UAVs, L k represents the k-th cluster after user clustering, E represents the set of edges between communication UAVs and clusters, and W represents the edge weights; Each time, select an unmatched vertex and search for the perfect matching of the current subgraph according to the depth-first search method. When all the edges in the subgraph satisfy f(i)+f(j) = w(i,j), the current matching result is the maximum weight matching of the current subgraph; f(i) and f(j) are the vertex labels of vertices i and j: f(j) = 0, j ∈ L k When the perfect matching of the current subgraph cannot be obtained, by modifying: f(i) = f(i)+Δ, i∈X f(j) = f(j)-Δ, j∈Y Among them, and represent the vertices that have been matched, and Δ represents the modification difference: Loop the above steps until the maximum weight perfect matching between the communication UAV and the clustering is obtained.

4. A two - layer UAV communication coverage method based on a deep reinforcement learning network according to claim 1, characterized in that, The deep reinforcement learning network in step (4) includes two Actor networks and eight Critic networks; two Actor networks are used to online-decide the flight actions and resource allocation actions at each moment to obtain the scheduling information of the communication UAV, power allocation, and the position of the control UAV at the current moment.

5. A two - layer UAV communication coverage method based on a deep reinforcement learning network according to claim 1, characterized in that, The implementation process of training the deep reinforcement learning network in step (4) is as follows: Train the two Actor networks to respectively output the trajectory actions and resource allocation actions of the control UAV, and the loss functions are respectively: Among them, and represent the track actor network parameters and the resource allocation actor network parameters; represents the policy of the track actor network at ; represents the Q value obtained by the critic network under the parameter θ i ; The loss function of the Critic network is defined as: Among them, θ i represents the critic network parameters; is the advantage function, expressed as: The loss function of the temperature coefficient of the entropy value is expressed as: Among them, is the target entropy constant; The parameters of the deep reinforcement learning network are trained and updated according to the above loss functions, and the target Critic network is updated slowly with the Critic network. The update strategy is: Among them, represents the parameters of the target critic network; ε represents the parameters for updating the Critic network.

6. The double-layer UAV communication coverage method based on a deep reinforcement learning network according to claim 3, characterized in that The weight W of the said edge is calculated by w(i,j) = -||u i -C j ||, i ∈ K, j ∈ L k where w(i,j) represents the weight value of the edge between i and j; u i represents the coordinate of the i-th communication UAV; C j represents the centroid coordinate of the j-th cluster.

7. A two-layer UAV communication coverage method based on a deep reinforcement learning network according to claim 1, characterized in that The fair energy efficiency reward r fee (n), the speed constraint violation penalty r vel (n), and the altitude constraint violation penalty r alt (n) are respectively: r vel = κ vel (n)l vel r alt = κ alt (n)l alt Among them, l fee is the fair energy efficiency reward coefficient, is the service fairness index, α k is the communication UAV scheduling information, R k (n) is the achievable transmission rate of the communication UAV, p(v J (n)) represents the energy consumption of the control UAV; κ vel represents the violation of speed constraint indicator, l vel represents the violation of speed constraint penalty coefficient; κ alt is the violation of altitude constraint indicator, l alt is the violation of altitude constraint penalty coefficient.

Images