Double-layer unmanned aerial vehicle communication coverage method based on deep reinforcement learning network
Through the dual-layer UAV communication coverage method based on deep reinforcement learning, the problem of deploying drone-assisted communication networks in dynamic environments is solved, the universality and exploration capabilities of the algorithm are improved, and high-efficiency and efficient UAV communication coverage is achieved. .
Patent Information
- Application Number
- CN202411206410.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-30
AI Technical Summary
The existing drone-assisted communication network is difficult to deploy online in dynamic environments, and the algorithm calculation complexity is high, so it cannot adapt to complex communication scenarios. The algorithms based on deep reinforcement learning are limited in generality and robustness, making it difficult to solve the problem of unmanned Cooperation and competition between machines.
Using a two-layer UAV communication coverage method based on deep reinforcement learning, by dividing the UAV into control UAV and communication UAV, clustering and matching according to user location information, building a deep reinforcement learning network, and using the Actor network and Critic networks optimize drone tracks and resource allocation to maximize fair energy efficiency.
It realizes the online deployment of UAV communication coverage in a dynamic environment, improves the universality and exploration capabilities of the algorithm, increases the cumulative reward by about 100%, and maintains high efficiency and efficiency under different users.
Smart Images

Figure CN119031379A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of unmanned aerial vehicle (UAV) trajectory planning, and specifically relates to a double-layer UAV communication coverage method based on a deep reinforcement learning network. Background Art
[0002] With the increasing number of users accessing the network, the communication infrastructure is obviously insufficient, and the wireless communication network needs higher energy efficiency and more reliable service quality. Due to the advantages of good channel conditions, high mobility, strong flexibility, easy deployment, low cost, and necessary onboard computing power, drones can be regarded as a supplement to ground communication facilities. With the development of various derivative solutions for auxiliary communication, the role of drones in auxiliary communication in communication systems is becoming more and more obvious. Whether in the current cellular network or in the future 6G wireless communication network, drones will play an indispensable role.
[0003] In order to give full play to the advantages of drones as aerial base stations, the network coverage performance can be significantly improved by jointly optimizing the drone trajectory and resource allocation. However, considering the limited onboard energy and computing power, energy efficiency is particularly important for drone-assisted communication networks. For single-drone-assisted communication networks, although they are easy to deploy, they are difficult to adapt to increasingly complex communication scenarios due to limited capabilities. Multi-drone-assisted communication networks can provide higher data rates, but the cooperation and competition between drones have generated many necessary consumption and challenges.
[0004] In existing research, many studies and attempts have been made on drone-assisted communication networks. Most of the work decomposes the original optimization problem into multiple convex optimization sub-problems for iterative solution. However, traditional algorithms based on convex optimization require offline calculations through global information to obtain solutions, which makes it impossible to deploy in dynamic environments. At the same time, the computational complexity of the algorithm also increases sharply with the scale of the optimization problem. Algorithms based on deep reinforcement learning can convert the optimization problem of drone-assisted communication into a Markov decision process and learn the optimal strategy by maximizing the cumulative reward. In algorithms based on deep reinforcement learning, the neural network structure of the algorithm is often related to the number of ground users, which limits the versatility and robustness of the algorithm; in optimization problems in continuous space, the huge exploration space is also difficult to solve. Summary of the invention
[0005] Purpose of the invention: The present invention provides a two-layer UAV communication coverage method based on a deep reinforcement learning network, which realizes UAV trajectory planning for ground user communication coverage by jointly optimizing UAV trajectory, user connection, communication UAV scheduling and power allocation.
[0006] Technical solution: The double-layer UAV communication coverage method based on deep reinforcement learning network described in the present invention comprises the following steps:
[0007] (1) UAVs are divided into control UAVs and communication UAVs, and users are clustered according to their location information at each moment;
[0008] (2) Based on the clustering results, the clusters are matched with the communication drones one by one to obtain the user connection information at each moment;
[0009] (3) Obtain the current position of the communication UAV under collision and velocity constraints through the position change of the cluster center;
[0010] (4) Construct a deep reinforcement learning network and train it; input the location information of the communication UAV as the state into the trained deep reinforcement learning network to obtain the dispatch information, power allocation and current location of the communication UAV;
[0011] (5) Repeat steps (1) to (4) at each moment, so as to maximize the fair energy efficiency of the drone's communication coverage in the area while satisfying the drone's speed constraint, drone energy constraint, and user service quality constraint.
[0012] Furthermore, the process of clustering users according to the location information of the users at each moment in step (1) is as follows:
[0013] Divide the service period T into N sub-time slots, each sub-time slot is represented by δ t =T / N; Initialize the centroid C of the current time slot n ; n represents the nth time slot. When n = 0, the k-means++ centroid initialization scheme is used to randomly initialize the first centroid. Select the remaining centroids; M represents the set of ground users, D(m) represents the shortest distance between user m and the selected nearest centroid; when n≠0, C n = {u k,n-1 ,k∈K}, K represents the communication drone set, u k,n-1 represents the three-dimensional Cartesian coordinates of the communication drone k in time slot n;
[0014] Calculate the three-dimensional Cartesian coordinate w of ground user m in time slot n m,n to C n , compare the distance to each centroid, and classify the user and the centroid closest to themselves into one category; for each cluster L k , according to the coordinates of the kth centroid at the nth time slot Update its centroid;
[0015] Calculate the distance between the new and old centroids to determine whether it is less than a preset threshold. If so, the calculation ends; otherwise, repeat the above clustering steps until the distance is less than the threshold.
[0016] Furthermore, the implementation process of step (2) is as follows:
[0017] The communication drone and cluster matching problem is expressed as a weighted complete bipartite graph G = (K; L k ; E; W), where K represents the communication drone set, L k represents the kth cluster after user clustering, E represents the set of edges between communication drones and clusters, and W represents the weight of the edge;
[0018] Each time an unmatched vertex is selected, a full match of the current subgraph is searched according to the depth-first search method. When all the edges in the subgraph satisfy f(i)+f(j)=w(i,j), the current matching result is the maximum weight matching of the current subgraph; f(i) and f(j) are the top labels of vertices i and j:
[0019]
[0020] f(j)=0,j∈L k
[0021] When a full match of the current subgraph cannot be obtained, modify:
[0022] f(i)=f(i)+Δ,i∈X
[0023] f(j)=f(j)-Δ,j∈Y
[0024] in, and Represents the matched vertices, and Δ represents the modification difference:
[0025]
[0026] The above steps are repeated until the maximum weight of the communication drone and the cluster is completely matched.
[0027] Furthermore, the deep reinforcement learning network in step (4) includes two Actor networks and eight Critic networks; two Actor networks are used to make online decisions on flight actions and resource allocation actions at each moment, respectively, to obtain the scheduling information, power allocation and current position of the communication drone.
[0028] Furthermore, the implementation process of constructing the deep reinforcement learning network in step (4) is as follows:
[0029] The trajectory planning problem of controlling the UAV is converted into a Markov decision process, where the state space is expressed as:
[0030]
[0031] Among them, q(n) is the position of the controlled drone at time n; represents the total service time of communication UAV k before time n; v J (n) is the speed of the drone at time n;
[0032] The action space is:
[0033]
[0034] Among them, λ v , and They represent the normalized controlled drone speed, the angle between the flight direction and the Z axis, and the angle between the projection of the flight direction on the XY plane and the X axis, respectively. k (n)} k∈K is a power allocation set;
[0035] The reward function is:
[0036] r(n)=r fee (n)+r vel (n)+r alt (n)
[0037] Among them, r fee (n) is the fair energy efficiency reward, r vel (n) is the penalty for violating the speed constraint, r alt (n) penalties for violating high-level constraints;
[0038] Based on the maximum entropy framework, the optimal strategy is learned to maximize the objective function:
[0039]
[0040] Among them, π* represents the optimal strategy, α represents the temperature coefficient of entropy, H(π(·|s t )) means in s t The entropy of taking strategy π in the state.
[0041] Furthermore, the training process of the deep reinforcement learning network in step (4) is as follows:
[0042] The two Actor networks are trained to output the trajectory action and resource allocation action of controlling the drone respectively, and the loss functions are:
[0043]
[0044] Among them, φ v and φ p Represents the network parameters of the track actor and the resource allocation actor; Represents the track actor network in φ v The following strategies; Indicates that the critic network has parameters θ i The Q value obtained under
[0045] The loss function of the Critic network is defined as:
[0046]
[0047] Among them, θ i Represents the critic network parameters; is the advantage function, expressed as:
[0048]
[0049] The loss function of the temperature coefficient of entropy is expressed as:
[0050]
[0051] in, is the target entropy constant;
[0052] The parameters of the deep reinforcement learning network are trained and updated according to the above loss function. The target critic network is slowly updated along with the critic network. The update strategy is:
[0053]
[0054] in, represents the parameters of the target critic network; ε represents the updated parameters of the critic network.
[0055] Furthermore, the weight W of the edge is given by w(i,j)=-||u i -C j ||,i∈K,j∈L k Calculated, where w(i,j) represents the weight of the edge between i and j; u i represents the coordinates of the i-th communication drone; C j represents the centroid coordinates of the jth cluster.
[0056] Furthermore, the fair energy efficiency reward r fee (n), speed constraint violation penalty r vel (n) and penalty for violating height constraints r alt (n) are:
[0057]
[0058] r vel =κ vel (n)l vel
[0059] r alt =κ alt (n)l alt
[0060] Among them, l fee is the fair energy efficiency reward coefficient, is the service fairness index, α k Dispatching information for communication drones, R k (n) is the achievable transmission rate of the communication UAV, p(vJ(n)) represents the energy consumption of the control UAV; κ vel Indicates speed constraint violation indicator, l vel represents the penalty coefficient for violating the speed constraint; κ alt is the altitude constraint violation indicator, l alt is the penalty coefficient for violating the height constraint.
[0061] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention first solves the problem that traditional algorithms such as convex optimization cannot be deployed online in dynamic environments, such as considering user mobility; by clustering and matching users, the neural network is decoupled from the number of users, so that the proposed algorithm can be deployed and implemented under different numbers of users without retraining after training, thereby improving the versatility of the algorithm; based on the framework of maximum entropy, the drone environment planning and power allocation are trained and learned through different Actor networks, thereby reducing the exploration space and enhancing the exploration capability. Compared with the DDPG algorithm, the cumulative reward is increased by about 100%. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 is an application scenario diagram of the present invention;
[0063] Figure 2 This is the simulation result diagram of the energy consumption of the communication UAV;
[0064] Figure 3 is a simulation result graph of the cumulative rewards of the algorithm during training;
[0065] Figure 4 It is a diagram of the cumulative fair energy efficiency simulation results when evaluating under different numbers of users. DETAILED DESCRIPTION
[0066] The present invention is further described in detail below with reference to the accompanying drawings.
[0067] The present invention proposes a two-layer UAV communication coverage method based on deep reinforcement learning network. The UAVs are divided into control UAVs and communication UAVs. The fair energy efficiency within the cycle is maximized by jointly optimizing the UAV trajectory, user connection, communication UAV scheduling and power allocation. The specific implementation process is as follows:
[0068] Step 1: Cluster users according to their location information at each moment.
[0069] The service period T is divided into N sub-time slots, each sub-time slot is denoted as δ t =T / N.
[0070] Initialize the centroid C of the current time slot n , where n represents the nth time slot. When n = 0, the k-means++ centroid initialization scheme is used to randomly initialize the first centroid, and then according to the probability formula Select the remaining centroid; where g cods represents the selection probability, M represents the ground user set, and D(m) represents the shortest distance between user m and the nearest centroid. When n≠0, C n = {u k,n-1 ,k∈K}, where K represents the set of communication drones, u k,n-1 Represents the three-dimensional Cartesian coordinates of the communication drone k in time slot n.
[0071] Calculate w m,n to C n The distance to each centroid is compared, and the user and the centroid closest to the user are divided into one category; where w m,n represents the three-dimensional Cartesian coordinates of ground user m in time slot n. For each cluster L k ,according to Update its centroid with the coordinates of the kth centroid at the nth time slot.
[0072] Calculate the distance between the new and old centroids to determine whether it is less than the threshold. If so, the calculation ends. Otherwise, repeat the above clustering steps until the distance is less than the threshold.
[0073] Step 2: Based on the clustering results, match the clusters with the communication drones one by one to obtain the user's scheduling information.
[0074] The communication drone and cluster matching problem is expressed as a weighted complete bipartite graph G = (K; L k ; E; W), where K represents the communication drone set, L k represents the kth cluster after user clustering, E represents the set of edges between communication drones and clusters, and W represents the weight of the edge, which is given by w(i,j)=-||u i-C j ||,i∈K,j∈L k Calculated, where w(i,j) represents the weight of the edge between i and j; u i represents the coordinates of the i-th communication drone; C j represents the centroid coordinates of the jth cluster.
[0075] Each time an unmatched vertex is selected, a full match of the current subgraph is searched according to the depth-first search method. When all the edges in the subgraph satisfy f(i)+f(j)=w(i,j), the current matching result is the maximum weight matching of the current subgraph. Where f(i) and f(j) are defined as the top labels of vertices i and j:
[0076]
[0077] f(j)=0,j∈L k
[0078] When a full match of the current subgraph cannot be obtained, modify:
[0079] f(i)=f(i)+Δ,i∈X,
[0080] f(j)=f(j)-Δ,j∈Y
[0081] in, and represents the matched vertices, and Δ represents the modified difference;
[0082]
[0083] The above steps are repeated until the maximum weight of the communication drone and the cluster is completely matched.
[0084] Step 3: Based on the obtained cluster centroid, the current position of the communication UAV is obtained under the constraints of UAV speed and collision.
[0085] Step 4: The location information of the communication UAV is input as the state into the pre-trained deep reinforcement learning network, including 2 Actor networks and 8 Critic networks. The 2 Actor networks are used to make online decisions on the flight actions and resource allocation actions at each moment, respectively, to obtain the scheduling information, power allocation and current position of the communication UAV.
[0086] The training process of the deep reinforcement learning network includes: converting the trajectory planning problem of controlling the UAV into a Markov decision process, designing the state space, action space and reward function of the model; decomposing the agent into two sub-agents, outputting the trajectory action and resource allocation action of controlling the UAV respectively; training the reinforcement learning network based on the maximum entropy framework to obtain a trained network. The specific process is as follows:
[0087] The trajectory planning problem of controlling the UAV is converted into a Markov decision process, where the state space s(n) is expressed as:
[0088]
[0089] Among them, q(n) is the position of the controlled drone at time n; represents the total service time of communication UAV k before time n; v J (n) is the speed of the drone at time n.
[0090] The action space a(n) is expressed as:
[0091]
[0092] Among them, λ v , and They represent the normalized controlled drone speed, the angle between the flight direction and the Z axis, and the angle between the projection of the flight direction on the XY plane and the X axis, respectively. k (n)} k∈K The power allocation set.
[0093] The reward function r(n) is:
[0094] r(n)=r fee (n)+r vel (n)+r alt (n)
[0095] Among them, r fee (n) is the fair energy efficiency reward, r vel (n) is the penalty for violating the speed constraint, r alt (n) Penalties for violating high-level constraints, where:
[0096]
[0097] r vel =κ vel (n)l vel
[0098] r alt =κ alt (n)l alt
[0099] Among them, l fee is the fair energy efficiency reward coefficient, is the service fairness index, α k Dispatching information for communication drones, R k (n) is the achievable transmission rate of the communication drone, p(v J (n)) represents the energy consumption of controlling the UAV; κ vel Indicates speed constraint violation indicator, l vel represents the penalty coefficient for violating the speed constraint; κ alt is the altitude constraint violation indicator, l alt is the penalty coefficient for violating the height constraint.
[0100] Based on the maximum entropy framework, the optimal strategy is learned to maximize the objective function:
[0101]
[0102] Among them, π* represents the optimal strategy, α represents the temperature coefficient of entropy, H(π(·|s t )) means in s t The entropy of taking strategy π in the state.
[0103] Train two Actor networks to output the trajectory action and resource allocation action of controlling the drone respectively, and the loss functions are:
[0104]
[0105] Among them, φv v and φ p Represents the network parameters of the track actor and the resource allocation actor; Represents the track actor network in φ v The following strategies; Indicates that the critic network has parameters θ i The Q value obtained is:
[0106] The loss function of the Critic network is defined as:
[0107]
[0108] Among them, θ i Represents the critic network parameters; is the advantage function, expressed as:
[0109]
[0110] The loss function of the temperature coefficient α of the entropy value is expressed as:
[0111]
[0112] in, is the target entropy constant.
[0113] The parameters of the deep reinforcement learning network are trained and updated according to the above loss function, where the target critic network is slowly updated with the critic network, and the update strategy is:
[0114]
[0115] in, represents the parameters of the target critic network; ε represents the parameters of the soft update.
[0116] Step 5: Repeat the above steps at each moment to maximize the energy efficiency of the drone's communication coverage in the area while satisfying the drone's speed constraints, drone energy constraints, and user service quality constraints.
[0117] Figure 1 The present invention provides a scenario in which a double-layer UAV-assisted communication network maximizes the energy efficiency of communication coverage within a region under the conditions of satisfying UAV speed constraints, UAV energy constraints, and user minimum rate requirement constraints. Figure 2 This is a simulation result diagram of the energy consumption of a communication drone using the present invention and the communication drone trajectory planning method based on K-means clustering. Compared with the k-means algorithm, the present invention takes into account the time-varying motion of ground users and optimizes the selection of the initial centroid, while the k-means algorithm only considers the internal distance. Based on this, the total flight distance of the communication drone using the present invention is reduced by 85%. Due to the reduction in the total flight distance, the total energy consumption of the communication drone using the present invention is 60% of that of the traditional k-means algorithm in the figure. At the same time, it is shown that the hovering energy consumption of the communication drone accounts for a large part of the total energy consumption.
[0118] Figure 3This is a simulation result diagram of the cumulative reward of the present invention and the three existing baseline methods during training. From top to bottom, they are the method proposed by the present invention (Multi UAVs Energy Efficiency Fair Communications, MUEEFC), the trajectory planning method under fixed power (MUEEFC-FP), the fixed trajectory method under power optimization (MUEEFC-FT), and the optimization method based on the DDPG deep reinforcement learning algorithm. Through sufficient training, the MUEEFC method proposed by the present invention is superior to the other three baseline methods. In the early stage of training, the cumulative fair energy efficiency of the optimization method based on the DDPG algorithm grows fastest. This is because DDPG only maximizes the expected return without expected entropy, which leads to poor exploration ability and convergence to local optimum. In the first 200 cycles, the cumulative fair energy efficiency of the trajectory planning method under fixed power is significantly higher than that of the MUEEFC method. There are two reasons to explain this phenomenon. First, compared with the trajectory planning method under fixed power, the MUEEFC method needs to consider the power allocation outside the trajectory design, which increases the difficulty of formulating a reasonable trajectory. Second, the flexible power allocation of the MUEEFC method increases fairness. During 800 training cycles, the MUEEFC method can gradually make appropriate actions for speed and power distribution, and the performance is significantly improved. At the same time, the cumulative fair energy efficiency of the fixed trajectory method under power optimization and the optimization method based on DDPG has no obvious improvement, indicating that the MUEEFC method attaches importance to trajectory design and has strong exploration capabilities. Due to the difference between deep reinforcement learning and supervised learning, deep reinforcement learning does not have clear label information, so there are often obvious oscillations in the training of all schemes. After 1200 cycles of training, the cumulative fair energy efficiency of the MUEEFC method converged without obvious improvement, but the optimization method based on DDPG still oscillated and did not converge due to the high-dimensional action space.
[0119] Figure 4 This is a diagram of the cumulative fair energy efficiency simulation results when the present invention, MUEEFC-FP and MUEEFC-FT are evaluated under different numbers of users. Each data point consists of five random seeds and the average value of ten evaluations under each random seed. When the number of ground users increases, the MUEEFC method proposed in this patent can be used without retraining, which greatly improves the versatility of the deep reinforcement learning method. As the number of ground users increases, the MUEEFC method always shows excellent performance, and its total energy efficiency is improved by about 33% compared with other benchmark algorithms.
[0120] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
Claims
1. A two-layer UAV communication coverage method based on deep reinforcement learning network, characterized in that: The following steps are involved: (1) UAVs are divided into control UAVs and communication UAVs, and users are clustered according to their location information at each moment; (2) Based on the clustering results, the clusters are matched with the communication drones one by one to obtain the user connection information at each moment; (3) Obtain the current position of the communication UAV under collision and velocity constraints through the position change of the cluster center; (4) Construct a deep reinforcement learning network and train it; input the location information of the communication UAV as the state into the trained deep reinforcement learning network to obtain the dispatch information, power allocation and current location of the communication UAV; (5) Repeat steps (1) to (4) at each moment, so as to maximize the fair energy efficiency of the drone's communication coverage in the area while satisfying the drone's speed constraint, drone energy constraint, and user service quality constraint.
2. According to claim 1, a double-layer UAV communication coverage method based on deep reinforcement learning network is characterized in that: The process of clustering users according to the location information of users at each moment in step (1) is as follows: Divide the service period T into N sub-time slots, each sub-time slot is represented by δ t =T / N; Initialize the centroid C of the current time slot n ; n represents the nth time slot. When n = 0, the k-means++ centroid initialization scheme is used to randomly initialize the first centroid. Select the remaining centroids; M represents the set of ground users, D(m) represents the shortest distance between user m and the selected nearest centroid; when n≠0, C n = {u k,n-1 ,k∈K}, K represents the communication drone set, u k,n-1 represents the three-dimensional Cartesian coordinates of the communication drone k in time slot n; Calculate the three-dimensional Cartesian coordinate w of ground user m in time slot n m,n to C n , compare the distance to each centroid, and classify the user and the centroid closest to themselves into one category; for each cluster L k , according to the coordinates of the kth centroid at the nth time slot Update its centroid; Calculate the distance between the new and old centroids to determine whether it is less than a preset threshold. If so, the calculation ends. Otherwise, the above clustering steps are repeated until the distance is less than the threshold.
3. According to the double-layer UAV communication coverage method based on deep reinforcement learning network according to claim 1, it is characterized in that: The implementation process of step (2) is as follows: The communication drone and cluster matching problem is expressed as a weighted complete bipartite graph G = (K; L k ; E; W), where K represents the communication drone set, L k represents the kth cluster after user clustering, E represents the set of edges between communication drones and clusters, and W represents the weight of the edge; Each time an unmatched vertex is selected, a full match of the current subgraph is searched according to the depth-first search method. When all the edges in the subgraph satisfy f(i)+f(j)=w(i, j), the current matching result is the maximum weight matching of the current subgraph; f(i) and f(j) are the top labels of vertices i and j: f(j)=0,j∈L When a full match of the current subgraph cannot be obtained, modify: f(i)=f(i)+Δ,i∈X f(j)=f(j)-Δ,j∈Y in, and Represents the matched vertices, and Δ represents the modification difference: The above steps are repeated until the maximum weight of the communication drone and the cluster is completely matched.
4. According to the double-layer UAV communication coverage method based on deep reinforcement learning network in claim 1, it is characterized in that: The deep reinforcement learning network in step (4) includes two Actor networks and eight Critic networks; two Actor networks are used to make online decisions on flight actions and resource allocation actions at each moment, respectively, to obtain the scheduling information, power allocation and current position of the communication drone.
5. According to the double-layer UAV communication coverage method based on deep reinforcement learning network in claim 1, it is characterized in that: The implementation process of building a deep reinforcement learning network in step (4) is as follows: The trajectory planning problem of controlling the UAV is converted into a Markov decision process, where the state space is expressed as: Among them, q(n) is the position of the controlled drone at time n; represents the total service time of communication UAV k before time n; v J (n) is the speed of the drone at time n; The action space is: Among them, λ v , and They represent the normalized controlled drone speed, the angle between the flight direction and the Z axis, and the angle between the projection of the flight direction on the XY plane and the X axis, respectively. k (n)} k∈K is a power allocation set; The reward function is: r(n)=r fee (n)+r vel (n)+r alt (n) Among them, r fee (n) is the fair energy efficiency reward, r vel (n) is the penalty for violating the speed constraint, r alt (n) penalties for violating high-level constraints; Based on the maximum entropy framework, the optimal strategy is learned to maximize the objective function: Among them, π * represents the optimal strategy, α represents the temperature coefficient of entropy, H(π(·|s t )) means in s t The entropy of taking strategy π in the state.
6. According to the double-layer UAV communication coverage method based on deep reinforcement learning network in claim 1, it is characterized in that: The training process of the deep reinforcement learning network in step (4) is as follows: The two Actor networks are trained to output the trajectory action and resource allocation action of controlling the drone respectively, and the loss functions are: Among them, φ v and φ p Represents the network parameters of the track actor and the resource allocation actor; Represents the track actor network in φ v The following strategies; Indicates that the critic network has parameters θ i The Q value obtained under The loss function of the Critic network is defined as: Among them, θ i Represents the critic network parameters; is the advantage function, expressed as: The loss function of the temperature coefficient of entropy is expressed as: in, is the target entropy constant; The parameters of the deep reinforcement learning network are trained and updated according to the above loss function. The target critic network is slowly updated along with the critic network. The update strategy is: in, represents the parameters of the target critic network; ε represents the parameters updated by the critic network.
7. A double-layer UAV communication coverage method based on deep reinforcement learning network according to claim 3, characterized in that: The weight W of the edge is given by w(i, j) = -||u i -C j ||, i∈K, j∈L k Calculated, where w(i, j) represents the weight of the edge between i and j; u i represents the coordinates of the i-th communication drone; C j represents the centroid coordinates of the jth cluster.
8. The double-layer UAV communication coverage method based on deep reinforcement learning network according to claim 5 is characterized in that: The fair energy efficiency reward r fee (n), speed constraint violation penalty r vel (n) and penalty for violating height constraints r alt (n) are: r vel =k vel (n)l vel r alt =k alt (n)l alt Among them, l fee is the fair energy efficiency reward coefficient, is the service fairness index, α k Dispatching information for communication drones, R k (n) is the achievable transmission rate of the communication drone, p(v J (n)) represents the energy consumption of controlling the UAV; κ vel Indicates speed constraint violation indicator, l vel represents the penalty coefficient for violating the speed constraint; κ alt is the altitude constraint violation indicator, l alt is the penalty coefficient for violating the height constraint.
Citation Information
Patent Citations
Unmanned aerial vehicle trajectory and power joint optimization method based on deep reinforcement learning
CN111263332A
Unmanned aerial vehicle network hovering position optimization method based on multi-agent deep reinforcement learning
CN111786713A
Forest unmanned aerial vehicle network deployment method and system based on hierarchical clustering
CN113543068A
Unmanned aerial vehicle cluster intelligent coverage method and system based on improved reinforcement learning algorithm
CN117202207A
Distributed Deep Reinforcement Learning Framework for Software-Defined Unmanned Aerial Vehicle Network Control
US20240205095A1
Cited By
Track design method and system for wireless energy transmission network of unmanned aerial vehicle cluster
CN119997029A
Trajectory design method and system of unmanned aerial vehicle cluster wireless energy transmission network
CN119997029B
