Multi-uav base station cooperative coverage path planning method based on deep reinforcement learning

Abstract

The application discloses a kind of multi-unmanned aerial vehicle base station cooperative coverage path planning methods based on deep reinforcement learning, steps include: first define the Markov model based on deep reinforcement learning, the Markov decision process five-tuple is modeled;Then according to modeling proposes deep deterministic policy gradient DDPG algorithm;Then the experience cache pool of DDPG algorithm is improved, by classifying the experience data stored in experience cache pool, the experience data obtained is put into different experience cache pool, and the improved DDPG algorithm can solve the problem of unstable convergence;Finally, a simulation environment is designed, the unmanned aerial vehicle group interacts with the environment, and training data is obtained. Through this method, the unmanned aerial vehicle group realizes the target task of cooperative coverage of ground nodes under the limitation of multiple constraints, and the method can make the unmanned aerial vehicle group have higher planning efficiency and lower flight cost.

Description

TECHNICAL FIELD

[0001] The application provides a multi-unmanned aerial vehicle base station cooperative coverage path planning method based on deep reinforcement learning, and belongs to the field of computer artificial intelligence. BACKGROUND

[0002] Unmanned aerial vehicles have the advantages of high mobility, flexible deployment and low cost, and have been widely used in terrain coverage, agricultural production, environmental reconnaissance, aerial rescue, disaster warning and other industries. Unmanned aerial vehicles can be used as air base stations to enhance the coverage range and performance of communication networks in various scenarios. When the ground communication network is interrupted unexpectedly, unmanned aerial vehicles can be quickly deployed, and the unmanned aerial vehicles will establish a communication link with the ground to transmit data and realize cooperative interaction with the ground network. The coverage path planning algorithm is an important technology to support the successful application of unmanned aerial vehicles in the above complex scenarios.

[0003] In the process of planning the path of unmanned aerial vehicles covering ground nodes, the energy constraint condition of the unmanned aerial vehicles needs to be considered, and at the same time, the unmanned aerial vehicles need to ensure signal transmission with ground base stations during task execution, but signal transmission will cause loss and affect the quality of service of coverage. On the other hand, due to the constraints of energy and communication, a single unmanned aerial vehicle is difficult to be applied to large-scale ground coverage tasks, and cooperative flight of multiple unmanned aerial vehicles is an effective solution to realize large-scale coverage tasks, which requires the unmanned aerial vehicles to keep communication connectivity at all times. Therefore, under the constraints of energy consumption limitation, communication connectivity distance limitation and signal transmission loss, how to efficiently realize cooperative coverage of ground nodes is a challenging theoretical and application problem. SUMMARY

[0004] To solve the problem of how to realize efficient cooperative coverage of the ground under multiple constraints, the application provides a multi-unmanned aerial vehicle base station cooperative coverage path planning method based on deep reinforcement learning, which specifically comprises the following steps:

[0005] Step one, define a Markov model: model the Markov decision process five-tuple (S, A, P, R, γ);

[0006] Step two, based on the Markov decision process five-tuple (S, A, P, R, γ) obtained in step one, design a deep deterministic policy gradient (DDPG) algorithm using basic deep reinforcement learning;

[0007] Step three, improve the experience cache pool of the DDPG algorithm, and classify the experience data stored in the experience cache pool to put the obtained experience data into different experience cache pools.

[0008] Step four, design a simulation environment, the UAV group interacts with the environment, obtains training data, samples training data for simulation training, and realizes the cooperative coverage path planning of the target ground node.

[0009] The specific steps of step one include:

[0010] Step 1.1, determine the state S of the UAV:

[0011] The entire target area is divided into I x J cells, and m fixed-position ground nodes and n UAVs flying at a fixed height H are randomly distributed in the area. The coordinates of UAV i at time t are represented as The position coordinates of the u-th ground node are represented as q u =(x u ,y u ). The fixed total energy of a UAV is e max , the energy consumption of a UAV moving one unit is e1, and the energy consumption of hovering to cover a ground node is e2, both e1 and e2 are constants, and the UAV must complete the task before the energy is exhausted. Therefore, the energy consumption of UAV i from the initial position to the position at time t is :

[0012]

[0013]

[0014] wherein, is the number of ground nodes covered by UAV i at time t;

[0015] The communication radius of each UAV is fixed as R s Due to the limitation of communication connectivity, UAV i must always remain within the communication radius of the nearest UAV j, and the formula is:

[0016] min(||p i -p j ||,i≠j)<=R s

[0017] The process of UAV signal propagation to the ground node will cause channel fading, and signal loss will affect the service quality of the covered ground node. If there are buildings, trees and other obstacles around, it will cause additional loss based on channel fading, and the probability formula of the line-of-sight (LoS) link between the UAV and the ground is as follows:

[0018]

[0019] wherein, f and g are constants related to the type of environment, H represents the height of the UAV, d iuis the horizontal distance between the ith UAV and the u-th ground node, and the formula is

[0020]

[0021] The probability formula of the non-line-of-sight (NLoS) link is

[0022] P NLoS = 1 - P LoS

[0023] The LoS and NLoS link loss models are

[0024]

[0025]

[0026]

[0027] where c is the propagation speed of light, f c is the carrier frequency, ω iu is the distance between the ith UAV and the u-th ground node, η LoS , η NLoS are the additional losses of the line-of-sight (LoS) link and the non-line-of-sight (NLoS) link. Under the LoS and NLoS models, the signal loss formula of the j-th ground node is as follows:

[0028]

[0029] L u ≤ κ

[0030] To ensure the quality of service in the process of covering the ground nodes by the UAV, the signal loss received by each ground node in the covering process must be less than or equal to a certain threshold k, and then the ground node will be successfully covered, otherwise the covering of the node fails.

[0031] The state contains the following parts: at time t, the position and energy consumption of the ith UAV, and the signal loss received by each ground node. Therefore, the state of the ith UAV at time t is as follows:

[0032]

[0033] Step 1.2, determine the action set A of the UAV:

[0034] The flight speed of the UAV i is fixed during the flight process, and the next moving direction can be a t ∈ (0, 2π) or the hovering action a t= 0. Where, hovering action means that the UAV needs to keep the current position unchanged after covering the ground node. So, the action of the UAV i is:

[0035] a t ∈ [0, 2π)

[0036] Step 1.3, define the state transition probability function P of the UAV reaching the next input state s' under the condition of the state s of the UAV at time t and taking action a:

[0037]

[0038] Step 1.4, determine the reward function R of the UAV:

[0039] Set the set of ground node coverage states B = {b1, b2,..., bu,..., bn}. Where b u is the coverage state of the u-th ground node, and is the Boolean domain {0, 1}. If b u = 1, the ground node has been covered by the UAV, and b u = 0, it has not been covered. The coverage rate is the ratio of the number of ground nodes that have been covered to the total number of ground nodes, and at time t, the coverage rate is: u ,...,b m}. Where b u is the coverage state of the u-th ground node, and is the Boolean domain {0, 1}. If b u = 1, the ground node has been covered by the UAV, and b u = 0, it has not been covered. The coverage rate is the ratio of the number of ground nodes that have been covered to the total number of ground nodes, and at time t, the coverage rate is:

[0040]

[0041] The coverage range of each UAV is a circle with a radius R c , and the coverage effect of the UAV on the target node decreases from the center to the periphery. When the UAV is directly above the ground node, the coverage effect is most obvious. The effect degree of the u-th ground node being covered for the first time is The formula is:

[0042]

[0043] Where λ is the coverage effect constant.

[0044] Planning the optimal path needs to realize the transition of the ground node from the initial state to the target state. The initial state of the ground node is the uncovered state, and the target state is the covered state by the UAV. The coverage efficiency E is designed as a cooperative formula of the coverage ground node rate and the coverage effect. The formula is: c The formula is:

[0045]

[0046] Define the reward function, which represents the feedback obtained after selecting a certain action in the current state of the UAV. The basic reward formula is:

[0047]

[0048] where the coverage increment: Δα t = α t - α t-1 , the energy consumption increment of the ith UAV:

[0049] If only positive reward values are given when the UAV swarm successfully completes the task, the reward is too sparse, and it is difficult to obtain good results after multiple rounds of training. Therefore, additional rewards and punishments are added so that the reward is no longer sparse. In the additional reward and punishment setting, when the overall coverage rate does not reach the expected value α ev , a -5 punishment will be made, and when the overall coverage rate reaches our expected value, no punishment will be made; the coverage reward of each ground node covered by the UAV swarm for the first time will be +0.1, and if a UAV exceeds the energy budget during flight, each UAV will be given a -1 punishment, and if the UAVs cannot be connected, a -1 punishment will be made. The additional reward and punishment amount is r extra , and the reward value calculation method is as follows:

[0050]

[0051] Step 1.5, define the discount factor γ, where γ ∈ (0, 1). Calculate the cumulative reward value during the entire process, and the reward value will be discounted over time. The larger the discount factor, the more emphasis on long-term returns.

[0052] The specific steps of step two include:

[0053] Step 2.1, use the actor-critic framework, one network is the actor Actor, and the other network is the critic Critic, and the two networks stimulate and compete with each other. Randomly initialize the network state-action value function Q(s, a | θ Q ) of the Critic network and the policy function μ(s, a | θ μ ) of the Actor network, copy the weights of the Critic network and the Actor network to the target network parameters of the respective networks, i.e., θ Q → θ Q′ , θ μ → θ μ′ , where θ Q , θ μ represent the Critic network parameters and the Actor network parameters, respectively, and θ Q′ , θ μ′ represent the Critic target network parameters and the Actor target network parameters, respectively.

[0054] Step 2.2, when the task starts, the initial state of the UAV i is

[0055]

[0056] As the task proceeds, a current state s t is made, an action a t is made, the formula is:

[0057] a t = μ(s t | θ μ ) + β

[0058] where β is a random noise. The action a t is performed, a reward r t and a new state s t+1 are obtained.

[0059] Step 2.3, obtain experience tuples (s t , a t , r t , s t+1 ) from step 2.2. Save the experience tuples in the experience pool, the newly saved experience tuples are stored in the first position of the experience pool, and the original experience tuples in the experience pool are moved one position at a time; randomly extract part of the samples from the experience pool for training, assuming that (s i , a i , r i , s i+1 ) is a batch of randomly sampled data, TDtarget training is performed, and the target network Y i is represented as:

[0060] Y i = r i + γQ'(s i+1 , μ'(s i+1 | θ μ′ ) | θ Q′ )

[0061] Where μ' represents the policy obtained by analyzing s i+1 , and Q' represents the state-action value obtained by taking the μ' policy at s i+1 .

[0062] Step 2.4, update the Critic network, calculate the minimum loss function L as:

[0063]

[0064] Where N represents the number of random samples extracted from the experience pool for action exploration.

[0065] Step 2.5, update the Actor network parameter θ μ , using the function of the policy gradient descent algorithm is:

[0066]

[0067] wherein denotes the Critic network state-action value function gradient, denotes the Actor network policy function gradient, μ(s i ) denotes the action policy selected at the input state s i of the Actor network, denotes the Critic network state-action value function at state s i , denotes the Actor network policy function at state s i .

[0068] Step 2.6, calculate the target network values with the copy network, and the weight parameters of these target networks are updated by tracking the learning network delay. At the same time, the corresponding Critic and Actor target networks are updated step by step using the current network parameters:

[0069] θ Q′ ← τθ Q + (1-τ)θ Q

[0070] θ μ′ ← τθ μ + (1-τ)θ μ

[0071] wherein τ denotes the update proportion coefficient, τ ∈ (0, 1).

[0072] The specific steps of Step Three include:

[0073] Step 3.1, divide the experience pool into M success and M failure , respectively, to store successful and failed flight experiences, and set a temporary experience pool M temp to store the latest flight experience. The size of M temp is fixed, and once it is full, the earliest experience in it will be taken out and stored in the experience pool M success according to the principle of first-in first-out, and the latest flight experience will continue to be stored in the experience pool M temp . Repeat this process, and finally according to the final state of the unmanned aerial vehicle, extract several experiences from the experience pools M success and M failure respectively, and train the neural network.

[0074] Step 3.2, in order to extract more valuable experiences from the experience pool M success , set the proportion sampling from the two experience pools:

[0075]

[0076] wherein, η success , η failure are the number of samples drawn from the experience pool M success and M failure respectively, ψ is the total number of samples, β∈[0,1] is the success sample rate, and represents the probability of drawing experience from the experience pool M success .

[0077] The technical scheme of the present application has the following advantages:

[0078] 1. The method of the present application establishes a multi-UAV cooperative coverage scenario model, and the UAV group interacts with the environment to obtain training data and autonomously plans an optimal path. The simulation environment in the process has high practical application value.

[0079] 2. The method of the present application uses the deep deterministic policy gradient (DDPG) algorithm, and improves the DDPG algorithm by classifying the experience data stored in the experience cache pool, effectively solving the continuous control problem of the UAV, improving the success rate of obtaining samples during the task process, and achieving better convergence effect.

[0080] 3. The method of the present application has better coverage efficiency, and at the same time realizes the overall energy consumption balance, so that the task flight cost is lower and the completion time is shorter.

[0081] Through this method, the UAV group realizes the target task of cooperative coverage of ground nodes under the restriction of multiple constraints. The method can make the UAV group have higher planning efficiency and lower flight cost. BRIEF DESCRIPTION OF DRAWINGS

[0082] Figure 1 is the flowchart of the whole method of the present application;

[0083] Figure 2 is a schematic diagram of the application scenario of the present application;

[0084] Figure 3 is a comparison effect diagram of the coverage efficiency of the UAV group under different coverage rates under four kinds of algorithms;

[0085] Figure 4 is a comparison effect diagram of the balance degree of energy used by the UAV group during flight under four kinds of algorithms. DETAILED DESCRIPTION

[0086] The technical scheme of the present application will be further described in detail below in combination with the drawings:

[0087] Those skilled in the art will understand that, unless otherwise defined, all terms used herein including technical and scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0088] The present application aims at the problems of more and more movement constraints of unmanned aerial vehicles (UAVs), high flight cost and poor movement continuity when the UAVs cooperatively perform coverage tasks, proposes an improved DDPG algorithm based on deep reinforcement learning, constructs a deep neural network to take continuous state and action space as network input, and uses an actor-critic mechanism to improve the amplitude of policy update. Meanwhile, the experience data stored in the experience cache pool is classified to improve the DDPG algorithm. Finally, the cooperative coverage path planning and dynamic adjustment of the UAV group are realized, and higher planning efficiency and lower flight cost are obtained.

[0089] The improved DDPG algorithm model and application structure are as shown in Figure 1 .

[0090] Specifically, the following steps are included:

[0091] Step one, define the Markov model: model the five-tuple (S, A, P, R, γ) of the Markov decision process, and the specific steps are as follows:

[0092] Step 1.1, determine the state S of the UAV:

[0093] The entire target area is divided into I x J cells, and m fixed-position ground nodes and n UAVs flying at a fixed height H are randomly distributed in the area. The coordinates of the UAV i at time t are represented as The position coordinates of the u-th ground node are represented as q u = (x u , y u ). The total energy of a UAV is e max , the energy consumption of a UAV moving one unit is e1, and the energy consumption of a UAV hovering to cover a ground node is e2, both e1 and e2 are constants, and the UAV must complete the task before the energy is exhausted. Therefore, the energy consumption of the UAV i from the initial position to the position at time t is :

[0094]

[0095]

[0096] wherein, is the number of ground nodes covered by UAV i at time t;

[0097] The communication radius of each UAV is fixed as R s Due to the limitation of communication connectivity, UAV i must always keep within the communication radius of the nearest UAV j, with the formula:

[0098] min(||p i -p j ||, i≠j) <= R s

[0099] The process of UAV propagating signals to ground nodes will cause channel fading, and signal loss will affect the quality of service when the ground nodes are covered. If there are buildings, trees and other obstacles around, it will bring additional loss on the basis of channel fading. The probability formula of the line-of-sight (LoS) link between the UAV and the ground is as follows:

[0100]

[0101] where f and g are constants related to the type of environment, H represents the height of the UAV, d iu is the horizontal distance between the ith UAV and the u-th ground node, with the formula:

[0102]

[0103] The probability formula of the non-line-of-sight (NLoS) link is as follows:

[0104] P NLoS = 1 - P LoS

[0105] The LoS and NLoS link loss models are as follows:

[0106]

[0107]

[0108]

[0109] where c is the propagation speed of light, f c is the carrier frequency, ω iu is the distance between the ith UAV and the u-th ground node, η LoS , η NLoS are the additional losses of the LoS and NLoS links. Under the LoS and NLoS models, the signal loss formula of the j-th ground node is as follows:

[0110]

[0111] L u ≤κ

[0112] In order to ensure the quality of service in the process of unmanned aerial vehicle covering ground nodes, the signal loss suffered by each ground node in the covering process must be less than or equal to a certain threshold k, then this ground node will be successfully covered, otherwise covering this node fails.

[0113] The state contains the following parts: at time t, the position and energy consumption of unmanned aerial vehicle i, and the signal loss suffered by each ground node. So the state of unmanned aerial vehicle i at time t is:

[0114]

[0115] Step 1.2, determine the action set A of unmanned aerial vehicle:

[0116] The flight speed of unmanned aerial vehicle i is fixed in the flight process, the next moving direction can be a t ∈(0, 2π) or hovering action a t = 0. Wherein, the hovering action refers to the unmanned aerial vehicle needs to keep the current position unchanged after covering the ground node. Therefore, the action of unmanned aerial vehicle i is:

[0117] a t ∈[0, 2π)

[0118] Step 1.3, define the state transition probability function P of unmanned aerial vehicle under the condition that the state s at time t and the action a are taken, which can reach the next input state s':

[0119]

[0120] Step 1.4, determine the reward function R of unmanned aerial vehicle:

[0121] Set the set of ground node covering states B = {b1, b2,..., bu,..., b u ,..., b m}. Where b u is the covering state of the u-th ground node, which is a Boolean field {0, 1}. If b u = 1, this ground node has been covered by unmanned aerial vehicle, b u = 0, which is not covered. The coverage rate is the ratio of the number of ground nodes that have been covered to the total number of ground nodes. At time t, the coverage rate is:

[0122]

[0123] The coverage range of each unmanned aerial vehicle is a radius of R cthe target node from the center to the periphery of the circle, and the coverage effect is most obvious when the UAV is directly above the ground node. The coverage effect degree of the u-th ground node when it is first covered The formula is:

[0124]

[0125] where λ is the coverage effect constant.

[0126] Planning the optimal path requires the ground nodes to be transformed from the initial state to the target state. The initial state of the ground node is the uncovered state, and the target state is the covered state by the UAV. The coverage efficiency E is designed as a cooperative formula of the coverage rate and the coverage effect. The coverage efficiency E c The formula is:

[0127]

[0128] The reward function is defined as the feedback obtained after selecting a certain action in the current state of the UAV. The basic reward formula is:

[0129]

[0130] where the coverage rate increment: Δα t = α t - α t-1 The energy consumption increment of the i-th UAV is:

[0131] If positive reward values are only given when the UAV swarm successfully completes the task, the reward is too sparse, and it is difficult to obtain good results after multiple rounds of training. Therefore, additional rewards and punishments are added so that the reward is no longer sparse. In the additional reward and punishment setting, when the overall coverage rate does not reach the expected value α ev , a -5 punishment will be made, and when the overall coverage rate reaches the expected value, no punishment will be made; the coverage reward of each ground node first covered by the UAV swarm will be +0.1, and if a UAV exceeds the energy consumption budget during the flight process, each UAV will be given a -1 punishment, and if the UAVs cannot be connected, a -1 punishment will be made. The additional reward and punishment amount is r extra , and the reward value calculation method is as follows:

[0132]

[0133] Step 1.5, define the discount factor γ, where γ ∈ (0, 1). Calculate the cumulative reward value in the entire process, and the reward value will be discounted over time. The larger the discount factor, the more emphasis on long-term returns.

[0134] Step two, based on the Markov decision process quintuple (S, A, P, R, γ) modeled in step one, a deep deterministic policy gradient (DDPG) algorithm using deep reinforcement learning is designed, the specific steps are as follows:

[0135] Step 2.1, adopt the actor-critic framework, one network is the actor, another network is the critic, two networks stimulate each other and compete with each other. Randomly initialize the network state-action value function Q(s, a|θ Q ) of the critic network and the policy function μ(s, a|θμ) of the actor network, copy the weights of the critic network and the actor network to the target network parameters of the respective networks, i.e. θ Q →θ Q′ , θ μ →θ μ′ , where θ Q , θ μ represent the parameters of the critic network and the actor network respectively, and θ Q′ , θ μ′ represent the target network parameters of the critic and the actor respectively.

[0136] Step 2.2, at the beginning of the task, the initial state of the unmanned aerial vehicle i is

[0137]

[0138] With the task, according to the current state s t , make action a t , the formula is:

[0139] a t = μ(s t |θμ) + β

[0140] Where β is random noise. Perform action a t , get reward r t and new state s t+1 .

[0141] Step 2.3, get experience (s t , a t , r t , s t+1 ) from step 2.2. Save the experience in the experience pool, the newly saved experience is stored in the first position of the experience pool, and the original experience in the experience pool is moved one position at a time; randomly extract part of the samples from the experience pool for training, assuming (s i , a i , r i , si+1 TDtarget training for a batch of randomly sampled data, target network Y i is represented as:

[0142] Y i = r + γQ'(s i , μ'(s i+1 | θ i+1 ) ′ ) Q′

[0143] where μ' represents the policy obtained by analyzing s i+1 , and Q' represents the state-action value obtained by taking the μ' policy at s i+1 .

[0144] Step 2.4, update the Critic network, calculate the minimum loss function L as:

[0145]

[0146] where N represents the number of random samples extracted from the experience pool for action exploration.

[0147] Step 2.5, update the Actor network parameter θ μ , using the policy gradient descent algorithm function

[0148]

[0149] where represents the Critic network state-action value function gradient, represents the Actor network policy function gradient, μ(s i ) represents the action policy selected at the input state s i of the Actor network, represents the Critic network state-action value function at state s i , and represents the Actor network policy function at state s i .

[0150] Step 2.6, calculate the target network values using the replica network, and the weight parameters of these target networks are updated by tracking the learning network delay. At the same time, using the current network parameters, update the corresponding Critic and Actor target networks step by step:

[0151] θ Q′ ← τθ Q + (1 - τ)θ Q

[0152] θμ​​′ ←τθμ+(1-τ)θμ

[0153] Where τ represents the update scaling factor, τ∈(0,1).

[0154] Step 3: Improve the experience cache pool of the DDPG algorithm by classifying the experience data stored in the experience cache pool and placing the acquired experience data into different experience cache pools. The improved DDPG algorithm can solve the problem of unstable convergence.

[0155] Step 3.1: Divide the experience pool into M success and M failure It stores both successful and failed flight experiences separately, and sets up a temporary experience pool M. temp Stores the latest flight experience. M temp The size is fixed. Once full, the earliest experience is retrieved and stored in the experience pool M according to the first-in, first-out principle. success In the middle, the latest flight experience will continue to be stored in the experience pool M. temp This process is repeated until, based on the final state of the drone, the result is determined from experience pool M. success and M failure Several pieces of experience are extracted from each of the data to train the neural network.

[0156] Step 3.2, in order to obtain experience from pool M success Extract more valuable and higher-quality experience from the two experience pools by sampling proportionally:

[0157]

[0158] Where, η success η failure They are from experience pool M success and M failure The number of samples drawn from the experience pool M, where ψ is the total number of samples, β∈[0,1] is the success rate, and represents the number of samples drawn from the experience pool M. success The probability of drawing experience from it.

[0159] Finally, a simulation environment was designed, in which the drone swarm interacted with the environment to obtain training data.

[0160] This invention can be applied in real-world scenarios. Drones act as aerial base stations, enhancing the coverage and performance of communication networks in various situations. When ground communication networks experience unexpected outages, drones can be quickly deployed. By covering ground targets, drones establish communication links with the ground to transmit data and simultaneously achieve collaborative interaction with the ground network. Examples of drone swarm collaborative coverage in planar scenarios include... Figure 2As shown: There are m fixed-position ground nodes and n unmanned aerial vehicles flying at a fixed height H in the region, all unmanned aerial vehicles take off from random positions at the same time, plan the optimal path of the unmanned aerial vehicle group under the restriction of multiple constraints to cooperatively cover the ground nodes, and provide fast, reliable and economically efficient data transmission and network communication for the ground.

[0161] Compared with the random algorithm, particle swarm algorithm and DDPG algorithm, the improved DDPG algorithm of the present application is superior to the foregoing algorithms in terms of coverage efficiency and energy consumption balance.

[0162] The random algorithm refers to that each unmanned aerial vehicle randomly selects a flight direction in the range of [0, 2pi) as the current action at each time, and if the new position exceeds the boundary of the target region, all unmanned aerial vehicles will give up this action and remain in place.

[0163] The particle swarm algorithm is a meta-heuristic algorithm, which is a commonly used method for finding the optimal path at present, and the optimal solution is found through multiple iterations of a group of random particles. In each iteration process, the particle can update itself by tracking two extreme values: the optimal solution found by itself and the optimal solution found by the whole population at present, or the extreme value of the neighbor of the particle.

[0164] Reference Figure 3 and Figure 4 By comparing the movement paths of the unmanned aerial vehicle group obtained by the four different algorithms, the coverage efficiency of the unmanned aerial vehicle group under different coverage rates and the balance degree of energy consumption during flight are observed, the improved DDPG algorithm proposed by the present application improves the training success rate and has faster convergence speed, realizes the maximization of coverage efficiency under the same conditions, effectively balances the flight energy consumption of each unmanned aerial vehicle, avoids the wood barrel effect of excessive energy consumption of a single unmanned aerial vehicle, and further reduces the flight time and cost of multiple unmanned aerial vehicles.

[0165] The above only describes some embodiments of the present application, and it should be pointed out that for ordinary skilled persons in the technical field, some improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A method for multi-UAV base station cooperative coverage path planning based on deep reinforcement learning, comprising the following steps: first, designing a deep reinforcement learning model, then in a simulation environment, a UAV group interacts with the environment to obtain training data, and the training data is sampled for simulation training, and finally the cooperative coverage path planning for a target ground node is realized. characterized in that The design of the deep reinforcement learning model comprises the following steps: Step one, defining a Markov model: modeling the constraint conditions of the UAV base station on the five-tuple (S, A, P, R, γ) of the Markov decision process; the UAV base station is a base station carried by a UAV, which is referred to as a UAV hereinafter; Step two, based on the five-tuple (S, A, P, R, γ) of the Markov decision process obtained in step one, a deep deterministic policy gradient (DDPG) algorithm is designed, which is a basic deep reinforcement learning algorithm; Step three, the experience cache pool of the DDPG algorithm is improved, and the experience data stored in the experience cache pool is classified, and the obtained experience data is put into different experience cache pools; In step one: Step 1.1, determining the state S of the UAV: There are m fixed-position ground nodes and n UAVs randomly distributed in the target area; The state S of the UAV i includes: at time t, the position where the UAV i is located ) and energy consumption , and the signal loss suffered by each ground node ; the state of the UAV i at time t is represented as: ; ) is the coordinate of the UAV i at time t; is the energy consumption of the UAV i from the initial position to the position at time t;​ Step 1.2, determining the action set A of the UAV: The unmanned aerial vehicle i has a fixed flight speed during flight, and a moving direction of a next flight action is , a t =0 represents a hovering action; wherein the hovering action refers to the unmanned aerial vehicle needing to keep a current position unchanged after covering a ground node; and then the action of the unmanned aerial vehicle i is: ; Step 1.3, define the state s of the drone at time t t and take action a t The state transition probability function P to the next input state s' from state s, given the conditions ; Step 1.4, determining the reward function R of the UAV: Set of ground node coverage states ; where b u is the coverage state of the u-th ground node, b u = 1 if the ground node has been covered by the drone, and b u = 0 if the ground node has not been covered by the drone. Coverage ratio α t is the ratio of the number of ground nodes that have been covered to the total number of ground nodes m, and The coverage ratio at time t is: ； The coverage range of each UAV is a circle with a radius R C , and the coverage effect of the UAV on the target ground node decreases from the center to the periphery; the effect degree φ(q u ) at which the u-th ground node is first covered is given by the formula: ， wherein is the coverage effect constant; d iu is the horizontal distance between the ith UAV and the u-th ground node; The optimal path planning needs to realize the transition of the ground nodes from the initial state to the target state. The initial state of the ground nodes is the uncovered state, and the target state is the covered state by the unmanned aerial vehicle. The coverage efficiency E is designed as a cooperative formula of the coverage ground node rate and the coverage effect c The formula is: ， Defining the reward function represents the feedback obtained by the UAV after selecting a certain action in the current state; the basic reward formula is: ， Wherein the coverage increment: , the energy consumption increment of the ith UAV: ; base reward The reward value of the reward function R; Step 1.5, defining the discount factor wherein ; the cumulative reward value over the entire process is calculated, the reward value will be discounted over time, the greater the discount factor, i.e. the greater the focus on long-term returns; In step two: Step 2.1, using the actor-critic architecture, one network is the actor, and the other network is the critic, and the two networks stimulate and compete with each other; network state-action value function of the Critic network is randomly initialized , the policy function of the Actor network ; the weights of the Critic network and the Actor network are copied to the target network parameters of the respective network, i.e. , wherein denote the Critic network parameters and the Actor network parameters, respectively, denote the Critic target network parameters and the Actor target network parameters, respectively; Step 2.2, when the task starts, the initial state of the UAV i is ， As the task proceeds, an action a t is made based on the current state s t , the formula is: , Where β is a random noise; Perform action a t , obtain reward r t and new state s t+1 ; Step 2.

3. Saving the experience bar from Step 2.2 into the experience pool ; saving the experience bar into the experience pool; Randomly extract part of the samples from the experience pool for training, assuming TD target training for a batch of randomly sampled data, target network Y i is represented as: ， wherein represents the policy i+1 analyzed, represents the state-action value i+1 adopted at s policy. Step 2.4, updating the Critic network, and calculating the minimum loss function L as: ， Where N represents the number of random samples extracted from the experience pool for action exploration; Step 2.5, updating the actor network parameters , a function using the policy gradient descent algorithm is: ， in This represents the gradient of the state-behavior value function of the Critic network. This represents the gradient of the Actor network policy function. Indicates the input state s in the Actor network i The action strategy selected at that time Represents state s i Time-based Critic network state-behavior value function Represents state s i Time-based Actor network policy function; Step 2.6, calculating the target network value using the replica network, and the weight parameters of these target networks are updated by tracking the learning network delay; at the same time, using the current network parameters, the corresponding Critic and Actor target networks are updated step by step: ， where τ represents an update proportionality coefficient, ; In step three: Step 3.1, divide the experience pool into M success and M failure , respectively store the success and failure of two flight experiences; extract several experiences from the experience pool M success and M failure , respectively, and train the deep reinforcement learning model; Step 3.2, setting the proportion of sampling from the two experience pools: ， where η success , η failure are the number of samples drawn from experience pools M success and M failure respectively, Ψ is the total number of samples, is the success sample rate, representing the probability of drawing an experience from experience pool M success .

2. The method of claim 1, wherein the method further comprises: In step 1.1, The whole target area is divided into I x J cells, and m fixed-position ground nodes and n unmanned aerial vehicles flying at a fixed height H are randomly distributed in the area, and the coordinate of unmanned aerial vehicle i at time t is represented as , and the position coordinates of the u-th ground node are represented as ; The fixed total energy of a UAV is e max The energy consumption of a UAV moving one unit is e1, and the energy consumption of a UAV hovering to cover one ground node is e2, both e1 and e2 are constants, and the UAV must complete the task before the energy is exhausted. Thus, the energy consumption of the drone i when flying from the initial position to the position at time t is: ， wherein, is the number of ground nodes that have been covered by the drone i at time t; The communication radius of each drone is fixed as R s Due to the limitation of communication connectivity, the drone i must always keep within the communication radius of the nearest drone j to itself, with the formula: ， p i and p j denote the position of drone i and drone j, respectively; The process of the UAV propagating signals to the ground node will cause channel fading, and the probability of the line-of-sight (LoS) link between the UAV and the ground is ， where f and g are constants related to the environment type, H represents the height of the UAV, d iu is the horizontal distance between the ith UAV and the u-th ground node, and the formula is: ， The probability of the non-line-of-sight (NLoS) link is ， The LoS and NLoS link loss model is ， where c is the propagation speed of light, f c is the carrier frequency, ω iu is the distance between the ith UAV and the u-th ground node, η LoS and η NLoS are the additional losses for the line-of-sight (LoS) and non-line-of-sight (NLoS) links, respectively. Under the LoS and NLoS model, the signal loss formula of the u-th ground node is as follows: ， In order to ensure the quality of service in the process of the UAV covering the ground node, it is required that the signal loss received by each ground node in the covering process must be less than or equal to a certain threshold k, and then the ground node will be successfully covered, otherwise the node will not be covered.

3. The method of claim 1, wherein the method further comprises: In step 1.4, an additional reward and punishment is added, and the sum of the basic reward and the additional reward and punishment is taken as the reward value of the reward function. In the extra reward and punishment setting, when the overall coverage does not reach the expected value a ev a negative increase punishment will be made, while when the overall coverage reaches the expected value, no punishment will be made; The coverage reward of each ground node covered by the first coverage of the UAV group will be positively increased; if a UAV exceeds the energy budget during flight in the process, a negative increase penalty will be given to each UAV; if the UAVs cannot be connected, a negative increase penalty will be given. The additional reward or penalty is r extra , the reward value .

4. The method of claim 1, wherein the method further comprises: In step 2.3, the experience bar is saved in the experience pool, the newly saved experience bar is stored in the first position of the experience pool, and the original experience bar in the experience pool is moved one position at a time.

5. The method of claim 1, wherein the method further comprises: In step 3.1, a temporary experience pool M is also set up temp stores the latest flight experience; M temp The size is fixed, once full, according to the principle of first in first out, the earliest one experience is taken out and stored in experience pool M success The latest flight experience will continue to be stored in experience pool M temp ; so on and so forth, finally according to the final state of the unmanned aerial vehicle, several experiences are extracted from experience pools M success and M failure , and the deep reinforcement learning model is trained.

Images