A method for air safety data collection and resource allocation based on DDPG

Abstract

The present invention discloses an aerial safety data collection and resource allocation method based on DDPG, including modeling the drone safety data collection process as a Markov decision process, constructing an optimization problem, constructing a deep reinforcement learning network framework, the policy network takes environmental information as input, and the speed vector of the drone at the next moment is output, and the critic network takes environmental information and the output of the policy network as input, and evaluates the output of the policy network. Construct a reward function in the reinforcement learning network, input the state information of the current time slot into the policy network, so as to obtain the action of the drone, the drone executes the action in the simulation environment, obtains the state information of the next time slot and the reward of the time slot, and then inputs the state information of the next time slot into the policy network, and repeats until the drone returns to the end point. The present invention takes into account the adaptive energy consumption threshold, and balances between returning to the end point and the data collection problem to maximize the total confidentiality rate.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communication networks and relates to an air safety data collection and resource allocation transmission solution based on DDPG. Background Art

[0002] Over the past few decades, drones have been widely used. With the continuous reduction in the cost of drones and their advantages such as high maneuverability, drones have become a solution for a wide range of commercial and civilian applications, such as traffic control, disaster detection, emergency rescue, and target tracking.

[0003] Drone-assisted communications are expected to play a key role in the next generation of wireless communication systems and provide good coverage and cost-effective green communication services. Due to their high mobility, low cost and on-demand deployment, drones can be deployed as temporary aerial base stations to provide broadband wireless connectivity to hotspots in many scenarios, such as when ground base stations are destroyed in disasters. Through drone trajectory design, a line-of-sight communication link can be established between drones and ground users, thereby improving the performance of the communication system and achieving high-quality, low-energy communication services. Therefore, drone trajectory design plays a decisive role in the design of drone communication systems.

[0004] Due to the broadcast nature of wireless channels and the high probability of line-of-sight in air-to-ground links, the risk of eavesdropping in air communication systems increases, which poses a major challenge to the security of wireless communications. Physical layer security is a perfect solution to achieve secure transmission of wireless communications.

[0005] Although UAV-assisted communication systems have many advantages over traditional ground users, many new design challenges need to be addressed to achieve effective UAV communication, such as random movement of users, energy-saving communication, flight safety of UAVs, resource allocation, trajectory design, etc. For this optimization problem, methods such as BCD and SCA are often used to obtain the optimal or suboptimal solution to the problem. However, as the system model becomes more complex and the environment changes dynamically, optimization problems often become difficult to solve.

[0006] However, deep reinforcement learning algorithms can search for optimal or suboptimal solutions from a larger strategy space by combining the perception ability of deep learning with the decision-making ability of reinforcement learning. As a result, how to combine deep reinforcement learning with drone trajectory design has become one of the research hotspots in drone communication system design.

[0007] Machine learning is an important subfield of artificial intelligence. According to the current mainstream classification method, machine learning algorithms can be divided into three types: supervised learning, unsupervised learning, and reinforcement learning, based on different training samples and feedback methods.

[0008] Reinforcement learning, also known as reinforcement learning, evaluation learning or enhanced learning, is one of the paradigms and methodologies of machine learning. It is used to describe and solve the problem of how intelligent agents learn strategies to maximize rewards or achieve specific goals during the interaction with the environment. Reinforcement learning learns how to map scenes to action strategies to obtain the maximum reward. That is, under a given environmental state, decide which action to choose to change the environment so as to maximize the reward. Unlike the supervised learning process, in the reinforcement learning process, the intelligent agent will not tell which action to take, but will learn the optimal or better strategy for the current task through trial and error through continuous interaction with the environment.

[0009] Reinforcement learning can generally be represented by a Markov decision process (MDP), the main elements of which include (S, A, R, P sa ,γ), S represents the state of the environment, A represents the action taken by the agent, R represents the reward, P sa represents the state transition probability, and γ represents the discount factor. The agent’s strategy π represents a mapping from the state space to the action space. t ∈S, take action a according to strategy π t ∈A, and then according to the state transition probability P sa Transition to next state s t+1 , and receive the reward value r from the environment feedback t ∈R. Figure 1 shown.

[0010] The goal of reinforcement learning is to continuously optimize the agent's strategy to obtain the maximum reward value. The value function and action value function of the agent are V(s t ) and Q(s t ,a t ), used to evaluate the agent in state s t The optimal strategy of the agent can be obtained by optimizing the value function. Summary of the invention

[0011] The present invention considers the problem of secure data collection of a quadrotor UAV communication system in the presence of multiple ground eavesdroppers with uncertain positions. Taking into account the flight energy consumption of the UAV, fairness among users and secure transmission, the frequency band allocation and three-dimensional trajectory of the UAV are jointly designed to maximize the overall fair confidentiality rate of the system. Because this problem is a non-convex multivariate joint problem, it is challenging to solve it using traditional optimization methods. Therefore, a deep reinforcement learning algorithm based on a deep deterministic policy gradient algorithm is proposed to solve the 3D flight trajectory and resource allocation strategy of the UAV. Therefore, the purpose of the present invention is to provide an aerial security data collection and resource allocation method based on DDPG.

[0012] A method for air safety data collection and resource allocation based on DDPG, comprising the following steps:

[0013] Step 1: Model the drone security data collection process as a Markov decision process. Under the conditions of limited energy and given fairness requirements, the total throughput of the system considering fairness and confidentiality is maximized by jointly optimizing the trajectory of the drone and the bandwidth allocation of all users. The optimization problem is constructed, including the communication model, channel model and energy consumption model of the optimization problem.

[0014] Construct a deep reinforcement learning network framework, including a policy network, a critic network, and a target network. The policy network takes environmental information as input and the velocity vector of the drone at the next moment as output. The critic network takes environmental information and the output of the policy network as input and evaluates the output of the policy network.

[0015] Step 2: Construct the reward function in the reinforcement learning network. In order to enable the drone to converge and learn strategies as quickly as possible, a reward function is formulated, which mainly includes energy consumption reward, destination arrival reward, confidentiality rate reward and constraint reward;

[0016] Step 3: Input the state information of the current time slot into the policy network to obtain the action of the drone. The drone executes the action in the simulation environment, obtains the state information of the next time slot and the reward of the time slot, and then inputs the state information of the next time slot into the policy network, and repeats this cycle until the drone returns to the end point. For the state information of each cycle, the instantaneous reward value is calculated using the reward function in step 2 and the rewards are accumulated to obtain the cumulative reward value of the current flight time.

[0017] Step 4: Obtain multiple samples through step 3 to build a sample pool, use the samples in the sample pool to train and update the policy network and critic network, and continue to obtain new samples in the same way as step 3 and put them into the sample pool until the cumulative reward value tends to be stable, at which time the policy network training is completed;

[0018] Step 5: Use the trained strategy network to optimize the UAV trajectory.

[0019] In step 1, the drone safety data collection process is modeled as a Markov decision process and an optimization problem is proposed:

[0020] like Figure 2 As shown, the present invention considers a drone-assisted aerial system, in which a drone S acts as a base station and uses frequency division multiple access (FDMA) to receive signals from K ground users U k Collect confidential information. M ground eavesdroppers E m Attempt to eavesdrop on confidential information independently. Assume that all devices are equipped with a single antenna and only partial location information about the eavesdropper is available on S. k Randomly move at the maximum speed. Flight period T t Divided equally into N = T t / δ t time slots, δ t represents the length of a time slot. Without loss of generality, the rectangular coordinate system is used to describe the positions of these nodes. The position of drone S in the nth time slot is represented by Q S [n] = [x s [n],y s [n],z s [n]] T ∈R 3×1 .U k and E m The horizontal positions in the nth time slot are expressed as and

[0021] set up The nth time slot is allocated to user U k The communication bandwidth, B represents the total bandwidth, then

[0022]

[0023] The speed and position of drone S at the n+1th time slot can be expressed as:

[0024] v[n+1]=v[n]+a c [n]δ t (2)

[0025]

[0026] in is a binary indicator if Then the direction of acceleration is the same as the direction of velocity. Otherwise, if Then the acceleration and velocity are in opposite directions, v d [n] and || a c [n]||represents the unit velocity vector and acceleration vector of the drone respectively.

[0027] The present invention considers a probabilistic path loss model, where the drone S and the user U k The probability of LoS connection between in time slot n is:

[0028]

[0029] Here η a and η b is a constant related to the type of propagation environment, Represents the distance from drone S to user U k The Euclidean distance from drone S to user U. k The path loss expression between LoS and NLoS links is:

[0030]

[0031]

[0032] The probability of a NLoS link is represents the LoS connection probability in time slot n, is the free space path loss, f c is the carrier frequency, v c is the speed of light, η LoS and η NLoS are the excess path losses of the LoS and NLoS links respectively. Finally, the distance between the drone S and the user U k The path loss expression between becomes:

[0033]

[0034] Substituting (3) and (4) into (5), we can obtain that in n time slots, the relationship between drone S and user U is k The channel gain is:

[0035]

[0036] So user U k The achievable rate is expressed as:

[0037]

[0038] Where N0 is the noise power spectral density, For user U k of transmission power.

[0039] Ground User U k With eavesdropping m The channel link takes into account both large-scale path loss and small-scale Rayleigh fading, and its channel gain can be expressed as:

[0040]

[0041] Where ρ0 represents the channel power gain at a reference distance of 1 m, ζ follows an exponential distribution with unit mean, α is the path loss exponent, and α>2. Represents the Euclidean distance between the user and the eavesdropper.

[0042] Since the location of the eavesdropping is uncertain, the and r E To represent the center and radius of the eavesdropping area. We consider the best case of eavesdropping channel gain or the worst case of user security when the eavesdropping location is uncertain, and scale it by using the triangle inequality:

[0043]

[0044] So user U k With eavesdropping m The channel gain is expressed as:

[0045]

[0046] And by scaling it up through Jason's inequality, we get E m The upper bound of the achievable eavesdropping rate is expressed as:

[0047]

[0048] Considering the worst safety situation, U k The achievable confidentiality rate is expressed as

[0049]

[0050] Since the achievable confidentiality rate is a function of the transmit power The concave function of In other words, when the maximum transmission power is P max The user K in time unit δ t The confidentiality rate within is less than the confidentiality rate threshold When , there is no need to establish a communication link between the drone and the user, and the user's transmission power should be 0. When the communication link between the drone and the user is established, the transmission power is P max .

[0051] Define a binary variable to indicate whether the UAV has established a communication link with user K.

[0052]

[0053] The instantaneous total confidentiality rate of all users in the Nth time slot is expressed as

[0054]

[0055] The cumulative confidentiality throughput of user K to the Nth time slot is expressed as

[0056]

[0057] The cumulative confidentiality throughput of all users to the Nth time slot is expressed as

[0058]

[0059] The energy consumption of a drone consists of two parts: one part is used for communication, and the other part is the propulsion energy that generates thrust to help the drone overcome resistance and gravity. In practice, the energy for communication is usually two orders of magnitude smaller than the energy for flight. Therefore, the present invention ignores the energy consumption for communication. Considering the propulsion energy consumption required for drone flight and hovering, it can be expressed as a function of its speed v:

[0060]

[0061] Among them, P B and P I The constant parameter representing the blade profile power and induced power in the hovering state, U tip is the tip speed of the rotor blade, and v0 is the average rotor induced speed during hovering. In addition, the parameters d0, s , ρ and A represent the fuselage drag ratio, rotor solidity, air density and rotor disk area, respectively.

[0062] The drone has a total energy E max , that is, E(0)=E max E(n) represents the remaining energy of the drone at time slot n. The remaining energy of the drone can be expressed as:

[0063]

[0064] In order to ensure that the drone can reach the destination, E(N)≥0.

[0065] In order to consider whether the system meets the fairness requirements, a binary variable I[n] is defined:

[0066]

[0067] in represents Jain's fairness index, k f Represents the target fairness threshold. and R cum I[n] represents the cumulative confidentiality throughput of user K to the Nth time slot and the cumulative confidentiality throughput of all users to the Nth time slot. I[n] = 1 and I[n] = 0 respectively indicate that the fairness among all users meets or does not meet the requirements. It must be noted that when I[n] = 0, it is necessary to set the user's total maximum confidentiality rate upload threshold R max Otherwise, the drone will tend to communicate with the nearest user and will not visit those users with lower confidentiality or farther away. In the nth time slot, the fairness coefficient of user K is defined as:

[0068]

[0069] where k gp1 >0 is the attenuation coefficient, k gp1 The larger the value, the slower the curve decays. gp1 The smaller the value, the faster the curve decays. Under the influence of the fairness coefficient, when the information uploaded by the user reaches the threshold, the drone should fly to the area with higher Collect data from users who do not upload data or users who do not upload data. Therefore, the fair total confidentiality rate of the system considered during the entire task is expressed as

[0070]

[0071] The goal of this paper is to maximize the fair confidentiality total throughput of the considered system by jointly optimizing the trajectory of the UAV and the bandwidth allocation of all users under the conditions of limited energy and given fairness requirements. The optimization problem is expressed as:

[0072]

[0073] stE[0]=E max ,E[N]≥0,(24b)

[0074]

[0075]

[0076]

[0077] v[0]=0,(24f)

[0078]

[0079]

[0080]

[0081]

[0082]

[0083] Where (24b) and (24c) represent the energy consumption constraint and location constraint of the UAV, respectively. and denote the initial and final positions of the UAV respectively. (24e) is the altitude constraint of the UAV flight, where z min and z max They represent the minimum and maximum altitudes allowed by the drone, respectively. (24f) and (24g) represent the initial speed constraint and the speed constraint of the drone, respectively. (24h) is the speed iteration constraint to be satisfied during the flight, (24i) represents the acceleration constraint, and a min and a max They represent the minimum and maximum acceleration of the UAV during flight. (24j) represents the bandwidth allocation constraint. (24k) represents the power and fairness constraints.

[0084] The optimization objective is a non-convex multivariable coupling problem with respect to trajectory and band allocation. And because the user moves randomly on the ground and the environment is highly dynamic, the optimization problem (P1) is difficult to solve using traditional optimization methods. Although the optimization problem can be simplified and converted into multiple convex subproblems through successive convex approximations for solution. However, in traditional optimization algorithms, with the increase in the number of optimization variables and flight time, the complexity of traditional optimization methods increases rapidly. Therefore, we will use the DRL algorithm to solve this problem with lower complexity, and use the powerful learning ability of DRL to obtain optimized solutions from a wide range of policy spaces. Considering that the optimization variables in the optimization problem of the present invention are continuous, we use the deep reinforcement learning algorithm DDPG to solve it.

[0085] In step 2, we construct the reward function in reinforcement learning:

[0086] In DRL, the reward signal can be used to evaluate the quality of behavior in a state, and we can transform the optimization problem into the maximization of cumulative rewards through reward design. The reward function proposed in this invention consists of several parts: energy consumption reward, energy consumption reward, destination arrival reward, confidentiality rate reward and constraint reward.

[0087] By setting an energy consumption penalty reward, the drone is encouraged to minimize the flight energy consumption by optimizing its speed and trajectory, thereby maximizing the total confidentiality rate. The energy consumption penalty is further expressed as:

[0088] r ec [n] = -κec P[n]δ t , (25)

[0089] Here ec is a constant, and the reward size of the energy consumption part is adjusted. P[n] represents the flight power of the drone in the nth time slot.

[0090] For the constraint of reaching the end point, the direct reward given by whether or not the end point is reached is very sparse. In order to solve the sparse reward problem, we use reward shaping to make the task of reaching the end point easier to learn. Through reward shaping, the reward for the end point related task is designed as:

[0091] Define a binary indicator ξ rd (n) is intended to describe the state where the drone is about to run out of energy and needs to return to its destination.

[0092]

[0093] When the UAV flight energy consumption is less than E min When rd (n)=1, otherwise it is ξ rd (n) = 0. It is worth noting that when E min When it is a fixed value, it is equivalent to the drone starting to return to the destination when the drone's energy consumption reaches an energy consumption threshold. However, this assumption ignores the relationship between the drone's current position and the destination, that is, no matter when the drone is close or far from the destination, the drone returns to the destination at the same energy consumption threshold. Obviously, when the drone is close to the destination, this energy consumption threshold is too abundant. At this time, if the energy consumption threshold is reduced, more confidentiality rates can be collected, thereby maximizing confidentiality energy efficiency. Therefore, the present invention considers a dynamic energy consumption threshold, which is defined as:

[0094]

[0095] The adaptive energy consumption threshold is divided into two parts. The first part is the minimum energy consumption of the drone from the current position to the end point. The flight speed here is This flight speed represents the minimum energy consumption of the drone per unit distance, that is, the minimum energy consumption to reach the destination. The second part E0 is the energy consumption compensation constant.

[0096] So the reward for reaching the destination is designed as:

[0097] r rd (n) = κ rd ·d dis ·ξ rd (n) (28)

[0098] ddis represents the difference between the current position of the UAV and the destination after moving in one time slot, κ rd is a constant used to adjust the reward size of endpoint-related tasks.

[0099] In summary, the continuous reward for reaching the destination is defined as:

[0100]

[0101] When rd When [n] = 1, the continuous reward related to the end point is divided into two parts. The first part is the displacement reward of the drone flying towards the end point. rd1 is a positive constant used to adjust the size of the displacement reward. The second part is the distance reward of the drone relative to the end point. The closer to the end point, the greater the reward. rd2 is a positive constant used to adjust the size of the distance reward. rd3 Used to adjust the slope of reward vs. distance.

[0102] In addition, when the energy is exhausted, a reward r should be set ar , indicating whether the drone has reached the target point, that is:

[0103]

[0104] κ ar and κ nar are the reward coefficients for reaching or not reaching the final location, respectively. When the battery is exhausted, if the drone reaches the destination ξ ar =1, otherwise ξ ar =0.

[0105] Furthermore, the confidentiality reward in the Nth time slot is defined as:

[0106]

[0107] where κ th is the bonus coefficient of confidentiality rate when energy consumption is sufficient, κ nth The reward coefficient is the energy consumption is about to run out and the drone needs to return to the end point. The reward coefficient is used to adjust the reward of the confidentiality rate. It is worth noting that κ th >κ nth That is, when the energy consumption is about to be exhausted, the proportion of the bonus coefficient of the confidentiality rate needs to be reduced so that the drone tends to return to the end point.

[0108] To further punish violations of acceleration and altitude constraints, the constraint reward is defined as:

[0109]

[0110] Here A [n] is a binary acceleration constraint index, where ξ A [n] = 1 means that the acceleration violates the constraint, otherwise ξ A [n] = 0. Similar ξ H [n] represents the binary height constraint index, and two negative constants κ A and κ H represent the specific penalty rewards for violating the acceleration constraint and height constraint, respectively.

[0111] In summary, the reward can be expressed as:

[0112] r[n] = r ec [n]+r rd [n]+r ar [n]+r sec [n]+r A [n]+r H [n]. (33)

[0113] The beneficial effects of the present invention are mainly in three aspects:

[0114] (1) A UAV-assisted secure data acquisition system is considered in the presence of uncertain locations of potential eavesdroppers. The three-dimensional trajectory and frequency band allocation of the UAV are also considered, and an optimization problem is proposed for the three-dimensional trajectory design and dynamic bandwidth allocation of the UAV in order to maximize the total fair throughput of the network while ensuring energy constraints, fairness, secure transmission, and flight conditions.

[0115] (2) In multiple eavesdropping scenarios with uncertain locations, a fairness policy constraint is defined under a reinforcement learning algorithm to achieve a trade-off between maximizing system throughput and fairness in communication between users.

[0116] (3) A DDPG-based reinforcement learning algorithm is developed to achieve continuous trajectory design and frequency band allocation. In the design of the endpoint reward, the present invention considers an adaptive energy consumption threshold and makes a trade-off between returning to the endpoint and data collection issues to maximize the total confidentiality rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0117] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0118] Figure 1 This is a schematic diagram of the reinforcement learning interaction;

[0119] Figure 2 Collect models for drone data;

[0120] Figure 3The confidentiality rate of each user in each time slot;

[0121] Figure 4 This is the change graph of the system fairness index;

[0122] Figure 5 It is a 2D flight trajectory diagram;

[0123] Figure 6 It is a 3D flight trajectory diagram;

[0124] Figure 7 Schematic diagram of the drone trajectory optimization and resource allocation system. DETAILED DESCRIPTION

[0125] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0126] The present invention provides a method for optimizing the trajectory of unmanned aerial vehicles (UAVs) under aerial safety data collection and a design for frequency band resource allocation, which uses a deep reinforcement learning algorithm based on DDPG to optimize the trajectory and frequency band allocation. Through interaction with the environment, continuous learning and optimization strategies are continuously implemented to maximize the fair confidentiality rate.

[0127] The implementation process of the present invention is divided into five steps:

[0128] Step 1: Build a deep reinforcement learning network framework, including a policy network, a critic network, and a target network, and model the system.

[0129] 1.1 The present invention uses a fully connected neural network structure as the framework of the network policy network, critic network and target network.

[0130] The policy network consists of a 6-layer fully connected neural network. The input is the current environment information: the channel gain between K users and the drone, the amount of confidential data uploaded by K users, the current location of the drone, the Euclidean distance from the drone to the destination, the flight speed of the drone, and the current remaining energy of the drone, with a total of (2K+6) dimensions. The output is the flight speed vector and bandwidth allocation of the drone in the next time slot. The flight speed vector is represented by a Cartesian coordinate system, so the output action has a total of (K+3) dimensions. In the policy network, except for the output layer, all use the Relu function as the activation function to prevent the gradient from disappearing. The flight speed vector in the output layer uses Tanh as the activation function, and the bandwidth allocation of K users in the output layer uses the Sigmod function as the activation function.

[0131] The critic network has the same structure as the policy network. The input is the output of the policy network, which is the flight speed vector and bandwidth allocation of the drone in the next time slot, and the current environment information. The output is one-dimensional data, which is used to judge the quality of the action.

[0132] The target network consists of two parts, the policy target network and the critic target network. It has the same structure as the policy network and the critic network and uses a slow update method called soft update. The parameters of the target network change little, and the gradient of the online network calculated during training is relatively stable, making it easier for training to converge.

[0133] 1.2 The present invention provides a method for optimizing drone trajectory and designing frequency band resource allocation under aerial safety data collection. The training process is carried out in a simulated urban model, and the drone data collection during the training process is also performed by a virtual drone in a virtual urban environment. Therefore, before building a deep reinforcement learning framework network, the urban environment modeling should be performed first.

[0134] like Figure 2 As shown, there are multiple users in the environment waiting for drone communication, and multiple illegal users are ready to eavesdrop on the user's data. In the present invention, the probabilistic path loss model is considered between the drone and the user. The channel gain between the drone and the user can be calculated by formulas (4)-(8). The channel link between the ground user and the eavesdropping considers both large-scale path loss and small-scale Rayleigh fading, and its channel gain is calculated by formulas (10)-(12). Therefore, the maximum confidentiality rate of each user for all eavesdropping can be calculated, that is, formula (14).

[0135] The present invention takes the energy consumption of the UAV into consideration, and therefore adopts the energy consumption model of a quad-rotor UAV, taking into account the propulsion energy consumption required for the flight and hovering of the UAV, which can be expressed as a function of its speed, i.e., formula (19). The flight energy consumption of the UAV can be calculated for each time slot through this formula.

[0136] Step 2: Constructing the Reward Function for Reinforcement Learning

[0137] In formulas (25)-(33), the reward function of the optimization problem is defined, which is used to evaluate the quality of the behavior in a state and refers to the learning goal. The reward mainly consists of several parts: energy consumption reward, energy consumption reward, destination arrival reward, confidentiality rate reward and constraint reward.

[0138] Step 3: In each time slot of the flight, the state information of the current time slot is input into the strategy network to obtain the action of the UAV, that is, the velocity vector of the UAV and the frequency band allocation strategy.

[0139] The drone performs the actions in the simulation environment, returns the state information of the next time slot and the reward of the time slot in the virtual environment, and then inputs the state information of the next time slot into the strategy network, and repeats this cycle until the flight ends.

[0140] In calculating the reward for this time slot, formulas (25)-(33) are used to calculate the reward for each part, thereby obtaining the total instantaneous reward for this time slot.

[0141] In calculating the energy consumption reward, the current UAV's flight energy consumption is calculated based on the adopted energy consumption model and the speed of the time slot, thereby obtaining the energy consumption reward.

[0142] In calculating the reward for reaching the destination, firstly, a binary indicator is obtained by formula (26) to determine whether it is necessary to return to the destination. If it is necessary to return to the destination, the displacement reward and the distance reward of the drone relative to the destination are calculated according to formula (29) to obtain the continuous reward for reaching the destination. After judging whether the energy consumption is exhausted and whether the reward is reached, the discrete reward for reaching the destination is obtained.

[0143] To further calculate the constraint reward, the acceleration vector is calculated by the velocity vector output by the current policy network and the velocity vector of the previous time slot, so as to determine whether the acceleration vector meets the constraint. Similarly, the action will be performed in the current time slot to determine whether the flight altitude of the drone meets the constraint, so as to obtain the constraint reward.

[0144] To further calculate the confidentiality rate bonus of the current time slot, firstly, the channel gain between each user and the drone is calculated based on the current drone location information and the location information of each user, that is, formula (8), and then the achievable rate between each user and the drone is calculated.

[0145] The worst eavesdropping rate of each user and all eavesdroppers is calculated by formula (13). The confidentiality rate of each user is calculated by formula (14).

[0146] The binary variable of each user is calculated by formula (15) to determine whether the security transmission threshold is met. If so, data transmission is performed at the maximum transmission power, otherwise the transmission power is 0.

[0147] Formula (21) is used to calculate whether the fairness index between users in the current simulation environment meets the minimum fairness threshold, and then the fairness coefficient of each user is calculated by formula (22). Finally, the total fair confidentiality rate of the current time slot is calculated by formula (23). The confidentiality rate reward of the current time slot is obtained by formula (31).

[0148] In summary, the rewards of each part are accumulated and the total reward of the current time slot is obtained by formula (33). Each state in the environment is also updated with this flight time slot.

[0149] Then input the state information of the next time slot into the policy network, and repeat this cycle until the drone returns to the destination or runs out of energy. For the state information of each cycle, use the reward function defined in step 2 to calculate the instantaneous reward value and accumulate the rewards to get the cumulative reward value of the current flight time.

[0150] Step 4: Through step 3, the network continuously interacts with the environment, obtains multiple interaction samples and stores them in the experience pool, and uses the samples in the experience pool to train and update the policy network and the critic network respectively. Before updating the network, the experience pool is first filled by interacting with the environment, and then a batch of samples are randomly selected to update the policy network and the critic network during each interaction process.

[0151] Critic network Q(s,a;θ Q ) can update its network parameter learning strategy by minimizing the mean square loss, and its loss function is expressed as:

[0152]

[0153] y t (n)=r(n)+γQ′(s(n+1),π′(s(n+1);θ π′ );θ Q′ ) (35)

[0154] L(θ Q ) is the loss function of the critic network, N b is the batch size, y t (n) is the target Q value, r(n) is the reward given by the environment during the state transition process, and γ is the future discount factor, which represents the trade-off between the current time slot reward and the future reward, 0≤γ≤1. π and θ Q refers to the parameters of the actor and critic networks, π′(s;θ π′ ) and Q′(s,a;θ Q′ ) denote the target networks of the policy and critic, respectively.

[0155] Policy network π(s;θ π ) can update its network parameters by minimizing the critic’s loss, and its loss function is expressed as:

[0156]

[0157] Update the network parameters of the target network through soft update technology:

[0158] θ Q′ ←εθ Q +(1-ε)θ Q′

[0159] θπ′ ←εθ π +(1-ε)θ π′

[0160] Here ε is the soft update parameter, ε<<1, θ π′ and θ Q′ Refers to the parameters of the policy target network and the critic target network. Using a slow update method called soft update, the parameters of the target network change little, the gradient of the online network is more stable during training, and the training is easier to converge.

[0161] While updating the network, continue to interact with the environment in the same way as in step 3 to obtain new samples and put them into the experience pool until the cumulative reward value stabilizes, at which point the policy network training is completed.

[0162] Step 5: Use the trained strategy network to optimize the UAV trajectory.

[0163] The above steps are repeated until the cumulative reward value tends to be stable, that is, the cumulative reward value reaches the maximum and is in a small range of fluctuations. Then, the policy network actually deployed to the drone can be obtained, and the optimized drone trajectory can be obtained.

[0164] After sufficient training, if the cumulative reward value no longer increases significantly, the training is stopped. The trained policy network is directly deployed to the optimization module, which is the actual drone base station platform. It is used to guide the drone base station to fly quickly and efficiently to maximize the optimization problem. The evaluation network and target network are only valid in the training phase and are not required for actual deployment.

[0165] Figure 3 After the training is completed, it is the confidentiality rate of each user in each time slot during the flight of the drone. Figure 4 This is the changing trend of the system fairness index during the UAV flight, which first increases and then decreases. Figure 5 and Figure 6 They are the 2D flight trajectory and 3D flight trajectory of the drone. In the 3D flight trajectory, the drone increases the channel gain between the drone and the user by changing the flight altitude, thereby collecting data faster.

[0166] like Figure 7 As shown in the figure, a UAV trajectory optimization system for IoT data collection is constructed, and the UAV collects data in the actual environment according to the final optimized trajectory and frequency band allocation. The constructed IoT data collection system mainly includes a deep reinforcement learning framework network, an environmental simulation module, a calculation module, and an optimization module;

[0167] The environment simulation module mainly includes an environment module and a drone module, which is used to simulate the data collection task of the drone on the user node in the simulation environment, and then obtain the state information that needs to be input into the policy network; input the current state information into the policy network, receive the drone action generated by the policy network, and the drone executes the action in the simulation environment to obtain the state information at the next moment, and then inputs it into the policy network, and so on;

[0168] The status information generated by the environment simulation module includes: the channel gain between K users and the drone, the amount of confidential data uploaded by K users, the current location of the drone, the Euclidean distance between the drone and the destination, the flight speed of the drone, and the current remaining energy of the drone, with a total of (2K+6) dimensions.

[0169] The calculation module is used to calculate the instantaneous reward value. For the state information of each cycle in the environment simulation module, the calculation module is used to calculate the instantaneous reward value, and the instantaneous reward value is accumulated to obtain the cumulative reward value; the instantaneous reward value can be calculated by formula (33).

[0170] The experience pool includes multiple samples, including the state information of each current time slot in the environment simulation module, the state information of the next time slot, the drone action information generated by the policy network, and the instantaneous reward value calculated by the calculation module for each cycle; the policy network is trained and updated using the samples in the sample pool; at the same time, new samples are obtained by looping and put into the sample pool; until the cumulative reward value tends to be stable, the policy network training is completed;

[0171] The optimization module is used to receive the trained policy network, obtain the final actual flight drone action information, and generate the optimized trajectory of the drone.

[0172] The present invention provides a UAV trajectory optimization method and a frequency band resource allocation design method for aerial safety data collection. After completing the UAV trajectory optimization training in the environmental model, the trained strategy network is deployed to the actual UAV base station to obtain the actual optimized trajectory and bandwidth allocation of the UAV. The UAV then collects data quickly, efficiently and safely in the actual environment according to the optimized trajectory and bandwidth allocation, and provides services for various Internet of Things nodes in the actual environment.

[0173] The above specific embodiments only describe the design principle of the present invention. The shapes and names of the components in the description can be different and are not limited. Therefore, technicians in the field of the present invention can modify or replace the technical solutions recorded in the above embodiments; and these modifications and replacements do not deviate from the creative purpose and technical solutions of the present invention and should all fall within the protection scope of the present invention.

Claims

1. A method for air safety data collection and resource allocation based on DDPG, characterized in that: The following steps are involved: Step 1: Model the drone security data collection process as a Markov decision process. Under the conditions of limited energy and given fairness requirements, the trajectory of the drone and the bandwidth allocation of all users are jointly optimized to maximize the total throughput of the fair confidentiality of the system. An optimization problem is constructed, including the communication model, channel model and energy consumption model of the optimization problem. The optimization problem is: stE[0]=E max ,E[N]≥0,(24b) v[0]=0,(24f) Among them, Q and B represent the trajectory of the drone and the bandwidth of K users respectively, and R sum represents the fair total confidentiality rate of the system during the entire mission, E[0] represents the energy consumption of the drone at the 0th time slot, i.e., the initial time, and E max represents the total energy of the drone, E[N] represents the flight energy consumption of the drone in the Nth time slot, Q S [0], Q S [N] represents the starting point and end point of the drone, Q S [n] represents the position of drone S in the nth time slot, v[n] represents the speed of drone S in the nth time slot, is a binary indicator if Then the direction of acceleration is the same as the direction of velocity if Then the acceleration and velocity are in opposite directions, v d [n] and || a c [n]||represents the unit velocity vector and acceleration vector of the drone, z[n] represents the flight altitude of the drone, v[0] represents the initial velocity of the drone, and v max Indicates the maximum flight speed of the drone. Indicates that the nth time slot is allocated to user U k The communication bandwidth Indicates whether the drone is with user U k Establish a communication link, represents the confidentiality rate threshold, Indicates U k The achievable confidentiality rate, I[n] represents a binary variable, represents Jain's fairness index, k f represents the target fairness threshold, δ t Indicates the length of a time slot; (24b) and (24c) represent the energy consumption constraint and location constraint of the UAV, respectively. and denote the initial and final positions of the UAV respectively, (24e) is the altitude constraint of the UAV flight, where z min and z max denote the minimum and maximum altitudes allowed by the drone, (24f) and (24g) denote the initial speed constraint and the speed constraint of the drone, (24h) is the speed iteration constraint to be satisfied during the flight, (24i) denotes the acceleration constraint, and a min and a max They represent the minimum and maximum acceleration of the UAV, (24j) represents the bandwidth allocation constraint, and (24k) represents the power and fairness constraints; Construct a deep reinforcement learning network framework, including a policy network, a critic network, and a target network. The policy network takes environmental information as input and the velocity vector of the drone at the next moment as output. The critic network takes environmental information and the output of the policy network as input and evaluates the output of the policy network. Step 2: Construct the reward function in the reinforcement learning network, including energy consumption reward, destination arrival reward, confidentiality rate reward and constraint reward; Step 3: Input the state information of the current time slot into the policy network to obtain the action of the drone. The drone executes the action in the simulation environment, obtains the state information of the next time slot and the reward of the time slot, and then inputs the state information of the next time slot into the policy network, and repeats this cycle until the drone returns to the end point. For the state information of each cycle, the instantaneous reward value is calculated using the reward function in step 2 and the rewards are accumulated to obtain the cumulative reward value of the current flight time. Step 4: Obtain multiple samples through step 3 to build a sample pool, use the samples in the sample pool to train and update the policy network and critic network, and continue to obtain new samples in the same way as step 3 and put them into the sample pool until the cumulative reward value tends to be stable, at which time the policy network training is completed; Step 5: Use the trained strategy network to optimize the UAV trajectory.

2. The method for air safety data collection and resource allocation based on DDPG according to claim 1, characterized in that: The channel gain between the drone and the user is calculated using the following formula: Drone S and user U k The probability of LoS connection between in time slot n is: Here η a and η b is a constant related to the type of propagation environment, Represents the distance from drone S to user U k The Euclidean distance from drone S to user U is k The path loss expression between LoS and NLoS links is: The probability of a NLoS link is represents the LoS connection probability in time slot n, is the free space path loss, η LoS and η NLoS are the excessive path losses of the LoS and NLoS links respectively; finally, the distance between the drone S and the user U k The path loss expression between becomes: In n time slots, drone S and user U k The channel gain is: f c is the carrier frequency, v c is the speed of light; The channel link between the ground user and the eavesdropper takes into account both large-scale path loss and small-scale Rayleigh fading, and its channel gain is: Where ρ0 represents the channel power gain at the reference distance of 1m, follows an exponential distribution with unit mean, α is the path loss exponent, represents the Euclidean distance between the user and the eavesdropper; and Respectively represent U k and E m The horizontal position at the nth time slot; UserU k With eavesdropping m The channel gain is expressed as: and r E They represent the center and radius of the eavesdropping area respectively; Thus, the maximum confidentiality rate of each user for all eavesdropping is calculated as: Represents user U k The achievable rate, Indicates E m The upper bound of the achievable eavesdropping rate; M represents the number of ground eavesdroppers, and m represents the mth eavesdropper.

3. The method for air safety data collection and resource allocation based on DDPG according to claim 1 is characterized in that: The remaining energy of the drone is expressed as: Among them, P B and P I The constant parameter representing the blade profile power and induced power in the hovering state, U tip is the tip speed of the rotor blade, v0 is the average rotor induced speed during hovering, and the parameters d0, s , ρ and A represent the fuselage drag ratio, rotor solidity, air density and rotor disk area, respectively.

4. The method for air safety data collection and resource allocation based on DDPG according to claim 1, characterized in that: The reward function includes: The energy consumption penalty is expressed as: r ec [n]=-k ec P[n]δ t , Here ec is a constant, and the reward size of the energy consumption part is adjusted. P[n] represents the flight power of the drone in the nth time slot; The continuous rewards for reaching the destination are: κ rd1 is a positive constant used to adjust the size of the displacement reward, κ rd2 is a positive constant used to adjust the size of the distance reward, κ rd3 Used to adjust the slope change of reward and distance, ξ rd (n) represents a binary indicator; When the energy is exhausted, set a reward r ar (n), indicating whether the drone has reached the target point, that is: κ ar is the reward coefficient for reaching the final position, κ nar is the reward coefficient for not reaching the final position. When the battery is exhausted, if the drone reaches the destination ξ ar =1, otherwise ξ ar =0; The confidentiality reward at the nth time slot is defined as: r sec [n]=((1-ξ rd [n])k th +ξ rd [n]k nth )I uk [n]R ins [n]d t , where κ th is the bonus coefficient of confidentiality rate when energy consumption is sufficient, κ nth is the bonus coefficient of the confidentiality rate when the energy consumption is about to be exhausted, R ins [n] is the instantaneous total confidentiality rate of all users in the nth time slot; To penalize violations of acceleration and altitude constraints, the constraint reward is defined as: r A [n]=-ξ A [n]k A , r H [n]=-ξ H [n]k H , Here A [n] is a binary acceleration constraint index, where ξ A [n] = 1 means that the acceleration violates the constraint, otherwise ξ A [n] = 0, similar ξ H [n] represents the binary height constraint index, and two negative constants κ A and κ H They represent the penalty rewards for violating the acceleration constraint and the height constraint respectively; In summary, the reward is expressed as: r[n]=r ec [n]+r rd [n]+r ar [n]+r sec [n]+r A [n]+r H [n]。 5. The method for air safety data collection and resource allocation based on DDPG according to claim 4 is characterized in that: In calculating the energy consumption reward, the current UAV flight energy consumption is calculated based on the adopted energy consumption model and the speed of the time slot, thereby obtaining the energy consumption reward; In calculating the reward for reaching the destination, first determine whether it is necessary to return to the destination based on the binary indicator. If it is necessary to return to the destination, then calculate the displacement reward and the distance reward of the drone relative to the destination position to obtain the continuous reward for reaching the destination. Then determine whether the energy consumption is exhausted and whether the destination is reached to obtain the discrete reward for reaching the destination. Calculate the constraint reward, and calculate the acceleration vector through the velocity vector output by the current policy network and the velocity vector of the previous time slot, so as to determine whether the acceleration vector meets the constraint, and determine whether the flight height of the drone meets the constraint, so as to obtain the constraint reward; Calculate the confidentiality rate bonus of the current time slot. First, calculate the channel gain between each user and the drone based on the current drone location information and each user's location information, and then calculate the achievable rate between each user and the drone. Calculate the worst eavesdropping rate of each user and all eavesdroppers, and calculate the confidentiality rate of each user; Calculate the binary variable of each user to determine whether it meets the security transmission threshold. If so, the data is transmitted at the maximum transmission power, otherwise the transmission power is 0. Calculate whether the fairness index between users in the current simulation environment meets the minimum fairness threshold, thereby calculating the fairness coefficient of each user, and finally calculate the total fair confidentiality rate of the current time slot to obtain the confidentiality rate reward of the current time slot; In summary, the rewards of each part are accumulated to get the total reward of the current time slot.

6. The method for air safety data collection and resource allocation based on DDPG according to claim 1, characterized in that: Step 4: The critic network Q(s,a;θ Q ) updates its network parameter learning strategy by minimizing the mean square loss, and its loss function is expressed as: y t (n)=r(n)+γQ′(s(n+1),π′(s(n+1);θ π′ );θ Q′ ) L(θ Q ) is the loss function of the critic network, N b is the batch size, y t (n) is the target Q value, r(n) is the reward given by the environment during the state transition process, γ is the future discount factor, which represents the trade-off between the current time slot reward and the future reward, 0≤γ≤1, where θ π and θ Q refers to the parameters of the policy and critic networks, π′(s;θ π′ ) and Q′(s,a;θ Q′ ) represent the target networks of the strategy and critic respectively; Policy network π(s;θ π ) updates its network parameters by minimizing the critic’s loss, and its loss function is expressed as: θ π′ and θ Q′ refers to the parameters of the policy target network and the critic target network.

Images