Cost evaluation reinforcement learning-based unmanned aerial vehicle trajectory optimization and power distribution method, system and device, and medium

Abstract

The invention discloses an unmanned aerial vehicle trajectory optimization and power distribution method, system and device based on cost evaluation reinforcement learning, and a medium, and the method comprises the steps: constructing an unmanned aerial vehicle data service scene under safety constraints, modeling a multi-objective optimization problem of unmanned aerial vehicle trajectory optimization and power distribution, and designing an unmanned aerial vehicle data service evaluation index; converting a multi-objective optimization problem into a constrained Markov decision problem; respectively designing a state space, an action space, an instant reward function and a corresponding cost function; constructing a cost evaluation critic network parallel to the reward evaluation critic network, training and updating network parameters of the unmanned aerial vehicle by using a cost evaluation SAC algorithm according to a constrained Markov decision problem, and calculating trajectory optimization and power distribution parameters of the unmanned aerial vehicle; the system, the equipment and the medium are used for implementing the method. According to the invention, ground user data service requirements of a data service scene under random and dynamic security constraints are met.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) communication technology, and specifically relates to a method, system, device, and medium for UAV trajectory optimization and power allocation based on cost assessment reinforcement learning. Background Technology

[0002] Unmanned aerial vehicles (UAVs) possess advantages such as flexible deployment, low cost, and ease of establishing line-of-sight links, making them a promising candidate for widespread application. In 6G networks, UAVs, acting as airborne base stations (A-BS), can provide excellent coverage and data services, becoming a crucial component of the future integrated terrestrial-ground network. Their advantages are particularly pronounced in emergency communications. When ground networks are damaged or overloaded, UAVs can be rapidly deployed to establish temporary communication networks in disaster areas and other emergency situations.

[0003] However, drone networks still face numerous challenges in data service applications. On the one hand, the locations and data needs of ground users are dynamic and constantly changing, making it crucial to meet these needs in a timely and efficient manner. On the other hand, with the diversification of drone application scenarios, safety issues also arise. For example, drones must cope with constantly changing weather conditions and physical obstacles such as buildings in cities. Furthermore, no-fly zones must be considered in many cases. Therefore, in practical applications, providing services under dynamic and safety conditions using drone networks requires further exploration and solutions.

[0004] Patent application CN202510206418.6 employs an improved particle swarm optimization algorithm for global pre-track planning, generating a global track suitable for the flight characteristics of a multi-vertical take-off and landing fixed-wing UAV. During actual flight, it combines deep reinforcement learning-based DDPG and an improved interferometric hydrodynamic system (IIFDS) algorithm to intelligently switch the UAV's flight mode and efficiently and automatically avoid multiple dynamic obstacles. However, this method is highly complex, has high training costs, and poor robustness, making it difficult to achieve a good balance between UAV performance and risk control. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the present invention aims to provide a method, system, device, and medium for UAV trajectory optimization and power allocation based on cost evaluation reinforcement learning. By constructing a cost evaluation critic network in parallel with a reward evaluation critic network and designing corresponding training algorithms, the invention enables UAV base stations to jointly optimize trajectory and power allocation in data service scenarios under security constraints. The present invention has good scalability and high system performance.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for optimizing trajectory and power allocation of unmanned aerial vehicle based on cost evaluation reinforcement learning, comprising the following steps:

[0008] Step 1, constructing a safe constraint unmanned aerial vehicle data service scene, modeling the multi-objective optimization problem of unmanned aerial vehicle trajectory optimization and power allocation, and designing unmanned aerial vehicle data service evaluation index, the unmanned aerial vehicle data service evaluation index includes: average information age, data service amount and unmanned aerial vehicle working cost;

[0009] Step 2, converting the multi-objective optimization problem established in step 1 into a constraint Markov decision problem, and designing state space, action space, immediate reward function and corresponding cost function respectively;

[0010] Step 3, constructing a cost evaluation Critic network parallel to the reward evaluation Critic network, using the cost evaluation SAC algorithm to train and update the network parameters of the unmanned aerial vehicle according to the constraint Markov decision problem obtained in step 2, and calculating the trajectory optimization and power allocation parameters of the unmanned aerial vehicle.

[0011] The specific method of step 1 includes:

[0012] Step 1.1, making specific assumptions for the safe constraint unmanned aerial vehicle data service scene; deploying unmanned aerial vehicles as air base stations in a specified area; the unmanned aerial vehicle communicates with multiple ground users based on the orthogonal frequency division multiplexing (OFDM) scheme; there are multiple dynamic no-fly zones in the scene, that is, the range of no-fly zone changes with time; the unmanned aerial vehicle decides the action trajectory and also decides the power allocation on each subchannel;

[0013] Step 1.2, establishing a communication link model between the unmanned aerial vehicle and the ground user; the transmission rate of the communication link is:

[0014]

[0015] Where B is the bandwidth of the frequency spectrum subband, P t represents the transmission power of the unmanned aerial vehicle on the communication link, n0 represents the power spectral density of additive white Gaussian noise, g n (t) represents the channel gain of the communication link between the unmanned aerial vehicle and the ground user n;

[0016] Step 1.3, designing the unmanned aerial vehicle data service evaluation index: average information age, data service amount and unmanned aerial vehicle working cost; wherein the average information age refers to the average information age of each ground user in each time slot, the data service amount refers to the cumulative data service amount of the unmanned aerial vehicle, and the unmanned aerial vehicle working cost refers to the total number of times the unmanned aerial vehicle enters the no-fly zone.

[0017] Step 1.4, combining the scenario assumptions of Step 1.1, the communication link channel model between the UAV and the ground user established in Step 1.2, and the UAV data service evaluation index designed in Step 1.3, yields the multi-objective optimization problem of UAV trajectory optimization and power allocation in the data service scenario under security constraints.

[0018] The specific method for step 2 includes:

[0019] Step 2.1, design the state space, including the state changes of the ground user and the drone, namely the relative distance between the drone and the ground user and the data demand, specifically:

[0020] S={Δx n ,Δy n ,q n}, n∈{1,2,...N}

[0021] Where, Δx n and Δy n Let q represent the relative distances between the drone and the ground user n on the x and y axes, respectively. n This represents the data requirement of ground user n;

[0022] Step 2.2, design the motion space. The UAV's motion includes the flight trajectory for each time slot and the power allocated to each sub-channel, specifically:

[0023] A={Δx(t),Δy(t),P f (t)},f∈{1,2,...F}

[0024] Where Δx and Δy represent the displacements of the UAV on the x-axis and y-axis, respectively, P f This represents the transmit power allocated on subcarrier f for UAV communication;

[0025] Step 2.3, the immediate reward function is expressed as:

[0026]

[0027] Where, k Z and k AoI These represent the data service volume coefficient and the average information age penalty factor for time period t, respectively.

[0028] Step 2.4, the cost function is used to evaluate the drone flight constraints, i.e., whether each drone enters the no-fly zone in time slot t. The cost function is expressed as:

[0029]

[0030] Where c(t) represents the cost of the drone in time slot t.

[0031] The specific method of step 3 includes:

[0032] Step 3.1, construct a cost evaluation Critic network in parallel with the reward evaluation Critic network, and design a cost evaluation Critic network training algorithm;

[0033] First, construct the cost evaluation Critic network; according to the strategy of the UAV π(θ), the long-term cost distribution of the UAV is assumed to be a Gaussian distribution:

[0034]

[0035] Where s and a represent the current environment state and the action of the UAV, respectively; at the same time, the variance of the Gaussian distribution is Therefore, two neural networks with parameters μ and η are used to estimate And

[0036] And The loss function J C (μ) and J V (η) are represented as:

[0037]

[0038] Where, is the cost evaluation value distribution target mean, specifically:

[0039]

[0040] is the cost evaluation value distribution target variance, specifically:

[0041]

[0042] Step 3.2, design the training algorithm of the Actor network and the reward evaluation Critic network, combine the cost evaluation Critic network training algorithm in step 3.1 to get the cost evaluation SAC algorithm, and use the cost evaluation SAC algorithm to train and update the network parameters of the UAV according to the constrained Markov decision problem obtained in step 2;

[0043] According to the cost evaluation Critic network, the expected long-term cost is replaced by the new safety indicator CVaR α with a risk level of α, specifically:

[0044]

[0045] where φ(·) and Φ(·) represent the probability density function and cumulative distribution function of the standard normal distribution respectively; the control strategy of the UAV aims to optimize the following function:

[0046]

[0047] where Z π (s t ) represents a partition function used to regularize the distribution, the adaptive entropy and safety risk weights of the UAV are represented by β and κ respectively; the loss function of the Actor network is represented as:

[0048]

[0049] According to the safety metric Γ π (s,a,α), the safety risk weight κ is optimized by minimizing the following loss function:

[0050]

[0051] For the reward evaluation Critic network, the network training algorithm is consistent with the SAC algorithm;

[0052] Step 3.3, calculate the UAV trajectory optimization and power allocation parameters;

[0053] The Actor network trained using the cost evaluation SAC algorithm is configured on the UAV, and after observing the current ground user state, the UAV calculates the UAV trajectory optimization and power allocation parameters through the Actor network, while avoiding the no-fly zone as much as possible, and providing data services for the ground users covered by the UAV.

[0054] A UAV trajectory optimization and power allocation system based on cost evaluation reinforcement learning, comprising:

[0055] A multi-objective optimization problem establishing module is used to construct a UAV data service scene under safety constraints, model a multi-objective optimization problem of UAV trajectory optimization and power allocation, and design a UAV data service evaluation index, wherein the UAV data service evaluation index includes: average information age, data service amount and UAV operating cost.

[0056] A Markov decision problem expression module is used to convert the multi-objective optimization problem into a constrained Markov decision problem, and design a state space, an action space, an immediate reward function and a corresponding cost function respectively.

[0057] The unmanned aerial vehicle trajectory optimization and power allocation parameter calculation module is configured to construct a cost evaluation critic network in parallel with a reward evaluation critic network, train and update network parameters of the unmanned aerial vehicle according to a constrained Markov decision problem using a cost evaluation SAC algorithm, and calculate the unmanned aerial vehicle trajectory optimization and power allocation parameters.

[0058] The unmanned aerial vehicle trajectory optimization and power allocation device based on cost evaluation reinforcement learning comprises:

[0059] The memory is configured to store a computer program of the unmanned aerial vehicle trajectory optimization and power allocation method based on cost evaluation reinforcement learning according to any one of claims 1-4, and the computer program is readable by the device.

[0060] The processor is configured to implement the unmanned aerial vehicle trajectory optimization and power allocation method based on cost evaluation reinforcement learning according to any one of claims 1-4 when the computer program is executed.

[0061] The computer readable storage medium stores a computer program, and the computer program can implement the unmanned aerial vehicle trajectory optimization and power allocation method based on cost evaluation reinforcement learning when executed by the processor.

[0062] Compared with the prior art, the unmanned aerial vehicle trajectory optimization and power allocation method based on cost evaluation reinforcement learning has the following beneficial effects:

[0063] 1、The step 1 of the present application establishes a data service scene under the assistance of a safe constraint of an unmanned aerial vehicle, innovatively considers the threat of a dynamic no-fly zone to data service of the unmanned aerial vehicle, and realizes the maximum satisfaction of data service demand of a ground user under the condition of avoiding the dynamic no-fly zone as much as possible through the cooperative optimization of a trajectory of the unmanned aerial vehicle and a launch power.

[0064] 2、The step 1 of the present application designs average information age, data service amount and unmanned aerial vehicle working cost evaluation indexes, and compared with traditional evaluation indexes such as a data transmission rate, the present application can more reflect data service quality of the unmanned aerial vehicle base station to the ground user in the maximum degree.

[0065] 3、The step 3 of the present application designs a separate safety evaluation critic network in parallel with a reward evaluation critic network, significantly improves the adaptability of the algorithm in solving the reinforcement learning problem with safety requirements through the network, and has good expansibility. In addition, the unmanned aerial vehicle constantly interacts with the environment through an optimization strategy of a conditional risk value CVaR, updates an optimized trajectory optimization and power allocation strategy, and this combination of the reward evaluation and the safety evaluation of the unmanned aerial vehicle strategy further reduces the complexity and training cost of the algorithm compared with the previous invention of multiple designs for the unmanned aerial vehicle strategy, and enhances the robustness of the algorithm to the random environment.

[0066] In summary, the present application sets up an evaluation index for ground users, and designs a cost evaluation critic network, so that the UAV can make real-time decisions on flight trajectory and power allocation strategy in the safe constraint data service scene of dynamic change of no-fly zone, while maximizing the data service quality of ground users as much as possible while minimizing the safety risk, with high scalability and excellent system performance. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 The present application is a UAV-assisted data service scene under safety constraints.

[0068] Figure 2 The present application proposes a cost evaluation SAC algorithm operation framework diagram.

[0069] Figure 3 The present application proposes a cost evaluation SAC algorithm for UAV reward convergence chart.

[0070] Figure 4 The present application proposes a cost evaluation SAC algorithm for the average information age of ground users.

[0071] Figure 5 The present application proposes a cost evaluation SAC algorithm for the change of UAV data service amount.

[0072] Figure 6 The present application proposes a cost evaluation SAC algorithm for the change of UAV operating cost. DETAILED DESCRIPTION

[0073] In order to more clearly understand the technical features, objectives and effects of the present application, the technical solutions thereof will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present application, not all, and should not be regarded as limiting the scope of the present application; based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0074] The present application proposes a separate cost evaluation critic network in parallel with the reward evaluation critic network, and the cost evaluation critic network and the reward evaluation critic network respectively evaluate the reward income and risk cost of the action of the UAV, and the actor network updates the parameters according to the evaluation results of the two critic networks to construct a loss function for parameter iteration, so that the UAV can realize real-time data service for ground users under safety risk constraints, which is more in line with the actual scene. The present application proposes a cost evaluation SAC algorithm, which can be used for trajectory optimization and power allocation in the UAV-assisted data service scene under safety constraints.

[0075] Referring to Figure 2 A method for optimizing trajectory and power allocation of a UAV based on cost evaluation reinforcement learning, comprising the following steps:

[0076] Step 1, constructing a UAV data service scene under safety constraints, modeling a multi-objective optimization problem of UAV trajectory optimization and power allocation, and designing a UAV data service evaluation index, the UAV data service evaluation index comprising: average information age, data service amount and UAV operating cost; specifically:

[0077] Step 1.1, making specific assumptions for the UAV data service scene under safety constraints; deploying a UAV as an air base in a specified area to provide data service support for ground users as much as possible without entering the no-fly zone; the UAV as an air base communicates with multiple ground users based on the orthogonal frequency division multiplexing (OFDM) scheme; there are multiple dynamic no-fly zones in the scene, i.e. the range of the no-fly zone changes over time; at the same time, the positions of some ground users change constantly, and their data demands change constantly over time; in addition, according to the TCP / IP protocol, the UAV provides data services by sending multiple data packets to the ground users; the UAV decides the action trajectory while also deciding the power allocation on each subchannel, in order to meet the data service demands of the ground users while avoiding dynamic no-fly zones as much as possible and reducing the operating cost of the UAV;

[0078] Step 1.2, establishing a communication link model between the UAV and the ground users; the transmission rate of the communication link is:

[0079]

[0080] where B is the bandwidth of the frequency spectrum subband, P t represents the transmission power of the UAV on the communication link, n0 represents the power spectral density of additive white Gaussian noise, g n (t) represents the channel gain of the communication link between the UAV and the ground user n, and is specifically:

[0081]

[0082] where μ NLoS is the additional attenuation coefficient of the NLoS link, γ0 represents the channel power gain at the reference distance d0 = 1m, f c and c represent the carrier frequency and the speed of light, respectively, d n (t) represents the distance between the UAV and the ground user n, α is the path loss exponent, θ n (t) represents the elevation angle between the UAV and the ground user n, denotes the LoS probability between the UAV and the ground user n, specifically,

[0083]

[0084] The LoS probability of channel conditions depends largely on the propagation environment. Wherein, a and b are constant values depending on the carrier frequency and the communication environment;

[0085] Step 1.3, design the UAV data service evaluation index: average information age, data service volume and UAV operating cost, including the following sub-steps:

[0086] Step 1.31, design the average information age evaluation index; the average information age of each ground user in each time slot is expressed as:

[0087]

[0088] Wherein, ψ i,j denotes the information age of ground user i at the end of time slot j, N denotes the number of ground users, and K denotes the total number of time slots;

[0089] Step 1.32, design the data service volume evaluation index; the service data volume of the UAV is expressed as:

[0090]

[0091] Wherein, z i denotes the cumulative data service volume of the UAV in time slot i;

[0092] Step 1.33, design the UAV operating cost evaluation index; the total number of times that the UAV enters the no-fly zone is defined as the cost of the UAV, specifically:

[0093]

[0094] Wherein, c i denotes the number of times that the UAV enters the no-fly zone in time slot i;

[0095] Step 1.4, combining the scene assumption of step 1.1, the communication link channel model between the UAV and the ground user established in step 1.2, and the UAV data service evaluation index designed in step 1.3, the multi-objective optimization problem of the UAV trajectory optimization and power allocation in the data service scene under the safety constraint is obtained, which is specifically described as:

[0096]

[0097] Among them, (a), (b), and (c) ensure the maximum flight speed constraint of the UAV and its location constraint; (d) ensure that each ground user occupies at most one subcarrier channel; (e) ensure that the total number of times the UAV enters the no-fly zone is less than the maximum threshold; (f) ensure that the number of ground users served by the UAV does not exceed the number of subcarriers; and (g) ensure that the power consumption of the UAV does not exceed the upper limit of the transmit power.

[0098] Step 2: Transform the multi-objective optimization problem established in Step 1 into a constrained Markov decision problem, and design the state space, action space, immediate reward function and corresponding cost function respectively.

[0099] Step 2.1, design the state space, including the state changes of the ground user and the drone, namely the relative distance between the drone and the ground user and the data demand, specifically:

[0100] S={Δx n ,Δy n ,q n}, n∈{1,2,...N}

[0101] Where, Δx n and Δy n Let q represent the relative distances between the drone and the ground user n on the x and y axes, respectively. n This represents the data requirement of ground user n;

[0102] Step 2.2, design the motion space. The UAV's motion includes the flight trajectory for each time slot and the power allocated to each sub-channel, specifically:

[0103] A={Δx(t),Δy(t),P f (t)},f∈{1,2,...F}

[0104] Where Δx and Δy represent the displacements of the UAV on the x-axis and y-axis, respectively, P f This represents the transmit power allocated on subcarrier f for UAV communication;

[0105] Step 2.3, the immediate reward function is expressed as:

[0106]

[0107] Where, k Z and k AoI These represent the data service volume coefficient and the average information age penalty factor for time period t, respectively.

[0108] Step 2.4, the cost function is used to evaluate the drone flight constraints, i.e., whether each drone enters the no-fly zone in time slot t. The cost function is expressed as:

[0109]

[0110] where c(t) represents the cost of UAV at time slot t.

[0111] Step 3, construct a cost evaluation Critic network in parallel with the reward evaluation Critic network, according to the constraint Markov decision problem obtained in step 2, use the cost evaluation SAC algorithm to train and update the network parameters of the UAV, and calculate the UAV trajectory optimization and power allocation parameters.

[0112] Step 3.1, construct a cost evaluation Critic network in parallel with the reward evaluation Critic network, and design a cost evaluation Critic training algorithm based on the Wasserstein distance;

[0113] In order to meet the safety constraints, first construct the cost evaluation Critic network; according to the policy of the UAV π(θ), the long-term cost distribution of the UAV is assumed to be a Gaussian distribution:

[0114]

[0115] where s and a represent the current environment state and the action of the UAV respectively; at the same time, the variance of the Gaussian distribution is Therefore, two neural networks with parameters μ and η are used to estimate and

[0116] In order to estimate the loss value of the cost evaluation critic network, the distance between the cost estimation value distributions can be measured by the simplified 2-Wasserstein distance, which is:

[0117]

[0118] where The time series difference error is obtained by calculating the 2-Wasserstein distance to update the cost evaluation critic network; therefore, and The loss function J C (μ) and J V (η) can be represented as:

[0119]

[0120] The loss function J C (μ) and J V (η) are minimized by the following formula:

[0121]

[0122] in, It is the target mean of the distribution of cost assessment values, specifically:

[0123]

[0124] Similarly, ΔV(s) t ,a t ,η) is represented as:

[0125]

[0126] in, It is the objective variance of the cost assessment value distribution, specifically:

[0127]

[0128] Because the number of times a drone enters a no-fly zone cannot exceed the prescribed maximum limit, this invention uses Conditional Value at Risk (CVaR) to replace the expected cost function and imposes corresponding constraints. The optimization strategy of Conditional Value at Risk (CVaR) makes p... π The tail of (C|s,a) does not exceed the maximum threshold d for the total number of times the drone enters the no-fly zone, reshaping the distribution of long-term costs. This transformation enables drones to evolve from initially aimless exploration to being able to move without exceeding the boundary d. Specifically, the conditional value of risk (CVaR) is defined as follows:

[0129]

[0130] Where α∈(0,1] represents the risk level in terms of security considerations; the smaller α is, the more the algorithm strategy will focus on avoiding risks and reducing costs; F C It is p π The cumulative distribution function of (C|s,a); according to the risk level α, the strategy π is safe if it satisfies the following conditions;

[0131]

[0132] Step 3.2: Design training algorithms for the Actor network and the reward evaluation Critic network. Combine the cost evaluation Critic network training algorithm based on Wasserstein distance from Step 3.1 to obtain the cost evaluation SAC algorithm. Based on the constrained Markov decision problem obtained in Step 2, use the cost evaluation SAC algorithm to train and update the network parameters of the UAV.

[0133] According to the cost assessment Critic network, the expected long-term cost can be represented by a new safety metric, CVaR, with a risk level of α. α Replace; specifically:

[0134]

[0135] where φ(·) and Φ(·) denote the probability density function and cumulative distribution function of the standard normal distribution, respectively, and the policy is optimized with respect to a given risk level a until π satisfies the following condition:

[0136]

[0137] The policy of the UAV is optimized by minimizing the following KL divergence:

[0138]

[0139] where Z π (s t ) denotes a partition function used to regularize the distribution; the KL divergence can be re-expressed as follows:

[0140]

[0141] where The adaptive entropy and safety risk weight of the UAV are denoted by β and κ, respectively; note that Z π (s t ) does not affect the update of θ. Therefore, the loss function of the Actor network is expressed as:

[0142]

[0143] According to the safety measure Γ π (s, a, a), the safety risk weight κ can be optimized by minimizing the following loss function:

[0144]

[0145] For the reward evaluation Critic network, its network training algorithm is consistent with the SAC algorithm.

[0146] To maintain the stability of online training, the target Actor network, the target reward evaluation Critic network, and the target cost evaluation Critic network have the same network structure as the online Actor network, the online reward evaluation Critic network, and the online cost evaluation Critic network, and the network parameters of the target Actor network, the target reward evaluation Critic network, and the target cost evaluation Critic network are updated using a soft target method with a small constant ξ, and the parameter update rules of the target Actor network θ', the target reward evaluation Critic network φ', and the target cost evaluation Critic network (μ', η') are as follows:

[0147]

[0148] Step 3.3, calculating the UAV trajectory optimization and power allocation parameters;

[0149] In the cost evaluation SAC algorithm, the cost evaluation Critic network and the reward evaluation Critic network respectively evaluate the reward income and risk cost of the action of the UAV, and the Actor network constructs a loss function according to the evaluation results of the cost evaluation Critic network and the reward evaluation Critic network for parameter iterative update. Finally, the trained Actor network is configured on the UAV. After observing the current state of the ground user, the UAV decides its flight trajectory and power allocation result on each sub-channel through the Actor network, so as to avoid the no-fly zone as much as possible while providing data services for the ground users covered.

[0150] Embodiment

[0151] The embodiment proposes a cost evaluation SAC algorithm, and applies it to the data service scene assisted by the UAV under the safety constraint, to solve the UAV trajectory optimization and communication resource allocation problems.

[0152] The research problem scene is as shown in Figure 1 , and the algorithm framework diagram is as shown in Figure 2 . The specific implementation method is introduced in three parts: the first part is the description of the simulation scene and the setting of the parameters, the second part is the specific application process of the cost evaluation SAC algorithm, and the third part is the simulation results and effect verification.

[0153] First part: simulation scene simulation and parameter setting;

[0154] The simulation experiment is carried out in a 500mx500m square area, in which the UAV flies at a fixed height of 100m to provide data services for 28 randomly distributed ground users. There are 7 no-fly zones in the area, and the position coordinates of the ground users are uniformly distributed. The Actor network, the reward evaluation Critic network and the cost evaluation Critic network of the algorithm are composed of an input layer, two hidden layers and an output layer. A total of 800 training is carried out, each training contains 40 steps, and performance test is carried out once every 50 training to evaluate the effectiveness of the agent.

[0155] Among them, the initial position of the ground user is uniformly distributed, and in order to simulate the dynamic change of the user position in the actual scene, the ground user will move back and forth along a certain trajectory. At the same time, in order to simulate the randomness of the data demand of the user in the actual scene, the data demand of each ground user in different time slots in the simulation experiment is set to follow the normal distribution X q ~N(5,1 2(MB). Meanwhile, the no-fly zone will also move at a certain speed, and its range will also change. In addition, according to the TCP / IP protocol, the UAV as a base station will provide data services by sending multiple data packets to the ground user. The working time of the UAV is divided into K time slots, and the length of each time slot is τ. The rest of the important parameters are set as shown in Table 1.

[0156] Table 1: Simulation experiment parameter settings

[0157]

[0158]

[0159] The second part: the practical application of the SAC algorithm in the scenario of data service assisted by the UAV under the safety constraint, which is specifically divided into the following 7 steps:

[0160] Step 1: initialize the actor network, reward evaluation critic network and cost evaluation critic network of the UAV, and their corresponding target network parameters, network update frequency, delay update frequency, experience pool and simulation environment parameters;

[0161] Step 2: judge whether the maximum number of iterations is reached, if yes, end the algorithm, output the optimal trajectory optimization and power allocation strategy, the maximum total reward value, the maximum data service amount, the minimum UAV working cost and the average information age, if not, execute step 3;

[0162] Step 3: judge whether the maximum number of steps is reached, if not, execute step 4, if yes, execute step 2;

[0163] Step 4: the agent obtains the current environment state s, inputs the state into the actor network of the agent, obtains the corresponding action a, and the environment state changes according to the action, the agent obtains the reward r obtained by the action and the next environment state s', and whether the next state is the final state end;

[0164] Step 5: store the experience [s, a, r, c, s', end] in the experience pool, judge whether the size of the experience pool is greater than the training batch sample number, if yes, execute step 6, if not, execute step 4;

[0165] Step 6: randomly select a batch of samples in the experience pool, calculate the loss function value of the actor network, reward evaluation critic network and cost evaluation critic network of the UAV, and update the parameters of the above networks;

[0166] Step 7: judge whether the target network update frequency is reached, if yes, copy the parameters of the main network to the target network according to the soft update strategy, if not, enter step 3.

[0167] Third part: simulation results and analysis;

[0168] The simulation results are shown in Figure 3 , Figure 4 , Figure 5 and Figure 6 .

[0169] From Figure 3 , by comparing the reward values of SAC algorithm and cost-evaluated SAC algorithm under different α values, it can be concluded that all algorithms converge to an effective strategy at the end of training. Compared with SAC algorithm, the reward of UAV after convergence of cost-evaluated SAC algorithm under α = 0.9, α = 0.5 and α = 0.1 is reduced by 1.3%, 4.6% and 7.1% respectively. Since SAC algorithm is not limited by safety constraints, it can converge to a better strategy and obtain a higher reward. At the same time, it is also observed that in the cost-evaluated SAC algorithm with different α values, the reward value at the end of training decreases with the decrease of α value. This is because, with the decrease of α value, the risk aversion degree is higher, and the safety constraints of the no-fly zone are more stringent. On the contrary, with the increase of α value, the risk aversion degree decreases, resulting in less stringent safety constraints and higher reward value after convergence.

[0170] Figure 4 The average information age of ground users during the training process of different algorithms is shown. The simulation results show that the final average information age of SAC algorithm is the smallest, and the average information age of cost-evaluated SAC algorithm increases with the decrease of α. Compared with SAC algorithm, the average information age of cost-evaluated SAC algorithm under α = 0.9, α = 0.5 and α = 0.1 is 1.9%, 2.1% and 4.1% higher respectively. Obviously, due to the safety constraints of the no-fly zone, the performance of data service of cost-evaluated SAC algorithm decreases slightly.

[0171] Figure 5 The data service amount of different algorithms during the training process is shown. Similar to the average information age performance, the data service amount of UAV using SAC algorithm is the highest, while the data service amount of cost-evaluated SAC algorithm decreases slightly. Compared with SAC algorithm, the data service amount of cost-evaluated SAC algorithm under α = 0.9, α = 0.5 and α = 0.1 is reduced by 2.7%, 4.6% and 6.9% respectively. Although the proposed algorithm shows slightly worse performance compared with SAC algorithm, the results shown in Figure 6 indicate that the safety risk is significantly reduced. The proposed algorithm effectively avoids the no-fly zone during data service.

[0172] Figure 6The working cost performance of the UAV in the training process is shown. Compared with the cost evaluation SAC algorithm, the SAC algorithm has obviously better and more stable performance in providing data services for ground users, but it does not meet the constraint condition. The cost of the SAC algorithm has no downward trend in the training process, and after convergence, it is 32.1%, 57.9% and 65.7% higher than that of the cost evaluation SAC algorithm in the case of α=0.9, α=0.5 and α=0.1, respectively. Obviously, the cost evaluation SAC algorithm adopts the safety design based on the constraint Markov decision process, which significantly reduces the frequency of entering the no-fly zone and the related safety risk. In addition, since the greater the risk level α is, the lower the risk aversion is, the influence of the safety constraint on the training process of the algorithm is minimized. On the contrary, a smaller α will lead to an increase in risk aversion and a greater influence of safety constraints. Therefore, as α decreases, the working cost of the UAV of the proposed algorithm will also decrease.

[0173] The application adopts the above-mentioned cost evaluation SAC algorithm, and applies it to the UAV-assisted data service scene under safety constraints to solve the UAV trajectory optimization and power allocation problems. Simulation results show that compared with the SAC algorithm, the proposed cost evaluation SAC algorithm significantly reduces the safety risk in the operation process of the UAV. The simulation results also prove the effectiveness of the proposed algorithm in controlling the safety risk in the operation process of the UAV.

[0174] The application also provides a UAV trajectory optimization and power allocation system based on cost evaluation reinforcement learning, comprising:

[0175] The multi-objective optimization problem establishing module is used to realize the construction of the UAV data service scene under safety constraints in step 1, model the multi-objective optimization problem of the UAV trajectory optimization and power allocation, and design the UAV data service evaluation index, wherein the UAV data service evaluation index comprises: average information age, data service amount and UAV working cost.

[0176] The Markov decision problem expression module is used to realize the conversion of the multi-objective optimization problem established in step 1 into a constraint Markov decision problem in step 2, and design the state space, action space, immediate reward function and corresponding cost function respectively.

[0177] The UAV trajectory optimization and power allocation parameter calculation module is used to realize the construction of a cost evaluation Critic network parallel to the reward evaluation Critic network in step 3, train and update the network parameters of the UAV using the cost evaluation SAC algorithm according to the constraint Markov decision problem obtained in step 2, and calculate the UAV trajectory optimization and power allocation parameters.

[0178] The application also provides a UAV trajectory optimization and power allocation device based on cost evaluation reinforcement learning, comprising:

[0179] Memory: a computer readable device for storing the computer program of the above-mentioned method for optimizing trajectory and power distribution of unmanned aerial vehicle based on cost evaluation reinforcement learning;

[0180] Processor: for executing the computer program to realize the method for optimizing trajectory and power distribution of unmanned aerial vehicle based on cost evaluation reinforcement learning.

[0181] The application further provides a computer readable storage medium, which stores a computer program, and the computer program can realize the method for optimizing trajectory and power distribution of unmanned aerial vehicle based on cost evaluation reinforcement learning when executed by a processor.

Claims

1. A method for UAV trajectory optimization and power allocation based on cost assessment reinforcement learning, characterized in that, Includes the following steps: Step 1: Construct a UAV data service scenario under security constraints, model a multi-objective optimization problem of UAV trajectory optimization and power allocation, and design UAV data service evaluation indicators, including: average information age, data service volume, and UAV operating cost. Step 2: Transform the multi-objective optimization problem established in Step 1 into a constrained Markov decision problem, and design the state space, action space, immediate reward function and corresponding cost function respectively. Step 3: Construct a cost evaluation Critic network in parallel with the reward evaluation Critic network. Based on the constrained Markov decision problem obtained in Step 2, use the cost evaluation SAC algorithm to train and update the network parameters of the UAV, and calculate the UAV trajectory optimization and power allocation parameters.

2. The method for UAV trajectory optimization and power allocation based on cost assessment reinforcement learning according to claim 1, characterized in that, The specific method of step 1 includes: Step 1.1: Make specific assumptions about the UAV data service scenario under security constraints; deploy UAVs as airborne base stations in a designated area; UAVs communicate with multiple ground users based on the Orthogonal Frequency Division Multiplexing (OFDM) scheme; there are multiple dynamic no-fly zones in the scenario, that is, the range of the no-fly zones changes over time; while deciding on the flight trajectory, the UAV also decides on the power allocation on each sub-channel. Step 1.2: Establish a communication link model between the UAV and the ground user; the transmission rate of the communication link is: Where B is the spectrum subband bandwidth, P t The value of g represents the transmit power of the UAV on this communication link, n0 represents the power spectral density of additive white Gaussian noise, and g represents the transmit power of the UAV on this communication link. n (t) represents the channel gain of the communication link between the UAV and ground user n; Step 1.3, design evaluation indicators for UAV data services: average information age, data service volume, and UAV operating cost; where average information age refers to the average information age of each ground user in each time slot, data service volume refers to the cumulative data service volume of UAVs, and UAV operating cost refers to the total number of times UAVs enter no-fly zones. Step 1.4, combining the scenario assumptions of Step 1.1, the communication link channel model between the UAV and the ground user established in Step 1.2, and the UAV data service evaluation index designed in Step 1.3, yields the multi-objective optimization problem of UAV trajectory optimization and power allocation in the data service scenario under security constraints.

3. The method for UAV trajectory optimization and power allocation based on cost assessment reinforcement learning as described in claim 1, characterized in that, The specific method for step 2 includes: Step 2.1, design the state space, including the state changes of the ground user and the drone, namely the relative distance between the drone and the ground user and the data demand, specifically: S={Δx n ,Δy n ,q n },n∈{1,2,...N} Where, Δx n and Δy n Let q represent the relative distances between the drone and the ground user n on the x and y axes, respectively. n This represents the data requirement of ground user n; Step 2.2, design the motion space. The UAV's motion includes the flight trajectory for each time slot and the power allocated to each sub-channel, specifically: A={Δx(t),Δy(t),P f (t)},f∈{1,2,...F} Where Δx and Δy represent the displacements of the UAV on the x-axis and y-axis, respectively, P f This represents the transmit power allocated on subcarrier f for UAV communication; Step 2.3, the immediate reward function is expressed as: Where, k Z and k AoI These represent the data service volume coefficient and the average information age penalty factor for time period t, respectively. Step 2.4, the cost function is used to evaluate the drone flight constraints, i.e., whether each drone enters the no-fly zone in time slot t. The cost function is expressed as: Where c(t) represents the cost of the drone in time slot t.

4. The method for UAV trajectory optimization and power allocation based on cost assessment reinforcement learning according to claim 1, characterized in that, The specific method for step 3 includes: Step 3.1: Construct a cost evaluation Critic network that runs parallel to the reward evaluation Critic network, and design a training algorithm for the cost evaluation Critic network. First, a cost assessment Critic network is constructed; based on the UAV's policy π(θ), the long-run cost distribution of the UAV is assumed to be a Gaussian distribution: Where s and a represent the current environmental state and the drone's actions, respectively; meanwhile, the variance of the Gaussian distribution... Therefore, two neural networks with parameters μ and η are used to estimate... and and loss function J C (μ) and J V (η) is represented as: in, It is the target mean of the distribution of cost assessment values, specifically: It is the objective variance of the cost assessment value distribution, specifically: Step 3.2: Design training algorithms for the Actor network and the reward evaluation Critic network. Combine the cost evaluation Critic network training algorithm from Step 3.1 to obtain the cost evaluation SAC algorithm. Based on the constrained Markov decision problem obtained in Step 2, use the cost evaluation SAC algorithm to train and update the network parameters of the UAV. Based on the cost assessment Critic network, the expected long-term cost is represented by a new safety metric, CVaR, with a risk level of α. α The replacement is as follows: Where φ(·) and Φ(·) represent the probability density function and cumulative distribution function of the standard normal distribution, respectively; the control strategy of the UAV aims to optimize the following function: Among them, Z π (s t ) represents the partitioning function used to regularize the distribution. The adaptive entropy and safety risk weights of the drone are represented by β and κ, respectively; the loss function of the Actor network is expressed as: According to the safety metric Γ π (s,a,α), the safety risk weight κ is optimized by minimizing the following loss function: For the Critic network, the network training algorithm is consistent with the SAC algorithm; Step 3.3: Calculate the UAV trajectory optimization and power allocation parameters; The Actor network trained using the cost-evaluation SAC algorithm is configured on the drone. After observing the current status of ground users, the drone uses the Actor network to calculate drone trajectory optimization and power allocation parameters, providing data services to the ground users it covers while avoiding no-fly zones as much as possible.

5. A cost-evaluation-based reinforcement learning-based UAV trajectory optimization and power allocation system based on the method of any one of claims 1 to 4, characterized in that, include: The multi-objective optimization problem establishment module is used to construct UAV data service scenarios under safety constraints, model multi-objective optimization problems of UAV trajectory optimization and power allocation, and design UAV data service evaluation indicators, including: average information age, data service volume and UAV operating cost. The Markov decision problem formulation module is used to transform multi-objective optimization problems into constrained Markov decision problems, and designs the state space, action space, immediate reward function and corresponding cost function respectively. The module for calculating UAV trajectory optimization and power allocation parameters is used to construct a cost evaluation Critic network that runs parallel to the reward evaluation Critic network. Based on the constrained Markov decision problem, the network parameters of the UAV are trained and updated using the cost evaluation SAC algorithm, and the UAV trajectory optimization and power allocation parameters are calculated.

6. A device for drone trajectory optimization and power allocation based on cost assessment reinforcement learning, characterized in that, include: Memory: A computer program for a method of drone trajectory optimization and power allocation based on cost assessment reinforcement learning as described in any one of claims 1-4, which is a computer-readable device; Processor: Used to implement the method for drone trajectory optimization and power allocation based on cost assessment reinforcement learning as described in any one of claims 1-4 when executing the computer program.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the method for drone trajectory optimization and power allocation based on cost assessment reinforcement learning as described in any one of claims 1-4.

Images