A UAV-assisted communication perception trajectory planning method based on reinforcement learning

Abstract

The present invention discloses a reinforcement learning-based method for UAV-assisted communication perception trajectory planning, which relates to the field of UAV technology. The method comprises: modeling the UAV dynamics and energy consumption model, the communication perception channel model, and the laser wireless charging model, and obtaining a trajectory planning problem based on the established models; establishing the trajectory planning problem as a Markov decision process, and designing a state space, action space, and reward function for the problem; designing a deep reinforcement learning network based on the Markov decision process, training it, and then using the trained deep reinforcement learning network to plan the trajectory of the UAV. The present invention reduces the difficulty of UAV control, and the planned trajectory is closer to the actual situation, and the considered scenario is closer to the actual situation; establishing the optimization problem as a Markov decision process and using deep reinforcement learning to solve it solves the problem that traditional optimization algorithms are not effective in solving complex constrained optimization problems.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a UAV-assisted communication perception trajectory planning method based on reinforcement learning. Background Art

[0002] With the rapid development of the drone industry and the large-scale commercial application of 5G communications, the cost and size of drones are constantly decreasing, and research on their use in auxiliary communication perception scenarios is increasing. Although the related applications of drones have brought great convenience, their own size and payload capacity limit the energy they can carry. How to improve the endurance of drones is also an urgent problem to be solved. Therefore, trajectory planning for energy-constrained drones to assist in communication perception has become a key technology. Considering drone dynamics in trajectory planning can make the trajectory closer to the actual drone flight situation and can optimize the optimal control sequence that directly acts on the motor. However, this will result in the optimization problem being established with too large a dimension and extremely complex constraints. Traditional optimization algorithms have certain difficulties in solving it. Deep reinforcement learning algorithms can learn and train the actions of intelligent agents in unknown environments and can effectively solve the complex optimization problems mentioned above.

[0003] Existing trajectory planning methods based on traditional optimization methods generally only consider the UAV mass kinematic equations and do not consider the UAV's energy constraints. These methods have the following disadvantages: (1) They do not consider the UAV's dynamic equations, resulting in poor tracking planning results; (2) They do not consider the UAV's energy constraints, resulting in planning results that consume more energy than the UAV can carry; (3) They are difficult to solve complex constraints such as UAV dynamics, laser charging, and mobile target perception requirements. Summary of the Invention

[0004] The present invention provides a UAV-assisted communication perception trajectory planning method based on reinforcement learning, which can solve the above problems.

[0005] In order to solve the above problems, the technical solutions adopted by the present invention are as follows:

[0006] The present invention provides a UAV-assisted communication perception trajectory planning method based on reinforcement learning, comprising:

[0007] Model the UAV dynamics and energy consumption model, communication perception channel model, and laser wireless charging model used, and solve the trajectory planning problem based on the established models;

[0008] The trajectory planning problem is established as a Markov decision process, and the state space, action space and reward function are designed for it;

[0009] A deep reinforcement learning network is designed based on the Markov decision process and trained. The trained deep reinforcement learning network is then used to plan the trajectory of the UAV.

[0010] As a further description of the above technical solution: the trajectory planning problem is:

[0011] P:

[0012]

[0013] C1:U imin ≤u i (t)≤U imax ,i=1,2,3,4

[0014] C2:x(0)=x0

[0015] C3:z(t)≥h min

[0016] C4:E total (t)≥0

[0017] C5:R t (t)-R s (t)≤0

[0018] C6:x(T)=x F ,y(T)=y F ,z(T)=z F

[0019] Among them, T is the optimized terminal time, Q t (T) is the estimation of the mutual information of the UAV’s perception of the target, C0 is the UAV’s dynamic constraint, x(t) is the state quantity at time t, u(t) is the control quantity at time t, C1 is the control quantity constraint, and U imin is the lower limit of the control quantity of the i-th motor, u i (t) is the control quantity of the i-th motor at time t, U imax is the upper limit of the control quantity of the i-th motor, C2 is the initial state constraint, x(0) is the state quantity at the initial moment, x0 is the initial state quantity value, C3 is the safe flight height constraint of the UAV, z(t) is the flight height of the UAV at time t, h min is the minimum safe altitude of the UAV, C4 is the remaining energy safety constraint of the UAV, E total (t) is the remaining energy of the UAV at time t, C5 is the communication constraint, R t (t) is the mutual information rate of the perception signal, R s (t) is the communication rate between the UAV and the base station, and C6 is the three-dimensional position coordinate of the UAV terminal.

[0020] As a further description of the above technical solution: The Markov decision process is described as a five-tuple in, is the system state set, is a set of actions, is the probability of transition between any state and action state, For the expected value of state transition given any state and action, is the discount factor; the Markov decision process change sequence is expressed as:

[0021] s t →a t →r t ,s t+1 →a t+1 →r t+1 ,s t+2 →……

[0022] Among them, s t is the state of the environment at the current time t, a t is the action selected by the drone at the current moment, r t is the action reward obtained, s t+1 For action a t The state of the environment at the next moment after the action.

[0023] As a further description of the above technical solution: the state space includes the relative position of the UAV and the base station, the relative position of the UAV and the perception target, the relative position of the UAV and the laser source, the relative position of the UAV and the end point, the UAV's speed, attitude, attitude quaternion, remaining energy, and perception and communication throughput.

[0024] As a further description of the above technical solution: the control quantity of the drone is used as an action in the action space.

[0025] As a further description of the above technical solution: the reward function is:

[0026] R tot =R d +R c +R h +R e +R g +R 3D

[0027] Among them, R d is the terminal reward function, R c is the process-aware reward function, R h is the height reward function, R e is the energy reward function, R g is the reward function of the flight process, R 3Dis the penalty function.

[0028] As a further description of the above technical solution: The deep reinforcement learning network is built based on the TD3 algorithm and uses two critic networks to calculate the Q value, including the actor network, critic network 1 and critic network 2 of the current network group, and the actor target network, critic target network 1 and critic target network 2 of the target network group;

[0029] The parameters defining the Actor network and the Actor target network are θ A and The output of the two networks for a given state s is expressed as μ(s|θ A )and The parameters of Critic Network 1 and Critic Network 2 are and Critic target network 1 and Critic target network 2 are and Then the benefit for a given state s and action a is expressed as

[0030] The loss function used to train the deep reinforcement learning network is:

[0031]

[0032] Among them, s t is the posture at time t, a t is the action at time t;

[0033]

[0034] done is the end mark of a single training, n ε To cut off the noise, is the total reward, is the discount factor, s t+1 For the posture of the next moment, are the parameters of the Critic target network;

[0035] During the training process, the update formula of the two Critic network parameters is:

[0036]

[0037] Among them, N b To select the number of samples from the experience pool, the Actor network uses the estimated value of Critic network 1 for delayed update. At this time, the gradient of the Actor network parameter update is expressed as:

[0038]

[0039] The update formula of the target network is:

[0040]

[0041] Among them, τ is the network update coefficient.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] Compared with existing UAV perception and communication trajectory planning methods, this method takes into account the UAV dynamics equation, and the planned trajectory is closer to the actual situation. The final optimal control sequence is the UAV motor speed, which reduces the difficulty of UAV control. It also takes into account the problem of UAV energy limitation and target movement, and the considered scenario is closer to the actual situation.

[0044] Compared with traditional optimization algorithms based on optimal control, establishing the optimization problem as a Markov decision process and using deep reinforcement learning to solve it solves the problem that traditional optimization algorithms are not good at solving complex constrained optimization problems.

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, embodiments of the present invention are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 This is a schematic diagram of the simple structure of a quadrotor drone;

[0048] Figure 2 This is the structure diagram of the deep reinforcement learning algorithm;

[0049] Figure 3 This is a comparison chart of drone flight trajectories;

[0050] Figure 4 This is a comparison chart of perception and communication rates;

[0051] Figure 5 This is a comparison chart of the speed changes of drones;

[0052] Figure 6 A comparison chart of drone motor speeds. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0054] An embodiment of the present invention provides a reinforcement learning-based quadrotor drone assisted communication perception trajectory planning method, which plans the trajectory of a quadrotor drone that supports laser charging while perceiving a moving target and communicating with a base station.

[0055] 1. Model the quadrotor UAV's dynamics and energy consumption model, communication perception channel model, and laser wireless charging model, and solve the trajectory planning problem based on the established models:

[0056] 1) For Figure 1 The quadrotor UAV dynamic model shown in is described as follows:

[0057]

[0058] in,

[0059] The state quantities x are respectively the UAV in the ground coordinate system O e X e Y e Z e Acceleration and velocity in

[0060] The drone orbits the body coordinate system O b X b Y b Z b Angle of rotation of the coordinate axis ω x ,ω y ,ω z ,

[0061] The attitude of the drone updates the quaternion q0,q1,q2,q3,

[0062] In the formula, Ω=ω1+ω2+ω3+ω4 is the sum of the UAV motor speeds, and the control variable u is converted from the motor speed, which can be specifically expressed as:

[0063]

[0064] Among them, ω1~ω4 are the speeds of the motor.

[0065] According to the analysis of the voltage and current during the operation of the UAV motor, the relationship between the power consumption and the speed of the i-th motor, i=1, 2, 3, 4 during the flight of the UAV is expressed as follows:

[0066]

[0067] Among them, c0~c4 are motor coefficients.

[0068] 2) When a drone perceives a target, it usually measures the perception effect by estimating the mutual information. At this time, the signal scattered back by the drone is expressed as:

[0069]

[0070] Among them, P t is the transmission power of the sensing signal, and the signal satisfies τ t is the signal return time, h t (t) is the channel gain, n(t) is the noise, assuming that there is only additive white Gaussian noise in the channel at this time, and its distribution obeys the Gaussian normal distribution Because the sensing target is a non-cooperative object, the radar channel gain is used to express the expected gain of the sensing channel as:

[0071]

[0072] Among them, G is the antenna gain of the UAV transmitting signal, and the gain is the same during transmission and reception, λ is the signal carrier wavelength, σ cross is the cross-sectional area of ​​the perceived target, d t (t)=||q(t)-q t (t)|| is the distance between the UAV and the target, F N is the noise coefficient of the UAV receiving antenna, L s is the signal processing loss, air propagation loss, etc. that the signal may suffer during transmission and reception. The mutual information rate of the perceived signal is expressed as:

[0073]

[0074] Therefore, the mutual information estimator of the UAV’s perception of the target is expressed as:

[0075]

[0076] in, is a constant parameter, and T is the flight time of the UAV.

[0077] The drone transmits the sensing data back to the base station. During this process, it is easy for the drone to establish a LoS channel when communicating with the base station.

[0078] The channel power is expressed as

[0079] Where β0 is the channel power at the reference distance, ds =||qq s || is the distance between the drone and the base station,

[0080] Assume that the UAV signal transmission power is P t , then the communication rate between the drone and the base station is expressed as:

[0081]

[0082] Among them, σ 2 is the Gaussian white noise power, γ1=P t β0 / σ 2 .

[0083] Similarly, the communication throughput between the drone and the base station is expressed as:

[0084]

[0085] 3) To solve the problem of limited energy for drones, we consider using laser wireless charging to replenish energy for drones. After a certain approximation, the power received by the drone is expressed as:

[0086]

[0087] in, It represents the loss coefficient of laser during propagation, which is mainly affected by the propagation distance, weather conditions and laser wavelength. The environmental parameter value is κ = 3.92 × 10 -4 , δ=550nm, ε=1.3.

[0088] d b =||qq b || is the distance between the drone and the laser source base station, where the wavelength λ is used b =810nm laser light source, P b =100W is the power provided by the laser source, d min =200m is the minimum distance for starting laser charging.

[0089] In the scenario considered by the present invention, the drone carries a certain initial energy E0=10 5 J starts from the starting point. During the flight, the motor consumes energy and is replenished by the laser source. The remaining energy at time t is expressed as:

[0090]

[0091] Among them, P0=10J is the additional energy consumption used for other functions such as communication.

[0092] 4) The present invention considers that a UAV flies from a starting point to an end point, senses a moving target, and transmits the sensed data to a base station. During the flight, a ground laser source provides energy for the UAV. Therefore, the optimization problem is described as:

[0093]

[0094] Among them, C0 is the UAV dynamic constraint, C1 is the control quantity constraint, C2 is the initial state constraint, C3 is the UAV safe flight altitude constraint, C4 is the UAV remaining energy safety constraint, C5 is the communication constraint, and C6 is the terminal position of the UAV flight.

[0095] 2. Establish the above optimization problem as a Markov decision process and design the state space, action space and reward function.

[0096] The Markov decision process is described as a five-tuple

[0097] in, System status collection Action Set is the probability of transition between any state and action state, For the expected value of state transition given any state and action, is the discount factor.

[0098] The Markov decision process change sequence is expressed as:

[0099] s t →a t →r t ,s t+1 →a t+1 →r t+1 ,s t+2 →… … (13)

[0100] Among them, s t is the state of the environment at the current time t, a t is the action selected by the drone at the current moment, r t is the action reward obtained, s t+1 For action a t The environmental state at the next moment t+1 after the action continues to cycle until the process ends.

[0101] 1) State Space

[0102] The state space is the state of the environment that the agent interacts with, and it is updated according to the action of the agent. In the trajectory planning problem, the objects that need to be considered in the environment are drones, base stations, sensing targets and laser sources. The states affected by the drone control variables include the drone's position, speed, attitude, attitude quaternion, and residual energy E. total and the perception and communication throughput Q t and Q s .

[0103] The factors that affect the control decision-making amount in the environment are mainly the base station location (x s ,y s ,z s ),

[0104] Target position (x t (t),y t (t),z t (t)), laser source position (x b ,y b ,z b ) and the end point coordinate position (x F ,y F ,z F ).

[0105] In order to reduce the factors that affect the control decision in the environment, relative positions are used to represent environmental variables.

[0106] is the relative position of the UAV and the base station,

[0107] is the relative position of the UAV and the sensing target,

[0108] is the relative position of the UAV and the laser source,

[0109] is the relative position of the UAV and the destination. The final system environment has a total of 25 states represented as follows:

[0110]

[0111] 2) Action Space

[0112] The actions in the action space need to be able to affect all states in the above state space, so the control variables {u1,u2,u3,u4} of the drone are selected as the actions in the action space.

[0113] 3) Reward Function

[0114] The trajectory planning problem aims to maximize the mutual information of terminal time perception and includes constraints on terminal position and process state. Therefore, the reward function is designed to be divided into the following parts:

[0115] (1) Terminal Rewards

[0116] The ultimate goal of the drone is to fly to the destination. In this process, it is affected by the target perception mutual information and the remaining total energy. Therefore, the terminal reward is designed as follows:

[0117]

[0118] Among them, d F =||qq F || is the distance between the drone and the end point, dis d is the switching distance, r maxd is the maximum distance reward, 1>ε d ,ε d1 ,ε d2 >0 is the reward coefficient. When the distance between the drone and the end point is greater than the switching distance, the drone needs to be forced to fly to the end point. Therefore, the distance from the end point is penalized and only part of the perception reward is given. When the drone reaches the end point, it obtains the maximum distance reward and the full perception reward. The planning process also requires the parameter ε d ,ε d1 ,ε d2 Make adjustments to ensure

[0119]

[0120] This allows the drone to receive increasing rewards as it flies towards the finish line.

[0121] (2) Process Perception Reward

[0122] In the process of the drone sensing the target, it is necessary to ensure that the constraint C5 is satisfied and the influence of the drone's remaining energy needs to be considered. Therefore, the reward function is designed as

[0123]

[0124] Among them, ε c +ε c1 =1 is the weight parameter. When the constraint C5 is satisfied, the drone will receive a reward, which is positively correlated with the remaining energy of the drone. When the constraint C5 is not satisfied, a penalty will be given, k c is the penalty weight.

[0125] (3) Flight altitude bonus

[0126] The drone needs to maintain a certain flight altitude during flight, so the flight altitude reward function is designed as

[0127] R h =r maxh +k h min(zh min ,0) (17)

[0128] Among them, r maxh is the maximum value of the height reward, and a penalty is given when the constraint C3 is not satisfied, k h is the penalty weight.

[0129] (4) Process Energy Reward

[0130] The drone needs to maintain as much residual energy as possible during flight, so the energy reward function is designed as

[0131] R e =k e (E total (t)-E total (t-dt)) (18)

[0132] Among them, k e is the energy penalty weight, and dt is the time step. During the flight, the UAV tries to minimize the difference in residual energy between adjacent moments, that is, consume as little energy as possible at each step and receive more laser energy.

[0133] (5) Flight process rewards

[0134] During the flight, we hope that the drone will fly towards the target when the remaining energy is high, increase the mutual information rate of perception, and fly to the destination as quickly as possible when the energy is low. Therefore, the flight reward function is set as

[0135]

[0136] Among them, d t,t-1 and d t,t is the distance between the drone and the target at the previous moment and the current moment, d F,t-1 and d F,t E is the distance between the drone and the destination at the previous moment and the current moment, s When the drone's energy is greater than the switching energy, it is rewarded to move closer to the target. When the energy is less, it is rewarded to fly towards the end point.

[0137] (6) End of Round Settings

[0138] Obviously, if the UAV wants to complete the task better, it needs to ensure that it flies within a certain space range. The flight range of the UAV in space is set to [x min ,xmax ],[y min ,y max ],[z min ,z max ] If the drone flies out of range, the training round will end and a penalty will be given.

[0139] R 3D =-r max3D (20)

[0140] In summary, in each training session, the total reward of the drone is

[0141] R tot =R d +R c +R h +R e +R g +R 3D (twenty one)

[0142] 3. Design a deep reinforcement learning network for the above Markov decision process and train it.

[0143] Traditional reinforcement learning algorithms have the problem of overestimating the Q value. This paper uses the TD3 algorithm to solve this problem. Two critic networks are used to calculate the Q value and the smaller value is updated to solve the problem of overestimation of the Q value.

[0144] The deep reinforcement learning network structure designed by the present invention is as follows Figure 2 As shown, there are a total of 6 neural networks, including the Actor network, Critic network 1 and Critic network 2 of the current network group, and the Actor target network, Critic target network 1 and Critic target network 2 of the target network group.

[0145] The Actor network takes the environment state variables as input and the drone control actions as output. The Critic network takes the environment state variables and the drone control actions as input and the estimated reward as output.

[0146] Since the action output is the normalized result, the following processing is required to obtain the actual control amount of the drone

[0147]

[0148] Among them, {a1, a2, a3, a4} is the output of the Actor network.

[0149] During the TD3 algorithm training process, random network parameters are first generated and defined. Figure 2 The parameters of the Actor network and the Actor target network are θA and The output of the two networks for a given state s can be expressed as μ(s|θ A )and The parameters of Critic Network 1 and Critic Network 2 are and Critic target network 1 and Critic target network 2 are and Then the benefit for a given state s and action a is expressed as In order to avoid the problem of overestimation of the critic network, two critic target networks are used to estimate it and finally take the minimum value as the estimation result. Its loss function can be expressed as

[0150]

[0151] in,

[0152]

[0153] In the above formula, done is the end mark of a single training, n ε To cut off the noise, it can be expressed as

[0154]

[0155] in, The mean is 0 and the variance is Gaussian noise, is the threshold for adding noise.

[0156] The update formula of the two critic network parameters can be expressed as

[0157]

[0158] Among them, N b In order to select the number of samples from the experience pool, the Actor network uses the estimated value of Critic network 1 for delayed update. At this time, the gradient of the Actor network parameter update is expressed as

[0159]

[0160] The target network also adopts the update strategy, and the update formula is expressed as

[0161]

[0162] The trajectory planning algorithm flow proposed by the present invention is shown in the following table.

[0163]

[0164] The following simulation is conducted on the quadrotor drone assisted communication perception trajectory planning method based on reinforcement learning described in the present invention:

[0165] In the simulation case, the drone was set to fly from a starting point to a destination, receiving wireless laser charging from a laser source, sensing moving targets, and communicating with a base station to transmit data.

[0166] The drone's flight range is set to [-100,600], [-100,500], [60,200],

[0167] The penalty for drone flying out of range is set to r max3D =100.

[0168] r in the terminal reward function maxd =1000,dis d =50,ε d =0.3,ε d1 =0.7,ε d2 =0.01.

[0169] ε in the process-aware reward function c =0.2,ε c1 =0.8.

[0170] h in the flight height reward function min =60m, r maxh =5,k h =10.

[0171] k in the energy reward function e =8.

[0172] E in the reward function of the flight process s =6.

[0173] Set the starting point of the drone to q0 = [500, 0, 100] T , the end point is q F =[0,0,100] T , the laser source position is q b =[250,200,0] T , the initial carried energy of the UAV is E0=10kJ;

[0174] The initial state of the drone is set to

[0175] x0=[500,0,100,0,0,0,0,0,0,1,0,0,0,0,0,10000] T ;

[0176] Perception target from q t0 =[500,400,0] T The object starts from the x-axis and moves at a speed of -10m / s.

[0177] The values ​​of other parameters used are shown in Table 1.

[0178] Table 1 Parameters required for simulation

[0179]

[0180]

[0181] The present invention compares the results of solving the optimization problem P using a traditional optimization method based on control parameterization. In this method, the optimization problem P is transformed into a general nonlinear programming problem through time scale transformation, control variable parameterization, and exact penalty function methods, and is solved using a gradient-based solution method.

[0182] Figure 3 The trajectory of the drone during target perception is shown in the figure. Figure 4 The comparison between the perceived mutual information rate and the real-time communication rate is shown in Figure 5 and Figure 6 The figure shows a comparison of the drone's speed changes and motor speed changes. The simulation results show that both methods enable the drone to perceive the target, maintain the required communication rate with the base station, and safely reach its destination. However, the control parameterization-based optimization method only yields 384 Mbits of mutual information perceptually, while the deep reinforcement learning algorithm achieves 1547 Mbits of mutual information perceptually, significantly higher than the control parameterization-based method.

[0183] Figure 4 The changes in perception and communication rates of the two methods are compared in the paper. The deep reinforcement learning algorithm proposed in this invention can achieve a higher target perception mutual information rate than the control parameterization method, which is consistent with the result of perceiving more information.

[0184] Figure 5 and Figure 6 The speed changes and control quantity changes obtained by optimizing the two methods are compared. Since the control parameterization method discretizes the control parameters, the motor speed control quantity obtained has mutations, while the control quantity obtained by the deep reinforcement learning algorithm of the present invention is continuous and has no mutations, which is more in line with the actual situation.

[0185] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A UAV-assisted communication perception trajectory planning method based on reinforcement learning, characterized in that: include: The UAV dynamics and energy consumption model, communication perception channel model and laser wireless charging model used are modeled, and the trajectory planning problem is obtained based on the established models: Among them, T is the optimized terminal time, Q t (T) is the estimation of the mutual information of the UAV’s perception of the target, C0 is the UAV’s dynamic constraint, x(t) is the state quantity at time t, u(t) is the control quantity at time t, C1 is the control quantity constraint, and U imin is the lower limit of the control quantity of the i-th motor, u i (t) is the control quantity of the i-th motor at time t, U imax is the upper limit of the control quantity of the i-th motor, C2 is the initial state constraint, x(0) is the state quantity at the initial moment, x0 is the initial state quantity value, C3 is the safe flight height constraint of the UAV, z(t) is the flight height of the UAV at time t, h min is the minimum safe altitude of the UAV, C4 is the remaining energy safety constraint of the UAV, E total (t) is the remaining energy of the UAV at time t, C5 is the communication constraint, R t (t) is the mutual information rate of the perception signal, R s (t) is the communication rate between the UAV and the base station, C6 is the three-dimensional position coordinate of the UAV terminal; The trajectory planning problem is established as a Markov decision process, and the state space, action space and reward function are designed for it; A deep reinforcement learning network is designed based on the Markov decision process and trained. The trained deep reinforcement learning network is then used to plan the trajectory of the UAV.

2. The UAV-assisted communication perception trajectory planning method based on reinforcement learning according to claim 1 is characterized in that: The Markov decision process is described as a five-tuple in, is the system state set, is a set of actions, is the probability of transition between any state and action state, For the expected value of state transition given any state and action, is the discount factor; the Markov decision process change sequence is expressed as: s t →a t →r t ,s t+1 →a t+1 →r t+1 ,s t+2 →…… Among them, s t is the state of the environment at the current time t, a t is the action selected by the drone at the current moment, r t is the action reward obtained, s t+1 For action a t The state of the environment at the next moment after the action.

3. The UAV-assisted communication perception trajectory planning method based on reinforcement learning according to claim 2 is characterized in that: The state space includes the relative position of the UAV and the base station, the relative position of the UAV and the perception target, the relative position of the UAV and the laser source, the relative position of the UAV and the end point, the UAV's speed, attitude, attitude quaternion, remaining energy, and perception and communication throughput.

4. The UAV-assisted communication perception trajectory planning method based on reinforcement learning according to claim 2 is characterized in that: The control quantities of the drone are taken as actions in the action space.

5. The UAV-assisted communication perception trajectory planning method based on reinforcement learning according to claim 2 is characterized in that: The reward function is: R tot =R d +R c +R h +R e +R g +R 3D Among them, R d is the terminal reward function, R c is the process-aware reward function, R h is the height reward function, R e is the energy reward function, R g is the reward function of the flight process, R 3D is the penalty function.

6. The UAV-assisted communication perception trajectory planning method based on reinforcement learning according to claim 1 is characterized in that: The deep reinforcement learning network is built based on the TD3 algorithm and uses two critic networks to calculate the Q value, including the Actor network, Critic network 1, and Critic network 2 of the current network group, and the Actor target network, Critic target network 1, and Critic target network 2 of the target network group; The parameters defining the Actor network and the Actor target network are θ A and The output of the two networks for a given state s is expressed as μ(s|θ A )and The parameters of Critic Network 1 and Critic Network 2 are and Critic target network 1 and Critic target network 2 are and Then the benefit for a given state s and action a is expressed as The loss function used to train the deep reinforcement learning network is: Among them, s t is the posture at time t, a t is the action at time t; done is the end mark of a single training, n ε To cut off the noise, is the total reward, is the discount factor, s t+1 For the posture of the next moment, are the parameters of the Critic target network; During the training process, the update formula of the two Critic network parameters is: Among them, N b To select the number of samples from the experience pool, the Actor network uses the estimated value of Critic network 1 for delayed update. At this time, the gradient of the Actor network parameter update is expressed as: The update formula of the target network is: Among them, τ is the network update coefficient.

Images