A method for optimizing a path of a UAV based on deep reinforcement learning, a storage medium and an apparatus

Abstract

A kind of unmanned aerial vehicle path optimization method based on deep reinforcement learning, storage medium and equipment, comprising S1: establishing the communication model of unmanned aerial vehicle and radio network, including scene model, base station antenna radiation model and signal model, S2: according to the communication model of step S1, the signal interruption probability of unmanned aerial vehicle in different positions is calculated, to build radio coverage probability graph E, S3: considering comprehensively the flight time of unmanned aerial vehicle and the communication coverage probability of different positions, set the optimization target of unmanned aerial vehicle flight path;Based on optimization target, the flight path of unmanned aerial vehicle is optimized using deep reinforcement learning algorithm.The flight path optimization of unmanned aerial vehicle is completed by the above scheme, avoid weak coverage area, to ensure the connectivity between unmanned aerial vehicle and base station;Deep reinforcement learning algorithm is used to improve the utilization rate of important sample, obtain more accurate return value, more effectively optimize the path of unmanned aerial vehicle.

Description

Technical Field

[0001] This invention relates to a technology for enhancing communication connectivity between unmanned aerial vehicles (UAVs) and base stations in cellular networks, belonging to the field of wireless information transmission; specifically, it relates to a UAV path optimization method, storage medium, and device based on deep reinforcement learning. Background Technology

[0002] In recent years, the combination of cellular networks and drones has attracted widespread attention. Firstly, cellular network infrastructure is globally distributed, providing cost-effective communication links and reducing limitations on communication range. Secondly, it can reduce latency, improve data transmission rates, and supplement positioning accuracy. However, cellular-connected drone communication also presents some challenges. Because existing cellular networks primarily serve ground users, base station antennas are typically tilted towards the ground, resulting in inadequate aerial communication coverage. Simultaneously, cellular-connected drones are susceptible to severe interference from other unrelated base stations. To ensure drone flight safety and mission efficiency, it is necessary to leverage the controllable mobility of drones to optimize their flight paths, avoiding areas with weak coverage, thereby ensuring connectivity between the drone and base stations.

[0003] Currently, the DQN algorithm, widely used in path planning, suffers from the following problems: low sample utilization. Specifically, resampling data during training in the replay experience pool leads to a limited variety of training samples due to the original random sampling mechanism. This results in a low rate of environmental exploration by the agent, making it prone to obtaining local optima and slowing down training. This invention proposes a UAV path optimization method, storage medium, and device based on deep reinforcement learning. This method replaces traditional uniform sampling with a priority experience replay mechanism, improving the utilization of important samples while maintaining sample diversity, obtaining more accurate reward values, and more effectively optimizing UAV paths. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by providing a method, storage medium, and device for optimizing drone paths based on deep reinforcement learning. By ensuring sample diversity while improving the utilization rate of important samples, it can obtain more accurate reward values ​​and optimize drone paths more effectively.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for optimizing drone paths based on deep reinforcement learning includes the following steps:

[0007] S1: Establish a communication model between the UAV and the radio network, including a scene model, a base station antenna radiation model, and a signal model.

[0008] (I) Scene Model:

[0009] Define the drone's flight area as D×D, and the height and location of obstacles within that area;

[0010] Define the position of the drone at time t as q(t) = (x t ,y t ),t∈[0,T],x t ∈[0,D],y t ∈[0,D], where x t and y t The x and y coordinates of the drone at time t represent the position of the drone; T represents the total flight time taken by the drone from the starting position to the ending position.

[0011] (II) Base station antenna radiation model:

[0012] Assume there are M base station sectors, and construct the base station antenna radiation model A. A (θ,φ), where the base station antenna is a multi-element uniform linear array;

[0013] (III) Signal Model:

[0014] Analyze the path loss model between the UAV at position q(t) and cell m, m∈M, including line-of-sight path loss. Non-line-of-sight path loss

[0015] S2: Calculate the probability of signal interruption for the drone in different locations, and construct a radio coverage probability map E accordingly. The specific method is as follows:

[0016] S2.1: Based on the communication model established in step S1, analyze the instantaneous signal power y received by the UAV from cell m at time t. m (t)

[0017] S2.2: According to y m (t) Calculate the signal-to-interference ratio (SIR) at time t;

[0018] S2.3: Measure the probability P of communication interruption between the UAV at its location q(t) at time t and each associated cell b(t) based on the signal-to-interference ratio (SIR(t)). out (q(t),b(t));

[0019] S2.4: Based on the measured communication interruption probability, determine the optimal associated cell b(t) at position q(t). * :

[0020]

[0021] S2.5: Therefore, the communication coverage probability at position q(t) = 1 - the interruption probability of the UAV at position q(t), and the communication coverage probability value is saved in the radio coverage probability map E; where the interruption probability of the UAV at position q(t) is P. out (q(t)):

[0022]

[0023] S3: Take into account the UAV flight time and the communication coverage probability at different locations to set the optimization target for the UAV flight path; based on the optimization target, use deep reinforcement learning algorithm to optimize the UAV flight path.

[0024] To optimize the above technical solution, the specific measures also include:

[0025] Furthermore, in step S1, A in the base station antenna radiation model A The construction process of (θ,φ) is as follows:

[0026] A A (θ,φ)=G E,max -min{-[A E,V (θ)+A E,H (φ)],A m}+10log 10 [1+ρ(|a·w T | 2 -1)]

[0027] In the formula, θ and φ are the elevation and azimuth angles of the base station antenna, respectively; G E,max A is the maximum directional gain of the antenna element in the main lobe direction. E,V (θ) and A E,H (φ) represent the vertical and horizontal radiation patterns of the antenna, respectively. m It is the ratio before and after, ρ is the correlation coefficient, a represents the amplitude vector, and w is the beamforming vector;

[0028] Parameter A E,V (θ) and A E,H The specific formula for calculating (φ) is:

[0029]

[0030]

[0031] In the formula, θ 3dB and φ 3dB These are the antenna's half-power beamwidths in the vertical and horizontal directions, respectively; SLA V It is the sidelobe level limitation of the antenna.

[0032] Furthermore, in step S1, the signal model considers the line-of-sight path loss. Non-line-of-sight path loss The calculations are as follows:

[0033]

[0034]

[0035] In the formula, d m (q(t)) is the distance between the UAV at position q(t) and cell m; f c is the carrier frequency; h is the altitude of the UAV at position q(t).

[0036] Furthermore, in step S2.1, the instantaneous signal power received by the UAV from cell m at position q(t) is y. m The specific calculation formula for (t) is as follows:

[0037]

[0038] In the formula, P m h is the transmission power of cell m; m β(q(t)) represents the channel power gain at time t; β(q(t)) represents the base station antenna gain at position q(t). It is a random variable representing the small-scale fading between the UAV and cell m at time t; This represents the large-scale channel power gain between the UAV at position q(t) and cell m. Where LoS link indicates a link on a line-of-sight path, and NLoS link indicates a link on a non-line-of-sight path.

[0039] Furthermore, the specific content of calculating the signal-to-interference ratio (SIR(t)) at time t in step S2.2 is as follows:

[0040]

[0041] In the formula, b(t) represents a certain associated cell of the UAV at time t; y b(t) (t) represents the instantaneous signal power received by the UAV from the associated cell b(t) at time t.

[0042] Further, in step S2.3, P is calculated. out The specific content of (q(t),b(t)) is as follows:

[0043] S2.3.1: Define the probability of communication interruption between a UAV at position q(t) and a certain associated cell b(t) as P. out (q(t),b(t)):

[0044]

[0045] In the formula, Pr(·) represents the probability of the event occurring; γ th The threshold is set when the signal-to-interference ratio (SIR(t)) is lower than γ. th At this time, the drone is considered to be in a state of communication interruption;

[0046] S2.3.2: Rewrite the signal-to-interference ratio SIR(t) as SIR(q(t),b(t)). The variable time t is changed to the corresponding UAV position q(t), associated cell b(t), and small-scale fading between the UAV and associated cell b(t). Define the interrupt indicator function as c(q(t),b(t),

[0047]

[0048] S2.3.3: Based on the content of step S2.3.2, change the interruption probability P of S2.3.1. out (q(t),b(t)) can be rewritten as Expected value:

[0049]

[0050] Then, the signal-to-interference ratio (SIR) between the UAV and each associated cell b(t) is measured J times within a certain time period to obtain the interruption probability between the UAV and each associated cell b(t) at that location:

[0051]

[0052] In the formula, Let j be the j-th measurement of small-scale fading between the UAV and the associated cell b(t) at time t.

[0053] Furthermore, in step S3, the specific content of the optimization objective for setting the UAV flight path is as follows:

[0054] S3.1: Constructing a continuous optimization objective function:

[0055]

[0056] stq(0)=q s

[0057] q(T)=q f

[0058] In the formula, T represents the flight time of the UAV from the starting point to the ending point; μ is a non-negative coefficient; q(0) represents the initial position of the UAV; q s q represents the initial position of the drone; q(T) represents the final position of the drone; q f Represents the destination of the drone;

[0059] S3.2: Discretize the continuous optimization problem from step S3.1 by dividing the UAV flight area into a series of adjacent grid points. The final objective function is equivalent to minimizing the weighted sum of the number of grid points N traversed by the UAV and the expected interruption probability.

[0060]

[0061] stq0 = q s

[0062] q N =q f

[0063] In the formula, q n This indicates the location of the UAV at point n (n∈N) in the grid; P out (q n ) represents the probability of the drone being interrupted at position n; q0 represents the starting position of the drone; q f This indicates the location of the drone's destination.

[0064] Furthermore, in step S3, the specific content of optimizing the UAV flight path using a deep reinforcement learning algorithm based on the optimization objective is as follows:

[0065] (I) First, conduct an actual test flight of the drone from the starting point to the finish line, as follows:

[0066] 1) Set the maximum number of iterations for the actual test flight of the UAV to N. epi The maximum number of grid points the drone visits in each iteration, also known as the maximum number of steps the drone takes, is called N. step Initialize the drone exploration probability ε→ε0, and set the drone exploration probability decay rate α and the drone's reward value R for reaching the destination. des The penalty value P for the drone flying out of the designated D×D area. ob A non-negative coefficient μ, a replay experience pool D with a capacity of C, a radio coverage probability map E, and a map neural network with its parameters ξ, a depth Q network with its parameters θ, and a target depth Q′ network with its parameters θ. - =θ; Sets and initializes the default data structure of the experience replay summation tree, and assigns priority p to each leaf node of the summation tree. i Initialize to p s That is, pi →p s ;

[0067] 2): The UAV begins to perform actual test flights; let the loop variable n be... epi =1, which represents the first iteration in the actual test flight mission;

[0068] 3): Initialize a sliding window W of size N1, and initialize the initial position q of the actual test flight of the UAV. n =q s At this time q n In this context, n=0 indicates that the number of grid points traversed by the drone or the number of steps explored by the drone at this time is 0;

[0069] 4): Select action v using the ε-greedy strategy n Specifically, it randomly selects an action in the action space with a probability of ε, and selects the optimal action with a probability of 1-ε.

[0070] 5): Execute action v n To obtain the position q of the drone in the next state. n+1 q obtained by measurement in radio coverage probability map E n+1 Location interruption probability P out (q n+1 Set a single-step reward R n :

[0071] R n =-1-μP out (q n+1 )

[0072] 6): (q) n ,v n ,R n ,q n+1 The reward is stored in a sliding window W; where, when n≥N1, the cumulative reward from step n-N1 to step n is calculated. Then, the position and action of step n-N1, the cumulative reward from step n-N1 to step n, and the position of step n. The data samples are stored in the summation tree nodes;

[0073] 7): Update the number of steps n for drone exploration, and repeat steps 4)-7) to obtain multiple data samples;

[0074] 8): Sample k nodes from the summation tree. The probability that each sample j will be sampled is... p j This indicates the priority of node sample j. This represents the sum of priorities of all nodes in the summation tree, and the weights ω of the loss function for the nodes in the summation tree. j=(P(j) / min i P(i)) -β β determines the impact of priority experience playback on the convergence result;

[0075] 9): Calculate the current reward value y for each sample j in the k-node samples of the summation tree. j The specific method is as follows:

[0076]

[0077] In the formula, This represents the cumulative reward for the drone from step j to step j+N1. Let S represent the drone's position at step j+N1, S represent the drone's flight area D×D, and γ represent the reward discount rate. This indicates that the target depth Q′ network is related to the depth Q network in q j+N1 Position selection of optimal action v * Evaluation reward value;

[0078] 10): Regarding the loss function Perform gradient descent and update the parameters θ of the deep Q-network; where ω j Represents the weights of the loss function. This indicates that the target Q′ network is related to the Q network in q j Position selection action v j Evaluation reward value;

[0079] 11): Based on the radio coverage probability map E, update the map neural network parameters ξ; then conduct a simulated test flight mission for the UAV:

[0080] Step 1: Initialize the initial position of the UAV simulated test flight mission The "~" symbol above each parameter indicates that the current state is a simulated test flight mission, distinguishing it from parameters in actual flight. In This indicates that the number of grid points traversed or the number of steps explored by the drone during the simulated flight mission is 0; a loop variable is set. This indicates the first iteration in the simulated flight test mission;

[0081] Step 2: Same as steps 4)-10), except that the interruption probability in step 5) is obtained by the prediction output of the map neural network, and the input parameter of the map neural network is the data in the radio coverage probability map E.

[0082] Step 3: Determine whether the drone has reached the destination, gone out of bounds, or reached the maximum number of steps N. step If yes, proceed to step 4; otherwise, let Repeat the loop until step 2;

[0083] Step 4: Analyze the number of iterations After each iteration B times, the network parameters θ for the target depth Q′ are updated. - →θ, then proceed to step 5;

[0084] Step 5: If Then the loop ends; if Then let And return to step 1;

[0085] 12) Returning to the actual test flight mission, determine whether the UAV has reached the destination, gone out of bounds, or reached the maximum number of steps N. step If so, proceed to step 13); otherwise, let n = n + 1 and the drone exploration probability ε → εα, and repeat the loop to step 4).

[0086] 13) Analyze the number of iterations n epi In each iteration B, the network parameters θ for the target depth Q′ are updated. - →θ; then proceed to step 14);

[0087] 14) If n epi =N epi If n epi <N epi Let n epi =n epi +1 and return to step 3);

[0088] (ii) The parameters θ of the target depth Q′ network can be continuously improved through the entire process of steps 1)-14). - Therefore, the target depth Q′ network is designed for the depth Q network for the UAV at a certain position q. n Select action v n The evaluation reward value gradually reaches the optimal level; at this point, no actual test flights are conducted. Instead, the drone is guided to fly from different starting points to different endpoints directly through the target depth Q′ network after final training. During this process, the target depth Q′ network guides the selection of flight actions, enabling the drone to select the action with the highest reward value and complete the path planning.

[0089] A computer-readable storage medium storing a computer program that causes a computer to perform the unmanned aerial vehicle (UAV) path optimization method as described in any of the preceding claims.

[0090] An electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the unmanned aerial vehicle path optimization method as described in any of the preceding claims.

[0091] The beneficial effects of this invention are:

[0092] 1. This application optimizes the flight path of the drone to avoid weak coverage areas, thereby ensuring the connectivity between the drone and the base station.

[0093] 2. This invention proposes a method, storage medium, and device for UAV path optimization based on deep reinforcement learning. By using a priority experience replay mechanism based on a summation tree, uniform sampling is broken, and samples with high learning efficiency are given greater sampling weights. Thus, this solution improves the utilization rate of important samples while ensuring sample diversity, obtains more accurate reward values, and optimizes UAV paths more effectively. Attached Figure Description

[0094] Figure 1 This is a schematic diagram of a communication scenario between the drone and a cellular network base station according to the present invention.

[0095] Figure 2 This is a schematic diagram of the UAV path optimization process based on deep reinforcement learning according to the present invention.

[0096] Figure 3 This is a schematic diagram of the flight path of a drone generated in a specific case of the present invention.

[0097] Figure 4 This is a schematic diagram comparing the completion time of a drone mission with other methods in a specific case of the present invention. Detailed Implementation

[0098] The following detailed description of the UAV path optimization method, storage medium, and device based on deep reinforcement learning, with reference to the accompanying drawings and embodiments, illustrates the present invention.

[0099] This case study uses a dense urban cellular network scenario, and the specific simulation parameters are shown in Table 1.

[0100] Table 1 Simulation Parameters

[0101] D 2 M 21 <![CDATA[f c ]]> 2GHz <![CDATA[D tol ]]> 20m <![CDATA[α bd ]]> 0.3 <![CDATA[h bs ]]> 25m h 100m <![CDATA[ε0]]> 0.5 <![CDATA[β bd ]]> 300 <![CDATA[G E,max ]]> 3dBi <![CDATA[P m ]]> 0.1W α 0.998 <![CDATA[σ bd ]]> 50 <![CDATA[A m ]]> 30dB <![CDATA[γ th ]]> 0dB <![CDATA[R des ]]> 200 <![CDATA[(x s ,and s )]]> random value ρ 1 J 1000 <![CDATA[P ob ]]> 10000 <![CDATA[(x f ,and f )]]> (1400，1600) <![CDATA[θ 3dB ]]> 65° μ 40 C 100000 V 8 <![CDATA[φ 3dB ]]> 65° <![CDATA[N epi ]]> 5000 <![CDATA[N1]]> 30 X 7 <![CDATA[SLA V ]]> 30dB <![CDATA[N step ]]> 300 γ 0.9999

[0102] This embodiment includes the following steps:

[0103] Step 1: Establish a communication model between the drone and the cellular network, including a scene model, an antenna model, and a signal model. The specific steps are as follows:

[0104] 1) such as Figure 1 As shown, a scenario model for establishing communication between drones and cellular networks in dense urban areas is presented. Specifically, it includes...

[0105] Step 1: Consider the flight area to be D×D km. 2The height and location of buildings are generated based on a statistical model recommended by the International Telecommunication Union. This model involves three parameters: α bd β bd and γ bd Among them, α bd β is the ratio of the building area to the total land area. bd γ is the average number of buildings per unit area; bd The value determines the height distribution of buildings, that is, it follows a mean of σ. bd Rayleigh distribution;

[0106] Step 2: Define the position of the UAV at time t as q(t) = (x t ,y t ),t∈[0,T],xt∈[0,D],yt∈[0,D],x t and y t This represents the x and y coordinates of the drone. q s =(x s ,y s ) and q f =(x f ,y f ) represent the starting position and the ending position respectively, and the drone's flight speed is Vm / s.

[0107] 2) Calculate the base station antenna radiation model

[0108] This scenario features X = 7 base stations located at (1000, 1000), (1577.4, 1333), (1000, 1666.7), (422.6, 1333.3), (422.6, 666.7), (1000, 333.3), and (1577.4, 666.7). There are 21 base station sectors, and the antennas are 8-element uniform linear arrays. All base stations have the same height h. bs =25m, the antenna array radiation pattern is modeled as follows

[0109] A A (θ,φ)=3-min{-[A E,V (θ)+A E,H [(φ)],30}+10log 10 [1+(|a·w T | 2 -1)]

[0110] Where a represents the amplitude vector, w is the beamforming vector, and A E,V (θ) and A E,H (φ) represent the vertical and horizontal radiation patterns of the antenna, respectively.

[0111]

[0112]

[0113] 3) Calculate the path loss model between the UAV at position q(t) and cell m. The specific method is as follows:

[0114]

[0115]

[0116] Where, d m (q(t)) is the distance between the UAV and cell m when the UAV is at position q(t).

[0117] Step 2: Calculate the interruption probability at each location and construct a radio coverage probability map. The specific method is as follows:

[0118] 1) Calculate the instantaneous signal power received by the UAV from cell m at position q(t).

[0119]

[0120] Among them, h m (t) is the channel power gain, and β(q(t)) represents the base station antenna gain at position q(t). It is a random variable representing the small-scale fading between the UAV and cell m at time t. The large-scale channel power gain between the UAV and cell m at position q(t) is determined as follows:

[0121]

[0122] 2) Calculate the signal-to-interference ratio at time t

[0123]

[0124] Where b(t) represents the associated cell of the UAV at time t, and y b(t) (t) represents the instantaneous signal power received from associated cell b(t) at time t.

[0125] 3) Based on the aforementioned interruption indication function, measure the SIR value of each cell 1000 times, and calculate the interruption probability of that location for each cell.

[0126]

[0127] 4) Determine the optimal associated base station based on the measured outage probability.

[0128]

[0129] 5) The communication coverage probability of the UAV at position q(t) = 1 - the interruption probability. Save the coverage probability value to the radio coverage probability map:

[0130]

[0131] Step 3: Based on the above analysis, construct the objective function that needs to be optimized. The specific method is as follows:

[0132] 1) Construct a continuous optimization objective function

[0133]

[0134] stq(0)=q s

[0135] q(T)=q f

[0136] Where T represents the flight time from the starting point to the destination, and μ is a non-negative coefficient. The higher the value, the more attention is paid to the connectivity of the drone, at the cost that the drone trajectory may be more circuitous.

[0137] 2) Discretize the continuous optimization problem by dividing the UAV flight area into a series of adjacent grid points. The final objective is equivalent to minimizing the weighted sum of the number of grid points N traversed by the UAV and the expected interruption probability:

[0138]

[0139] stq0 = q s

[0140] q N =q f

[0141] Step 4: As Figure 2 As shown, the drone path is optimized using a deep reinforcement learning algorithm. The specific steps are as follows:

[0142] 1) Initialize the variables according to the parameters in Table 1; where D tol This is expressed as the fault tolerance distance;

[0143] 2) The drone begins its flight mission, and the loop variable n represents the number of iterations. epi =1;

[0144] 3) Initialize a sliding window W of size 30, assuming the actual initial position q s = (1000, 1000), the number of drone exploration steps n = 0;

[0145] 4) Select an action using an ε-greedy strategy, assuming the selected action is v.n = (0, 10);

[0146] 5) Execute the action to obtain the next state q. n+1 = (1000, 1010), the current interruption probability P is obtained by measurement. out (q n+1 ), calculate the single-step reward R n =-1-40P out (q n+1 );

[0147] 6) (q) n ,v n ,R n ,q n+1 The data is stored in a sliding window W, and R is calculated when n ≥ 30. (n-30):n and (q) n-30 ,v n-30 ,R (n-30):n ,q n Store the samples in a summation tree; repeat steps 4)-6) to obtain multiple samples;

[0148] 7) Sample 32 samples (q) from the summation tree. j ,v j ,R j:j+30 ,q j+30 ),...,(q j+32 ,v j+32 ,R j+32:j+30+32 ,q j+30+32 The probability of each sample being sampled is based on p j Indicates the priority of sample j. This represents the sum of priorities of all nodes in the summation tree. The weights ω of the loss function are calculated. j =(P(j) / min i P(i)) -0.4 ;

[0149] 8) Calculate the current reward value for each sample, using the following method:

[0150]

[0151] Where S represents the drone's flight area. This indicates that the target Q′ network is relative to the Q network in... Optimal action v for position selection * Evaluation reward value;

[0152] 9) Regarding the loss function Perform gradient descent and update the Q-network parameters θ;

[0153] 10) Add the measured interruption probability data to E and update the map network parameter ξ;

[0154] 11) Perform the simulated flight step (the simulated flight is independent of the actual flight described above; the purpose of the simulated flight is to create more samples to update the parameters of the two networks, thereby improving the training effect), where the loop variable represents the number of simulated flight episodes. The specific steps are as follows:

[0155] Step 1: Randomly initialize the initial position of the simulation

[0156] Step 2: Same as 4)-10), except that the interruption probability of 5) is predicted by the map neural network model at this time;

[0157] Step 3: When the destination is reached, the boundary is crossed, or the maximum number of steps is reached, proceed to Step 4; otherwise, let... Repeat step two;

[0158] Step 4: Every 5 iterations, set the target Q′ network parameters θ. - →θ;

[0159] Step 5: If Then the loop ends; if Then let And return to step one, where

[0160] 12) If the endpoint is reached, the boundary is exceeded, or the maximum number of steps is reached, execute 13); otherwise, let n = n + 1, ε → 0.998ε and return to 4);

[0161] 13) Every B iterations, set the target Q′ network parameters θ. - →θ;

[0162] 14) If n epi If n = 5000, the loop ends; if n epi If n < 5000, then let n epi =n epi +1 and return 3).

[0163] The effect obtained in this embodiment can be achieved through… Figures 3-4 The specific data obtained in the simulation experiment will be further explained. It can be seen that: 1) Figure 3 The paths of the last 100 drones from random starting points to ending points are given. As can be seen from the graph, the drones avoided areas with weak coverage and flew along paths with a higher probability of communication interruption; 2) Figure 4The paper presents a comparison of the method of this invention with other algorithms in terms of flight time, communication interruption time, and the weighted sum of the two. It can be seen that the present invention can complete the task with the least weighted sum time and achieve the expected purpose.

[0164] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for optimizing the path of a drone based on deep reinforcement learning, characterized in that, Includes the following steps: S1: Establish a communication model between the UAV and the radio network, including a scene model, a base station antenna radiation model, and a signal model. (I) Scene Model: Define the drone's flight area as D×D, and the height and location of obstacles within that area; Define the position of the drone at time t as q(t) = (x t ,y t ),t∈[0,T],x t ∈[0,D],y t ∈[0,D], where x t and y t The x and y coordinates of the drone at time t represent the position of the drone; T represents the total flight time taken by the drone from the starting position to the ending position. (II) Base station antenna radiation model: Assume there are M base station sectors, and construct the base station antenna radiation model A. A (θ,φ), where the base station antenna is a multi-element uniform linear array; (III) Signal Model: Analyze the path loss model between the UAV at position q(t) and cell m, m∈M, including line-of-sight path loss. Non-line-of-sight path loss S2: Calculate the probability of signal interruption for the drone in different locations, and construct a radio coverage probability map E accordingly. The specific method is as follows: S2.1: Based on the communication model established in step S1, analyze the instantaneous signal power y received by the UAV from cell m at time t. m (t) S2.2: According to y m (t) Calculate the signal-to-interference ratio (SIR) at time t; S2.3: Measure the probability P of communication interruption between the UAV at its location q(t) at time t and each associated cell b(t) based on the signal-to-interference ratio (SIR(t)). out (q(t),b(t)); S2.4: Based on the measured communication interruption probability, determine the optimal associated cell b(t) at position q(t). * : S2.5: Therefore, the communication coverage probability at position q(t) = 1 - the interruption probability of the UAV at position q(t), and the communication coverage probability value is saved in the radio coverage probability map E; where the interruption probability of the UAV at position q(t) is P. out (q(t)): S3: Take into account the UAV flight time and the communication coverage probability at different locations to set the optimization target for the UAV flight path; based on the optimization target, use deep reinforcement learning algorithm to optimize the UAV flight path.

2. The UAV path optimization method based on deep reinforcement learning according to claim 1, characterized in that, In step S1, A in the base station antenna radiation model A The construction process of (θ,φ) is as follows: A A (θ,φ)=G E,max -min{-[A E,V (θ)+A E,H (φ)],A m }+10log 10 [1+ρ(|a·w T | 2 -1)] In the formula, θ and φ are the elevation and azimuth angles of the base station antenna, respectively; G E,max A is the maximum directional gain of the antenna element in the main lobe direction. E,V (θ) and A E,H (φ) represent the vertical and horizontal radiation patterns of the antenna, respectively. m It is the ratio before and after, ρ is the correlation coefficient, a represents the amplitude vector, and w is the beamforming vector; Parameter A E,V (θ) and A E,H The specific formula for calculating (φ) is: In the formula, θ 3dB and φ 3dB These are the half-power beamwidths of the antenna in the vertical and horizontal directions, respectively; SLA V It is the sidelobe level limitation of the antenna.

3. The UAV path optimization method based on deep reinforcement learning according to claim 1, characterized in that, In step S1, the signal model considers the line-of-sight path loss. Non-line-of-sight path loss The calculations are as follows: In the formula, d m (q(t)) is the distance between the UAV at position q(t) and cell m; f c is the carrier frequency; h is the altitude of the UAV at position q(t).

4. The UAV path optimization method based on deep reinforcement learning according to claim 1, characterized in that, In step S2.1, the instantaneous signal power received by the UAV from cell m at position q(t) is y. m The specific calculation formula for (t) is as follows: In the formula, P m h is the transmission power of cell m; m β(q(t)) represents the channel power gain at time t; β(q(t)) represents the base station antenna gain at position q(t). It is a random variable representing the small-scale fading between the UAV and cell m at time t; This represents the large-scale channel power gain between the UAV at position q(t) and cell m. Where LoS link indicates a link on a line-of-sight path, and NLoS link indicates a link on a non-line-of-sight path.

5. The UAV path optimization method based on deep reinforcement learning according to claim 1, characterized in that, The specific content of calculating the signal-to-interference ratio (SIR(t)) at time t in step S2.2 is as follows: In the formula, b(t) represents a certain associated cell of the UAV at time t; y b(t) (t) represents the instantaneous signal power received by the UAV from the associated cell b(t) at time t.

6. The UAV path optimization method based on deep reinforcement learning according to claim 1, characterized in that, In step S2.3, calculate P out The specific content of (q(t),b(t)) is as follows: S2.3.1: Define the probability of communication interruption between a UAV at position q(t) and a certain associated cell b(t) as P. out (q(t),b(t)): In the formula, Pr(·) represents the probability of the event occurring; γ th The threshold is set when the signal-to-interference ratio (SIR(t)) is lower than γ. th At this time, the drone is considered to be in a state of communication interruption; S2.3.2: Rewrite the signal-to-interference ratio SIR(t) as follows: The variable time t is changed to the corresponding UAV position q(t), associated cell b(t), and small-scale fading between the UAV and associated cell b(t). Define the interrupt indicator function as follows S2.3.3: Based on the content of step S2.3.2, change the interruption probability P of S2.3.

1. out (q(t),b(t)) can be rewritten as Expected value: Then, the signal-to-interference ratio (SIR) between the UAV and each associated cell b(t) is measured J times within a certain time period to obtain the interruption probability between the UAV and each associated cell b(t) at that location: In the formula, Let j be the j-th measurement of small-scale fading between the UAV and the associated cell b(t) at time t.

7. The UAV path optimization method based on deep reinforcement learning according to claim 1, characterized in that, In step S3, the specific content of the optimization objective for setting the UAV flight path is as follows: S3.1: Constructing a continuous optimization objective function: s.t.q(0)=q s q(T)=q f In the formula, T represents the flight time of the UAV from the starting point to the ending point; μ is a non-negative coefficient; q(0) represents the initial position of the UAV; q s q represents the initial position of the drone; q(T) represents the final position of the drone; q f Represents the destination of the drone; S3.2: Discretize the continuous optimization problem from step S3.1 by dividing the UAV flight area into a series of adjacent grid points. The final objective function is equivalent to minimizing the weighted sum of the number of grid points N traversed by the UAV and the expected interruption probability. s.t.q0=q s q N =q f In the formula, q n This indicates the location of the UAV at point n (n∈N) in the grid; P out (q n ) represents the probability of the drone being interrupted at position n; q0 represents the starting position of the drone; q f This indicates the location of the drone's destination.

8. The UAV path optimization method based on deep reinforcement learning according to claim 1, characterized in that, In step S3, the specific content of optimizing the UAV flight path using a deep reinforcement learning algorithm based on the optimization objective is as follows: (I) First, conduct an actual test flight of the drone from the starting point to the finish line, as follows: 1) Set the maximum number of iterations for the actual test flight of the UAV to N. epi The maximum number of grid points the drone visits in each iteration, also known as the maximum number of steps the drone takes, is called N. step Initialize the drone exploration probability ε→ε0, and set the drone exploration probability decay rate α and the drone's reward value R for reaching the destination. des The penalty value P for the drone flying out of the designated D×D area. ob A non-negative coefficient μ, a replay experience pool D with a capacity of C, a radio coverage probability map E, and a map neural network with its parameters ξ, a depth Q network with its parameters θ, and a target depth Q′ network with its parameters θ. - =θ; Sets and initializes the default data structure of the experience replay summation tree, and assigns priority p to each leaf node of the summation tree. i Initialize to p s That is, p i →p s ; 2): The UAV begins to perform actual test flights; let the loop variable n be... epi =1, which represents the first iteration in the actual test flight mission; 3): Initialize a sliding window W of size N1, and initialize the initial position q of the actual test flight of the UAV. n =q s At this time q n In this context, n=0 indicates that the number of grid points traversed by the drone or the number of steps explored by the drone at this time is 0; 4): Select action v using the ε-greedy strategy n Specifically, it randomly selects an action in the action space with a probability of ε, and selects the optimal action with a probability of 1-ε. 5): Execute action v n To obtain the position q of the drone in the next state. n+1 q obtained by measurement in radio coverage probability map E n+1 Location interruption probability P out (q n+1 Set a single-step reward R n : R n =-1-μP out (q n+1 ) 6): (q) n ,v n ,R n ,q n+1 The reward is stored in a sliding window W; where, when n≥N1, the cumulative reward from step n-N1 to step n is calculated. Then, the position and action of step n-N1, the cumulative reward from step n-N1 to step n, and the position of step n. The data samples are stored in the summation tree nodes; 7): Update the number of steps n for drone exploration, and repeat steps 4)-7) to obtain multiple data samples; 8): Sample k nodes from the summation tree. The probability that each sample j will be sampled is p j This indicates the priority of node sample j. This represents the sum of priorities of all nodes in the summation tree, and the weights ω of the loss function for the nodes in the summation tree. j =(P(j) / min i P(i)) -β β determines the impact of priority experience playback on the convergence result; 9): Calculate the current reward value y for each sample j in the k-node samples of the summation tree. j The specific method is as follows: In the formula, This represents the cumulative reward for the drone from step j to step j+N1. Let S represent the drone's position at step j+N1, S represent the drone's flight area D×D, and γ represent the reward discount rate. This indicates that the target depth Q′ network is compared with the depth Q network in... Position selection of optimal action v * Evaluation reward value; 10): Regarding the loss function Perform gradient descent and update the parameters θ of the deep Q-network; where ω j Represents the weights of the loss function, This indicates that the target Q′ network is related to the Q network in q j Position selection action v j Evaluation reward value; 11): Based on the radio coverage probability map E, update the map neural network parameters ξ; then conduct a simulated test flight mission for the UAV: Step 1: Initialize the initial position of the UAV simulated test flight mission The "~" symbol above each parameter indicates that the current state is a simulated flight test mission, distinguishing it from parameters in actual flight. In This indicates that the number of grid points traversed or the number of steps explored by the drone during the simulated flight mission is 0; a loop variable is set. This indicates the first iteration in the simulated flight test mission; Step 2: Same as steps 4)-10), except that the interruption probability in step 5) is obtained by the prediction output of the map neural network, and the input parameter of the map neural network is the data in the radio coverage probability map E. Step 3: Determine whether the drone has reached the destination, gone out of bounds, or reached the maximum number of steps N. step If yes, proceed to step 4; otherwise, let Repeat the loop until step 2; Step 4: Analyze the number of iterations After each iteration B times, the network parameters θ for the target depth Q′ are updated. - →θ, then proceed to step 5; Step 5: If Then the loop ends; if Then let And return to step 1; 12) Returning to the actual test flight mission, determine whether the UAV has reached the destination, gone out of bounds, or reached the maximum number of steps N. step If so, proceed to step 13); otherwise, let n = n + 1 and the drone exploration probability ε → εα, and repeat the loop to step 4). 13) Analyze the number of iterations n epi In each iteration B, the network parameters θ for the target depth Q′ are updated. - →θ; then proceed to step 14); 14) If n epi =N epi If n epi <N epi Let n epi =n epi +1 and return to step 3); (ii) The parameters θ of the target depth Q′ network can be continuously improved through the entire process of steps 1)-14). - Therefore, the target depth Q′ network is designed for the depth Q network for the UAV at a certain position q. n Select action v n The evaluation reward value gradually reaches the optimal level. At this point, no actual test flights are conducted. Instead, the drone is guided to fly from different starting points to different endpoints directly through the target depth Q′ network after final training. During this process, the target depth Q′ network guides the selection of flight actions, enabling the drone to select the action with the highest reward value and complete the path planning.

9. A computer-readable storage medium storing a computer program, characterized in that, The computer program causes the computer to execute the UAV path optimization method as described in any one of claims 1-8.

10. An electronic device, characterized in that, include: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the UAV path optimization method as described in any one of claims 1-8.

Images