Airborne network deployment method based on two-stage Markov process

Abstract

The present invention provides an airborne network deployment method based on a two-stage Markov process, belonging to the field of drone-assisted communication networks. The method comprises: establishing a system model comprising an urban environment model, a drone model, and a channel model; defining an optimization problem for minimizing task execution time; designing the state space, action space, and reward function corresponding to the Markov process for the Mobile Steering System (MTS) and the Markov process for the Multi-Stage System (MMS) respectively; training the MTS and MMS models of drones in an urban environment using a deep reinforcement learning algorithm to obtain the optimization objective; and executing a two-stage drone deployment based on the MTS and MMS models to complete the data collection task from drone takeoff to network coverage completion. The method can support the safe and long-term deployment of multiple drones, achieve three-dimensional trajectory planning and efficient deployment of drones in urban environments, and thus achieve network coverage over large areas.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle (UAV)-assisted communication networks, and in particular relates to an airborne network deployment method based on a two-stage Markov process. Background Art

[0002] With the rapid development of UAV (Unmanned Aerial Vehicle) technology, drone-assisted networks, with their high flexibility, strong maneuverability, and low manufacturing costs, are effectively driving the development of future communication networks. They will also help promote the provision of wide-coverage, multi-functional drone-assisted communication services in 6G networks. Specifically, drones can serve as mobile base stations in cellular networks or as aerial base stations to provide network coverage for ground nodes and collect ground data. By designing drone flight trajectories and combining them with the coordinated deployment of multiple drones, network coverage and communication system performance can be further expanded. In recent years, the rapid development of embedded technology, wireless communication technology, and production technology has enabled the expansion of drone deployment methods from simple planning-based deployment methods to model-based optimization theory deployment methods, and further intelligent deployment methods based on massive data and artificial intelligence (AI) technology, greatly improving the system performance of drone-assisted communication networks.

[0003] However, in urban environments, where these systems are deployed, the efficient deployment of drone-assisted networks faces numerous challenges. First, obstruction of communication links by urban buildings can severely impact both link quality and communication performance. Second, the random distribution of urban buildings along drone flight paths can pose significant safety risks. Balancing drone flight safety and system performance in urban environments remains an open question requiring further research. Summary of the Invention

[0004] The purpose of this invention is to propose an airborne network deployment method based on a two-stage Markov process, which can support long-term safe deployment, realize three-dimensional trajectory planning and efficient deployment of drones in urban environments, and thus achieve network coverage in large areas.

[0005] The present invention is achieved through the following technical solutions:

[0006] The airborne network deployment method based on a two-stage Markov process includes the following steps:

[0007] Step S1: Establish a system model including an urban environment model, a UAV model, and a channel model. The urban environment model involves a mission area, a building set, and a ground user set.

[0008] Step S2: Based on the system model, define an optimization problem of minimizing the task execution time, which is subject to the constraints of the UAV's position and maneuverability, the task execution time, the obstacle avoidance, and the UAV network coverage and transmission power limit;

[0009] Step S3: Based on the system model, the state space, action space, and reward function corresponding to the Markov process of the MTS and the state space, action space, and reward function corresponding to the Markov process of the MMS are designed. During the mission execution, the UAV needs to pass through the MTS and MMS in sequence to complete the network coverage mission in the urban environment;

[0010] Step S4: Based on the MTS Markov process and MMS Markov process designed in step S3, the MTS model and MMS model of the UAV in the urban environment are trained respectively by deep reinforcement learning algorithm. During the MTS model training process, the goal is to minimize the MTS mission time T MTS During the MMS model training process, the goal is to minimize the MMS task time T MMS The sum of the task time of the two stages is the target T of the optimization problem in step S2. mission =T MTS +T MMS ;

[0011] Step S5: Based on the MTS model and MMS model obtained in step S4, a two-stage UAV deployment is performed to complete the data collection task from the UAV taking off from the starting position to the completion of network coverage.

[0012] Furthermore, in the step S1, in the urban environment model, the task area It is a square space with length, width and height of L, W and H respectively. K b buildings, collections of buildings The kth b The building vector is Assume that in the mission area There are ground users randomly distributed on the ground, and the ground user set The position of the gth ground user at time t is represented by p g,t =[x g,t ,y g,t ,z g,t ], in the mission area In the process, the UAV will start from the fixed take-off point p0=[0,0,0] and complete the mission area. Network coverage, where k b ∈[1,K b ], Represents building k b The minimum and maximum values ​​of the coordinates of the area occupied in the x-axis direction, Represents building k b The minimum and maximum values ​​of the coordinates of the area occupied in the y-axis direction, h b Represents building k b height, z g,t =0.

[0013] Furthermore, in the step S1, in the UAV model, the UAV is equipped with a downward-looking antenna, the coverage angle of the antenna is Θ, and the UAV flying altitude is H′, then the ground network coverage radius is R C =H′·tan(Θ), the positions of ground users and drones that meet the network coverage requirements should satisfy ||p g,t -p u,t ||≤R C In the UAV model, there are K UAVs that update their positions according to discrete time slots. The position of the selected updated UAV in time slot t is p u,t =[x u,t ,y u,t ,z u,t ]=T0·a u,t ⊙v+p u,t-1 ,in, a u,t =[a x,t ,a y,t ,a z,t ] represents the forward direction of the UAV in the x, y, and z axes, v = [v x ,v y ,v z ] represents the flight speed of the UAV in the x, y, and z axis directions, and T0 represents the time slot length.

[0014] Further, in the step S1, in the channel model, the air-to-ground channel includes a direct link and an indirect link. For the direct link, the channel gain of the link is The channel signal-to-noise ratio is For a non-direct link, the channel gain of the link is The channel signal-to-noise ratio is d u,g ≥d0, where d0=1 represents unit distance, d u,g is the distance between the UAV and the corresponding ground user, η LoS is the path loss index under direct link, f c Indicates the carrier frequency, Y LoS is the random reflection fading distribution parameter under direct link, G LoS is the characteristic constant of the communication channel under the direct link, ηNLoS is the path loss index under non-direct link, Y NLoS is the random reflection fading distribution parameter under non-direct link, G NLoS is the characteristic constant of the communication channel under the non-direct link.

[0015] In the further step S2, the optimization problem is expressed as Among them, C1-C3 are the position and maneuver constraints of the UAV, C4 is the task execution time constraint, C5 is the obstacle avoidance constraint, and C6-C8 are the UAV network coverage and transmission power limitations. represents the set of drones, u represents the current drone, and T represents the upper limit of the task execution time. represents the signal-to-noise ratio of the communication link between the UAV and the ground user without considering interference, h u,g is the channel gain of the communication link between the UAV and the ground user, p is the signal transmission power, is the channel noise power, γ th represents the minimum signal-to-noise ratio for network access, γ u,g ≥γ th When the ground user coverage index c is met g,t =1, indicating that the ground user g is covered by the UAV network in time slot t, otherwise c g,t = 0, indicating that the ground user g is not covered by the UAV network in time slot t, s g,t Represents the ground user service index, s g,t =1 indicates that the ground user has completed network access and uploaded and sent its own data, otherwise s g,t =0,η g,t It represents the ground user access index, which is defined as the indication of whether the ground user wants to access the network when it is covered by the UAV network. g,t =1 and s g,t = 0, the ground user g needs to access the network, that is, η g,t =1, otherwise η g,t =0.

[0016] Furthermore, in step S3, defining the Markov process of MTS specifically includes:

[0017] Step S31, design the state space of MTS: the state space vector expression of MTS is Among them, p u,tar represents the target position, d t =||p u,t -p u,tar ||2 represents the distance between the UAV and the target position at time slot t, d0=||p u,t＝0 -p u,tar||2 represents the initial distance between the UAV and the target position, p u,t＝0 Indicates the position of the drone at the starting moment, o t Indicates the obstacle environment information around the drone. t is a 26-dimensional vector representing the environmental information, specifically the distance from the current position p of the drone to the surrounding environment. u,t When the next move is made in the mission area, whether there is a building in each moving direction, when the UAV moves, T0·[v x ,v y ,v z ] is the side length, and moves in the order of the formula [x,y,z]={1,0,-1;1,0,-1;1,0,-1} on the x, y, and z axes. In this formula, x, y, and z cannot be 0 at the same time. 1 means moving forward in that direction, and -1 means moving backward in that direction. If there is a building in a certain moving direction, then o t The corresponding element value is 1, otherwise it is 0;

[0018] Step S32, design the action space of MTS: In MTS, the action of the drone in time slot t is with a u,t The corresponding relationship is shown as [x,y,z]={1,0,-1; 1,0,-1; 1,0,-1}. The value corresponding to a u,t The indices of the different elements in , i.e.

[0019] Step S33: Design the profit function of MTS as in, is the distance gain, which indicates the change in the distance between the UAV and the target point at each time step after the maneuver, is a static penalty, which indicates that the action output by the drone in a certain time slot is the benefit of maintaining static conditions. is the crash penalty, which indicates the benefit when there are obstacles within the range of the drone's movement in the time slot T0. This is an out-of-bounds penalty, which indicates that the drone has exceeded the mission area during movement. range, is the reward for reaching the destination, which indicates the reward obtained by the drone when it reaches the target position. ω1, ω2, ω3, ω4, and ω5 are all constant terms.

[0020] Further in step S3, defining the Markov process of MMS specifically includes:

[0021] Step S34, design the state space of MMS: the state space vector expression of MMS is in, is the information age of adjacent time slots, κ c K is the information age accumulated in a single time slot when the ground user is served by the UAV. c is the number of all served users in the current time slot t, χ loss represents the information age offset of served users, K is the number of ground users;

[0022] Step S35, design the action space of MMS: In MMS, the action of the drone in time slot t is with a u,t The corresponding relationship is shown as [x,y,z]={1,0,-1;1,0,-1;0}. The value corresponding to a u,t The indices of the different elements in , i.e.

[0023] Step S36: Design the profit function of MMS as Among them, Tt represents the remaining time after completing the network coverage of all users as the reward for task completion, p beyond Represents the penalty imposed on a drone for crossing the boundary.

[0024] Furthermore, in step S4, the action space of MTS and the action space of MMS do not interfere with each other, and the MTS model training and the MMS model training are deployed in the same virtual environment and performed simultaneously.

[0025] Further, the step S5 comprises the following steps:

[0026] Step S51: Perform K-means clustering on ground users based on the number of drones N, and assign the mission areas where different ground user subsets are located to corresponding drones. The drones matching the corresponding user subsets will be responsible for the user network coverage of the set;

[0027] Step S52: Change the cluster center c of the user subset corresponding to the drone. u The flight height H' above the location, execute MMS, and select the optimal height H with the least mission execution time o , as the flight altitude of the UAV under MMS in real deployment;

[0028] Step S53, with [c u ,H o ] is the target position, the UAV takes off, performs MTS, and maneuvers toward the target position;

[0029] Step S54: When the UAV enters the corresponding task sub-area, MMS is executed and the UAV starts to perform the network coverage task until the network coverage is completed.

[0030] The present invention has the following beneficial effects:

[0031] 1. The present invention first establishes a system model including an urban environment model, a drone model and a channel model. Secondly, based on the system model, it defines the optimization problem of minimizing the task execution time. Then, based on the system model, the Markov process of MTS and the Markov process of MMS are designed respectively. The MTS model and the MMS model are obtained through deep reinforcement learning algorithm training, and then the goal of the optimization problem of minimizing the task execution time is obtained. Finally, according to the MTS model and the MMS model, a two-stage drone deployment is performed to complete the data collection task from the drone taking off from the starting position to the completion of network coverage. This solves the problem of buildings in urban environments blocking the drone auxiliary communication network link and affecting the safety of the drone flight link in the existing technology, and minimizes the time for the drone to complete network coverage while ensuring the safety of the drone flight. It also realizes the optimization iteration of the drone deployment strategy based on artificial intelligence and data experience, and improves the deployment strategy performance compared with traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The present invention will be described in further detail below with reference to the accompanying drawings.

[0033] Figure 1 Flowchart of the present invention.

[0034] Figure 2 This is a graph showing the results of the MTS model training convergence process of the present invention.

[0035] Figure 3 This is a graph showing the results of the MMS model training convergence process of the present invention.

[0036] Figure 4 This is the result diagram of the time it takes for the MTS model of the present invention to deploy drones to reach the target point.

[0037] Figure 5 This is a graph showing the network coverage time results of the MMS model of the present invention deploying drones to complete the task.

[0038] Figure 6 This is a bird's-eye view of the trajectory covered by the drone network of the present invention.

[0039] Figure 7 This is a three-dimensional result diagram of the trajectory of the drone network coverage of the present invention. DETAILED DESCRIPTION

[0040] like Figure 1 As shown, the airborne network deployment method based on the two-stage Markov process includes the following steps:

[0041] Step S1: Establish a system model including an urban environment model, a UAV model, and a channel model. The urban environment model involves a mission area, a building set, and a ground user set.

[0042] Specifically, in the urban environment model, the mission area It is a square space with length, width and height of L, W and H respectively. K b buildings, collections of buildings The kth b The building vector is The vector dimension is 1×5, assuming that in the task area There are ground users randomly distributed on the ground, and the ground user set The position of the gth ground user at time t is represented by p g,t =[x g,t ,y g,t ,z g,t ], in the mission area In the process, the UAV will start from the fixed take-off point p0=[0,0,0] and complete the mission area. network coverage, of which Represents building k b The minimum and maximum values ​​of the coordinates of the area occupied in the x-axis direction, Represents building k b The minimum and maximum values ​​of the coordinates of the area occupied in the y-axis direction, h b Represents building k b height, z g,t =0.

[0043] Specifically in this embodiment, L=500m, W=500m, H=200m, K b =120, number of ground users T0 = ​​0.25s, urban building area range is 1m 2 -17m 2 , the height range is 10m-120m.

[0044] In the UAV model, the UAV is equipped with a downward-looking antenna, the coverage angle of the antenna is Θ, and the flight altitude of the UAV is H′, then the ground network coverage radius is R C =H′·tan(Θ), the positions of ground users and drones that meet the network coverage requirements should satisfy ||p g,t -p u,t ||≤R C ,The UAV can also perceive the surrounding environment to obtain ,information about obstacles in the surrounding environment;

[0045] In the drone model, there are K drones that update their positions according to discrete time slots. Assume that a u,t is the flight direction vector of the UAV at time t, v v 、v h Represent the horizontal and vertical speeds of the drone respectively. The position of the drone at time slot t is represented by p u,t =T0·a u,t ⊙v+p u,t-1 , where v = [v x ,v y ,v z ] represents the flight speed of the drone in the x, y, and z axes, specifically v x =v y =v v , v z =v h , a u,t =[a x,t ,a y,t ,a z,t ]∈{0,±1} represents the update strategy selected to update the UAV’s forward direction in the x, y, and z axes. +1 represents forward movement in this direction, 0 represents stationary movement in this direction, and -1 represents backward movement in this direction. u,t ⊙v is defined as T0 represents the time slot length.

[0046] Specifically in this example, K = 5, v v =10m / s,v h =5m / s, Θ=[-60°,60°], H′=120m.

[0047] In the channel model, the air-to-ground channel includes direct links (Line-of-Sight, LoS) and non-direct links (NonLoS, NLoS). In the urban environment model, the system will automatically detect the communication link between the drone and the ground user. u,g Is there a building blocking the link? u,g The channel gain is h u,g , ignoring interference, the link signal-to-noise ratio (SNR)γ u,g Modeled as Where p is the signal transmission power, is the link noise power.

[0048] For a direct link, the channel gain is Among them, c LoS is the channel multiple response under the condition of unit distance d0=1, d u,g is the distance between the UAV and the corresponding ground user, ηLoS is the path loss index, Y LoS is the channel random reflection fading distribution parameter under LoS channel.

[0049] c LoS The calculation expression is in, is the free space loss;

[0050] Therefore, the channel gain can be further expressed as

[0051] The channel signal-to-noise ratio is expressed as Where d0=1 represents the unit distance, d u,g is the distance between the UAV and the corresponding ground user, η LoS is the path loss index under direct link, f c Indicates the carrier frequency, Y LoS is the random reflection fading distribution parameter under direct link, G LoS Communication channel characteristic constants under direct link.

[0052] The expressions of the link channel gain and channel signal-to-noise ratio of the indirect link are the same as those of the direct link, where the path loss exponent, random reflection fading distribution parameter and communication channel characteristic constant under the indirect link are expressed as η NLoS 、Y NLoS and G NLoS , that is, the link channel gain of the non-direct link is The channel signal-to-noise ratio is

[0053] Specifically in this embodiment, p=0.1dBW, σ 2 =-75dBm, γ th =-0dB,η LoS =0.1dB,η NLoS =21dB, f c =2.4GHz, Y LoS and Y NLoS are modeled as Rayleigh distribution parameters with expected values ​​equal to 10dB and 5dB, respectively.

[0054] Step S2: Based on the system model, define an optimization problem of minimizing the task execution time, which is subject to the constraints of the UAV's position and maneuverability, the task execution time, the obstacle avoidance, and the UAV network coverage and transmission power limit;

[0055] First, define intermediate variables such as target location, ground user coverage index, ground user service index, ground user access index, and maximum access limit:

[0056] Target position pu,tar It is defined as the center of gravity of the ground user and the starting point of the UAV network coverage mission;

[0057] Ground user coverage index c g,t It is defined as indicating whether the ground user g is covered by the UAV network in time slot t, specifically: if the link l u,g The SNR satisfies γ u,g ≥γ th , then c g,t =1, otherwise c g,t =0, where γ th Indicates the minimum signal-to-noise ratio for network access;

[0058] Ground user service indicators g,t It is defined as whether the ground user has completed network access and completed uploading and sending of its own data. If so, s g,t =1, otherwise s g,t =0;

[0059] Ground user access indication η g,t It is defined as an indication of whether to intervene in the network when the ground user is covered by the drone network. Specifically, when c g,t =1 and s g,t = 0, ground user g needs to access the network, η g,t =1, otherwise η g,t =0;

[0060] Maximum access limit N max Defined as the maximum number of ground users that a drone can access;

[0061] Based on the system model and the above intermediate variables, the optimization problem of minimizing task execution time is expressed as Among them, C1-C3 are the position and maneuver constraints of the UAV, C4 is the task execution time constraint, C5 is the obstacle avoidance constraint, and C6-C8 are the UAV network coverage and transmission power limitations. Represents the set of drones, u represents the current drone, and T represents the upper limit of the task execution time.

[0062] Specifically in this embodiment, N max =20.

[0063] Step S3: Based on the system model, the state space, action space, and reward function corresponding to the Markov process of the MTS and the state space, action space, and reward function corresponding to the Markov process of the MMS are designed. During the mission execution, the UAV needs to pass through the MTS and MMS in sequence to complete the network coverage mission in the urban environment;

[0064] The Markov process that defines MTS specifically includes:

[0065] Step S31, design the state space of MTS: The state space of MTS is composed of the position information of the UAV, the distance information between the UAV and the target position, and the obstacle information around the UAV. At different time slots t, the state space of MTS is composed of the feature vector s t Representation, the state space vector expression of MTS is Among them, p u,tar represents the target position, d t =||p u,t -p u,tar ||2 represents the distance between the UAV and the target position at time slot t, d0=||p u,t＝0 -p u,tar ||2 represents the initial distance between the UAV and the target position, p u,t＝0 Indicates the position of the drone at the starting moment, o t Indicates the obstacle environment information around the drone. t is a 26-dimensional vector representing the environmental information, specifically the distance from the current position p of the drone to the surrounding environment. u,t When the next move is made in the mission area, whether there is a building in each moving direction, when the UAV moves, T0·[v x ,v y ,v z ] is the side length, and moves in the order of the formula [x,y,z]={1,0,-1;1,0,-1;1,0,-1} on the x, y, and z axes. In this formula, x, y, and z cannot be 0 at the same time. 1 means moving forward in that direction, and -1 means moving backward in that direction. If there is a building in a certain moving direction, then o t The corresponding element value is 1, otherwise it is 0;

[0066] Step S32, design the action space of MTS: In MTS, the action of the drone in time slot t is That is, the direction of the drone's movement a u,t , with a u,t The corresponding relationship is shown as [x,y,z]={1,0,-1; 1,0,-1; 1,0,-1}. The value corresponding to a u,t The indices of the different elements in , i.e.

[0067] Step S33: Design the profit function of MTS as in, is the distance gain, which indicates the change in the distance between the UAV and the target point at each time step after the maneuver, is a static penalty, which indicates that the action output by the drone in a certain time slot is the benefit of maintaining static conditions. is the crash penalty, which indicates the benefit when there are obstacles within the range of the drone's movement in the time slot T0. This is an out-of-bounds penalty, which indicates that the drone has exceeded the scope of the mission area S during its movement. is the reward for reaching the destination, which indicates the reward obtained by the drone when it reaches the target position. ω1, ω2, ω3, ω4, and ω5 are all constant terms.

[0068] Specifically in this embodiment, [ω1,ω2,ω3,ω4,ω5]=[10,10,25,20,200].

[0069] The Markov process that defines MMS specifically includes:

[0070] Step S34, design the state space of MMS: The state space of MMS includes location information, network coverage information and information age information, which is represented by vector s at different time slots t. t Representation, the state space vector expression of MMS is in, is the information age of adjacent time slots, κ c K is the information age accumulated in a single time slot when the ground user is served by the UAV. c is the number of all served users in the current time slot t, χ loss represents the information age offset of served users, K is the number of ground users;

[0071] Step S35, design the action space of MMS: In MMS, the action of the drone in time slot t is The direction of the drone is a u,t , with a u,t The corresponding relationship is shown as [x,y,z]={1,0,-1;1,0,-1;0}. The value corresponding to a u,t The indices of the different elements in , i.e.

[0072] Step S36: Design the profit function of MMS as Among them, Tt represents the remaining time after completing the network coverage of all users as the reward for task completion, p beyond Represents the penalty imposed on a drone for crossing the boundary.

[0073] Specifically in this embodiment, κ c =10,χ loss =1.

[0074] Step S4: Based on the MTS Markov process and MMS Markov process designed in step S3, the MTS model and MMS model of the UAV in the urban environment are trained respectively by deep reinforcement learning algorithm. During the MTS model training process, the goal is to minimize the MTS mission time T MTS During the MMS model training process, the goal is to minimize the MMS task time T MMS The sum of the task time of the two stages is the target T of the optimization problem in step S2. mission =T MTS +T MMS ;

[0075] During the MTS training process, the drone has a fixed initial position p0 and a randomly assigned target position p u,tar During the MMS training process, the distribution of ground users is fixed and the initial position of the drone is randomly assigned to complete network coverage. The action space of MTS and the action space of MMS do not interfere with each other, so the MTS model training and MMS model training can be deployed in the same virtual environment and carried out simultaneously. By configuring the experience replay capacity D, the decay rate δ, and the number of sampling batches D s The training of MTS and MMS models is carried out with parameters such as learning rate α.

[0076] Specifically in this example, the deep reinforcement algorithm Double Deep Q-Learning (DDQN) algorithm is used for training, D = 500000, δ = 0.99, D s =512,α=0.0004.

[0077] Step S5: Based on the MTS model and MMS model obtained in step S4, a two-stage UAV deployment is performed to complete the data collection task from the UAV taking off from the starting position to the completion of network coverage;

[0078] The specific steps include:

[0079] Step S51: Perform K-means clustering on ground users based on the number of drones N, and assign the mission areas where different ground user subsets are located to corresponding drones. The drones matching the corresponding user subsets will be responsible for the user network coverage of the set;

[0080] Step S52: Change the cluster center c of the user subset corresponding to the drone. u The flight height H' above the location, execute MMS, and select the optimal height H with the least mission execution time o , as the flight altitude of the UAV under MMS in real deployment;

[0081] Step S53, with [c u ,Ho ] is the target position, the UAV takes off, performs MTS, and maneuvers toward the target position;

[0082] Step S54: When the UAV enters the corresponding task sub-area, MMS is executed and the UAV starts to perform the network coverage task until the network coverage is completed.

[0083] Specifically in this embodiment, H o =120m,c u Depends on the distribution of ground users.

[0084] Figure 2 In , the horizontal axis is the number of training rounds, which ranges from 1 to 5000, and the vertical axis is the round profit, that is, the function value of the MTS profit function in different rounds. Figure 2 It can be seen that after training, the MTS model tends to converge, which proves the effectiveness of the MTS model training of the present invention.

[0085] Figure 3 In , the horizontal axis is the number of training rounds, which ranges from 1 to 3500, and the vertical axis is the round profit, that is, the function value of the MMS profit function in different rounds. Figure 3 It can be seen that after training, the MMS model tends to converge, which proves the effectiveness of the MMS model training of the present invention.

[0086] Figure 4 The average mission execution time of the UAV from a fixed starting point to 20 random target points is counted in the figure. The horizontal axis of the figure is the different methods of the UAV to the target location, and the vertical axis is the MTS mission execution time in seconds. Figure 4 It can be seen that the MTS model increases the time consumption compared to the straight-line maneuvering method that completely ignores obstacles, but reduces the task time by nearly 40 seconds compared to the coordinate translation method that avoids buildings in a completely straight line. This proves the effectiveness of the MTS model of the present invention and realizes the safe deployment of drones.

[0087] Figure 5 In the figure, the horizontal axis is the altitude of the UAV, which is 100, 20, 300, 400, and 500 in order, in meters, and the vertical axis is the mission execution time for MMS to achieve network coverage of all ground users. Figure 5 The average task execution time of the UAV to complete the ground user network coverage under the premise of 20 random location departures was compared among the scanning method, ant colony algorithm and MMS model. Figure 5 It can be seen that the MMS model takes the shortest time, while the scanning method takes the longest time, which proves the effectiveness of the MMS model of the present invention in minimizing the task execution time.

[0088] Figure 6In the figure, the horizontal axis is the x-axis of the mission area, and the vertical axis is the y-axis of the mission area. The solid line is the UAV's maneuver trajectory, the squares are buildings, and the triangles are ground users. Figure 7 In the figure, the horizontal axis is the x-axis and y-axis of the mission area, and the vertical axis is the z-axis of the mission area. The model is the maneuvering trajectory of the drone, and the cuboid is the building model. Figure 6 and Figure 7 It can be seen that after the drone departs from the starting position in the lower left corner, it goes to different locations in the mission area to perform network coverage. The first half of the maneuvering trajectory is MTS. Since the drone is flying at a low altitude, it needs to avoid buildings to prevent collisions, so the drone trajectory has more turns. When the drone reaches the target location at height H o After that, the MTS trajectory becomes almost a straight line. When the drone reaches the mission area, it switches to MMS and begins the network coverage mission. You can see that the drone's trajectory begins to turn or turn around, corresponding to the process of the drone hovering over the mission area to complete the network coverage of ground users.

[0089] In summary, the MTS model and MMS model of the present invention can be reasonably matched to optimize the deployment of UAV-assisted communication networks in urban environments, which has theoretical and practical guiding significance.

[0090] The above description is merely a preferred embodiment of the present invention and therefore cannot be used to limit the scope of the present invention. In other words, equivalent changes and modifications made according to the scope of the patent application and the contents of the specification should still fall within the scope of the patent of the present invention.

Claims

1. A method for deploying an airborne network based on a two-stage Markov process, characterized by: The steps include: Step S1: Establish a system model including an urban environment model, a UAV model, and a channel model. The urban environment model involves a mission area, a building set, and a ground user set. Step S2: Based on the system model, define an optimization problem of minimizing the task execution time, which is subject to the constraints of the UAV's position and maneuverability, the task execution time, the obstacle avoidance, and the UAV network coverage and transmission power limit; Step S3: Based on the system model, the state space, action space, and reward function corresponding to the Markov process of the MTS and the state space, action space, and reward function corresponding to the Markov process of the MMS are designed. During the mission execution, the UAV needs to pass through the MTS and MMS in sequence to complete the network coverage mission in the urban environment; Step S4: Based on the MTS Markov process and MMS Markov process designed in step S3, the MTS model and MMS model of the UAV in the urban environment are trained respectively by deep reinforcement learning algorithm. During the MTS model training process, the goal is to minimize the MTS mission time T MTS During the MMS model training process, the goal is to minimize the MMS task time T MMS The sum of the task time of the two stages is the target T of the optimization problem in step S2. mission =T MTS +T MMS ; Step S5: Based on the MTS model and MMS model obtained in step S4, a two-stage UAV deployment is performed to complete the data collection task from the UAV taking off from the starting position to the completion of network coverage; In the step S1, in the urban environment model, the task area It is a square space with length, width and height of L, W and H respectively. K b buildings, collections of buildings The kth b The building vector is Assume that in the mission area There are ground users randomly distributed on the ground, and the ground user set The position of the gth ground user at time t is represented by p g,t =[x g,t ,y g,t ,z g,t ], in the mission area In the process, the UAV will start from the fixed take-off point p0=[0,0,0] and complete the mission area. Network coverage, where k b ∈[1,K b ], Represents building k b The minimum and maximum values ​​of the coordinates of the area occupied in the x-axis direction, Represents building k b The minimum and maximum values ​​of the coordinates of the area occupied in the y-axis direction, h b Represents building k b height, z g,t =0; In step S1, in the UAV model, the UAV is equipped with a downward-looking antenna, the coverage angle of the antenna is Θ, the UAV flying height is H′, and the ground network coverage radius is R C =H′·tan(Θ), the positions of ground users and drones that meet the network coverage requirements should satisfy ||p g,t -p u,t ||≤R C In the UAV model, there are K UAVs that update their positions according to discrete time slots. The position of the selected updated UAV in time slot t is p u,t =[x u,t ,y u,t ,z u,t ]=T0·a u,t ⊙v+p u,t-1 ,in, a u,t =[a x,t ,a y,t ,a z,t ] represents the forward direction of the UAV in the x, y, and z axes, v = [v x ,v y ,v z ] represents the flight speed of the UAV in the x, y, and z axis directions, and T0 represents the time slot length; In step S1, in the channel model, the air-to-ground channel includes a direct link and an indirect link. For the direct link, the channel gain of the link is The channel signal-to-noise ratio is d u,g ≥d0, for non-direct links, the channel gain of the link is The channel signal-to-noise ratio is d u,g ≥d0, where d0=1 represents unit distance, d u,g is the distance between the UAV and the corresponding ground user, η LoS is the path loss index under direct link, f c Indicates the carrier frequency, Y LoS is the random reflection fading distribution parameter under direct link, G LoS is the characteristic constant of the communication channel under the direct link, η NLoS is the path loss index under non-direct link, Y NLoS is the random reflection fading distribution parameter under non-direct link, G NLoS is the characteristic constant of the communication channel under the non-direct link.

2. The airborne network deployment method based on a two-stage Markov process according to claim 1, characterized in that: In step S2, the optimization problem is expressed as Among them, C1-C3 are the position and maneuver constraints of the UAV, C4 is the task execution time constraint, C5 is the obstacle avoidance constraint, and C6-C8 are the UAV network coverage and transmission power limitations. represents the set of drones, u represents the current drone, and T represents the upper limit of the task execution time. represents the signal-to-noise ratio of the communication link between the UAV and the ground user without considering interference, h u,g is the channel gain of the communication link between the UAV and the ground user, p is the signal transmission power, is the channel noise power, γ th represents the minimum signal-to-noise ratio for network access, γ u,g ≥γ th When the ground user coverage index c is met g,t =1, indicating that the ground user g is covered by the UAV network in time slot t, otherwise c g,t = 0, indicating that the ground user g is not covered by the UAV network in time slot t, s g,t Represents the ground user service index, s g,t =1 indicates that the ground user has completed network access and uploaded and sent its own data, otherwise s g,t =0,η g,t It represents the ground user access index, which is defined as the indication of whether the ground user wants to access the network when it is covered by the UAV network. g,t =1 and s g,t = 0, the ground user g needs to access the network, that is, η g,t =1, otherwise η g,t =0.

3. The airborne network deployment method based on a two-stage Markov process according to claim 2, characterized in that: In step S3, defining the Markov process of MTS specifically includes: Step S31, design the state space of MTS: the state space vector expression of MTS is Among them, p u,tar represents the target position, d t =||p u,t -p u,tar ||2 represents the distance between the UAV and the target position at time slot t, d0=||p u,t＝0 -p u,tar ||2 represents the initial distance between the UAV and the target position, p u,t＝0 Indicates the position of the drone at the starting moment, o t Indicates the obstacle environment information around the drone. t is a 26-dimensional vector representing the environmental information, specifically the distance from the current position p of the drone to the surrounding environment. u,t When the next move is made in the mission area, whether there is a building in each moving direction, when the UAV moves, T0·[v x ,v y ,v z ] is the side length, and moves in the order of the formula [x,y,z]={1,0,-1;1,0,-1;1,0,-1} on the x, y, and z axes. In this formula, x, y, and z cannot be 0 at the same time. 1 means moving forward in that direction, and -1 means moving backward in that direction. If there is a building in a certain moving direction, then o t The corresponding element value is 1, otherwise it is 0; Step S32, design the action space of MTS: In MTS, the action of the drone in time slot t is with a u,t The corresponding relationship is shown as [x,y,z]={1,0,-1; 1,0,-1; 1,0,-1}. The value corresponding to a u,t The indices of the different elements in , i.e. Step S33: Design the profit function of MTS as in, is the distance gain, which indicates the change in the distance between the UAV and the target point at each time step after the maneuver, is a static penalty, which indicates that the action output by the drone in a certain time slot is the benefit of maintaining static conditions. is the crash penalty, which indicates the benefit when there are obstacles within the range of the drone's movement in the time slot T0. This is an out-of-bounds penalty, which indicates that the drone has exceeded the mission area during movement. range, is the reward for reaching the destination, which indicates the reward obtained by the drone when it reaches the target position. ω1, ω2, ω3, ω4, and ω5 are all constant terms.

4. The airborne network deployment method based on a two-stage Markov process according to claim 3 is characterized in that: In step S3, defining the Markov process of MMS specifically includes: Step S34, design the state space of MMS: the state space vector expression of MMS is in, is the information age of adjacent time slots, κ c K is the information age accumulated in a single time slot when the ground user is served by the UAV. c is the number of all served users in the current time slot t, χ loss represents the information age offset of served users, K is the number of ground users; Step S35, design the action space of MMS: In MMS, the action of the drone in time slot t is with a u,t The corresponding relationship is shown as [x,y,z]={1,0,-1;1,0,-1;0}. The value corresponding to a u,t The indices of the different elements in , i.e. Step S36: Design the profit function of MMS as Among them, Tt represents the remaining time after completing the network coverage of all users as the reward for task completion, p beyond Represents the penalty imposed on a drone for crossing the boundary.

5. The airborne network deployment method based on a two-stage Markov process according to claim 4, characterized in that: In step S4, the action space of MTS and the action space of MMS do not interfere with each other, and the MTS model training and the MMS model training are deployed in the same virtual environment and performed simultaneously.

6. The airborne network deployment method based on a two-stage Markov process according to claim 5, characterized in that: The step S5 comprises the following steps: Step S51: Perform K-means clustering on ground users based on the number of drones N, and assign the mission areas where different ground user subsets are located to corresponding drones. The drones matching the corresponding user subsets will be responsible for the user network coverage of the set; Step S52: Change the cluster center c of the user subset corresponding to the drone. u The flight height H' above the location, execute MMS, and select the optimal height H with the least mission execution time o , as the flight altitude of the UAV under MMS in real deployment; Step S53, with [c u ,H o ] is the target position, the UAV takes off, performs MTS, and maneuvers toward the target position; Step S54: When the UAV enters the corresponding task sub-area, MMS is executed and the UAV starts to perform the network coverage task until the network coverage is completed.

Images