Unmanned aerial vehicle auxiliary network path optimization method and system based on deep reinforcement learning

Optimizing the UAV path through deep reinforcement learning has solved the challenges of UAV trajectory planning in dynamic environments, achieved efficient communication services and energy consumption reduction, and adapted to changes in user needs.

CN119094975AActive Publication Date: 2024-12-06NAT UNIV OF DEFENSE TECH

Patent Information

Application Number
CN202411119870.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-12-06
Estimated Expiration
2044-08-15

AI Technical Summary

Technical Problem

In a dynamically changing environment, existing drone-assisted communication systems are difficult to effectively optimize drone paths to meet the dynamic mobility and service quality needs of ground users, especially how to reasonably plan drone trajectory under energy limitations to provide high efficiency Communication services are a challenge.

Method used

Using a method based on deep reinforcement learning, the drone base station downlink scenario is constructed, the user dynamic movement is simulated using the Gauss Markov model, and the joint optimization model is generated, and network training is accelerated through dual competition deep Q networks and imagination mechanisms, and the network training is optimized. UAV paths to minimize energy consumption and improve transmission efficiency.

Benefits of technology

It significantly reduces the energy consumption of drones, improves transmission efficiency and service quality in dynamic environments, adapts to changes in user needs, and achieves more efficient drone path planning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094975A_ABST
    Figure CN119094975A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of electromagnetic spectrum, and particularly discloses an unmanned aerial vehicle auxiliary network path optimization method and system based on deep reinforcement learning, and the method comprises the steps: constructing a typical unmanned aerial vehicle base station downlink scene, and simulating the dynamic movement behavior of a user according to a Gaussian Markov model; performing system modeling analysis according to the downlink scene of the unmanned aerial vehicle base station, and generating a joint optimization model by taking minimization of energy consumption as a target; according to the method, the optimal solution of the joint optimization model is solved through a deep reinforcement learning algorithm, the double-competition deep Q network and an imagination mechanism (IM) are fused, the double-Q network and a competition structure are adopted to process the over-estimation problem of a value function, the IM is utilized to spread information in a cross-round mode to accelerate network training, and the optimal solution of the joint optimization model is solved. And changing electromagnetic environments and user requirements can be dynamically adapted.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of electromagnetic spectrum technology, and in particular to a drone-assisted network path optimization method and system based on deep reinforcement learning. Background Art

[0002] In recent years, drone-assisted communications have attracted widespread attention due to their flexibility and affordability, as well as their potential to enhance coverage capabilities in various scenarios. Their superior mobility has enabled drone-assisted communication systems to play an important role in emergency communications and broadband connectivity in remote areas. For example, when natural disasters severely damage ground communication infrastructure, drones can serve as temporary base stations to quickly restore ground users' connections to the backbone network. In addition, compared with traditional ground communication systems, drones more frequently establish line-of-sight (LoS) links with ground users, providing better communication services. However, ensuring the quality of service that meets ground users when providing services to users under energy constraints remains a major challenge. The dynamic movement of users leads to constantly changing signal quality, and the different service requirements of different users put higher demands on drone trajectory planning and service strategies. Therefore, how to reasonably plan the flight path of drones in a dynamically changing environment to meet the needs of all users has become one of the key research issues.

[0003] Traditional UAV trajectory planning problems can usually be abstracted into traveling salesman problems and picking problems. By solving these problems, the initial trajectory of the UAV can be obtained. For example, some studies have focused on the trajectory and resource allocation of UAVs to minimize the time consumption of specific tasks. The problem is discretized using a set of line segments to represent the actual trajectory, and the block coordinate descent algorithm is used to decouple it into a resource allocation subproblem and a trajectory optimization subproblem. Other studies have jointly optimized the three-dimensional UAV trajectory and speed-time profile under the constraints of onboard energy, speed, acceleration, and completion time to maximize the total user throughput. The original problem is decoupled into two subproblems. First, the trajectory is optimized considering the speed constraint, and in the second subproblem, the speed and time of each time slot are optimized. Compared with traditional convex optimization algorithms, algorithms based on reinforcement learning may have higher performance and can adapt to more complex environments.

[0004] Reinforcement learning is a machine learning algorithm that updates and iterates existing models by continuously using new data. Its characteristic is that in the optimization process, it can not only use existing data, but also obtain new data through exploration of the environment, so as to achieve the goal of real-time dynamic optimization. Some studies have proposed a distributed trajectory optimization algorithm based on dual-stream attention multi-agent actor-critic to control multiple drones to achieve on-demand coverage of wireless communication networks. Other studies have used a MADDPG algorithm to improve the fairness of the covered area for drones to respond quickly to sudden and emergency situations.

[0005] Although reinforcement learning has shown great potential in drone-assisted wireless networks, many current studies are still based on static user service quality. In addition, many existing studies rarely consider the issue of drone flight data sampling when training algorithms. However, the actual network environment is constantly changing, the user's location is also moving in real time, and the drone should make the best use of data samples. This requires that the drone path optimization can be reasonably adjusted according to the user's location and service status. Therefore, there is an urgent need for a more in-depth exploration of how to efficiently implement reinforcement learning methods in a constantly changing environment to achieve drone path optimization. Summary of the invention

[0006] To achieve the purpose of the present invention, the present application provides a UAV-assisted network path optimization method based on deep reinforcement learning, comprising:

[0007] Step S1: construct a typical drone base station downlink scenario and simulate the user's dynamic movement behavior based on the Gauss-Markov model;

[0008] Step S2: Based on the UAV base station downlink scenario, a system modeling analysis is performed to generate a joint optimization model with the goal of minimizing energy consumption;

[0009] Step S3: Use a deep reinforcement learning algorithm to solve the optimal solution of the joint optimization model, integrate the dual competitive deep Q network and the imagination mechanism, and use IM to propagate information across rounds to accelerate network training.

[0010] In some specific embodiments, the step S1 includes:

[0011] Step S11: The UAV and the user mobile node select a direction and speed to move from the current location to a new location;

[0012] Step S12: the user mobile node moves at a constant time interval each time, and calculates a new direction and speed of the user mobile node;

[0013] Step S13: If the drone or user mobile node reaches the simulation boundary, it bounces back from the simulation boundary, the rebound angle is determined by the incident direction, and continues to move along this path, repeating the above steps until the end.

[0014] In some specific embodiments, in step S2, the joint optimization model is determined according to the following formula:

[0015]

[0016] In the formula, C1~C3 represent the flight area of ​​the UAV; C4 represents that the UAV needs to fly to the destination in the last time slot; C5 represents the UAV onboard energy limit; C6 represents the user's minimum transmission data limit; C7 represents the user's signal-to-noise ratio limit; c k [t] represents the connection status between the user and the drone; t represents the time slot; k represents the number of users, represents the amount of data transmitted by the drone to the user, u[t] represents the user's location, represents the amount of data remaining to be transmitted, ω[t] represents the position of the drone, ω end represents the final position of the drone, E[t] represents the remaining energy in the current time slot, and E max Indicates the maximum onboard energy.

[0017] In some specific embodiments, step S3 includes:

[0018] Step S31: determining the data transmission rate between the UAV and the user according to the transmission power of the UAV and the signal-to-noise ratio between the UAV and the ground user in the time slot, and determining the energy consumption of the UAV by using a closed analytical boost power consumption model;

[0019] Step S32: setting the state function, action function and reward function of the drone;

[0020] Step S33: determining the loss of the drone data in the D3QN neural network according to the error between the predicted Q value and the target Q value, and updating the neural network;

[0021] Step S34: Calculate the difference between the drone state-action pairs according to the drone state function and the drone action function through the IM mechanism, and update the Critic and IM networks accordingly.

[0022] In some specific embodiments, in step S3, the state function of the drone is:

[0023]

[0024] Where u[t] represents the user location, represents the amount of data remaining to be transmitted, ω[t] represents the position of the drone, ω end represents the final position of the drone, E[t] represents the remaining energy in the current time slot, and E max Indicates the maximum onboard energy.

[0025] In some specific embodiments, the user's action function is: (+x d ,0,0),(-x d ,0,0),(0,+y d ,0),(0,-y d,0),(0,0,+z d ), (0,0,-z d ) and (0,0,0), representing left, right, forward, backward, up, down, and hover behaviors respectively.

[0026] In some specific embodiments, the reward function includes:

[0027]

[0028] r2[t]=-κ 2a E[t]-κ 2b D sum

[0029]

[0030] In the formula, κ i ,i={1,2a,2b,3,4,5} is the reward parameter, D sum Relative to the total amount of data transferred The normalization function of is determined according to the following formula:

[0031]

[0032] In the formula, α and β are normalization parameters;

[0033] In some specific embodiments, the overall reward and punishment design is determined according to the reward function as follows:

[0034]

[0035] In some specific embodiments, the closed analytical propulsion power consumption model of the UAV is expressed as follows:

[0036]

[0037] In the formula, P0, P i are two constants representing the blade power and induced power in the hovering state, U tip is the tip speed of the rotor blade, v0 and ν are the rotor induced speed and fuselage drag ratio in the hovering state, ρ and s are the air density and rotor disk area, A is the rotor disk area, and V is the speed of the UAV at time slot t.

[0038] To achieve the same invention purpose, the present application also provides a drone-assisted network path optimization system based on deep reinforcement learning, including:

[0039] Behavior simulation module: used to build a typical drone base station downlink scenario and simulate the user's dynamic movement behavior based on the Gauss-Markov model;

[0040] Optimization model generation module: used to perform system modeling analysis according to the UAV base station downlink scenario, and generate a joint optimization model with the goal of minimizing energy consumption;

[0041] Model solving module: used to solve the optimal solution of the joint optimization model through a deep reinforcement learning algorithm, and train the drone to learn the optimal solution through a DQN neural network, a dual competitive deep Q network, and an IM mechanism.

[0042] Beneficial effects of the above technical solution:

[0043] Aiming at the joint optimization problem of transmission efficiency and energy consumption of UAV-assisted ground mobile users, this paper proposes a UAV path-assisted planning algorithm based on deep reinforcement learning. By integrating the imagination mechanism and the dual competitive deep Q network, the advantages of the IM mechanism for cross-round information propagation, the dual Q network and the competitive structure are effectively utilized to efficiently adapt to the dynamically changing electromagnetic environment and user needs. Under the premise of meeting the user service quality, the energy consumption of the UAV is significantly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A flowchart of a method for UAV-assisted network path optimization based on deep reinforcement learning provided by an embodiment of the present invention;

[0046] Figure 2 A model diagram of a ground user serving by a drone provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of a drone network learning curve provided by an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of a remaining energy curve of a drone under different algorithms provided by an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of the remaining energy of a drone using different algorithms under different numbers of users provided by an embodiment of the present invention;

[0050] Figure 6 A schematic diagram of the remaining energy of a drone under different amounts of transmitted data provided by an embodiment of the present invention;

[0051] Figure 7 A schematic diagram of the structure of a drone-assisted network path optimization system based on deep reinforcement learning provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0053] Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0054] Embodiment 1

[0055] An embodiment of the present invention provides a method for optimizing a UAV-assisted network path based on deep reinforcement learning. Figure 1 As shown, including:

[0056] Step S1: Construct a typical drone base station downlink scenario and simulate the user's dynamic movement behavior based on the Gauss-Markov model.

[0057] In a specific embodiment of the present invention, the step S1 comprises:

[0058] Step S11: The UAV and the user mobile node select a direction and speed to move from the current location to a new location;

[0059] Step S12: the user mobile node moves at a constant time interval each time, and calculates a new direction and speed of the user mobile node;

[0060] Step S13: If the drone or user mobile node reaches the simulation boundary, it bounces back from the simulation boundary, the rebound angle is determined by the incident direction, and continues to move along this path, repeating the above steps until the end.

[0061] Step S2: Based on the UAV base station downlink scenario, perform system modeling analysis to generate a joint optimization model with the goal of minimizing energy consumption.

[0062] In a specific embodiment of the present invention, in step S2, the joint optimization model is determined according to the following formula:

[0063]

[0064] In the formula, C1~C3 represent the flight area of ​​the UAV; C4 represents that the UAV needs to fly to the destination in the last time slot; C5 represents the UAV onboard energy limit; C6 represents the user's minimum transmission data limit; C7 represents the user's signal-to-noise ratio limit; c k [t] represents the connection status between the user and the drone; t represents the time slot; k represents the number of users, represents the amount of data transmitted by the drone to the user, u[t] represents the user's location, represents the amount of data remaining to be transmitted, ω[t] represents the position of the drone, ω end represents the final position of the drone, E[t] represents the remaining energy in the current time slot, and E max Indicates the maximum onboard energy.

[0065] In a specific embodiment of the present invention, the amount of data transmitted by the drone to the user is expressed as:

[0066]

[0067] In the formula, c k [t] indicates whether the user is connected to the drone, R k [t] represents the transmission rate between the UAV and the user in time slot t.

[0068] In a specific embodiment of the present invention, the transmission rate between the drone and the user in time slot t is determined according to the following formula:

[0069] R k [t] = Blog2(1+γ k [t])

[0070] In the formula, γ k [t] is the user signal-to-noise ratio, and B is the user bandwidth.

[0071] Step S3: Solve the optimal solution of the joint optimization model through a deep reinforcement learning algorithm, and train the drone to learn the optimal solution through a DQN neural network, a dual competitive deep Q network, and an IM mechanism.

[0072] Step S31: determining the data transmission rate between the UAV and the user according to the transmission power of the UAV and the signal-to-noise ratio between the UAV and the ground user in the time slot, and determining the energy consumption of the UAV by using a closed analytical boost power consumption model;

[0073] Step S32: setting the state function, action function and reward function of the drone;

[0074] Step S33: determining the loss of the drone data in the D3QN neural network according to the error between the predicted Q value and the target Q value, and updating the neural network;

[0075] Step S34: Calculate the difference between the drone state-action pairs according to the drone state function and the drone action function through the IM mechanism, and update the Critic and IM networks accordingly.

[0076] In a specific embodiment of the present invention, the state function of the drone is:

[0077]

[0078] Where u[t] represents the user location, represents the amount of data remaining to be transmitted, ω[t] represents the position of the drone, ω end represents the final position of the drone, E[t] represents the remaining energy in the current time slot, and E max Indicates the maximum onboard energy.

[0079] In a specific embodiment of the present invention, the user's action function is: (+x d ,0,0),(-x d ,0,0),(0,+y d ,0),(0,-y d ,0),(0,0,+z d ), (0,0,-z d ) and (0,0,0), representing left, right, forward, backward, up, down, and hover behaviors respectively.

[0080] In a specific embodiment of the present invention, the reward function includes:

[0081]

[0082] r2[t]=-κ 2a E[t]-κ 2b D sum

[0083]

[0084] In the formula, κ i ,i={1,2a,2b,3,4,5} is the reward parameter, D sum Relative to the total amount of data transferred The normalization function of is determined according to the following formula:

[0085]

[0086] In the formula, α and β are normalization parameters;

[0087] In a specific embodiment of the present invention, the overall reward and punishment design is determined according to the reward function as follows:

[0088]

[0089] In a specific embodiment of the present invention, the closed analytical propulsion power consumption model of the UAV is expressed as follows:

[0090]

[0091] In the formula, P0, P i are two constants representing the blade power and induced power in the hovering state, U tip is the tip speed of the rotor blade, v0 and ν are the rotor induced speed and fuselage drag ratio in the hovering state, ρ and s are the air density and rotor disk area, A is the rotor disk area, and V is the speed of the UAV at time slot t.

[0092] Reference Figure 2 As shown, in a downlink communication scenario, within the limited area R, the total energy of one aircraft is E max The drone acts as a base station to provide services to K ground users, K = {1,…,K}. Assume that the target area R is a [0,X max ]×[0,Y max ] is a rectangular area. E[t] represents the energy consumed by the drone at time slot t, where E[0] = 0, T t It is represented as the total flight time slot of the drone (each time slot t has the same duration δ t ), where t∈T,T={0,1,…,T t}, It is the minimum amount of data that the UAV needs to transmit to the ground user k. Charging stations are set up near the starting point and the end point of the UAV flight. The UAV needs to consume the maximum energy E max Before the flight, complete all ground user communication tasks and fly to the destination end , otherwise the mission is considered failed. This application takes minimizing the energy consumed by the drone to complete the communication mission and fly to the destination as the optimization goal.

[0093] The channel model between the drone base station and the ground user is a probabilistic path loss model. The probability of line of sight between the drone and the ground user k at time t is:

[0094]

[0095] Where a and b are constants related to the propagation environment type, and θ is the elevation angle, which can be expressed as

[0096]

[0097] The LoS loss and NLoS loss between the UAV and the ground user k can be expressed as:

[0098]

[0099]

[0100] Where f is the carrier frequency, c is the speed of light, and η LoS , η NLoS are the average additional losses for LoS and NLoS links, respectively.

[0101] The loss between the UAV and the ground user k can be expressed as:

[0102]

[0103] Given the UAV transmission power P, the signal-to-noise ratio (SNR) between the UAV and the ground user k in time slot t can be expressed as:

[0104]

[0105] In the formula, n0 is the noise power spectral density. In order to ensure the quality of service (QoS) between the UAV and user k, this paper assumes that the SNR of the kth ground in the tth time slot should exceed a certain threshold γ m i n :

[0106] γ k [t]≥γ min

[0107] The closed analytical propulsion power consumption model of the UAV is expressed as follows:

[0108]

[0109] In the formula, P0, P i are two constants representing the blade power and induced power in the hovering state, U tip is the tip speed of the rotor blade, v0 and ν are the rotor induced speed and fuselage drag ratio in the hovering state, ρ and s are the air density and rotor disk area, A is the rotor disk area, and V is the speed of the UAV at time slot t.

[0110] The total energy consumption of the drone is E[T t ]for:

[0111]

[0112] The loss of the DQN neural network is calculated based on the error between the predicted Q value and the target Q value, and then the gradient descent method is used to update the parameters θ of the neural network. The predicted Q value is described as follows:

[0113]

[0114] Where γ is the discount rate. The target Q value is described as follows:

[0115]

[0116] The loss function is described as follows:

[0117]

[0118] In the dual Q network, two independent networks are used, and the update of the target Q value does not depend on the predicted Q value of the same network. The target Q value is redefined as follows:

[0119]

[0120] In the competitive neural network, the action value function is decomposed into the state value function and the advantage function, and its Q value is redefined as follows:

[0121]

[0122] In the formula, φ, are the parameters for the advantage function and the state-value function respectively, and A is the number of all possible actions.

[0123] The IM mechanism includes five steps: 1) First, randomly sample from the experience replay pool to obtain another set of experience samples (s', a'); 2) Then input (s, a) and (s', a') into the feature encoder to obtain the feature (q, q'); 3) Use the similarity function to calculate the feature (q, q') to obtain the similarity vector v; 4) Pass the similarity vector v to the inference network to obtain the difference between the vectors d(s, a, s', a'); 5) Finally, use Q θ (s,a)+d and Q θ The mean squared error (MSE) of (s',a') is used as the loss function to update the Critic network and IM.

[0124] The simulation was built using the pytorch library in Python. D3QN contains two hidden layers, each of which contains 256 neural units, and Relu is selected as the activation function, the learning rate is 0.0001, and the discount rate is 0.99. The hidden layers of the multi-head encoder and the difference reasoning network of the IM structure have 256 and 64 neural units respectively. The similarity function uses cosine similarity and the activation function is ReLU.

[0125] Figure 3The learning curve of the drone network is shown. The average reward value is used as a metric because it is a common method for evaluating reinforcement learning algorithms and can effectively reflect the overall performance of the algorithm. It can be seen that the agent trained by the IM-D3QN algorithm with the IM mechanism reached a platform area with good performance after about 1100 rounds, while the number of rounds required for the agent trained using the ordinary D3QN algorithm to reach the same reward value was about 1400 rounds, and the training efficiency was improved by about 21.4%. After the IM-D3QN with the IM mechanism continuously trains the agent, more information can be transferred to other states across rounds, and the algorithm can learn sample information more effectively and then have better performance.

[0126] In order to better demonstrate the performance of the algorithm, the proposed algorithm is compared with the Greedy algorithm and the KMeans algorithm. For the Greedy algorithm, the drone selects the nearest ground user who has not completed the communication task as the next target. When the drone completes the tasks of all users, it flies directly to the end point. For the KMeans algorithm, all ground users are clustered into several clusters. The drone first flies to the center of the nearest cluster until the tasks of all users in the cluster are completed, and then the drone selects the next nearest cluster. Similarly, when the tasks of all users are completed, it flies directly to the end point.

[0127] Figure 4 The curves of the remaining energy of the drone after completing the task under the three different algorithms as the number of iterations change are shown in detail. It is observed that in the early stage of IM-D3QN training, the drone attempts to complete the communication task in the flight area until the energy is exhausted, and its remaining energy is lower than that of the other two algorithms. However, as the number of training times increases, the performance of IM-D3QN gradually emerges, not only surpassing the greedy algorithm and KMeans algorithm, but also after about 1100 iterations, the remaining energy tends to be stable and close to the optimal state.

[0128] Figure 5 The comparison of the remaining energy of drones after completing tasks under different numbers of users using different algorithms is shown. It can be seen that the drones trained by the IM-D3QN algorithm have the largest remaining energy after completing tasks. Traditional greedy algorithms have difficulty coping with situations with a large number of users. D3QN and DQN based on deep reinforcement learning can better find suitable paths to reduce energy loss and complete communication tasks.

[0129] Figure 6The four algorithms are shown to complete the tasks of drones under different data transmission amounts. It can be observed that the agent trained by the IM-D3QN algorithm has the largest overall residual energy under different data transmission conditions. The drones controlled by the greedy algorithm and the KMeans algorithm need to stay for a longer time when they reach the relevant user or cluster center to complete the data transmission. When the amount of data required to be transmitted is large, the energy consumed increases significantly.

[0130] The above test results show that the solution method proposed in this example has good performance, especially in terms of energy efficiency using IM-D3QN. The effect of using the method proposed in this application is significantly better than the traditional DQN algorithm and the other two traditional optimization algorithms.

[0131] Embodiment 2

[0132] An embodiment of the present invention provides a UAV-assisted network path optimization system based on deep reinforcement learning, referring to Figure 7 As shown, including:

[0133] Behavior simulation module 10: used to build a typical drone base station downlink scenario and simulate the user's dynamic movement behavior based on the Gauss-Markov model;

[0134] The optimization model generation module 20 is used to perform system modeling analysis according to the UAV base station downlink scenario, and generate a joint optimization model with the goal of minimizing energy consumption;

[0135] Model solving module 30: used to solve the optimal solution of the joint optimization model through a deep reinforcement learning algorithm, and train the drone to learn the optimal solution through a DQN neural network and a dual competitive deep Q network.

[0136] In a specific embodiment of the present invention, the behavior simulation module 10 is used to perform the following steps:

[0137] Step S11: The UAV and the user mobile node select a direction and speed to move from the current location to a new location;

[0138] Step S12: the user mobile node moves at a constant time interval each time, and calculates a new direction and speed of the user mobile node;

[0139] Step S13: If the drone or user mobile node reaches the simulation boundary, it bounces back from the simulation boundary, the rebound angle is determined by the incident direction, and continues to move along this path, repeating the above steps until the end.

[0140] In a specific embodiment of the present invention, in step S2, the joint optimization model is determined according to the following formula:

[0141]

[0142] In the formula, C1~C3 represent the flight area of ​​the UAV; C4 represents that the UAV needs to fly to the destination in the last time slot; C5 represents the UAV onboard energy limit; C6 represents the user's minimum transmission data limit; C7 represents the user's signal-to-noise ratio limit; c k [t] represents the connection status between the user and the drone; t represents the time slot; k represents the number of users, represents the amount of data transmitted by the drone to the user, u[t] represents the user's location, represents the amount of data remaining to be transmitted, ω[t] represents the position of the drone, ω end represents the final position of the drone, E[t] represents the remaining energy in the current time slot, and E max Indicates the maximum onboard energy.

[0143] In a specific embodiment of the present invention, the model solving module 30 is used to perform the following steps:

[0144] Step S31: determining the data transmission rate between the UAV and the user according to the transmission power of the UAV and the signal-to-noise ratio between the UAV and the ground user in the time slot, and determining the energy consumption of the UAV by using a closed analytical boost power consumption model;

[0145] Step S32: setting the state function, action function and reward function of the drone;

[0146] Step S33: determining the loss of the drone data in the D3QN neural network according to the error between the predicted Q value and the target Q value, and updating the neural network;

[0147] Step S34: Calculate the difference between the drone state-action pairs according to the drone state function and the drone action function through the IM mechanism, and update the Critic and IM networks accordingly.

[0148] In a specific embodiment of the present invention, in the model solving module 30, the state function of the drone is:

[0149]

[0150] Where u[t] represents the user location, represents the amount of data remaining to be transmitted, ω[t] represents the position of the drone, ω end represents the final position of the drone, E[t] represents the remaining energy in the current time slot, and E max Indicates the maximum onboard energy.

[0151] In a specific embodiment of the present invention, the user's action function is: (+x d ,0,0),(-xd ,0,0),(0,+y d ,0),(0,-y d ,0),(0,0,+z d ), (0,0,-z d ) and (0,0,0), representing left, right, forward, backward, up, down, and hover behaviors respectively.

[0152] In a specific embodiment of the present invention, the reward function includes:

[0153]

[0154] r2[t]=-κ 2a E[t]-κ 2b D sum

[0155]

[0156] In the formula, κ i ,i={1,2a,2b,3,4,5} is the reward parameter, D sum Relative to the total amount of data transferred The normalization function of is determined according to the following formula:

[0157]

[0158] In the formula, α and β are normalization parameters;

[0159] In a specific embodiment of the present invention, the overall reward and punishment design is determined according to the reward function as follows:

[0160]

[0161] In a specific embodiment of the present invention, the amount of data transmitted by the drone to the user is expressed as:

[0162]

[0163] In the formula, c k [t] indicates whether the user is connected to the drone, R k [t] represents the transmission rate between the UAV and the user in time slot t.

[0164] In a specific embodiment of the present invention, the transmission rate between the drone and the user in time slot t is determined according to the following formula:

[0165] R k [t] = Blog2(1+γ k [t])

[0166] In the formula, γk [t] is the user signal-to-noise ratio, and B is the user bandwidth.

[0167] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

[0168] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referenced to each other. The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing terminal device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps of the functions specified in one or more boxes. Although the preferred embodiments of the embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention. Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0169] The method and device provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

[0170] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "one specific embodiment" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A UAV-assisted network path optimization method based on deep reinforcement learning, characterized in that: include: Step S1: construct a typical drone base station downlink scenario and simulate the user's dynamic movement behavior based on the Gauss-Markov model; Step S2: Based on the UAV base station downlink scenario, a system modeling analysis is performed to generate a joint optimization model with the goal of minimizing energy consumption; Step S3: Use a deep reinforcement learning algorithm to solve the optimal solution of the joint optimization model, integrate the dual competitive deep Q network and the imagination mechanism, and use IM to propagate information across rounds to accelerate network training.

2. The method for UAV-assisted network path optimization based on deep reinforcement learning according to claim 1 is characterized in that: The step S1 comprises: Step S11: The UAV and the user mobile node select a direction and speed to move from the current location to a new location; Step S12: the user mobile node moves at a constant time interval each time, and calculates a new direction and speed of the user mobile node; Step S13: If the UAV or user mobile node reaches the simulation boundary, it bounces back from the simulation boundary, the rebound angle is determined by the incident direction, and continues to move along this path, repeating the above steps until the end.

3. The method for UAV-assisted network path optimization based on deep reinforcement learning according to claim 1 is characterized in that: In step S2, the joint optimization model is determined according to the following formula: C4:ω[T t ]=ω end In the formula, C1~C3 represent the flight area of ​​the UAV; C4 represents that the UAV needs to fly to the destination in the last time slot; C5 represents the UAV onboard energy limit; C6 represents the user's minimum transmission data limit; C7 represents the user's signal-to-noise ratio limit; c k [t] represents the connection status between the user and the drone; t represents the time slot; k represents the number of users, represents the amount of data transmitted by the drone to the user, u[t] represents the user's location, represents the amount of data remaining to be transmitted, ω[t] represents the position of the drone, ω end represents the final position of the drone, E[t] represents the remaining energy in the current time slot, and E max Indicates the maximum onboard energy.

4. The method for UAV-assisted network path optimization based on deep reinforcement learning according to claim 1 is characterized in that: Step S3 includes: Step S31: determining the data transmission rate between the UAV and the user according to the transmission power of the UAV and the signal-to-noise ratio between the UAV and the ground user in the time slot, and determining the energy consumption of the UAV by using a closed analytical boost power consumption model; Step S32: setting the state function, action function and reward function of the drone; Step S33: determining the loss of the drone data in the D3QN neural network according to the error between the predicted Q value and the target Q value, and updating the neural network; Step S34: Calculate the difference between the drone state-action pairs according to the drone state function and the drone action function through the IM mechanism, and update the Critic and IM networks accordingly.

5. The method for UAV-assisted network path optimization based on deep reinforcement learning according to claim 4 is characterized in that: The state function of the drone is: Where u[t] represents the user location, represents the amount of data remaining to be transmitted, ω[t] represents the position of the drone, ω end represents the final position of the drone, E[t] represents the remaining energy in the current time slot, and E max Indicates the maximum onboard energy.

6. The method for UAV-assisted network path optimization based on deep reinforcement learning according to claim 5 is characterized in that: The user's action function is: (+x d ,0,0),(-x d ,0,0),(0,+y d ,0),(0,-y d ,0),(0,0,+z d ), (0,0,-z d ) and (0,0,0), representing left, right, forward, backward, up, down, and hover behaviors respectively.

7. The method for UAV-assisted network path optimization based on deep reinforcement learning according to claim 6 is characterized in that: The reward function includes: r2[t]=-κ 2a E[t]-k 2b D sum In the formula, κ i ,i={1,2a,2b,3,4,5} is the reward parameter, D sum Relative to the total amount of data transferred The normalization function of is determined according to the following formula: Where α and β are normalization parameters.

8. The method for UAV-assisted network path optimization based on deep reinforcement learning according to claim 7 is characterized in that: According to the reward function, the overall reward and punishment design is determined as follows:

9. The method for UAV-assisted network path optimization based on deep reinforcement learning according to claim 4 is characterized in that: The closed analytical propulsion power consumption model of the UAV is expressed as follows: In the formula, P0, P i are two constants representing the blade power and induced power in the hovering state, U tip is the tip speed of the rotor blade, v0 and ν are the rotor induced speed and fuselage drag ratio in the hovering state, ρ and s are the air density and rotor disk area, A is the rotor disk area, and V is the speed of the UAV at time slot t.

10. A UAV-assisted network path optimization system based on deep reinforcement learning, characterized in that: include: Behavior simulation module: used to build a typical drone base station downlink scenario and simulate the user's dynamic movement behavior based on the Gauss-Markov model; Optimization model generation module: used to perform system modeling analysis according to the UAV base station downlink scenario, and generate a joint optimization model with the goal of minimizing energy consumption; Model solving module: used to solve the optimal solution of the joint optimization model through a deep reinforcement learning algorithm, and train the drone to learn the optimal solution through a DQN neural network, a dual competitive deep Q network, and an IM mechanism.

Citation Information

Patent Citations

  • Model construction method and device, task allocation method and device, equipment and medium

    CN113032904A

  • Computing task unloading method based on deep reinforcement learning in electric power internet of things

    CN114065963A

  • Unmanned aerial vehicle trajectory planning method based on deep reinforcement learning in interference environment

    CN116774723A

  • Real-time spectrum resource optimization method and system based on deep reinforcement learning

    CN117768902A

  • Unmanned aerial vehicle auxiliary relay method for emergency scene

    CN117792481A

Cited By

  • Unmanned aerial vehicle group spectrum resource sharing method and device under centralized architecture

    CN119743762A

  • A method and device for sharing spectrum resources of drone swarm under centralized architecture

    CN119743762B