User scheduling and drone altitude control method in wearable device charging network

Abstract

The present invention discloses a method for user scheduling and drone altitude control in a wearable device charging network. The present invention collaboratively considers the charging requirements of smart wearable devices and the limited battery capacity of drones, models the network throughput and energy consumption in a real-time charging network for drone-assisted wearable devices, and describes the optimization problem with the goal of maximizing network throughput. The present invention transforms the optimization problem into a constrained Markov decision process, and uses a proximal policy optimization (PPO)-CMDP algorithm method based on Lagrangian primal dual strategy optimization to solve the optimal smart wearable device time slot scheduling decision and drone altitude adjustment strategy, ensuring that the drone's battery energy is within a limited range and meets the device's charging requirements. The present invention provides a new method for user scheduling and drone altitude control in a smart wearable device charging network.

Description

Technical Field

[0001] The present invention relates to a method for user device time slot scheduling and drone altitude control in a drone-assisted intelligent wearable device charging network, and in particular to a method for jointly optimizing drone altitude control and user device time slot scheduling under the constraints of drone coverage and limited battery energy, and capable of charging intelligent wearable devices within the range allowed by the drone battery capacity to meet the charging needs of the devices. Background Art

[0002] Smart wearable devices can monitor various health data of patients in real time, thereby reducing contact with patients. However, the battery energy of smart wearable devices is limited. The emergence of wireless power transfer (WPT) technology makes wireless charging of smart wearable devices possible. Unlike the traditional fixed network infrastructure, drones have the advantages of easy deployment and flexibility, and drones can be used as carriers of WPT. However, the existing research considers that the served users perform tasks in sequence within the drone coverage area and only serve one user in a period of time; and it is generally assumed that the drone operates at a fixed horizontal altitude, and the impact of the drone's altitude adjustment on the number of covered devices is not considered. How to comprehensively consider the drone altitude change and user equipment time slot scheduling, and effectively solve the problem of changes in user equipment allowed to be served due to changes in drone altitude and allocating services to user equipment in different time slots to meet user needs needs further exploration by researchers. Summary of the invention

[0003] The main purpose of the present invention is to address the shortcomings of existing research, aiming at maximizing network throughput, and propose a method for jointly optimizing the altitude control of drones and the time slot scheduling of user devices under the constraints of the coverage range and limited battery energy of drones. First, according to the limited coverage range of drones and the service demand factors of wearable devices, the present invention establishes an optimization model that maximizes the network throughput of wearable devices and ensures that the battery energy of drones is non-negative; secondly, in order to solve the time slot scheduling of wearable devices and the altitude control of drones in energy-constrained drone-assisted networks, the problem is converted into a constrained Markov decision process (CMDP); finally, the present invention proposes a proximal policy optimization (PPO)-CMDP algorithm based on Lagrangian primal dual strategy optimization, which provides a new and effective solution for drones to decide the moving distance to adjust the altitude and the wearable devices to select services within the current coverage range.

[0004] The technical solution adopted by the present invention is: a method for user scheduling and drone altitude control in a wearable device charging network, comprising the following steps:

[0005] Step (1): Construct a real-time charging model for drone-assisted wearable devices, including network structure, channel model, and energy consumption model.

[0006] Step (2): Based on the real-time charging model of drone-assisted wearable devices in step (1), the optimization problem P1 is constructed with the goal of maximizing network throughput.

[0007] Step (3): Convert the wearable device time slot scheduling and drone altitude decision problem in step (2) into a constrained Markov decision problem.

[0008] Step (4): Solve the constrained Markov decision problem in step (3).

[0009] The beneficial effects of the present invention include:

[0010] The present invention provides a method for scheduling user device time slots and controlling the altitude of drones in a drone-assisted smart wearable device charging network based on WPT. The charging requirements of smart wearable devices and the limited battery capacity of drones are collaboratively considered. The present invention describes the optimization problem with the goal of maximizing network throughput, transforms the optimization problem into a constrained Markov decision process, and uses a proximal policy optimization (PPO)-CMDP algorithm method based on Lagrangian primal dual strategy optimization to solve the optimal smart wearable device time slot scheduling decision and drone altitude adjustment strategy. Experimental results demonstrate the high efficiency of the algorithm proposed in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A real-time charging model for drone-assisted smart wearable devices constructed for the present invention;

[0012] Figure 2 A comparison chart of the expected total reward between the PPO-CMDP algorithm disclosed in the present invention and the Trust Region Policy Optimization (TRPO) algorithm and the Actor-Critic (AC) algorithm;

[0013] Figure 3 is the changing trend of Lagrange multiplier with the number of iterations;

[0014] Figure 4 The relationship between service completion time and the number of wearable devices;

[0015] Figure 5 is the relationship between total energy consumption and the number of wearable devices;

[0016] Figure 6 is the relationship between the total energy consumption and the number of time slots;

[0017] Figure 7 is the relationship between the service fairness rate and the number of time slots. DETAILED DESCRIPTION

[0018] like Figure 1 As shown in Figure 1, before the mission begins, the drone is fully charged at the charging station. When the smart wearable device sends a charging request, the drone flies directly above the center of the user cluster to provide service.

[0019] The present invention provides a method for scheduling user equipment time slots and controlling the height of a drone in a drone-assisted intelligent wearable device charging network, the method comprising:

[0020] Step (1): Construct a real-time charging model for drone-assisted wearable devices, including network structure, channel model, and energy consumption model;

[0021] Step (2): Based on the real-time charging model of drone-assisted wearable devices in step (1), the optimization problem is described with the goal of maximizing network throughput;

[0022] Step (3): Convert the wearable device time slot scheduling and drone altitude decision problem in step (2) into a constrained Markov decision process (CMDP);

[0023] Step (4): Solve the optimization problem P2 in step (3).

[0024] The present invention constructs a downlink drone-assisted system for wireless charging of smart wearable device networks. In the target user cluster area A, there are (I represents the number of wearable devices) smart wearable devices and a drone. Considering the limited coverage of the drone, it can provide services to the smart wearable devices in the hotspot area by adjusting its altitude. The energy demand of the smart wearable devices in the user cluster is denoted by g_{i}. We assume that most of the transmission power of the drone is concentrated directly below the drone with an aperture angle of α.

[0025] Since charging smart wearable devices is a delay-sensitive task, for example, in medical emergencies, it is necessary to monitor various physiological data of patients in real time. Charging smart wearable devices can ensure their long-term effective health monitoring. The time slot-frame structure is adopted to divide the time into a set of frames. Energy transfer should be at L max Completed in L max represents the maximum service completion time. Each frame consists of K time slots, and the set of time slots is represented by Indicates that the duration of each time slot is δ. As the altitude of the drone changes, the set of wearable devices currently covered by the drone is The quantity is C k,l≤I. In addition, the smart wearable devices scheduled in this time slot are defined as a service device group. The optional candidate service device group is recorded as The maximum number is The number and set of the currently served device group z in time slot k of frame l are denoted by C k,z,l and express.

[0026] Assume that smart wearable devices are randomly distributed in hot spots, and their location coordinates are (x i ,y i ,0). The charging station is located outside the drone coverage cluster, (x p ,y p ,0) is considered to be the location of the charging pile. In addition, the three-dimensional position coordinates of the drone are in is the height of the drone in the user cluster. Then, the radius of the disk area covered by the drone is Therefore, drones and smart wearable devices The three-dimensional distance between The spectrum occupied by the drone backhaul link and the drone-user device link does not interfere with each other, and the communication link between the drone and the user device is easily affected by the environment. The path loss from the drone to the smart wearable device is expressed as PL i,u ,Right now: Among them, f c is the carrier frequency assigned to the smart wearable device, c is the speed of light, is the additional loss when the line-of-sight (LoS) link and the non-line-of-sight (NLoS) link have different values. In addition, since the drone may serve multiple smart wearable devices in the same time slot, there is inevitable interference between the devices, which is defined as: Where P t represents the transmission power, G j,u represents the channel gain between the remaining users in the current service user group except device i and the drone. Then, the signal interference signal-to-noise ratio (SINR) from the drone to the smart wearable device is expressed as: Among them is Channel gain, σ 2 is the variance of additive white Gaussian noise (AWGN). According to Shannon’s theorem, the total throughput of the wearable device network is defined as:

[0027]

[0028] Where B is the system bandwidth.

[0029] The drone consists of an initial energy of E initIn each time slot k in the l frame, the drone moves in the vertical direction at a certain speed η. A distance, h max Represents the maximum distance to move and then interact with the environment while hovering at the new position. The calculation of the propulsion power of the drone flight is expressed as follows:

[0030]

[0031] Among them, u tip is the tip speed of the rotor, v0 is the average induced speed of the rotor, d0 is the fuselage drag ratio, ρ is the air density, s is the rotor solidity, and J is the rotor disc area. s They represent blade section power and induced power respectively.

[0032] From the above formula, it can be seen that the UAV flight propulsion power consists of three parts, the first term represents the blade profile, the second term represents the induced power, and the third term represents the additional energy power. Then the UAV flight energy consumption is Among them l f is the flight time for the drone to adjust its altitude, expressed as

[0033] The hovering energy consumption of the UAV is the blade profile power P0 and the induced power P in the hovering state (ie, η = 0). s So it can be expressed as:

[0034]

[0035] Where θ is the drag coefficient, V is the blade angular velocity, and R is the rotor radius. The variable W is the weight of the drone, W = mg, where m is the mass of the drone and g is the acceleration of gravity. The variable ι is the incremental correction factor. Assuming that the drone is charged in parallel with the wearable device while hovering, the hovering time of the drone is l h Equal to the charging service time l c , the total energy consumption of drone charging and hovering can be expressed as:

[0036]

[0037] Where P c is the communication power of the drone when charging the smart wearable device. In summary, the battery energy of the drone in time frame l is The energy collected by smart wearable devices is Where μ is the WPT energy conversion coefficient.

[0038] The decision variable p z,k,l Defined as the scheduling indicator of wearable devices, where p z,k,l=1 indicates wearable device group is selected by the drone for service in time slot k of frame l. Another decision variable represents the distance that the drone adjusts its altitude in time slot k, and the positive and negative signs indicate the direction of movement. The change in the altitude position of the drone will affect the service quality of the wearable device, thereby affecting the throughput of the wearable network. Since the battery capacity of the drone is limited, the optimization problem with the goal of maximizing throughput is expressed as:

[0039]

[0040] st

[0041] C1:

[0042] C2:

[0043] C3:

[0044] C4:

[0045] C5:

[0046] C6:

[0047] Constraints C1 and C2 restrict the decision variables p z,k,l and Constraint C3 states that no more than one set of wearable devices can be scheduled in a time period. Constraint C4 ensures that the charging needs of all wearable devices are within L max Constraint C5 limits the altitude of the drone to the allowed range [H min ,H max ] Constraint C6 ensures that the battery energy change of the drone is lower than the threshold E thre In the considered problem, the altitude position of the UAV is closely connected with the scheduling of wearable devices, and the problem has energy constraints on the actions.

[0048] The wearable device time slot scheduling and drone altitude decision problem is transformed into a constrained Markov decision process (CMDP), where the immediate reward is defined as the total throughput of the wearable device network, that is: The penalty constraint is defined as the residual energy difference of the UAV in adjacent time periods, i.e. Considering the maximization of the long-term discounted network throughput, a discount factor β is introduced, so the CMDP problem can be stated as follows:

[0049]

[0050] st

[0051] represents the expected value of the Markov strategy β l It refers to the discount factor of cumulative returns.

[0052] When solving the optimization problem P2 in step (3), the present invention proposes a nested PPO-CMDP algorithm to solve the problem by constructing a Lagrangian-based reward and punishment function to relax the constrained optimization problem. In this algorithm, an actor-criticism scheme is used to improve the strategy, update the parameters of a given Lagrangian on a faster time scale, and update the parameters of the Lagrangian on a slower time scale.

[0053] In view of the above analysis, the state of the system of the present invention consists of three parts: the current altitude of the drone, the remaining battery capacity of the drone, and the unmet energy demand of the smart wearable device, as described below:

[0054] The position of the drone directly affects the number of smart wearable devices served per frame, and thus greatly affects the maximization objective in P1. When the drone is providing services to smart wearable devices, we only care about the drone in frame l<L max The altitude position of time slot k on the UAV is within the movement boundary within the altitude limit, so

[0055] The remaining energy of the drone’s battery is another key factor. When the drone’s battery is fully charged, it is expected that there will be little impact on the drone’s mobility. However, when the drone’s battery capacity approaches the energy threshold E thre This factor is expected to have a significant impact on the mobility of the UAV when it is in use. We will learn from experience to ensure that the battery energy variation of the UAV when serving smart wearable devices is within the limits, i.e.

[0056] The remaining energy demand of the smart wearable device is also part of the environmental state. When the unserved energy that will charge the smart wearable device in time frame l is in and Is the drone in strategy π(S l |A l ) is the energy delivered to the smart wearable device in time frame l under . Since the energy demand of the smart wearable device affects the remaining battery capacity of the drone, this information as an important state will greatly increase the nonlinearity of the learning task.

[0057] By collecting the above three components, the state S l Can be defined as: Note that the next state depends only on the current state S. l and the current policy. Therefore, the state S l The transfer process complies with the requirements of MDP.

[0058] The action of the drone consists of two parts. One part is the distance moved at the height during the time period l. in The other is the user time slot allocation of the drone in time period l, p l ={p 1,l ,…,p K,l}, where p k,l ∈{1,…,z,…Z k,l}, p k,l = z represents the candidate group z of user equipment to be served in time slot k of time period l. Therefore, the action space is defined as

[0059] The specific solution process of step (4) is as follows: an agent is deployed on the drone to interact with the wearable device in a series of actions. The reward is observed and the optimal altitude adjustment decision and wearable device scheduling strategy are stably learned. The present invention uses a method based on the Proximal Policy Optimization (PPO) algorithm to solve this CMDP problem. This algorithm uses the primal-dual optimization technique. The objective function of the PPO algorithm is as follows:

[0060]

[0061] in is the generalized advantage estimate (GAE), which is the key to balancing the effects of bias and variance. CLIP (ζ) represents the objective function of the PPO algorithm used to update the policy network parameters, represents the expected cumulative return estimate, A l represents the action space, S l represents the state space, π ζ represents the new strategy, π ζold represents the old policy, represents the broad advantage estimate, clip represents the clipping function, ∈ represents the hyperparameter used to adjust the clipping, represents the Lagrangian penalty reward function, β represents the discount factor, and λ represents a hyperparameter for balancing bias and variance. Represents the state value function of the value network, S l+k+1 It indicates the next state at a certain moment in the future as k increases.

[0062] The second is to learn the state-value function by minimizing the mean squared error of the policy.

[0063]

[0064] in represents the loss function used to update the value network parameters, The expected multi-step return estimate, β k represents the discount factor in the cumulative return, It represents the Lagrangian penalty reward function at a certain moment in the future as k increases, Defined as the Lagrangian penalty reward function, represents the reward function, represents the penalty function, and γ represents the Lagrange multiplier. At the same time, the penalty critic has a penalty value function parameter ω q It is calculated in a similar way to the state value function.

[0065]

[0066] Finally, the Lagrangian parameters are improved by penalizing the actor by performing stochastic gradient ascent,

[0067] γ l represents the Lagrange multiplier at frame l, ε represents the learning rate of the Lagrange multiplier, represents the state-value function that penalizes the critic.

[0068] This paper proposes a nested PPO-CMDP algorithm to solve this problem by constructing a Lagrangian-based reward and punishment function to relax the constrained optimization problem. In this algorithm, an actor-critic scheme is used to improve the strategy, update the parameters of a given Lagrangian on a faster time scale, and update the parameters of the Lagrangian on a slower time scale. The pseudo code of the PPO-CMDP algorithm is shown in Table 1.

[0069]

[0070]

[0071] like Figure 2 As shown in Figure 2, as training progresses, the agent eventually converges under all three methods and obtains a higher expected total reward. The PPO-CMDP scheme converges faster than the other two schemes.

[0072] like Figure 3 The Lagrange multiplier changes with the number of iterations. As the training progresses, the Lagrange multipliers of the three methods converge. The Lagrange multiplier of the TRPO algorithm is slightly higher than that of the PPO-CMDP algorithm, followed by the AC algorithm.

[0073] like Figure 4 The relationship between the service completion time and the number of wearable devices is shown. In all three cases, the total service time increases with the increase in the number of wearable devices. However, the solution of PPO-CMDP has the shortest total service time due to the increase in the number of wearable devices. The randomness of the number of devices currently covered by the drone and the randomness of selecting the devices to be served, as well as the need to charge different devices to meet these devices, leads to the increase in service time.

[0074] like Figure 5 The relationship between the total energy consumption and the number of wearable devices is shown. Using the PPO-CMDP based algorithm, the total energy consumption is smaller than the baseline algorithm. In addition, the total energy consumption increases with the number of wearable devices. This is because as the number of wearable devices increases, more charging energy is required and more charging time is required to obtain more energy to ensure the sustainable use of wearable devices.

[0075] like Figure 6 The relationship between total energy consumption and the number of time slots is shown. When the number of time slots is small, the charging task needs to be completed in a shorter specified time. Then the drone serves multiple user devices in one time slot, and the communication interference between user devices leads to a large total energy consumption. As the number of time slots increases, the total energy consumption will decrease. However, because the drone has a longer time to serve the smart wearable device, the total energy consumption will increase.

[0076] like Figure 7 The relationship between the service fairness rate and the number of time slots is shown. The service fairness rate is defined as the percentage of service collection energy of all wearable devices to the total energy demand. The experimental results show that as the number of time slots increases, the service fairness rates of the three methods will increase. The proposed PPO-CMDP algorithm has a high service fairness rate close to 99%, TRPO has a second service fairness rate close to 92%, and the AC algorithm performs poorly with a service fairness rate of 85%.

[0077] The above description is a specific embodiment of the present invention and the technical principles used. If the changes made according to the concept of the present invention do not exceed the spirit covered by the description and drawings, they should still fall within the scope of protection of the present invention.

Claims

1. User scheduling and drone altitude control methods in wearable device charging networks, It is characterized in that The following steps are involved: Step (1): Construct a real-time charging model for drone-assisted wearable devices, including network structure, channel model, and energy consumption model; The network structure: In the target user cluster area A, there are Smart wearable devices and a drone, I represents the number of wearable devices, and the energy demand of smart wearable devices in the user cluster is expressed in g i It means that most of the UAV’s transmission power is concentrated directly below the UAV, with an aperture angle of α; The channel model adopts a time slot-frame structure, dividing time into a set of frames. Energy transfer in L max Completed in L max represents the maximum service completion time. Each frame consists of K time slots, and the set of time slots is represented by Indicates that the duration of each time slot is δ. As the altitude of the drone changes, the set of wearable devices currently covered by the drone is The quantity is C k,l ≤I, the smart wearable devices scheduled in this time slot are defined as a service device group, and the optional candidate service device groups are recorded as The maximum number is The number and set of the currently served device group z in time slot k of frame l are denoted by C k,z,l and express; Smart wearable devices are randomly distributed in hot spots, and their location coordinates are (x i ,y i ,0), the charging pile is located outside the drone coverage cluster, (x p ,y p ,0) is considered to be the location of the charging pile, and the three-dimensional position coordinates of the drone are in is the height of the drone in the user cluster, then the radius of the disk area covered by the drone is Drones and smart wearable devices The three-dimensional distance between The path loss from the drone to the smart wearable device is expressed as PL i,u ,Right now: Among them, f c is the carrier frequency assigned to the smart wearable device, c is the speed of light, It is the additional loss when the line-of-sight link and the non-line-of-sight link have different values; since the drone may serve multiple smart wearable devices in the same time slot, there is inevitable interference between the devices, which is defined as: Where P t represents the transmission power, G j,u represents the channel gain between the remaining users in the current service user group except device i and the drone; the signal interference signal-to-noise ratio from the drone to the smart wearable device is expressed as: in is the channel gain, σ 2 is the variance of additive white Gaussian noise; the total throughput of the wearable device network is defined as: Where B is the system bandwidth; The energy consumption model: The drone has an initial energy of E init The battery is powered, and in each time slot k in the l frame, the drone moves in the vertical direction at a certain speed η A certain distance, and then interact with the environment while hovering at the new position, the flight energy consumption of the drone is Among them l f is the flight time for the drone to adjust its altitude, expressed as P f is the propulsion power of the UAV flight; The hovering energy consumption of the drone is expressed as: Where θ is the drag coefficient, V is the blade angular velocity, R is the rotor radius, the variable W is the weight of the drone, W = mg, where m is the mass of the drone, g is the acceleration of gravity, the variable ι is the incremental correction factor, ρ is the air density, s is the rotor solidity, and J is the rotor disc area; the total energy consumption of the drone charging and hovering is expressed as: Where P c is the communication power when the drone charges the smart wearable device, l c It is the drone charging service time; In summary, the battery energy of the drone in time frame l is The energy collected by smart wearable devices is Where μ is the WPT energy conversion coefficient; Step (2): Based on the real-time charging model of drone-assisted wearable devices in step (1), an optimization problem P1 is constructed with the goal of maximizing network throughput; The decision variable p z,k,l Defined as the scheduling indicator of wearable devices, where p z,k,l =1 indicates wearable device group The other decision variable is represents the distance that the drone adjusts its altitude in time slot k, and the positive and negative signs represent the direction of movement; the optimization problem P1 is: st Constraints C1 and C2 restrict the decision variables p z,k,l and The range of,constraint C3 indicates that more than one set of wearable devices cannot be arranged in a time period,g i represents the energy demand of smart wearable devices in the user cluster, and constraint C4 ensures that the charging demand of all wearable devices is within L max The constraint C5 limits the height position of the UAV to the allowed range [H min ,H max ], constraint C6 ensures that the battery energy change of the drone is lower than the threshold E thre ; Step (3): Convert the wearable device time slot scheduling and drone altitude decision problem in step (2) into a constrained Markov decision problem: represents the expectation of the Markov strategy, Indicates the battery capacity of the drone, E thre represents the energy threshold of the drone, β l It refers to the discount factor of the cumulative return; Step (4): Solve the constrained Markov decision problem in step (3).

2. The method for user scheduling and drone altitude control in a wearable device charging network according to claim 1, Features: In step (4), a method based on an approximate strategy optimization algorithm is used to solve the problem The objective function of the algorithm is as follows: in is the generalized advantage estimate, which is the key to balancing the effects of bias and variance; L CLIP (ζ) represents the objective function of the PPO algorithm used to update the policy network parameters, represents the expected cumulative return estimate, A l represents the action space, S l represents the state space, π ζ represents the new strategy, π ζold represents the old policy, represents the broad advantage estimate, clip represents the clipping function, ∈ represents the hyperparameter used to adjust the clipping, represents the Lagrangian penalty reward function, β represents the discount factor, and λ represents a hyperparameter for balancing bias and variance. Represents the state value function of the value network, S l+k+1 Indicates the next state at a certain moment in the future as k increases; Secondly, the state value function is learned by minimizing the mean square error of the strategy; in represents the loss function used to update the value network parameters, β k represents the discount factor in the cumulative return, It represents the Lagrangian penalty reward function at a certain moment in the future as k increases, Defined as the Lagrangian penalty reward function, represents the reward function, represents the penalty function, γ represents the Lagrange multiplier; At the same time, the penalty critic has a penalty function parameter ω q Optimize Finally, the Lagrangian parameters are improved by penalizing the actor by performing stochastic gradient ascent, γ l represents the Lagrange multiplier at frame l, ε represents the learning rate of the Lagrange multiplier, represents the state-value function that penalizes the critic.

Images