A method for collecting data and transmitting energy from unmanned aerial vehicles

Abstract

The present application discloses a method for collecting data and transmitting energy from a drone, and relates to the field of mobile computing technology. The method includes: constructing a data collection model, an energy collection model, and an energy consumption model based on a drone system; constructing a multi-objective drone flight optimization model based on the data collection model, the energy collection model, and the energy consumption model, with the goal of minimizing the average information age of all smart devices, minimizing the total energy consumption of the drone, and maximizing the average energy collected by all smart devices; solving the multi-objective drone flight optimization model by alternating between an evolutionary algorithm and a reinforcement learning algorithm, so as to obtain a drone flight behavior decision, which is one of flying, hovering and collecting data, and hovering and transmitting energy. The present application can obtain a solution with better quality that simultaneously considers the minimum information age, the minimum drone energy consumption, and the maximum energy collection.

Description

Technical Field

[0001] The present application relates to the field of mobile computing technology, and in particular to a method for collecting data and transmitting energy from an unmanned aerial vehicle. Background Art

[0002] With the rapid development of 5G technology and the Internet of Things, the number of smart devices has increased exponentially, and the computing tasks generated have gradually become more complex and diverse. On this basis, centralized services such as cloud computing have emerged. Mobile Edge Computing (MEC) is proposed to alleviate the peak computing pressure on the cloud. Some computing tasks of smart devices can be directly offloaded to the closer edge server for calculation, while also reducing processing latency. Due to the flexibility and mobility of Unmanned Aerial Vehicles (UAVs), they can provide reliable connections in wireless networks and are therefore considered to be cost-effective and energy-saving solutions. However, due to the energy limitations of smart devices and drones, this service model is unsustainable.

[0003] To solve the above problems, current research is increasingly focusing on energy harvesting technologies, such as wireless power transmission. For drones that require higher power, base stations can be used for directional laser charging; for smart devices scattered everywhere, wireless radio frequency transmission is a more efficient way to charge. At the same time, to ensure the timeliness of data collection, researchers introduced the concept of information age to measure the time difference between when data is generated and when it is received. Inspired by the growing attention to information age and energy harvesting research, most current research on drone-assisted mobile edge computing systems focuses on a single issue, limiting the breadth of research, such as only reducing information age and only improving energy harvesting efficiency. However, reducing information age and improving energy harvesting efficiency conflict in the system. Reducing information age requires drones to interact frequently with more IoT devices, while improving energy harvesting efficiency requires drones to remain stationary for longer to ensure sufficient charging time. Previous research has largely ignored this conflict. Summary of the invention

[0004] The purpose of this application is to provide a method for collecting data and transmitting energy from a drone, which can obtain a better quality solution that simultaneously considers the minimum information age, the minimum drone energy consumption, and the maximum amount of collected energy.

[0005] To achieve the above objectives, the present application provides the following solution: The present application provides a method for collecting data and transmitting energy using a drone, comprising the following steps.

[0006] Construct an unmanned aerial vehicle system; the unmanned aerial vehicle system includes a unmanned aerial vehicle, a base station, an aerial platform and multiple intelligent devices; the unmanned aerial vehicle is used to collect data collected by the intelligent devices and unload the data to the aerial platform, and transmit energy to the intelligent devices; the base station is used to transmit energy to the unmanned aerial vehicle.

[0007] Based on the drone system, a data collection model, an energy collection model and an energy consumption model are constructed; the data collection model is constructed based on the information age corresponding to the data collected by the smart device.

[0008] Based on the data collection model, the energy collection model and the energy consumption model, a multi-objective UAV flight optimization model is constructed with the goals of minimizing the average information age of all smart devices, minimizing the total energy consumption of UAVs and maximizing the average energy collected by all smart devices.

[0009] The multi-objective UAV flight optimization model is solved by alternating between an evolutionary algorithm and a reinforcement learning algorithm to obtain a flight behavior decision of the UAV; the flight behavior decision is one of flying, hovering and collecting data, and hovering and transmitting energy.

[0010] Preferably, the multi-objective UAV flight optimization model is solved by alternating between an evolutionary algorithm and a reinforcement learning algorithm to obtain the flight behavior decision of the UAV, including:

[0011] For the multi-objective UAV flight optimization model, an initial population is determined based on Markov decision making; the initial population includes multiple agents, each of which includes a weight vector, a strategy network, and a value network; the weight vector is used to characterize the importance of multiple optimization objectives, and the multiple optimization objectives include the average information age of all smart devices, the total energy consumption of the UAV, and the average energy collected by all smart devices;

[0012] In the current iteration, if the number of iterations does not reach the preset number and is within the preset range of reinforcement learning execution rounds, multi-objective proximal strategy optimization is performed on multiple agents in the initial population to obtain a new population;

[0013] If the number of iterations does not reach the preset number of times and is within the preset evolution operation execution round range, the evolution operation is performed on the initial population and the elite population to obtain a new population; wherein the rounds within the preset reinforcement learning execution round range and the rounds within the preset evolution operation execution round range are alternately arranged in order of size;

[0014] Evaluate a target vector for each agent in the new population; the target vector is determined based on a vector value reward corresponding to the agent in each time slot;

[0015] The elite population is updated based on the new population, and then the new population is merged with the buffer population to obtain an updated buffer population, and the new population is assigned to the initial population to obtain an updated initial population;

[0016] If the number of iterations does not meet the preset multiple condition, then enter the next iteration;

[0017] If the number of iterations meets the preset multiple condition, then based on the target vector of each agent in the new population, a population update operation is performed on the updated buffer population to delete some agents to obtain an optimized population, the optimized population is assigned to the initial population to obtain an updated initial population, and then the next iteration is entered;

[0018] If the number of iterations reaches a preset number, the elite population is marked as the output population;

[0019] According to the output population, flight behavior decisions of the UAV in multiple time slots are determined.

[0020] Preferably, performing multi-objective proximal strategy optimization on multiple agents in the initial population to obtain a new population includes:

[0021] For any agent in the initial population, a policy network is used to perform multiple environmental interactions to obtain a transfer sample corresponding to each environmental interaction; the transfer sample includes a current state, a next state, a vector value reward and an agent action; wherein the current state includes the state of each smart device and the state of the drone; the state of each smart device includes the distance between the smart device and the drone, the information age of the smart device and the remaining energy of the smart device; the state of the drone includes the coordinates of the drone, the number of tasks collected in the drone unloading queue and the number of environmental interactions; the agent action is a flight behavior decision of the drone; the vector value reward is determined based on multiple optimization objectives;

[0022] According to the transfer samples corresponding to multiple environmental interactions, the advantage function is calculated using the following formula :

[0023] ;

[0024] Based on the advantage function and the transfer samples corresponding to multiple environmental interactions, the following formula is used to update the strategy network Parameters:

[0025] ;

[0026] Based on the transfer samples corresponding to multiple environmental interactions, the value network is updated using the following formula Parameters:

[0027] ;

[0028] in, is the discount factor, T is the number of time slots, and an environment interaction is performed in each time slot. is a hyperparameter, is the weight vector, For the The vector value reward for sub-environment interaction, For the The state of the sub-environment interaction, For the The state of the sub-environment interaction, l is the difference in the number of steps used to calculate the advantage function; Training loss function for the policy network, For expectations, for Update the previous policy network, For the The state of the sub-environment interaction, For the Agent actions for sub-environment interactions, Represents the clipping function, Restricted to Within the scope, is a hyperparameter; Training loss function for the value network, For the Vector-valued rewards for sub-environment interactions;

[0029] A new population is determined based on the initial population, the updated parameters of the policy network, and the updated parameters of the value network.

[0030] Preferably, in the drone system, the drone uses time division multiple access technology to collect data from the smart device, and the drone uses frequency division multiple access technology to transmit energy to the smart device; based on the drone system, a data collection model, an energy collection model and an energy consumption model are constructed, including:

[0031] Determine, according to the relative position of the drone and the smart device in the drone system, a channel transmission rate from the smart device to the drone and a channel transmission rate from the drone to the high-altitude platform;

[0032] Based on the channel transmission rate from the smart device to the drone, determine the delay of the smart device sending data to the drone; based on the channel transmission rate from the drone to the high-altitude platform, determine the delay of the drone unloading data to the high-altitude platform;

[0033] Determine the data collection model based on the information age defined by the smart device and the time delay of the drone unloading data to the high-altitude platform;

[0034] Constructing a drone energy collection sub-model according to the drone system, and determining a smart device energy collection sub-model according to the relative position of the drone and the smart device; the drone energy collection sub-model and the smart device energy collection sub-model constitute an energy collection model;

[0035] Based on the drone system, combined with the delay of the smart device sending data to the drone, the delay of the drone unloading data to the high-altitude platform and the flight status of the drone, the drone flight energy consumption, drone data collection energy consumption, drone data unloading energy consumption and drone transmission energy consumption to the smart device are determined, and an energy consumption model is constructed.

[0036] Preferably, the multi-objective UAV flight optimization model includes an objective function and constraints; the objective function is:

[0037] ;

[0038] in, is the average information age of smart devices, , K is the total number of smart devices, T is the number of time slots, For the time slot Smart devices Information age; is the total energy consumption of the UAV, , For the time slot Energy consumption of drones; The average energy collected by smart devices, , For smart devices In time slot Energy collected;

[0039] The constraints are:

[0040] ;

[0041] ;

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] ;

[0047] in, For drones in time slots Flight behavior decisions, For drones in time slots The flight direction, is the set of time slots, , are the horizontal and vertical coordinates of the drone’s coordinates, , are the coordinate limits, For drones and smart devices In time slot The horizontal distance within The data collection range of the drone, For the energy collection range, For smart device collection, The power received by the drone from the base station. is the duration of the time slot.

[0048] Preferably, the relative position of the drone and the smart device in the drone system includes an elevation angle and a distance between the drone and the smart device;

[0049] Among them, the calculation formula for the channel transmission rate from the smart device to the drone is:

[0050] ;

[0051] ; ;

[0052] in, For the time slot Smart devices k Channel transmission rate to the drone, For the time slot Smart devices to the bandwidth between drones, For smart devices The data transmission power, For the time slot Drones and smart devices The channel power gain between For smart devices The channel noise power of For the time slot Drones and smart devices BetweenLoS probability, based on the time slot Drones and smart devices The elevation angle between them is determined; is the additional attenuation coefficient of the non-line-of-sight link, For the time slot Drones and smart devices Between NLoS Probability, is the channel power gain at a reference distance of 1m, For the time slot Drones and smart devices The distance between is the path loss index;

[0053] The calculation formula for the channel transmission rate from the UAV to the high-altitude platform is:

[0054] ;

[0055] in, For the time slot The channel transmission rate from the drone to the high-altitude platform, The bandwidth between the drone and the high-altitude platform, is the unloaded power of the UAV, For the time slot The channel power gain between the UAV and the high-altitude platform, is the channel noise power between the UAV and the high-altitude platform.

[0056] Preferably, the data collection model includes:

[0057] When the high-altitude platform receives the smart device When the data is , the corresponding information age is expressed as the following function model: ;

[0058] When the aerial platform does not receive the smart device When the data is , the corresponding information age is expressed as the following function model: ;

[0059] in, The latency of offloading data from drones to high-altitude platforms; For the time slot Smart devices The information age, is the duration of the time slot, For the time slot +1 Smart Device The information age.

[0060] Preferably, the flight status of the drone includes the flight speed of the drone (t); The calculation model of the UAV flight energy consumption is:

[0061] ;

[0062] ;

[0063] in, For the time slot The energy consumption of drone flight, is the duration of the time slot, For the time slot The propulsion power consumption of the drone, and are the rotor profile power and rotor blade tip speed of the UAV in the hovering state, and are the induced power and average induced speed of the rotor of the UAV in the hovering state, is the fuselage drag ratio, is the air density, is the rotor stability, is the rotor disc area;

[0064] The calculation model of the energy consumption of the drone data collection is:

[0065] ;

[0066] in, For the time slot Energy consumption of drone data collection; It is a collection of smart devices within the scope of data collection. For smart devices k The latency of sending data to the drone;

[0067] The calculation model of the energy consumption of drone data offloading is:

[0068] ;

[0069] in, For the time slot of drone data offloading energy consumption, The latency of offloading data from drones to high-altitude platforms;

[0070] The calculation model of energy consumption of transmitting from the drone to the smart device is:

[0071] ;

[0072] in, For the time slot The drone transmits energy consumption to smart devices. The power received by the drone from the base station; is a binary variable. If the condition is true, then ;otherwise ; For drones in time slots Hover and transfer energy, For drones in time slots Hover and collect data;

[0073] The energy consumption model is:

[0074] ;

[0075] in, For the time slot Energy consumption of drones.

[0076] According to the specific embodiments provided by the present application, the present application has the following technical effects: the present application provides a method for collecting and transmitting data from a drone, based on a drone system, constructs a data collection model, an energy collection model and an energy consumption model, and then constructs a multi-objective drone flight optimization model with the goal of minimizing the average information age of all smart devices, minimizing the total energy consumption of the drone and maximizing the average energy collected by all smart devices. So far, the present application has taken into account the three optimized solutions for drone data collection and energy transmission, namely, the minimum information age, the minimum drone energy consumption and the maximum energy collected. Then, the evolutionary algorithm and the reinforcement learning algorithm are used in an alternating manner to solve the problem and obtain the flight behavior decision of the drone. The above-mentioned alternating solution method can make the final solution result have better convergence and diversity, and can provide decision makers with richer and better quality alternative solutions, which can minimize the system information age and drone energy consumption while maximizing the amount of energy collected by smart devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0078] Figure 1 A flowchart of a method for collecting and transmitting energy from a drone is provided in accordance with an embodiment of the present application.

[0079] Figure 2A schematic diagram of the structure of a drone system provided in one embodiment of the present application.

[0080] Figure 3 A schematic diagram of an algorithm flow for alternating processing of an evolutionary algorithm and a reinforcement learning algorithm provided in another embodiment of the present application.

[0081] Reference numerals: drone-1, base station-2, high-altitude platform-3, smart device-4, flight area-5. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0083] This application solves the problem of data collection and energy transmission, and can simultaneously minimize the information age and energy consumption of drones, improve energy collection efficiency, and obtain high-quality non-dominated solution sets, providing decision support for decision makers in the context of multi-party games. Simulation experiments and data analysis show that compared with the proximal policy optimization (PPO) algorithm, this application has significant advantages in algorithm performance and can obtain a better non-dominated solution set.

[0084] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0085] In an exemplary embodiment, Figure 1 As shown, a method for collecting drone data and transmitting energy is provided, which is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In an embodiment of the present application, the method includes the following steps 101 to 104.

[0086] Step 101, constructing a drone system; the drone system includes a drone, a base station, a high-altitude platform and multiple smart devices; the drone is used to collect data collected by the smart device and unload it to the high-altitude platform, and transmit energy to the smart device; the base station is used to transmit energy to the drone.

[0087] In an application example, Figure 2As shown, multiple smart devices 4 are randomly distributed in the flight area 5, and the smart devices 4 are used to monitor environmental information, such as humidity and temperature. The flight area 5 can be a rectangular area. The smart device 4 cannot access the ground cellular communication system, so it is necessary to use the drone 1 to collect data and unload the data to the high-altitude platform 3 for remote processing. The drone 1 provides wireless charging for the smart device 4 through wireless power transfer (WPT) technology. The drone 1 can be a rotary-wing drone. Since the rotary-wing drone can hover at a low altitude near the smart device 4, a rotary-wing drone is used in an application example to provide computing services, data collection and energy transmission for the smart device 4. The drone 1 will collect the data generated by the smart device 4 or wirelessly charge it in its coverage area along a predetermined trajectory. The base station 2 wirelessly charges the drone 1 through a laser beam to maintain its flight mission. The high-altitude platform 3 floats in the stratosphere at a fixed height in a quasi-static manner. The high-altitude platform 3 has abundant computing resources and can serve as a supplementary solution for the drone's task offloading.

[0088] For the above-mentioned drone system, relevant settings need to be made before subsequent data processing: The drone system of this application adopts a discrete time system, and the duration of each time slot is . Assume that the entire mission collection process of the drone lasts time slots, then record is the set of time slots in a single flight mission of the UAV. Assume that the smart device The generated data size follows a pruned Gaussian distribution, expressed as , and are the mean and standard deviation, and are the minimum and maximum data sizes.

[0089] Use Cartesian coordinate system to monitor drones, high-altitude platforms, base stations and smart devices The positions of are modeled and expressed as , , and In the time slot , the flight behavior decision of the drone is one of flying, hovering and collecting data, hovering and transmitting energy. For drones in time slots The flight behavior decision , the drone moves in the horizontal direction Level flight; if , then the drone is in the time slot Hover and collect data; , then the drone is in the time slot Hover and transfer energy. is a binary variable. If the condition is true, then ;otherwise Then the drone is in the time slot The coordinates are updated to:

[0090] .

[0091] in, Indicates that the drone is in the time slot The flight speed of the drone is constant. The side length of the flight area in this application is and , is a rectangular area.

[0092] Step 102: construct a data collection model, an energy collection model and an energy consumption model based on the drone system; the data collection model is constructed based on the information age corresponding to the data collected by the smart device.

[0093] In an application example, in the drone system, the drone uses time division multiple access technology to collect data from the smart device, and the drone uses frequency division multiple access technology to transmit energy to the smart device. Based on this, step 102 includes the following steps (21) to (25).

[0094] (21) Determine the channel transmission rate from the smart device to the drone and the channel transmission rate from the drone to the high-altitude platform based on the relative position of the drone and the smart device in the drone system; the relative position of the drone and the smart device in the drone system includes the elevation angle and distance between the drone and the smart device.

[0095] Specifically, it is assumed that the drone is equipped with two antennas: one for receiving data from or transmitting energy to smart devices, and the other for communicating with the high-altitude platform. Due to the proximity of smart devices, simultaneous data transmission to the drone may cause mutual interference. Therefore, this application adopts time division multiple access technology for data collection. On the other hand, the drone adopts frequency division multiple access to transmit energy to smart devices to ensure that the smart devices can receive sufficient energy from the drone. In an actual mobile edge computing environment, communications in the air and on the ground may be affected by obstacles and other factors. Therefore, a probabilistic path loss model that includes line-of-sight (LoS) and non-line-of-sight (NLoS) conditions is considered. In the time slot , drones and smart devices LoS probability between for:

[0096] .

[0097] in, and are two constants that depend on the carrier frequency and the environment, For drones and smart devices The elevation angle between For drones and smart devices Based on the above calculation formula, we can know the NLoS probability Based on the above probability calculation, drones and smart devices In time slot The calculation formula of the channel power gain between is:

[0098] .

[0099] in, is the additional attenuation coefficient of the NLoS link (i.e., non-line-of-sight link), is the channel power gain at a reference distance of 1m is the path loss index. Based on this, smart devices k The calculation formula for the channel transmission rate to the drone is:

[0100] .

[0101] in, For the time slot Smart devices k Channel transmission rate to the drone, For the time slot Smart devices to the bandwidth between drones, For smart devices The data transmission power, For smart devices The channel noise power, For the time slot Drones and smart devices Between NLoS Probability.

[0102] There are almost no obstacles between the drone and the high-altitude platform, so the transmission follows the free space path loss model. The channel gain can be expressed as ,in is the distance between the UAV and the high-altitude platform. , the calculation formula of the channel transmission rate from the UAV to the high-altitude platform is:

[0103] .

[0104] in, For the time slot The channel transmission rate from the drone to the high-altitude platform, The bandwidth between the drone and the high-altitude platform, is the unloaded power of the UAV, For the time slot The channel power gain between the UAV and the high-altitude platform, is the channel noise power between the UAV and the high-altitude platform.

[0105] (22) Based on the channel transmission rate from the smart device to the UAV, determine the delay for the smart device to send data to the UAV; based on the channel transmission rate from the UAV to the high-altitude platform, determine the delay for the UAV to unload data to the high-altitude platform.

[0106] Specifically, For smart devices The delay of sending data to the drone. After receiving the data, the drone will unload it to the high-altitude platform for remote processing. The corresponding delay of the drone unloading data to the high-altitude platform is .

[0107] (23) Determine a data collection model based on the information age defined by the smart device and the time delay of the drone unloading data to the high-altitude platform.

[0108] Specifically, the information age indicator is used to measure the timeliness of collected data, which is defined as the age of smart devices. The time from when the latest data was generated to the current moment. Smart devices The information age is: .in, Indicates that drones are connected from smart devices The time when the data was collected.

[0109] To ensure the largest possible channel transmission rate, it is assumed that the drone only collects data from smart devices within a specific elevation range. The set of these smart devices is represented as:

[0110] .

[0111] in, For smart devices The horizontal distance from the drone, Indicates the data collection range of the drone, For smart devices The remaining energy, The energy threshold for each smart device to transmit data. The remaining energy of the smart device must be greater than or equal to this value to transmit data. collect The data of the device, the number of tasks to be unloaded from the UAV queue in the next time slot is expressed as In the time slot , smart devices The remaining energy is updated as:

[0112] .

[0113] Based on the above analysis, the data collection model includes: when the high-altitude platform receives the smart device When the data is , the corresponding information age is expressed as the following function model: When the aerial platform does not receive the smart device When the data is , the corresponding information age is expressed as the following function model: .in, For the time slot +1 Smart Device The information age.

[0114] (24) Constructing a UAV energy collection sub-model based on the UAV system, and determining a smart device energy collection sub-model based on the relative position of the UAV and the smart device; the UAV energy collection sub-model and the smart device energy collection sub-model constitute an energy collection model.

[0115] Specifically, in the time slot , drones use frequency division multiple access technology to wirelessly charge smart devices, is the set of smart devices within the energy collection range, expressed as: .in, is the energy collection range. This application adopts the linear energy collection model, so the smart device In time slot Energy harvesting power The calculation formula is:

[0116] .

[0117] in, is the wireless transmission power of the drone. , smart devices The remaining energy after charging is: .

[0118] in, and The energy conversion efficiency and maximum battery capacity of smart devices. In time slot The energy collected is .

[0119] Due to the high power consumption and limited battery capacity of drones in wireless power transmission, in order to ensure that the drone can maintain normal flight status during the entire mission, this application uses the base station as a laser charging system to wirelessly charge the drone. The received power of the received energy collected by the drone from the base station is:

[0120] .

[0121] in, , and The conversion efficiency of power to laser, the laser transmission efficiency and the laser to power conversion efficiency. It is the transmission power used by the base station for wireless power transmission.

[0122] (25) Based on the drone system, combined with the delay of the smart device sending data to the drone, the delay of the drone unloading data to the high-altitude platform and the flight status of the drone, determine the drone flight energy consumption, drone data collection energy consumption, drone data unloading energy consumption and drone transmission energy consumption to the smart device, and construct an energy consumption model. The flight status of the drone includes the drone flight speed (t).

[0123] When the rotor drone is moving at speed ( t ) in flight, its propulsion power consumption is defined as:

[0124] .

[0125] in, It consists of blade profile, induced power and parasite power. and are the rotor profile power and rotor blade tip speed of the UAV in the hovering state. In the induced power, and are the induced power and average induced speed of the rotor of the UAV in the hovering state. In the parasitic power, is the fuselage drag ratio, is the air density, is the rotor stability, is the rotor disc area. The calculation model of the UAV flight energy consumption is: ;in, For the time slot The energy consumption of drone flight, For the time slot Propulsion power consumption of the drone.

[0126] The energy consumption of wireless information transfer (WIT) by drones includes data collection energy consumption and unloading energy consumption. The calculation model of drone data collection energy consumption is:

[0127] .

[0128] in, For the time slot Energy consumption of drone data collection; It is a collection of smart devices within the scope of data collection. For smart devices k The latency of sending data to the drone.

[0129] The calculation model of the energy consumption of drone data offloading is:

[0130] .

[0131] in, For the time slot of drone data offloading energy consumption, Reduce the latency of offloading data from drones to high-altitude platforms.

[0132] The calculation model of energy consumption of transmitting from the drone to the smart device is:

[0133] .

[0134] in, For the time slot The drone transmits energy consumption to smart devices. The power that the drone receives energy from the base station.

[0135] The energy consumption model is:

[0136] .

[0137] in, For the time slot Energy consumption of drones.

[0138] Step 103, based on the data collection model, the energy collection model and the energy consumption model, a multi-objective UAV flight optimization model is constructed with the goal of minimizing the average information age of all smart devices, minimizing the total energy consumption of UAVs and maximizing the average energy collected by all smart devices. The multi-objective UAV flight optimization model includes an objective function and constraints. The objective function is:

[0139] .

[0140] in, is the average information age of smart devices, , K is the total number of smart devices, T is the number of time slots, For the time slot Smart devices Information age; For drones The total energy consumption of each time slot is , For the time slot Energy consumption of drones; For smart devices The average energy collected after time slots, , For smart devices In time slot Collected energy.

[0141] The constraints are:

[0142] C1: .

[0143] C2: .

[0144] C3: .

[0145] C4: .

[0146] C5: .

[0147] C6: .

[0148] C7: .

[0149] in, For drones in time slots Flight behavior decisions, For drones in time slots The flight direction, , are the horizontal and vertical coordinates of the drone’s coordinates, , are the coordinate limits, For drones and smart devices In time slot The horizontal distance within The data collection range of the drone, For the energy collection range, The power received by the drone from the base station. is the duration of the time slot.

[0150] Constraints C1 and C2 specify the decision space of the drone. Constraints C3 and C4 limit the flight area of ​​the drone. Constraints C5 and C6 define the scope of data collection and energy collection, respectively. Constraint C7 indicates that the energy provided by the base station to the drone should be sufficient to meet the energy consumption requirements of the drone throughout the flight mission.

[0151] Step 104, using an evolutionary algorithm and a reinforcement learning algorithm to alternately process the multi-objective UAV flight optimization model to obtain a flight behavior decision of the UAV; the flight behavior decision is one of flying, hovering and collecting data, and hovering and transmitting energy.

[0152] In an application example, this application uses a drone data collection and energy transmission method based on evolutionary reinforcement learning to minimize information age and drone energy consumption, maximize the amount of energy collected, and obtain high-quality non-dominated solutions to provide decision support for decision makers in a multi-party game context. Figure 3 , step 104 includes the following steps (41) to (49).

[0153] (41) For the multi-objective UAV flight optimization model, based on Markov decision making, the initial population is determined The initial population includes multiple agents, each of which contains a weight vector, a strategy network and a value network. i Agent 〉. The weight vector is used to characterize the importance of multiple optimization objectives, including the average information age of all smart devices, the total energy consumption of drones, and the average energy collected by all smart devices. i Weight vector , , and They correspond to the three objectives in the objective function respectively.

[0154] In the actual application algorithm, set the elite population and buffer populations .generate uniformly distributed weight vector ;initialization Policy Network , each policy network is used to make decisions based on the observed environment state; initialization Value Network , each value network is used to estimate the value of the observed environment state. Set the iteration variable .

[0155] (42) In the current iteration, if the number of iterations does not reach the preset number , and is within the preset reinforcement learning execution round range, then a multi-objective proximal strategy optimization is performed on multiple agents in the initial population to obtain a new population .

[0156] In an application instance, set up a new population , for the initial population Any agent , using the policy network Perform multiple environmental interactions to obtain transfer samples corresponding to each environmental interaction ); the transfer sample includes the current state , next state , vector value reward and agent actions ; Wherein, the current state includes the state of each smart device and the state of the drone, that is, ; The status of each smart device includes the distance between the smart device and the drone , the information age of smart devices and the remaining energy of smart devices ,Right now The state of the drone includes the coordinates of the drone ( ), the number of tasks collected in the drone offloading queue and the number of interactions with the environment ,Right now The agent action is the flight behavior decision of the UAV, that is, The vector value reward is determined based on a plurality of optimization objectives; ,in With three optimization objectives Corresponding.

[0157] Trajectory data can be obtained based on the transfer samples corresponding to multiple environmental interactions , the advantage function is calculated using the following formula :

[0158] .

[0159] Based on the advantage function and the transfer samples corresponding to multiple environmental interactions, the following formula is used to update the strategy network Parameters:

[0160] .

[0161] Based on the transfer samples corresponding to multiple environmental interactions, the value network is updated using the following formula Parameters:

[0162] .

[0163] in, is the discount factor, T is the number of time slots, and an environment interaction is performed in each time slot. l To calculate the step difference of the advantage function, is a hyperparameter, which is an additional hyperparameter introduced to balance the variance and bias of the advantage function estimate; when When , the advantage function only calculates the current difference, at which point the variance is lower but the bias is higher; when When , the advantage function calculates the difference produced at each time slot, which results in a higher variance and lower bias. is the weight vector, For the The vector value reward for sub-environment interaction, For the The state of the sub-environment interaction, For the The state of the sub-environment interaction; Training loss function for the policy network, For expectations, for Update the previous policy network, is the state of the t-th environmental interaction, is the agent action in the t-th environment interaction, Represents the clipping function, Restricted to Within the scope, As a hyperparameter, this approach can ensure that the update range of the policy network is limited; Training loss function for the value network, For the Vector-valued rewards for sub-environment interactions.

[0164] A new population is determined based on the initial population, the updated parameters of the policy network, and the updated parameters of the value network.

[0165] (43) If the number of iterations does not reach the preset number and is within the preset evolution operation execution round range, the evolution operation is performed on the initial population and the elite population to obtain a new population. ; Among them, the rounds within the preset reinforcement learning execution round range and the rounds within the preset evolution operation execution round range are arranged alternately in order of size.

[0166] In an application example, Each agent in ,from Find the Best matching agent , where index .remember is the number of parameters in the policy network, The first parameters, is the crossover probability, is the mutation probability.

[0167] The specific evolution operations are as follows: for two strategies and No. Parameters, in the crossover operation, randomly generate random numbers between (0,1) Random (0,1), if Random (0,1) , then set No. Parameters ; otherwise set No. Parameters In the mutation operation, a random number between (0,1) is generated Random (0,1), if Random (0,1) , then for the crossover and No. Parameters Each of them performs severe Gaussian perturbation, reset parameters or weak Gaussian perturbation with probability 5%, 50% or 90% respectively. Finally, set the new population , and output the population. In the evolutionary operation of this step, the number of new populations will increase.

[0168] In another practical application, taking the total number of rounds as 20 as an example, the preset range of reinforcement learning execution rounds is 1 to 9, 11 to 19; the preset range of evolution operation execution rounds is 10 and 20. The above two ranges are only examples to illustrate the alternating settings of the two, and relevant technicians can adjust the two round ranges as needed.

[0169] (44) Evaluate the target vector of each agent in the new population ; The target vector is determined based on the vector value reward corresponding to the agent in each time slot.

[0170] In an application example, the policy network of the agent is used Interact with the environment, in each time slot, Input to Output action , the drone takes action After that, it will receive a vector value reward from the environment , and transfer to the next state .So After the time slot ends, you will get Vector-valued rewards ( , this Add the vectors to get the target vector ,in" ” is vector addition.

[0171] (45) The elite population is updated based on the new population. Specifically, The policy network of each agent in ,from Deleted The strategy dominated by There is no strategy governing , then Add to Then the new population is merged with the buffer population to obtain an updated buffer population, and the new population is assigned to the initial population to obtain an updated initial population, that is, setting , .

[0172] (46) If the number of iterations does not meet the preset multiple condition, the next iteration is entered. The preset multiple condition is set in the present application based on the situation that the buffer population contains a large number of agents, where the number of agents may be several times the initial number, and the spatial range of the buffer population is limited. Before the number of agents reaches the limit, some agents need to be deleted. In an application example, a ten-fold value can be set, that is, when the remainder of the number of iterations divided by ten is zero, it indicates that the preset multiple condition is met, otherwise it does not meet the multiple condition.

[0173] (47) If the number of iterations meets the preset multiple condition, then based on the target vector of each agent in the new population, a population update operation is performed on the updated buffer population to delete some agents to obtain an optimized population, and the optimized population is assigned to the initial population to obtain an updated initial population, and then the next iteration is entered, that is, setting .

[0174] In an application example, first generate uniformly distributed weight vector, denoted by As the reference point, is the upper limit of the energy consumption of the drone in any time slot. The reference point is set to the vertex position of the optimization target, which can ensure that the results are in a reasonable relative position and facilitate the division of the set. Diversity, using multiple cache collections to divide Each agent in the buffer population is divided evenly into cache collections, each cache collection can only store Agent. For the cache collection, .

[0175] against Each agent in , calculate the vector , set the index ,Will Save to Then calculate For each cache collection ,according to The distances of each agent in are sorted in descending order, retaining the first agents and delete the other redundant agents.

[0176] set up , set the optimization population to an empty set, for each weight vector , set the index ,replace Agents in The weight vector is , and add to the optimized population. Finally, the optimized population is output as the population .

[0177] (48) If the number of iterations reaches a preset number, the elite population is marked as the output population.

[0178] (49) Determine the flight behavior decisions of the UAV in multiple time slots based on the output population.

[0179] To verify the effectiveness of this application, this application also provides a specific application example, and configures the experimental parameters and test scenarios as follows: the flight mission of the drone lasts for 2 minutes, and the duration of each time slot is 1 second, so the number of time slots is . The horizontal length of the rectangular area and the vertical side length Both are set to 200 m. Smart devices are evenly distributed in the area, and the maximum tolerable information age value of each smart device is , the data size of smart devices has increased from =0.1 MB to =0.5 MB. The altitude of the high-altitude platform is 20 km. At the initial time slot, the drone takes off from the center of the rectangular area, and its flight speed m / s. The reference point is set to Table 1 shows the setting values ​​of other parameters.

[0180] Table 1

[0181]

[0182] In terms of algorithm parameters, the number of agents The strategy network and value network of each agent adopt a 3-layer fully connected neural network, with 500 neurons in each layer and a learning rate of 3×10 -4 , the learning rate of the value network is 6×10 -4 The total number of iterations is 200, and the algorithm performs an evolution operation every 10 generations. The number of cache sets , its scale .

[0183] The number of smart devices and the drone's flight altitude As two system parameters are varied to generate test scenarios, consider , six test scenarios can be produced, as shown in Table 2.

[0184] Table 2

[0185]

[0186] The following two multi-objective algorithm performance indicators are used to compare the present application scheme with the original proximal policy optimization (PPO): Inverted Generational Distance (IGD) and Generational Distance (GD). IGD can simultaneously measure the diversity and convergence of the non-dominated solution set obtained by an algorithm, while GD can measure how close the non-dominated solution set obtained by the algorithm is to the real non-dominated solution set. The smaller the value of both IGD and GD indicators, the better the multi-objective performance of the algorithm.

[0187] Table 3 shows the IGD and GD values ​​of PPO and the solution of this application in six test scenarios. It can be found that the solution of this application can obtain smaller IGD and GD values ​​in all six test scenarios, which shows that the Pareto front obtained by the solution of this application has better convergence and diversity, and can provide decision makers with richer and better quality alternative solutions, which can minimize the system information age and drone energy consumption while maximizing the amount of energy collected by smart devices. Therefore, the experimental results in Table 3 verify the effectiveness and feasibility of the solution of this application.

[0188] Table 3

[0189]

[0190] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0191] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0192] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for collecting data and transmitting energy from a drone, characterized in that: The drone data collection and energy transmission method comprises: Constructing a drone system; the drone system includes a drone, a base station, a high-altitude platform and a plurality of intelligent devices; the drone is used to collect data collected by the intelligent device and unload it to the high-altitude platform, and transmit energy to the intelligent device; the base station is used to transmit energy to the drone; Based on the UAV system, a data collection model, an energy collection model and an energy consumption model are constructed; the data collection model is constructed based on the information age corresponding to the data collected by the smart device; Based on the data collection model, the energy collection model and the energy consumption model, a multi-objective UAV flight optimization model is constructed with the goals of minimizing the average information age of all smart devices, minimizing the total energy consumption of UAVs and maximizing the average energy collected by all smart devices; The multi-objective UAV flight optimization model is solved by alternating between an evolutionary algorithm and a reinforcement learning algorithm to obtain a flight behavior decision of the UAV; the flight behavior decision is one of flying, hovering and collecting data, and hovering and transmitting energy; The multi-objective UAV flight optimization model is solved by alternating between an evolutionary algorithm and a reinforcement learning algorithm to obtain the flight behavior decision of the UAV, including: For the multi-objective UAV flight optimization model, an initial population is determined based on Markov decision making; the initial population includes multiple agents, each of which includes a weight vector, a strategy network, and a value network; the weight vector is used to characterize the importance of multiple optimization objectives, and the multiple optimization objectives include the average information age of all smart devices, the total energy consumption of the UAV, and the average energy collected by all smart devices; In the current iteration, if the number of iterations does not reach the preset number and is within the preset range of reinforcement learning execution rounds, multi-objective proximal strategy optimization is performed on multiple agents in the initial population to obtain a new population; If the number of iterations does not reach the preset number of times and is within the preset evolution operation execution round range, the evolution operation is performed on the initial population and the elite population to obtain a new population; wherein the rounds within the preset reinforcement learning execution round range and the rounds within the preset evolution operation execution round range are alternately arranged in order of size; Evaluate a target vector for each agent in the new population; the target vector is determined based on a vector value reward corresponding to the agent in each time slot; The elite population is updated based on the new population, and then the new population is merged with the buffer population to obtain an updated buffer population, and the new population is assigned to the initial population to obtain an updated initial population; If the number of iterations does not meet the preset multiple condition, then enter the next iteration; If the number of iterations meets the preset multiple condition, then based on the target vector of each agent in the new population, a population update operation is performed on the updated buffer population to delete some agents to obtain an optimized population, the optimized population is assigned to the initial population to obtain an updated initial population, and then the next iteration is entered; If the number of iterations reaches a preset number, the elite population is marked as the output population; Determining flight behavior decisions of the UAV in a plurality of time slots according to the output population; The data collection model includes: When the high-altitude platform receives the smart device When the data is , the corresponding information age is expressed as the following function model: ; When the aerial platform does not receive the smart device When the data is , the corresponding information age is expressed as the following function model: ; in, The latency of offloading data from drones to high-altitude platforms; For the time slot Smart devices The information age, is the duration of the time slot, For the time slot +1 Smart Device Information age; The flight status of the drone includes the flight speed of the drone (t); The calculation model of the UAV flight energy consumption is: ； ； in, For the time slot The energy consumption of drone flight, For the time slot The propulsion power consumption of the drone, and are the rotor profile power and rotor blade tip speed of the UAV in the hovering state, and are the induced power and average induced speed of the rotor of the UAV in the hovering state, is the fuselage drag ratio, is the air density, is the rotor stability, is the rotor disc area; The calculation model of the energy consumption of the drone data collection is: ； in, For the time slot Energy consumption of drone data collection; A collection of smart devices within the data collection scope. For smart devices k The delay in sending data to the drone, For smart devices The data transmission power, For the time slot Drones and smart devices The channel power gain between The calculation model of the energy consumption of drone data offloading is: ； in, For the time slot of drone data offloading energy consumption, The delay of offloading data from drones to high-altitude platforms, is the unloaded power of the UAV; The calculation model of energy consumption of transmitting from the drone to the smart device is: ； in, For the time slot The drone transmits energy consumption to smart devices. The power received by the drone from the base station; is a binary variable. If the condition is true, then ;otherwise ; For drones in time slots Hover and transfer energy, For drones in time slots Hover and collect data; The energy consumption model is: ； in, For the time slot Energy consumption of drones.

2. The method for collecting drone data and transmitting energy according to claim 1, characterized in that: Performing multi-objective proximal strategy optimization on multiple agents in the initial population to obtain a new population, including: For any agent in the initial population, a policy network is used to perform multiple environmental interactions to obtain a transfer sample corresponding to each environmental interaction; the transfer sample includes a current state, a next state, a vector value reward and an agent action; wherein the current state includes the state of each smart device and the state of the drone; the state of each smart device includes the distance between the smart device and the drone, the information age of the smart device and the remaining energy of the smart device; the state of the drone includes the coordinates of the drone, the number of tasks collected in the drone unloading queue and the number of environmental interactions; the agent action is a flight behavior decision of the drone; the vector value reward is determined based on multiple optimization objectives; According to the transfer samples corresponding to multiple environmental interactions, the advantage function is calculated using the following formula : ； Based on the advantage function and the transfer samples corresponding to multiple environmental interactions, the following formula is used to update the strategy network Parameters: ； Based on the transfer samples corresponding to multiple environmental interactions, the value network is updated using the following formula Parameters: ； in, is the discount factor, T is the number of time slots, and an environment interaction is performed in each time slot. is a hyperparameter, is the weight vector, For the The vector value reward for sub-environment interaction, For the The state of the sub-environment interaction, For the The state of the sub-environment interaction, l is the difference in the number of steps used to calculate the advantage function; Training loss function for the policy network, For expectations, for Update the previous policy network, For the The state of the sub-environment interaction, For the Agent actions for sub-environment interactions, represents the clipping function, Restricted to Within the scope, is a hyperparameter; Training loss function for the value network, For the Vector-valued rewards for sub-environment interactions; A new population is determined based on the initial population, the updated parameters of the policy network, and the updated parameters of the value network.

3. The method for collecting drone data and transmitting energy according to claim 1, characterized in that: In the drone system, the drone uses time division multiple access technology to collect data from the smart device, and the drone uses frequency division multiple access technology to transmit energy to the smart device; Based on the UAV system, a data collection model, an energy collection model and an energy consumption model are constructed, including: Determine, according to the relative position of the drone and the smart device in the drone system, a channel transmission rate from the smart device to the drone and a channel transmission rate from the drone to the high-altitude platform; Based on the channel transmission rate from the smart device to the drone, determine the delay of the smart device sending data to the drone; based on the channel transmission rate from the drone to the high-altitude platform, determine the delay of the drone unloading data to the high-altitude platform; Determine the data collection model based on the information age defined by the smart device and the delay of unloading data from the drone to the high-altitude platform; Constructing a drone energy collection sub-model according to the drone system, and determining a smart device energy collection sub-model according to the relative position of the drone and the smart device; the drone energy collection sub-model and the smart device energy collection sub-model constitute an energy collection model; Based on the drone system, combined with the delay of the smart device sending data to the drone, the delay of the drone unloading data to the high-altitude platform and the flight status of the drone, the drone flight energy consumption, drone data collection energy consumption, drone data unloading energy consumption and drone transmission energy consumption to the smart device are determined, and an energy consumption model is constructed.

4. The method for collecting drone data and transmitting energy according to claim 1, characterized in that: The multi-objective UAV flight optimization model includes an objective function and constraints; the objective function is: ； in, is the average information age of smart devices, , K is the total number of smart devices, T is the number of time slots, For the time slot Smart devices Information age; is the total energy consumption of the UAV, , For the time slot Energy consumption of drones; The average energy collected by smart devices, , For smart devices In time slot Energy collected; The constraints are: ； ； ； ； ； ； ； in, For drones in time slots Flight behavior decisions, For drones in time slots The flight direction, is the set of time slots, , are the horizontal and vertical coordinates of the drone’s coordinates, , are the coordinate limits, For drones and smart devices In time slot The horizontal distance within The data collection range of the drone, For the energy collection range, For smart device collection, The power received by the drone from the base station. is the duration of the time slot.

5. The method for collecting drone data and transmitting energy according to claim 3, characterized in that: The relative position of the drone and the smart device in the drone system includes an elevation angle and a distance between the drone and the smart device; Among them, the calculation formula for the channel transmission rate from the smart device to the drone is: ； ； ； in, For the time slot Smart devices k Channel transmission rate to the drone, For the time slot Smart devices to the bandwidth between drones, For smart devices The data transmission power, For the time slot Drones and smart devices The channel power gain between For smart devices The channel noise power of For the time slot Drones and smart devices Between LoS probability, based on the time slot Drones and smart devices The elevation angle between them is determined; is the additional attenuation coefficient of the non-line-of-sight link, For the time slot Drones and smart devices Between NLoS Probability, is the channel power gain at a reference distance of 1m, For the time slot Drones and smart devices The distance between is the path loss index; The calculation formula for the channel transmission rate from the UAV to the high-altitude platform is: ； in, For the time slot The channel transmission rate from the drone to the high-altitude platform, The bandwidth between the drone and the high-altitude platform, is the unloaded power of the UAV, For the time slot The channel power gain between the UAV and the high-altitude platform, is the channel noise power between the UAV and the high-altitude platform.

Images