Unloading strategy generation method, device, terminal and medium based on drone assistance

Abstract

The present invention discloses a method, device, terminal and medium for generating an unloading strategy based on the assistance of a drone. The method includes: constructing an edge computing network model assisted by multiple drones, and determining the computational delay information and energy consumption information generated by drones in the edge computing network model during the task computing process; determining the total cost function of the task computing process based on the computational delay information and the energy consumption information; optimizing the total cost function based on a preset QEMU algorithm to obtain the total cost of the target system, and the total cost of the target system is used to reflect the target movement strategy and target unloading strategy of the drone. The present invention takes into account the mobility of user devices in edge computing scenarios, and uses the QEMU algorithm to optimize the total cost function, so that the generated target movement strategy and target unloading strategy are suitable for dynamic environments.

Description

Technical Field

[0001] The present invention relates to the field of Internet of Things communication technology, and in particular to a method, device, terminal and medium for generating an unloading strategy based on the assistance of a drone. Background Art

[0002] Mobile edge computing (MEC) can improve the quality of service (QoS) within wireless access networks by distributing computing resources to the edge of the network. Common mobile edge computing methods include unmanned aerial vehicle (UAV)-assisted MEC, which can provide flexible deployment in dynamic scenarios by using drones as mobile edge servers.

[0003] However, in existing drone-assisted MEC systems, the mobility and differentiated mobility patterns of user equipments (UE) are often ignored, resulting in inaccurate planning of mobility patterns of highly mobile UE devices such as smart vehicles, wearable devices, and logistics robots in actual smart city and IoT application scenarios, affecting the model accuracy and reliability of drone-assisted MEC systems. At the same time, existing drone-assisted MEC systems also ignore the problem of offloading efficiency. For example, when drones fly along different routes, they will connect to different UEs and receive various computing requests, resulting in communication delays and energy consumption, resulting in low computing resource offloading efficiency.

[0004] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in response to the above-mentioned defects of the prior art, a method, device, terminal and medium for generating an offloading strategy based on drone assistance are provided, aiming to solve the problem that the drone-assisted MEC system in the prior art does not take into account the mobility and differentiated mobility modes of user equipment and the offloading efficiency, resulting in insufficient model accuracy of the drone-assisted MEC system and low computing resource offloading efficiency.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention provides a method for generating an unloading strategy based on drone assistance, wherein the method comprises:

[0008] Constructing a multi-UAV-assisted edge computing network model, and determining computing delay information and energy consumption information generated by UAVs in the edge computing network model during task computing;

[0009] Determining a total cost function of the task calculation process based on the calculation delay information and the energy consumption information;

[0010] Based on the preset QEMU algorithm, the total cost function is optimized to obtain the total cost of the target system, and the total cost of the target system is used to reflect the target movement strategy and target unloading strategy of the UAV.

[0011] In one implementation, the step of constructing a multi-UAV-assisted edge computing network model and determining computing delay information and energy consumption information generated by UAVs in the edge computing network model during task computing includes:

[0012] Acquire the current task number of the drone, and determine the observation situation of the drone based on the current task number, where the observation situation includes a first observation situation and a second observation situation;

[0013] Based on the observed conditions, calculation delay information and energy consumption information generated during the task calculation process are determined.

[0014] In one implementation, determining the computing delay information and energy consumption information generated during the task computing process based on the observed situation includes:

[0015] When the observation situation is the first observation situation, determining first calculation delay information and first energy consumption information, wherein the first delay information includes a transmission delay of the task offloading from the UE to the drone and a calculation delay of the drone, and the first energy consumption information includes energy consumption of the task offloading from the UE to the drone, calculation energy consumption of the drone, and propulsion energy consumption of the drone movement;

[0016] When the observation situation is the second observation situation, the second calculation delay information and the second energy consumption information are determined, wherein the second calculation delay information includes the transmission delay of the task unloading from the UE to the UAV, the calculation delay of the task unloading from the UAV to the base station and the calculation delay of the base station, and the second energy consumption information includes the energy consumption of the task unloading from the UE to the UAV, the calculation energy consumption of the task unloading from the UAV to the base station and the propulsion energy consumption of the UAV movement.

[0017] In one implementation, the transmission delay of the task offloading from the UE to the drone is:

[0018] Among them, D m (t) is the data size of the task, R um (t) is the offload data rate;

[0019] The computational delay of the drone is:

[0020] Among them, β um (t) is the indicator variable of the UAV, C m(t) is the number of CPU cycles required for each bit of data, f um (t) The computing resources allocated to UAV u for task m;

[0021] The computational delay of the task offloading from the UAV to the base station is:

[0022]

[0023] The calculation delay of the base station is:

[0024] Among them, F K The computing resources allocated to each task for the base station;

[0025] The energy consumption of offloading the task from UE to UAV is:

[0026] Among them, P M is the UE’s transmit power, P U r is the received power of the UAV, is the number of UEs that offload tasks to drones in time slot t, The transmission delay of the task offloading from UE to UAV;

[0027] The calculation energy consumption of the UAV is:

[0028] Where κ = 10 -26 , is a hardware-related constant, Calculation delay for drones;

[0029] The computing energy consumption of the task offloaded from the UAV to the base station is:

[0030] in, is the transmit power of the UE during time slot t, is the number of tasks unloaded from the UAV to the base station during time slot t, Computational latency for offloading tasks from the drone to the base station;

[0031] The propulsion energy consumption of the UAV motion is:

[0032] Among them, P pro (v u (t)) is the propulsion power of the UAV, v u (t) is the constant speed of uniform flight in the time slot (t, t+Δt).

[0033] In one implementation, the total cost function is optimized based on a preset QEMU algorithm to obtain a total cost of a target system, and the total cost of the target system is used to reflect a target movement strategy and a target unloading strategy of the drone, including:

[0034] The total cost of the system is optimized using a distributed partially observable Markov decision process;

[0035] In the optimization process, a reward function of the drone is set, and rewards and punishments are performed on the drone based on the reward function. The reward function is:

[0036]

[0037] Among them, η1 is a hyperparameter related to system cost, η2 is the collision constraint, η3 is the unloading failure constraint, η4 ​​is the no service constraint, η5 is the service compensation, C u (t) is the total system cost, is the number of drones within the drone safety range, is the number of tasks that failed to be unloaded on the drone, |U| is the number of drones, is the number of UEs that offload tasks to drones in time slot t, ε u (t) is the number of tasks in the task queue of drone u in the current time period t.

[0038] In one implementation, the total system cost is optimized based on a preset QEMU algorithm to obtain a target movement strategy and a target unloading strategy of the drone, wherein the target movement strategy and the target unloading strategy are used to reflect the minimized total system cost, including:

[0039] Based on the preset QEMU algorithm, the optimization process is divided into a pre-training phase and an exploration phase;

[0040] In the pre-training stage, an expert strategy algorithm is used to generate expert demonstration experience samples, and a loss function of the behavior clone of the demonstration sample is determined, and the value network is updated based on the loss function, wherein the expert strategy algorithm includes a preset greedy sailfish algorithm;

[0041] In the exploration phase, a greedy exploration strategy is adopted to perform autonomous exploration to obtain the target movement strategy.

[0042] In one implementation, the loss function of the behavior cloning is:

[0043] L BC (θ u )=E[(μ u (o u (t))-au (t)) 2 ], where E is the number of pre-training rounds, μ u (.) is the actor network, o u (t) is the local observation of UAV u at time t, a u (t) is the action of UAV u at time t.

[0044] In a second aspect, an embodiment of the present invention further provides a device for generating an unloading strategy based on drone assistance, wherein the device comprises:

[0045] An edge computing network model construction module is used to construct a multi-drone assisted edge computing network model and determine the computing delay information and energy consumption information generated by the drones in the edge computing network model during the task computing process;

[0046] A total cost function determination module, used to determine the total cost function of the task calculation process based on the calculation delay information and the energy consumption information;

[0047] The target system total cost determination module is used to optimize the total cost function based on a preset QEMU algorithm to obtain the target system total cost, and the target system total cost is used to reflect the target movement strategy and target unloading strategy of the UAV.

[0048] In the third aspect, an embodiment of the present invention also provides a terminal, wherein the terminal includes a memory, a processor, and a drone-assisted unloading policy generation program stored in the memory and executable on the processor, and when the processor executes the drone-assisted unloading policy generation program, the steps of the drone-assisted unloading policy generation method of any one of the above-mentioned schemes are implemented.

[0049] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein a drone-assisted unloading strategy generation program is stored on the computer-readable storage medium, and when the drone-assisted unloading strategy generation program is executed by a processor, the steps of the drone-assisted unloading strategy generation method described in any one of the above-mentioned schemes are implemented.

[0050] Beneficial effects: Compared with the prior art, the present invention provides a method, device, terminal and medium for generating an unloading strategy based on drone assistance, the method comprising: constructing a multi-drone-assisted edge computing network model, and determining the computational delay information and energy consumption information generated by the drones in the edge computing network model during the task computing process; determining the total cost function of the task computing process based on the computational delay information and the energy consumption information; optimizing the total cost function based on a preset QEMU algorithm to obtain the total cost of the target system, and the total cost of the target system is used to reflect the target movement strategy and target unloading strategy of the drone. The present invention takes into account the mobility of user devices in edge computing scenarios, and uses the QEMU algorithm to optimize the total cost function, so that the generated target movement strategy and target unloading strategy are suitable for dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flowchart of a specific implementation of a method for generating an unloading strategy based on drone assistance provided in an embodiment of the present invention.

[0052] Figure 2 This is a diagram of the edge computing network model architecture in a smart city IoT scenario provided by an embodiment of the present invention.

[0053] Figure 3 A probability density diagram of transmission delay and computation delay of a task provided by an embodiment of the present invention.

[0054] Figure 4 A framework diagram of the QEMU algorithm provided by an embodiment of the present invention.

[0055] Figure 5 A visualization diagram of a simulation experiment environment provided by an embodiment of the present invention.

[0056] Figure 6 A schematic diagram of the impact of different algorithms on average reward returns provided in an embodiment of the present invention.

[0057] Figure 7 A schematic diagram of the impact on various performances when changing the number of drones and user devices provided in an embodiment of the present invention.

[0058] Figure 8 The cumulative probability distribution diagram of the uninstallation failure rate under different algorithms provided in the embodiment of the present invention.

[0059] Fig. 9 A schematic diagram of the impact of energy consumption and delay with different weight ratios on algorithm performance provided by an embodiment of the present invention.

[0060] Fig.10 A functional schematic diagram of a drone-assisted unloading strategy generation device provided in an embodiment of the present invention.

[0061] Fig.11 A functional block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0063] The present embodiment provides a method for generating an unloading strategy based on drone assistance. In specific implementation, the present embodiment first constructs a multi-drone-assisted edge computing network model, and determines the computational delay information and energy consumption information generated by the drones in the edge computing network model during the task calculation process. Then, based on the computational delay information and the energy consumption information, the total cost function of the task calculation process is determined. Finally, based on the preset QEMU algorithm, the total cost function is optimized to obtain the total cost of the target system, and the total cost of the target system is used to reflect the target movement strategy and target unloading strategy of the drone. In this embodiment, the mobility of user devices in the edge computing scenario is taken into account, and the QEMU algorithm is used to optimize the total cost function so that the generated target movement strategy and target unloading strategy are suitable for dynamic environments.

[0064] The drone-assisted uninstallation strategy generation method of this embodiment can be applied to a terminal, which can be a smart device such as a mobile phone or a computer, or an external device connected to the drone-assisted uninstallation strategy generation device, or a device built into the drone-assisted uninstallation strategy generation device. Figure 1 As shown in , the unloading strategy generation method based on drone assistance of this embodiment includes the following steps:

[0065] Step S100: construct a multi-drone assisted edge computing network model, and determine the computing delay information and energy consumption information generated by the drones in the edge computing network model during the task computing process.

[0066] like Figure 2As shown, in this embodiment, a multi-UAV assisted edge computing network model in discrete time slots is first constructed, including a group of UAVs (abbreviated as U), a group of user equipment UE (abbreviated as M) and a group of base stations K, wherein several user equipment UEs are also referred to as UEs, and can be divided into three categories H, L and F according to the motion characteristics of UE: Category H represents high-speed UEs along the urban road network, such as vehicles and equipment installed on vehicles; Category L represents low-speed UEs not along the urban road network, such as pedestrians carrying user equipment and intelligent robots; Category F represents fixed UEs, such as smart homes, etc., wherein M = H∪L∪F. When the UAV takes off from the base station, the network is established. In the edge computing network, UEs are randomly distributed in a square area with a side length of s. Multiple UAVs fly over the area and communicate directly with the UE to provide edge computing services. In each time slot t, each UE generates a computationally intensive task Wm(t) that needs to be offloaded, wherein the data size of the task and the number of CPU cycles required for each bit of data are respectively represented by D m (t) and C m It can be seen that the multi-drone-assisted edge computing network model constructed in this embodiment is a composite heterogeneous mobile model, which takes into account the UE mobility in actual smart city and IoT application scenarios, making the model training environment closer to the real scenario.

[0067] During specific implementation, in order to facilitate the determination of the computing delay information and energy consumption information generated by the UAV in the edge computing network model during the task calculation process, the UE motion model and the UAV motion model will be designed preferentially in this embodiment.

[0068] Specifically, this embodiment considers three UE motion models. In the high-speed UE motion model, UEh∈H moves at a constant speed V on a city street. h Movement, UEh selects a destination and moves to it along the shortest path. After reaching the destination, UEh stays at the current location for a specified time th, then selects another destination and repeats this process. UEh uses the improved random waypoint (RWP-Ci) model for urban streets, which integrates the exploration mechanism and the return mechanism. The exploration mechanism is expressed as: UEh can choose an intersection that has never been reached as the destination with a probability of Among them, n S is the number of arrival points, ρ h ∈(0,1], ψ>0. The return mechanism is expressed as: UEh takes P old The probability of selecting the previously reached intersection as the destination is, where P old =1-P new .

[0069] In the low-speed UE motion model, UE l∈L does not depend on the road network. In this embodiment, the Gauss-Markov model is used to capture the motion pattern of UE l, and the speed of UE l at time t is recorded as v l (t), v l (t+1) is calculated as: in, α、v l and σ l denote the asymptotic mean and standard deviation of memory level and speed respectively. Subsequently, the coordinates of UE l at time t, pl(t)=[xl(t),yl(t)], are updated to pl(t+1)=pl(t)+v l (t)Δt, Δt is the time interval. Further, in order to prevent UE l from leaving the specified area of ​​the edge computing network model, when the calculated pl(t+1) is outside the specified area, UE l is controlled to maintain the current position.

[0070] In the fixed UE motion model, UEf∈F can be devices such as smart furniture, which are randomly distributed in the designated area of ​​the edge computing network model and remain stationary.

[0071] In the UAV motion model, this embodiment assumes that the UAV flies at a fixed altitude Z and a maximum speed V max , the motion of drone u at time t is represented by the tuple (v u (t),θ u (t)) indicates that the constant speed and direction angle of uniform flight in the time slot (t, t+Δt) are v u (t)∈[0,V max ] and θ u (t)∈[-π,π], the flight distance is Δd u (t) = v u (t)Δt. Based on this, the propulsion power of the UAV motion in this embodiment can be expressed as:

[0072] Among them, P0 is the blade profile power when the drone is hovering, V tip is the tip speed of the rotor blade, V is the actual uniform flight speed of the UAV, and P i and v0 represent the induced power and average rotor induced speed under hovering conditions, d0, ρ, r s and d They represent fuselage drag ratio, air density, rotor density and rotor disk area respectively.

[0073] In the specific implementation process, the user equipment (UE) can improve the service quality of mobile edge computing by sending the computing tasks it carries to the network edge (drone and base station) to complete the computing. Therefore, the task offloading process of this embodiment first includes the process of offloading the computing tasks from the UE to the drone. Specifically, if the coordinates of drone u at time t are marked as X u (t), expressed as (x u (t), y m (t), Z), then the position of UEM is X m (t), is expressed as (x m (t), y m (t), 0), the location of base station k is represented as (x k ,y k , 0). Since the service area of ​​the drone is a circular area, the coverage radius of the drone when working is r c = Z / tan(θ), where θ represents the maximum coverage angle. The elevation angle between UAVu and UEm at time t is θ um (t), at this time, the probability of establishing line-of-sight (LoS) and non-line-of-sight (NLoS) connections can be expressed as:

[0074]

[0075] Among them, a and b are constants determined by the communication environment.

[0076] In the process of offloading the computing task from UE to UAV, the channel gain between UAV u and UEm is:

[0077] Where, K0=(4πfc / c)2, represents the channel power gain at the reference distance d0 = 1m, fc is the carrier frequency, c is the speed of light, μ LoS and μ NLoS is the attenuation factor link between LoS and NLoS, d um (t) is the horizontal distance between u and m.

[0078] The offload data rate of computing tasks from UE to UAV is:

[0079] Among them, B U is the bandwidth of the drone, is the number of UEs that offload computing tasks to u in time slot t. In this embodiment, it is assumed that bandwidth is shared equally among all UEs. M is the transmit power of the UE, is the additive white Gaussian noise power of the UAV communication.

[0080] The transmission delay of computing tasks offloaded from UE to UAV is:

[0081] The energy consumption of offloading computing tasks from UE to UAV is: Among them, D m (t) is the data size of the task, R um (t) is the unloaded data rate, P M is the UE's transmit power, is the received power of the UAV, is the number of UEs that offload tasks to drones in time slot t, The transmission delay of the task offloading from UE to UAV.

[0082] In the above process, a drone can only provide offloading services to UEs within its coverage area, i.e., d um (t)<r, and in one time slot, each UE can only offload tasks to one UAV. Specifically, this embodiment defines the offloading indicator variable as ξum(t). When the computing task of UEm is served by UAV u, ξum(t)=1, and when the computing task of UEm is served by the base station, ξum(t)=0. Assuming that each UE can be served by at most one UAV at any given time, then When the value is 0, it means that the UE is currently in the drone coverage blind spot and no drone can be used to offload computing tasks; otherwise, the value is 1. When the UE is in the overlapping coverage area of ​​multiple drones, the UE selects the drone with the smallest transmission delay: Among them, U m A collection of available drones.

[0083] In one implementation, step S100 of this embodiment includes the following steps:

[0084] Step S101, obtaining the current number of tasks of the drone, and determining the observation situation of the drone based on the current number of tasks, wherein the observation situation includes a first observation situation and a second observation situation;

[0085] Step S102: Based on the observed situation, determine the calculation delay information and energy consumption information generated during the task calculation process.

[0086] Since the computing task in this embodiment can be unloaded from the UE to the UAV and completed in the UAV, or it can be unloaded from the UE to the UAV and then the computing capacity of the UAV is full, so the computing task needs to be unloaded from the UAV to the base station and completed in the base station, so this embodiment first needs to determine the observation situation of the current UAV, and then further determine the computing delay information and energy consumption information generated in the task calculation process more accurately based on the corresponding observation situation.

[0087] In one implementation, step S102 of this embodiment specifically includes the following steps:

[0088] Step S112: When the observation situation is the first observation situation, determine first calculation delay information and first energy consumption information, wherein the first delay information includes the transmission delay of the task offloading from the UE to the drone and the calculation delay of the drone, and the first energy consumption information includes the energy consumption of the task offloading from the UE to the drone, the calculation energy consumption of the drone, and the propulsion energy consumption of the drone movement;

[0089] Step S122: When the observation situation is the second observation situation, determine the second calculation delay information and the second energy consumption information, wherein the second calculation delay information includes the transmission delay of the task unloading from the UE to the UAV, the calculation delay of the task unloading from the UAV to the base station, and the calculation delay of the base station; the second energy consumption information includes the energy consumption of the task unloading from the UE to the UAV, the calculation energy consumption of the task unloading from the UAV to the base station, and the propulsion energy consumption of the UAV movement.

[0090] In the specific implementation, the task queue model following the first-in-first-out (FIFO) strategy is adopted in this embodiment to determine the observation status of the drone. Specifically, each drone has a pre-set limited capacity ∈ max , when the current number of tasks passed to the drone does not exceed ∈ max , the task is calculated in the UAV first. At this time, the observation of the UAV is the first observation. When the number of current tasks passed to the UAV exceeds ∈ max When , the drone must offload the tasks that exceed the capacity to the nearest base station. At this time, the drone's mobile strategy is the second observation situation. It can be seen that the drone in this embodiment retains the offloading ratio decision option of directly processing the task or offloading it to the base station for the second time. In each time slot, the offloading ratio decision made by the drone is expressed as δ u (t)∈[0,1]. Let ∈ u (t) represents the number of tasks in the task queue of UAV u in the current time slot, and defines the indicator variable β of UAVu um (t) to decide whether to uninstall W m(t), for example, at time slot t, UAV u locally processes the first task ν in its queue u (t), then determine β um (t) = 1, if the task is offloaded to the base station, then determine In the process of offloading the task to the base station, the computational delay of the task offloading from the drone to the base station is:

[0091] The computing energy consumption of offloading computing tasks from the UAV to the base station is:

[0092]

[0093] Among them, β um (t) is the indicator variable of the drone, D m (t) is the data size of the task, R um (t) is the offload data rate, is the transmit power of the UE during time slot t, is the number of tasks unloaded from the UAV to the base station during time slot t, Computational latency for offloading tasks from the drone to the base station.

[0094] In one implementation, when the computing task of this embodiment is completed in a drone, the task offloading process of this embodiment needs to consider the task computing process of the drone in order to obtain the computing delay information and energy consumption information of the drone.

[0095] Specifically, in a traditional FIFO queue, the task calculation process of a UAV is usually executed in an arranged order. However, this sequential execution method may cause subsequent tasks to time out when applied to an MEC server. In order to ensure fair service to each user, this embodiment incorporates a parallel processing mechanism into the UAV-MEC, which allows tasks in the queue to be executed in parallel to a certain extent. Since tasks do not arrive at the UAV-MEC at the same time in the actual process, in order to avoid introducing certain program errors by using the parallel processing mechanism here, this embodiment analyzes this error, such as Figure 3 As shown in the experimental analysis results, the unit difference between the transmission delay and the calculation delay of the task is about 10 3 , the transmission delay of the task is significantly less than the calculation delay. Therefore, the error caused by the use of the parallel processing mechanism in this embodiment can be considered to be negligible and within an acceptable range. Therefore, the total computing resource F U Fair distribution among all currently ongoing tasks. Based on the assumption of fair distribution of total computing resources, the computing resources allocated to task m by drone u are f um (t), Then, the calculation delay of the drone is:

[0096] The computational energy consumption of the drone is:

[0097]

[0098] Among them, β um (t) is the indicator variable of the drone, D m (t) is the data size of the task, C m (t) is the number of CPU cycles required for each bit of data, κ=10 -26 , is a hardware-related constant, Calculation delay for the drone.

[0099] In another implementation, when the computing task of this embodiment needs to be offloaded from the drone to the base station for a second time, the task offloading process of this embodiment also needs to consider the task calculation process of the base station in order to obtain the calculation delay information of the base station.

[0100] In specific implementation, the drone will give priority to offloading the computing task to the base station with the smallest transmission delay, that is, the base station with the closest horizontal distance. At this time, the calculation delay of the base station is:

[0101]

[0102] Among them, β um (t) is the indicator variable of the drone, D m (t) is the data size of the task, C m (t) is the number of CPU cycles required for each bit of data, F K The computing resources allocated to each task for the base station.

[0103] Step S200: Determine a total cost function of the task calculation process based on the calculation delay information and the energy consumption information.

[0104] Specifically, the total cost of the task calculation process in this embodiment includes the total calculation delay and total energy consumption of UAV u within time t. u (t)) can be expressed as:

[0105]

[0106] in, To calculate the transmission delay of the task offloaded from UE to UAV, The calculation delay for the drone, To calculate the computational delay of offloading the task from the UAV to the base station, is the calculated delay of the base station.

[0107] Total energy consumption (E u (t)) can be expressed as: in, and They represent the mission processing energy consumption and flight propulsion energy consumption of the UAV respectively. The mission processing energy consumption and flight propulsion energy consumption of the UAV are:

[0108]

[0109] in, To calculate the energy consumption of offloading tasks from UE to UAV, The computing energy consumption of offloading computing tasks from drones to base stations, is the computing energy consumption of the UAV, P pro (v u (t)) is the propulsion power of the UAV, v u (t) is the constant speed of uniform flight in the time slot (t, t+Δt).

[0110] Furthermore, this embodiment can allocate weights between the total computation delay and the total energy consumption of the UAV u within time t, and express the total cost function (C u (t)), that is, C u (t) = ω1E u (t)+ω2τ u (t), where ω1 and ω2 represent the weights of the relative importance of energy consumption and computational latency, respectively.

[0111] Step S300: Based on a preset QEMU algorithm, the total cost function is optimized to obtain the total cost of the target system, where the total cost of the target system is used to reflect the target movement strategy and target unloading strategy of the UAV.

[0112] After the total cost function in the edge computing network model is determined, since the mobility of UEs, differentiated mobility models, and unloading failures in the model are also considered in this embodiment, in order to achieve efficient drone unloading decisions and flight trajectory control, this embodiment needs to optimize the relevant parameters between the total cost functions. Specifically, this embodiment expresses the optimization problem of the relevant parameters between the total cost functions as follows:

[0113]

[0114] This embodiment optimizes the movement decision of the drone while finding the minimum cost (v u (t),θ u (t)) and the unloading rate decision δ u(t), in this process, the optimization problem also needs to meet the following constraints:

[0115] st ω1+ω2=1

[0116]

[0117] Among them, D min Defined as the minimum safety distance limited to prevent collisions between drones, V represents the set of vertices in the specified square area, and also represents the location of the base station. According to the definition, the drone starts working from the base station location and is required to remain in the specified area. In order to prevent unloading failure, the last restriction is used to ensure that the UE is always within the service range of the drone during the transmission of the calculation results. Each drone naturally acts as an intelligent agent, which makes this optimization problem suitable for exploration and solution within the framework of the multi-agent reinforcement learning algorithm (MARL algorithm), which includes the traditional MADDPG algorithm and MATD3 algorithm.

[0118] In one implementation, step S300 of this embodiment specifically includes the following steps:

[0119] Step S301, optimizing the total cost of the system by using a distributed partially observable Markov decision process;

[0120] Step S302: During the optimization process, a reward function of the drone is set, and rewards and punishments are performed on the drone based on the reward function. The reward function is:

[0121]

[0122] Among them, η1 is a hyperparameter related to system cost, η2 is the collision constraint, η3 is the unloading failure constraint, η4 ​​is the no service constraint, η5 is the service compensation, C u (t) is the total system cost, is the number of drones within the drone safety range, is the number of tasks that failed to be unloaded on the drone, |U| is the number of drones, is the number of UEs that offload tasks to drones in time slot t, ε u (t) is the number of tasks in the task queue of drone u in the current time period t.

[0123] Specifically, the total cost joint optimization of the drone-assisted MEC system in this embodiment can be expressed as a distributed partially observable Markov decision process (Dec-POMDP):<N,S,A,P,R,O,n,γ> , where N is the set of agents, S is the state set, A is the action set, P is the state transition function, R is the reward function shared by all agents, O is the observation set, n is the number of agents, and γ is the discount factor. Specifically as follows: Agent: Each drone is regarded as an agent, and N is a finite set of n = |U| agents. State: The state at time t includes the location information, motion state, and connection state of all drones and UEs, expressed as s(t)∈S. Action: Drone decisions include mobile strategies and task offloading ratios. At time slot t, the action of drone u is expressed as a u (t) = {v u (t), θ u (t), δ u (t)},a u (t)∈A, and the global action is expressed as State transition function: When the agent interacts with the environment by performing an action a(t), the state is updated according to the transition function P(s(t+1)|s(t), a(t)). Observation: The observation set is denoted as O, and the local observation o of drone u at time t is u (t)∈O is the partial information obtained from s(t), including the positions of all UEs and other UAVs relative to m, the motion states of all agents, and the unloading decisions between all UEs and u. Formally,

[0124] Preferably, this embodiment also sets a reward function for the drone during the above optimization process, and rewards or punishes the drone based on the reward function. u (t)), η1 is a hyperparameter related to the system cost, and η2 constitutes a collision constraint. If the distance between two drones is lower than a predetermined safety parameter, the two drones will be penalized. is the number of drones within the drone safety range. η3 is the unloading failure constraint, is the number of tasks on the UAV that suffer offloading failures. η3 plays a role in preventing offloading failures and aims to force the UAV to dynamically adjust to the changes in the UE location, which in turn reduces the occurrence of connection interruptions to ensure the robust execution of tasks on the UAV. η4 contributes to the no-service constraint, which imposes a penalty on all UAVs if any UE is in an out-of-service state. η5 is attributed to the service compensation, which offsets the no-service penalty with a value proportional to the current number of UEs served by UAVu. Cu (t) is the total system cost, |U| is the number of drones, is the number of UEs that offload tasks to the drone in time slot t. Therefore, this embodiment defines the action value function based on the expected reward of the drone agent as follows, where γ∈[0,1):

[0125] In one implementation, step S300 of this embodiment further includes the following steps:

[0126] Step S303: Based on the preset QEMU algorithm, the optimization process is divided into a pre-training phase and an exploration phase;

[0127] Step S304, using an expert strategy algorithm in the pre-training stage to generate expert demonstration experience samples, and determining a loss function of the behavior clone of the demonstration sample, and updating the value network based on the loss function, wherein the expert strategy algorithm includes a preset greedy sailfish algorithm;

[0128] Step S305: In the exploration phase, a greedy exploration strategy is used to perform autonomous exploration to obtain the target movement strategy.

[0129] like Figure 4 As shown in , the preset QEMU algorithm is adopted in this embodiment, and the processing process of the above optimization problem is divided into a pre-training stage and an exploration stage.

[0130] In the QEMU algorithm, for each agent, we first randomly initialize the agent with weight θ u Actor Network and with weights {ω u,i} i＝1,2 The critic network {Q u,i (s,a1,a2,...,a U )} i＝1,2 The target network is represented as and are initialized as copies of the actor network and the critic network respectively. In order to improve sample efficiency and stabilize the training process, each drone uses a capacity of 10 5 The target network is gradually updated using the soft update mechanism defined by the parameter τ using the experience replay buffer D. The target network not only facilitates smoother training, but also defines the optimization objective of the critic network as:

[0131]

[0132] In the pre-training phase, this embodiment quickly improves the performance of the action network to a fairly high level through expert strategy demonstration. At the same time, a "warm-up" period is designed for the critic network to prevent a catastrophic drop in performance at the beginning of the exploration phase. As the algorithm transitions to the exploration phase, the model of this embodiment breaks away from the expert strategy and uses the ε greedy exploration strategy to autonomously explore potential better decisions. In this process, since high-quality expert data is difficult to obtain, this embodiment proposes an expert algorithm called Greedy-Sailfish Algorithm (GSF), which combines the Sailfish algorithm with the Greedy Algorithm, where the Greedy Algorithm is used to guide flight direction decisions, while the subsequent unloading rate and flight speed decisions are regarded as simplified constrained optimization problems and are optimized using the Sailfish Algorithm.

[0133] As shown in the following Algorithm 2, if ∈ m (t)≠0, then u selects the UE m farthest from it in its offloading queue f . Then, u moves at the maximum speed V max Fly to m f , to prevent m f Uninstallation failed. Otherwise, if ∈ m (t) = 0, then u selects the nearest unserved UE m from the global c , and to m c Fly away. Given Using the Sailfish Optimizer to determine the flight speed v u and the unloading ratio δ u Decision making requires a reformulation of the problem and its associated constraints:

[0134]

[0135] st ω1+ω2=1

[0136]

[0137] The global action execution function is expressed as f(x)=g(s(t),a(t)). For a given state s(t) and global action a(t), the function g returns the value of f(x) and performs forward and backtracking in a computer simulation experimental environment.

[0138] In order to address the challenges of training with randomly initialized network parameters in a sparse reward space, a pre-training round Epretraining is performed at the initial stage of training. As shown in Algorithm 1 above, in the pre-training stage, the expert strategy GSF algorithm is used instead of the actor network for decision making. This process generates expert demonstration experience samples, which are then stored in a buffer. At each iteration, a tuple (s(t), ou (t),a u (t),r u (t),s(t+1),o u A random mini-batch B consisting of samples from (t+1)) is used to update the network. Specifically, behavior cloning pre-training is performed on the actor network, and the learning objective aims to minimize the difference between the decisions made by the policy network and the decisions made by the expert policy.

[0139] In one implementation, the loss function of behavior cloning in this embodiment is defined as follows:

[0140] L BC (θ u )=E[(μ u (o u (t))-a u (t)) 2 ], where E is the number of pre-training rounds, μ u (.) is the actor network, o u (t) is the local observation of UAV u at time t, a u (t) is the action of UAV u at time t.

[0141] Behavior cloning pre-training of the actor network eliminates the inefficiency of exploring better behaviors only through random interactions with the environment when the policy is poor. Instead, these policies quickly reach a high level by imitating expert algorithms, establishing a strong learning starting point from the beginning and promoting more effective exploration in high-reward areas. The critic network is then warmed up to mitigate the overly biased value estimates of the untrained (cold start) critic network. This bias may cause the model to forget well-performing policies and experience severe performance degradation. The loss function for warm-up training using expert-guided experience is defined as follows:

[0142]

[0143] The network training in this stage basically follows the conventional online MATD3 algorithm. In order to improve performance through fine-tuning and prevent overfitting of the expert strategy, the attenuated ε greedy exploration strategy is adopted in this embodiment. When the model chooses exploration, Gaussian noise is introduced into the output of the policy network. The action is calculated as follows:

[0144]

[0145] The value of ε gradually decays as the number of training iterations increases, which means that the intensity of exploration decreases as the network performance improves. Expert strategies are no longer used at this stage. Instead, the QEMU algorithm uses its own strategy network μ u (o) Interact with the environment to capture experience. Various methods are used to reduce the overestimation bias of the critic network, including the dual critic mechanism and adding noise to the predictions of the target actor network. By taking the lower value of the two critic network outputs, the critic network y u The original optimization objective is reshaped into y′ u , as shown below:

[0146]

[0147] It is worth noting that despite the warm-up process during pre-training, it is still impossible to ensure that the current critic network has reached a sufficiently high performance level. In addition, the experience used in the warm-up phase is generated by the expert strategy rather than the policy network itself, resulting in a different distribution. Therefore, during the exploration process, the critic network may still provide incorrect guidance to the actor network, leading to the degradation of the policy network. To alleviate this problem, the QEMU algorithm adopts a delayed update strategy for the policy network, allowing the trainer to have time to wait for the critic network to stabilize. Specifically, as shown in line 22 of Algorithm 1, after each TD update of the critic network, the actor network is updated according to the policy gradient defined as:

[0148]

[0149] In addition, in order to prove that the optimization processing of the total cost function by the QEMU algorithm in this embodiment is effective, this embodiment also conducts relevant experiments for verification. Figure 5 As shown, the simulation area is a square with a side length of s = 250m. There is a base station in each corner. The grid street network allows UEs to move at high speed in the area. 10 drones take off from the base station, with a flight altitude of Z = 100m and a maximum speed of V max =10m / s. In each random process, from the moment the drone takes off, the simulation iterates for 50 steps, recorded as T=50. Since multiple drones take off from the same starting point and since this experiment ignores the process of drone ascent, the collision problem is not considered in the first 5 steps, that is, the value of η2 is set to 0, the edge angle of the coverage area is set to θ=50°, there are 200 UEs in total, and the ratio of the three types of UEs (|H|:|L|:|F|) is 2:5:1.

[0150] Figure 5The figure shows a scene in which a drone has just taken off from a base station, started to form a network and covered the UEs in the area. Assuming that the experimental environment in this experiment is symmetrical and each base station and drone are isomorphic, the 10 drones can be simply and fairly allocated to the four base stations: the base stations in the upper left and lower left corners each accommodate 3 drones, and the base stations in the upper right and lower right corners each accommodate 2 drones. The drones "converge" from the four corners to the center, gradually reducing the area of ​​the unserviced area in the center of the area, thereby improving the service coverage of the system. The simulation in this embodiment is performed using Python and PyTorch. In both the behavior network and the critic network, four fully connected hidden layers are used, with the number of neurons being [400, 800, 800, 400]. All networks are based on 10 -5 The learning rate is , and the Adam optimizer is used for updating. For the decaying ε greedy exploration strategy, ε is initialized to 0.8 and decays at a rate of 0.999. In addition, σ ε and σ r They are set to 0.2×c and 0.2 respectively, and the strategy update frequency TD is fixed to 5. The main parameters of the simulation network environment are shown in Table 1 below.

[0151]

[0152] Table 1 Network environment parameters

[0153] from Figure 6 It can be seen that the rewards of the QEMU algorithm are improved by 36.62% and 62.47% compared with the traditional MADDPG and MATD3 algorithms, respectively. Compared with MATD3, QEMU only needs 36.59% of the episodes in pre-training to converge to the rewards of the Naive-Greedy algorithm. When transitioning from the pre-training stage to the exploration stage, the performance of the policy network drops slightly, about 23.89% of the previous training rewards. This is attributed to the warm-up operation performed on the critic network and the delayed update operation in the actor network training, which prevents the network from further catastrophic degradation. Subsequently, after only 1,000 rounds, it quickly recovered to performance comparable to the Naive-Greedy algorithm and further explored potential superior solutions, surpassing the MADDPG algorithm and MATD3 algorithm on the baseline.

[0154] It is worth noting that, as shown in Table 2, although the rewards obtained by the QEMU algorithm in the experiment did not exceed the expert algorithm GSF designed in this embodiment, the QEMU algorithm showed significant superiority in practical usability compared with GSF. First, Table 2 compares the average time spent on each decision in the algorithm simulation. Obviously, the decision time of the QEMU algorithm remains within an acceptable range, usually less than 10 milliseconds, while the decision time required by GSF always exceeds hundreds of milliseconds. Another fundamental reason is that GSF is based on the SFO metaheuristic algorithm, which requires the objective function to be known and can be repeatedly substituted for optimization. In the experimental simulation, the action decision, that is, the solution to the problem, can be repeatedly substituted into the virtual environment for optimization, and the objective function value can be obtained by forward and backward tracking. However, in practical applications, forward and backward are actually impossible. Therefore, the algorithm lacks practicality, which indirectly highlights the advantages of the QEMU algorithm.

[0155]

[0156] Table 2 Average time for the agent to complete a decision in each iteration (in ms)

[0157] exist Figure 7 In the experiment, the simulation results consistently show that the algorithm of this embodiment outperforms the baseline in various indicators by changing the number of drones and UEs. Service coverage refers to the proportion of users within the coverage of the drone, reflecting the basic network deployment and service coverage capabilities of the drone. It is observed that the service coverage of QEMU exceeds 95%, which is on par with GSF and surpasses all other baseline algorithms. It is worth noting that compared with traditional MARL algorithms (i.e., MADDPG algorithm and MATD3 algorithm), QEMU shows obvious energy-saving advantages and always shows the lowest system energy consumption in all scenarios. As the number of UEs increases, the energy consumption of the traditional MADDPG algorithm exceeds that of the Greedy algorithm. In contrast, the energy consumption of QEMU not only remains at a low level, but even exceeds GSF by 5.26% in the scenario of 1000 UEs. This further verifies the effectiveness of the enhancements of this embodiment in adapting to large-scale and large-scale applications. The performance of QEMU is particularly noteworthy in reducing the offloading failure rate, which makes a great contribution to achieving the lowest average user delay performance. Compared with MADDPG and MATD3 algorithms, the latency is reduced by 28.13% to 40.67%, and the offloading failure rate is reduced by 22.23% to 44.74%, respectively.

[0158] Figure 8is the cumulative probability distribution function of the offloading failure rate, and the dotted line represents the average value of the offloading failure rate. It can be seen that the QEMU algorithm ensures a lower offloading failure rate in more instances and achieves the lowest average offloading failure rate among all baseline algorithms. It should be noted that although the offloading failure rate of the Random algorithm is significantly lower than that of the GSF expert algorithm, this does not necessarily mean that the Random algorithm is more effective in avoiding offloading failures than the GSF algorithm. This phenomenon is actually because the premise of offloading failure is to start task offloading, such as Figure 7 As shown in the first figure on the upper left, the Random algorithm cannot achieve high user coverage, resulting in users being out of service, thus avoiding a large number of unloading failure events. In contrast, the QEMU algorithm achieves a high user coverage comparable to the GSF algorithm while also ensuring a low unloading failure rate. This proves that the reward mechanism designed in this embodiment has a positive impact, effectively enhances QoS, and alleviates the problem of unloading failure.

[0159] Fig. 9 In this embodiment, the policy's preference for energy consumption and QoS is fine-tuned by adjusting the weights ω1 and ω2, further demonstrating the flexibility and controllability of the algorithm. For QEMU, by increasing ω1, the system's energy consumption can be further reduced by 20.55% at the expense of QoS. Vice versa, increasing ω2 will further improve service coverage by 2.74%, reduce offloading failure rate by 10.16%, and reduce latency by 11.24%. This emphasizes the algorithm's adaptability to various optimization objectives and its ability to balance conflicting performance indicators. QEMU's fine-tuning capabilities can optimize system performance based on dynamic environments and user needs. By appropriately adjusting ω1 and ω2, operators can effectively manage the balance between energy efficiency and service quality to meet diverse application requirements. In MEC systems where effective resource management and excellent QoS are critical, this flexibility makes QEMU an ideal solution for the future.

[0160] In summary, this embodiment provides a method for generating an unloading strategy based on drone assistance. During specific implementation, this embodiment first constructs a multi-drone-assisted edge computing network model, and determines the calculation delay information and energy consumption information generated by the drones in the edge computing network model during the task calculation process. Then, based on the calculation delay information and the energy consumption information, the total cost function of the task calculation process is determined. Finally, based on the preset QEMU algorithm, the total cost function is optimized to obtain the total cost of the target system, and the total cost of the target system is used to reflect the target movement strategy and target unloading strategy of the drone. This embodiment takes into account the mobility of user devices in edge computing scenarios, and uses the QEMU algorithm to optimize the total cost function, so that the generated target movement strategy and target unloading strategy are suitable for dynamic environments.

[0161] Based on the above embodiments, the present invention also provides a device for generating an unloading strategy based on drone assistance, such as Fig.10 As shown in , the unloading strategy generation device based on drone assistance includes: an edge computing network model construction module 10, a total cost function determination module 20, and a target system total cost determination module 30. Specifically, the edge computing network model construction module 10 is used to construct an edge computing network model assisted by multiple drones, and determine the calculation delay information and energy consumption information generated by the drones in the edge computing network model during the task calculation process. The total cost function determination module 20 is used to determine the total cost function of the task calculation process based on the calculation delay information and the energy consumption information. The target system total cost determination module 30 is used to optimize the total cost function based on a preset QEMU algorithm to obtain the total cost of the target system, and the target system total cost is used to reflect the target movement strategy and target unloading strategy of the drone.

[0162] In one implementation, the edge computing network model building module 10 includes:

[0163] A mobile strategy determination unit, configured to obtain the current number of tasks of the drone, and determine an observation situation of the drone based on the current number of tasks, wherein the observation situation includes a first observation situation and a second observation situation;

[0164] The calculation delay information and energy consumption information determination unit is used to determine the calculation delay information and energy consumption information generated during the task calculation process based on the observation situation.

[0165] In one implementation, the mobile strategy determination unit includes:

[0166] a first observation situation information determination subunit, configured to determine first calculation delay information and first energy consumption information when the observation situation is the first observation situation, wherein the first delay information includes a transmission delay of the task unloaded from the UE to the drone and a calculation delay of the drone, and the first energy consumption information includes energy consumption of the task unloaded from the UE to the drone, calculation energy consumption of the drone, and propulsion energy consumption of the drone movement;

[0167] The second observation situation information determination subunit is used to determine the second calculation delay information and the second energy consumption information when the observation situation is the second observation situation, wherein the second calculation delay information includes the transmission delay of the task unloaded from the UE to the UAV, the calculation delay of the task unloaded from the UAV to the base station and the calculation delay of the base station; the second energy consumption information includes the energy consumption of the task unloaded from the UE to the UAV, the calculation energy consumption of the task unloaded from the UAV to the base station and the propulsion energy consumption of the UAV movement.

[0168] In one implementation, the transmission delay of the task offloading from the UE to the drone is:

[0169] Among them, D m (t) is the data size of the task, R um (t) is the offload data rate;

[0170] The computational delay of the drone is:

[0171] Among them, β um (t) is the indicator variable of the UAV, C m (t) is the number of CPU cycles required for each bit of data, f um (t) The computing resources allocated to UAV u for task m;

[0172] The computational delay of the task offloading from the UAV to the base station is:

[0173]

[0174] The calculation delay of the base station is:

[0175] Among them, F K The computing resources allocated to each task for the base station;

[0176] The energy consumption of offloading the task from UE to UAV is:

[0177] Among them, P M is the UE's transmit power, is the received power of the UAV, is the number of UEs that offload tasks to drones in time slot t, The transmission delay of the task offloading from UE to UAV;

[0178] The calculation energy consumption of the UAV is:

[0179] Where κ = 10 -26 , is a hardware-related constant, Calculation delay for drones;

[0180] The computing energy consumption of the task offloaded from the UAV to the base station is:

[0181] in, is the transmit power of the UE during time slot t, is the number of tasks unloaded from the UAV to the base station during time slot t, Computational latency for offloading tasks from the drone to the base station;

[0182] The propulsion energy consumption of the UAV motion is:

[0183] Among them, P pro (v u (t)) is the propulsion power of the UAV, v u (t) is the constant speed of uniform flight in the time slot (t, t+Δt).

[0184] In one implementation, the target system total cost determination module 30 includes:

[0185] A system total cost optimization unit, used for optimizing the system total cost by using a distributed partially observable Markov decision process;

[0186] A reward function setting unit is used to set a reward function of the drone during the optimization process, and to reward or punish the drone based on the reward function, wherein the reward function is:

[0187]

[0188] Among them, η1 is a hyperparameter related to system cost, η 2 is the collision constraint, η 3 is the offloading failure constraint, η4 ​​is the no-service constraint, η5 is the service compensation, C u (t) is the total system cost, is the number of drones within the drone safety range, is the number of tasks that failed to be unloaded on the drone, |U| is the number of drones, is the number of UEs that offload tasks to drones in time slot t, ε u (t) is the number of tasks in the task queue of drone u in the current time period t.

[0189] In one implementation, the target system total cost determination module 30 further includes:

[0190] The optimization process segmentation unit is used to divide the optimization process into a pre-training phase and an exploration phase based on a preset QEMU algorithm;

[0191] A pre-training stage optimization unit, used to use an expert strategy algorithm in the pre-training stage to generate expert demonstration experience samples, determine a loss function of the behavior clone of the demonstration sample, and update the value network based on the loss function, wherein the expert strategy algorithm includes a preset greedy sailfish algorithm;

[0192] The exploration phase optimization unit is used to adopt a greedy exploration strategy to perform autonomous exploration in the exploration phase to obtain the target movement strategy.

[0193] In one implementation, the loss function of the behavior cloning is:

[0194] Where E is the number of pre-training rounds, μ u (.) is the actor network, o u (t) is the local observation of UAV u at time t, a u (t) is the action of UAV u at time t.

[0195] The working principles of each module in the drone-assisted unloading strategy generation device of this embodiment are the same as the principles of each step in the above method embodiment, and will not be repeated here.

[0196] Based on the above embodiment, the present invention further provides a terminal, the principle block diagram of the terminal can be as follows: Fig.11 The terminal may include one or more processors 100 ( Fig.11 Only one is shown in the figure), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a drone-assisted unloading strategy generation program. When one or more processors 100 execute the computer program 102, the various steps in the drone-assisted unloading strategy generation method embodiment can be implemented. Alternatively, when one or more processors 100 execute the computer program 102, the functions of the various modules / units in the drone-assisted unloading strategy generation device embodiment can be implemented, which is not limited here.

[0197] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0198] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the electronic device. Further, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0199] Those skilled in the art will understand that Fig.11 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0200] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double operational data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0201] In summary, the present invention discloses a method, device, terminal and medium for generating an unloading strategy based on drone assistance, the method comprising: constructing a multi-drone-assisted edge computing network model, and determining the computational delay information and energy consumption information generated by the drones in the edge computing network model during the task computing process; determining the total cost function of the task computing process based on the computational delay information and the energy consumption information; optimizing the total cost function based on a preset QEMU algorithm to obtain the total cost of the target system, and the total cost of the target system is used to reflect the target movement strategy and target unloading strategy of the drone. The present invention takes into account the mobility of user devices in edge computing scenarios, and uses the QEMU algorithm to optimize the total cost function, so that the generated target movement strategy and target unloading strategy are suitable for dynamic environments.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating an unloading strategy based on drone assistance, characterized in that: The method comprises: Constructing a multi-UAV-assisted edge computing network model, and determining computing delay information and energy consumption information generated by UAVs in the edge computing network model during task computing; Determining a total cost function of the task calculation process based on the calculation delay information and the energy consumption information; Based on the preset QEMU algorithm, the total cost function is optimized to obtain the total cost of the target system. The total cost of the target system is used to reflect the target movement strategy and target unloading strategy of the UAV, including: Based on the preset QEMU algorithm, the optimization process is divided into a pre-training phase and an exploration phase; In the pre-training stage, an expert strategy algorithm is used to generate expert demonstration experience samples, and a loss function of the behavior clone of the expert demonstration experience samples is determined. The value network is updated based on the loss function. The expert strategy algorithm includes a preset greedy sailfish algorithm, which is obtained by combining a greedy algorithm and a sailfish algorithm. The greedy algorithm is used to guide flight direction decisions, and the sailfish algorithm is used to optimize unloading rate and flight speed decisions. The loss function of the behavior clone is: L BC (θ u )=E[(μ u (o u (t))-a u (t)) 2 ], where E is the number of pre-training rounds, μ u (.) is the actor network, o u (t) is the local observation of UAV u at time t, a u (t) is the action of UAV u at time t; In the exploration phase, a greedy exploration strategy is adopted to perform autonomous exploration to obtain the target movement strategy.

2. The method for generating an unloading strategy based on drone assistance according to claim 1, characterized in that: The constructing of a multi-UAV assisted edge computing network model and determining computing delay information and energy consumption information generated by UAVs in the edge computing network model during task computing includes: Acquire the current task number of the drone, and determine the observation situation of the drone based on the current task number, where the observation situation includes a first observation situation and a second observation situation; Based on the observed conditions, calculation delay information and energy consumption information generated during the task calculation process are determined.

3. The method for generating an unloading strategy based on drone assistance according to claim 2 is characterized in that: The determining, based on the observed situation, the calculation delay information and the energy consumption information generated during the task calculation process includes: When the observation situation is the first observation situation, determining first calculation delay information and first energy consumption information, wherein the first calculation delay information includes a transmission delay of the task offloading from the UE to the drone and a calculation delay of the drone, and the first energy consumption information includes energy consumption of the task offloading from the UE to the drone, calculation energy consumption of the drone, and propulsion energy consumption of the drone movement; When the observation situation is the second observation situation, the second calculation delay information and the second energy consumption information are determined, wherein the second calculation delay information includes the transmission delay of the task unloading from the UE to the UAV, the calculation delay of the task unloading from the UAV to the base station and the calculation delay of the base station, and the second energy consumption information includes the energy consumption of the task unloading from the UE to the UAV, the calculation energy consumption of the task unloading from the UAV to the base station and the propulsion energy consumption of the UAV movement.

4. The method for generating an unloading strategy based on drone assistance according to claim 3 is characterized in that: The transmission delay of the task offloading from UE to UAV is: Among them, D m (t) is the data size of the task, R um (t) is the offload data rate; The computational delay of the drone is: Among them, β um (t) is the indicator variable of the UAV, C m (t) is the number of CPU cycles required for each bit of data, f um (t) The computing resources allocated to UAV u for task m; The computational delay of the task offloading from the UAV to the base station is: The calculation delay of the base station is: Among them, F K The computing resources allocated to each task for the base station; The energy consumption of offloading the task from UE to UAV is: Among them, P M is the UE’s transmit power, P U r is the received power of the UAV, is the number of UEs that offload tasks to drones in time slot t, The transmission delay of the task offloading from UE to UAV; The calculation energy consumption of the UAV is: Where κ = 10 -26 , is a hardware-related constant, Calculation delay for drones; The computing energy consumption of the task offloaded from the UAV to the base station is: in, is the transmit power of the UE during time slot t, is the number of tasks unloaded from the UAV to the base station during time slot t, Computational latency for offloading tasks from the drone to the base station; The propulsion energy consumption of the UAV motion is: Among them, P pro (v u (t)) is the propulsion power of the UAV, v u (t) is the constant speed of uniform flight in the time slot (t, t+Δt).

5. The method for generating an unloading strategy based on drone assistance according to claim 4 is characterized in that: The total cost function is optimized based on the preset QEMU algorithm to obtain the total cost of the target system, and the total cost of the target system is used to reflect the target movement strategy and target unloading strategy of the UAV, including: The total cost of the system is optimized using a distributed partially observable Markov decision process; In the optimization process, a reward function of the drone is set, and rewards and punishments are performed on the drone based on the reward function. The reward function is: Among them, η1 is a hyperparameter related to system cost, η2 is the collision constraint, η3 is the unloading failure constraint, η4 ​​is the no service constraint, η5 is the service compensation, C u (t) is the total system cost, is the number of drones within the drone safety range, is the number of tasks that failed to be unloaded on the drone, |U| is the number of drones, is the number of UEs that offload tasks to drones in time slot t, ε u (t) is the number of tasks in the task queue of drone u in the current time period t.

6. A drone-assisted unloading strategy generation device, characterized in that: The device comprises: An edge computing network model construction module is used to construct a multi-drone assisted edge computing network model and determine the computing delay information and energy consumption information generated by the drones in the edge computing network model during the task computing process; A total cost function determination module, used to determine the total cost function of the task calculation process based on the calculation delay information and the energy consumption information; The target system total cost determination module is used to optimize the total cost function based on a preset QEMU algorithm to obtain the target system total cost, and the target system total cost is used to reflect the target movement strategy and target unloading strategy of the UAV, including: Based on the preset QEMU algorithm, the optimization process is divided into a pre-training phase and an exploration phase; In the pre-training stage, an expert strategy algorithm is used to generate expert demonstration experience samples, and a loss function of the behavior clone of the expert demonstration experience samples is determined. The value network is updated based on the loss function. The expert strategy algorithm includes a preset greedy sailfish algorithm, which is obtained by combining a greedy algorithm and a sailfish algorithm. The greedy algorithm is used to guide flight direction decisions, and the sailfish algorithm is used to optimize unloading rate and flight speed decisions. The loss function of the behavior clone is: L BC (θ u )=E[(μ u (o u (t))-a u (t)) 2 ], where E is the number of pre-training rounds, μ u (.) is the actor network, o u (t) is the local observation of UAV u at time t, a u (t) is the action of UAV u at time t; In the exploration phase, a greedy exploration strategy is adopted to perform autonomous exploration to obtain the target movement strategy.

7. A terminal, characterized in that: The terminal includes a memory, a processor, and a drone-assisted unloading policy generation program stored in the memory and executable on the processor. When the processor executes the drone-assisted unloading policy generation program, the steps of the drone-assisted unloading policy generation method as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a drone-assisted unloading strategy generation program. When the drone-assisted unloading strategy generation program is executed by the processor, the steps of the drone-assisted unloading strategy generation method according to any one of claims 1 to 5 are implemented.

Images