A method for multi-UAV task offloading and resource allocation in edge computing system
Through deep deterministic policy gradient optimization neural network optimization, the problem of difficult coverage of areas for multiple drone networks is solved, the service fairness and energy consumption of IoT devices are minimized, and network quality is improved .
Patent Information
- Application Number
- CN202110657342.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-11
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-06-11
AI Technical Summary
In remote areas or disaster relief scenarios, multiple drone networks are difficult to continuously cover the entire area, resulting in ground equipment service fairness and network life problems. Traditional edge computing systems are poor in flexibility and difficult to dynamically adjust.
Deep deterministic strategy gradient optimization neural network is used to optimize drone trajectory, offloading strategies and computing resource allocation strategies, and obtain the optimal strategy by training neural networks to ensure fairness in IoT service and minimize system energy consumption.
It effectively reduces system costs, improves the quality of wireless network services, overcomes the dimensional limitations of traditional deep reinforcement learning methods, accelerates the optimization convergence process, and realizes the trade-off between service fairness and energy consumption.
Smart Images

Figure CN113395654B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of Internet of Things computing offloading and resource allocation, and more specifically, to a method for multi-UAV task offloading and resource allocation of an edge computing system. Background Art
[0002] With the development of the Internet of Things, emerging diversified mobile applications such as augmented reality, face recognition, mobile online games, virtual reality and other emerging technologies are constantly developing, and the requirements for delay processing and computational offloading are also increasing. However, the computing processing power of mobile devices is insufficient and the battery capacity is limited, making it difficult to meet the requirements of low latency and low energy consumption. Modern industrial Internet of Things smart devices are often limited in computing power due to their small form factor and strict production cost constraints. However, since smart devices often generate some computing tasks that need to be processed at a certain frequency, if they are not processed in time, the tasks will be discarded, resulting in irreversible consequences. Therefore, a method is needed to solve this problem.
[0003] Mobile edge computing has been recognized as one of the promising technologies to address this challenge. It provides cloud computing services to edge networks, and mobile users can offload their computing tasks to the edge network. Unlike mobile cloud computing, network edge devices in mobile edge computing can perform cloud-like computing and are deployed near users. It can help users save energy and provide low-latency services to ensure the security of mobile applications. However, mobile edge computing deployed in fixed locations has poor flexibility and is difficult to adjust dynamically. While drone networks have the advantages of easy deployment, strong flexibility and wide application range, they also face some challenges, such as their very limited battery capacity and relatively weak computing power.
[0004] In recent years, with the reduction of drone costs and the improvement of their capabilities, drones not only play an important role in the military battlefield, but are also increasingly widely used in the civilian field, mainly involving surveillance and monitoring, disaster relief and emergency response, precision agriculture, and network coverage. Therefore, the deployment of drone-based mobile edge computing is more flexible and rapid, which makes them particularly suitable for certain specific application requirements or responding to unexpected events. Moreover, drones have high-speed mobility, so drones can dynamically and close to the mobile devices they need to serve, significantly improving the throughput and communication capabilities of the network. Secondly, compared with the ground-to-ground link on the ground, the air-to-ground link in the drone network has the characteristics of line-of-sight transmission, so it can provide higher link capacity. Therefore, drone-based mobile edge computing is one of the current research focuses.
[0005] Artificial intelligence has been one of the hottest research directions in recent years. Combining it with edge computing based on drones can give this architecture great capabilities. Compared with ordinary edge computing, edge intelligent computing can effectively solve problems such as connection strategies and resource management in drone edge computing by using intelligent methods, and can also provide computing services for intelligent applications run by ground users or drone users. The edge intelligent computing based on drones is equipped with high-performance chips and artificial intelligence algorithms such as deep reinforcement learning on drones, which enables them to have good decision-making ability while having computing power. The drone can make the best decision to execute the next action based on environmental information such as the current channel status, ground node distribution status and the drone's own position. At the same time, it takes advantage of its own strong mobility to provide computing offloading services for multiple user devices in an area more quickly and flexibly. For example, Chinese patent publication number: CN111585637A, publication date: 2020-08-25, discloses a drone task offloading and resource allocation method based on an edge computing system, the edge computing system includes: a plurality of drones and a plurality of Fog nodes; the drone task offloading and resource allocation method includes the following steps: constructing an objective function for optimizing the benefits of the drone according to the problem of unloading tasks from drones to Fog nodes in the edge computing system and the problem of allocating computing resources from Fog nodes to drones; calculating the objective function to obtain the optimal solution of drone task offloading and computing resource allocation, and according to the optimal solution of drone task offloading and computing resource allocation, obtaining the optimization plan of drone benefits.
[0006] However, in some remote areas or disaster relief scenarios, due to the high cost and limited number of drones, it is not realistic to have enough drones to cover all areas all the time. Therefore, drones must keep flying to ensure that all areas are covered as much as possible within a reasonable time. Therefore, it is very important to ensure the fairness of ground equipment services, thereby extending the life of the entire network and improving the network's service quality. Summary of the invention
[0007] In order to solve the above-mentioned problems, the present invention provides a method for multi-UAV task offloading and resource allocation of an edge computing system, which optimizes the trajectory, offloading strategy and computing resource allocation strategy of the UAV through a deterministic policy gradient optimization neural network, thereby minimizing the system energy consumption while ensuring the fairness of the service of the IoT smart devices.
[0008] In order to achieve the above-mentioned purpose of the present invention, the technical scheme adopted is as follows:
[0009] A method for multi-UAV task offloading and resource allocation of an edge computing system, the method comprising the following steps:
[0010] S1: Formulate system state S t, the current position of the drone, the relative distance of the drone, the relative distance between the drone and the IoT smart device, and the service times of the IoT smart device are combined into the system state S t ;
[0011] S2: Build a deep deterministic policy gradient optimization neural network and initialize the neural network parameters including weights, biases, learning rates, and the number of neural network layers;
[0012] S3: Set the system state S t Input into the deep deterministic policy gradient optimization neural network for training to obtain system actions, wherein the system actions include the unloading decision of the IoT smart device, the flight direction of the drone, the flight distance of the drone, and the computing resources allocated by the drone to the corresponding IoT smart device;
[0013] S4: The edge computing system performs corresponding actions according to the system action obtained in step S3, and obtains the reward value of the system action according to the instant reward function;
[0014] S5: Continue to train the parameters of the deep deterministic policy gradient optimization neural network according to the obtained reward value until the reward value tends to be stable, thereby completing the training of the deep deterministic policy gradient optimization neural network and obtaining the optimal strategy π through training. The strategy π includes the optimal drone trajectory, unloading plan and computing resource allocation plan.
[0015] Preferably, the deep deterministic policy gradient optimization neural network includes an action neural network and a critic neural network;
[0016] The action neural network transforms the system state S t As input, output action A t , we get a strategy π defined as a mapping π: S t →A t ; Wherein, t represents the time slot;
[0017] For any action A t , a critic neural network is used to evaluate the expected long-term reward to guide action A t To develop in a better direction, it is defined as a mapping Q(S t ,A)→R.
[0018] Furthermore, at the decision time slot t, the action neural network receives the system state S t , and generates action A t , and get the next network state S t+1 ;
[0019] The instantaneous reward function R of the Markov process tEquivalent to taking an action from the current state S t Arriving at the next state S t+1 the price paid;
[0020] Then, (S t ,A t ,R t ,S t+1 ) is stored in the memory; in each training stage, samples are selected from the memory to learn and update the parameters of the action neural network and the critic neural network to maximize the long-term reward, that is, the optimal objective function, which is defined as maxE π,s [R(S t+1 |S t ,π(A t ))].
[0021] Furthermore, for the action neural network, samples Ω = {(S t , A t )|t∈Γ}, where Γ represents the sample index; the Adam optimizer in machine learning is used to minimize the cost function L(θ u ) direction to update the parameters θ of the action neural network u ,in, u(S t ) represents the current action neural network input S t the output produced;
[0022] By optimizing the minimization of the cost function L(θ u ) to update the parameters θ of the action neural network u , learn the optimal strategy π, and thus complete the training process.
[0023] Furthermore, for the critic neural network, the sample obtained from the memory is represented as Ω Γ ={(S t ,A t ,R t ,S t+1 )|t∈Γ}, where Γ represents the sample index; define the loss function, L(θ Q )=E Γ (R t +γmaxQ(S t ',A t ')-Q(S t ,A t )) 2 , where γ is the discount factor;
[0024] By optimizing the loss function L(θ Q) to update the parameters θ of the critic neural network Q , learn the optimal strategy π, and thus complete the training process.
[0025] Furthermore, we define the system state S t and action A t They are
[0026] S t ∈{(X k [n],Y k [n],H),d k,k' [n],d k,m [n],u m [n], k∈K, m∈M}
[0027]
[0028] In the formula, (X k [n],Y k [n],H) represents the current position of the drone; d k,k' [n] represents the relative distance between two drones at the nth time slot; d k,m [n] represents the relative distance between drone k and IoT smart device m at the nth time slot; u m [n] represents the proportion of times IoT smart device m is served in the nth time slot; s m,k [n] represents the binary offloading decision of IoT smart device m to drone k in the nth time slot, s m,k [n]∈{0,1}, when k=0,s m,k [n] = 0 means to choose to perform calculations locally; k ≠ 0, s m,k [n] = 1 means that the whole process is offloaded to UAV k for processing, otherwise s m,k [n] = 0;
[0029] d k [n] represents the flight distance of UAV k, represents the flight direction of UAV k, f k,m [n] represents the computing resources allocated by drone k to the IoT smart device m that is selected for offloading.
[0030] Furthermore, the instant reward function calculates the reward value, and the calculation formula is as follows:
[0031]
[0032] Where δ represents the fairness index, T[n] represents the time delay of the complete unloading calculation process of a time slot; E[n] represents the system energy consumption of the complete unloading calculation process of a time slot; μ1 represents the weight factor of time, and μ2 represents the weight factor of energy consumption;
[0033] in, δ∈(0,1), the larger the δ, the better the fairness of the service, and the proportion of times each IoT smart device is served T represents the longest time slot length of the drone service, T m Indicates the number of times each IoT smart device is served within T time slot lengths.
[0034] Furthermore, the time delay T[n] of the complete unloading calculation process of a time slot is calculated as follows:
[0035]
[0036] Among them, the binary uninstall variable s m,k [n]∈{0,1} represents the unloading decision of the nth time slot, when k=0,s m,k [n] = 0 means to choose to perform the calculation locally; when k≠0, s m,k [n] = 1 means that the whole calculation is offloaded to UAV k for execution, otherwise s m,k [n] = 0;
[0037] represents the execution time delay of the computing task of IoT smart device m when it is executed locally, represents the computing resources executed locally by IoT smart device m in the nth time slot, Z m [n] represents the computing resources required by the IoT smart device m to perform the task in the nth time slot;
[0038] Indicates the delay in the unloading processing time when the IoT smart device m chooses to offload the computing task to the drone k. The unloading processing time delay includes the delay time caused by the IoT smart device m uploading the task to the drone The time of performing the mission on UAV k Among them, R m,k [n] represents the upload rate of IoT smart device m uploading data to drone k in the nth time slot; f k,m [n] represents the computing resources allocated to ground device m by UAV k in the nth time slot; H m [n] represents the task data size of IoT smart device m in the nth time slot;
[0039] The upload rate R m,k [n]The calculation formula is as follows:
[0040]
[0041] In the formula, p mis the transmission power of device m when uploading computing task data to the drone,
[0042] is the distance from drone k to IoT smart device m;
[0043] [X k [n],Y k [n],H] represents the geographical location of the drone, W m =[x m ,y m ] represents the geographical location of the IoT smart device; β represents the channel gain per unit distance.
[0044] Furthermore, the system energy consumption E[n] of the complete unloading calculation process of a time slot is calculated as follows:
[0045]
[0046] In the formula, represents the energy consumption of IoT smart device m during local computing, represents the transmission energy consumption of IoT smart device m uploading data to drone k in the nth time slot; represents the computing energy consumption of the drone; represents the flight energy consumption of the UAV in time slot n;
[0047] in, γ m Indicates the chip capacitance parameter of IoT smart device m;
[0048] p m represents the transmission power of IoT smart device m;
[0049] γ k represents the chip capacitance parameter of drone k;
[0050]
[0051] Since the computing resources allocated by the drone to the IoT smart devices selected for processing cannot exceed its own limitations, i.e. Among them, F k represents the total number of computing resources of UAV k.
[0052] Furthermore, the optimal objective function obtained through training is
[0053]
[0054]
[0055] C2:
[0056] C3:d k,k' [n]≥d min ,k∈K,n∈N
[0057] C4:
[0058] C5:
[0059] C6:μ1+μ2=1
[0060] Among them, C1 means that the drone cannot fly out of its service area; C2 means the flight distance and flight direction constraints of the drone; C3 means the minimum distance constraint between two drones; C4 means that an IoT smart device can only be connected to one drone; C5 means that the computing resources allocated to the IoT smart device do not exceed the resource limit of the drone server itself; C6 means the weight constraint of energy consumption and time delay.
[0061] The beneficial effects of the present invention are as follows:
[0062] The present invention applies deep deterministic policy gradient optimization neural network to the technical field of multi-UAV computing offloading and resource allocation. UAVs can learn the optimal trajectory, offloading and resource allocation strategy, which can effectively reduce the cost of the entire system and improve the quality of wireless network services.
[0063] In the optimization problem of the present invention, the deep deterministic policy gradient optimization neural network used can effectively overcome the dimensionality limitation of the state behavior space of the traditional deep reinforcement learning method, accelerate the convergence process of the network, and make the optimization more efficient.
[0064] At the same time, this embodiment takes into account the service fairness of ground IoT smart devices, designs a service fairness index in the immediate reward function, and forms a fractional proportional relationship with the trade-off between system energy consumption and time delay. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 This is the multi-drone assisted IoT smart device computing offloading and resource allocation scenario diagram described in Example 1.
[0066] Figure 2 It is a flowchart of the method for multi-UAV task offloading and resource allocation of the edge computing system described in Example 1.
[0067] Figure 3 It is a framework diagram of the deep deterministic policy gradient optimization neural network described in Example 1. DETAILED DESCRIPTION
[0068] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0069] Example 1
[0070] like Figure 1 As shown. This is a scenario diagram of multi-drone assisted IoT smart device computing offloading and resource allocation. In the figure, M IoT smart devices are distributed in the area. K drones are used to assist ground IoT smart devices in processing computing tasks. The computing tasks generated by IoT smart devices can be calculated locally and offloaded to drones with powerful computing and storage functions for processing. Considering that the coverage and communication range of a single drone are limited, this embodiment considers the scenario of multiple drones. Considering that IoT smart devices are sensitive to the time delay of computing tasks and the energy consumption of the entire edge computing system, this embodiment takes computing time delay and system energy consumption as the main optimization goals. The system energy consumption mainly includes local computing energy consumption of IoT smart devices, offloading transmission energy consumption, and computing energy consumption and flight energy consumption of drones. Because the number of drones is limited, they need to fly continuously to cover all IoT smart devices. Therefore, this embodiment also considers the service fairness of IoT smart devices. Therefore, it is necessary to optimize the trajectory of drones to achieve service fairness. Here, a deep deterministic policy gradient optimization neural network is used to learn the optimal action decision. The above two optimization goals are constructed as a reinforcement learning reward function to maximize the long-term return of reinforcement learning, that is, to maximize our optimization objective function.
[0071] Specifically, this embodiment provides a method for multi-UAV task offloading and resource allocation of an edge computing system, wherein the method comprises the following steps:
[0072] S1: Formulate system state S t , the current position of the drone, the relative distance of the drone, the relative distance between the drone and the IoT smart device, and the service times of the IoT smart device are combined into the system state S t ;
[0073] S2: Build a deep deterministic policy gradient optimization neural network and initialize the neural network parameters including weights, biases, learning rates, and the number of neural network layers;
[0074] S3: Set the system state S t Input into the deep deterministic policy gradient optimization neural network for training to obtain system actions, wherein the system actions include the unloading decision of the IoT smart device, the flight direction of the drone, the flight distance of the drone, and the computing resources allocated by the drone to the corresponding IoT smart device;
[0075] S4: The edge computing system performs corresponding actions according to the system action obtained in step S3, and obtains the reward value of the system action according to the instant reward function;
[0076] S5: Continue to train the parameters of the deep deterministic policy gradient optimization neural network according to the obtained reward value until the reward value tends to be stable, thereby completing the training of the deep deterministic policy gradient optimization neural network and obtaining the optimal strategy π through training. The strategy π includes the optimal drone trajectory, unloading plan and computing resource allocation plan.
[0077] In a specific embodiment, since the traditional Q-learning and DQN methods are complex in dimension and are not suitable for the resource allocation problem of control and computational offloading of drones, the deep deterministic policy gradient optimization neural network (DDPG) used here has solved many challenging problems in various fields with large action spaces. The deep deterministic policy gradient optimization neural network includes an action neural network and a critic neural network;
[0078] In the action neural network, instead of outputting the expected reward or the probability of all possible actions, we have a policy π; the action neural network transforms the system state S t As input, output action A t , we get a strategy π defined as a mapping π: S t →A t ; Wherein, t represents the time slot;
[0079] For any action A t , a critic neural network is used to evaluate the expected long-term reward to guide action A t To develop in a better direction, it is defined as a mapping Q(S t ,A)→R.
[0080] In the edge computing system, the edge computing system senses the network status of drones and IoT smart devices and integrates them into the system status S t The deep deterministic policy gradient optimization neural network is triggered periodically, and at the decision time slot t, the action neural network receives the system state S t , and generates action A t , and get the next network state S t+1 .
[0081] The instantaneous reward function of a Markov process is equal to taking an action from the current state S t Arriving at the next state S t+1 The reward R t ;
[0082] Then, (S t,A t ,R t ,S t+1 ) is stored in the memory; in each training stage, samples are selected from the memory to learn and update the parameters of the action neural network and the critic neural network to maximize the long-term reward, that is, the optimal objective function, which is defined as maxE π,s [R(S t+1 |S t ,π(A t ))].
[0083] In a specific embodiment, for an action neural network, a sample Ω={(S t , A t )|t∈Γ}, where Γ represents the sample index; the Adam optimizer in machine learning is used to minimize the cost function L(θ u ) direction to update the parameters θ of the action neural network u ,in, u(S t ) represents the current action neural network input S t the output produced;
[0084] By optimizing the minimization of the cost function L(θ u ) to update the parameters θ of the action neural network u , learn the optimal strategy π, and thus complete the training process.
[0085] In a specific embodiment, for the critic neural network, the sample obtained from the memory is represented as Ω Γ ={(S t ,A t ,R t ,S t+1 )|t∈Γ}, where Γ represents the sample index; define the loss function, L(θ Q )=E Γ (R t +γmaxQ(S t ',A t ')-Q(S t ,A t )) 2 , where γ is the discount factor;
[0086] By optimizing the loss function L(θ Q ) to update the parameters θ of the critic neural network Q , learn the optimal strategy π, and thus complete the training process.
[0087] This embodiment updates the parameters of the two networks by optimizing the two functions, learns the optimal strategy, and completes the training process. According to the final system action, the trajectory planning and computation offloading of the UAV and the allocation of its resources are performed.
[0088] In a specific embodiment, the system state S is defined as t and action A t They are
[0089] S t ∈{(X k [n],Y k [n],H),d k,k' [n],d k,m [n],u m [n], k∈K, m∈M}
[0090]
[0091] In the formula, (X k [n],Y k [n],H) represents the current position of the drone; d k,k' [n] represents the relative distance between two drones at the nth time slot; d k,m [n] represents the relative distance between drone k and IoT smart device m at the nth time slot; u m [n] represents the proportion of times IoT smart device m is served in the nth time slot; s m,k [n] represents the binary offloading decision of IoT smart device m to drone k in the nth time slot, s m,k [n]∈{0,1}, when k=0,s m,k [n] = 0 means to choose to perform calculations locally; k ≠ 0, s m,k [n] = 1 means that the whole process is offloaded to UAV k for processing, otherwise s m,k [n] = 0;
[0092] d k [n] represents the flight distance of UAV k, represents the flight direction of UAV k, f k,m [n] represents the computing resources allocated by drone k to the IoT smart device m that is selected for offloading.
[0093] In a specific embodiment, one of our optimization goals is to minimize the total energy consumption and computational time delay of IoT smart devices and drones by optimizing offloading decisions and drone trajectories. However, this may lead to an unfair process, so in order to avoid the situation where some IoT smart devices are served in many time slots while other IoT smart devices are not served at all, a fairness indicator is defined.
[0094] Assume that there are M IoT smart devices on the ground, define the service ratio of each IoT smart device Where T represents the longest time slot length of the drone service, T m represents the number of times each IoT smart device is served within T time slots. Our goal is to make u m However, there will be an unfair situation that in most or even all time slots, a certain IoT smart device is served, while the rest are not. Therefore, we need to solve the problem of fairness of service.
[0095] Here we define a fairness index. Given an unloading control strategy, the corresponding fairness index is: δ∈(0,1), the larger δ is, the better the fairness of the service is.
[0096] The instant reward function is calculated based on the fairness index δ to obtain the reward value, and the calculation formula is as follows:
[0097]
[0098] Where δ represents the fairness index, T[n] represents the time delay of the complete unloading calculation process of a time slot; E[n] represents the system energy consumption of the complete unloading calculation process of a time slot; μ1 represents the weight factor of time, and μ2 represents the weight factor of energy consumption;
[0099] In a specific embodiment, in the scenario we consider, there are K drones providing computing offloading services for M IoT smart devices on the ground. Since the computing and storage capabilities of IoT smart devices are limited, IoT smart devices can either process computing tasks locally or offload them to drones for processing. Assume that each IoT smart device on the ground will generate a simple computing task information, defined as S m ={H m ,Z m The IoT smart device first sends brief information about the task to the drone, and the drone makes a decision to choose whether to perform calculations locally or offload them to the selected drone for processing. m represents the task data size of IoT smart device m, Z m It represents the computing resources required by IoT smart device m to perform the task.
[0100] Define k∈{0,1,2...,K}. This embodiment uses binary unloading variable s m,k [n]∈{0,1} represents the unloading decision of the nth time slot. m,k[n] = 0 means to choose to perform the calculation locally. k≠0,s m,k [n] = 1 means that the whole process is offloaded to UAV k for processing, otherwise s m,k [n]=0.
[0101] In a specific embodiment, the geographical location of drone k is defined as: k [n],Y k [n],H], where H is the fixed height of UAV k. Define d k [n] is the flight distance of UAV k in the nth time slot, is the direction (angle) of UAV k’s flight in the nth time slot, is the speed of UAV k in the nth time slot, T d is the time slot length.
[0102] To ensure that the distance between two adjacent drones is not less than the minimum distance,
[0103]
[0104] In a specific embodiment, assume that the geographical location of drone k is [X k [n],Y k [n],H], the geographical location of the IoT smart device is W m =[x m ,y m ]. Assuming that the simultaneous unloading of IoT smart devices has no impact on the channel bandwidth, the upload rate R of IoT smart device m uploading data to drone k in the nth time slot is m,k [n] is:
[0105]
[0106] Where B represents the wireless channel bandwidth, p m is the transmission power when the IoT smart device m uploads computing task data to the drone, is the distance from drone k to IoT smart device m; β represents the channel gain per unit distance.
[0107] In a specific embodiment, a local computing model is constructed. If the IoT smart device chooses to execute task S locally m ={H m ,Z m}, then the execution time of the computing task of IoT smart device m in local execution is The calculation formula is as follows:
[0108]
[0109] In the formula, represents the computing resources executed locally by IoT smart device m in the nth time slot, Z m [n] represents the computing resources required for the IoT smart device m to perform the task in the nth time slot.
[0110] The energy consumption of the IoT smart device m during local computing The calculation formula is as follows:
[0111]
[0112] In the formula, γ m Represents the chip capacitance parameters of the IoT smart device m.
[0113] In a specific embodiment, a computational model is constructed to offload tasks to a drone.
[0114] In the nth time slot, if the IoT smart device m chooses to offload the computing task to the drone k for processing, the entire offloading process includes three steps. First, the IoT smart device m uploads the computing task data to the drone k; then the drone k allocates computing resources to perform the task; and finally, the drone k returns the received results to the IoT smart device m on the ground.
[0115] Since the return rate of the wireless network is generally much higher than the upload rate, and the returned data is much smaller than the uploaded data, the return time delay and energy consumption are ignored here. Then the offloading processing time delay is mainly composed of two parts. The first part is the delay caused by the IoT smart device m uploading the task to the drone. R m,k [n] represents the upload rate of IoT smart device m uploading data to drone k in the nth time slot, H m [n] represents the amount of task data of IoT smart device m in the nth time slot; the second part is the time to execute the task on drone k Among them, f k,m [n] represents the computing resources allocated to ground device m by UAV k in the nth time slot.
[0116] The IoT smart device m chooses to offload the computing task to the drone k for processing. The system energy consumption generated by the offloading is also mainly composed of three parts. The first part is the transmission energy consumption of the IoT smart device m uploading data to the drone k in the nth time slot. The second part is to show the computing energy consumption of the drone, where p m represents the transmission power of IoT smart device m; Represents the flight energy consumption of the drone in n time slots where γk represents the chip capacitance parameter of drone k; the third part is the flight energy consumption of the drone in time slot n:
[0117]
[0118] Since the computing resources allocated by drone k to the selected processing device cannot exceed its own limit, that is, F k represents the total number of computing resources of UAV k.
[0119] According to the above analysis, we can know that the time delay T[n] for the complete unloading calculation process of a time slot is expressed as follows:
[0120]
[0121] The system energy consumption E[n] of a complete unloading calculation process in a time slot is expressed as follows:
[0122]
[0123] In a specific embodiment, until the reward value becomes stable and the training of the neural network is completed, according to the task requested by the IoT smart device, under the constraints of the drone computing resources and various physical conditions, the optimized objective function obtained through training is:
[0124]
[0125]
[0126] C2:
[0127] C3:d k,k' [n]≥d min ,k∈K,n∈N
[0128] C4:
[0129] C5:
[0130] C6:μ1+μ2=1
[0131] Among them, C1 means that the position of the drone cannot fly out of the service area; C2 means the flight distance and flight direction constraints of the drone; C3 means the minimum distance constraint between two drones; C4 means that an IoT smart device can only be connected to one drone; C5 means that the computing resources allocated to the IoT smart device do not exceed the resource limitations of the drone server itself; C6 means the weight constraints of energy consumption and time delay.
[0132] This will lead to the best drone trajectory, offloading scheme and computing resource allocation scheme, which can improve the service fairness of IoT smart devices and the performance of wireless networks while reducing energy consumption and computing time.
[0133] Obviously, the above embodiments of the present invention are only examples for clearly explaining the present invention, and are not intended to limit the implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.
Claims
1. A method for multi-UAV task offloading and resource allocation in an edge computing system, characterized in that: The method comprises the following steps: S1: Formulate system state S t , the current position of the drone, the relative distance of the drone, the relative distance between the drone and the IoT smart device, and the service times of the IoT smart device are combined into the system state S t ; S2: Build a deep deterministic policy gradient optimization neural network and initialize the neural network parameters including weights, biases, learning rates, and the number of neural network layers; S3: Set the system state S t Input into the deep deterministic policy gradient optimization neural network for training to obtain system actions, wherein the system actions include the unloading decision of the IoT smart device, the flight direction of the drone, the flight distance of the drone, and the computing resources allocated by the drone to the corresponding IoT smart device; S4: The edge computing system performs corresponding actions according to the system action obtained in step S3, and obtains the reward value of the system action according to the instant reward function; S5: Continue to train the parameters of the deep deterministic policy gradient optimization neural network according to the obtained reward value until the reward value tends to be stable, thereby completing the training of the deep deterministic policy gradient optimization neural network and obtaining the optimal strategy π through training. The strategy π includes the optimal drone trajectory, unloading plan and computing resource allocation plan.
2. The method for multi-UAV task offloading and resource allocation of the edge computing system according to claim 1, characterized in that: The deep deterministic policy gradient optimization neural network includes an action neural network and a critic neural network; The action neural network transforms the system state S t As input, output action A t , we get a strategy π defined as a mapping π: S t →A t ; Wherein, t represents the time slot; For any action A t , a critic neural network is used to evaluate the expected long-term reward to guide action A t To develop in a better direction, it is defined as a mapping Q(S t ,A)→R; where R represents the immediate reward function.
3. The method for multi-UAV task offloading and resource allocation of the edge computing system according to claim 2, characterized in that: At decision time slot t, the action neural network receives the system state S t , and generates action A t , and get the next network state S t+1 ; The instantaneous reward function R of the Markov process t Equivalent to taking an action from the current state S t Arriving at the next state S t+1 the price paid; Then, (S t ,A t ,R t ,S t+1 ) is stored in the memory; in each training stage, samples are selected from the memory to learn and update the parameters of the action neural network and the critic neural network to maximize the long-term reward, that is, the optimal objective function, which is defined as maxE π,s [R(S t+1 |S t ,π(A t ))], which is used to represent the expected value of maximizing the long-term return R under the strategy π and the system state S.
4. The method for multi-UAV task offloading and resource allocation of the edge computing system according to claim 3, characterized in that: For the action neural network, samples Ω = {(S t , A t )|t∈Γ}, where Γ represents the sample index; the Adam optimizer in machine learning is used to minimize the cost function L(θ u ) direction to update the parameters θ of the action neural network u ,in, u(S t ) represents the current action neural network input S t The resulting output, E Γ represents the expected value over all possible sample indices Γ; By optimizing the minimization of the cost function L(θ u ) to update the parameters θ of the action neural network u , learn the optimal strategy π, and thus complete the training process.
5. The method for multi-UAV task offloading and resource allocation of the edge computing system according to claim 3, characterized in that: For the critic neural network, the samples taken from the memory are represented as Ω Γ ={(S t ,A t ,R t ,S t+1 )|t∈Γ}, where Γ represents the sample index; define the loss function, L(θ Q )=E Γ (R t +γmaxQ(S t ',A t ')-Q(S t ,A t )) 2 , where γ is the discount factor, E Γ represents the expected value over all possible sample indices Γ; By optimizing the loss function L(θ Q ) to update the parameters θ of the critic neural network Q , learn the optimal strategy π, and thus complete the training process.
6. The method for multi-UAV task offloading and resource allocation of an edge computing system according to any one of claims 4 or 5, characterized in that: Define the system state S t and action A t They are S t ∈{(X k [n],Y k [n],H),d k,k' [n],d k ,m[n],u m [n],k∈K,m∈M} In the formula, (X k [n],Y k [n],H) represents the current position of the drone; d k,k' [n] represents the relative distance between two drones at the nth time slot; d k,m [n] represents the relative distance between drone k and IoT smart device m at the nth time slot; u m [n] represents the proportion of times IoT smart device m is served in the nth time slot; s m,k [n] represents the binary offloading decision of IoT smart device m to drone k in the nth time slot, s m,k [n]∈{0,1}, when k=0,s m,k [n] = 0 means to choose to perform calculations locally; k ≠ 0, s m,k [n] = 1 means that the whole process is offloaded to UAV k for processing, otherwise s m,k [n] = 0; d k [n] represents the flight distance of UAV k, represents the flight direction of UAV k, f k,m [n] represents the computing resources allocated by drone k to the IoT smart device m that is selected for offloading; K represents the set of drones, and M represents the set of IoT smart devices.
7. The method for multi-UAV task offloading and resource allocation of the edge computing system according to claim 6, characterized in that: The reward value of the system action is obtained according to the instant reward function, and the calculation formula is as follows: Where δ represents the fairness index, T[n] represents the time delay of the complete unloading calculation process of a time slot; E[n] represents the system energy consumption of the complete unloading calculation process of a time slot; μ1 represents the weight factor of time, and μ2 represents the weight factor of energy consumption; in, The larger δ is, the better the fairness of the service is. The proportion of times each IoT smart device is served is T represents the longest time slot length of the drone service, T m Indicates the number of times each IoT smart device is served within T time slot lengths.
8. The method for multi-UAV task offloading and resource allocation of the edge computing system according to claim 7, characterized in that: The time delay T[n] of the complete unloading calculation process of a time slot is calculated as follows: represents the execution time delay of the computing task of IoT smart device m when it is executed locally, represents the computing resources executed locally by IoT smart device m in the nth time slot, Z m [n] represents the computing resources required by the IoT smart device m to perform the task in the nth time slot; Indicates the delay in the unloading processing time when the IoT smart device m chooses to offload the computing task to the drone k. The unloading processing time delay includes the delay time caused by the IoT smart device m uploading the task to the drone The time of performing the mission on UAV k Among them, R m,k [n] represents the upload rate of IoT smart device m uploading data to drone k in the nth time slot; f k,m [n] represents the computing resources allocated to ground device m by UAV k in the nth time slot; H m [n] represents the task data size of IoT smart device m in the nth time slot; The upload rate R m,k [n]The calculation formula is as follows: In the formula, p m is the transmission power of device m when uploading computing task data to the drone, is the distance from drone k to IoT smart device m; [X k [n],Y k [n],H] represents the geographical location of the drone, W m =[x m ,y m ] represents the geographical location of the IoT smart device; β represents the channel gain per unit distance; H is the fixed height of the drone k, and B represents the wireless channel bandwidth.
9. The method for multi-UAV task offloading and resource allocation of the edge computing system according to claim 8, characterized in that: The system energy consumption E[n] of a complete unloading calculation process in a time slot is calculated as follows: In the formula, represents the energy consumption of IoT smart device m during local computing, represents the transmission energy consumption of IoT smart device m uploading data to drone k in the nth time slot; represents the computing energy consumption of the drone; represents the flight energy consumption of the UAV in time slot n; in, γ m Indicates the chip capacitance parameter of IoT smart device m; p m represents the transmission power of IoT smart device m; γ k Represents the chip capacitance parameter of drone k.
10. The method for multi-UAV task offloading and resource allocation of the edge computing system according to claim 9, characterized in that: The optimal objective function obtained through training is: C3:d k,k' [n]≥d min ,k∈K,n∈N C6:μ1+μ2=1 Among them, C1 indicates that the position of the drone cannot fly out of the service area; C2 indicates the flight distance and flight direction constraints of the drone; C3 indicates the minimum distance constraint between two drones; C4 indicates that an IoT smart device can only be connected to one drone; C5 indicates that the computing resources allocated to the IoT smart device do not exceed the resource limitations of the drone server itself; C6 indicates the weight constraints of energy consumption and time delay; N indicates the time slot set.
