MAPPO-Att-based multi-unmanned aerial vehicle edge network task unloading and resource allocation method

Abstract

The invention provides a multi-unmanned aerial vehicle edge network task unloading and resource allocation method based on MAPPO-Att, and relates to the technical field of unmanned aerial vehicle resource allocation. The method comprises the following steps: firstly, constructing a multi-layer heterogeneous edge computing network model, and computing average path loss between an unmanned aerial vehicle group and Internet of Things equipment; then designing an Internet of Things task unloading model, and calculating the total delay cost of the unloading task of the Internet of Things equipment; designing an energy cost and cost target optimization function model of the unloading task of the Internet of Things equipment; an MAPPO-Att algorithm is adopted to determine an optimal selection strategy of task unloading and resource allocation, the task unloading and resource allocation problems in an A2G network are solved, and the optimal strategy is searched to maximize long-term return, so that the calculation cost and effective resource allocation are reduced, and the overall training cost is reduced through a centralized training and decentralized execution technology; and completing unmanned aerial vehicle cluster resource allocation simulation until the task completion rate reaches a set value.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle resource allocation, and in particular to a multi-unmanned aerial vehicle edge network task unloading and resource allocation method based on MAPPO-Att. Background Art

[0002] The current fifth generation (5G) and beyond 5G (B5G) technologies are designed to improve the quality of service (QoS) and increase the productivity and network capacity of Internet of Things (IoT) networks. This will reduce the cost of computing by using current network technologies and machine learning methods. The integration of IoT and machine learning is a promising paradigm that will accelerate future network technologies and increase the reusability of IoT in different fields. In the current 5G and B5G network technologies, the number of ultra-dense heterogeneous IoT devices at the edge network is growing exponentially. IoT devices generate a lot of traffic, and they cannot process all the data by themselves due to lack of computing resources, battery life, and network coverage.

[0003] Mobile Edge Computing (MEC) is a promising model that can improve the QoS and network performance of edge devices better than cloud computing infrastructure. IoT devices can offload computationally intensive data to the nearest MEC with ultra-low latency (ULL) and low power consumption.

[0004] However, due to the mobility of user equipment (UE), MEC base stations (BS) with computing servers may be overloaded by handling large amounts of traffic from IoT devices or damaged due to inevitable natural disasters or temporary failures. Therefore, traditional MEC technologies are not suitable for guaranteeing the required QoS, network availability, and scalability. Therefore, recent drone technology has become part of MEC in cellular communication environments to assist terrestrial networks, acting as macro base stations (MBS) or aerial base stations (ABS).

[0005] The multi-agent deep reinforcement learning (MADRL) approach can solve the current real-world (multi-agent environment) IoT edge network problems. It provides distributed resource management and resource allocation for computation offloading in IoT edge networks; each IoT device can take actions based on the local information it observes. MADRL has been applied to different problems in traditional MEC networks, such as handling risk-aware energy, channel access, and computation offloading. In drone networks, the MADRL approach has been applied to optimize simultaneous target assistance and path planning (STAP), dynamic resource allocation for ground users in drone networks. These works use stochastic countermeasures to formulate the problem and apply the MADRL algorithm to solve the formulated problem. However, these works mainly focus on MEC networks or drone networks, and do not consider network congestion and emergency situations in drone-assisted edge IoT network scenarios in air-to-ground (A2G) network environments. Multiple agents can collaborate based on their prior knowledge of system parameters to reduce computational costs and increase the probability of association (offloading) within computational nodes. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a multi-UAV edge network task unloading and resource allocation method based on MAPPO-Att in view of the deficiencies of the above-mentioned prior art, so as to realize task unloading and resource allocation of a multi-UAV edge network.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: a multi-UAV edge network task offloading and resource allocation method based on MAPPO-Att, comprising the following steps:

[0008] Step 1: Build a multi-layer heterogeneous edge computing network model based on the A2G network and calculate the average path loss between the drone swarm UCH and the IoT device IoT;

[0009] The IoT edge network is realized by deploying drone flying base stations; drone groups are organized into clusters, which complete tasks on the ground edge network; drones also act as aerial MEC base stations, providing different services for ground computing nodes;

[0010] A group of IoT devices is represented as l∈L={1,2,...,K}, a group of drones is represented as u∈U={1,2,...,Y}, and a group of MEC base stations is represented as n∈N={1,2,...,M}. When an IoT device chooses to offload tasks to a computing node on a UCH or MEC base station with access channel m, the set of active computing nodes in the MEC base station is represented as When IoT device l chooses to offload tasks to the MEC base station or UCH with access channel m, the uplink rate is expressed as:

[0011]

[0012] Among them, g lx is the channel gain from IoT device l to computing node x on channel m; p l is the transmission power of IoT device l, σ2 is the noise power, and B represents the channel bandwidth.

[0013] Each channel has two states in each time slot t, available or disconnected, and is represented as Used to indicate the state of channel m in time slot t; if Then the channel of BS or UCH is available and IoT devices can offload tasks; otherwise, the channel is unavailable, which means that the computing node is overloaded or fails; the number of IoT devices associated with one of the UCH or MEC base stations is limited, expressed as Where Q represents the maximum allowed number of IoT devices whose tasks are offloaded and associated with the UCH or MEC base station;

[0014] The IoT device will first determine the sub-channel status The value of determines whether the subchannel is available, and then selects a suitable computing node on the available subchannel for task offloading. Therefore, it is assumed that each IoT device can only use the allocated spectrum or channel m to select a computing node in any time slot t, and is expressed as:

[0015]

[0016] in, represents the task offloading decision variable; the task offloading decision variable is related to the computing energy consumption. Selecting different channels through the decision variable will affect the success probability of the entire link;

[0017] The communication link of the A2G network is modeled by a probabilistic path loss model, which includes line-of-sight LoS and non-line-of-sight NLoS with different occurrence probabilities; at time slot t, the LoS link probability P between UCH and IoT device l Los It is expressed as:

[0018]

[0019] Among them, ι and ρ are constants that depend on the environment, D lu (t) represents the distance between UCH and IoT device l, H u Indicates the height of UCH;

[0020] Then the NLoS link probability is calculated as P NLos (t) = 1-P LoS (t);

[0021] The path loss from UCH to the ground IoT device l is further calculated as follows:

[0022]

[0023] in, represents the path loss of the LoS link between UCH and IoT device l, η LoS represents the additional attenuation factor of the LoS link associated with the path, represents the path loss of the NLoS link between UCH and IoT device l, η NLoS represents the additional attenuation factor of the NLoS link associated with the path, represents the free space path loss, f is the frequency carrier, c is the speed of light;

[0024] Average path loss between UCH and IoT device l in time slot t It is expressed as:

[0025]

[0026] Step 2: Design an IoT task offloading model and calculate the total delay cost of offloading tasks of IoT devices.

[0027] Each IoT device has different forms of applications, and the size of the task is generated in time slot t; each computing task has three profiles: the size of the input task d l , the computing power required to process the task c l and the maximum time delay T l ;

[0028] When an IoT device decides to offload tasks to related computing nodes based on the current offloading strategy and information from other IoT devices, the latency cost is calculated in three stages: transmission time latency, execution time latency, and result sending latency.

[0029] Assume that the A2G network has a multi-channel access scheme in OFDMA technology; let W be a working frequency band that can be divided into equal sub-channels; each computing node can serve at most L IoT devices in time slot t; in each computing node, the set of available sub-channels is defined as m∈ω={1,2,3,...,M};

[0030] The task offloading decision variable at time slot t with uplink subchannel scheduling is defined as

[0031] When IoT device l offloads the input task to the computing node, IoT device l also calculates the transmission power of the uploaded task; the set of IoT devices that offload intensive tasks to the MEC base station or UCH is defined as: According to the current offloading policy at time slot t, the input task d l The transmission power of IoT device l uploaded to MEC base station or UCH is expressed as p l (t), where P represents the transmission power, represents the maximum value of the transmission power; in the process of transmitting the input task to the nearest computing node accessing different subchannels, the IoT device suffers from inter-cell interference; therefore, the signal interference plus noise ratio SINR from the IoT device l to the computing node x on the subchannel m is defined according to the following situation at time slot t;

[0032] The rate at which IoT device l offloads tasks to the MEC base station is calculated as:

[0033] r ln (t) = Blog2(1+γ ln ) (6)

[0034] In the formula, r ln (t) represents the rate at which IoT device l offloads tasks to the MEC base station, γ ln represents the signal-to-noise ratio of IoT device l at time slot t;

[0035] Channel gain between IoT device l and UCH Among them, g o is the channel power gain at the reference distance of 1m, H u is the height of UCH, D lu (t) represents the distance between the IoT device and the UCH or MEC base station;

[0036] IoT device l checks whether the drone group is within the network coverage before unloading the task. The specific judgment is: D lu ≤H u (t)tanθ u , where θ u Indicates the tilt angle of the IoT device;

[0037] Further calculation shows that the offload data rate of IoT device l offloading the input task to UCH is:

[0038]

[0039] In the formula, r ul (t) represents the transmission rate between IoT device l and user computing device u in time slot t, G0 is the gain coefficient, represents the power gain factor, p lu (t) is the transmission power of IoT device l in time slot t;

[0040] like Then the offloading transmission delay cost between IoT device l and UCH based on the current offloading strategy in time slot t is Calculated as:

[0041]

[0042] in, represents the task offloading decision variable, d l (t) represents the task data volume of IoT device l in time slot t;

[0043] The execution time delay of offloading from IoT device l to the computing node is calculated based on the current offloading strategy and the information of other IoT devices in time slot t, as well as the computing resource blocks and UCH capacity of the local MEC base station;

[0044] The computing node on each resource block of the local MEC base station is shared by all IoT devices in time slot t; and the computing resources are set to be evenly distributed to all IoT devices; then the total delay cost of the offloading task of IoT device l is expressed as:

[0045]

[0046] In the formula, represents the total delay cost of the IoT device l loading task, d l (t) represents the amount of task data of IoT device l in time slot t, c l (t) represents the computing resource requirement of IoT device l in time slot t, r nl (t) represents the transmission rate between IoT device l and MEC base station n in time slot t, f ln (t) represents the computing resource capacity of MEC base station n in time slot t, r ul (t) represents the transmission rate between IoT device l and UCH device u in time slot t, f lu (t) represents the computing resource capacity of UCH device u in time slot t;

[0047] Step 3: Calculate the energy consumption cost of IoT devices when offloading tasks to computing nodes at different stages, and design the energy cost and cost target optimization function model of IoT device offloading tasks;

[0048] First, each IoT device and UCH are set to use discrete transmission power control; the transmission power value of each IoT device is represented by the vector P l T ={p1,...,pz}, where p z represents the transmission power allocated to the zth subchannel; at time slot t, each IoT device selects its transmission power as Indicates that IoT device l decides to offload the input task, otherwise Where Z represents different communication options;

[0049] Based on the power constraint and the current offloading strategy, the IoT device selects the transmission power from the UCH or selects the local task execution; the power value of each UCH communicating with the selected IoT device l is expressed as P u T ={p1,...,p z}, the binary value of the power value is expressed as

[0050] The IoT device l or UCH selects and allocates the power level in each time slot t. The limit of the total transmission power for a given link and IoT device is expressed as:

[0051]

[0052] in, It represents the transmission power allocation value between the communication link and the IoT device, P max Limitations on the total transmit power on a given link and IoT device;

[0053] Based on the transmission power control strategy, calculate the energy consumption of the input task; if the transmission power of the IoT device And the IoT device determines the local computing input task, then the energy consumption during the local computing Calculated as:

[0054]

[0055] in, represents the energy consumption during local computing of IoT devices, ρl represents the computing energy efficiency coefficient of IoT device l, f l (t) represents the computation frequency or speed of IoT device l in time slot t;

[0056] Otherwise, the transmission power of IoT device l IoT device l decides to offload the task to UCH or MEC base station. The transmission energy consumption of IoT device l to offload the task to UCH or MEC base station is expressed as:

[0057]

[0058] in, represents the transmission energy consumption of IoT devices in time slot t, represents the task offloading decision variable, p lx (t) represents the transmission power of the IoT device in time slot t;

[0059] If an IoT device is selected or associated with a UCH based on the current offloading policy and other IoT device information, the energy consumption is also calculated based on the UCH constraints;

[0060] set up is the flight energy consumption of UCH, which depends on the flight distance and speed of UCH; then, the flight energy consumption at time slot t Calculated as:

[0061]

[0062] in, is the UCH flight energy consumption, x u (t″), y u (t″), x u (t'),y u (t') represents the position coordinates of UCH at time t", t' respectively, V u (t) represents the flight speed of the UAV at time slot t;

[0063] The total energy cost of calculating the offloading task of IoT device l is expressed as:

[0064]

[0065] In the formula, represents the total energy consumption cost of the IoT device offloading task, represents the task offloading decision variable, represents the transmission energy consumed by IoT device l to transmit task data to MEC base station n in time slot t, represents the computing energy consumed by the base station server n when performing tasks for the IoT device l in time slot t, represents the transmission energy consumed by IoT device l to transmit task data to UCH device u in time slot t, represents the computational energy consumed by the UCH device u when performing tasks for the IoT device in time slot t;

[0066] In order to control the fairness coverage of UCH, the fairness level between UCHs serving IoT devices and between IoT devices is defined; the fairness level between UCHs is r u (t)∈[0,1] is defined as:

[0067]

[0068] Among them, βu (t)∈[0,1] represents the relative number of IoT devices served by UCH at time slot t,

[0069]

[0070] The total computational cost of IoT device l is expressed as: The total computational cost of all IoT devices is: Z lx (t)=∑ l∈L Z l (t);

[0071] according to UCH's Fairness Level u (t) and UCH resources, formulate the cost objective optimization function P1 that minimizes energy consumption, delay and resource allocation cost, expressed as:

[0072] P1 minZ lx (t)(16a)

[0073] st:

[0074] C2:

[0075] C3:

[0076] C4:r u (t)∈[0,1](16e)

[0077] in, Indicates the maximum computing capacity of computing node x;

[0078] Among them, constraints C1 and C2 indicate that the computation is offloaded to at most one computation node x, and the subchannel is available and allocated in time slot t; constraint C3 represents the maximum computation capacity of computation node x, respectively; constraint C4 represents the fairness level of UCH in time slot t;

[0079] Step 4: Based on the cost objective optimization function designed in step 3, the multi-agent proximal policy optimization algorithm MAPPO-Att with an attention mechanism is used to determine the optimal selection strategy for task offloading and resource allocation, solving the problem of task offloading and resource allocation in the A2G network.

[0080] Based on the cost objective optimization function designed in step 3, the optimization problem is transformed into a random game model to deal with the complexity and dynamics of the A2G network; IoT device l selects an available node with resource availability, computing resources, and transmission power;

[0081] The IoT devices in the A2G network are regarded as intelligent agents, and the intelligent agents receive a reward to evaluate the performance of the selected behavior; the computational cost Z of the intelligent agent is l (t) as the reward function r l (t), which is a negative reward; where the goal of the learning agent is to make the total reward θ by choosing the appropriate action at time slot t l (t) is maximized; therefore, the sum of rewards is expressed as:

[0082]

[0083] Where ι is the time slot index starting from time step t, and λ = [0,1] is the discount factor;

[0084] The optimal action selection strategy is expressed as:

[0085]

[0086] in, Indicates the value of the independent variable t corresponding to the minimum value of the total reward;

[0087] The process of determining the optimal selection strategy for task offloading and resource allocation based on the multi-agent proximal strategy optimization algorithm with the introduction of the self-attention mechanism is as follows:

[0088] (1) Input the information data obtained from the interaction between the intelligent agent and the environment into the convolutional network module for convolution processing and then extract features;

[0089] The information data includes: environmental status information, equipment and network resource status, energy consumption information, and location information;

[0090] Among them, the environmental status information includes: channel status, path loss, channel gain; the device and network resource status is the available computing resources, channel status and allocated bandwidth of the computing node; the energy consumption information is the flight energy consumption or hovering energy consumption of the UCH; the location information is the relative position or distance between the IoT device and the UCH, the altitude, flight speed and location update information of the UCH;

[0091] (2) The extracted features are input into the attention encoding module, and the weight calculation and feature enhancement are performed through the attention network to capture the relevance of information;

[0092] (3) Input them into the Actor-Critic algorithm framework of reinforcement learning for strategy improvement and training;

[0093] The convolutional network module extracts features from the information data obtained by the interaction between the agent and the environment, and designs a two-layer convolutional network to extract deep multi-dimensional information from the information data; the first layer of the convolutional network has 3 input channels, 32 output channels, 4 convolution kernels, and a step size of 1; the second layer of the convolutional network has 32 input channels, 64 output channels, and 4 convolution kernels;

[0094] The attention encoding module consists of a multi-head attention network, two fully connected layers, a dropout layer and a batch-norm layer; the number of heads used in the multi-head attention network is 8; the first fully connected layer uses 64 input channels and 2048 output channels to map the input feature data from 64 dimensions to 2048 dimensions; the second fully connected network uses 2048 input channels and 64 output channels; the multi-dimensional feature information output by the convolutional network module enters the multi-head attention network for weight superposition, and uses the fully connected layer for data adjustment; the eps value used by the two norm layers is 10 -5 ; And use the dropout layer to prevent overfitting;

[0095] In the Actor-Critic algorithm framework, the Actor network obtains the probability array of the current action selection by inputting the processed feature information, and the Critic network evaluates and estimates the current state and returns a state evaluation value; the number of input channels of the first fully connected layer in the Actor network is 64, and the number of output channels is 64; the number of input channels of the second fully connected layer is 64, and the number of output channels is 7; the number of input channels of the first fully connected layer in the Critic network is 64, and the number of output channels is 64; the number of input channels of the second fully connected layer is 64, and the number of output channels is 1.

[0096] Step 5: Repeat step 4 to find the optimal strategy to maximize long-term returns, thereby reducing computing costs and effective resource allocation, and reducing overall training costs through centralized training and decentralized execution techniques; until the task completion rate reaches the set value, the drone cluster resource allocation simulation is completed.

[0097] The beneficial effects of adopting the above technical solution are: the MAPPO-Att-based multi-UAV edge network task offloading and resource allocation method provided by the present invention designs an IoT edge network supported by multiple UAVs, realizes dynamic task offloading and resource allocation in a collaborative environment; and models the collaborative computing offloading, resource allocation and related problems that provide services to different IoT devices in the IoT edge network. The above problem is expressed as a random game, assuming that each agent makes decisions based on local real-time observations and current strategies, selects resources, selects task offloading, and shares information with other agents. By adopting the improved proximal policy optimization (MAPPO-Att) method, multiple agents in the A2G network environment can maximize long-term returns by finding the optimal strategy, thereby reducing computational costs and effective resource allocation, and reducing overall training costs and effective resource allocation through centralized training and decentralized execution techniques. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 A flowchart of a method for task offloading and resource allocation in a multi-UAV edge network based on MAPPO-Att is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0099] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0100] In this embodiment, a multi-UAV edge network task offloading and resource allocation method based on MAPPO-Att is described as follows: Figure 1 As shown, the following steps are included:

[0101] Step 1: Build a multi-layer heterogeneous edge computing network model based on the A2G network and calculate the average path loss between the UAV cluster (UCH) and the Internet of Things devices (IoT devices);

[0102] The ground computing nodes provide services to a large number of multi-layer heterogeneous IoT devices. Some IoT devices (sensors on the car, body sensors, wearable devices) are mobile. Perhaps, these devices may be far away from the network coverage, and part of the ground network may have failures, resulting in degraded network performance. Therefore, the IoT edge network is realized by deploying drone flying base stations. The drone swarm is organized into clusters, and the clusters complete tasks on the ground edge network. The drones also act as aerial MEC (mobile edge computing) servers to provide different services to the ground computing nodes.

[0103] In this embodiment, a group of IoT devices is represented as l∈L={1,2,...,K}, a group of UAV swarms (UCH) is represented as u∈U={1,2,...,Y}, and a group of MEC base stations (BS) is represented as n∈N={1,2,...,M}. When the IoT device chooses to offload the task to the computing node on the UCH (UAV swarm) or MEC base station with access channel m, a group of active computing nodes in the MEC base station is represented as When IoT device l chooses to offload tasks to the MEC base station or UCH with access channel m, the uplink rate is expressed as:

[0104]

[0105] Among them, g lx is the channel gain from IoT device l to computing node x on channel m; p l is the transmission power of IoT device l, σ2 is the noise power, and B represents the channel bandwidth.

[0106] Each channel has two states in each time slot t, available or disconnected, and is represented as Used to indicate the state of channel m in time slot t; if Then the channel of the MEC base station or UCH is available and the IoT device can offload the task; otherwise, the channel is unavailable, which means that the computing node is overloaded or failed. The number of IoT devices associated with one of the UCH or MEC base stations is limited and can be expressed as Where Q represents the maximum allowed number of IoT devices whose tasks are offloaded and associated with the UCH or MEC base station.

[0107] The IoT device will first determine the sub-channel status The value of determines whether the subchannel is available, and then selects a suitable computing node (UCH or MEC base station) on the available subchannel for task offloading. Therefore, it is assumed that each IoT device can only use the allocated spectrum or channel m to select a computing node in any time slot t, which is expressed as:

[0108]

[0109] in, represents the task offloading decision variable; the task offloading decision variable is related to the computing energy consumption. Selecting different channels through the decision variable will affect the success probability of the entire link;

[0110] The communication channel of the A2G network depends on the altitude, elevation angle and propagation environment type. The communication link of the A2G network is modeled by the probabilistic path loss (PL) model, which contains line-of-sight LoS and non-line-of-sight NLoS with different occurrence probabilities. At time slot t, the LoS link probability P between UCH and IoT device l is Los It is expressed as:

[0111]

[0112] Among them, ι and ρ are constants that depend on the environment, D lu (t) represents the distance between UCH and IoT device l, H u Indicates the height of UCH;

[0113] Then the NLoS link probability is calculated as P NLos (t) = 1-P LoS (t);

[0114] The path loss from UCH to the ground IoT device l is further calculated as follows:

[0115]

[0116] in, represents the path loss of the LoS link between UCH and IoT device l, η LoS represents the additional attenuation factor of the LoS link associated with the path, represents the path loss of the NLoS link between UCH and IoT device l, η NLoS represents the additional attenuation factor of the NLoS link associated with the path, represents the free space path loss, f is the frequency carrier and c is the speed of light.

[0117] Average path loss between UCH and IoT device l in time slot t It is expressed as:

[0118]

[0119] Step 2: Design an IoT task offloading model and calculate the total delay cost of offloading tasks of IoT devices.

[0120] Each IoT device has different forms of applications, and the size of the task is generated in time slot t; each computing task has three profiles: the size of the input task d l , the computing power (CPU cycles) required to process the task l and the maximum time delay T l ;

[0121] When an IoT device decides to offload tasks to related computing nodes based on the current offloading strategy and information from other IoT devices, the latency cost is calculated in three stages: transmission time latency, execution time latency, and result sending latency.

[0122] Assume that the A2G network has a multi-channel access scheme in Orthogonal Frequency Division Multiple Access (OFDMA) technology; let W be a working frequency band that can be divided into equal sub-channels; each computing node can serve at most L IoT devices in time slot t; in each computing node, the set of available sub-channels is defined as m∈ω={1,2,3,...,M};

[0123] The task offloading decision variable at time slot t with uplink subchannel scheduling is defined as

[0124] When IoT device l offloads the input task to the computing node, IoT device l also calculates the transmission power of the uploaded task; the set of IoT devices that offload intensive tasks to the BS or UCH is defined as: According to the current offloading policy at time slot t, the input task d l The transmission power of IoT device l uploaded to BS or UCH is denoted as p l (t), where P represents the transmission power, represents the maximum value of the transmission power; in the process of transmitting the input task to the nearest computing node accessing different subchannels, the IoT device suffers from inter-cell interference; therefore, the signal interference plus noise ratio (SINR) from the IoT device l to the computing node x on the subchannel m at time slot t is defined according to the following situation;

[0125] The rate at which IoT device l offloads tasks to the BS is calculated as:

[0126] r ln (t) = Blog2(1+γ ln ) (6)

[0127] In the formula, r ln (t) represents the rate at which IoT device l offloads tasks to the MEC base station (BS), γ ln represents the signal-to-noise ratio (SNR) of IoT device l at time slot t;

[0128] Channel gain between IoT device l and UCH Among them, g o is the channel power gain at the reference distance of 1m, H u is the height of UCH, D lu(t) represents the distance between the IoT device and the UCH or BS. Even with the UCH channel strength or SINR, the IoT device l will check whether the drone group is within the network coverage before unloading the task. The specific judgment is: D lu ≤H u (t)tanθ u , where θ u Indicates the tilt angle of the IoT device, that is, the signal propagation angle.

[0129] Further calculation shows that the offload data rate of IoT device l offloading the input task to UCH is:

[0130]

[0131] In the formula, r ul (t) represents the transmission rate between IoT device l and user computing device u in time slot t, G0 is the gain coefficient, G0≈2.2846, represents the power gain factor, p lu (t) is the transmission power of IoT device l in time slot t.

[0132] like Then the offloading transmission delay cost between IoT device l and UCH based on the current offloading strategy in time slot t is Calculated as:

[0133]

[0134] in, represents the task offloading decision variable, d l (t) represents the task data volume of IoT device l in time slot t.

[0135] The execution time delay of offloading from IoT device l to the computing node is calculated based on the current offloading strategy and the information of other IoT devices in time slot t, as well as the computing resource blocks and UCH capacity of the local MEC base station;

[0136] The computing node on each resource block of the local MEC base station is shared by all IoT devices in time slot t; and the computing resources are set to be evenly distributed to all IoT devices; then the total delay cost of the offloading task of IoT device l is expressed as:

[0137]

[0138] In the formula, represents the total delay cost of the IoT device l loading task, d l (t) represents the amount of task data of IoT device l in time slot t, c l(t) represents the computing resource requirement of IoT device l in time slot t, r nl (t) represents the transmission rate between IoT device l and MEC base station n in time slot t, f ln (t) represents the computing resource capacity of MEC base station n in time slot t, r ul (t) represents the transmission rate between IoT device l and UCH device u in time slot t, f lu (t) represents the computing resource capacity of UCH device u in time slot t.

[0139] Step 3: Calculate the energy consumption cost of IoT devices when offloading tasks to computing nodes at different stages, and design the energy cost and cost target optimization function model of IoT device offloading tasks;

[0140] First, each IoT device and UCH are set to use discrete transmission power control; the transmission power value of each IoT device is expressed as a vector where p z represents the transmission power allocated to the zth subchannel; at time slot t, each IoT device selects its transmission power as Indicates that IoT device l decides to offload the input task, otherwise Where Z represents different communication options;

[0141] Based on the power constraint and the current offloading strategy, the IoT device selects the transmission power from the UCH or chooses to perform local tasks. Power resources are crucial for drones, so the power value of each UCH communicating with the selected IoT device l is expressed as The binary value of this power is expressed as

[0142] The IoT device l or UCH selects and allocates the power level in each time slot t. The limit of the total transmission power for a given link and IoT device is expressed as:

[0143]

[0144] in, It represents the transmission power allocation value between the communication link and the IoT device, P max Limitations on total transmit power on a given link and device;

[0145] Based on the above transmission power control strategy, the energy consumption of the input task is calculated; if the transmission power of the IoT device And the IoT device determines the local computing input task, then the energy consumption during the local computing Calculated as:

[0146]

[0147] in, represents the energy consumption during local computing of IoT devices, ρl represents the computing energy efficiency coefficient of IoT device l, f l (t) represents the computation frequency or speed (in cycles per second) of IoT device l at time slot t.

[0148] Otherwise, the transmission power of IoT device l IoT device l decides to offload the task to UCH or BS. The transmission energy consumption of IoT device l offloading the task to UCH or BS is expressed as:

[0149]

[0150] in, represents the transmission energy consumption of IoT devices in time slot t, represents the task offloading decision variable, p lx (t) represents the transmission power of the IoT device in time slot t.

[0151] If an IoT device is selected or associated with a UCH based on the current offloading strategy and other IoT device information, the energy consumption is also calculated based on the UCH constraints, such as the energy consumption of flight, hovering, and execution of time slot t. For simplicity, in this embodiment, the flight and hovering energy consumption of the drone swarm can be referred to interchangeably. Assume is the UCH flight energy consumption, which depends on the flight distance and speed of the UCH. Then, the flight energy consumption at time slot t is Calculated as:

[0152]

[0153] in, is the UCH flight energy consumption, x u (t″), y u (t″), x u (t'),y u (t') represents the position coordinates of UCH at time t", t' respectively, V u (t) represents the flight speed of the UAV at time slot t;

[0154] The total energy cost of calculating the offloading task of IoT device l is expressed as:

[0155]

[0156] In the formula, represents the total energy consumption cost of the IoT device offloading task, represents the task offloading decision variable, represents the transmission energy consumed by IoT device l to transmit task data to MEC base station n in time slot t, represents the computing energy consumed by the base station server n when performing tasks for the IoT device l in time slot t, represents the transmission energy consumed by IoT device l to transmit task data to UCH device u in time slot t, It represents the computational energy consumed by UCH device u when performing tasks for IoT devices in time slot t.

[0157] The SDN controller (which stands for software defined network, controls network resources through software, and is responsible for communicating with UCH devices, IoT devices, etc. in the A2G communication network to help optimize task allocation and energy consumption management) maintains continuous communication with UCH, not a single drone. IoT devices also learn from the environment, experience, and experience of other IoT devices to reconfigure. This can minimize losses and computational complexity. In order to control the fairness coverage of UCH, the fairness level between UCHs serving IoT devices and between IoT devices is defined; the fairness level between UCHs is r u (t)∈[0,1] is defined as:

[0158]

[0159] Among them, β u (t)∈[0,1] represents the relative number of IoT devices served by UCH at time slot t,

[0160] The total computational cost of IoT device l is expressed as: The total computational cost of all IoT devices is: Z lx (t)=∑ l∈L Z l (t).

[0161] according to UCH's Fairness Level u (t) and UCH resources, formulate the cost objective optimization function P1 that minimizes energy consumption, delay and resource allocation cost, expressed as:

[0162] P1 minZ lx (t) (16a)

[0163] st:

[0164] C2:

[0165] C3:

[0166] C4:r u(t)∈[0,1](16e)

[0167] in, Indicates the maximum computing capacity of computing node x;

[0168] Here, constraints C1 and C2 indicate that the computation is offloaded to at most one computation node x, and the subchannel is available and allocated in time slot t. Constraint C3 indicates the maximum computation capacity of computation node x, respectively. Constraint C4 indicates the fairness level of UCH in time slot t.

[0169] Since the above optimization problem is a non-deterministic polynomial hard problem, we transform the optimal computation offloading and effective resource allocation problem into a stochastic game problem and solve it using the MADRL technique, which will be discussed in Step 4.

[0170] Step 4: Based on the cost objective optimization function designed in step 3, the multi-agent proximal policy optimization algorithm MAPPO-Att (Multi-Agent Proximal Policy Optimization-Attention) with an attention mechanism is used to determine the optimal selection strategy for task offloading and resource allocation to solve the problem of task offloading and resource allocation in the A2G network.

[0171] Based on the cost objective function designed in step 3, since the optimization problem in (16a) is a complex continuous action space problem, the optimization problem is transformed into a random game model to deal with the complexity and dynamics of the A2G network; the IoT device l selects an available node with resource availability, computing resources, and transmission power.

[0172] The IoT devices in the A2G network are regarded as intelligent agents, and the intelligent agents receive a reward to evaluate the performance of the selected behavior; the computational cost Z of the intelligent agent is l (t) as the reward function r l (t), which is a negative reward. The goal of the learning agent in deep reinforcement learning DRL is to select appropriate actions at time slot t so that the total reward θ l (t) is maximized; therefore, the sum of rewards is expressed as:

[0173]

[0174] Where ι is the time slot index starting from time step t, and λ = [0,1] is the discount factor;

[0175] The optimal action selection strategy is expressed as:

[0176]

[0177] in, Indicates the value of the independent variable t corresponding to the minimum sum of rewards.

[0178] To this end, the present invention adopts a combination of a multi-agent deep reinforcement learning (MADRL) method and an attention mechanism Att to solve the problem of task offloading and resource allocation in A2G networks.

[0179] In traditional wireless communication and computing systems, single-agent deep reinforcement learning (DRL) algorithms are often used to solve relatively simple, small-scale problems. However, with the continuous expansion of the application of air-to-ground (A2G) networks, the complexity of the network environment has been greatly improved. The A2G network environment not only involves extensive spatial coverage, but also challenges such as multi-agent interaction and dynamic time slot changes. This multi-dimensional complexity makes it difficult for traditional single-agent DRL to respond effectively, especially in terms of resource management, task offloading, and communication scheduling. More advanced strategies are needed to coordinate the various resources and devices in the system. In order to solve these problems, the present invention constructs a hierarchical A2G network to cope with complex multi-agent dynamic environments.

[0180] Layer-1 is the application layer on the cloud, which has multiple cloud service applications and supports multiple application systems.

[0181] Layer-2 is the control layer, which contains the SDN controller, which is used to centrally manage the data layer devices and edge computing servers. This layer uses the OpenFlow protocol. The SDN controller can also allocate resources such as spectrum, computing, and transmission power according to different network requirements.

[0182] Layer-3 is the data layer controlled by the SDN controller, including switches, multiple BSs, gateways, and UCH. The controller manipulates resource allocation, computation offloading decisions, and other decisions.

[0183] Layer-4 is a hyper-dense heterogeneous IoT device network consisting of basic end-user smart devices.

[0184] In the scenario of the present invention, each IoT device is regarded as an agent in the learning process. The MAPPO algorithm adopts a centralized training and decentralized execution model, and each IoT device l of layer-4 is an autonomous agent. The IoT device is called an agent, and any agent is represented as l. The agent has an Actor network as a decentralized execution, and the Critic network is the centralized training part. Centralized training (Critic network and Actor network) is mainly performed on the UCH or BS, and sometimes on the central controller in time slot t.

[0185] MAPPO-based algorithms are suitable for multi-agent environments, where agents can learn collaboratively and improve network performance. The MAPPO algorithm is one of the MADRL schemes. It is an improved Actor-Critic network that can handle dynamic environments and is suitable for cooperative strategy learning of multiple agents with continuous action spaces. The Critic network also uses the behavioral strategies of other agents, and agents can make decisions independently using their strategies and observations.

[0186] In order to ensure the QoS of IoT devices and improve the network performance, a swarm of drones is deployed to act as ABS and flight MEC (FMEC) servers to provide different services to IoT devices at the IoT device network level. In this work, we focus on minimizing the overall task computation cost from the perspective of energy, latency and considering drone resource constraints in a dynamic A2G network environment.

[0187] MAPPO is an extension of the PPO algorithm in the multi-agent domain. It is an Actor-Critic algorithm that uses the CTDE paradigm to train random policies in an online policy update manner. MAPPO trains two independent neural networks: an Actor network with parameters θ and an Actor network with parameters If the agents are homogeneous, these network parameters can be shared among all agents. Each agent can also have its own pair of actor and critic networks.

[0188] Actor network, denoted as π θ , the agent can observe A categorical distribution mapped to a discrete action space can also be mapped to a mean and standard deviation vector of a multivariate Gaussian distribution that samples actions in a continuous action space. The Actor Network is trained to maximize the objective:

[0189]

[0190] in It is the advantage function calculated by Generalized Advantage Estimation (GAE), S is the policy entropy, σ is the entropy coefficient, ε is a small parameter representing how far the new policy is allowed to be from the old policy, and clip means clipping. θ represents the normal distribution of the new strategy, Represents the old strategy normal distribution.

[0191] The Critic network can be expressed as is trained to minimize the loss function:

[0192]

[0193] in is the cumulative discounted reward, ε is a small parameter representing how far the new strategy is allowed to be from the old strategy, Represents the value function, which is used to estimate the state under the current policy (represented by the parameter φ) The expected return (accumulated future rewards), is the state of the environment at a certain time step k in an episode or trajectory i. In the above two formulas, B represents the batch size and n represents the number of agents.

[0194] The algorithm works as follows: Each agent downloads the trained actor network weights from the central trainer and loads them into its own actor network. At the beginning of each time slot, the agent obtains the current state of the environment based on local observations. Each agent performs actions independently in a decentralized manner and obtains the current reward and transition state according to the policy (i.e., new policy or old policy). The agent stores this data in its local experience replay buffer for subsequent policy updates. When the agent performs an action, it takes actions and obtains rewards based on the observed state of the local environment and the current policy. When the communication channel is available and unblocked, the agent uploads its local information to the base station for continuous training; if the communication channel is limited or overloaded, the agent uploads the information to a nearby drone cluster. The agent then uses the uploaded information to update the policy (including the new policy and the old policy) and decides whether to offload the task through the subchannel or perform local calculations based on the current observations and policies at time slot t. The state, action, reward, transition state, and other data of all agents at each time slot t are stored in the global experience replay buffer for global training and optimization.

[0195] The attention mechanism is a technology used to improve the performance of neural network models, which can be mainly divided into hard attention mechanism, soft attention mechanism and self-attention mechanism. The hard attention mechanism uses binary weights of 0 or 1 to decide which areas to focus on, so some irrelevant items can be discarded directly, which is very effective in tasks such as image cropping in the image field, but some information may also be lost. In contrast, soft attention uses differentiable weight values ​​to express the degree to which a region is paid attention to, so all information can be taken into account, but the computational cost is high. The self-attention mechanism is a mechanism that determines the weight of each input item based on the internal relationship of the input item, and has the advantage of parallel computing. The key to the attention mechanism is to find the relationship between the input data. Its core lies in the scoring mechanism. The most commonly used is the scaled dot product model, and the calculation formula is as follows:

[0196]

[0197] In the self-attention mechanism, the three vectors Q, K, and V are all transformed from the common input vector. Q represents the query vector, which is a representation vector related to the task, and other input information needs to calculate similarity with it; K represents the key vector, which is used to calculate similarity with Q; d k represents the dimension of K, is a scaling factor used to avoid a rapid decrease in the softmax function gradient caused by a too fast increase in the calculated value. V represents the value vector, which is obtained by multiplying the attention feature vector with the probability distribution generated after the normalization of the Softmax function.

[0198] The basic process of the self-attention mechanism can be divided into three steps. The first step is to use the query vector to score other input information key vectors. The calculation formula is as follows:

[0199]

[0200] The second step is to normalize the scoring results to obtain the attention weights:

[0201]

[0202] where α i is the attention weight, softmax(s i ) indicates normalization of the score, exp(s i ) is to take the exponential function of the scoring result to ensure that all values ​​are positive, ∑ j exp(s j ) is the sum of the indices of all scoring results.

[0203] The third step is to perform weighted summation on the value vectors of all input information according to the attention weights to obtain the attention feature vector:

[0204]

[0205] where α i represents the weight, V i Indicates the value of the input information.

[0206] The multi-head attention mechanism allows the model to jointly focus on information in different representation subspaces at different positions, thereby improving the model's expressiveness. For example, in image classification tasks, we can regard an image as a sequence matrix, and then use a multi-head attention mechanism to process different local information in the image. In this way, the model can simultaneously focus on different representation subspaces at different positions, thereby better understanding the content of the image and improving the accuracy of classification. Specifically, take the multi-head scaled dot product attention network as an example. The query vector, key vector, and value vector first pass through a fully connected layer to obtain the hidden state vector, and then these three hidden state vectors are respectively input into multiple scaled dot product attention networks and processed multiple times. Each attention network outputs an attention representation vector, and then multiple attention representation vectors are spliced ​​together and input into a fully connected layer for fusion to obtain the final attention representation vector. The calculation formula is expressed as follows:

[0207]

[0208] Among them, h represents the number of attention heads, W is the parameter of the fully connected layer in the network, and head i Represents the calculation result of the i-th attention head. Multi-head attention divides the original input into h subspaces, calculates the attention vector independently, and then concatenates all the subspace results as output. MultiHead(Q,K,V) represents the multi-head attention mechanism, where Q (query), K (key), and V (value) are the three input matrices. Through the multi-head attention mechanism, the input is divided into multiple attention heads and merged after parallel attention calculation. Concat(head1,...,head h ) concatenates all attention heads into a large vector, W 0 Indicates mapping the concatenated vector back to the original dimension. i Q represents the query matrix of the head, Q multiplied by the query weight matrix W i Q Post-generation, KW i K represents the key matrix of the th head, K multiplied by the key weight matrix W i K After generation, VW i V Represents the value matrix of the th head, V multiplied by the value weight matrix W i V After generation.

[0209] The steps of using the multi-agent proximal strategy optimization algorithm with self-attention mechanism to determine the optimal selection strategy for task offloading and resource allocation are as follows:

[0210] (1) Input the information data obtained from the interaction between the intelligent agent and the environment into the convolutional network module for convolution processing and then extract features;

[0211] The information data includes: environmental status information, equipment and network resource status, energy consumption information and location information.

[0212] Among them, the environmental status information includes: channel status (whether the channel is available, the current link status (line of sight LoS or non-line of sight NLoS), path loss (average path loss from the device to the drone base station or mobile edge computing node), channel gain (communication gain between the device and the UCH or computing node); the device and network resource status is the available computing resources, channel status, and allocated bandwidth of the computing node. The energy consumption information is the flight energy consumption or hovering energy consumption of the UCH; the location information is the relative position or distance between the IoT device and the UCH, the altitude, flight speed, and location update information of the UCH.

[0213] (2) The extracted features are input into the attention encoding module, and the weight calculation and feature enhancement are performed through the attention network to capture the relevance of information and achieve multivariate decoupling or partial decoupling to a certain extent;

[0214] (3) Input them into the Actor-Critic algorithm framework of reinforcement learning for strategy improvement and training.

[0215] In this embodiment, the convolutional network module extracts features from the information data obtained by the interaction between the agent and the environment. Considering the computational complexity and overfitting problems, a two-layer convolutional network is designed to extract deep multi-dimensional information of the data. The first layer of the convolutional network has 3 input channels, 32 output channels, 4 convolution kernels, and a step size of 1. The second layer of the convolutional network has 32 input channels, 64 output channels, and 4 convolution kernels.

[0216] The attention encoding module consists of a multi-head attention network, two fully connected layers, a dropout layer, and a batch-norm layer. The number of heads used in the multi-head attention network is 8. The first fully connected layer uses 64 input channels and 2048 output channels to map the input data from 64 dimensions to 2048 dimensions. The second fully connected network uses 2048 input channels and 64 output channels. The multi-dimensional feature information output by the convolutional network module enters the multi-head attention network for weight superposition, and the fully connected layer is used for data adjustment. The EPS value used by the two-layer norm is 10 -5 . And use the dropout layer to prevent overfitting.

[0217] In the Actor-Critic algorithm framework, the Actor network obtains the probability array of the current action selection through the input processed feature information, and the Critic network evaluates and estimates the current state and returns a state evaluation value. The first fully connected layer in the Actor network has 64 input channels and 64 output channels. The second fully connected layer has 64 input channels and 7 output channels. The first fully connected layer in the Critic network has 64 input channels and 64 output channels. The second fully connected layer has 64 input channels and 1 output channel.

[0218] The content of step 4 above can be collectively referred to as a multi-agent proximal strategy optimization algorithm that introduces a self-attention mechanism. The multi-agent proximal strategy optimization algorithm that introduces a self-attention mechanism can ultimately determine the optimal selection strategy. The pseudo code of the algorithm is shown in Table 1.

[0219] Table 1 Multi-agent proximal strategy optimization algorithm with self-attention mechanism

[0220]

[0221]

[0222] Step 5: Repeat step 4 to find the optimal strategy to maximize long-term returns, thereby reducing computing costs and effective resource allocation, and reducing overall training costs through centralized training and decentralized execution techniques; until the task completion rate reaches the set value, the drone cluster resource allocation simulation is completed.

[0223] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A multi-UAV edge network task offloading and resource allocation method based on MAPPO-Att, characterized by: The following steps are involved: Step 1: Build a multi-layer heterogeneous edge computing network model based on the A2G network and calculate the average path loss between the drone swarm UCH and the IoT device IoT; Step 2: Design an IoT task offloading model and calculate the total delay cost of offloading tasks from IoT devices; Step 3: Calculate the energy consumption cost of IoT devices when offloading tasks to computing nodes at different stages, and design the energy cost and cost target optimization function model of IoT device offloading tasks; Step 4: Based on the cost objective optimization function designed in step 3, the multi-agent proximal policy optimization algorithm MAPPO-Att with an attention mechanism is used to determine the optimal selection strategy for task offloading and resource allocation, solving the problem of task offloading and resource allocation in the A2G network. Step 5: Repeat step 4 to find the optimal strategy to maximize long-term returns, thereby reducing computing costs and effective resource allocation, and reducing overall training costs through centralized training and decentralized execution techniques; until the task completion rate reaches the set value, the drone cluster resource allocation simulation is completed.

2. According to claim 1, a method for task offloading and resource allocation of multi-UAV edge network based on MAPPO-Att is characterized by: The specific method of step 1 is: The IoT edge network is realized by deploying drone flying base stations; drone groups are organized into clusters, which complete tasks on the ground edge network; drones also act as aerial MEC base stations, providing different services for ground computing nodes; A group of IoT devices is represented as l∈L={1,2,...,K}, a group of drones is represented as u∈U={1,2,...,Y}, and a group of MEC base stations is represented as n∈N={1,2,...,M}. When an IoT device chooses to offload tasks to a computing node on a UCH or MEC base station with access channel m, the set of active computing nodes in the MEC base station is represented as When IoT device l chooses to offload tasks to the MEC base station or UCH with access channel m, the uplink rate is expressed as: Among them, g lx is the channel gain from IoT device l to computing node x on channel m; p l is the transmission power of IoT device l, σ2 is the noise power, and B represents the channel bandwidth; Each channel has two states in each time slot t, available or disconnected, and is represented as Used to indicate the state of channel m in time slot t; if Then the channel of BS or UCH is available and IoT devices can offload tasks; otherwise, the channel is unavailable, which means that the computing node is overloaded or fails; the number of IoT devices associated with one of the UCH or MEC base stations is limited, expressed as Where Q represents the maximum allowed number of IoT devices whose tasks are offloaded and associated with the UCH or MEC base station; The IoT device will first determine the sub-channel status The value of determines whether the subchannel is available, and then selects a suitable computing node on the available subchannel for task offloading. Therefore, it is assumed that each IoT device can only use the allocated spectrum or channel m to select a computing node in any time slot t, and is expressed as: in, represents the task offloading decision variable; the task offloading decision variable is related to the computing energy consumption. Selecting different channels through the decision variable will affect the success probability of the entire link; The communication link of the A2G network is modeled by a probabilistic path loss model, which includes line-of-sight LoS and non-line-of-sight NLoS with different occurrence probabilities; at time slot t, the LoS link probability P between UCH and IoT device l Los It is expressed as: Among them, ι and ρ are constants that depend on the environment, D lu (t) represents the distance between UCH and IoT device l, H u Indicates the height of UCH; Then the NLoS link probability is calculated as P NLos (t) = 1-P LoS (t); The path loss from UCH to the ground IoT device l is further calculated as follows: in, represents the path loss of the LoS link between UCH and IoT device l, η LoS represents the additional attenuation factor of the LoS link associated with the path, represents the path loss of the NLoS link between UCH and IoT device l, η NLoS represents the additional attenuation factor of the NLoS link associated with the path, represents the free space path loss, f is the frequency carrier, c is the speed of light; Average path loss between UCH and IoT device l in time slot t It is expressed as:

3. According to claim 2, a method for task offloading and resource allocation of multi-UAV edge network based on MAPPO-Att is characterized in that: The specific method of step 2 is: Each IoT device has different forms of applications, and the size of the task is generated in time slot t; each computing task has three profiles: the size of the input task d l , the computing power required to process the task c l and the maximum time delay T l ; When an IoT device decides to offload tasks to related computing nodes based on the current offloading strategy and information from other IoT devices, the latency cost is calculated in three stages: transmission time latency, execution time latency, and result sending latency. Assume that the A2G network has a multi-channel access scheme in OFDMA technology; let W be a working frequency band that can be divided into equal sub-channels; each computing node can serve at most L IoT devices in time slot t; in each computing node, the set of available sub-channels is defined as m∈ω={1,2,3,...,M}; The task offloading decision variable at time slot t with uplink subchannel scheduling is defined as When IoT device l offloads the input task to the computing node, IoT device l also calculates the transmission power of the uploaded task; the set of IoT devices that offload intensive tasks to the MEC base station or UCH is defined as: According to the current offloading policy at time slot t, the input task d l The transmission power of IoT device l uploaded to MEC base station or UCH is expressed as p l (t), where P represents the transmission power, represents the maximum value of the transmission power; in the process of transmitting the input task to the nearest computing node accessing different subchannels, the IoT device suffers from inter-cell interference; therefore, the signal interference plus noise ratio SINR from the IoT device l to the computing node x on the subchannel m is defined according to the following situation at time slot t; The rate at which IoT device l offloads tasks to the MEC base station is calculated as: r ln (t)6Blog2(1+γ ln ) (6) In the formula, r ln (t) represents the rate at which IoT device l offloads tasks to the MEC base station, γ ln represents the signal-to-noise ratio of IoT device l at time slot t; Channel gain between IoT device l and UCH Among them, g o is the channel power gain at the reference distance of 1m, H u is the height of UCH, D lu (t) represents the distance between the IoT device and the UCH or MEC base station; IoT device l checks whether the drone group is within the network coverage before unloading the task. The specific judgment is: D lu ≤H u (t)tanθ u , where θ u Indicates the tilt angle of the IoT device; Further calculation shows that the offload data rate of IoT device l offloading the input task to UCH is: In the formula, r ul (t) represents the transmission rate between IoT device l and user computing device u in time slot t, G0 is the gain coefficient, represents the power gain factor, p lu (t) is the transmission power of IoT device l in time slot t; like Then the offloading transmission delay cost between IoT device l and UCH based on the current offloading strategy in time slot t is Calculated as: in, represents the task offloading decision variable, d l (t) represents the task data volume of IoT device l in time slot t; The execution time delay of offloading from IoT device l to the computing node is calculated based on the current offloading strategy and the information of other IoT devices in time slot t, as well as the computing resource blocks and UCH capacity of the local MEC base station; The computing node on each resource block of the local MEC base station is shared by all IoT devices in time slot t; and the computing resources are set to be evenly distributed to all IoT devices; then the total delay cost of the offloading task of IoT device l is expressed as: In the formula, represents the total delay cost of the IoT device l loading task, d l (t) represents the amount of task data of IoT device l in time slot t, c l (t) represents the computing resource requirement of IoT device l in time slot t, r nl (t) represents the transmission rate between IoT device l and MEC base station n in time slot t, f ln (t) represents the computing resource capacity of MEC base station n in time slot t, r ul (t) represents the transmission rate between IoT device l and UCH device u in time slot t, f lu (t) represents the computing resource capacity of UCH device u in time slot t.

4. According to claim 3, a method for task offloading and resource allocation based on MAPPO-Att multi-UAV edge network is characterized in that: The specific method of step 3 is: First, each IoT device and UCH are set to use discrete transmission power control; the transmission power value of each IoT device is represented by the vector P l T ={p1,...,p z }, where p z represents the transmission power allocated to the zth subchannel; at time slot t, each IoT device selects its transmission power as Indicates that IoT device l decides to offload the input task, otherwise Where Z represents different communication options; Based on the power constraint and the current offloading strategy, the IoT device selects the transmission power from the UCH or chooses local task execution; The power value of each UCH communicating with the selected IoT device l is expressed as The binary value of this power is expressed as The IoT device l or UCH selects and allocates the power level in each time slot t. The limit of the total transmission power for a given link and IoT device is expressed as: in, It represents the transmission power allocation value between the communication link and the IoT device, P max Limitations on the total transmit power on a given link and IoT device; Based on the transmission power control strategy, calculate the energy consumption of the input task; if the transmission power of the IoT device And the IoT device determines the local computing input task, then the energy consumption during the local computing Calculated as: in, represents the energy consumption during local computing of IoT devices, ρl represents the computing energy efficiency coefficient of IoT device l, f l (t) represents the computation frequency or speed of IoT device l in time slot t; Otherwise, the transmission power of IoT device l IoT device l decides to offload the task to UCH or MEC base station. The transmission energy consumption of IoT device l to offload the task to UCH or MEC base station is expressed as: in, represents the transmission energy consumption of IoT devices in time slot t, represents the task offloading decision variable, p lx (t) represents the transmission power of the IoT device in time slot t; If an IoT device is selected or associated with a UCH based on the current offloading policy and other IoT device information, the energy consumption is also calculated based on the UCH constraints; set up is the flight energy consumption of UCH, which depends on the flight distance and speed of UCH; then, the flight energy consumption at time slot t Calculated as: in, is the UCH flight energy consumption, x u (t″), y u (t″), x u (t'),y u (t') represents the position coordinates of UCH at time t", t' respectively, V u (t) represents the flight speed of the UAV at time slot t; The total energy cost of calculating the offloading task of IoT device l is expressed as: In the formula, represents the total energy consumption cost of the IoT device offloading task, represents the task offloading decision variable, represents the transmission energy consumed by IoT device l to transmit task data to MEC base station n in time slot t, represents the computing energy consumed by the base station server n when performing tasks for the IoT device l in time slot t, represents the transmission energy consumed by IoT device l to transmit task data to UCH device u in time slot t, represents the computational energy consumed by the UCH device u when performing tasks for the IoT device in time slot t; In order to control the fairness coverage of UCH, the fairness level between UCHs serving IoT devices and between IoT devices is defined; the fairness level between UCHs is r u (t)∈[0,1] is defined as: Among them, β u (t)∈[0,1] represents the relative number of IoT devices served by UCH at time slot t, The total computational cost of IoT device l is expressed as: The total computational cost of all IoT devices is: Z lx (t)=∑ l∈L Z l (t); according to UCH's Fairness Level u (t) and UCH resources, formulate the cost objective optimization function P1 that minimizes energy consumption, delay and resource allocation cost, expressed as: P1 minZ lx (t)(16a) C4:r u (t)∈[0,1](16e)where Indicates the maximum computing capacity of computing node x; Among them, constraints C1 and C2 indicate that the computation is offloaded to at most one computing node x, and the subchannel is available and allocated in time slot t; constraint C3 represents the maximum computing capacity of computing node x; constraint C4 represents the fairness level of UCH in time slot t.

5. The method for task offloading and resource allocation of multi-UAV edge network based on MAPPO-Att according to claim 4 is characterized in that: The specific method of step 4 is: Based on the cost objective optimization function designed in step 3, the optimization problem is transformed into a random game model to deal with the complexity and dynamics of the A2G network; IoT device l selects an available node with resource availability, computing resources, and transmission power; The IoT devices in the A2G network are regarded as intelligent agents, and the intelligent agents receive a reward to evaluate the performance of the selected behavior; the computational cost Z of the intelligent agent is l (t) as the reward function r l (t), which is a negative reward; where the goal of the learning agent is to make the total reward θ by choosing the appropriate action at time slot t l (t) is maximized; therefore, the sum of rewards is expressed as: Where ι is the time slot index starting from time step t, and λ = [0,1] is the discount factor; The optimal action selection strategy is expressed as: in, Indicates the value of the independent variable t corresponding to the minimum value of the total reward; The process of determining the optimal selection strategy for task offloading and resource allocation based on the multi-agent proximal strategy optimization algorithm with the introduction of the self-attention mechanism is as follows: (1) Input the information data obtained from the interaction between the intelligent agent and the environment into the convolutional network module for convolution processing and then extract features; The information data includes: environmental status information, equipment and network resource status, energy consumption information, and location information; Among them, the environmental status information includes: channel status, path loss, channel gain; the device and network resource status is the available computing resources, channel status and allocated bandwidth of the computing node; the energy consumption information is the flight energy consumption or hovering energy consumption of the UCH; the location information is the relative position or distance between the IoT device and the UCH, the altitude, flight speed and location update information of the UCH; (2) The extracted features are input into the attention encoding module, and the weight calculation and feature enhancement are performed through the attention network to capture the relevance of information; (3) Input them into the Actor-Critic algorithm framework of reinforcement learning for strategy improvement and training.

6. The method for task offloading and resource allocation of multi-UAV edge network based on MAPPO-Att according to claim 5 is characterized in that: The convolutional network module performs feature extraction on the information data obtained by the interaction between the intelligent agent and the environment, and designs a two-layer convolutional network to extract deep multi-dimensional information of the information data; the first layer of the convolutional network has 3 input channels, 32 output channels, 4 convolution kernel size, and a step size of 1; the second layer of the convolutional network has 32 input channels, 64 output channels, and 4 convolution kernel size.

7. The method for task offloading and resource allocation of multi-UAV edge network based on MAPPO-Att according to claim 6 is characterized in that: The attention encoding module consists of a multi-head attention network, two fully connected layers, a dropout layer and a batch-norm layer; the number of heads used in the multi-head attention network is 8; the first fully connected layer uses 64 input channels and 2048 output channels to map the input feature data from 64 dimensions to 2048 dimensions; the second fully connected network uses 2048 input channels and 64 output channels; the multi-dimensional feature information output by the convolutional network module enters the multi-head attention network for weight superposition, and uses the fully connected layer for data adjustment; the eps value used by the two norm layers is 10 -5 ; and use the dropout layer to prevent overfitting.

8. The method for task offloading and resource allocation of multi-UAV edge network based on MAPPO-Att according to claim 7 is characterized in that: In the Actor-Critic algorithm framework, the Actor network obtains the probability array of the current action selection by inputting the processed feature information, and the Critic network evaluates and estimates the current state and returns a state evaluation value; the number of input channels of the first fully connected layer in the Actor network is 64, and the number of output channels is 64; the number of input channels of the second fully connected layer is 64, and the number of output channels is 7; the number of input channels of the first fully connected layer in the Critic network is 64, and the number of output channels is 64; the number of input channels of the second fully connected layer is 64, and the number of output channels is 1.

Images