Task joint offloading and migration method for air-ground collaborative mobile edge computing

Abstract

The present application belongs to the field of mobile group computing technology. The present application provides a method for joint task offloading and migration for air-ground collaborative mobile edge computing. The disclosed embodiment takes into account the scenarios of resource heterogeneity, limited resources, limited communication range and dynamic network fluctuations under edge computing composed entirely of drone-vehicle networks; the method models the joint offloading and migration problem of DAG tasks and expresses it as a problem of minimizing the average delay of user DAG tasks; the method takes into account the different characteristics of UAVs and vehicles, and the different conditions for triggering task migration; the method takes into account the situation where the same DAG task may undergo multiple migrations, and takes into account the shortest transmission path for task migration, task upload, and calculation result download.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of mobile group computing technology, and in particular to a task joint offloading and migration method for air-ground collaborative mobile edge computing. Background Art

[0002] With the transition to sixth-generation mobile communication technology (6G), the rapid expansion of smart devices and the explosive growth of new applications have brought about the demand for computational tasks, such as video stream analysis and extraction in smart home monitoring systems and sensor data processing in IoT systems. Due to the limited processing capabilities of smart devices, computational tasks need to be offloaded to the edge for processing. Computational tasks are tasks submitted to edge servers. The edge servers only need to return the processing results of the computational tasks to the user within the delivery deadline. With one submission and one delivery, there is no secondary interaction between the user and the server, making computational tasks relatively less sensitive to latency.

[0003] In the traditional field of edge computing, BSs are generally used as edge servers. However, BS resources are limited, so BSs are generally used to handle highly latency-sensitive services such as VR and AR that require continuous interaction. Many existing studies consider using drones and vehicles as edge servers to assist BSs in providing services to users. In order to reduce the pressure on the BS, and considering that mobile servers are the future development trend and that mobile servers have good processing capabilities and can handle computing tasks with relatively low latency sensitivity, this application proposes a solution that is completely independent of the BS, namely, dispatching a group of drones equipped with lightweight CPUs and GPUs, and a self-organizing network of roadside parked vehicles as mobile MEC servers to handle computing tasks.

[0004] None of the related technologies considers both the heterogeneous resources of mobile MEC servers and the ability to process certain complex computing tasks at the granularity of subtasks. Typically, complex computing tasks can be modeled as typical task flows, taking into account the interdependencies of tasks. For example, a video navigation task involves graphics, face detection, camera preview, and video processing, and can be represented as a dependency graph of 14 tasks. If migration decisions can be made at the granularity of subtasks rather than entire tasks, the performance gains from offloading to edge servers can be amplified.

[0005] Therefore, it is necessary to improve one or more problems existing in the above-mentioned related technical solutions.

[0006] It should be noted that this section is intended to provide background or context for the technical solutions of the present disclosure stated in the claims. The description herein is not admitted to be prior art by virtue of being included in this section. Summary of the Invention

[0007] In order to avoid the shortcomings of the existing technology, the present application provides a task joint offloading and migration method for air-ground collaborative mobile edge computing to solve XX problems existing in the existing technology.

[0008] According to an embodiment of the present disclosure, a method for joint task offloading and migration for air-ground collaborative mobile edge computing is provided, the method comprising:

[0009] Build an MEC scenario; the MEC scenario includes M edge servers, K users, and 1 SDN controller. The edge servers include drones or vehicles.

[0010] Based on the joint offloading and migration of users' DAG tasks in MEC scenarios, an MDP problem model is constructed;

[0011] The MDP problem model is transformed into a Markov decision process, and the four-tuple<S,A,P,R> Represented by; where S is the state set of the Markov decision process, A is the action set of the Markov decision process, P is the state transition probability of the Markov decision process, and R is the reward of the Markov decision process;

[0012] The DJOMD-PPO algorithm is used to solve the Markov decision process to obtain the minimized average delay of user DAG tasks.

[0013] Furthermore, in the MEC scenario, each user corresponds to a DAG task, and each subtask of the DAG task is processed by a different resource type. The user's DAG tasks arrive in a Poisson distribution over time. After the DAG task arrives, its predecessor subtasks are offloaded to the edge server with corresponding resources in topological order for processing; after the predecessor subtask is processed, the subsequent subtask begins to be processed. Each edge server processes one task, and the SDN controller makes an offloading decision at the beginning of each moment; when the edge server cannot process the corresponding subtask, the SDN controller makes a migration decision to migrate the subtask to other edge servers with corresponding resources.

[0014] Furthermore, in the MEC scenario, the task scheduling plan, the calculation process of the response time of the user DAG request, the task start time, the task execution time, the migration trigger condition and the task migration time, the task download time and the communication model are set.

[0015] Furthermore, the objective function of the MDP problem model is:

[0016]

[0017] Among them, k represents the kth user, T resp The response time for user requests.

[0018] Furthermore, the state set Among them, S1 represents the user basic information domain, including the coordinates of all users and the CPU cycles required for all user tasks. S2 represents the user DAG task information domain, including the input data size, output data size, and required resource type of each subtask in all user DAG requests. S3 represents the status of the edge server composed of drones and vehicles, including the geographic location of all drones and vehicles, the computing power of all drones and vehicles, the current power of all drones, the CPU clock frequency of all drones and vehicles, and the resource types owned by all drones and vehicles. S4 represents the network bandwidth of each link in the drone-vehicle edge computing network.

[0019] Each action a in the action set A t It is an M×N two-dimensional decision matrix The value of each element in the two-dimensional decision matrix is ​​0 or 1. i∈M,j∈K means that the SDN controller does not make a decision at time t. i∈M,j∈K means that the decision made by the SDN controller for user j at time t is served by edge server i, and at the same time, a user's DAG task request is served by an edge server;

[0020] The state transition probability P: S × A × S → [0, 1] represents the state transition probability distribution, p0: S → [0, 1] is the distribution of the initial state s0, the initial state s0 is obtained by taking action a t Based on the probability P(s t+1 |s t ,a t ) from s t Transfer to s t+1 ;

[0021] Reward R: r t : represents the immediate reward received by the SDN controller after making a decision for the user request at time t; the goal of the MDP problem model is to find the optimal action in each step so that the expected cumulative reward R episode Maximum, the reward function is set as follows:

[0022]

[0023] When the round ends, that is, when all users' DAG requests are executed, the average latency of all users' DAG requests is calculated, and the reward setting is modeled based on the average latency; in other cases, the reward is set to 0.

[0024] Furthermore, the DJOMD-PPO algorithm is used to solve the Markov decision process to obtain the minimized average delay of the user DAG task, including:

[0025] The hybrid graph neural network model in the DJOMD-PPO algorithm is used to extract features from S3 and S4 to obtain the first feature; wherein the hybrid graph neural network model includes a graph convolutional neural network (GCN) and a graph Gaussian mixture neural network (GMM);

[0026] Use the hybrid graph neural network model to extract features from S1 to obtain the second feature;

[0027] Use the hybrid graph neural network model to extract features from S2 to obtain the third feature;

[0028] The first, second, and third features are fused to output the minimized average delay of the user DAG task.

[0029] Furthermore, the DJOMD-PPO algorithm includes:

[0030] Initialize network parameters and experience replay buffer, and reset the environment state;

[0031] Collect sample data from multiple stem groups according to the strategy and calculate the advantage function

[0032] Optimize and update the first network parameter θ based on the reward function e and the second network parameter ψ e ;

[0033] For a set of sample data, perform P updates;

[0034] The first network parameter θ of the updated strategy e Copy to The second network parameter ψ of the updated value network e Copy to

[0035] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0036] In an embodiment of the present disclosure, through the above-mentioned task joint offloading and migration method for air-ground collaborative mobile edge computing, the method takes into account the scenarios of resource heterogeneity, limited resources, limited communication range and dynamic network fluctuations under edge computing composed entirely of drone-vehicle networking; the method models the joint offloading and migration problem of DAG tasks and expresses it as the problem of minimizing the average delay of user DAG tasks; the method takes into account the different characteristics of UAVs and vehicles, and the conditions for triggering task migration are different; the method takes into account the situation where the same DAG task may be migrated multiple times, and considers the shortest transmission path for task migration, task upload, and calculation result download. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0038] Figure 1 A diagram showing the steps of a method for joint task offloading and migration for air-ground collaborative mobile edge computing in an exemplary embodiment of the present disclosure;

[0039] Figure 2 A scene diagram in an exemplary embodiment of the present disclosure is shown;

[0040] Figure 3 Another scene diagram in an exemplary embodiment of the present disclosure is shown;

[0041] Figure 4 A framework diagram of the DJOMD-PPO algorithm in an exemplary embodiment of the present disclosure is shown;

[0042] Figure 5 A performance graph of the DJOMD-PPO algorithm is shown in an exemplary embodiment of the present disclosure when 20 users receive services from an edge server;

[0043] Figure 6 A performance graph of the DJOMD-PPO algorithm is shown in the case where 15 users receive services from the edge server in an exemplary embodiment of the present disclosure;

[0044] Figure 7 A performance graph of the DJOMD-PPO algorithm is shown in an exemplary embodiment of the present disclosure when 10 users receive services from an edge server;

[0045] Figure 8 FIG. 4 shows a performance diagram of the DJOMD-PPO algorithm when five users receive services from an edge server in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0047] In addition, the accompanying drawings are merely schematic illustrations of embodiments of the present disclosure and are not necessarily drawn to scale. Like reference numerals in the figures represent like or similar parts, and thus repeated descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically separate entities.

[0048] This example embodiment provides a task joint offloading and migration method for air-ground collaborative mobile edge computing. Figure 1 As shown in , the task joint offloading and migration method for air-ground collaborative mobile edge computing may include: steps S101 to S104.

[0049] Step S101: Constructing an MEC scenario; wherein the MEC scenario includes M edge servers, K users, and 1 SDN controller, and the edge servers include drones or vehicles;

[0050] Step S102: Construct an MDP problem model based on the joint offloading and migration problem of the user's DAG tasks in the MEC scenario;

[0051] Step S103: Convert the MDP problem model into a Markov decision process and<S,A,P,R> Represented by; where S is the state set of the Markov decision process, A is the action set of the Markov decision process, P is the state transition probability of the Markov decision process, and R is the reward of the Markov decision process;

[0052] Step S104: using the DJOMD-PPO algorithm to solve the Markov decision process to obtain the minimized average delay of the user DAG tasks.

[0053] Through the above-mentioned task joint offloading and migration method for air-ground collaborative mobile edge computing, this method takes into account the scenarios of resource heterogeneity, limited resources, limited communication range and dynamic network fluctuations under edge computing composed entirely of drone-vehicle networks; this method models the joint offloading and migration problem of DAG tasks and expresses it as the problem of minimizing the average delay of user DAG tasks; this method takes into account the different characteristics of UAVs and vehicles, and the different conditions for triggering task migration; this method considers the situation where the same DAG task may be migrated multiple times, and considers the shortest transmission path for task migration, task upload, and calculation result download.

[0054] Below, we will refer to Figures 1 to 8 Each step of the above-mentioned task joint offloading and migration method for air-ground collaborative mobile edge computing in this example implementation is described in more detail.

[0055] In step S101 and step S102, the constructed drone-vehicle MEC environment is provided with M mobile servers to serve the task requests of K users. The M servers in the Mobile Edge Computing (MEC) system are represented as U = {U1,…,U M},U m ={r m ,f m ,kind m ,loc m ,h m ,p m ,θ m ,rad m}, where r m ∈{0,1,…,X} means that each server has different heterogeneous resources, and different values ​​represent different resource types; f m Indicates the CPU clock frequency of the server; kind m is a binary variable, kind m 0 means it is a drone, kind m 1 represents a vehicle; loc m Represents the horizontal coordinate of the server; h m Represents the vertical height of the server; p m represents the transmission power of the server; θ m Indicates the computing power of the server, that is, the number of CPU cycles required to process each bit of data; rad m Indicates the radius of the server coverage. Each user can be represented by a tuple as <loc k ,p k ,G k >, where loc kIndicates the two-dimensional coordinates of the user, p k represents the transmission power of user k, G k =(N k ,E k ) represents the user's DAG task request, where N = {n1, n2, ..., n j} is a collection of subtasks, each subtask n j ∈N consists of triples <input j ,resource j ,output j > indicates input j Represents subtask n j Input data size, resource j Represents subtask n j The resource type required for execution, output j Represents subtask n j The processing result data size, E is the set of data flow edges. i,j (e i,j ∈E), task i is the predecessor of task j, and task j is the successor of task i. Due to task dependency, a task can only be executed after receiving the processing results of all its predecessor tasks, where pre(i) represents the set of predecessor tasks of task i. Figure 2 Taking request 2 of the DAG as an example, the set of predecessor tasks of task 5 is pre(5)={3, 4}, so task 5 must wait for data from tasks 3 and 4 before it can be executed.

[0056] The inter-arrival interval of user task requests follows a Poisson distribution with an inter-arrival interval of λ. When a user request is generated, the SDN controller decides to assign the first subtask of the request to a specific mobile MEC server that has the resources required for that subtask. Each time a subtask is completed, the SDN controller also determines the target MEC server for the next subtask. The following defines the considered scenario in more detail.

[0057] Task scheduling plan:

[0058] For a user request G with n subtasks k , the scheduling plan is defined as in represents the scheduling decision at time step t. Specifically, Indicates that task i is scheduled to computing node m. Figure 1 As shown, the scheduling decision at time step 0 is as follows: (1) Task 1 of user request 1 is scheduled to ES1, and (2) Task 1 of user request 2 is scheduled to ES9. To better illustrate, user request Gk The task schedule at time step t is defined as in Represents the scheduling plan of task i, using a one-dimensional vector Indicates that y m represents the mth server, y m ∈{0,1},y m =1 means task i is executed on the mth server, It means that task i has not started executing yet or is executing on a certain server at time t. According to the scheduling decisions at different time steps, the final scheduling plan can be obtained. In addition, T t represents the delay of the user request at time step t, where T t The detailed calculation process of will be given below. Based on the above content, the response time of user request is defined as T resp , which represents the time interval from task submission to completion of the last subtask.

[0059] The calculation process of the response time of user DAG requests is as follows:

[0060] As mentioned above, T t represents the delay of a user request at time step t. Since subtasks may be executed in parallel on different mobile servers, T t represents the maximum end time of all completed tasks at time step t. Therefore, T t Can be defined as

[0061]

[0062] Among them D t is the set of completed tasks at time step t. end (i) is the end time of task i, defined as

[0063] t end (i) = t start (i)+t exe (i)+t mig (i)+t down (i)

[0064] where t exe (i), t start (i), t mig (i), t down (i) are the execution time, start time, migration time, and download time of task i, respectively.

[0065] Task start time:

[0066] The start time of task i depends on two aspects. First, task i should receive the execution results of all its predecessor tasks. Second, the server that executes task i should be available. Therefore, the start time of task i is defined as

[0067]

[0068] Among them, t avl (m) is the time when node m becomes available. tran (i, j) represents the time required to transfer the intermediate data of task i’s predecessor task j to the execution server of task i, which is similar to the offloading delay of traditional service migration. At time t, it is defined as

[0069]

[0070] Set Path l,m (t) represents the data transmission path from server l to server m. Task i and its predecessor task j are scheduled to nodes m and l respectively. If points m and l are the same node, there is no data transmission time because data transmission can be achieved through shared memory. When i is the first task requested by the user, l represents the user who initiated the request. i,j is the data transferred from task i to j, that is, the data size after task i is executed. TR s,s+1 It represents the transmission rate of the channel between nodes s and s+1. The calculation formula of the transmission rate is shown in the communication model.

[0071] Task execution time:

[0072] The execution time of task i depends on the computing power of server m and the computational load of task i. However, considering that the server may not be able to continue processing the task due to its own reasons, which will trigger task migration, this application considers that after the task migration, the site can be restored to continue running. The migrated data is the intermediate data, that is, task i may be executed on multiple nodes. For the convenience of calculation, it is defined as

[0073]

[0074] Among them, c m Indicates the amount of task data processed on server m.

[0075] Migration trigger conditions and task migration time:

[0076] In this application, the conditions that trigger migration can be mainly divided into two categories.

[0077] The first category is to consider the endurance of drones. The drone is currently processing a task. Because the drone is about to run out of power (the power level is lower than the threshold α), it needs to migrate the current task to another server with corresponding resources. The power prediction formula of drone m is as follows:

[0078]

[0079] Among them, E m (t) is the remaining power of drone m at time t, is the initial charge of drone m, is the hovering power consumption of the drone, It is the CPU power consumption of the drone, which is a function of the CPU frequency.

[0080] The challenge posed by this application when drones trigger migrations due to battery issues is determining an algorithm for selecting the appropriate target server based on location, network access, and load. Research indicates that existing work selects migration targets based on a partial combination of location, network access, and load. This application, however, considers all of these factors, making it challenging.

[0081] Figure 3 This diagram illustrates the first migration scenario. As the drone's battery runs low, it needs to dynamically find a new server with the necessary resources. It then migrates intermediate data and any remaining unprocessed data to the new server for continued processing. When selecting a new target server, consider its location, computing power, and network bandwidth.

[0082] The second category is vehicles that are processing tasks. Since the vehicles are temporarily parked on the roadside, they may move away at any time. At this time, considering the limited coverage of the SDN central controller and the fast movement speed of the vehicles, once they move out of the coverage of the SDN central controller, the calculation results cannot be returned to the user. Therefore, it is necessary to predict the vehicle's movement trajectory and determine whether to make a migration decision based on whether the predicted vehicle position exceeds the coverage of the SDN central controller.

[0083] The challenge posed by the migration of roadside parked vehicles due to movement is how to make a reasonable trade-off between relay backhaul and migration. This application considers that the vehicle can complete the task before moving out of the coverage of the SDN central controller. Considering that migration requires additional overhead, there is no need to migrate at this time; otherwise, the task should be migrated in advance before the vehicle leaves the SDN central controller. A key difficulty is how to determine how long the vehicle moves before triggering migration, because the vehicle movement trajectory is complex. To simplify the problem, and this application focuses on the unloading / migration decision, it is assumed that the vehicle movement trajectory is known.

[0084] Task i may trigger multiple migrations. The migration time of task i at time step t is defined as follows:

[0085]

[0086] Here, set represents the set of servers that have hosted task i.

[0087] Task download time:

[0088] When the last task requested by the user is completed, the calculation results need to be sent to the user, so the task download time is defined as follows:

[0089]

[0090] Among them, res(i) represents the size of the calculation result of task i, v represents the server node hosting the task, and u represents the user.

[0091] Communication model:

[0092] In this application, only large-scale fading, i.e., communication path loss, is considered, and small-scale fading or fast fading is not considered, because the time scale of decision making is not small. Every time a decision is made, it remains valid until the next decision time. Without loss of generality, this application assumes that the decision will not be changed within a single time slot, so no decision will be made within the time scale of fast fading. For air-to-ground (A2G) communication, the path loss between the drone and the ground node is given by:

[0093]

[0094] Where h and r represent the altitude of the drone and the horizontal distance between the drone and the ground terminal device, respectively. f (unit Hz) is the carrier frequency, and c (unit m / s) is the speed of light. LoS represents the line-of-sight transmission probability of the A2G link, which is given by:

[0095]

[0096] where (a, b, η LoS ,η NLoS ) is a parameter related to the environment, which is equal to (4.88, 0.43, 0.1, 21) in the suburban environment.

[0097] Based on the path loss expression, the Ground-UAV data rate and UAV-Ground data rate are given by the following two equations:

[0098]

[0099] Among them, B represents the bandwidth of WiFi, EB and E U Represent the transmission power of ground equipment and UAV respectively, σ 2 represents the power of Gaussian noise.

[0100] Air-to-air (A2A) communication is mainly based on line-of-sight transmission, so the free-space path loss model is used, as shown below:

[0101] PL A2A (d) = 32.45 + 20logf c +20logd

[0102] where f c is the carrier frequency (MHz), and d is the distance between UAVs (km). Therefore, the UAV-UAV data transmission rate is given by:

[0103]

[0104] Ground-to-ground (G2G) communication is primarily based on line-of-sight transmission, so the free-space path loss model is used. The path loss for G2G communication is shown below:

[0105]

[0106] where f G is the carrier frequency (Hz), and d is the distance between ground devices (m). Therefore, the GROUND-GROUND data transmission rate is given by:

[0107]

[0108] Based on the above description, considering that the transmission rate involves different types of channels, the summary is as follows: Let TR a,b represents the actual transmission rate of the channel between source node a and destination node b, which is a nonlinear function as follows:

[0109]

[0110] Among them E a , σ 2 , B, and PL represent the transmission power, noise power, channel bandwidth, and composite fading vector of source node a, respectively. The fading vector PL is determined based on the source node a and the target node b. If both the source node and the target node are drones, PL = PL_A2A; if both the source node and the target node are ground equipment, PL = PL G2G Otherwise PL=PL A2G .

[0111] Problem Definition:

[0112] In order to improve user QoS, the optimization goal of this application is to minimize the response time of all user DAG task requests. The objective function is defined as:

[0113]

[0114] Given the volatility of networks and the complexity of DAG task scheduling, choosing a task offloading / migration decision solution presents a combinatorial explosion. Traditional heuristic algorithms take a long time to find a suitable migration solution, which doesn't fully meet user requirements. To address this issue, deep reinforcement learning (DRL) is employed to implement adaptive offloading / migration decisions for user DAG tasks in dynamic network environments. This means that the DRL stored in the SDN controller at each moment will make the appropriate decision.

[0115] In step S103, the problem is transformed into a Markov decision process:

[0116] First, this application transforms the average delay minimization problem defined above into a Markov decision process consisting of a four-tuple<S,A,P,R> Indicates that each element is explained as follows:

[0117] a) State set: S is the state set of the Markov decision process. The design of the state is crucial. In order to reduce the difficulty of the intelligent agent in understanding the environment, it is better to extract and classify the environmental information and use different neural network modules for feature extraction instead of directly inputting all environmental information into the neural network. To this end, this application defines the state set S as It includes three elements: S1 represents the user basic information domain, which contains the coordinates of all users and the CPU cycles required for all user tasks; S2 represents the user DAG task information domain, which contains the input and output data size and required resource type of each subtask in all user DAG requests; S3 represents the status of the edge server composed of drones and roadside parked vehicles, which includes the geographic location of all drones / vehicles, the computing power of all drones / vehicles, the current power of all drones, the computing power of all drones / vehicles, the CPU clock frequency of all drones / vehicles, and the resource types owned by all drones / vehicles; S4 represents the network bandwidth size of each link in the network link of the drone-vehicle edge computing network.

[0118] b) Action set: A, each action a in the action set (space) t It is an M×N two-dimensional decision matrix The value of each element in this two-dimensional decision matrix can only be 0 and 1, for example i∈M,j∈K, which reflects that the decision made by the central controller for user j at time t is served by server i. Similarly, it ensures that a user's DAG request is served by at most one server at the same time.

[0119] c) State transition probability: P:S×A×S→[0,1] represents the state transition probability distribution, and p0:S→[0,1] is the distribution of the initial state s0. State s0 is changed by taking action a t Based on the probability P(s t+1 |s t ,a t ) from s t Transfer to s t+1 .

[0120] d) Reward: r t : represents the immediate reward received by the central controller after making a decision for the user request at time t. The goal of the central controller is to minimize the average delay of all users' DAG requests by solving the constructed MDP problem. That is, the goal of solving the MDP is to find the optimal action at each step so that the expected cumulative reward R episode Maximum, so the reward function is set as follows:

[0121]

[0122] The reward function is considered in two cases: 1. At the end of the round, that is, when all users' DAG requests are executed, the average delay of all users' DAG requests can be calculated, so the reward setting is modeled based on the average delay; 2. In other cases, the reward is set to 0.

[0123] In step S104, the characteristics of the problem raised in this application are analyzed and solved using the PPO algorithm.

[0124] The PPO algorithm is a policy-based reinforcement learning algorithm. It is an improvement on the policy gradient (PG) algorithm and retains the advantages of excellent performance in continuous state space and continuous action space.

[0125] In PPO, for a network structure where the policy network and the value network parameters do not share, the objective function is usually optimized for the policy and value networks respectively. The following formula is the objective function of the policy network, θ e are the parameters of the policy network:

[0126]

[0127] The objective function of the value network is as follows:

[0128]

[0129] where ψ e Represents the parameters of the value network. is the state value estimated by the value network. V target is the actual return, usually estimated by sampling trajectories. is the mean squared error of the value function.

[0130] For Gym environments or some simple scenarios, the Actor network or Critic network of the traditional PPO algorithm is generally built based on a multi-layer perceptron (MLP), and the neural network model is relatively simple. In the joint DAG task offloading / migration scenario considered in this application, based on the state space and scenario characteristics defined in step 1, we proposed a set of neural network models suitable for computing task migration, such as Figure 4 As shown. First, considering that the MEC network composed of drones / vehicles has a dynamic network and each drone / vehicle has computing resources and other characteristic attributes, the data is naturally presented as a graph structure. The points in the graph represent each drone / vehicle, and each edge in the graph represents the bandwidth value of the wireless communication network of the two servers. Based on this feature, it is reasonable to use a graph neural network to extract edge server features. This application proposes a hybrid graph neural network model that combines a graph convolutional neural network GCN and a graph Gaussian mixture neural network GMM. It extracts features from S3 and S4 defined by the state in the Markov decision process. By combining different neural network structures, this application can better capture the complex characteristics of the network, thereby improving decision-making performance. Then, the user's basic information S1 is represented by a low-dimensional vector, and feature extraction can be performed through a simple MLP. The user's DAG task information S2 is represented by a low-dimensional vector, and feature extraction can be performed through a simple MLP. The extracted features are then fused to output the corresponding action or value function. This application names the PPO algorithm DJOMD-PPO (PPO-Based DAGTaskJointOffloadingandMigrationDecisionAlgorithm).

[0131] The pseudo code of the DJOMD-PPO algorithm is shown in Table 1. It describes the training process of the DJOMD-PPO algorithm in detail. First, the network parameters and the experience replay buffer are initialized and the environment state is reset. After collecting enough samples according to the strategy, the advantage function is calculated. Then optimize and update the network parameters θ based on the reward function e , ψ e For a set of sample data, P updates are required. Finally, the network parameters θ of the updated strategy are e Copy to The parameter ψ of the value network eCopy to

[0132] Table 1 Pseudocode of DJOMD-PPO algorithm

[0133]

[0134] In a specific embodiment, the present application conducted a large number of simulation experiments. The experimental settings and numerical results will be introduced below to demonstrate the effectiveness of the proposed solution.

[0135] Experimental setup

[0136] The experiments in this paper were conducted on an Intel(R) Xeon(R) Gold 6148CPU@2.40GHz, 128G memory, and GeForce RTX3090 device based on Python 3.8 and Pytorch 1.10.

[0137] This paper builds a virtual simulation environment and deploys 25 drones and vehicle servers in a 2500m*2500m square area. The servers can fully cover the area, and users can enjoy the services provided by the servers in the area. At each time t, it detects which users' DAG tasks need to be offloaded or migrated.

[0138] Drone server settings: Assume that the drone's CPU frequency is 5 GHz, the drone's communication radius is 500 meters, the drone's computing power is randomly generated in the range [1500 CPU Circles / byte, 2000 CPU Circles / byte] according to a uniform distribution, the drone's transmission power is 20 mW, and the drone's flight altitude is 70 meters.

[0139] Vehicle server settings: Assume that the vehicle's CPU frequency is 5 GHz, the communication radius is 500 meters, the computing power of the vehicle server is randomly generated in the range [1500 CPU Circles / byte, 2000 CPU Circles / byte] according to a uniform distribution, and the transmission power of the vehicle server is 20 mW.

[0140] Edge server network settings: The network rate unit of the communication link of the edge computing network composed of drones and vehicle servers is MHZ, which is randomly generated according to the normal distribution with mean μ=μ1 and standard deviation σ=σ1. μ1 and σ1 are expected to be set according to the specific experiment. By default, μ1=80 and σ1=25.

[0141] User settings: The CPU cycles requested by the user's DAG task are randomly generated in the range [1.5e8, 2e8]. The input data size of the task is randomly generated in the range [3MB, 5MB]. The calculation result size is randomly generated in the range [1.5MB, 2.5MB]. The computing power of the user's mobile device is 1.5mW.

[0142] For the designed DJOMD-PPO algorithm, we consider different user scales, namely 5 users, 10 users, 15 users, and 20 users receiving services from the edge server. Figures 5 to 8 As shown in the figure, it is the performance diagram of the DJOMD-PPO algorithm under different user scales; Figure 5 Performance graph of the DJOMD-PPO algorithm for 20 users receiving services from the edge server; Figure 6 Performance graph of the DJOMD-PPO algorithm for 15 users receiving services from the edge server; Figure 7 Performance graph of the DJOMD-PPO algorithm for 10 users receiving services from the edge server; Figure 8 Figure 2 shows the performance of the DJOMD-PPO algorithm when 5 users are served by the edge server.

[0143] Through the above-mentioned task joint offloading and migration method for air-ground collaborative mobile edge computing, this method takes into account the scenarios of resource heterogeneity, limited resources, limited communication range and dynamic network fluctuations under edge computing composed entirely of drone-vehicle networks; this method models the joint offloading and migration problem of DAG tasks and expresses it as the problem of minimizing the average delay of user DAG tasks; this method takes into account the different characteristics of UAVs and vehicles, and the different conditions for triggering task migration; this method considers the situation where the same DAG task may be migrated multiple times, and considers the shortest transmission path for task migration, task upload, and calculation result download.

[0144] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly indicate the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0145] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0146] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A task joint offloading and migration method for air-ground collaborative mobile edge computing, characterized in that: The method includes: Build an MEC scenario; the MEC scenario includes M edge servers, K users, and 1 SDN controller. The edge servers include drones or vehicles. Based on the joint offloading and migration of users' DAG tasks in MEC scenarios, an MDP problem model is constructed; The MDP problem model is transformed into a Markov decision process, and the four-tuple<S,A,P,R> Represented as follows; where S is the state set of the Markov decision process, A is the action set of the Markov decision process, P is the state transition probability of the Markov decision process, and R is the reward of the Markov decision process; state set , Indicates the user basic information domain, including the coordinates of all users and the CPU cycles required for all user tasks. Represents the user DAG task information domain, including the input data size, output data size, and required resource type of each subtask in all user DAG requests. The state of the edge server composed of drones and vehicles, including the geographic location of all drones and vehicles, the computing power of all drones and vehicles, the current power of all drones, the CPU clock frequency of all drones and vehicles, and the resource types owned by all drones and vehicles. Represents the network bandwidth of each link in the UAV-Vehicle Edge Computing Network; The DJOMD-PPO algorithm is used to solve the Markov decision process to obtain the minimized average delay of the user DAG tasks. Specifically, it includes: Using the hybrid graph neural network model in the DJOMD-PPO algorithm and Perform feature extraction to obtain a first feature; wherein the hybrid graph neural network model includes a graph convolutional neural network (GCN) and a graph Gaussian mixture neural network (GMM); Using hybrid graph neural network model Perform feature extraction to obtain a second feature; Using hybrid graph neural network model Perform feature extraction to obtain a third feature; The first, second, and third features are fused to output the minimized average delay of the user DAG task.

2. The task joint offloading and migration method for air-ground collaborative mobile edge computing according to claim 1 is characterized in that: In the MEC scenario, each user corresponds to a DAG task. Each subtask of the DAG task is processed by different resource types. The user's DAG tasks arrive in a Poisson distribution over time. After the DAG task arrives, its predecessor subtasks are offloaded to the edge server with corresponding resources in topological order for processing. After the preceding subtask is processed, the subsequent subtask begins to be processed. Each edge server processes one task, and the SDN controller makes an offloading decision at the beginning of each moment. When the edge server cannot process the corresponding subtask, the SDN controller makes a migration decision to migrate the subtask to another edge server with corresponding resources.

3. The task joint offloading and migration method for air-ground collaborative mobile edge computing according to claim 2 is characterized in that: The objective function of the MDP problem model is: in, k Indicates the k users, The response time for user requests.

4. The task joint offloading and migration method for air-ground collaborative mobile edge computing according to claim 3 is characterized in that: Each action in the action set A It is an M×N two-dimensional decision matrix , the value of each element in the two-dimensional decision matrix is ​​0 or 1, Indicates that the SDN controller does not make a decision at time t. Indicates that at time t, the SDN controller is user The decision is made by the edge server Provide services, and at the same time, a user's DAG task request is served by one edge server; State transition probability P: [0, 1] represents the state transition probability distribution, is the initial state The distribution of the initial state By taking action Based on probability from Transfer to ; Reward R: : represents the immediate reward received by the SDN controller after making a decision for the user request at time t; The goal of the model to solve the MDP problem is to find the optimal action at each step so that the expected cumulative reward Maximum, the reward function is set as follows: When the round ends, that is, when all users' DAG requests are executed, the average latency of all users' DAG requests is calculated, and the reward setting is modeled based on the average latency; in other cases, the reward is set to 0.

5. The task joint offloading and migration method for air-ground collaborative mobile edge computing according to claim 4 is characterized in that: The DJOMD-PPO algorithm includes: Initialize network parameters and experience replay buffer, and reset the environment state; Collect sample data from multiple stem groups according to the strategy and calculate the advantage function ; Optimize and update the first network parameters based on the reward function and the second network parameters For a set of sample data, perform P updates; The first network parameter of the updated strategy Copy to , the second network parameter of the network after the update value Copy to .

Images