Real-time dependent task offloading method based on deep reinforcement learning

Abstract

The application relates to a real-time dependent task offloading method based on deep reinforcement learning, which comprises the following steps: step S1: based on a system model of task offloading, training an offloading operation Q value prediction model in a runtime environment by using a DQN algorithm; step S2: the offloading operation Q value prediction model predicts the Q values of different offloading operations according to the computing capacity of the computing nodes, the transmission rate between the computing nodes and the offloading scheme of the application, and then selects a suitable offloading operation by comparing the corresponding Q values; and step S3: repeating step S2, gradually determining the execution position of each task through feedback iteration. The application can well adapt to different cloud edge environments and efficiently generate an offloading scheme.

Description

Technical Field

[0001] This invention relates to the field of cloud-edge collaborative computing, specifically to a real-time dependency task offloading method based on deep reinforcement learning. Background Technology

[0002] With the rise of intelligent technologies, an increasing number of computationally intensive applications (such as autonomous driving, facial recognition, and augmented reality) are being developed to meet people's needs. Furthermore, the computing platforms for these applications are no longer limited to smartphones and laptops, but have gradually expanded to wearable devices, vehicles, drones, and other intelligent devices. Although these mobile devices are becoming increasingly powerful, they are still constrained by size and weight, limiting their processing power, memory capacity, and battery capacity. Most mobile devices are still unable to handle the ever-increasing number of computationally intensive tasks in a short period of time.

[0003] Compute offloading is an effective way to solve the problem of limited resources on mobile devices. It involves sending computationally intensive tasks from the local machine to a remote device for execution, leveraging remote resources to extend local resources. Computational resources are distributed across mobile devices, edge servers, and cloud servers, with significant differences in resource volume across each platform. For an application, the available computational resources are often scattered across multiple different platforms and dynamically change with location. The application's offloading scheme determines which computational tasks should run on which platform, but no single offloading scheme is the final solution. When the application's runtime environment changes, the offloading scheme needs to be adjusted accordingly to provide better performance. Therefore, runtime decisions are needed on whether to offload computational tasks and on which platforms. Existing research mostly uses heuristic or search algorithms to find suitable offloading schemes, which can take tens or even hundreds of seconds. Considering the mobility of mobile devices, application runtime environments will frequently change, and selecting a suitable offloading scheme faces a combinatorial explosion problem. Therefore, finding efficient offloading scheme generation methods to meet the real-time requirements of computational offloading remains a highly challenging research area. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a real-time dependency-based task unloading method based on deep reinforcement learning, which can efficiently generate unloading schemes to meet the real-time requirements of computational unloading.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A real-time dependency-based task offloading method based on deep reinforcement learning includes the following steps:

[0007] Step S1: Based on the system model of task unloading, train the Q-value prediction model of unloading operation using the DQN algorithm in the runtime environment;

[0008] Step S2: Unloading operation Q-value prediction model. Based on the computing power of computing nodes, the transmission rate between computing nodes and the application's unloading scheme, predict the Q-value of different unloading operations, and then select the appropriate unloading operation by comparing their corresponding Q-values.

[0009] Step S3: Repeat step S2, and gradually determine the execution position for each task through feedback iteration.

[0010] Furthermore, the system model for task unloading includes a system model and a task model, specifically:

[0011] The system model includes mobile devices (MD), edge servers (ES), and cloud servers (CS). The set of computing nodes is represented by V = {MD, ES, CS}, and the computing power of each node is denoted by f. k (k∈V) represents the data transfer rate between different computing nodes, denoted by v. k，l (k, l∈V) represents;

[0012] The task model is as follows: an application is represented by a directed acyclic graph G = (N, E), where N = {1, 2, ..., n} represents the set of subtasks, n is the number of subtasks, and the computational cost of each task is denoted by c. i (i∈N) represents; E={e i，j |i, j∈N, i<j} represents the set of directed edges that depend on subtasks. For each edge e i，j For directed edges ∈ E, subtask i is called the direct predecessor of subtask j, and subtask j is called the direct successor of subtask i; furthermore, for each e i，j Directed edges ∈ E and their weights d i，j Related, d i，j This represents the amount of data transferred from subtask i to subtask j; pre(i) and suc(i) represent the set of direct predecessor tasks and the set of direct successor tasks of subtask i, respectively. A subtask can only start execution after receiving the processing results of all its predecessor tasks.

[0013] Furthermore, define a binary variable x. ik Let x represent the uninstallation scheme. ik =1 indicates that subtask i is assigned to computing node k, otherwise x ik =0; Since each subtask can only be assigned to one computing node in the network, the following definition applies:

[0014]

[0015] Furthermore, for any subtask j∈N, two conditions must be met before execution can begin: first, the assigned computing node must be available, meaning no other subtasks are currently executing on that node; second, the available time of the node assigned to subtask j must be sufficient. The following constraints must be met:

[0016]

[0017] in Let i be the completion time of subtask i;

[0018] Second, subtask j should be ready, meaning it has received the processing results of all preceding subtasks. The ready time for task j is... Defined as:

[0019]

[0020] If subtask j and one of its predecessor subtasks i∈P(j) are assigned to different computing nodes k and l respectively, communication latency needs to be considered. In this case, the second term on the right-hand side of the constraint will be zero;

[0021] Taking both conditions into account, the start time of subtask j is defined as:

[0022]

[0023] The end time of subtask j is defined as:

[0024]

[0025] Use D 1：t This represents the set of all subtasks that were successfully completed at time step t.

[0026] The cumulative execution delay T of the application at time step t 1：t Defined as:

[0027]

[0028] An application is considered complete if and only if all of its n subtasks complete successfully. When all n subtasks of an application complete successfully, then D... 1：n = {1, 2, ..., n}; therefore, the total execution latency T of the application is... 1：n Calculated using the following formula:

[0029]

[0030] For an application with n tasks, DEP = (dep(1), dep(2), ..., dep(n)) represents the application's unloading scheme; where dep(i) ∈ {1, 2, 3} represents the execution location of task i ∈ N, namely terminal device, edge server and cloud server respectively.

[0031] The objective function is defined as:

[0032] Minimize T 1：n .

[0033] Furthermore, the DQN algorithm acquires state s in the cloud-edge environment during runtime, selects action a through an ε-greedy policy, and then receives the reward value r obtained after the environment executes action a, as well as the next state s′. Next, the DQN algorithm stores (s, a, r, s′) obtained at each step into an experience storage pool. Typically, the capacity of the experience storage pool in the DQN algorithm is pre-set. When the storage threshold is reached, the neural network parameters are updated. The loss function of the neural network is as follows:

[0034] Loss=(r+γmaxQ(s′,a′;ω′)-Q(s,a;ω)) 2

[0035] Where γ is the discount factor; Q(s, a; ω) is the output of "EvalNet", which calculates the Q value of the current state action pair, and ω is the DNN weight of "EvalNet"; maxQ(s′, a′; ω′) is the output of "TargetNet", which calculates the maximum Q value when the action a′ is performed in the next state s′, and ω′ is the DNN weight of "TargetNet".

[0036] Furthermore, step S2 specifically includes:

[0037] First, randomly initialize the weights ω of the EvalNet neural network and the weights ω′ of the TargetNet neural network (row 3);

[0038] For each training cycle, the current offloading scheme DEP cur Both the current state s and the current response time T will be initialized;

[0039] During the algorithm training process, the execution position of each subtask is determined sequentially through the ε-greedy strategy. One subtask is randomly selected from all the unloading schemes with a probability of ε, and the unloading scheme with the largest Q value in EvalNet is selected with a probability of 1-ε.

[0040] Next, perform action a and obtain a new response time T′, calculate the reward r and update the current response time T, and observe the next state s′;

[0041] Then, (s, a, r, s′) is placed into the experience storage pool, and m samples are randomly selected from the experience pool to calculate the target value;

[0042] Next, the Loss is obtained based on the mean squared error loss function, and the weights ω of the EvalNet neural network are updated using the Adam optimizer. After reaching the set C rounds of iterations, the weights ω' = ω of the TargetNet neural network are updated.

[0043] Finally, update the current status.

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] This invention can adapt well to different cloud edge environments and efficiently generate offloading schemes. Attached Figure Description

[0046] Figure 1 This is a system model for task unloading in one embodiment of the present invention;

[0047] Figure 2 This is an example of a task unloading process in one embodiment of the present invention;

[0048] Figure 3 This is a diagram of the DQN algorithm architecture in one embodiment of the present invention;

[0049] Figure 4 This is a DAG with different task sizes in one embodiment of the present invention;

[0050] Figure 5 This is a performance comparison of DODQ and the ideal solution in different scenarios in one embodiment of the present invention;

[0051] Figure 6 This invention relates to the correctness of the uninstallation operation under different task scales in a DODQ scenario.

[0052] Figure 7 This is a performance comparison of DODQ with other classic methods in different scenarios in one embodiment of the present invention. Detailed Implementation

[0053] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0054] In this embodiment, reference Figure 1 The system model for task offloading consists of a mobile device (MD), an edge server (ES), and a cloud server (CS). The set of compute nodes is represented by V = {MD, ES, CS}, and the computing power of each compute node is denoted by f. k(k∈V) represents the value. Typically, due to size and weight constraints, mobile devices have the weakest computing power. Edge servers have stronger computing power than terminal devices, but weaker than remote cloud servers. The data transfer rate between different computing nodes is denoted by v. k，l (k, l∈V) indicates that, since the edge server is deployed close to the mobile device, while the remote cloud server and the mobile device are located far apart, the data transmission rate between the mobile device and the edge server is faster than the data transmission rate between the remote cloud server and the mobile device and the edge server.

[0055] like Figure 1 As shown, an application can be represented by a directed acyclic graph G = (N, E), where N = {1, 2, ..., n} represents the set of subtasks, n is the number of subtasks, and the computational cost of each task is denoted by c. i (i∈N) represents; E={e i，j |i, j∈N, i<j} represents the set of directed edges that depend on subtasks. For each edge e i，j For directed edges ∈ E, we say that subtask i is the direct predecessor of subtask j, and subtask j is the direct successor of subtask i. Furthermore, for each e... i，j Directed edges ∈ E and their weights d i，j Related, d i，j This represents the amount of data transferred from subtask i to subtask j. We use pre(i) and suc(i) to represent the set of direct predecessor tasks and the set of direct successor tasks of subtask i, respectively. A subtask can only begin execution after receiving the processing results of all its predecessor tasks. For example, the set of direct predecessor tasks and the set of direct successor tasks of a subtask are pre(9) = {2, 4, 5} and suc(9) = {10}, respectively. Therefore, subtask 9 must receive the processing results of subtasks 2, 4, and 5 before it can begin execution. Furthermore, for subtask 10, which has no direct successor tasks, we call subtask 10 the final task.

[0056] In this embodiment, we can process the task locally or offload the subtask to an edge server or a remote cloud server for execution. Therefore, we define a binary variable x. ik Let x represent the uninstallation scheme. ik =1 indicates that subtask i is assigned to computing node k, otherwise x ik =0. Since each subtask can only be assigned to one computing node in the network, the following definition applies:

[0057]

[0058] Furthermore, for any subtask j∈N, two conditions must be met before execution can begin. First, the assigned computing node must be available, meaning no other subtasks are currently executing on that node. Second, the availability time of the node assigned to subtask j. The following constraints must be met:

[0059]

[0060] in Let be the completion time of subtask i.

[0061] Second, subtask j should be ready, meaning it has received the processing results from all preceding subtasks. Task j ready time. Defined as:

[0062]

[0063] If subtask j and one of its predecessor subtasks i∈P(j) are assigned to different computing nodes k and l respectively, we need to consider communication latency. Otherwise, data transfer between them can be achieved through shared memory without communication latency. In this case, the second term on the right-hand side of the constraint will be zero.

[0064] Taking both conditions into account, the start time of subtask j is defined as:

[0065]

[0066] Furthermore, the end time of subtask j is defined as:

[0067]

[0068] It's important to note that tasks in a Directed Acyclic Graph (DAG) are executed in parallel; therefore, the application's latency will equal the longest time taken to complete a task in its dependency chain. Using D... 1：t This represents the set of all subtasks successfully completed at time step t. Furthermore, the cumulative execution delay T of the application at time step t is... 1：t It can be defined as:

[0069]

[0070] An application is considered complete if and only if all of its n subtasks complete successfully. When all n subtasks of an application complete successfully, then D... 1：n = {1, 2, ..., n}. Therefore, the total execution latency T of the application is... 1：n It can be calculated using the following formula:

[0071]

[0072] For an application with n tasks, let DEP = (dep(1), dep(2), ..., dep(n)) represent the application's uninstallation scheme. Here, dep(i) ∈ {1, 2, 3} represents the execution location of task i ∈ N, namely the terminal device, edge server, and cloud server, respectively. Since the mobile application is deployed on the terminal device, it is assumed that the first task of the application is executed on the mobile device, i.e., dep(1) = 1.

[0073] Different uninstallation methods (DEP) may lead to T 1：n With different values ​​for T, in order to achieve a better computational offloading effect in a cloud-edge collaborative environment, the optimization objective is to minimize T as much as possible. 1：n Therefore, the objective function is defined as:

[0074] Minimize T 1：n

[0075] Based on the above definitions, the MEC runtime environment consists of 2-tuples.<F,V> Composition. As shown in Table 1.1, where F = (f MD f ES f CS V represents the computing power of different computing nodes. MD，ES v MD，CS v ES，CS The data transfer rate between computing nodes is represented by ). DEP = (dep(1), dep(2), ..., dep(n)) represents the application offloading scheme, where dep(i) represents the execution location of the i-th task. 1：n T represents the application response time under the corresponding uninstallation scheme. 1：n The smaller the value, the better the calculation and unloading effect.

[0076] Table 1.1 Runtime Environment and Application Unloading Scheme Including Performance Metrics

[0077]

[0078] In this embodiment, a real-time dependency-aware task offloading method with Deep Qnetworks (DODQ) based on deep reinforcement learning is provided, including the following steps:

[0079] Step S1: Train the Q-value prediction model for offloading operations using the DQN algorithm in the runtime environment. The training data includes the computing power F of the compute nodes, the data transfer rate V between compute nodes, the application offloading scheme, and the corresponding application latency, as shown in Table 3. The Q-value prediction model for offloading operations can evaluate the Q-value of offloading operations under different runtime environments. Therefore, when the current system state (computing power of compute nodes, data transfer rate between compute nodes, and application offloading scheme) is input, the model can accurately predict the Q-value of different offloading operations.

[0080] Step S2: Unloading operation Q-value prediction model. Based on the computing power of computing nodes, the transmission rate between computing nodes and the application's unloading scheme, predict the Q-value of different unloading operations, and then select the appropriate unloading operation by comparing their corresponding Q-values.

[0081] Step S3: Repeat step S2, and gradually determine the execution position for each task through feedback iteration.

[0082] In this embodiment, DEP is used. t =(dep t (1), dep t (2), ..., dep t (n) represents the unloading scheme applied at time step t, where dept(i)∈{0,1,2,3}(i∈N) corresponds to the execution status of task i at this time. Specifically, when dept... t When (i) = 0, it means that task i has not been executed at time step t. For example... Figure 2 As shown, at this time, DEP3 = (1, 2, 2, 0, ..., 0), indicating that at the 3rd time step, task 1 is executed on the terminal device, task 2 and task 3 are executed on the edge server, and subsequent tasks have not yet been executed.

[0083] Table 2.1 illustrates the application uninstallation process (n=10) in a specific scenario. First, the application's first task is executed on the mobile device, with the corresponding DEP1 = {1,0,0,0,0,0,0,0,0,0}, T 1:1 =0.069s. Next, task 2 is offloaded to the edge server for execution, at which point the corresponding DEP2 = {1,2,0,0,0,0,0,0,0,0}, T 1:2 =0.183s. Similarly, the execution position of each task is determined sequentially. Finally, when DEP... 1:10 When the value is {1,2,2,1,1,3,2,2,2,2}, it means that all tasks in the application have been completed. Therefore, we can obtain the application's response time T. 1:n It takes 0.54 seconds.

[0084] Table 2.1 Application uninstallation process in a certain scenario

[0085]

[0086] Reinforcement learning is an algorithmic model that learns optimally in a specific scenario. It models all real-world problems as interactions between an agent and its environment. At each time step of this interaction, the agent receives the state of the environment and chooses an appropriate response. Then, in the next time step, the agent receives a reward and a new state based on the feedback from the environment. Since the goal of reinforcement learning is to maximize cumulative reward, it is typically modeled using Markov Decision Processes (MDPs). More specifically, an MDP can be defined as a 4-tuple, denoted as […].<S、A、T、R> , where S is the state space, A is the action space, T is the state transition function, and R is the reward function.

[0087] Based on the problem definition, in the proposed computational unloading problem, the corresponding state space S, action space A, state transition function T, and reward function R are defined as follows:

[0088] State Space: The state space is represented by S, where s∈S represents a potential state. To comprehensively consider the characteristics of the operating environment and the unloading scheme, s is defined as a triple. <F,V,DEP cur Therefore, s represents the current system state of the runtime environment, which consists of the computing power of different computing nodes, the data transfer rate between computing nodes, and the current application's offloading scheme.

[0089] Action Space: The action space is represented as A = {a1, a2, a3}, where one action a (a∈A) determines the execution location of the task to be decided. Specifically, a∈{a1, a2, a3} respectively represent executing the current task on the terminal device, offloading the current task to the edge server for execution, and offloading the current task to the cloud server for execution.

[0090] State transition function: The state transition function is represented by T(s, a), and the function returns the next state after performing action a in state s. For example, as shown in Table 2.1, when in state s =<F,V,(1,0,0,...,0)> Execute action a2, next state s′=<F,V,(1,2,0,...,0)> It can be observed that Task 2 is offloaded to the edge server for execution.

[0091] Reward function: To guide the RL agent to minimize the application's response time, the reward function is defined as:

[0092] R(s, a) = TT′

[0093] R(s, a) represents the reward received by the RL agent after performing action a in the current state s, where T corresponds to the application latency under the current offloading scheme, and T′ represents the application latency after performing action a. For example, as shown in Table 2.1, the reward for task 2 being offloaded to the edge server is r = R(s, a) = TT′ = T 1：1 -T 1：2 = -0.114.

[0094] refer to Figure 3 This is a diagram of the DQN algorithm architecture. The DQN agent obtains a state s (s∈S) in the cloud-edge environment during runtime, then selects an action a (a∈A) using an ε-greedy policy. It then receives the reward value r obtained after executing action a and the next state s′. Next, the DQN algorithm stores the (s, a, r, s′) obtained at each step into an experience storage pool. Typically, the capacity of the experience storage pool in the DQN algorithm is pre-set. When the storage threshold is reached, the neural network parameters are updated. The loss function of the neural network is as follows:

[0095] Loss=(r+γmaxQ(s′,a′;ω′)-Q(s,a;ω)) 2

[0096] Where γ is the discount factor; Q(s,a;ω) is the output of "EvalNet", which calculates the Q-value of the current state action pair, and ω is the DNN weight of "EvalNet"; maxQ(s',a';ω') is the output of "TargetNet", which calculates the maximum Q-value when performing action a' in the next state s', and ω' is the DNN weight of "TargetNet".

[0097]

[0098] Based on the above definition, the DQN algorithm is used to evaluate the Q-value of different offloading operations. The key steps of this algorithm are shown in Algorithm 1. First, the weights ω of the EvalNet neural network and the weights ω' of the TargetNet neural network are randomly initialized (row 3). For each training cycle, the current offloading scheme DEP... curThe current state s and the current response time T are initialized (lines 5-6). During algorithm training, the execution position of each subtask is determined sequentially using an ε-greedy strategy. An unloading scheme is randomly selected from all unloading schemes with a probability of ε, and the unloading scheme with the largest Q-value in EvalNet is selected with a probability of 1-ε (line 8). Next, action a is executed, and a new response time T' is obtained. The reward r is calculated, and the current response time T is updated. The next state s' is observed (line 9). Then, (s, a, r, s') is placed in the experience pool, and m samples are randomly selected from the experience pool to calculate the target value. These samples may come from different cloud-edge operating environments to ensure sufficient learning (lines 10-11). Next, the loss is obtained according to the mean squared error loss function, and the weights ω of the EvalNet neural network are updated using the Adam optimizer. After reaching the set C iterations, the weights ω' = ω of the TargetNet neural network are updated (lines 12-14). Finally, the current state is updated (line 15).

[0099] 2.2 Runtime Decisions for Unloading Operations

[0100]

[0101]

[0102] The decision-making process for uninstallation operations takes place at runtime, with the main steps outlined in Algorithm 2. For each task in the mobile application, the current uninstallation scheme is first initialized (lines 9-10). Next, the Q-value of each uninstallation operation is evaluated by calling the Q-value prediction model (lines 12-14). Finally, the uninstallation operation with the largest Q-value is selected, and the uninstallation scheme is updated (lines 15-17). Therefore, the execution position of each task is determined sequentially during the decision-making process.

[0103] Through an iterative feedback control process, the optimal unloading scheme can be executed incrementally in the runtime cloud edge environment. The feedback control operation will continue until all tasks of the mobile application are completed.

[0104] Example 1:

[0105] In this embodiment, to simulate the diversity of applications, four task graphs with task sizes n = {10, 15, 20, 25} were constructed. The structure of the constructed task graphs is as follows: Figure 4 As shown.

[0106] For each application G = (N, E), the computational cost of each subtask is c. i (i∈N) follows a uniform distribution within [50, 500]Mcycle. And for each ei,j For directed edges ∈ E, the data transfer amount from subtask i to subtask j follows a uniform distribution within [0, 1000] KB. Furthermore, the computational power F(f) of different computing nodes... MD ,f ES ,f CS The data transfer rate V(v) between computing nodes MD,ES ,v MD,CS ,b ES,CS The parameters follow a uniform distribution. Table 3.1 lists the detailed settings for the simulation parameters.

[0107] Table 3.1 Simulation Parameter Settings

[0108]

[0109]

[0110] The proposed DODQ is implemented using TensorFlow 2.3.0. DODQ uses a fully connected DNN consisting of one input layer, two hidden layers, and one output layer, where each hidden layer has 128 hidden neurons. The memory pool size M, training batch size m, discount factor γ, and learning rate of the Adam optimizer are set to 15000, 64, 0.9, and 0.001, respectively. Furthermore, different training epochs are set for task graphs of different sizes; the training epochs for task sizes n = 10, 15, 20, and 25 are set to 10000, 15000, 20000, and 25000, respectively.

[0111] Based on the above settings, 10 different cloud edge environment scenarios with varying F and V values ​​were simulated, as shown in Table 3.2. For each scenario, a runtime decision algorithm was executed to achieve adaptive task offloading in different cloud edge environments.

[0112] Table 3.2 Different Scenarios in Cloud Edge Environment

[0113]

[0114] In this embodiment, the effectiveness of DODQ for adaptive task offloading was evaluated under the different scenarios described in Table 3.2. Specifically, we will compare the performance of DODQ with the ideal solution in these scenarios. By combining management experience and local validation, the ideal offloading solution with the shortest response time is obtained in different scenarios. However, finding the ideal solution in practice is not feasible because it requires exhausting all possibilities, resulting in unacceptably high complexity. For example, for an application with n tasks, after the application is launched on the terminal device (i.e., the first task is executed on the terminal device), we need to determine the execution location for the remaining n-1 tasks (i.e., execute the tasks on the terminal device, or offload the tasks to an edge server or cloud server for execution). Therefore, we must consider a total of 3 options from the given F and V. n-1 Performance tests were conducted on each uninstallation scheme. For example... Figure 5 As shown, DODQ achieves response times close to the ideal solution in different scenarios. When n=10, DODQ achieves optimal performance in scenarios 1, 5, 6, and 8. In other cases, the performance gap between DODQ and the ideal solution remains below 4%. These results validate that DODQ achieves optimal or near-optimal task offloading performance in different cloud edge environments.

[0115] Taking scenario 1 (n=10) as an example, the task unloading process using DODQ is illustrated. As shown in Table 3.3, when the initial unloading scheme is DEP... cur = (1,0,0,0,0,0,0,0,0), the predicted Q-value of the unloading operation a=2 is higher than that of other unloading operations. Therefore, the task currently pending decision (i.e., the second subtask) is unloaded to the edge server for execution, hence the unloading scheme DEP. cur The corresponding update is (1,2,0,0,0,0,0,0,0). Next, in each step, select and execute the unloading operation with the highest predicted Q value. Finally, when DEP... cur When the value is (1,2,2,1,1,3,2,2,2), the unloading operation is no longer required, and the decision-making process for task unloading is complete.

[0116] Table 3.3 Calculation and unloading process in Scenario 1 (n=10)

[0117]

[0118]

[0119] In this embodiment, Action Accuracy (AAR) is used to measure the correctness of the unloading operation during the decision-making process, and is defined as:

[0120]

[0121] Where O and A represent the number of all uninstallation operations that need to be performed and the number of correct uninstallation operations, respectively. An uninstallation operation is considered correct if it brings the current uninstallation plan close to the ideal plan.

[0122] like Figure 6 As shown, DODQ can make highly accurate unloading operation decisions under the Q-value prediction model based on DQN. Specifically, the average AAR can reach 94.8% under different task sizes.

[0123] In this embodiment, the performance of DODQ is compared with rule-based, machine learning-based, and Q-learning methods to further evaluate DODQ's advantages in task offloading. Rule-based methods select the optimal DAG partition point and offload the remaining tasks from mobile devices to the cloud. ML-based methods use particle swarm optimization and genetic algorithms (PSO-GA) to search for offloading schemes based on the application's predicted response time under different environments. Q-learning stores each state-action pair and its corresponding Q-value in a Q-table to maximize the cumulative reward of the offloading scheme, which can handle offloading problems with small state spaces well. When the runtime environment changes, Q-learning needs to retrain the task offloading decision model to better adapt to the new environment.

[0124] like Figure 7 As shown, DODQ's response time is 13-30% better than rule-based methods in different scenarios. Rule-based methods involve expert-set segmentation rules, but these predefined rules cannot be well applied to different scenarios. Therefore, rule-based methods cannot effectively adapt to task offloading problems in dynamic cloud edge environments. Especially when the environment is complex (e.g., n=25), DODQ's advantage over rule-based methods becomes very significant. Furthermore, DODQ's performance is 6-8% better than ML-based methods. In particular, we evaluated the prediction accuracy of response time in ML-based methods, and the accuracy was only 72.5% with a 15% model error allowance. This is because ML-based methods require sufficient training data to develop accurate prediction models. However, in the absence of training data, inaccurate predictions lead to inefficient task offloading. In addition, Q-learning achieves response times comparable to DODQ in different scenarios without considering training time. However, when the runtime environment changes, Q-learning needs to retrain the task offloading decision model for the new environment, which will result in excessive training time.

[0125] Furthermore, we evaluated the convergence times of DODQ, Rule-based, ML-based, and Q-learning methods under different task scales, and the results are shown in Table 3.4. Specifically, using Rule-based methods, there is no training time because the rules are predefined. However, as... Figure 7 As shown, rule-based methods perform the worst in reducing response time. Among these methods, ML-based methods utilize DNNs to search for offloading schemes, requiring predictions of the performance of a large number of offloading schemes, resulting in the longest convergence time. In contrast, DODQ adapts well to dynamic cloud edge environments and generates optimal / near-optimal offloading schemes within milliseconds. Furthermore, Q-learning suffers from a high-dimensional state space problem, especially as the task size increases, because it records all state-action pairs and their corresponding Q-values ​​in a Q-table. Simultaneously, when the runtime environment changes, Q-learning needs to retrain the task offloading decision model to achieve better adaptability. These factors contribute to the excessively long convergence time of Q-learning. For example, when the number of tasks is 25, the convergence time of Q-learning exceeds 1000 seconds, which is much longer than DODQ. The above results demonstrate the advantages of DODQ in achieving low response time and high training efficiency.

[0126] Table 3.4 Convergence time of different methods under different task sizes

[0127]

[0128]

[0129] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. A real-time dependency-based task offloading method based on deep reinforcement learning, characterized in that, Includes the following steps: Step S1: Based on the system model of task unloading, train the Q-value prediction model of unloading operation using the DQN algorithm in the runtime environment; Step S2: Unloading operation Q-value prediction model. Based on the computing power of computing nodes, the transmission rate between computing nodes and the application's unloading scheme, predict the Q-value of different unloading operations, and then select the appropriate unloading operation by comparing their corresponding Q-values. Step S3: Repeat step S2, and gradually determine the execution position for each task through feedback iteration; The system model for task unloading includes a system model and a task model, specifically: The system model includes mobile device MD, edge server ES, and cloud server CS, using This represents a set of computing nodes, where the computing power of each node is expressed in terms of... Indicates; the data transfer rate between different computing nodes is represented by express; The task model, specifically, is as follows: an application is represented by a directed acyclic graph. It means that, among them Represents a set of subtasks. The number of subtasks, and the computational cost of each task. express; Represents the set of directed edges that represent dependencies between subtasks. For a given edge... A directed edge is called a subtask. It is a subtask The direct precursor mission, sub-mission It is a subtask The direct successor task; in addition, each Directed edges and weights Related, Indicates from subtask Transfer to subtask Data volume; using and To represent subtasks A subtask has a set of direct predecessor tasks and a set of direct successor tasks. A subtask can only begin execution after it has received the processing results of all its predecessor tasks.

2. The real-time dependency-based task offloading method according to claim 1, characterized in that, Define a binary variable To represent the uninstallation scheme, if Indicates subtasks Assigned to compute nodes ,on the contrary Since each subtask can only be assigned to one computing node in the network, the following definition applies: Furthermore, for any subtask Execution will only begin if two conditions are met: First, the allocated compute node must be available, meaning no other subtasks are currently executing on that compute node; second, the subtasks must be available. Available time of the allocated nodes The following constraints must be met: in For subtasks Completion time; Second, subtasks It should be ready, meaning it has received the processing results of all predecessor subtasks. Readiness time Defined as: If subtask And one of its precursor tasks They were assigned to different computing nodes. and Communication latency needs to be considered. In this case, the second term on the right-hand side of the constraint will be zero. Taking both conditions into account, the subtask The start time is defined as: The end time of subtask j is defined as: use Indicates the first The set of all subtasks that were successfully completed at each time step; The application in The cumulative execution delay of each time step Defined as: An application is considered complete if and only if all of its components are complete. When all subtasks of an application are successfully completed, all of them are complete. When all tasks are successfully completed, at this time Therefore, the total execution latency of the application Calculated using the following formula: For a person with Application of each task, using Indicates the application's uninstallation method; in Represents the mission The execution locations are respectively the terminal device, the edge server, and the cloud server; The objective function is defined as: 。 3. The real-time dependency-based task offloading method according to claim 1, characterized in that, The DQN algorithm obtains state s in the cloud-edge environment during runtime, and then... Strategy selection action Then you will receive the environmental execution action. The reward value obtained later And the next state Next, the DQN algorithm will obtain the results from each step. The data is stored in the experience storage pool. The capacity of the experience storage pool in the DQN algorithm is preset. When the storage threshold is reached, the neural network parameters are updated. The loss function of the neural network is as follows: in, It is a discount factor; This is the output of "EvalNet", which calculates the Q-value of the current state-action pair. These are the DNN weights of "EvalNet"; It is the output of "TargetNet", which is computed in the next state. Execute action The maximum Q value at that time These are the DNN weights of "TargetNet".

4. The real-time dependency-based task offloading method according to claim 1, characterized in that, Step S2 specifically involves: First, random initialization. weights of a neural network , weights of a neural network ; For each training cycle, the current unloading scheme Current status and current response time All will be initialized; During algorithm training, through The strategy determines the execution location of each subtask sequentially, in order to The probability is randomly selected from all uninstallation options, so that... Probability of choice The uninstallation scheme with the highest Q value; Next, perform the action. And obtain a new response time Calculate rewards And update the current response time. Observe the next state ; After that, Place the samples into the experience storage pool, and randomly select m samples from the experience pool to calculate the target value; Next, the Loss is obtained based on the mean squared error loss function, and the Adam optimizer is used to update the result. weights of a neural network And update after reaching the set C-round iteration. weights of a neural network ; Finally, update the current status.

Images