A dual-decision maker task offloading method based on meta-reinforcement learning

Abstract

The application discloses a double-decider task offloading method based on meta-reinforcement learning, two offloading deciders are set to learn and optimize the task offloading strategy from different angles, one of the deciders is responsible for deciding whether to offload the task and where to offload the task, and the other focuses on resource allocation details, namely, a double-decider offloading model (MA-DDTO model) based on meta-reinforcement learning is developed, and the two deciders generate an optimal action based on information such as task characteristics and device states, and continuously adjust the strategy according to real-time feedback to realize efficient task execution in an edge computing environment. The application can solve the problems of high time delay and high energy consumption caused by large task scale.

Description

Technical Field

[0001] The present invention relates to the field of mobile edge computing technology, and in particular to a dual-decision-maker task offloading method based on meta-reinforcement learning. Background Art

[0002] The rapid development of mobile edge computing (MEC) technology has enabled edge devices to possess increasing computing and storage capabilities, enabling them to perform more complex tasks. As edge computing scenarios become more complex, effectively allocating and offloading tasks to different edge devices has become a significant challenge. Current task offloading methods are primarily based on heuristic or approximate algorithms, but these solutions rely heavily on expert knowledge or accurate mathematical models of the MEC system, which may need to be updated whenever the MEC system's environment changes. Although the subsequently proposed deep reinforcement learning (DRL) can address these complex issues through trial-and-error and model adjustment, DRL algorithms typically require large amounts of data and computing resources for training. Edge devices have limited computing and storage resources and cannot sustain the highly computationally intensive training process required by DRL algorithms. Furthermore, the DRL algorithm training process typically takes a long time, while edge computing environments may require real-time or near-real-time decision responses. Summary of the Invention

[0003] The present invention aims to solve the problem that the existing edge computing task offloading method based on reinforcement learning is slow to adapt to the new environment, and provides a dual-decision-maker task offloading method based on meta-reinforcement learning.

[0004] To solve the above problems, the present invention is achieved through the following technical solutions:

[0005] A dual-decision-maker task offloading method based on meta-reinforcement learning includes the following steps:

[0006] Step 1: Collect user task information and device status information;

[0007] Step 2: Build a dual-decision-maker offloading model based on meta-reinforcement learning, which consists of two relatively independent sequence-to-sequence models. The first sequence-to-sequence model is deployed on the user terminal in the edge computing offloading environment, and the second sequence-to-sequence model is deployed on the designated edge server in the edge computing offloading environment.

[0008] Step 3: Calculate the estimated completion time of each task based on the user task information and device status information, and sort all tasks based on the estimated completion time to obtain the task embedding sequence;

[0009] Step 4: Initialize the initial strategy parameters and meta-parameters of the first sequence-to-sequence model;

[0010] Step 5: Input the task embedding sequence into the first sequence-to-sequence model, and obtain the first offloading decision corresponding to the task embedding sequence by optimizing the policy parameters and meta-parameters of the first sequence-to-sequence model. The first offloading decision includes the offloading decision of all tasks in the task embedding sequence, that is, the decision to offload the task to the user terminal, edge server, or cloud server for execution;

[0011] Step 6: Determine the offloading decision of each task in the first offloading decision: if the offloading decision of the task is to offload to the local user terminal or cloud server for execution, then no further decision is required for the task; if the offloading decision of the task is to offload to the edge server for execution, then further decision is required for the task;

[0012] Step 7: Use the optimized strategy parameters and meta-parameters of the first sequence-to-sequence model as the initial strategy parameters and meta-parameters of the second sequence-to-sequence model for further decision-making;

[0013] Step 8: Input the task embedding sequence into the second sequence-to-sequence model. By optimizing the policy parameters and meta-parameters of the second sequence-to-sequence model, a second offloading decision corresponding to the task embedding sequence is obtained. The second offloading decision includes the offloading decision for the task in the task embedding sequence that requires further decision-making, that is, the decision on which edge server to offload the task to.

[0014] Step 9: Output the first offloading decision obtained in step 5 and the second offloading decision obtained in step 8 as the decision result of task offloading.

[0015] In the above scheme, user task information includes the task data size, the resources required to run the task, and the dependencies between tasks; device status information includes the computing power of the device, namely the user terminal, edge server and cloud server, as well as the transmission rate between devices.

[0016] In the above scheme, the estimated completion time of task i is Time i for:

[0017]

[0018] in, is the estimated completion time of the i-th task executed locally, i.e., on the user terminal, is the estimated completion time of the i-th task executed on the edge server, is the estimated completion time of the i-th task executed on the cloud server;

[0019]

[0020] In the above formula:

[0021] is the time when the i-1th task starts running on the user terminal, The time when the i-1th task ends running on the user terminal;

[0022] is the time when the jth task ends on the user terminal, is the end time of the jth task on the edge server, is the time when the jth task ends on the cloud server;

[0023] is the start time of sending the i-1th task to the edge server, The end time of sending the i-1th task to the edge server;

[0024] is the time when the edge server starts processing the i-1th task, The end time of the edge server processing the i-1th task;

[0025] is the end time of sending the i-th task to the edge server, is the end time of sending the jth task to the edge server;

[0026] is the start time of sending the i-1th task to the cloud server, The end time of sending the i-1th task to the cloud server;

[0027] is the time when the cloud server starts processing the i-1th task, The end time of the i-1th task processed by the cloud server;

[0028] is the end time of sending the i-th task to the cloud server, is the end time of sending the jth task to the cloud server;

[0029] f UE is the computing power of the user terminal, f ES is the computing power of the edge server, f CS is the computing power of the cloud server; r ES is the link transmission rate from the user device to the edge server, r CS is the link transmission rate from the user device to the cloud server; ci is the data size of the i-th task, v i The resources required to run the i-th task; parent(i) is the parent task of the i-th task, i = 1, 2, ..., n, and n is the total number of tasks.

[0030] In step 3 above, it is necessary to convert the estimated completion time of the task into the task weight to sort all tasks to obtain the task embedding sequence; the weight of the i-th task is weight i for:

[0031]

[0032] Where, Time i is the estimated completion time of the i-th task, weight j is the weight of the j-th task, child(i) represents the subtask of the i-th task, and K is the set of leaf nodes.

[0033] Compared with existing technologies, this invention sets up two offloading decision makers, each learning and optimizing task offloading strategies from different perspectives. One decision maker is responsible for deciding whether and where to offload tasks, while the other focuses on the details of resource allocation. That is, a dual-decision maker offloading model based on meta-reinforcement learning (MA-DDTO model) is developed. These two models jointly generate optimal actions based on information such as task characteristics and device status, and continuously adjust strategies based on real-time feedback to achieve efficient task execution in edge computing environments. This invention can solve the problems of high latency and high energy consumption caused by large task scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the dual-decision-maker task offloading method based on meta-reinforcement learning. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific examples and the accompanying drawings.

[0036] See also Figure 1 , a dual-decision-maker task offloading method based on meta-reinforcement learning, comprising the following steps:

[0037] Step 1: Collect user task information and device status information.

[0038] User task information includes the task data size, the resources required to run the task, and the dependencies between tasks. User task information D is expressed as:

[0039] D=(T,E)

[0040] Where T represents the task set, T={t1, t2,…, t n}, task t i ={c i , v i}, c i is the task data size, v i is the resource required to run the task, n is the total number of tasks. E represents the directed edge set of the dependency relationship between tasks, Directed edges Represented as task t i and task t j Directed edges between i t j The parent task, t j It is t i subtasks.

[0041] Due to the constraints of dependency, a subtask can not be executed until all its parent tasks are completed. In D=(T, E), a task without subtasks is called an exit task.

[0042] Device status information includes the computing power of devices (user terminals, edge servers, and cloud servers) and the transmission rate between devices. The device status information N is expressed as:

[0043] N=(F,R)

[0044] Where F represents the computing power set, F={f UE , f ES , f CS}, f UE 、f ES and f Cs are the computing capabilities of the user equipment (UE), edge server (ES), and cloud server (CS), respectively. R represents the link transmission rate set, R = {r ES 、r CS}, r ES and r CS are the link transmission rates from user equipment to edge server and from user equipment to cloud server, respectively.

[0045] Step 2: Build a dual-decision-maker offloading model based on meta-reinforcement learning and deploy it in the edge computing offloading environment.

[0046] The dual decision maker offloading module constructed by the present invention based on meta-reinforcement learning consists of two relatively independent sequence-to-sequence models (Seq2Seq). Each decision maker generates a decision sequence by the corresponding sequence-to-sequence model. Each sequence-to-sequence model consists of an encoder and a decoder implemented by a recurrent neural network. The encoder uses the task embedding sequence as the state s t Input, encoder output e t As the input of the decoder, the output of the decoder d t Contains the state s t Uninstall decision a t , used to generate the policy function π(a t |s t ) and state value function v π (s t ), and calculate the immediate reward sequence r t .

[0047] The output of the encoder is t for:

[0048] e t =f en (s t , e t-1 )

[0049] Among them, f en (*) indicates an encoding function.

[0050] The output of the decoder d t for:

[0051] d t =f de (d t-1 , a t )

[0052] Among them, f de (*) indicates a decoding function.

[0053] When the state s t The decision is a t When its immediate reward sequence r t It is the reciprocal of the estimated completion time of each task, that is:

[0054]

[0055] The edge computing offloading environment involved in the present invention includes a user terminal, multiple edge servers, and a cloud server. A first sequence-to-sequence model is deployed on the user terminal and is used to decide whether to offload tasks to a local server, an edge server, or a cloud server. A second sequence-to-sequence model is deployed on a specific edge server among the multiple edge servers and is used to decide to which edge server the task should be offloaded.

[0056] Step 3: Calculate the estimated completion time of each task based on the user task information and device status information, and sort all tasks based on the estimated completion time to obtain the task embedding sequence.

[0057] The tasks are converted into a directed acyclic graph (DAG), and then converted into a task embedding sequence according to the expected completion time of each task. The time when the i-th task starts running on the user terminal, edge server, and cloud server is defined as and The time (estimated completion time) when the i-th task ends running on the user terminal, edge server, and cloud server is defined as and Where i = 0, 1,…, n, and n is the total number of tasks.

[0058] Since tasks are interdependent, the start time of each task is only related to the parent task, so the end time of a task is affected by its parent task and the transmission channel. and Set to 0. If the parent task is uninstalled, there are three cases:

[0059] ① If the task is executed locally (on the user terminal), there is no transmission delay. Therefore, the time when the i-th task ends on the user terminal is for:

[0060]

[0061] The time when the i-th task starts running on the user terminal for:

[0062]

[0063] Among them, when i=1, there is no need to calculate by formula Direct Order Based on the given Calculate by formula

[0064] When the user terminal chooses to offload the task to the local execution, the offloading time of the task only includes the computing power of the user terminal. Therefore, the offloading time of the task is equal to the computing delay of the task in the local, that is, the time when the i-th task is offloaded to the local for:

[0065]

[0066] ② If the task is offloaded to the server edge server, the task can be transferred to the server only when its parent task is completed and the transmission channel is available. To this end, the time when the i-th task ends on the edge server is for:

[0067]

[0068] The end time of sending the i-th task to the edge server for:

[0069]

[0070] The start time of sending the i-th task to the edge server for:

[0071]

[0072] Among them, when i=1, there is no need to calculate by formula Direct Order Based on the given Calculate by formula

[0073] The end time when the edge server completes processing of the i-th task for:

[0074]

[0075] The time when the edge server starts processing the i-th task for:

[0076]

[0077] Among them, when i=1, there is no need to calculate by formula Direct Order Based on the given Calculate by formula

[0078] When the user terminal chooses to offload the task to the edge server for execution, the task offloading time includes the transmission delay of the user terminal transmitting the task to the edge server, the calculation delay of the task on the edge server, and the transmission delay of the edge server returning the calculation result of the task to the user terminal. In view of the fact that the amount of data of the calculation result returned by the edge server to the user terminal is usually very small, for the sake of simplicity, the present invention does not consider the transmission delay of the edge server returning the calculation result of the task to the user terminal when calculating the task offloading time. For this reason, the task offloading time is equal to the sum of the transmission delay of the user terminal transmitting the task to the edge server and the calculation delay of the task on the edge server, that is, the time when the i-th task is offloaded to the edge server. for:

[0079]

[0080] The transmission delay from the user terminal to the edge server is for:

[0081]

[0082] The computational delay of the i-th task on the edge server for:

[0083]

[0084] ③ If the task is offloaded to the server cloud server, the task can be transferred to the server only when its parent task is completed and the transmission channel is available. To this end, the time when the i-th task ends on the cloud server is for:

[0085]

[0086] The end time of sending the i-th task to the cloud server for:

[0087]

[0088] The start time of sending the i-th task to the cloud server for:

[0089]

[0090] Among them, when i=1, there is no need to calculate by formula Direct Order Based on the given Calculate by formula

[0091] The end time when the cloud server completes processing of the i-th task for:

[0092]

[0093] The time when the cloud server starts processing the i-th task for:

[0094]

[0095] Among them, when i=1, there is no need to calculate by formula Direct Order Based on the given Calculate by formula

[0096] When the user terminal chooses to offload the task to the cloud server for execution, the task offloading time includes the transmission delay of the user terminal transmitting the task to the cloud server, the calculation delay of the task on the cloud server, and the transmission delay of the cloud server returning the calculation result of the task to the user terminal. In view of the fact that the amount of data of the calculation result returned by the cloud server to the user terminal is usually very small, for the sake of simplicity, the present invention does not consider the transmission delay of the cloud server returning the calculation result of the task to the user terminal when calculating the task offloading time. For this reason, the task offloading time is equal to the sum of the transmission delay of the user terminal transmitting the task to the cloud server and the calculation delay of the task on the cloud server, that is, the time when the i-th task is offloaded to the cloud server. for:

[0097]

[0098] The transmission delay from the user terminal to the cloud server is for:

[0099]

[0100] The computational latency of the i-th task on the cloud server for:

[0101]

[0102] In summary, based on the completion time of the i-th task on different devices (user terminal, edge server and cloud server), the estimated completion time Time of the i-th task is calculated. i for:

[0103]

[0104] The present invention can be directly based on the estimated completion time T of the i-th task iTo sort the tasks to generate a task embedding sequence, but considering the dependencies between tasks, the present invention can also be based on the estimated completion time Time of the i-th task i Assign different weights to each task i , and based on the weight of the task i To sort the tasks to generate a task embedding sequence, where the weight of the i-th task is weight i for:

[0105]

[0106] In the formula, weight j is the weight of the j-th task, K is the set of leaf nodes, indicating no subtasks, and child(i) represents the subtask of the i-th task.

[0107] The tasks are sorted according to their weights. The higher the weight, the lower the priority. Thus, a DAG can form a task embedding sequence T = [t1, t2, ..., t n ]. Among them, each task t i The embedding contains three elements: (1) Task index and normalized task feature vector: This part of the embedding contains the unique identifier of the task (i.e., the task index) and the attributes or features of the task, which include v i 、c i etc. (2) Parent Task Index Vector: This embedding contains the identifiers of the parent tasks that the current task directly depends on; in a DAG, the parent task must be completed before the current task can start. (3) Subtask Index Vector: This embedding contains the identifiers of the subtasks that the current task directly depends on; similarly, subtasks can only be started after the current task is completed.

[0108] Step 4: Initialize the initial strategy parameters of the first sequence-to-sequence model and meta-parameter θ outer .

[0109] Step 5: Embed the task sequence as state s t Input into the first sequence-to-sequence model to obtain the first offloading decision corresponding to the task embedding sequence, where the first offloading decision includes the offloading decisions a1 of all tasks in the task embedding sequence t , that is, the decision of offloading the task to the user terminal, edge server or cloud server for execution.

[0110] Set T = [t1, t2, ..., t n ] as state s t Input into the first sequence-to-sequence model (the agent’s neural network Seq2Seq1) to generate an approximate policy function π(a1t |s t ) and state value function v π (s t ), where state s t The corresponding first uninstallation decision a1 t :

[0111] a1 t =[a1 t1 , a1 t2 ,...,a1 tn ]

[0112] a1 ti ={d0,d1,d2}∈A, i=1,2,…,n

[0113] Among them, d0, d1, and d2 respectively represent offloading tasks to the local, designated edge server, and cloud server for execution.

[0114] When the task t i The decision is a1 t The instant reward r obtained at that moment t for:

[0115]

[0116] Among them, Time i Indicates the estimated completion time of the i-th task.

[0117] In the first sequence-to-sequence model, according to π(a1 t |s t ),v π (s t ), r t Calculate the PPO shear objective function Sum value function loss Execute the inner loop to update the policy parameters until When convergence is reached, the inner loop of this round ends and is saved as θ′ inner ; Using the θ′ saved in the inner loop inner By gradient ascent method, θ outer Update the parameters of the outer loop, and the updated θ outer It will affect the inner loop of the outer loop until θ outer The outer loop ends when convergence is reached. At this point, all tasks obtain the stable decision a1 given by Seq2Seq1 t .

[0118] Step 6: Determine the uninstallation decision of each task in the first uninstallation decision:

[0119] If the task is offloaded to the local user terminal or cloud server for execution (a1 ti =d0 or a1 ti =d2), then no further decision is required for this task;

[0120] If the task is offloaded to the edge server (a1 ti =d1), then the task requires further decision-making.

[0121] Step 7: Optimize the strategy parameters of the first sequence-to-sequence model and meta-parameter θ outer As the initial policy parameters of the second sequence-to-sequence model and meta-parameter θ outer Make further decisions.

[0122] Step 8: Input the task embedding sequence into the second sequence-to-sequence model to obtain the second offloading decision corresponding to the task embedding sequence, where the second offloading decision includes the offloading decision of the task that requires further decision in the task embedding sequence, that is, the decision on which edge server to offload the task to.

[0123] Set T = [t1, t2, ..., t n ] as state s t Input into the second sequence-to-sequence model (the agent’s neural network Seq2Seq2) to generate an approximate policy function π(a2 t |s t ) and state value function v π (s t ), where state s t The corresponding first uninstallation decision a2 t :

[0124] a2 t =[a2 t1 , a2 t2 ,...,a2 tn ]

[0125] a2 ti ={d j}∈A,,i=1,2,…,n

[0126] Among them, d j They represent offloading the task to the jth edge server for execution, j∈m, where m is the sum of the number of adjacent edge servers and the edge server. When j=0, it represents offloading to the edge server.

[0127] When the task t i The decision is a2 t The instant reward r obtained at that momentt for:

[0128]

[0129] Among them, Time i Indicates the estimated completion time of the i-th task.

[0130] In the second sequence-to-sequence model, according to π(a2 t |s t ),v π (s t ), r t Calculate the PPO shear objective function Sum value function loss Execute the inner loop to update the policy parameters until When convergence is reached, the inner loop of this round ends and is saved as θ′ inner ; Using the θ′ saved in the inner loop inner By gradient ascent method, θ outer Update the parameters of the outer loop, and the updated θ outer It will affect the inner loop of the outer loop until θ outer The outer loop ends when convergence is reached. At this point, all tasks obtain the stable decision a2 given by Seq2Seq2 t .

[0131] Step 9: Output the first offloading decision obtained in step 5 and the second offloading decision obtained in step 8 as the decision result of task offloading.

[0132] This paper combines meta-learning and reinforcement learning techniques to solve the task offloading optimization problem in edge computing environments. By using meta-learning algorithms to learn and summarize historical task execution data, it guides reinforcement learning agents to intelligently decide task offloading strategies between different edge devices, thereby improving the performance and efficiency of edge computing systems.

[0133] It should be noted that although the embodiments of the present invention described above are illustrative, they are not intended to limit the present invention. Therefore, the present invention is not limited to the above-mentioned specific embodiments. Without departing from the principles of the present invention, any other embodiments obtained by those skilled in the art under the guidance of the present invention are deemed to be within the protection of the present invention.

Claims

1. A dual-decision-maker task offloading method based on meta-reinforcement learning, characterized by: The steps are as follows: Step 1: Collect user task information and device status information; Step 2: Build a dual-decision-maker offloading model based on meta-reinforcement learning, which consists of two relatively independent sequence-to-sequence models. The first sequence-to-sequence model is deployed on the user terminal in the edge computing offloading environment, and the second sequence-to-sequence model is deployed on the designated edge server in the edge computing offloading environment. Step 3: Calculate the estimated completion time of each task based on the user task information and device status information, and sort all tasks based on the estimated completion time to obtain the task embedding sequence; Step 4: Initialize the initial strategy parameters and meta-parameters of the first sequence-to-sequence model; Step 5: Input the task embedding sequence into the first sequence-to-sequence model, and obtain the first offloading decision corresponding to the task embedding sequence by optimizing the policy parameters and meta-parameters of the first sequence-to-sequence model. The first offloading decision includes the offloading decision of all tasks in the task embedding sequence, that is, the decision to offload the task to the user terminal, edge server, or cloud server for execution; Step 6: Determine the offloading decision of each task in the first offloading decision: if the offloading decision of the task is to offload to the local user terminal or cloud server for execution, then no further decision is required for the task; if the offloading decision of the task is to offload to the edge server for execution, then further decision is required for the task; Step 7: Use the optimized strategy parameters and meta-parameters of the first sequence-to-sequence model as the initial strategy parameters and meta-parameters of the second sequence-to-sequence model for further decision-making; Step 8: Input the task embedding sequence into the second sequence-to-sequence model. By optimizing the policy parameters and meta-parameters of the second sequence-to-sequence model, a second offloading decision corresponding to the task embedding sequence is obtained. The second offloading decision includes the offloading decision for the task in the task embedding sequence that requires further decision-making, that is, the decision on which edge server to offload the task to. Step 9: Output the first offloading decision obtained in step 5 and the second offloading decision obtained in step 8 as the decision result of task offloading.

2. The dual-decision-maker task offloading method based on meta-reinforcement learning according to claim 1 is characterized in that: User task information includes the task data size, the resources required to run the task, and the dependencies between tasks; device status information includes the computing power of the device, namely the user terminal, edge server and cloud server, as well as the transmission rate between devices.

3. The dual-decision-maker task offloading method based on meta-reinforcement learning according to claim 1 or 2, characterized in that: The estimated completion time of task i i for: in, is the estimated completion time of the i-th task executed locally, i.e., on the user terminal, is the estimated completion time of the i-th task executed on the edge server, is the estimated completion time of the i-th task executed on the cloud server; In the above formula: is the time when the i-1th task starts running on the user terminal, The time when the i-1th task ends running on the user terminal; is the time when the jth task ends on the user terminal, is the end time of the jth task on the edge server, is the time when the jth task ends on the cloud server; is the start time of sending the i-1th task to the edge server, The end time of sending the i-1th task to the edge server; is the time when the edge server starts processing the i-1th task, The end time of the edge server processing the i-1th task; is the end time of sending the i-th task to the edge server, is the end time of sending the jth task to the edge server; is the start time of sending the i-1th task to the cloud server, The end time of sending the i-1th task to the cloud server; is the time when the cloud server starts processing the i-1th task, The end time of the i-1th task processed by the cloud server; is the end time of sending the i-th task to the cloud server, is the end time of sending the jth task to the cloud server; f UE is the computing power of the user terminal, f ES is the computing power of the edge server, f CS is the computing power of the cloud server; r ES is the link transmission rate from the user device to the edge server, r CS is the link transmission rate from the user device to the cloud server; c i is the data size of the i-th task, v i The resources required to run the i-th task; parent(i) is the parent task of the i-th task, i = 1, 2, ..., n, and n is the total number of tasks.

4. The dual-decision-maker task offloading method based on meta-reinforcement learning according to claim 1 is characterized in that: In step 3, the estimated completion time of the task needs to be converted into the task weight to sort all tasks to obtain the task embedding sequence; The weight of the i-th task is weight i for: Where, Time i is the estimated completion time of the i-th task, weight j is the weight of the j-th task, child(i) represents the subtask of the i-th task, and K is the set of leaf nodes.

Images