Double-decision-maker task unloading method based on meta-reinforcement learning

Abstract

The invention discloses a double-decision-maker task unloading method based on meta reinforcement learning, which is characterized in that two unloading decision makers are set, task unloading strategies are respectively learned and optimized from different angles, one decision maker is responsible for deciding whether or not and where to unload tasks, and the other decision maker focuses on resource allocation details; according to the method, a meta-reinforcement learning-based dual decision maker unloading model (MA-DDTO model) is developed, an optimal action is generated based on task characteristics, equipment states and other information, and a strategy is continuously adjusted according to real-time feedback so as to realize efficient task execution in an edge computing environment. According to the invention, the problems of high time delay and large energy consumption caused by large task scale can be solved.

Description

Technical Field

[0001] The present invention relates to the field of mobile edge computing technology, and in particular to a dual-decision-maker task offloading method based on meta-reinforcement learning. Background Art

[0002] The rapid development of mobile edge computing (MEC) technology has enabled edge devices to have more and more computing and storage capabilities, so that they can perform more complex tasks. With the complexity of edge computing scenarios, how to effectively allocate and offload tasks to different edge devices has become an important challenge. Current task offloading methods are mainly based on heuristic or approximate algorithms, but these solutions rely heavily on expert knowledge or accurate mathematical models of MEC systems. Whenever the environment of the MEC system changes, the expert knowledge or mathematical models may need to be updated accordingly. Although the subsequently proposed Deep Reinforcement Learning (DRL) can solve the above complex problems through trial and error and adjustment of models, the DRL algorithm usually requires a large amount of data and computing resources for training, while the computing and storage resources of edge devices are limited, and it is difficult to withstand the highly computationally intensive training process required by the DRL algorithm. In addition, the training process of the DRL algorithm usually takes a long time, while the edge computing environment may require real-time or near real-time decision responses. Summary of the invention

[0003] The present invention aims to solve the problem that the existing edge computing task offloading method based on reinforcement learning is slow to adapt to the new environment, and provides a dual decision-maker task offloading method based on meta-reinforcement learning.

[0004] To solve the above problems, the present invention is achieved through the following technical solutions:

[0005] A dual-decision-maker task offloading method based on meta-reinforcement learning includes the following steps:

[0006] Step 1: Collect user task information and device status information;

[0007] Step 2: construct a dual-decision-maker offloading model based on meta-reinforcement learning, which consists of two relatively independent sequence-to-sequence models. The first sequence-to-sequence model is deployed on the user terminal in the edge computing offloading environment, and the second sequence-to-sequence model is deployed on the designated edge server in the edge computing offloading environment.

[0008] Step 3: Calculate the estimated completion time of each task based on the user task information and device status information, and sort all tasks based on the estimated completion time of the tasks to obtain a task embedding sequence;

[0009] Step 4: Initialize the initial strategy parameters and meta-parameters of the first sequence-to-sequence model;

[0010] Step 5: Input the task embedding sequence into the first sequence-to-sequence model, and obtain the first offloading decision corresponding to the task embedding sequence by optimizing the policy parameters and meta-parameters of the first sequence-to-sequence model, wherein the first offloading decision includes the offloading decision of all tasks in the task embedding sequence, that is, the decision to offload the task to the user terminal, edge server or cloud server for execution;

[0011] Step 6: Determine the offloading decision of each task in the first offloading decision: if the offloading decision of the task is to offload to the local user terminal or cloud server for execution, then the task does not need further decision; if the offloading decision of the task is to offload to the edge server for execution, then the task needs further decision;

[0012] Step 7: Use the optimized strategy parameters and meta-parameters of the first sequence-to-sequence model as the initial strategy parameters and meta-parameters of the second sequence-to-sequence model for further decision-making;

[0013] Step 8: Input the task embedding sequence into the second sequence-to-sequence model, and obtain the second offloading decision corresponding to the task embedding sequence by optimizing the policy parameters and meta-parameters of the second sequence-to-sequence model, wherein the second offloading decision includes the offloading decision of the task that needs further decision in the task embedding sequence, that is, the decision of which edge server to offload the task to execute;

[0014] Step 9: Output the first offloading decision obtained in step 5 and the second offloading decision obtained in step 8 as the decision result of task offloading.

[0015] In the above scheme, user task information includes the task data size, the resources required to run the task and the dependencies between tasks; device status information includes the computing power of the device, i.e. the user terminal, edge server and cloud server, and the transmission rate between devices.

[0016] In the above scheme, the estimated completion time of the i-th task is Time i for:

[0017]

[0018] in, is the estimated completion time of the i-th task executed locally, i.e., on the user terminal, is the estimated completion time of the i-th task executed on the edge server, is the estimated completion time of the i-th task executed on the cloud server;

[0019]

[0020] In the above formula:

[0021] is the time when the i-1th task starts running on the user terminal, The time when the i-1th task ends running on the user terminal;

[0022] is the time when the jth task ends on the user terminal, is the end time of the jth task on the edge server, is the time when the jth task ends on the cloud server;

[0023] is the start time of sending the i-1th task to the edge server, The end time of sending the i-1th task to the edge server;

[0024] is the time when the edge server starts processing the i-1th task, The end time of the edge server processing the i-1th task;

[0025] is the end time of sending the i-th task to the edge server, is the end time of sending the jth task to the edge server;

[0026] is the start time of sending the i-1th task to the cloud server, is the end time of sending the i-1th task to the cloud server;

[0027] is the time when the cloud server starts processing the i-1th task, The end time of the i-1th task processed by the cloud server;

[0028] is the end time of sending the i-th task to the cloud server, is the end time of sending the jth task to the cloud server;

[0029] f UE is the computing power of the user terminal, f ES is the computing power of the edge server, f CS is the computing power of the cloud server; r ES is the link transmission rate from the user device to the edge server, r CS is the link transmission rate from the user device to the cloud server; ci is the data size of the i-th task, v i are the resources required to run the i-th task; parent(i) is the parent task of the i-th task, i = 1, 2, …, n, and n is the total number of tasks.

[0030] In step 3 above, the estimated completion time of the task needs to be converted into the task weight to sort all the tasks to obtain the task embedding sequence; the weight of the i-th task is i for:

[0031]

[0032] Where Time i is the estimated completion time of the i-th task, weight j is the weight of the j-th task, child(i) represents the subtask of the i-th task, and K is the set of leaf nodes.

[0033] Compared with the prior art, the present invention sets two offloading decision makers to learn and optimize the task offloading strategy from different perspectives. One decision maker is responsible for deciding whether and where to offload the task, and the other focuses on the details of resource allocation, that is, developing a dual decision maker offloading model (MA-DDTO model) based on meta-reinforcement learning, jointly generating optimal actions based on information such as task characteristics and device status, and continuously adjusting strategies based on real-time feedback to achieve efficient task execution in edge computing environments. The present invention can solve the problems of high latency and high energy consumption caused by large task scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the dual-decision-maker task offloading method based on meta-reinforcement learning. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with specific examples and with reference to the accompanying drawings.

[0036] See also Figure 1 , a dual-decision-maker task offloading method based on meta-reinforcement learning, comprising the following steps:

[0037] Step 1: Collect user task information and device status information.

[0038] User task information includes the task data size, the resources required to run the task, and the dependencies between tasks. User task information D is expressed as:

[0039] D=(T,E)

[0040] Where T represents the task set, T = {t1, t2, ..., t n}, task t i ={c i , v i},c i is the task data size, v i is the resource required to run the task, n is the total number of tasks. E represents the directed edge set of the dependency relationship between tasks, Directed Edge Represented as task t i and task t j The directed edge between i t j The parent task, t j Yes i subtasks.

[0041] Due to the constraints of dependency, a subtask can not be executed until all its parent tasks are completed. In D = (T, E), the task without subtasks is called exit task.

[0042] The device status information includes the computing power of the devices (user terminals, edge servers, and cloud servers) and the transmission rate between devices. The device status information N is expressed as:

[0043] N=(F,R)

[0044] Where F represents the computing capability set, F = {f UE , f ES , f CS}, f UE 、f ES and f Cs are the computing capabilities of the user equipment (UE), edge server (ES) and cloud server (CS). R represents the link transmission rate set, R = {r ES 、r CS}, r ES and r CS They are the link transmission rates from user equipment to edge server and from user equipment to cloud server, respectively.

[0045] Step 2: Build a dual decision-maker offloading model based on meta-reinforcement learning and deploy it in the edge computing offloading environment.

[0046] The dual decision maker unloading module constructed by the present invention based on meta-reinforcement learning consists of two relatively independent sequence-to-sequence models (Seq2Seq), and each decision maker generates a decision sequence by a corresponding sequence-to-sequence model. Each sequence-to-sequence model consists of an encoder and a decoder implemented by a recursive neural network. The encoder uses the task embedding sequence as the state s t Input, encoder output e t As the input of the decoder, the output of the decoder d t Contains the state s t The uninstall decision a t , used to generate the policy function π(a t |s t ) and the state value function v π (s t ), and calculate the immediate reward sequence r t .

[0047] The output of the encoder is t for:

[0048] e t =f en (s t , e t-1 )

[0049] Among them, f en (*) indicates encoding function.

[0050] The output of the decoder is d t for:

[0051] d t =f de (d t-1 , a t )

[0052] Among them, f de (*) indicates a decoding function.

[0053] When the state t The decision is a t When its immediate reward sequence r t It is the reciprocal of the estimated completion time of each task, that is:

[0054]

[0055] The edge computing offloading environment involved in the present invention includes a user terminal, multiple edge servers and a cloud server. The first sequence-to-sequence model is deployed on the user terminal, and the first sequence-to-sequence model is used to decide whether to offload the task to a local, edge server or cloud server. The second sequence-to-sequence model is deployed on a specified edge server among multiple edge servers, and the second sequence-to-sequence model is used to decide which edge server the task is offloaded to.

[0056] Step 3: Calculate the estimated completion time of each task based on the user task information and device status information, and sort all tasks based on the estimated completion time to obtain a task embedding sequence.

[0057] The tasks are converted into a directed acyclic graph (DAG), and then converted into a task embedding sequence according to the estimated completion time of each task. The time when the i-th task starts running on the user terminal, edge server, and cloud server is defined as and The time (estimated completion time) when the i-th task ends running on the user terminal, edge server, and cloud server is defined as and Where i=0,1,…,n, and n is the total number of tasks.

[0058] Since tasks are interdependent, the start time of each task is only related to the parent task, so the end time of a task is affected by its parent task and the transmission channel. and Set to 0. If the parent task is uninstalled, it can be divided into the following three cases:

[0059] ① If the task is executed locally (on the user terminal), there is no transmission delay. Therefore, the time when the i-th task ends on the user terminal is for:

[0060]

[0061] The time when the i-th task starts running on the user terminal for:

[0062]

[0063] When i=1, there is no need to calculate by formula Direct Order Based on the given Calculate by formula

[0064] When the user terminal chooses to offload the task to the local execution, the task offloading time only includes the computing power of the user terminal. Therefore, the task offloading time is equal to the task's local computing delay, that is, the time for the i-th task to be offloaded to the local for:

[0065]

[0066] ② If the task is offloaded to the server edge server, the task can be transferred to the server only when its parent task is completed and the transmission channel is available. To this end, the time when the i-th task ends on the edge server is for:

[0067]

[0068] The end time of sending the i-th task to the edge server for:

[0069]

[0070] The start time of sending the i-th task to the edge server for:

[0071]

[0072] When i=1, there is no need to calculate by formula Direct Order Based on the given Calculate by formula

[0073] The end time when the edge server completes processing of the i-th task for:

[0074]

[0075] The time when the edge server starts processing the i-th task for:

[0076]

[0077] When i=1, there is no need to calculate by formula Direct Order Based on the given Calculate by formula

[0078] When the user terminal chooses to offload the task to the edge server for execution, the task offloading time includes the transmission delay of the user terminal transmitting the task to the edge server, the calculation delay of the task on the edge server, and the transmission delay of the edge server returning the calculation result of the task to the user terminal. Given that the amount of data of the calculation result returned by the edge server to the user terminal is usually very small, for the sake of simplicity, the present invention does not consider the transmission delay of the edge server returning the calculation result of the task to the user terminal when calculating the task offloading time. To this end, the task offloading time is equal to the sum of the transmission delay of the user terminal transmitting the task to the edge server and the calculation delay of the task on the edge server, that is, the time when the i-th task is offloaded to the edge server. for:

[0079]

[0080] The transmission delay from the user terminal to the edge server for:

[0081]

[0082] The computation delay of the i-th task on the edge server for:

[0083]

[0084] ③ If the task is offloaded to a server cloud server, the task can only be transferred to the server when its parent task is completed and the transmission channel is available. To this end, the time when the i-th task ends on the cloud server is for:

[0085]

[0086] The end time of sending the i-th task to the cloud server for:

[0087]

[0088] The start time of sending the i-th task to the cloud server for:

[0089]

[0090] When i=1, there is no need to calculate by formula Direct Order Based on the given Calculate by formula

[0091] The end time when the cloud server completes processing of the i-th task for:

[0092]

[0093] The time when the cloud server starts processing the i-th task for:

[0094]

[0095] When i=1, there is no need to calculate by formula Direct Order Based on the given Calculate by formula

[0096] When the user terminal chooses to offload the task to the cloud server for execution, the task offloading time includes the transmission delay of the user terminal transmitting the task to the cloud server, the calculation delay of the task on the cloud server, and the transmission delay of the cloud server returning the calculation result of the task to the user terminal. Given that the amount of data of the calculation result returned by the cloud server to the user terminal is usually very small, for the sake of simplicity, the present invention does not consider the transmission delay of the cloud server returning the calculation result of the task to the user terminal when calculating the task offloading time. To this end, the task offloading time is equal to the sum of the transmission delay of the user terminal transmitting the task to the cloud server and the calculation delay of the task on the cloud server, that is, the time when the i-th task is offloaded to the cloud server. for:

[0097]

[0098] The transmission delay from the user terminal to the cloud server for:

[0099]

[0100] The computational latency of the i-th task on the cloud server for:

[0101]

[0102] In summary, based on the completion time of the i-th task on different devices (user terminal, edge server and cloud server), the estimated completion time Time of the i-th task is calculated. i for:

[0103]

[0104] The present invention can be directly based on the estimated completion time T of the i-th task iTo sort the tasks to generate a task embedding sequence, but considering the dependencies between tasks, the present invention can also be based on the estimated completion time Time of the i-th task i Assign different weights to each task. i , and based on the task weight Weight i To sort the tasks to generate a task embedding sequence, where the weight of the i-th task is i for:

[0105]

[0106] In the formula, weight j is the weight of the j-th task, K is the leaf node set, indicating no subtask, and child(i) represents the subtask of the i-th task.

[0107] The tasks are sorted according to their weights. The higher the weight, the lower the priority. Thus, a DAG can form a task embedding sequence T = [t1, t2, ..., t n ]. Among them, each task t i The embeddings contain three elements: (1) Task index and normalized task feature vector: This part of the embedding contains the unique identifier of the task (i.e., the task index) and the attributes or features of the task. These features include v i 、c i Etc. (2) Parent task index vector: This part of the embedding represents the identifier of the parent task that the current task directly depends on; in DAG, the parent task must be completed before the current task starts. (3) Subtask index vector: This part of the embedding contains the identifier of the subtask that the current task directly depends on; similarly, the subtask can only be started after the current task is completed.

[0108] Step 4: Initialize the initial strategy parameters of the first sequence-to-sequence model and meta-parameter θ outer .

[0109] Step 5: Embed the task sequence as state s t Input into the first sequence-to-sequence model to obtain the first offloading decision corresponding to the task embedding sequence, where the first offloading decision includes the offloading decisions a1 of all tasks in the task embedding sequence t , that is, the decision of offloading the task to the user terminal, edge server or cloud server for execution.

[0110] Let T = [t1, t2, ..., t n ] as state s t Input to the first sequence-to-sequence model (the agent’s neural network Seq2Seq1) to generate an approximate policy function π(a1t |s t ) and the state value function v π (s t ), where state s t The corresponding first uninstallation decision a1 t :

[0111] a1 t =[a1 t1 , a1 t2 , ..., a1 tn ]

[0112] a1 ti ={d0,d1,d2}∈A, i=1,2,…,n

[0113] Among them, d0, d1, and d2 respectively represent offloading the task to the local, designated edge server, and cloud server for execution.

[0114] When the task i The decision is a1 t The instant reward r obtained at that moment t for:

[0115]

[0116] Among them, Time i Indicates the estimated completion time of the i-th task.

[0117] In the first sequence-to-sequence model, according to π(a1 t |s t ),v π (s t ), r t Calculate the PPO cutting objective function Sum value function loss Execute the inner loop to update the policy parameters until When convergence is reached, the inner loop ends and is saved as θ′ inner ; Use the θ′ saved in the inner loop inner By using the gradient ascent method, θ outer Update the parameters of the outer loop, and the updated θ outer Will affect the inner loop of the outer loop until θ outer The outer loop ends when convergence is reached. At this point, all tasks obtain the stable decision a1 given by Seq2Seq1 t .

[0118] Step 6: Determine the uninstallation decision of each task in the first uninstallation decision:

[0119] If the task is offloaded to the local user terminal or cloud server for execution (a1 ti =d0 or a1 ti =d2), then no further decision is required for this task;

[0120] If the task is offloaded to the edge server for execution (a1 ti =d1), then the task requires further decision making.

[0121] Step 7: Optimize the strategy parameters of the first sequence-to-sequence model and meta-parameter θ outer As the initial policy parameters of the second sequence-to-sequence model and meta-parameter θ outer Make further decisions.

[0122] Step 8: Input the task embedding sequence into the second sequence-to-sequence model to obtain the second offloading decision corresponding to the task embedding sequence, wherein the second offloading decision includes the offloading decision of the task that requires further decision in the task embedding sequence, that is, the decision on which edge server to offload the task to execute.

[0123] Let T = [t1, t2, ..., t n ] as state s t Input to the second sequence-to-sequence model (the agent’s neural network Seq2Seq2) to generate an approximate policy function π(a2 t |s t ) and the state value function v π (s t ), where state s t The corresponding first uninstallation decision a2 t :

[0124] a2 t =[a2 t1 , a2 t2 , ..., a2 tn ]

[0125] a2 ti ={d j}∈A,,i=1,2,…,n

[0126] Among them, d j They represent offloading the task to the jth edge server for execution, j∈m, where m is the sum of the number of adjacent edge servers and the edge server. When j=0, it represents offloading to the edge server.

[0127] When the task i The decision is a2 t The instant reward r obtained at that momentt for:

[0128]

[0129] Among them, Time i Indicates the estimated completion time of the i-th task.

[0130] In the second sequence-to-sequence model, according to π(a2 t |s t ),v π (s t ), r t Calculate the PPO cutting objective function Sum value function loss Execute the inner loop to update the policy parameters until When convergence is reached, the inner loop ends and is saved as θ′ inner ; Use the θ′ saved in the inner loop inner By using the gradient ascent method, θ outer Update the parameters of the outer loop, and the updated θ outer Will affect the inner loop of the outer loop until θ outer The outer loop ends when convergence is reached. At this point, all tasks obtain the stable decision a2 given by Seq2Seq2 t .

[0131] Step 9: Output the first offloading decision obtained in step 5 and the second offloading decision obtained in step 8 as the decision result of task offloading.

[0132] The present invention combines meta-learning and reinforcement learning techniques to solve the task offloading optimization problem in the edge computing environment. By using the meta-learning algorithm to learn and summarize the historical task execution data, the reinforcement learning agent is guided to intelligently decide the task offloading strategy between different edge devices, thereby improving the performance and efficiency of the edge computing system.

[0133] It should be noted that although the embodiments of the present invention described above are illustrative, they are not intended to limit the present invention, and therefore the present invention is not limited to the above specific embodiments. Without departing from the principles of the present invention, any other embodiments obtained by those skilled in the art under the guidance of the present invention are deemed to be within the protection of the present invention.

Claims

1. A dual-decision-maker task offloading method based on meta-reinforcement learning, characterized in that: The steps include: Step 1: Collect user task information and device status information; Step 2: construct a dual-decision-maker offloading model based on meta-reinforcement learning, which consists of two relatively independent sequence-to-sequence models. The first sequence-to-sequence model is deployed on the user terminal in the edge computing offloading environment, and the second sequence-to-sequence model is deployed on the designated edge server in the edge computing offloading environment. Step 3: Calculate the estimated completion time of each task based on the user task information and device status information, and sort all tasks based on the estimated completion time of the tasks to obtain a task embedding sequence; Step 4: Initialize the initial strategy parameters and meta-parameters of the first sequence-to-sequence model; Step 5: Input the task embedding sequence into the first sequence-to-sequence model, and obtain the first offloading decision corresponding to the task embedding sequence by optimizing the policy parameters and meta-parameters of the first sequence-to-sequence model, wherein the first offloading decision includes the offloading decision of all tasks in the task embedding sequence, that is, the decision to offload the task to the user terminal, edge server or cloud server for execution; Step 6: Determine the offloading decision of each task in the first offloading decision: if the offloading decision of the task is to offload to the local user terminal or cloud server for execution, then the task does not need further decision; if the offloading decision of the task is to offload to the edge server for execution, then the task needs further decision; Step 7: Use the optimized strategy parameters and meta-parameters of the first sequence-to-sequence model as the initial strategy parameters and meta-parameters of the second sequence-to-sequence model for further decision-making; Step 8: Input the task embedding sequence into the second sequence-to-sequence model, and obtain the second offloading decision corresponding to the task embedding sequence by optimizing the policy parameters and meta-parameters of the second sequence-to-sequence model, wherein the second offloading decision includes the offloading decision of the task that needs further decision in the task embedding sequence, that is, the decision of which edge server to offload the task to execute; Step 9: Output the first offloading decision obtained in step 5 and the second offloading decision obtained in step 8 as the decision result of task offloading.

2. According to the dual decision maker task offloading method based on meta-reinforcement learning according to claim 1, it is characterized in that: User task information includes the task data size, the resources required to run the task, and the dependencies between tasks; device status information includes the computing power of the device, namely the user terminal, edge server and cloud server, and the transmission rate between devices.

3. A dual-decision maker task offloading method based on meta-reinforcement learning according to claim 1 or 2, characterized in that: The estimated completion time of the i-th task i for: in, is the estimated completion time of the i-th task executed locally, i.e., on the user terminal, is the estimated completion time of the i-th task executed on the edge server, is the estimated completion time of the i-th task executed on the cloud server; In the above formula: is the time when the i-1th task starts running on the user terminal, The time when the i-1th task ends running on the user terminal; is the time when the jth task ends on the user terminal, is the end time of the jth task on the edge server, is the time when the jth task ends on the cloud server; is the start time of sending the i-1th task to the edge server, The end time of sending the i-1th task to the edge server; is the time when the edge server starts processing the i-1th task, The end time of the edge server processing the i-1th task; is the end time of sending the i-th task to the edge server, is the end time of sending the jth task to the edge server; is the start time of sending the i-1th task to the cloud server, is the end time of sending the i-1th task to the cloud server; is the time when the cloud server starts processing the i-1th task, The end time of the i-1th task processed by the cloud server; is the end time of sending the i-th task to the cloud server, is the end time of sending the jth task to the cloud server; f UE is the computing power of the user terminal, f ES is the computing power of the edge server, f CS is the computing power of the cloud server; r ES is the link transmission rate from the user device to the edge server, r CS is the link transmission rate from the user device to the cloud server; c i is the data size of the i-th task, v i The resources required to run the i-th task; parent(i) is the parent task of the i-th task, i = 1, 2, …, n, and n is the total number of tasks.

4. The dual-decision-maker task offloading method based on meta-reinforcement learning according to claim 1 is characterized in that: In step 3, the estimated completion time of the task needs to be converted into the task weight to sort all tasks to obtain the task embedding sequence; The weight of the i-th task is i for: Where Time i is the estimated completion time of the i-th task, weight j is the weight of the j-th task, child(i) represents the subtask of the i-th task, and K is the set of leaf nodes.

Images