Multi-task offloading method, system and device based on deep reinforcement learning

Abstract

The present application discloses a multi-task offloading method, system and device based on deep reinforcement learning, the method comprising: constructing a multi-access edge computing task offloading scenario, modeling the user terminal computing task with a directed acyclic graph, converting the task into a sequence using an encoder, sorting and extracting features using a greedy algorithm; setting a structured state space model based on features, defining a state transformation matrix, introducing a time scale parameter for discretization, and constructing a deep reinforcement learning network architecture based on an attention mechanism through a global convolution process; defining the state space and action space using the constructed network architecture, constructing a reward function, using an approximate policy optimization algorithm for policy optimization, and introducing a generalized advantage estimation method to reduce the policy gradient variance. Through the scheme of the present application, it is possible to quickly adapt to dynamic offloading scenarios and have efficient offloading task reasoning capabilities for new learning tasks.

Description

Technical Field

[0001] The present application relates to the field of machine learning, and in particular to a multi-task offloading method, system and device based on deep reinforcement learning. Background Art

[0002] In recent years, the rapid development of sensing and communication technologies has opened up many application possibilities for the realization of a true digital age, and various types of UE (User Equipment) that carry applications have become an indispensable part of human life. Cloud computing technology has become one of the candidates for meeting the computing needs of resource-intensive applications running in UE due to its abundant computing power, but the cloud cannot meet the latency requirements of these applications due to the propagation delay between UE and the cloud. Therefore, resource-intensive mobile applications pose huge challenges to traditional cloud data centers in terms of computing, storage, and network resources. Specifically, applications such as face recognition, augmented reality, virtual reality, self-driving cars, and industrial Internet of Things have all created huge network transmission bottlenecks on the user side and the cloud computing side, ultimately resulting in very high service delays and poor user experience. Multi-access edge computing has become a key technical approach to solving the above problems. Edge computing technology extends cloud computing capabilities to edge nodes closer to the user side, and offloads the computing-intensive tasks of applications from user-side devices to edge computing hosts on the nearest network nodes, which can significantly alleviate network congestion and service delays.

[0003] However, edge computing servers are not as resource-rich as cloud servers. Especially in dynamic edge environments with limited resources, the problem of efficient multi-task offloading is multifaceted. On the one hand, recent studies have emphasized fine-grained modeling of business processes to characterize the complex internal structure of each computing task and its dependencies. For example, a VR device application contains tasks such as multi-camera array video stitching, equirectangular projection, video extraction, video projection, and rendering display. These tasks are interdependent and need to be executed in sequence. Therefore, efficient task conversion by modeling directed acyclic graphs is a key issue in edge computing. On the other hand, algorithms that need to be continuously optimized enable the system to make reasonable offloading decisions autonomously and find the best balance between computing latency and energy consumption. In recent years, offloading strategy mechanisms based on reinforcement learning have become the mainstream research method. However, with the increasing complexity of edge computing applications and architectures, they are usually faced with exponentially growing search spaces and heavy computing burdens. Computation offloading services still face challenges in achieving better performance.

[0004] Therefore, there is an urgent need for a technical solution that can quickly adapt to dynamic offloading scenarios and have efficient offloading task reasoning capabilities for new learning tasks. Summary of the invention

[0005] In order to solve the deficiencies of the prior art, the embodiments of the present application provide a multi-task offloading method, system and device based on deep reinforcement learning. The present application solves the technical problems of low computational efficiency on long sequences in the prior art.

[0006] The embodiment of the present application provides a multi-task offloading method based on deep reinforcement learning, including: constructing a multi-access edge computing task offloading scenario, modeling the user terminal computing task with a directed acyclic graph, converting the task into a sequence using an encoder, sorting and extracting features through a greedy algorithm; setting a structured state space model based on the feature, defining a state transformation matrix, introducing a time scale parameter for discretization, and constructing a deep reinforcement learning network architecture based on an attention mechanism through a global convolution process; defining the state space and action space using the constructed network architecture, constructing a reward function, using an approximate policy optimization algorithm for policy optimization, and introducing a generalized advantage estimation method to reduce the policy gradient variance; initializing network parameters, obtaining sample batches from a data set, generating action and value estimates, calculating a generalized advantage estimate, and updating policy network parameters to minimize the loss function; performing a directed acyclic graph conversion on an offline data set, calculating an initial strategy, performing an iterative training process, generating action-value pairs and calculating an advantage function, updating network weights until convergence, and outputting a final training strategy for task offloading.

[0007] In a possible implementation method, a multi-access edge computing task offloading scenario is constructed, a directed acyclic graph is modeled for the computing tasks of the user terminal, an encoder is used to convert the tasks into sequences, and a greedy algorithm is used to sort and extract features, including: constructing a multi-access edge computing task offloading scenario, including multiple mobile terminal applications and multiple heterogeneous edge servers; collecting the computing tasks of the user terminals and modeling them as a directed acyclic graph; using a directed acyclic graph encoder to convert the tasks into sequences; prioritizing the nodes in the sequence through a greedy algorithm; and extracting features from the sorted serialized tasks to form a task representation.

[0008] In a possible implementation, a structured state space model is set based on features, a state transformation matrix is ​​defined, a time scale parameter is introduced for discretization, and a deep reinforcement learning network architecture based on an attention mechanism is constructed through a global convolution process, including: setting a structured state space model based on task representation; defining a hidden state transformation matrix and an input-output linear transformation matrix; introducing a time scale parameter to discretize the state space model; designing a global convolution process to calculate a structured convolution kernel; constructing a deep reinforcement learning network architecture based on an attention mechanism, and setting a multi-layer superimposed encoder.

[0009] In a possible implementation, the state space and action space are defined using the constructed network architecture, a reward function is constructed, an approximate policy optimization algorithm is used for policy optimization, and a generalized advantage estimation method is introduced to reduce the variance of the policy gradient, including: using the constructed network architecture to define the state space and represent the task in matrix form; designing the action space for decision-making tasks to be executed locally or on an edge server; constructing a reward function and using the normalized negative value of the task completion time as a reward; using an approximate policy optimization algorithm for policy optimization and using truncation to limit the difference between the new and old policies; introducing a generalized advantage estimation method to calculate the advantage function to reduce the variance of the policy gradient and construct an optimization target.

[0010] In a possible implementation, network parameters are initialized, sample batches are obtained from a data set, action and value estimates are generated, generalized advantage estimates are calculated, and policy network parameters are updated to minimize the loss function, including: initializing the designed policy network and value function network parameters; obtaining a batch of samples from the processed data set; using the current policy to generate actions and corresponding value estimates for the obtained samples; based on the generated actions and value estimates, calculating the generalized advantage estimate for each time step; and updating the policy network parameters based on the calculated advantage estimate to minimize the policy loss and value function loss.

[0011] In a possible implementation, a directed acyclic graph is converted to an offline data set, an initial strategy is calculated, an iterative training process is performed, action-value pairs are generated and an advantage function is calculated, the network weights are updated until convergence, and a final training strategy is output for task offloading, including: a directed acyclic graph is converted to the obtained offline data set; the initial strategy is calculated using the initialized network parameters; the iterative training process is started, and batch samples are obtained from the converted data set in each iteration; the current strategy is applied to generate action-value pairs and calculate the advantage function; the network weights are updated according to the calculation results, the iteration is repeated until convergence, and the final trained strategy is output; and the trained model is deployed to offload edge computing tasks.

[0012] In a possible implementation, feature extraction is performed on the sorted serialized tasks to form task representations, including: ,in, express The final time consumed by the node, express The time consumed by the node, Representation Node The time consumption, Representation Node The collection of parent nodes.

[0013] In one possible implementation, a generalized advantage estimation method is introduced to calculate the advantage function to reduce the variance of the policy gradient and construct an optimization objective, including:

[0014] ,in, Indicates that in the initial strategy The expectations below, represents the sum from time step 0 to n, represents the ratio of the new and old strategies, In the old strategy The advantage function value calculated below is, Represents the clipping function. To limit the ratio between the new and old strategies, Restricted to Within the range, Indicates the difference between the old and new restriction policies.

[0015] The embodiment of the present application also provides a multi-task offloading system using the method described in any of the above embodiments, including: a sampling unit, a computing unit and a task offloading unit; wherein the sampling unit is used to construct a multi-access edge computing task offloading scenario, and perform directed acyclic graph modeling on the user terminal computing task to obtain a data set; the computing unit is used to convert the task into a sequence using an encoder, sort it by a greedy algorithm and perform feature extraction; a structured state space model is set based on the feature, a state transformation matrix is ​​defined, a time scale parameter is introduced for discretization, and a deep reinforcement learning network architecture based on an attention mechanism is constructed through a global convolution process; and a deep reinforcement learning network architecture based on an attention mechanism is constructed by using a deep convolution process. The constructed network architecture defines the state space and action space, constructs a reward function, uses an approximate policy optimization algorithm for policy optimization, and introduces a generalized advantage estimation method to reduce the policy gradient variance; initializes network parameters, obtains sample batches from a data set, generates action and value estimates, calculates generalized advantage estimates, and updates policy network parameters to minimize the loss function; performs a directed acyclic graph conversion on the offline data set, calculates the initial policy, performs an iterative training process, generates action-value pairs and calculates the advantage function, updates the network weights until convergence, and outputs the final training policy; the task offloading unit is used to deploy a model based on the final training policy to offload edge computing tasks.

[0016] An embodiment of the present application also provides a multi-task offloading device based on deep reinforcement learning, comprising: a processor, a memory, and a system bus; wherein the processor and the memory are connected via the system bus; the memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute the method described in any one of the above embodiments.

[0017] In a multi-task offloading method, system and device based on deep reinforcement learning as provided above, the embodiment of the present application can quickly adapt to dynamic offloading scenarios through a method based on a reinforcement learning fusion architecture, combined with a fixed state set of pre-learned sequence tasks and a parallel computing task offloading framework, and has efficient offloading task reasoning capabilities for new learning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of a multi-task offloading method based on deep reinforcement learning provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a reinforcement learning network architecture provided in an embodiment of the present application;

[0021] Figure 3 A schematic block diagram of a task offloading system provided in an embodiment of the present application;

[0022] Figure 4 A schematic block diagram of a task offloading device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0024] Those skilled in the art can understand that the terms "first", "second" and the like in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor represent the necessary logical order between them. It should also be understood that in the embodiments of the present disclosure, "multiple" can refer to two or more, and "at least one" can refer to one, two or more. It should also be understood that for any component, data or structure mentioned in the embodiments of the present disclosure, in the absence of explicit limitation or contrary revelation given in the context, it can generally be understood as one or more. In addition, the term "and / or" in the present disclosure is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are an "or" relationship. It should also be understood that the description of each embodiment in the present disclosure emphasizes the differences between the embodiments, and the same or similar parts can refer to each other. For the sake of brevity, they will not be repeated one by one.

[0025] At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. The techniques, methods and devices known to ordinary technicians in the relevant fields may not be discussed in detail, but where appropriate, the techniques, methods and devices should be considered part of the specification. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] Figure 1A flow chart of a multi-task offloading method based on deep reinforcement learning provided in an embodiment of the present application. It should be noted that in order to solve the problem of mobile edge computing load scheduling under multi-tasks and complex dependencies, the embodiment of the present application establishes a PPO reinforcement learning algorithm based on the Mamba architecture to achieve task scheduling with the least time consumption. First, according to the actual scheduling background, a cloud service scheduling model is established, and the optimization objective function with the least time consumption is disclosed. Then, the dependency relationship between tasks is characterized by DAG to facilitate the solution of the model. Finally, the embodiment of the present application introduces the core PPO algorithm for solving the model and discloses the algorithm model for solving task scheduling. At the same time, the embodiment of the present application designs a network structure for reinforcement learning based on the Mamba structure to increase the training and reasoning speed of reinforcement learning.

[0028] like Figure 1 As shown, at step S101, a multi-access edge computing task offloading scenario is constructed, a directed acyclic graph is modeled for the user terminal computing task, an encoder is used to convert the task into a sequence, and the sequence is sorted and feature extraction is performed through a greedy algorithm.

[0029] It is important to understand that mobile edge computing (MEC) uses cloud services to solve the pressure problems caused by current traffic and computing needs. It is one of the current emerging network paradigms. By deploying MEC hosts to couple computing resources, it can effectively reduce the burden on the return link and reduce service delays.

[0030] Generally, cloud services can be considered to be composed of multiple virtual machines that are uniformly deployed. Task scheduling is usually composed of n interdependent subtasks, which can be expressed as Each subtask Both include the memory size required for the task execution, denoted as , memory size after execution . For local operation delay Only with the computing power of For subtasks , whose local delay , for each cloud service offloading delay, which includes the uplink delay , cloud processing delay , downlink delay , where the uplink and downlink delays are related to the memory and transmission capabilities of the task and can be expressed linearly as , ,in , Respectively represents the uplink and downlink transmission capacity. Cloud processing time ,in Represents the capabilities of cloud services. The delay of cloud offloading can be expressed as :

[0031] ;

[0032] Subtasks The local delay time can be expressed as:

[0033] ;

[0034] Based on this model, it is an NP-hard problem to determine whether each subtask should be run locally or offloaded to the cloud service while ensuring the dependencies between tasks and taking the total delay of the task as the minimum.

[0035] In the embodiment of the present application, the DAG is encoded, and the DAG can be converted into a Sequence to overcome the difficulty of feature extraction. After a greedy algorithm, the nodes can be sorted on the basis of local optimality, and the sorting result is applied to encode each node, which can effectively convert the DAGs into a Sequence. Further, it includes: constructing a multi-access edge computing task offloading scenario, including multiple mobile terminal applications and multiple heterogeneous edge servers; collecting computing tasks of user terminals and modeling them as directed acyclic graphs; using a directed acyclic graph encoder to convert tasks into sequences; prioritizing nodes in the sequence through a greedy algorithm; extracting features from the sorted serialized tasks to form task representations.

[0036] Among them, feature extraction is performed on the sorted serialized tasks to form task representation, including: ,in, express The final time consumed by the node, express The time consumed by the node, Representation Node The time consumption, Representation Node The collection of parent nodes.

[0037] Specifically, in DAGs, Represents a DAG graph, V is a directed set of nodes, E is a set of directed edges, and in the embodiment of the present application, each node There are five inherent characteristics that can be used It is expressed as follows:

[0038] ;

[0039] Each edge has an inherent connection and can represent the relationship between two nodes. It can be represented as follows:

[0040] ;

[0041] Encoding method: Convert DAGs to sequences. Different node features not only include their own features, but also include the connection relationship between the node and other nodes in this embodiment of the application. In this embodiment of the application, due to the simulator generation control, the connection between a single node and other nodes does not exceed 6. Therefore, the connection with the current node can be The connected parent node index collection , child node index collection Splicing is performed as the feature of the node. Connections with less than 6 connections can be filled to the specified dimension. For sets with more than 6 connections, the first 6 are taken. The way to convert DAGs to Sequence can be expressed as follows:

[0042] ;

[0043] where the vector With vector Represents the index collection of child nodes and parent nodes respectively.

[0044] Through the DAG representation method, all sub-nodes in the task are represented to effectively represent the execution status of the nodes. However, considering the dependencies between tasks, the embodiment of the present application first converts it into a sequential model according to the greedy algorithm idea, processes it according to the minimum time consumption of each step, and calculates the completion time of each sub-task:

[0045] ;

[0046] in, is the local delay mentioned above. According to the dependency relationship, the minimum time consumption of the current node is the sum of its own minimum time consumption and the maximum parent node time consumption. The time consumption of each subtask can be calculated recursively, and the order of the nodes can be sorted according to the time consumption, and the relationship between task nodes can be converted into a sequential model. Through this representation method, each task k can be represented as an n×17 matrix, recorded as , n represents the number of subtasks of the task.

[0047] In step S102, a structured state space model is set based on the features, a state transformation matrix is ​​defined, a time scale parameter is introduced for discretization, and a deep reinforcement learning network architecture based on the attention mechanism is constructed through a global convolution process (e.g. Figure 2). These include: setting up a structured state space model based on task representation; defining the hidden state transformation matrix and the input-output linear transformation matrix; introducing a time scale parameter to discretize the state space model; designing a global convolution process and calculating the structured convolution kernel; building a deep reinforcement learning network architecture based on the attention mechanism and setting up a multi-layer superposition encoder.

[0048] Specifically, considering the relationship between tasks in practice, the transformer model can focus on different components of the input sequence from multiple perspectives to better capture global information. For strong dependencies between sequences such as DAG, this structure can be used to extract features from information. Transformer uses its self-attention mechanism as its core, and uses multi-head attention to globally model the sequence to capture long-range dependencies. As the length of the sequence increases, its complexity is proportional to its length. Mamba has an advantage in processing long sequences and can demonstrate higher practicality and efficiency.

[0049] In step S103, the constructed network architecture is used to define the state space and action space, a reward function is constructed, an approximate policy optimization algorithm is used to perform policy optimization, and a generalized advantage estimation method is introduced to reduce the policy gradient variance.

[0050] To meet the prerequisite of task node dependency, the decision for each child node is to minimize the total time consumption. The single node is largely affected by the attributes of other nodes. In the decision-making process, not only the minimum time consumption of the current execution node should be considered, but also the idle state of the device. The number of nodes to be searched by brute force is When , the computational complexity of solving its solution is , the embodiment of the present application adopts a greedy algorithm as the baseline algorithm. The greedy algorithm considers the optimal solution in each step to achieve local optimality in the current state. In the implementation of the present application, the dependency relationship is converted into sequential modeling through DAG. Because reinforcement learning is based on reward signals, reinforcement learning guides the decision-making process through optimization based on reward signals, making the entire sequential decision-making process more effective. Guided by reward signals, reinforcement learning can balance long-term cumulative benefits and short-term immediate rewards to obtain better performance. The embodiment of the present application uses the reinforcement learning PPO algorithm to solve the sequential model.

[0051] Furthermore, it can include: using the constructed network architecture to define the state space and represent the task in matrix form; designing the action space for deciding whether the task is executed locally or on the edge server; constructing a reward function and using the normalized negative value of the task completion time as the reward; using an approximate policy optimization algorithm for policy optimization and using truncation to limit the difference between the new and old policies; introducing a generalized advantage estimation method to calculate the advantage function to reduce the variance of the policy gradient and construct an optimization objective.

[0052] Among them, the generalized advantage estimation method is introduced to calculate the advantage function to reduce the variance of the policy gradient and construct the optimization objectives, including:

[0053] ,

[0054] in, Indicates that in the initial strategy The expectations below, represents the sum from time step 0 to n, represents the ratio of the new and old strategies, In the old strategy The advantage function value calculated below is, Represents the clipping function. To limit the ratio between the new and old strategies, Restricted to Within the range, Indicates the difference between the old and new restriction policies.

[0055] Specifically, all child nodes of each task are characterized by DAG and combined according to the order calculated by the greedy algorithm, using the matrix Represents a task, specifically, here represents k tasks, n represents n subtasks, and d represents the dimension, that is, a large task consists of n d-dimensional subtasks, using the matrix In reinforcement learning, state transition is random, and its randomness comes from the environment, which can be expressed as follows:

[0056] , it is important to understand that in reinforcement learning, represents the probability, Indicates the current state. Indicates the next state, Indicates the current The action under the state. The capital letter after the formula indicates a set, for example: Represents one of the future state sets. Unlike traditional reinforcement learning, the node ordering between subtasks here is fixed, so this model-based reinforcement learning can represent a large task as an episode, and the length of the episode is equal to the number of nodes. The embodiment of the present application stores data in the form of a Replay Buffer.

[0057] The task subnode decision object mainly determines whether the load is executed locally and in MEC. , where 0 means local execution and 1 means execution on MEC. In traditional reinforcement learning, the execution of the current action depends only on the previous state, as shown below: .in, express The action of the moment, Indicates decision making, express The state of the moment means: Moment of action , depending on Status at all times Decisions made. In practice, the decision of a node is related to the dependencies between nodes. Unlike the Markov property in traditional reinforcement learning, the action decision is not only related to the current state, but also to all states in the round. Therefore, the action decision in the embodiment of the present application can be expressed as: ,in, Indicates subtasks, Indicates decision making, Representation Task , Representation Task Each subtask The action decision depends on the state of the overall task . Different from the previous one, in this task, each task has multiple different subtasks, so one state corresponds to multiple actions.

[0058] The reinforcement learning PPO algorithm is introduced when solving tasks, and a multi-task offloading model based on reinforcement learning is disclosed. A reasonable reward function has a significant impact on the convergence and effect of the algorithm. The embodiment of the present application designs a reward function for the offloading conditions of different nodes. According to the greedy algorithm, the task subnodes are sorted, and the time consumption of each subnode is calculated according to the actions of the subnodes. At the same time, considering the subnodes There are dependencies between tasks, and the actual time of their child nodes Index collection with parent node Any node in Completion status Related to the idle state of the device. Representation Node Completion time point, so the time taken for a single child node It can be expressed as follows:

[0059] ;

[0060] In practice, there are many nodes with a reward value of 0, and the optimization goal of the embodiment of the present application is to minimize the total time consumption of all nodes. Therefore, the opposite of the time consumption is used as the reward value. In order to reduce the convergence of the algorithm due to the large difference in reward values, the maximum and minimum normalization is used to process the reward values. It can be expressed as:

[0061] ;

[0062] To train the algorithm, the present embodiment uses PPO clip to train the data set and calculates its ratio through Replay Buffer:

[0063] ;

[0064] in Represents the decision of Replay Buffer. By clipping the ratio, it is limited to between.

[0065] In order to reduce the variance of the gradient estimation in the policy gradient and reduce the samples required for training, the embodiment of the present application uses GAE to calculate the advantage function. The advantage function is It is expressed as follows:

[0066] ;

[0067] in is the discount factor, Represents a subtask Reward value, Represents the child node The state value after is estimated, Represents the child node The state value after is estimated, represents the GAE coefficient, Indicates the total number of subtasks.

[0068] The final optimization objective can be written as:

[0069] .

[0070] At step S104, the network parameters are initialized, a sample batch is obtained from the data set, action and value estimates are generated, a generalized advantage estimate is calculated, and the policy network parameters are updated to minimize the loss function. This includes: initializing the designed policy network and value function network parameters; obtaining a batch of samples from the processed data set; using the current policy to generate actions and corresponding value estimates for the obtained samples; based on the generated action and value estimates, the generalized advantage estimate is calculated for each time step; based on the calculated advantage estimate, the policy network parameters are updated to minimize the policy loss and value function loss.

[0071] At step S105, the offline data set is converted into a directed acyclic graph, the initial strategy is calculated, an iterative training process is performed, action-value pairs are generated and the advantage function is calculated, the network weights are updated until convergence, and the final training strategy is output for task offloading. This includes: the obtained offline data set is converted into a directed acyclic graph; the initial strategy is calculated using the initialized network parameters; the iterative training process is started, and batch samples are obtained from the converted data set in each iteration; the current strategy is applied to generate action-value pairs and calculate the advantage function; the network weights are updated according to the calculation results, the iteration is repeated until convergence, and the final trained strategy is output; the trained model is deployed to offload the edge computing task.

[0072] Specifically, based on the above modeling and algorithm, the embodiment of the present application designs a PPO load scheduling algorithm based on the mamba network structure (steps S104-S105, where step S104 describes the process of a single network update, and step S105 describes the process of the entire training cycle). The detailed process of the algorithm is shown in Table 1 below:

[0073] Table 1 PPO scheduling algorithm for complex relationship nodes based on Mamba structure

[0074]

[0075] Figure 2 A schematic diagram of a reinforcement learning network architecture provided in an embodiment of the present application. In an embodiment of the present application, a model based on structured state space (SSM), in particular, an S4 network architecture (a neural network architecture designed specifically for long sequence modeling) and a Mamba model are used. In an embodiment of the present application, based on the DAG data structure, a reinforcement learning network architecture based on Mamba is designed in an embodiment of the present application, and its structure is as follows: Figure 2 shown.

[0076] This model can effectively handle sequential or continuous systems, in which the input sequence changes over time. , and produces an output , and through the hidden state Transform the input using a matrix Describe the hidden state, and The linear transformation of the input and hidden state can be expressed as follows:

[0077] ;

[0078] ;

[0079] S4 and mamba combined with time scale parameters , discretize the above formula, , , after discretization, the time step of the system is It can be expressed as: , .

[0080] Finally, the output is calculated through a global convolution process, which designs the structured convolution kernel and uses , this kernel can represent the input sequence to the output sequence, that is , where m is the input length of the sequence. Through transformation, the discretization of continuous dynamic decomposition is realized.

[0081] Furthermore, the embodiment of the present application uses a simulator to generate data and applies a synthetic DAG generation program, which includes 5 DAGs generation parameters. The generated DAG parameters are shown in Table 2. Through the simulated data set, in the greedy benchmark test, DAGs graphs of different node sizes were generated by the simulator. Each node has a total of 1,300 random graphs with the same structure but different edge and node weights, and the models are trained for 2,000 epochs on a single number of nodes.

[0082] Table 2 DAGs generator environment configuration

[0083]

[0084] As shown in Table 2, by setting parameters, DAGs are applied to generate a sufficient number of data sets to evaluate the algorithm model. In the embodiment of this application, the network of Embedding + Encoder + FFN is used as the main architecture of the algorithm, and the Encoder is implemented by a multi-layer transformer, the number of stacking layers is set to 3, multi-head attention is used, and the head is set to 4. The initial learning rate is 1e -4The AdamW optimizer is used to train the agent, and the weight decay is 0.05. The model used in the embodiment of the present application is trained on a single 3090Ti with 24G-RAM. The replaybuffer space size of the reinforcement learning training is 200, the decay factor gamma is set to 0.99, the clipping factor (clip eps) of PPO is set to 0.2, the weights of the policy loss and value loss are set to 1 and 0.25 respectively, the entropy loss weight of the action probability is set to 0.01, and the environmental factors of MEC transmission are set as shown in Table 3 below. 2000 iterations are trained on the generated Dags dataset.

[0085] Table 3 Experimental environment settings

[0086]

[0087] Consistent with the existing scheduling evaluation standards, the embodiment of the present application uses the total time to complete the entire task (makesplan) as an evaluation indicator, which reflects the overall scheduling efficiency. The calculation formula is as follows: .

[0088] In order to verify the effectiveness of the algorithm proposed in the embodiment of the present application, the embodiment of the present application selected the most representative methods in the field of reinforcement learning as comparison algorithms from the two aspects of DAGs encoding method and Mamba-PPO performance, and used the traditional greedy algorithm as the benchmark algorithm to verify the effectiveness of the algorithm of the embodiment of the present application, including A2C, DQN, Greedy.

[0089] A2C algorithm: is a reinforcement learning method that combines policy gradient and value function. The algorithm generates actions through an Actor network and evaluates state values ​​through a Critic network, thereby training the Actor and Critic networks to maximize the cumulative reward.

[0090] DQN algorithm: combines the traditional Q-learning algorithm with deep learning, using the deep learning network to approximate the Q-value function, thereby learning a strategy to maximize the cumulative reward in each state.

[0091] Greedy algorithm: Greedy algorithm considers the estimated completion time of each task and selects the algorithm strategy with the best decision in the current state at each step, hoping to reach the global optimal solution through the local optimal decision. It determines whether the task is executed locally or on the cloud server.

[0092] As shown in Table 4, by comparing A2C, DQN and PPO under three different encodings, the experimental results show the algorithm performance under three different encodings. The experimental results show that the effect of PPO and the first type of encoding is the best when the number of nodes is 20-100, while when the number of nodes is 120, the first type of encoding under DQN is the best, but it is still insufficient compared to the baseline. Overall, the first type of algorithm has the best effect.

[0093] Table 4 Experimental performance comparison under multiple nodes

[0094]

[0095] To verify the comparison between Mamba-based PPO and transformer, we compared the task nodes from 20 to 130, and replaced the Transformer in the PPO network structure with the Mamba architecture. The experimental results are shown in Table 5:

[0096] Table 5 Comparison of algorithms under three transformers and Mamba

[0097]

[0098] By evaluating different node DAGs, and replacing Transformer with the mamba architecture for experimental comparison, the experimental results are shown in Table 5. From a single row, Table 5 reflects that under the same node, the PPO algorithm model based on the mamba architecture is superior to the Transformer-based one in terms of the evaluation index of completion time. From the columns of Table 5, at different nodes, the attention mechanism based on mamba performs better on DAGs with a large number of nodes. This shows the effectiveness of the attention PPO algorithm based on the mamba mechanism disclosed in the embodiment of the present application at different nodes.

[0099] In summary, the embodiments of the present application establish a new framework model and processing paradigm. By refining complex task modeling, sequence conversion and algorithm design, an efficient reasoning algorithm for long sequence tasks with theoretical analysis is disclosed, and comprehensive experiments are conducted. Specifically: The embodiments of the present application disclose for the first time a method based on a reinforcement learning fusion architecture (D2S-MPPO) to solve the problem of computational offloading, and achieve rapid adaptation to dynamic offloading scenarios; it has high sample efficiency and efficient offloading task reasoning capabilities for new learning tasks. For the first time in the field of edge computing, a parallel computing task offloading framework based on the Mamba architecture is designed to improve the learning efficiency of the optimal offloading strategy for sequence tasks. A new idea is disclosed, which uses the dag2seq computing framework to efficiently pre-learn a fixed state set of sequence tasks to reduce the amount of computation of the state transfer function in reinforcement learning, thereby improving the overall computing efficiency.

[0100] Figure 3 A schematic block diagram of a task offloading system provided for an embodiment of the present application. It should be understood that the system shown in the figure is exemplary and non-restrictive. This means that the system architecture involved is not limited to a specific form or design, but is presented as an example. In other words, the architecture shown in the figure can be regarded as a way of expression to clearly describe related concepts and relationships, and does not exclude other forms of architecture. Therefore, when interpreting the architecture in the picture, it should be understood that the model is flexible and diverse, and its purpose is to provide an exemplary description rather than a restrictive provision on a specific form.

[0101] The embodiment of the present application discloses a task offloading system, including: a sampling unit 301, a computing unit 302 and a task offloading unit 303; wherein the sampling unit 301 is used to construct a multi-access edge computing task offloading scenario, and perform directed acyclic graph modeling on the user terminal computing task to obtain a data set; the computing unit 302 is used to convert the task into a sequence using an encoder, sort it by a greedy algorithm and perform feature extraction; a structured state space model is set based on the feature, a state transformation matrix is ​​defined, a time scale parameter is introduced for discretization, and a deep reinforcement learning network architecture based on an attention mechanism is constructed through a global convolution process; and a deep reinforcement learning network architecture based on an attention mechanism is constructed by using the constructed The network architecture defines the state space and action space, constructs a reward function, uses an approximate policy optimization algorithm for policy optimization, and introduces a generalized advantage estimation method to reduce the policy gradient variance; initializes network parameters, obtains sample batches from a data set, generates action and value estimates, calculates generalized advantage estimates, and updates policy network parameters to minimize the loss function; performs a directed acyclic graph conversion on the offline data set, calculates the initial policy, performs an iterative training process, generates action-value pairs and calculates the advantage function, updates the network weights until convergence, and outputs the final training policy; the task offloading unit 303 is used to deploy a model based on the final training policy to offload edge computing tasks.

[0102] Figure 4 A schematic block diagram of a task offloading device provided in an embodiment of the present application. Further, an embodiment of the present application also provides a multi-task offloading device based on deep reinforcement learning, The electronic device includes: one or more processors 1101, a memory 1102, and an interface for connecting various components, including a high-speed interface and a low-speed interface. The various components are connected to each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in the embodiments of the present application, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required in the embodiments of the present application.

[0103] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0104] The input device 1103 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0105] To provide interaction with a user, the electronic device may be a computer. The computer has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0106] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any of the above methods.

[0107] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0108] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0109] It should also be noted that, in the embodiments of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "comprise one..." do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.

[0110] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined in the embodiments of the present application can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown in the embodiments of the present application, but will conform to the widest range consistent with the principles and novel features disclosed in the embodiments of the present application.

Claims

1. A multi-task offloading method based on deep reinforcement learning, characterized in that: include: Construct a multi-access edge computing task offloading scenario, model the user terminal computing tasks as a directed acyclic graph, use an encoder to convert the tasks into sequences, sort them using a greedy algorithm, and perform feature extraction; A structured state space model is set based on features, a state transformation matrix is ​​defined, a time scale parameter is introduced for discretization, and a deep reinforcement learning network architecture based on an attention mechanism is constructed through a global convolution process; including: setting a structured state space model based on task representation; defining a hidden state transformation matrix and an input-output linear transformation matrix; introducing a time scale parameter to discretize the state space model; designing a global convolution process to calculate a structured convolution kernel; constructing a deep reinforcement learning network architecture based on an attention mechanism, and setting a multi-layer superposition encoder; The constructed network architecture is used to define the state space and action space, construct a reward function, use an approximate policy optimization algorithm for policy optimization, and introduce a generalized advantage estimation method to reduce the variance of the policy gradient; which includes: using the constructed network architecture to define the state space and represent the task in matrix form; designing the action space for decision-making tasks to be executed locally or on edge servers; constructing a reward function and using the normalized negative value of the task completion time as a reward; using an approximate policy optimization algorithm for policy optimization and using truncation to limit the difference between the new and old policies; introducing a generalized advantage estimation method to calculate the advantage function to reduce the variance of the policy gradient and construct an optimization target; Among them, the generalized advantage estimation method is introduced to calculate the advantage function to reduce the variance of the policy gradient and construct the optimization objectives, including: ， in, Indicates that in the initial strategy The expectations below, represents the sum from time step 0 to n, represents the ratio of the new and old strategies, In the old strategy The advantage function value calculated below is, Represents the clipping function. To limit the ratio between the new and old strategies, Restricted to Within the range, Indicates the difference between the old and new strategies for limiting; Initialize network parameters, get sample batches from the dataset, generate action and value estimates, compute generalized advantage estimates, and update policy network parameters to minimize the loss function; The offline dataset is converted into a directed acyclic graph, the initial strategy is calculated, an iterative training process is performed, action-value pairs are generated and the advantage function is calculated, the network weights are updated until convergence, and the final training strategy is output for task offloading.

2. The multi-task offloading method according to claim 1, characterized in that: in, Construct a multi-access edge computing task offloading scenario, model the user terminal computing tasks as a directed acyclic graph, use an encoder to convert the tasks into sequences, sort them using a greedy algorithm, and perform feature extraction, including: Build a multi-access edge computing task offloading scenario, including multiple mobile terminal applications and multiple heterogeneous edge servers; Collect computing tasks of user terminals and model them as directed acyclic graphs; Use a directed acyclic graph encoder to convert tasks into sequences; Prioritize the nodes in the sequence using a greedy algorithm; Feature extraction is performed on the sorted serialized tasks to form task representation.

3. The multi-task offloading method according to claim 1, characterized in that: in, Initialize network parameters, obtain sample batches from the dataset, generate action and value estimates, calculate generalized advantage estimates, and update policy network parameters to minimize the loss function, including: Initialize the designed policy network and value function network parameters; Get a batch of samples from the processed dataset; Use the current strategy to generate actions and corresponding value estimates for the acquired samples; Based on the generated action and value estimates, compute a generalized advantage estimate for each time step; Based on the computed advantage estimate, the policy network parameters are updated to minimize the policy loss and the value function loss.

4. The multi-task offloading method according to claim 1, characterized in that: in, Convert offline datasets into directed acyclic graphs, calculate initial strategies, perform iterative training, generate action-value pairs and calculate advantage functions, update network weights until convergence, and output the final training strategy for task offloading, including: Perform directed acyclic graph conversion processing on the obtained offline data set; Calculate the initial strategy using the initialized network parameters; Start the iterative training process, where each iteration gets a batch of samples from the transformed dataset; Apply the current strategy to generate action-value pairs and calculate the advantage function; Update the network weights according to the calculation results, repeat the iteration until convergence, and output the final trained strategy; Deploy trained models to offload edge computing tasks.

5. The multi-task offloading method according to claim 2, characterized in that: in, Perform feature extraction on the sorted serialized tasks to form task representations, including: ， in, express The final time consumed by the node, express The time consumed by the node, Representation Node The time consumption, Representation Node The collection of parent nodes.

6. A multi-task unloading system using the method according to any one of claims 1 to 5, characterized in that: include: Sampling unit, computing unit and task offloading unit; among them, The sampling unit is used to construct a multi-access edge computing task offloading scenario and perform directed acyclic graph modeling on the user terminal computing task to obtain a data set; The computing unit is used to convert the task into a sequence using an encoder, sort it by a greedy algorithm and perform feature extraction; set a structured state space model based on the features, define a state transformation matrix, introduce a time scale parameter for discretization, and construct a deep reinforcement learning network architecture based on an attention mechanism through a global convolution process; define the state space and action space using the constructed network architecture, construct a reward function, use an approximate policy optimization algorithm for policy optimization, and introduce a generalized advantage estimation method to reduce the policy gradient variance; initialize network parameters, obtain sample batches from a data set, generate action and value estimates, calculate generalized advantage estimates, and update policy network parameters to minimize the loss function; perform directed acyclic graph conversion on the offline data set, calculate the initial strategy, perform an iterative training process, generate action-value pairs and calculate the advantage function, update the network weights until convergence, and output the final training strategy; The task offloading unit is used to deploy the model based on the final training strategy to offload the edge computing task.

7. A multi-task offloading device based on deep reinforcement learning, characterized in that: include: A processor, a memory, and a system bus; wherein the processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 5.

Images