An edge computing DAG task offloading method and system based on graph attention reinforcement learning
By constructing a Markov decision process model based on graph attention reinforcement learning, and combining graph convolutional networks and external attention mechanisms, the complexity and dynamism of DAG tasks in edge computing environments are addressed, and efficient task offloading and scheduling are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGXI UNIV
- Filing Date
- 2026-03-05
- Publication Date
- 2026-07-21
AI Technical Summary
In edge computing environments, the complexity, heterogeneity, and dynamism of DAG tasks make it difficult for traditional algorithms to schedule them effectively. Existing DAG task offloading methods are inefficient in dynamic environments and have poor model scalability.
We employ a graph attention-based reinforcement learning approach to construct a Markov decision process model. By combining a graph convolutional network and an external attention mechanism, we generate an offloading policy through a dual-stream encoder and an interactive decoder, and train the agent using the PPO algorithm to adapt to the dynamic environment.
It significantly reduces the completion time of DAG tasks, improves the robustness and scalability of system scheduling, and enables efficient scheduling in complex dependent tasks and dynamic environments.
Smart Images
Figure CN122431810A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to an edge computing DAG task offloading method and system based on graph attention reinforcement learning. Background Technology
[0002] With the development of IoT and edge computing technologies, the applications hosted on terminal devices are becoming increasingly complex. Various latency-sensitive applications, such as autonomous driving perception, industrial vision inspection, and real-time high-definition video analysis, typically consist of multiple subtasks with strict logical dependencies. This task structure is modeled as a Directed Acyclic Graph (DAG). Unlike independent tasks, the execution of subtasks in a DAG task must satisfy the timing constraints of predecessors and successors, and different subtasks may require different computational resources (CPU-intensive or data-intensive).
[0003] Due to the limited local computing power and battery capacity of terminal devices (such as robots, AGVs, and smart cameras), it is difficult to meet the real-time requirements of complex DAG tasks. Task offloading technology, by migrating computational tasks to edge servers for execution, has become key to resolving this contradiction. However, in edge computing environments, computing nodes are typically highly heterogeneous (different nodes have different computing power and bandwidth) and dynamic (node availability changes over time). How to efficiently map each subtask to a suitable computing node while satisfying DAG dependency constraints, in order to minimize task completion time, is an NP-hard problem with extremely high complexity.
[0004] Edge computing technology, by deploying edge servers on infrastructure to provide localized computing power support to terminals, is an effective way to solve the above problems. However, in edge computing environments, DAG task offloading faces many challenges:
[0005] (1) Complexity and heterogeneity of task structure: DAG tasks have high topological entropy, irregular structure, and huge differences in dependency patterns between different tasks (such as video analysis and path planning).
[0006] (2) Dynamism of the environment and resource constraints: Computing nodes (edge servers and idle collaborative computing units) have highly heterogeneous computing capabilities, and the number of available nodes changes dynamically with the workflow and mobility, causing traditional static programming-based algorithms to fail.
[0007] (3) Limitations of existing technologies: Traditional heuristic algorithms (such as HEFT) rely on precise prior knowledge of the environment and are difficult to adapt to dynamic environments; existing DRL methods usually use fully connected layers or RNNs to process task features, tending to "flatten" the graph structure and destroy the topological dependency information between tasks; and the input and output dimensions of existing DRL models are fixed and lack scalability. Once the number of devices changes, the model often needs to be retrained. Summary of the Invention
[0008] To address the aforementioned issues, this invention provides an edge computing DAG task offloading method and system based on graph attention reinforcement learning, which significantly reduces task offloading completion time in dynamic environments and improves the robustness and scalability of system scheduling.
[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0010] An edge computing DAG task offloading method based on graph attention reinforcement learning includes the following steps:
[0011] S1. Construct an edge computing system model, wherein the terminal device layer in the edge computing system model generates DAG tasks with directed acyclic graph dependencies;
[0012] S2. The unloading and scheduling problem of the DAG task is modeled as a Markov decision process model, and the state space, action space and reward function are defined, wherein the reward function takes minimizing the total completion time of the task as the optimization objective.
[0013] S3. Construct a deep reinforcement learning agent based on graph attention reinforcement learning to process the Markov decision process model through a dual-stream encoder and an interactive decoder to obtain the unloading decision action for the current subtask in the DAG task.
[0014] S4. Execute the unloading decision action of the current subtask in the simulation environment to obtain experience data, and store the experience data in the experience replay pool; train the agent through the Proximal Policy Optimization (PPO) algorithm and update the network parameters until the Markov Decision Process model converges to obtain the trained agent.
[0015] S5. Deploy the trained agent in the edge computing system model to perform online offloading and scheduling of the DAG tasks generated in real time.
[0016] Further, in step S2, the Markov decision process model is: ,in For state space, For spatial actions, For the transition probability, This is the reward function.
[0017] Furthermore, the state space is for any decision time. ,state It contains all the information necessary to make the current optimal decision, and ,in, This represents the feature matrix of the DAG task currently awaiting scheduling; This is a real-time status matrix of the port environment; This refers to the position information of the current subtask in the topological sorting sequence; This represents the hidden state of the decoder at the previous moment;
[0018] The spatial action To use the currently available computing resource pool Select a target node for the currently scheduled subtask; ;
[0019] The reward function is based on a dense reward mechanism with negative feedback based on the completion time increment, which will reward... Defined as ,in; The estimated completion time for the subtask; The completion time of the precursor mission; A collection of precursor missions.
[0020] Further, in step S3, the dual-stream encoder includes a task encoder and an environment encoder. The task encoder extracts the local topological features of the DAG task through a graph convolutional network and captures the global dependencies between tasks through an external attention mechanism to generate task feature embeddings.
[0021] The environmental encoder extracts features from the attributes of each computing node using a multilayer perceptron to generate an environmental resource embedding with permutation invariance.
[0022] The interactive decoder includes an LSTM unit and a query generation network. The LSTM unit is used for embedding the action and receiving the hidden state of the previous time step, and embedding the task features of the current subtask to be scheduled to output the current decoding state.
[0023] The query generation network is used to fuse the decoding state with the features of the current subtask to generate a dynamic task query vector.
[0024] The task query vector is multiplied by the feature vector of each computing node in the environmental resource embedding to calculate the matching score, and the probability distribution of the unloading action is generated by the Softmax function to obtain the unloading decision action of the current subtask.
[0025] Furthermore, in step S3, before processing the Markov decision process model, the priority of all subtasks in the DAG task is sorted by the heterogeneous earliest completion time algorithm to generate a scheduling sequence.
[0026] Furthermore, the task encoder extracts local topological features through a multi-layered graph convolutional network, and the feature propagation of the graph convolutional network is as follows:
[0027] Formula (1)
[0028] in, For feature propagation in graph convolutional networks; Let be the adjacency matrix of self-loops. , It is an adjacency matrix. The feature matrix of the DAG task; for The degree matrix; For the first The node feature matrix of the layer; For the first The learnable weight matrix of the layer; It is the ReLU activation function; after... After propagation in a layered graph convolutional network, the embedding vector of each node... It aggregated its Information about neighboring nodes is used to encode local dependencies and obtain the local topological features;
[0029] The external attention mechanism captures global dependencies between tasks through key and value storage units, calculates the similarity between the local topological features and the key storage units, and generates an attention map after normalization.
[0030] Formula (2)
[0031] in, For attention maps; Local topological features; For key storage units;
[0032] The value storage units are weighted and fused using the attention map to generate enhanced features:
[0033] Formula (3)
[0034] in, For enhanced features; For value storage units;
[0035] The enhanced features are fused through a linear layer to obtain a matrix of task feature embeddings;
[0036] The environment encoder is related to the environment state matrix. The first in Line, encoded as In order to obtain a set of environmental resource characteristics .
[0037] Furthermore, the decoding order of the interactive decoder follows the scheduling sequence, and the decoding steps of the interactive decoder include:
[0038] S3.1 in the Step-by-step LSTM cell updates internal state:
[0039] Formula (4)
[0040] in, This represents the hidden state of an LSTM cell; Long Short-Term Memory Network (LSTM) with LSTM cells; Embed the action selected in the previous moment; The feature vector of the currently scheduled task;
[0041] S3.2 Based on the hidden state and the feature vector of the currently scheduled task, a task query vector is generated through a fully connected layer:
[0042] Formula (5)
[0043] in, For task query vectors;
[0044] S3.3 Perform explicit attention matching between the task query vector and all resource feature vectors output by the environment encoder:
[0045] Formula (6)
[0046] in, For matching scores; For all resource feature vectors output by the environment encoder; All are learnable parameters;
[0047] S3.4 Softmax normalize the matching score to obtain the action probability distribution on all available nodes, and thus obtain the unloading decision action for the current subtask:
[0048] Formula (7)
[0049] in, The probability distribution of actions across all available nodes; For the first The matching score of each computing node; Sum the matching scores for all computed nodes.
[0050] Further, in step S4, the training steps of the agent include:
[0051] S4.1 Initialize the Actor network and Critic network, and input the mean pooling vectors of task feature embedding and environmental resource embedding;
[0052] S4.2 Interactive sampling is performed in the simulation environment. At the beginning of each round, the topology of the DAG task and the location of the port equipment are randomly generated. The agent executes a decision step according to the current strategy to complete the full scheduling of the DAG task. Experience data is obtained according to the process of executing the decision step, and the experience data is stored in the experience replay pool. Multiple rounds of sampling are repeated until the experience replay pool is full.
[0053] S4.3 The dominance value at each time step is calculated using generalized dominance estimation, and parameters are introduced... A trade-off is made between bias and variance to calculate the target value, and the target value is used to train the Critic network.
[0054] S4.4 Extract data in batches from the experience replay pool, perform gradient descent to update parameters, update the Actor network by maximizing the PPO-Clip objective function, and update the Critic network by minimizing the mean squared error loss function;
[0055] S4.5 Clear the experience replay pool and repeat steps S4.2 to S4.4 until the cumulative reward curve tends to stabilize or the preset number of training rounds is reached to obtain a trained agent.
[0056] Furthermore, in step S5, the Actor network parameters of the trained agent are extracted and deployed to the TOS control center or edge gateway of the smart port.
[0057] An edge computing DAG task offloading system based on graph attention reinforcement learning includes a processor and a storage device. The processor is configured to have one or more components, and the storage device is used to store one or more programs to execute the edge computing DAG task offloading method based on graph attention reinforcement learning as described in claim 1.
[0058] The beneficial effects of this invention are:
[0059] Since the optimization problem of DAG task unloading aims to find the optimal unloading strategy, which is an NP-hard combinatorial optimization problem with complex constraints and is subject to the highly dynamic nature of the port environment, traditional heuristic algorithms are difficult to adapt. This invention models the serialized unloading scheduling problem of DAG tasks as a Markov decision process model. By combining graph convolutional networks with external attention mechanisms, it not only captures the local dependencies of DAG tasks but also extracts the global structural priors, effectively solving the feature extraction problem of complex dependent tasks. The design of the environment encoder and interactive decoder makes the model permutation invariant to the number of computing nodes, and can adapt to the dynamic increase or decrease of the number of computing nodes without retraining. The explicit query-resource matching mechanism in the interactive decoder simulates the matching process of task demand and resource supply, which has higher decision efficiency and interpretability compared with traditional black-box output. This invention addresses the challenges of task scheduling difficulties caused by the high heterogeneity of devices, complex DAG task dependencies, and dynamic environmental changes in edge computing environments. It constructs a dependency-based task offloading framework based on graph attention reinforcement learning. This framework extracts features of tasks and environmental resources through a decoupled dual-stream encoder architecture, generates offloading strategies using a query-resource interactive decoder, and finally trains the framework using the PPO algorithm, achieving efficient collaborative processing of complex tasks. This invention can effectively perceive complex task structures, significantly reduce task completion time in dynamic environments, and is applicable to various edge computing collaborative scenarios such as smart ports and smart factories. Attached Figure Description
[0060] Figure 1 This is a flowchart of a preferred embodiment of the present invention, which describes an edge computing DAG task offloading method based on graph attention reinforcement learning.
[0061] Figure 2 This is a flowchart of the agent training process for an edge computing DAG task offloading method based on graph attention reinforcement learning, according to a preferred embodiment of the present invention.
[0062] Figure 3 This is a block diagram of the encoding and decoding structure of an edge computing DAG task offloading method based on graph attention reinforcement learning according to a preferred embodiment of the present invention.
[0063] Figure 4 This is a flowchart illustrating the online deployment and scheduling process of an edge computing DAG task offloading method based on graph attention reinforcement learning, according to a preferred embodiment of the present invention. Detailed Implementation
[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.
[0065] A preferred embodiment of the present invention provides a method for offloading edge computing DAG tasks based on graph attention reinforcement learning, comprising the following steps:
[0066] S1. Construct an edge computing system model, and generate DAG tasks with directed acyclic graph dependencies in the terminal device layer of the edge computing system model.
[0067] In this embodiment, the edge computing system model aims to digitally reproduce the operational scenarios of a modern container terminal, covering the entire lifecycle process from task generation, wireless transmission, edge computing to result feedback. The system architecture is logically divided into three layers: the port terminal equipment layer, the edge computing layer, and the cloud control layer.
[0068] The port terminal equipment layer is the source of computing tasks, consisting of intelligent machinery and sensing devices distributed across various work areas within the port area. Specifically, the terminal equipment cluster... This includes, but is not limited to: Intelligent Guided Vehicles (IGVs) and Automated Guided Vehicles (AGVs) operating in horizontal transport areas; machine vision terminals on automated rail-mounted gantry cranes (ARMGs) or tire-mounted gantry cranes (RTGs) operating in yards; onboard units of smart trucks responsible for container number identification at gates; and drones or robots responsible for port security patrols. When performing tasks such as autonomous driving path planning, multi-channel high-definition video stream analysis, LiDAR point cloud processing, and container damage detection, these devices generate a large number of computationally intensive and latency-sensitive data processing requests. Unlike simple, independent tasks, these workflows typically consist of multiple sub-tasks with logical sequences. For example, a typical "container number identification and location" task might include four sub-tasks: image preprocessing, character segmentation, OCR recognition, and coordinate verification, which must be executed sequentially. Therefore, this embodiment models each application task as a DAG. Let the first... The task is Represents the set of subtask nodes. This represents the set of directed edges. Each subtask It has two key attributes , representing the number of CPU clock cycles (in Gigacycles) required to complete the subtask and the amount of data (in Megabytes) generated by the subtask and to be passed to the successor node, respectively. Directed edge Subtasks yes The direct front drive means Must Execution can only begin after the previous execution is complete and its output data has been received. This topology introduces strict timing constraints.
[0069] The edge computing layer consists of multi-access edge computing (MEC) servers deployed on port infrastructure, designed to provide low-latency, local computing power to terminals. This embodiment defines a computing resource pool. It is highly heterogeneous, containing three types of nodes: The first type is local execution nodes, which are the terminal devices that generate tasks (such as the onboard industrial control computer of an AGV), and their computing power is relatively weak; the second type is fixed edge nodes, including high-performance edge servers deployed in the electrical rooms of the quay cranes, the smart poles in the yard, and the gate equipment rooms, denoted as... These first-type nodes have strong computing power but limited coverage; the third type is mobile collaborative nodes, which are nearby AGVs or intelligent trucks in an idle state. They can share idle computing power through V2V (Vehicle-to-Vehicle) technology, forming a dynamic "Vehicular Cloud". Due to the dynamic nature of port operations, the available computing node set... As time changes, for example, when an AGV leaves the light pole coverage area or enters a dormant state, its availability as a computing node immediately disappears.
[0070] In terms of communication and transmission models, this embodiment considers the complex electromagnetic environment within the port. Terminal devices and edge servers are connected via a 5G industrial private network or C-V2X communication technology. Considering that stacked metal containers and large quay cranes within the port area can cause severe obstruction and multipath fading effects on wireless signals, channel gain... It is modeled as a combination of Rayleigh fading (for non-line-of-sight (NLOS) paths) and Rice fading (for line-of-sight (LOS) paths), superimposed with distance-based path loss. According to Shannon's theorem, the terminal... With computing nodes Maximum transmission rate between Represented as:
[0071] in, For channel bandwidth, For terminal transmission power, The power of Gaussian white noise. This represents the interference power from other concurrent communication links. For dependent transmissions in a DAG task, if the predecessor task... and subsequent tasks If the data is scheduled to execute on the same compute node, the data transmission latency is negligible; if it is scheduled to different nodes, communication latency will occur. .
[0072] In terms of the computational model, nodes are defined. Its computing power is (CPU frequency, in GHz). Subtasks At the node The execution time is Since a computing node can only process one task at a time (or uses time-sharing multiplexing), tasks need to be queued on the node. Definition (Available Time) is the node The earliest time when all assigned tasks have been completed. Subtasks Actual start time It depends on two conditions: first, all data from the preceding tasks has been transmitted and arrived; second, the computing nodes... The previous tasks have been completed. That is:
[0073]
[0074] in It is a collection of precursor missions. This is the completion time of the precursor mission. Sub-task The completion time is .
[0075] The optimization objective of this embodiment is to minimize the total system completion time Makespan for all DAG tasks, while satisfying the aforementioned dependency and resource constraints. Makespan is defined as the maximum completion time of all exit nodes (nodes without successors) in the DAG. Formally, the optimization problem aims to find the optimal offloading strategy. , making Minimize. This is an NP-hard combinatorial optimization problem with complex constraints, and it is also subject to the highly dynamic nature of the port environment. Traditional heuristic algorithms are difficult to adapt to, so this invention introduces a deep reinforcement learning method to solve it.
[0076] S2. The unloading and scheduling problem of DAG tasks is modeled as a Markov decision process model, and the state space, action space and reward function are defined. The reward function takes minimizing the total completion time of the task as the optimization objective.
[0077] In step S2, the Markov decision process model is as follows: ,in For state space, For spatial actions, For the transition probability, This is the reward function.
[0078] The state space is for any decision time. ,state It contains all the information necessary to make the current optimal decision, and ,in, This is the feature matrix of the DAG task currently awaiting scheduling. It not only includes the static properties (computational cost) of each subtask Data volume It also embeds the topological structure information of the task (such as its position on the critical path and the number of predecessors and successors) through graph neural networks. Let the real-time state matrix of the port environment be given, and assume that the current system has... There are available computation nodes, and the matrix dimension is [number]. Each row represents a node's feature vector, containing: node type (local / edge / collaborative), current CPU utilization, and estimated available time. ), and the channel quality (SNR or estimated bandwidth) between the node and the task initiating terminal. This provides the position information of the current subtask in the topological sorting sequence, used to indicate the scheduling progress; This is the hidden state of the decoder at the previous time step, used to capture the temporal dependencies of historical scheduling actions. The state space in this embodiment includes the topology information of the DAG task, the computational and data volume of each subtask, and the available computing power, bandwidth status, and estimated available time of the edge computing nodes at the current time step.
[0079] Spatial movement To use the currently available computing resource pool Select a target node for the currently scheduled subtask; The action space is discrete, and the index... Corresponding to the Each computing node. For example, Represents local execution. This means offloading to the nearest light pole base station. This indicates that the AGV will be unloaded to a nearby idle AGV.
[0080] In this embodiment, the computing nodes include local terminal devices that generate tasks, nearby collaborating terminal devices, and roadside edge servers. Local terminal devices include smart trucks, automated guided vehicles (AGVs), yard cranes, and intelligent inspection robots.
[0081] The reward function is based on a dense reward mechanism with negative feedback based on the completion time increment, which distributes the reward... Defined as ,in; The estimated completion time for the subtask; The completion time of the precursor mission; A collection of precursor missions.
[0082] The design of the reward function directly determines the convergence direction of the agent. In this embodiment, at time... When the agent assigns subtasks Assigned to node Then, the system determines the current resource status. Calculate the estimated completion time of this subtask based on the transmission rate. .award The definition effectively penalizes the "incremental latency" of a task, i.e., the additional time overhead incurred during task execution (including queuing time, transmission time, and computation time). Furthermore, to prevent agents from choosing actions that lead to dependency deadlocks or severe timeouts, a significant negative penalty is applied when an action violates the constraints. At the end of an episode, if the Makespan of the entire DAG is better than the benchmark algorithm (such as HEFT), an additional positive reward is given. This is to encourage global optimization.
[0083] S3. Construct a deep reinforcement learning agent based on graph attention reinforcement learning to process the Markov decision process model through a dual-stream encoder and an interactive decoder to obtain the unloading decision action for the current subtask in the DAG task.
[0084] In step S3, before processing the Markov decision process model, the priority of all subtasks in the DAG task is sorted by the heterogeneous earliest completion time algorithm to generate a scheduling sequence.
[0085] Because DAG tasks must satisfy the constraint that the successor can only start after the predecessor has completed its execution, they cannot be scheduled arbitrarily out of order. Before inputting the data into the neural network, this embodiment first uses the upward ranking method in the Heterogeneous Earliest Finish Time (HEFT) algorithm. The value sorts all subtasks in the DAG by priority. The value reflects the "longest path length" from the task to the exit node and is a key indicator of the task's urgency. Subtasks of The calculation is performed recursively:
[0086] ,in, It is a task Average computation time across all available computing nodes It is a task arrive Average communication transmission time It is the set of direct successors. For the exit node, After the calculation is complete, all subtasks will be processed according to... Sort values in descending order to generate a scheduling sequence. This step simplifies the complex graph scheduling problem into a fixed-order sequence decision problem, greatly reducing the exploration difficulty of reinforcement learning.
[0087] In step S3, as Figure 2 As shown, the dual-stream encoder includes a task encoder and an environment encoder. The task encoder extracts local topological features of the DAG task through a graph convolutional network and captures global dependencies between tasks through an external attention mechanism to generate task feature embeddings.
[0088] The environmental encoder extracts features from the attributes of each computing node using a multilayer perceptron to generate an environmental resource embedding with permutation invariance.
[0089] The interactive decoder includes an LSTM unit and a query generation network. The LSTM unit is used for embedding the action of the previous time step and receiving the hidden state. It also embeds the action based on the task characteristics of the current subtask to be scheduled and outputs the current decoding state.
[0090] The query generation network is used to fuse the decoding state with the features of the current subtask to generate a dynamic task query vector;
[0091] The task query vector is multiplied by the feature vector of each computing node in the environmental resource embedding to calculate the matching score. The probability distribution of the unloading action is generated by the Softmax function to obtain the unloading decision action of the current subtask.
[0092] In step S3, the task encoder extracts local topological features through a multi-layer graph convolutional network. In this embodiment, the DAG task feature matrix is: The adjacency matrix is The feature propagation of a graph convolutional network is as follows:
[0093] Formula (1)
[0094] in, For feature propagation in graph convolutional networks; Let be the adjacency matrix of self-loops. , It is an adjacency matrix. The feature matrix of the DAG task; for The degree matrix; For the first The node feature matrix of the layer; For the first The learnable weight matrix of the layer; It is the ReLU activation function; after... After propagation in a layered graph convolutional network, the embedding vector of each node... It aggregated its Jump to neighbor information to encode local dependencies and obtain local topological features.
[0095] The computational complexity of traditional Self-Attention is The computational overhead is high when there are many tasks. This embodiment introduces two learnable shared memory unit matrices. and , The size of the memory unit is much smaller than the number of tasks. The external attention mechanism captures global dependencies between tasks through key and value storage units, calculates the similarity between local topological features and key storage units, and generates an attention map after normalization.
[0096] Formula (2)
[0097] in, For attention maps; Local topological features; For key storage units;
[0098] Enhanced features are generated by weighted fusion of value storage units using attention maps.
[0099] Formula (3)
[0100] in, For enhanced features; For value storage units;
[0101] The enhanced features are fused through a linear layer to obtain a matrix of task feature embeddings;
[0102] and Shared across all training samples, it acts as a global prior knowledge base, capable of memorizing common task structure patterns in smart ports, such as the pipeline structure specific to OCR recognition tasks. This endows the model with strong generalization ability, enabling it to handle unseen DAG topologies. The final output task embedding matrix is denoted as... .
[0103] The environmental encoder for the environmental state matrix The first in Line, encoded as In order to obtain a set of environmental resource characteristics .
[0104] To address the dynamic changes in the number of computing nodes, Given that the environment is a variable and there are no fixed topological connections between nodes, this embodiment designs an environment encoder with permutation invariance. This encoder does not use a fully connected layer to process the entire environment matrix; instead, it employs a multilayer perceptron (MLP) with shared weights to independently process the feature vector of each computation node. For the environment state matrix... The first in row (i.e., the first row) Attribute vectors of each node The encoding formula is: The output yields a set of environmental resource characteristics. This point-by-point processing design makes the model insensitive to the order and number of input nodes. Regardless of whether there are 5 or 50 available nodes within the port, the model can reuse the same set of parameters for feature extraction, greatly enhancing the system's scalability.
[0105] The interactive decoder is responsible for generating an unloading strategy based on the current task and resource states. This embodiment employs an LSTM-based pointer network architecture and introduces a "query-resource" interaction mechanism. The interactive decoder decodes according to the scheduling sequence, and the decoding steps include:
[0106] S3.1 in the Step-by-step LSTM cell updates internal state:
[0107] Formula (4)
[0108] in, This represents the hidden state of an LSTM cell; Long Short-Term Memory Network (LSTM) with LSTM cells; Embed the action selected in the previous moment; The feature vector of the task to be scheduled; in this embodiment, the input includes the action embedding selected in the previous time step. and the feature vector of the currently scheduled task .
[0109] S3.2 Based on the hidden state and the feature vector of the currently scheduled task, a task query vector is generated through a fully connected layer:
[0110] Formula (5)
[0111] in, This is the task query vector; semantically, the task query vector represents the ideal computing resource requirements of the current subtask in the current historical context.
[0112] S3.3 Perform explicit attention matching between the task query vector and all resource feature vectors output by the environment encoder, and then... All resource feature vectors output by the environment encoder Explicit attention matching is performed. The matching score is calculated using the Bahdanau Attention method:
[0113] Formula (6)
[0114] in, For matching scores; For all resource feature vectors output by the environment encoder; All of these are learnable parameters, and learnable parameters intuitively reflect the first... The degree of matching between a computing node and the current task requirements; for example, a computing-intensive task will have a high matching score with a high-frequency edge server.
[0115] S3.4 Softmax normalizes the matching scores to obtain the action probability distribution across all available nodes, thus determining the unloading decision action for the current subtask:
[0116] Formula (7)
[0117] in, The probability distribution of actions across all available nodes; For the first The matching score of each computing node; Sum the matching scores for all computed nodes.
[0118] S4. Execute the unloading decision action of the current subtask in the simulation environment to obtain experience data, store the experience data in the experience replay pool, train the agent through the near-end policy optimization (PPO) algorithm, and update the network parameters until the Markov decision process model converges to obtain the trained agent.
[0119] In step S4, as Figure 3 As shown, the training steps for an agent include:
[0120] S4.1 Initialize the Actor network and Critic network, and input the mean pooling vectors of task feature embedding and environmental resource embedding.
[0121] In step S4.1, the Actor network includes a dual-stream encoder and a decoder, with the following parameters: The Critic network is a value assessment network with the following parameters: The Critic network is used to estimate the value function of the current state. Its input is the mean pooling vector of task embedding and resource embedding.
[0122] S4.2 Interactive sampling is performed in the simulation environment. At the beginning of each round, the topology of the DAG task and the location of the port equipment are randomly generated. The agent executes decision steps according to the current strategy to complete the full scheduling of a DAG task. Experience data is obtained based on the execution of decision steps and stored in the experience replay pool. Multiple rounds of sampling are repeated until the experience replay pool is full.
[0123] In step S4.2, interactive sampling is performed in the smart port simulation environment. At the start of each episode, the DAG task topology and port equipment locations are randomly generated. The agent then performs actions according to the current strategy. implement Each step involves decision-making to complete the full scheduling of a DAG. In the decision-making step... Intelligent agent observes state Output action distribution and sample to obtain The environment returns a reward after performing an action. and the next state Empirical data The data is then stored in the experience replay pool. Multiple episodes are collected repeatedly until the experience replay pool is full.
[0124] S4.3 The dominance value at each time step is calculated using generalized dominance estimation, and parameters are introduced... A trade-off is made between bias and variance to calculate the target value, which is then used to train the Critic network.
[0125] In step S4.3, the Temporal Difference Error (TD) is first calculated: ;
[0126] Then, the dominance value is calculated recursively:
[0127] Simultaneously calculate the target value It is used to train the Critic network.
[0128] S4.4 Extract data in batches from the experience replay pool, perform gradient descent to update parameters, update the Actor network by maximizing the PPO-Clip objective function, and update the Critic network by minimizing the mean squared error loss function.
[0129] In step S4.5, data is drawn in mini-batch from the experience replay pool, and gradient descent is performed to update the parameters. Actor network update: Maximize the PPO-Clip objective function. Define probability ratios
[0130] Formula (8)
[0131] The second term is the policy entropy regularization term, used to encourage exploration and prevent premature convergence to a local optimum. Critic network update: Minimize the mean squared error loss function. . The total loss function is Updated via backpropagation algorithm and .
[0132] S4.5 Clear the experience replay pool and repeat steps S4.2 to S4.4 until the cumulative reward curve tends to stabilize or the preset number of training rounds is reached to obtain a trained agent.
[0133] S5. Deploy the trained agent in the edge computing system model to perform online offloading and scheduling of DAG tasks generated in real time.
[0134] In step S5, the Actor network parameters of the trained agent are extracted and deployed to the TOS control center or edge gateway of the smart port.
[0135] In step S5, as Figure 4 As shown, the online execution process includes:
[0136] (1) Task triggering: When an AGV or quay crane in the port generates a new DAG operation request, the scheduling engine is triggered.
[0137] (2) Status awareness: Real-time load status and network bandwidth information of all available computing nodes (including idle vehicles and edge servers) are retrieved through the port IoT platform.
[0138] (3) Model Inference: Input the task DAG structure and resource state into the trained model. The model first calculates... The data is sorted, and then features are extracted by the encoder. The decoder outputs the unloading target node with the highest probability for each subtask.
[0139] (4) Command distribution: The system generates specific control commands (such as "send the data packet of task node 3 to IP:192.168.1.105 for execution") based on the unloading strategy output by the model, and distributes them to the corresponding terminal devices through the 5G network.
[0140] (5) Closed-loop feedback: After the task is completed, the actual completion time is recorded and the data is stored as offline samples for subsequent fine-tuning and continuous learning of the model.
[0141] This embodiment also discloses an edge computing DAG task offloading system based on graph attention reinforcement learning, including a processor and a storage device. The processor is configured with one or more programs, and the storage device is used to store one or more programs to execute the edge computing DAG task offloading method based on graph attention reinforcement learning of claim 1.
Claims
1. A method for offloading edge computing DAG tasks based on graph attention reinforcement learning, characterized in that, Includes the following steps: S1. Construct an edge computing system model, wherein the terminal device layer in the edge computing system model generates DAG tasks with directed acyclic graph dependencies; S2. The unloading and scheduling problem of the DAG task is modeled as a Markov decision process model, and the state space, action space and reward function are defined, wherein the reward function takes minimizing the total completion time of the task as the optimization objective. S3. Construct a deep reinforcement learning agent based on graph attention reinforcement learning to process the Markov decision process model through a dual-stream encoder and an interactive decoder to obtain the unloading decision action for the current subtask in the DAG task. S4. Execute the unloading decision action of the current subtask in the simulation environment to obtain experience data, and store the experience data in the experience replay pool; The agent is trained by using the Proximal Policy Optimization (PPO) algorithm and the network parameters are updated until the Markov Decision Process model converges, thus obtaining a trained agent. S5. Deploy the trained agent in the edge computing system model to perform online offloading and scheduling of the DAG tasks generated in real time.
2. The edge computing DAG task offloading method based on graph attention reinforcement learning according to claim 1, characterized in that: In step S2, the Markov decision process model is as follows: ,in For state space, For spatial actions, For the transition probability, This is the reward function.
3. The edge computing DAG task offloading method based on graph attention reinforcement learning according to claim 2, characterized in that: The state space represents any decision time. ,state It contains all the information necessary to make the current optimal decision, and ,in, This represents the feature matrix of the DAG task currently awaiting scheduling; This is a real-time status matrix of the port environment; This refers to the position information of the current subtask in the topological sorting sequence; This represents the hidden state of the decoder at the previous moment; The spatial action To use the currently available computing resource pool Select a target node for the currently scheduled subtask; ; The reward function is based on a dense reward mechanism with negative feedback based on the completion time increment, which will reward... Defined as ,in; The estimated completion time for the subtask; The completion time of the precursor mission; A collection of precursor missions.
4. The edge computing DAG task offloading method based on graph attention reinforcement learning according to claim 1, characterized in that: In step S3, the dual-stream encoder includes a task encoder and an environment encoder. The task encoder extracts the local topological features of the DAG task through a graph convolutional network and captures the global dependencies between tasks through an external attention mechanism to generate task feature embeddings. The environmental encoder extracts features from the attributes of each computing node using a multilayer perceptron to generate an environmental resource embedding with permutation invariance. The interactive decoder includes an LSTM unit and a query generation network. The LSTM unit is used for embedding the action and receiving the hidden state of the previous time step, and embedding the task features of the current subtask to be scheduled to output the current decoding state. The query generation network is used to fuse the decoding state with the features of the current subtask to generate a dynamic task query vector. The task query vector is multiplied by the feature vector of each computing node in the environmental resource embedding to calculate the matching score, and the probability distribution of the unloading action is generated by the Softmax function to obtain the unloading decision action of the current subtask.
5. The edge computing DAG task offloading method based on graph attention reinforcement learning according to claim 4, characterized in that: In step S3, before processing the Markov decision process model, the priority of all subtasks in the DAG task is sorted by the heterogeneous earliest completion time algorithm to generate a scheduling sequence.
6. The edge computing DAG task offloading method based on graph attention reinforcement learning according to claim 5, characterized in that: The task encoder extracts local topological features through a multi-layer graph convolutional network, and the feature propagation of the graph convolutional network is as follows: Official (1) in, For feature propagation in graph convolutional networks; Let be the adjacency matrix of self-loops. , It is an adjacency matrix. The feature matrix of the DAG task; for The degree matrix; For the first The node feature matrix of the layer; For the first The learnable weight matrix of the layer; It is the ReLU activation function; after... After propagation in a layered graph convolutional network, the embedding vector of each node... It aggregated its Information about neighboring nodes is used to encode local dependencies and obtain the local topological features; The external attention mechanism captures global dependencies between tasks through key and value storage units, calculates the similarity between the local topological features and the key storage units, and generates an attention map after normalization. Official (2) in, For attention maps; Local topological features; For key storage units; The value storage units are weighted and fused using the attention map to generate enhanced features: Official (3) in, For enhanced features; For value storage units; The enhanced features are fused through a linear layer to obtain a matrix of task feature embeddings; The environment encoder is related to the environment state matrix. The first in Line, encoded as In order to obtain a set of environmental resource characteristics .
7. The edge computing DAG task offloading method based on graph attention reinforcement learning according to claim 6, characterized in that: The interactive decoder decodes in the order of the scheduling sequence, and the decoding steps of the interactive decoder include: S3.1 in the Step-by-step LSTM cell updates internal state: Official (4) in, This represents the hidden state of an LSTM cell; Long Short-Term Memory Network (LSTM) with LSTM cells; Embed the action selected in the previous moment; The feature vector of the currently scheduled task; S3.2 Based on the hidden state and the feature vector of the currently scheduled task, a task query vector is generated through a fully connected layer: Official (5) in, For task query vectors; S3.3 Perform explicit attention matching between the task query vector and all resource feature vectors output by the environment encoder: Official (6) in, For matching scores; For all resource feature vectors output by the environment encoder; All are learnable parameters; S3.4 Softmax normalize the matching score to obtain the action probability distribution on all available nodes, and thus obtain the unloading decision action for the current subtask: Official (7) in, The probability distribution of actions across all available nodes; For the first The matching score of each computing node; Sum the matching scores for all computed nodes.
8. The edge computing DAG task offloading method based on graph attention reinforcement learning according to claim 4, characterized in that: In step S4, the training steps of the agent include: S4.1 Initialize the Actor network and Critic network, and input the mean pooling vectors of task feature embedding and environmental resource embedding; S4.2 Interactive sampling is performed in the simulation environment. At the beginning of each round, the topology of the DAG task and the location of the port equipment are randomly generated. The agent executes a decision step according to the current strategy to complete the full scheduling of the DAG task. Experience data is obtained according to the process of executing the decision step, and the experience data is stored in the experience replay pool. Multiple rounds of sampling are repeated until the experience replay pool is full. S4.3 The dominance value at each time step is calculated using generalized dominance estimation, and parameters are introduced... A trade-off is made between bias and variance to calculate the target value, and the target value is used to train the Critic network. S4.4 Extract data in batches from the experience replay pool, perform gradient descent to update parameters, update the Actor network by maximizing the PPO-Clip objective function, and update the Critic network by minimizing the mean squared error loss function; S4.5 Clear the experience replay pool and repeat steps S4.2 to S4.4 until the cumulative reward curve tends to stabilize or the preset number of training rounds is reached to obtain a trained agent.
9. The edge computing DAG task offloading method based on graph attention reinforcement learning according to claim 8, characterized in that: In step S5, the Actor network parameters of the trained agent are extracted and deployed to the TOS control center or edge gateway of the smart port.
10. An edge computing DAG task offloading system based on graph attention reinforcement learning, characterized in that: The device includes a processor and a storage device, wherein the processor is configured with one or more processors and the storage device is used to store one or more programs for executing the edge computing DAG task offloading method based on graph attention reinforcement learning as described in claim 1.