Task offloading method and system using deep reinforcement learning and attention mechanism

Abstract

This invention relates to a task offloading method and system utilizing deep reinforcement learning and an attention mechanism, and belongs to the field of task offloading technology. This method involves acquiring and preprocessing data, including edge server information, user information, and task information. The preprocessed data is then fed into a trained task offloading model, and task offloading is achieved based on the Dedicated Performance Assessment (DPAQN) algorithm. The DPAQN algorithm exhibits significant advantages in optimizing the overall performance of task offloading, outperforming existing algorithms by an average of approximately 20.71% to 30.39%.

Description

Technical Field

[0001] The present invention relates to a task offloading method and system using deep reinforcement learning and attention mechanism, and belongs to the technical field of task offloading. Background Art

[0002] In recent years, edge computing has rapidly emerged as an emerging computing paradigm. Its core advantage lies in migrating data processing and computing tasks to the edge of the network, close to end devices. This layout significantly reduces latency and energy consumption during task computation, making it crucial for applications requiring immediate response times and enabling efficient processing of resource-intensive and time-sensitive tasks. Edge computing also reduces the burden on data centers, reduces network congestion, and enhances data security and privacy by localizing data processing. This architecture not only accelerates system response times but also provides a solid foundation for building smarter and more efficient Internet of Things (IoT) applications. Therefore, edge computing demonstrates significant potential for task offloading and is expected to become a key development direction in future computing technology.

[0003] With the rapid development of edge computing, computation offloading is becoming a key technology in edge computing. Offloading computing tasks from IoT terminals to edge environments can effectively address deficiencies in computing performance and energy efficiency. However, while offloading tasks to edge environments brings convenience, it also brings a series of challenges:

[0004] 1) How to flexibly schedule heterogeneous resources of edge servers to adapt to different tasks, including differences in computing, processing speed, and offloading requirements, to ensure that tasks are matched with the best server.

[0005] 2) How to maintain the flexibility and adaptability of task offloading strategies in the face of network fluctuations and changes in user demand to maintain system stability and performance.

[0006] 3) How to strike a balance between improving performance and reducing energy consumption to achieve economic and environmental sustainability of task offloading.

[0007] In response to the above challenges, researchers have proposed a variety of solutions, which are mainly divided into traditional methods and artificial intelligence (AI) methods.

[0008] Traditional methods, including mathematical optimization, Lyapunov optimization, and heuristic algorithms, each have advantages in ensuring optimality and stability, but also suffer from scalability, complexity, and suboptimal solutions. Peng et al. proposed an intelligent computation offloading and resource allocation method (EECCT) based on end-edge-cloud collaboration to optimize computation offloading and resource allocation in the Industrial Internet of Things (IIoT). Nguyen et al. studied a fast and efficient algorithm (FEA) to maximize the total computation rate of wireless powered mobile edge computing (WPMEC) in IoT networks by jointly optimizing the transmission power, backscatter coefficient, time division ratio, and computation mode decision of an OFDMA system. Wu et al. proposed a nonlinear exponential inertia-weighted particle swarm optimization (PSO) algorithm for solving edge-cloud collaborative multi-task computation offloading models. By dynamically adjusting the inertia weight, this algorithm can overcome the premature convergence of the standard PSO and effectively avoid falling into local optimal solutions. Zhang et al. proposed a new task scheduling scheme (GA-EC) based on a genetic algorithm (GA) for the efficient scheduling of resource-intensive tasks in edge computing environments. The advantage of this solution is that it can achieve multi-objective optimization and improve the robustness and adaptability of the system through dynamic scheduling strategies.

[0009] Traditional methods are advantageous in rapid deployment and low-complexity scenarios due to their simplicity, but they are ill-suited to the dynamic nature of multi-user and resource-constrained environments. In contrast, AI-based methods, through their adaptability and predictive capabilities, provide real-time decision support in resource-constrained and complex network environments, effectively processing high-dimensional data and learning feature representations to optimize task offloading. However, AI methods also face challenges in edge computing. These include the limited computing and storage resources of edge devices, which restrict model complexity, and the high demands on algorithm robustness and real-time performance due to network instability and frequent device changes.

[0010] Artificial intelligence-based methods, such as Q-learning, DRL, DQN, SVM, and ANN, have developed rapidly since 2019. They can dynamically evaluate network conditions and user behavior, enabling intelligent task allocation and resource optimization. Li et al. proposed Q-learning-based and DRL-based models, respectively, using the sum of all latency and energy consumption as the overall optimization objective, effectively reducing costs. Xiao et al. designed a DRL-based mobile resource offloading scheduling for edge cloud environments to reduce latency and energy consumption. An actor-network is designed to select scheduling policies, and a critic-network updates actor-network parameters to improve computational performance. Wang et al. proposed an algorithm based on meta-reinforcement learning (MRL) to solve the task offloading problem. This algorithm can quickly adapt to new environments with small sample sizes and effectively reduce task latency. Liu et al. proposed a DQN-based multi-workflow scheduling method that can cope with time-varying performance fluctuations in edge service performance. Sheng et al. proposed a deep reinforcement learning-based method for IoT edge computing task scheduling. By modeling the problem as an MDP and applying a policy-based deep reinforcement learning algorithm, this method performs well on multiple evaluation metrics. Heidari et al. combined Double Q-learning and Deep Post-Decision State (PDS) learning to propose a new learning method called D2QDPO3 to deal with the problem of dynamic computation offloading in IoT edge computing.

[0011] Deep learning automatically extracts features, reinforcement learning learns optimal strategies through interaction, and support vector machines (SVMs) and neural networks process high-dimensional data to optimize task offloading decisions. However, the complexity of models that can be deployed using AI-based methods is often limited by the limited computing and storage resources of edge devices. Summary of the Invention

[0012] In response to the shortcomings of the existing technology, this paper proposes a task offloading method using deep reinforcement learning and attention mechanism;

[0013] The present invention also proposes a task offloading system using deep reinforcement learning and attention mechanism.

[0014] In order to solve the task offloading problem more practically, in the present invention:

[0015] 1) Considering the heterogeneity of edge servers and user diversity, a task offloading strategy based on directed acyclic graph (DAG) is proposed to optimize the task offloading problem in edge computing environment.

[0016] 2) This paper applies the asymptotic rectangular window attention mechanism to edge computing task offloading for the first time, which improves decision quality. Combining this with deep reinforcement learning to form the DPAQN algorithm is expected to make new contributions to optimizing task offloading in edge computing environments.

[0017] 3) The proposed DPAQN algorithm comprehensively considers three metrics: total task completion time, energy consumption, and throughput, aiming to develop a more efficient and energy-efficient offloading algorithm. Comparisons with five other algorithms demonstrate that DPAQN improves the overall performance of task offloading.

[0018] The technical solution of the present invention is:

[0019] Task offloading methods using deep reinforcement learning and attention mechanisms include:

[0020] Data acquisition and preprocessing; data includes edge server information, user information, and task information;

[0021] The preprocessed data is input into the trained task offloading model, and task offloading is achieved based on the DPAQN algorithm.

[0022] Further preferably, the pretreatment comprises:

[0023] Noise addition;

[0024] Dynamic environment simulation: By continuously generating new tasks, a dynamic edge computing environment is simulated to ensure the continuity of task flow;

[0025] Parameter settings.

[0026] According to the preferred embodiment of the present invention, the objective function in the task offloading model is Expressed as:

[0027] ;

[0028] in, ; Represents the maximum completion time of all completed tasks, Represents the number of completed tasks; Represents the total completion time of the task;

[0029] ; Represents the maximum completion time of all completed tasks, Represents the total energy consumption of the task;

[0030] w1,w2,w3 are weights, Refers to the total throughput.

[0031] More preferably, the values ​​of w1, w2, and w3 are 0.25, 0.5, and 0.25 respectively.

[0032] According to the preferred embodiment of the present invention, the total task completion time TT includes the transmission time and running time of all tasks; it is specifically defined as:

[0033] ;

[0034] Where n is the number of users, l is the number of subtasks for each user, represents the data volume of the i-th task of user device k, represents the bandwidth allocated to the i-th task of user device k, is the transmission time of the i-th task of user device k, represents the time required for the local execution of the i-th task of user device k, represents the ratio of local and remote execution rates, is the running time of the i-th task of user device k;

[0035] The total energy consumption of a task includes the communication energy consumption and operation energy consumption of all tasks; it is specifically defined as:

[0036] ;

[0037] in, is the transmission power of the ith task of user equipment k, represents the communication time of the i-th task of user device k; represents the communication energy consumption of the i-th task of user equipment k, that is, ; represents the running energy consumption of the i-th task of user device k.

[0038] According to the preferred embodiment of the present invention, in the edge computing environment, task offloading refers to the process of migrating computing tasks from user devices to edge servers or edge nodes for execution. The task offloading decision is regarded as an MDP, which is represented by a five-tuple To express;

[0039] Where s is the state space, which is used to describe the state of the system environment. According to the current state, a task is selected from all tasks waiting to be assigned. After the task is offloaded to the edge server, the next state is obtained.

[0040] a is the action space, including all actions selected in the current state;

[0041] r is the immediate reward, which is used to evaluate the feedback of the action;

[0042] p is the state transition probability, which describes the probability of transitioning from one state to another;

[0043] It is a discount factor that measures the importance of immediate rewards and long-term rewards. It belongs to the interval [0,1], where 0 means only considering immediate rewards and 1 means that long-term rewards and immediate rewards are equally important.

[0044] More preferably, the calculation formula of r is as follows:

[0045] ;

[0046] in, is the transmission time of the i-th task of user device k, is the running time of the i-th task of user device k, is the communication energy consumption of the i-th task of user device k, ) is the running energy consumption of the i-th task of user device k.

[0047] Preferably, according to the present invention, the task offloading model includes an action network and a target network;

[0048] The action network and target network have the same structure;

[0049] In the action network, it is used to evaluate the Q value of each action in the current state and select the optimal action. Its parameters are continuously updated through training;

[0050] In the target network, it is used to provide a stable training target. The parameters of the target network are copied from the action network Q network every C steps.

[0051] According to the present invention, the action network preferably includes several neural network layers, specifically including: an input layer, a first hidden layer, a second hidden layer, an asymptotic rectangular window attention mechanism, and an output layer;

[0052] The input layer of the action network receives the feature vector, which is processed by the linear transformation and relu activation function of the first hidden layer before being output to the second hidden layer. After the second hidden layer, the asymptotic rectangular window attention mechanism is introduced. The asymptotic rectangular window attention mechanism divides the output of the second hidden layer into multiple windows and performs a weighted summation of the features within each window. Specifically,

[0053] 1) First, the features within each rectangular window, i.e., the features processed by the linear transformation and ReLU activation function of the first hidden layer, are transformed into feature representations through linear transformation. Then, nonlinear processing is performed using the tanh activation function. Next, the attention weight within each window is obtained by calculating the dot product similarity between the feature representation and the learnable vector and normalizing it through the Softmax function. Finally, the weighted features of all windows are concatenated to form the output of the asymptotic rectangular window attention mechanism.

[0054] 2) The output of the asymptotic rectangular window attention mechanism passes through the first hidden layer, the second hidden layer, and the asymptotic rectangular window attention mechanism again to further extract and abstract features;

[0055] The predicted Q value of each action is output through the last linear transformation plus the relu activation function. The predicted Q value represents the expected return of taking each possible action in the current state.

[0056] Preferably, according to the present invention, task offloading is implemented based on the DPAQN algorithm, including:

[0057] First, initialize the edge computing environment and DQN parameters and state s;

[0058] Using the current state s as input, the action network evaluates the predicted Q-values ​​of all possible actions, i.e. ; and select action a with the highest Q value to execute;

[0059] Execute action a to interact with the environment, observe the reward r, and obtain the next state s' and the iteration end flag done;

[0060] Then, the current state s, action a, reward r, next state s' and iteration end flag done are stored in the experience replay buffer for subsequent learning; a small batch of data is randomly sampled from the experience replay for training;

[0061] Take the next state s' as the input of the target network Q' and calculate the future ;

[0062] Combine the reward r and the iteration end flag done to calculate , as shown below;

[0063] ;

[0064] in, It is the parameter of the target network, and the value of done is 0 or 1;

[0065] The action network receives the state s and action a in the mini-batch sampling, and the Q value of the action network is combined with the action a in the mini-batch sampling to obtain ; Then use the Mean Squared Error (MSE) loss function to calculate and loss , as shown below;

[0066] ;

[0067] m represents the number of samples in the mini-batch sampling;

[0068] Update the parameters of the action network through back propagation according to the obtained Loss, and copy the parameters of the action network Q to the target network Q' every C steps;

[0069] Determine whether the termination condition is met. If so, the process ends; if not, the steps after returning to the initial state continue.

[0070] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a task offloading method using deep reinforcement learning and an attention mechanism are implemented.

[0071] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a task offloading method using deep reinforcement learning and an attention mechanism.

[0072] A task offloading system using deep reinforcement learning and attention mechanism, including:

[0073] The data acquisition and preprocessing module is configured to: data acquisition and preprocessing;

[0074] The task offloading module is configured to input the preprocessed data into the trained task offloading model and implement task offloading based on the DPAQN algorithm.

[0075] The beneficial effects of the present invention are:

[0076] 1. Enhanced data security and privacy protection: By offloading tasks to edge servers, the present invention can process data close to the data source, reducing the amount of data transmitted in the network, thereby enhancing data security and privacy protection.

[0077] 2. Comprehensively Optimizing Task Offloading Performance: The proposed DPAQN algorithm comprehensively considers three key metrics: total task completion time, energy consumption, and throughput. By introducing the asymptotic rectangular window attention mechanism into the DQN, it significantly improves the quality of task offloading decisions. Experimental results demonstrate that the DPAQN algorithm offers significant advantages in optimizing the overall performance of task offloading, outperforming existing algorithms (such as DQN, Double DQN, Dueling DQN, Prioritized Replay, and PPO) by an average of approximately 20.71% to 30.39%.

[0078] 3. Improved system resource utilization: This invention takes into account the heterogeneity of edge servers and the diversity of users. Through a directed acyclic graph (DAG)-based task offloading strategy, tasks are rationally distributed to different edge servers for execution, achieving optimal resource allocation. This not only improves edge server resource utilization, but also reduces the burden on data centers and network congestion.

[0079] 4. Improving Algorithm Generalization: Comparative experiments using different random seeds and a variety of user configurations verified the DPAQN algorithm's excellent generalization. These results demonstrate that DPAQN maintains stable performance across diverse scenarios, alleviating the common weaknesses of weak robustness and poor generalization in AI algorithms and demonstrating its applicability to a wide range of practical applications.

[0080] 5. Boosting the Intelligent Development of the Industrial Internet of Things: This invention has significant application value within the Industrial Internet of Things (IIoT). Intelligent manufacturing systems require processing large amounts of sensor and device data, placing high demands on energy consumption, task completion time, and throughput. The DPAQN algorithm aims to optimize the overall performance of task offloading and dynamically adjust task allocation, potentially supporting efficient and intelligent production processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 A schematic diagram of the overall architecture for task offloading in edge computing environments;

[0082] Figure 2 This is a flowchart of task offloading based on DQN;

[0083] Figure 3 Schematic diagram of action network structure;

[0084] Figure 4 Schematic diagram of TEH for five different algorithms under different numbers of users Figure 1 ;

[0085] Figure 5 Schematic diagram of TEH for five different algorithms under different numbers of users Figure 2 . DETAILED DESCRIPTION

[0086] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0087] Example 1

[0088] Task offloading methods using deep reinforcement learning and attention mechanisms include:

[0089] Data acquisition and preprocessing; data includes edge server information, user information, and task information;

[0090] Edge server information, including the following:

[0091] Basic information: ID, number of users, basic rate ratio of local and remote execution, user list object, total bandwidth, system status refresh frequency, uninstallation frequency, bandwidth allocation method;

[0092] Real-time information: list of tasks being executed, list of tasks being transmitted, remote waiting list, user priority list, occupied bandwidth, system running time, current system CPU usage, last system action, last system score;

[0093] Log information: offloaded task records, total transmission time, total transmission energy consumption, total execution time, total number of offloaded tasks, number of failed tasks, and total rewards.

[0094] User information, including the following:

[0095] Basic information: ID, user task list, task transmission data volume, task transmission time distribution, task local execution time, task CPU occupancy, initial task number, task transmission and local execution energy consumption, task deadline, user priority;

[0096] Log information: throughput, energy consumption, and total communication time.

[0097] Task information, including the following:

[0098] Basic information: user ID, task ID, number of task instructions, size of transmitted data, upload time, download time, local processing time, CPU resource requirements, transmission energy consumption, local execution energy consumption, task type, and task status;

[0099] Real-time information: remaining remote execution time of the task, remaining transmission time of the task, bandwidth allocated to the task, task start time, task completion time, task unloading start time, task execution time, and task transmission time.

[0100] The preprocessed data is input into the trained task offloading model, and task offloading is achieved based on the DPAQN algorithm.

[0101] Example 2

[0102] The task offloading method using deep reinforcement learning and attention mechanism described in Example 1 is different in that:

[0103] Preprocessing, including:

[0104] Noise addition: To simulate the randomness and uncertainty in a real environment, random noise (ξ) is added to the computation time of the task, ranging from 1 to 5. This noise simulates the computation time fluctuations that may exist in a real environment.

[0105] Dynamic environment simulation: By continuously generating new tasks (when a task is completed, a new task is immediately generated), a dynamic edge computing environment is simulated to ensure that the task flow is continuous;

[0106] Parameter settings. Based on the requirements of the simulation environment, we set some key parameters, such as network bandwidth (50 Mbps), number of user devices (ranging from 20 to 90), minimum computation time unit (25 milliseconds), and minimum data unit (6 milliseconds).

[0107] Objective function in the task offloading model Expressed as:

[0108] ;

[0109] in, ; Represents the maximum completion time of all completed tasks, Represents the number of completed tasks; Represents the total completion time of the task;

[0110] ; Represents the maximum completion time of all completed tasks, Represents the total energy consumption of the task;

[0111] w1,w2,w3 are weights, Refers to the total throughput.

[0112] It is a comprehensive optimization objective used to measure the performance of the task offloading strategy in the edge computing environment. The total completion time of the task ( )、Total energy consumption( ) and total throughput ( ) into a single objective function. Among them, w1, w2, and w3 are weight coefficients, which respectively represent the importance of different indicators.

[0113] The values ​​of w1, w2, and w3 are 0.25, 0.5, and 0.25, respectively. This approach aims to optimize the energy consumption of task offloading, while also optimizing the total completion time and throughput of offloaded tasks, striving to build an energy-efficient and efficient offloading model. A smaller TEH value indicates better overall performance.

[0114] In order to transform the multi-objective optimization problem into a single-objective optimization problem, the weighted sum method (WSM) is introduced, that is, the values ​​of the three objective functions are normalized and a set of weights are used to sum them. , , to form a new overall optimization objective. Weight , , The range is , and the sum of the three is 1, which reflects the importance of different goals.

[0115] Based on the overall model described above, we offload pending DAG-based tasks to edge servers, aiming to comprehensively optimize task completion time, energy consumption, and throughput. The optimization problem is described below.

[0116] The total task completion time TT includes the transmission time and running time of all tasks; it is specifically defined as:

[0117] ;

[0118] Where n is the number of users, l is the number of subtasks for each user, represents the data volume of the i-th task of user device k, represents the bandwidth allocated to the i-th task of user device k, is the transmission time of the i-th task of user device k, represents the time required for the local execution of the i-th task of user device k, represents the ratio of local and remote execution rates, is the running time of the i-th task of user device k;

[0119] The total energy consumption of a task includes the communication energy consumption and operation energy consumption of all tasks; it is specifically defined as:

[0120] ;

[0121] in, is the transmission power of the i-th task of user equipment k, which depends on the transmission distance and the power control strategy of the device. represents the communication time of the i-th task of user device k; represents the communication energy consumption of the i-th task of user equipment k, that is, ; represents the energy consumption of the i-th task of user device k. It is pre-set based on user settings and task characteristics.

[0122] Throughput refers to the amount of data successfully transmitted within a certain period of time. For each user, the total amount of data for all tasks and the total time of the simulation are recorded. At the end of the simulation, the total amount of data for each user is divided by the simulation time to obtain the average throughput. This paper defines the total throughput as .

[0123] This paper aims to leverage the advantages of artificial intelligence algorithms while taking into account the limited resources of edge devices and network instability. It designs a DPAQN algorithm that combines deep reinforcement learning with the asymptotic rectangular window attention mechanism. This algorithm comprehensively considers the realities of task offloading in edge computing environments, while simultaneously optimizing latency, energy consumption, and throughput, striving to make the model as stable and efficient as possible. Experiments ultimately demonstrate its effectiveness.

[0124] In an edge computing environment, task offloading is a key optimization problem, which involves allocating computing tasks between terminal devices and edge servers. The design goal of the present invention is to perform efficient and energy-saving task offloading in an edge computing environment while taking into account the heterogeneity of edge servers and the diversity of users. The heterogeneity of edge servers is reflected in the differences in computing resources, processing speed, and task offloading requirements of edge servers. Since different edge servers usually have different processing capabilities, storage capacity, and network bandwidth, they show diversity in task execution and resource allocation. In addition, the number of tasks that need to be offloaded, the time required for offloading, and the energy consumption are also different for different user devices. However, most of the current research on task offloading methods assumes that edge servers are homogeneous devices, and does not take into account the differences in user devices in actual application scenarios.

[0125] Figure 1This paper presents the overall architecture of task offloading in an edge computing environment. It includes different subtasks to be processed, different user devices, and heterogeneous edge servers. The relationships between subtasks can be represented using a directed acyclic graph (DAG). The vertices of the DAG represent subtasks, and the directed edges between vertices represent the dependencies between subtasks. In the DAG, subtasks without dependencies are executed in parallel to improve efficiency. Subtasks with dependencies, on the other hand, are executed serially. During task execution, subtasks can be dynamically scheduled based on the DAG topology to adapt to resource changes and task execution status. Therefore, before a user device sends a task request, an analyzer performs a detailed analysis of the task, including assessing resource requirements and resolving dependencies. Then, based on a predetermined offloading strategy, these subtasks are assigned to heterogeneous edge servers, which receive and process the tasks based on their own characteristics and current status. Finally, the execution results are returned to the user device through a feedback mechanism to ensure accurate and timely data transmission.

[0126] In edge computing environments, task offloading refers to the process of migrating computing tasks from user devices to edge servers or edge nodes for execution. Markov decision process is a mathematical framework used to model decision-making problems in uncertain environments. In edge computing, task offloading decisions are viewed as an MDP, represented by a five-tuple To express;

[0127] Where s is the state space, which is used to describe the state of the system environment. According to the current state, a task is selected from all tasks waiting to be assigned. After the task is offloaded to the edge server, the next state is obtained.

[0128] a is the action space, including all actions selected in the current state; after selecting the subtask to be executed, the present invention needs to select an edge server to execute the task. Therefore, the index of each edge server is an action.

[0129] r is the immediate reward, which is used to evaluate the feedback of the action. Because the optimization goal of the present invention comprehensively considers the total completion time, energy consumption and throughput of the task, the immediate reward of the present invention also takes these three factors into consideration.

[0130] p is the state transition probability, which describes the probability of transitioning from one state to another. In task offloading, it involves the probability of task transitioning between different offloading decisions.

[0131] It is a discount factor that measures the importance of immediate rewards and long-term rewards. It belongs to the interval [0,1], where 0 means only considering immediate rewards and 1 means that long-term rewards and immediate rewards are equally important.

[0132] Because the optimization goal of the present invention comprehensively considers the total completion time, energy consumption, and throughput of the task, the instant reward of the present invention also takes these three factors into consideration. The calculation formula of r is as follows:

[0133] ;

[0134] in, is the transmission time of the i-th task of user device k, is the running time of the i-th task of user device k, is the communication energy consumption of the i-th task of user device k, ) is the running energy consumption of the i-th task of user device k.

[0135] The task offloading model includes an action network and a target network;

[0136] The action network and target network have the same structure;

[0137] In the action network, it is used to evaluate the Q value of each action in the current state and select the optimal action. Its parameters are continuously updated through training;

[0138] In the target network, it is used to provide a stable training target. The parameters of the target network are copied from the action network Q network every C steps.

[0139] The target network Q' is a copy of the action network. Its structure is the same as the action network, but the parameters are updated less frequently. The role of the target network is to provide a stable training target to avoid frequent updates of the action network that may lead to unstable training.

[0140] like Figure 3 As shown in the figure, the action network includes several neural network layers, including: input layer, first hidden layer, second hidden layer, asymptotic rectangular window attention mechanism, and output layer; the attention mechanism is integrated into it to enhance the model's ability to capture key information.

[0141] The input layer of the action network receives a feature vector, which is a collection of attributes describing the task, the state of the edge server, and the characteristics of the user device. After the linear transformation and ReLU activation function processing of the first hidden layer, the output is passed to the second hidden layer. After the second hidden layer, the asymptotic rectangular window attention mechanism is introduced. The asymptotic rectangular window attention mechanism divides the output of the second hidden layer into multiple windows and performs a weighted sum of the features within each window to highlight important features. Specifically, it means:

[0142] 1) First, the features within each rectangular window, after undergoing the linear transformation and ReLU activation function of the first hidden layer, are transformed into feature representations. Then, nonlinear processing is performed using the tanh activation function to enhance the expressive power of the features. Next, the dot product similarity between the feature representation and the learnable vector is calculated and normalized using the Softmax function to obtain attention weights within each window. These weights are used to weight the features within the window, thereby highlighting the key features within the window. A learnable vector is a vector parameter that can be automatically learned and adjusted during model training. In the asymptotic rectangular window attention mechanism, this learnable vector performs a dot product similarity calculation with the feature representation within each rectangular window to obtain the attention weights within each window. Specifically, this learnable vector can be viewed as a query vector, representing the key feature directions that the model focuses on during the current task offloading decision. By learning this learnable vector, the model can automatically identify which features are more important in the current decision scenario and assign them higher attention weights. Finally, the weighted features of all windows are concatenated to form the output of the asymptotic rectangular window attention mechanism; this process not only effectively captures local features, but also enables the model to better adapt to the extraction of multi-scale features by gradually processing windows at different positions.

[0143] 2) The output of the asymptotic rectangular window attention mechanism passes through the first hidden layer, the second hidden layer, and the asymptotic rectangular window attention mechanism again to further extract and abstract features; providing richer information representation for subsequent decision-making while further enhancing the network's ability to capture local features.

[0144] The final linear transformation, followed by a ReLU activation function, outputs a predicted Q-value for each action. The predicted Q-value represents the expected reward for each possible action in the current state. The entire network is trained using gradient descent to minimize the difference between the predicted Q-value and the target Q-value, enabling better decision-making. The target Q-value, calculated using the Bellman equation, serves as the target value during DQN training and guides network parameter updates.

[0145] Through this structure, the task offloading model can focus on different parts of the input features through the asymptotic rectangular window attention mechanism while maintaining the good adaptability of the DQN algorithm to dynamic environments, thereby better learning and prediction.

[0146] To optimize the aforementioned TEH metric and achieve energy-efficient and efficient task offloading in edge computing environments, this paper proposes a DPAQN algorithm that combines the asymptotic rectangular window attention mechanism with the DQN. The DPAQN algorithm's fundamental framework is to model the aforementioned problem as a Markov decision process (MDP), forming a DQN-based task offloading algorithm. The DPAQN algorithm is then integrated with the asymptotic rectangular window attention mechanism.

[0147] Task offloading is achieved based on the DPAQN algorithm, including:

[0148] First, initialize the edge computing environment and DQN parameters and state s (such as Figure 2 (center ①);

[0149] Edge computing refers to a distributed computing architecture where computing tasks can be executed on edge devices (such as edge servers) close to the data source. This includes heterogeneous edge servers with limited resources, user devices with diverse requirements, tasks with different attributes, and fluctuating network environments.

[0150] DQN parameters, specifically: number of neural network layers, number of neurons in each layer, weights and biases in the neural network, learning rate, discount factor, experience replay buffer size, target network update frequency, mini-batch size, and exploration rate;

[0151] Status s, specifically including: task characteristics, edge server status, network status, and user device status;

[0152] Using the current state s as input, the action network evaluates the predicted Q-values ​​of all possible actions, i.e. ; and select the action a with the highest Q value to execute (such as Figure 2 (as shown in ②);

[0153] Executing action a interacts with the environment, observing reward r, and obtaining the next state s' and the iteration end flag done. This process involves: first, selecting the optimal action a based on the current state s and the evaluation results of the action network. This action typically involves offloading the task to an edge server. Next, the system executes action a, and the task begins transmission. Meanwhile, reward r is calculated based on the task's completion time, energy consumption, and throughput. As the task executes, the system state changes, including updates to the task queue, changes in edge server load, changes in network status, and resource usage on user devices. These changes constitute the next state s'. Finally, the iteration end flag done is set based on whether the iteration termination condition is met. If the task is completed or the termination condition is met, done = 1; otherwise, done = 0. This process enables the DQN algorithm to dynamically adjust the task offloading strategy through interaction with the environment to optimize the task's comprehensive TEH metric.

[0154] Then, the current state s, action a, reward r, the next state s' and the iteration end flag done are stored in the experience replay buffer (such as Figure 2 (as shown in ③ in the figure) for subsequent learning; a small batch of data is randomly sampled from the experience replay for training;

[0155] The next state s' is used as the input of the target network Q' (e.g. Figure 2 As shown in ④), calculate the future ;include:

[0156] The Q value of the optimal action a' predicted by the target network at the next state s'. This value is calculated by the target network Q' and represents the maximum future reward that can be obtained in state s'. is the maximum value among these Q values, namely: ;

[0157] Combined with the reward r and the iteration end flag done (such as Figure 2 Calculate as shown in ⑤ , as shown below;

[0158] ;

[0159] in, It is the parameter of the target network, and the value of done is 0 or 1; The target Q value calculated by the network. The target Q value combines the immediate reward r and the discounted value of future rewards to calculate the loss function.

[0160] The action network receives the state s and action a in the mini-batch sampling (such as Figure 2 As shown in ⑥), the Q value of the action network (as Figure 2⑦ in the middle) combined with action a in the mini-batch sampling (as shown in Figure 2 (shown in ⑧) to obtain ; Then use the Mean Squared Error (MSE) loss function to calculate and loss (like Figure 2 (as shown in point ⑨ in the middle), as shown below;

[0161] ;

[0162] m represents the number of samples in the mini-batch sampling;

[0163] Update the parameters of the action network through back propagation according to the obtained Loss, and copy the parameters of the action network Q to the target network Q' every C steps (such as Figure 2 (as shown in ⑩ in the middle); to stabilize the training process.

[0164] Determine whether the termination condition is met. If so, the process ends; if not, the steps after returning to the initial state continue.

[0165] In this experiment, the present invention constructs a dynamic simulation environment for edge computing to study the offloading decision of user tasks on heterogeneous edge servers. The dataset used is based on the real measurement data of offloading in image recognition calculations. At the same time, the impact of changes in the network and other system environments in the real environment on the execution and transit time of tasks is considered, and simulation data is generated by adding noise. The environment initializes a certain number of users, and each user is assigned tasks with different attributes. These attributes include the number of instructions of the task, the size of the transmitted data, the expected running time, energy consumption, and CPU resource requirements. Taking into account that in some common scenarios such as industrial automation or enterprises, an edge server is usually responsible for a small number of users, so the present invention tests eight cases with the number of users being 20, 30, 40, 50, 60, 70, 80, and 90. The specific parameters are shown in Table 1.

[0166] Table 1 Experimental configuration parameters;

[0167]

[0168] The performance of the proposed DPAQN algorithm is averaged using the average TEH of the application. It is compared with the DQN, DoubleDQN, DuelingDQN, Prioritized Replay, and PPO algorithms. The performance improvement is calculated using the average value, and the average performance improvement can be defined as:

[0169] ;

[0170] in, and are the average TEH obtained by other algorithms and the average TEH obtained by the DPAQN algorithm, respectively.

[0171] This paper compares the DPAQN algorithm with five algorithms: DQN, Double Q-learing, DuelingDQN, Prioritized RePlay, and Proximal Policy Optimization (PPO), and tests the TEH index values ​​of four edge servers with different configurations scheduling different numbers of user tasks on the same dataset. , the smaller the TEH value, the better the performance. The TEH values ​​of each algorithm in these eight cases on the four different edge servers are as follows Figure 4 shown.

[0172] Simulation results show that, overall, DPAQN outperforms DQN by an average of approximately 24.51096269% in TEH across four edge servers, by approximately 23.87575690% in Double Q-learning, by approximately 23.52633988% in Dueling DQN, by approximately 30.39486988% in Prioritized RePlay, and by approximately 28.82079803% in PPO. This demonstrates that the proposed DPAQN algorithm has advantages in optimizing the overall performance of task offloading in edge computing environments. Specifically, due to the dynamic nature of network status and resource availability in edge computing environments, and the sensitivity of data in edge computing environments, the TEH values ​​of different algorithms vary significantly depending on the number of users. However, across all eight scenarios, DPAQN consistently outperforms the other five algorithms, with the exception of the case with 20 users, where its TEH value is higher than that of PPO. Therefore, the DPAQN algorithm is expected to achieve more efficient and energy-saving offloading and optimize the load balancing of edge servers to a certain extent.

[0173] At the same time, in order to verify the generalization of the results of the present invention, the present invention also conducted comparative experiments under different random seeds, and the results are as follows: Figure 5As shown in the figure, DPAQN outperforms DQN by an average of approximately 28.93454619%, Double Q-learning by an average of approximately 20.71041776%, Dueling DQN by an average of approximately 29.19596098%, Prioritized RePlay by an average of approximately 28.71283886%, and PPO by an average of approximately 29.46298988%. This confirms the good generalization performance of DPAQN to a certain extent. This shows that the DPAQN algorithm proposed in this invention can, to a certain extent, alleviate the problems of weak robustness and poor generalization of artificial intelligence algorithms.

[0174] In the Industrial Internet of Things (IIoT) environment, smart manufacturing systems need to process large amounts of data generated by sensors and devices to enable real-time monitoring of production processes, fault prediction, and quality control. These tasks place high demands on energy consumption and task completion time, while also requiring sufficient throughput to ensure production efficiency. The DPAQN algorithm can optimize task offloading strategies, offloading some compute-intensive tasks to edge servers, potentially reducing device energy consumption and improving task completion speed. For example, in a smart factory, sensors can send data to edge servers for analysis, and the DPAQN algorithm dynamically adjusts task offloading decisions based on current network status and resource availability. This optimization approach is expected to reduce device energy consumption while increasing overall system throughput, contributing to a more efficient and intelligent production process.

[0175] This paper considers the heterogeneity of edge servers and the diversity of users, and proposes the DPAQN algorithm to address the problem of task offloading in edge computing environments. It retains the common advantages of artificial intelligence algorithms while alleviating, to a certain extent, the instability they often exhibit. Experiments were conducted on the model, taking into account three metrics: task completion time, energy consumption, and throughput. Comparisons with five algorithms—DQN, Double DQN, Dueling DQN, Prioritized Replay, and PPO—demonstrated the algorithm's excellent overall performance, outperforming the other five algorithms by an average of 20.71%-30.39%.

[0176] Example 3

[0177] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the task offloading method using deep reinforcement learning and attention mechanism described in Example 1 or 2 are implemented.

[0178] Example 4

[0179] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the task offloading method using deep reinforcement learning and attention mechanism described in Example 1 or 2 are implemented.

[0180] Example 5

[0181] A task offloading system using deep reinforcement learning and attention mechanism, including:

[0182] The data acquisition and preprocessing module is configured to: data acquisition and preprocessing;

[0183] The task offloading module is configured to input the preprocessed data into the trained task offloading model and implement task offloading based on the DPAQN algorithm.

Claims

1. A task offloading method using deep reinforcement learning and attention mechanism, characterized in that: include: Data acquisition and preprocessing; data includes edge server information, user information, and task information; The preprocessed data is input into the trained task offloading model to implement task offloading based on the DPAQN algorithm; Task offloading is achieved based on the DPAQN algorithm, including: First, initialize the edge computing environment and DQN parameters and state s; Using the current state s as input, the action network evaluates the predicted Q values ​​of all possible actions, i.e. Q'act; and selects the action a with the highest Q value to execute; Execute action a to interact with the environment, observe the reward r, and obtain the next state s' and the iteration end flag done; Then, the current state s, action a, reward r, next state s' and iteration end flag done are stored in the experience replay buffer for subsequent learning; a small batch of data is randomly sampled from the experience replay for training; Take the next state s' as the input of the target network Q' and calculate the future Q'max; Combine the reward r and the iteration end flag done to calculate Qtar as shown below; Qtar=r+(1-done)*γ*Q′max; Among them, Qtar is the parameter of the target network, and the value of done is 0 or 1; The action network receives the state s and action a in the mini-batch sampling. The Q value of the action network is combined with the action a in the mini-batch sampling to obtain Qact. Then the loss Loss of Qact and Qtar is calculated using the Mean Squared Error loss function, as shown below. m represents the number of samples in the mini-batch sampling; Update the parameters of the action network through back propagation according to the obtained Loss, and copy the parameters of the action network Q to the target network Q' every C steps; Determine whether the termination condition is met. If so, the process ends; if not, the process continues after returning to the initial state. The action network consists of several neural network layers, including: input layer, first hidden layer, second hidden layer, asymptotic rectangular window attention mechanism, and output layer; The input layer of the action network receives the feature vector, which is processed by the linear transformation and relu activation function of the first hidden layer before being output to the second hidden layer. After the second hidden layer, the asymptotic rectangular window attention mechanism is introduced. The asymptotic rectangular window attention mechanism divides the output of the second hidden layer into multiple windows and performs a weighted summation of the features within each window. Specifically, 1) First, the features within each rectangular window, i.e., the features processed by the linear transformation and ReLU activation function of the first hidden layer, are transformed into feature representations through linear transformation. Then, nonlinear processing is performed using the tanh activation function. Next, the attention weight within each window is obtained by calculating the dot product similarity between the feature representation and the learnable vector and normalizing it with the Softmax function. Finally, the weighted features of all windows are concatenated to form the output of the asymptotic rectangular window attention mechanism. 2) The output of the asymptotic rectangular window attention mechanism passes through the first hidden layer, the second hidden layer, and the asymptotic rectangular window attention mechanism again to further extract and abstract features; The predicted Q value of each action is output through the last linear transformation plus the relu activation function. The predicted Q value represents the expected return of taking each possible action in the current state.

2. The task offloading method using deep reinforcement learning and attention mechanism according to claim 1, characterized in that: Preprocessing, including: noise addition; dynamic environment simulation: by continuously generating new tasks, simulating a dynamic edge computing environment to ensure that the task flow is continuous; parameter setting.

3. The task offloading method using deep reinforcement learning and attention mechanism according to claim 1, characterized in that: The objective function min TEH in the task offloading model is expressed as: min T EH=w1*||TT||+w2*||TE||-w3*||TH||; in, Represents the maximum completion time of all completed tasks, FinishTaskN represents the number of completed tasks; TT represents the total completion time of the task; represents the maximum energy consumption of all completed tasks, and TE represents the total energy consumption of the task; w1, w2, w3 are weights, and TH refers to the total throughput; The values ​​of w1, w2, and w3 are 0.25, 0.5, and 0.25 respectively.

4. The task offloading method using deep reinforcement learning and attention mechanism according to claim 1, characterized in that: The total task completion time TT includes the transmission time and running time of all tasks; it is specifically defined as: Where n is the number of users, l is the number of subtasks for each user, represents the data volume of the i-th task of user device k, represents the bandwidth allocated to the i-th task of user device k, is the transmission time of the i-th task of user device k, represents the time required for the local execution of the i-th task of user device k, RHO represents the ratio of local and remote execution rates, is the running time of the i-th task of user device k; The total energy consumption of a task includes the communication energy consumption and operation energy consumption of all tasks; it is specifically defined as: in, is the transmission power of the ith task of user equipment k, represents the communication time of the i-th task of user device k; represents the communication energy consumption of the i-th task of user equipment k, that is, represents the running energy consumption of the i-th task of user device k.

5. The task offloading method using deep reinforcement learning and attention mechanism according to claim 1, characterized in that: In the edge computing environment, task offloading refers to the process of migrating computing tasks from user devices to edge servers or edge nodes for execution. The task offloading decision is regarded as an MDP, represented by a five-tuple<s,a,r,p,γ> To express; Where s is the state space, which is used to describe the state of the system environment. According to the current state, a task is selected from all tasks waiting to be assigned. After the task is offloaded to the edge server, the next state is obtained. a is the action space, including all actions selected in the current state; r is the immediate reward, which is used to evaluate the feedback of the action; p is the state transition probability, which describes the probability of transitioning from one state to another; γ is a discount factor that measures the importance of immediate rewards and long-term rewards. It belongs to the interval [0,1], where 0 means only considering immediate rewards and 1 means that long-term rewards and immediate rewards are equally important.

6. The task offloading method using deep reinforcement learning and attention mechanism according to claim 5, characterized in that: The calculation formula for r is as follows: in, is the transmission time of the i-th task of user device k, is the running time of the i-th task of user device k, is the communication energy consumption of the i-th task of user device k, is the running energy consumption of the i-th task of user device k.

7. The task offloading method using deep reinforcement learning and attention mechanism according to claim 1, characterized in that: The task offloading model includes an action network and a target network; The action network and target network have the same structure; In the action network, it is used to evaluate the Q value of each action in the current state and select the optimal action. Its parameters are continuously updated through training; In the target network, it is used to provide a stable training target. The parameters of the target network are copied from the action network Q network every C steps.

8. A task offloading system using deep reinforcement learning and attention mechanism, characterized by: include: The data acquisition and preprocessing module is configured to: data acquisition and preprocessing; The task offloading module is configured as; The preprocessed data is input into the trained task offloading model to implement task offloading based on the DPAQN algorithm; Task offloading is achieved based on the DPAQN algorithm, including: First, initialize the edge computing environment and DQN parameters and state s; Using the current state s as input, the action network evaluates the predicted Q values ​​of all possible actions, i.e. Q'act; and selects the action a with the highest Q value to execute; Execute action a to interact with the environment, observe the reward r, and obtain the next state s' and the iteration end flag done; Then, the current state s, action a, reward r, next state s' and iteration end flag done are stored in the experience replay buffer for subsequent learning; a small batch of data is randomly sampled from the experience replay for training; Take the next state s' as the input of the target network Q' and calculate the future Q'max; Combine the reward r and the iteration end flag done to calculate Qtar as shown below; Qtar=r+(1-done)*γ*Q′max; Among them, Qtar is the parameter of the target network, and the value of done is 0 or 1; The action network receives the state s and action a in the mini-batch sampling. The Q value of the action network is combined with the action a in the mini-batch sampling to obtain Qact. Then the loss Loss of Qact and Qtar is calculated using the Mean Squared Error loss function, as shown below. m represents the number of samples in the mini-batch sampling; Update the parameters of the action network through back propagation according to the obtained Loss, and copy the parameters of the action network Q to the target network Q' every C steps; Determine whether the termination condition is met. If so, the process ends; if not, the process continues after returning to the initial state. The action network consists of several neural network layers, including: input layer, first hidden layer, second hidden layer, asymptotic rectangular window attention mechanism, and output layer; The input layer of the action network receives the feature vector, which is processed by the linear transformation and relu activation function of the first hidden layer before being output to the second hidden layer. After the second hidden layer, the asymptotic rectangular window attention mechanism is introduced. The asymptotic rectangular window attention mechanism divides the output of the second hidden layer into multiple windows and performs a weighted summation of the features within each window. Specifically, 1) First, the features within each rectangular window, i.e., the features processed by the linear transformation and ReLU activation function of the first hidden layer, are transformed into feature representations through linear transformation. Then, nonlinear processing is performed using the tanh activation function. Next, the attention weight within each window is obtained by calculating the dot product similarity between the feature representation and the learnable vector and normalizing it with the Softmax function. Finally, the weighted features of all windows are concatenated to form the output of the asymptotic rectangular window attention mechanism. 2) The output of the asymptotic rectangular window attention mechanism passes through the first hidden layer, the second hidden layer, and the asymptotic rectangular window attention mechanism again to further extract and abstract features; The predicted Q value of each action is output through the last linear transformation plus the relu activation function. The predicted Q value represents the expected return of taking each possible action in the current state.

Images