A method and system for DAG dependent task edge offloading based on predictive dynamic caching and TD3 reinforcement learning

Abstract

This invention discloses a method for edge offloading of DAG-dependent tasks based on predictive dynamic caching and TD3 reinforcement learning. The edge server uses LSTM to learn the historical task request sequence and predicts the probability distribution of the task request in the (t+1)th time slot. Based on the prediction results and 0-1 knapsack dynamic programming, predictive dynamic caching is implemented. The normalized DAG task state, cache hit state, edge computing resources, cache resources, mobile user location, and channel state are input into the TD3 reinforcement learning model, and the optimal continuous offloading ratio is output. The local execution latency, uplink transmission latency, edge execution latency, and corresponding energy consumption are calculated according to the offloading ratio. Based on the predecessor and successor dependencies of the DAG, the earliest start time and earliest completion time of each subtask are calculated. Under the premise of satisfying the maximum latency constraint of all tasks, the average latency and total energy consumption of the edge computing system are minimized.

Description

Technical Field

[0001] This invention belongs to the field of reinforcement learning and wireless communication network technology, and more specifically, relates to a method and system for edge offloading of DAG dependent tasks based on predictive dynamic caching and TD3 reinforcement learning. Background Technology

[0002] With the rapid development of IoT technology, 6G mobile communication, and immersive interactive applications, a large number of computationally intensive and latency-sensitive applications such as VR / AR, autonomous driving, real-time video analytics, and the Industrial Internet have emerged. Mobile terminal devices have significant bottlenecks in computing power, storage capacity, and battery life, making it difficult to independently complete complex computing tasks. Mobile edge computing (MEC) provides low-latency, high-bandwidth computing services to end users by deploying computing and caching resources at the network edge. However, many complex tasks in actual business are broken down into subtasks of directed acyclic graphs (DAGs) with predecessor-successor dependencies. How to implement efficient edge offloading and scheduling of DAG tasks has become a core research direction for MEC system optimization.

[0003] Existing edge offloading optimization methods for DAG tasks mainly include numerical programming heuristic optimization methods, basic deep reinforcement learning scheduling methods, and cache offloading discrete cooperative scheduling methods. Numerical programming heuristic optimization methods construct mathematical models with latency and energy consumption constraints, and rely on integer programming and greedy strategies to solve for subtask offloading positions and cache allocation schemes. They often use binary full offloading and static cache deployment modes to complete resource scheduling. Basic deep reinforcement learning scheduling methods mainly use algorithms such as Deep Deterministic Policy Gradient (DDPG) to transform offloading and cache actions into Markov decision spaces, and rely on agents to interact with the dynamic environment to iteratively update scheduling strategies. Cache offloading discrete cooperative scheduling methods separate cache replacement and task offloading into two major optimization stages, solve decision variables independently for each stage, and then combine them to achieve separate management of edge storage and computing resources.

[0004] However, the three existing edge offloading optimization methods for DAG tasks mentioned above generally have some non-negligible drawbacks:

[0005] (1) Existing numerical programming heuristic optimization methods generally adopt coarse-grained binary offloading, which cannot perform fine-grained partial offloading of subtasks, resulting in low resource utilization of terminals and edge nodes; in addition, the lack of future business traffic prediction capability of the supporting static cache leads to insufficient cache hit rate and large additional transmission delay overhead.

[0006] (2) Existing basic reinforcement learning methods have problems such as Q-value overestimation, training oscillation, and slow convergence speed. When faced with the high-dimensional optimization space composed of DAG dependency constraints, multi-user competition, and dynamic cache coupling, the optimization stability and scheduling accuracy are greatly reduced.

[0007] (3) Existing cache unloading discrete collaborative scheduling methods cannot form a closed-loop collaborative optimization because they sever the decision-making relationship between cache replacement and task unloading. They also fail to fully exploit the parallel execution characteristics of DAG subtasks for joint resource allocation, resulting in limited global optimality and scheduling execution efficiency. Summary of the Invention

[0008] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method and system for DAG-dependent task edge offloading based on predictive dynamic caching and TD3 reinforcement learning. Its purpose is to solve the following problems: Firstly, existing numerical programming heuristic optimization methods generally employ coarse-grained binary offloading, failing to perform fine-grained partial offloading of subtasks, resulting in low resource utilization at terminals and edge nodes. Secondly, the lack of future service traffic prediction capabilities in the accompanying static cache leads to insufficient cache hit rate and high additional transmission latency overhead. Thirdly, existing basic reinforcement learning methods experience a significant decrease in optimization stability and scheduling accuracy when facing high-dimensional optimization spaces composed of DAG dependency constraints, multi-user competition, and dynamic cache coupling. Fourthly, existing discrete-time collaborative scheduling methods for cache offloading, by severing the decision-making relationship between cache replacement and task offloading, fail to form a closed-loop collaborative optimization and do not fully exploit the parallel execution characteristics of DAG subtasks for joint resource allocation, resulting in limited global optimality and scheduling efficiency.

[0009] To achieve the above objectives, according to one aspect of the present invention, a method for edge offloading of DAG-dependent tasks based on predictive dynamic caching and TD3 reinforcement learning is provided. This method is applied to an edge computing system comprising multiple edge servers and multiple mobile users, and includes the following steps:

[0010] (1) The nth edge server obtains the total number T of time slots in the edge computing system and sets the time slot number t=1;

[0011] (2) The nth edge server determines whether t is greater than the total number of time slots T in the edge computing system obtained in step (1). If it is, proceed to step (8); otherwise, proceed to step (3), where n∈[1, N] and N represents the total number of edge servers in the edge computing system.

[0012] (3) The nth edge server obtains from the mth mobile user the DAG graph of the mobile user in the tth time slot, the parameters of the i-th subtask corresponding to the DAG graph, the dependencies between subtasks, and the position coordinates of the m-th mobile user to which the i-th subtask belongs in the time slot t, including the x-axis coordinates. and y-axis coordinates Obtain the all-local processing latency and all-local processing energy consumption of the i-th subtask corresponding to the DAG graph, and obtain the remaining computing resources of the n-th edge server in the (t-1)-th time slot. Remaining cache resources The sequence of historical task request counts, p(t-L+1), p(t-L+2), ..., p(t), consists of node load rate and the number of historical task requests over L consecutive time slots; where m∈[1,M], M represents the total number of mobile users in the edge computing system, i∈[1,K], K is the number of all subtasks corresponding to the DAG graph of the m-th mobile user in the t-th time slot, and the parameters of the i-th subtask include the amount of input data for the i-th subtask. The total number of CPU cycles required to complete the i-th subtask And the maximum tolerable delay of the i-th subtask. The value of L ranges from 3 to 20;

[0013] (4) The nth edge server obtains the edge computing system utility based on the all-local processing latency and all-local processing energy consumption of the i-th subtask of the m-th mobile user obtained in step (3). ;

[0014] (5) The nth edge server will use the edge computing system utility obtained in step (4) Feedback will be sent to all mobile users.

[0015] Preferably, the DAG-dependent task edge unloading method further includes the following steps after step (3) and before step (4):

[0016] (A) The nth edge server normalizes the input data volume of the i-th subtask corresponding to the DAG graph obtained in step (3), the total number of CPU cycles required to complete the i-th subtask, the location coordinates of the mobile user to which the i-th subtask belongs in time slot t, the remaining computing resources, remaining cache resources of the nth edge server in time slot t-1, and the historical task request sequence of the preceding L consecutive time slots, respectively, so as to obtain the normalized input data volume of the i-th subtask, the normalized total number of CPU cycles required, the normalized location coordinates, the normalized remaining computing resources, the normalized remaining cache resources of the nth edge server in time slot t-1, and the normalized historical task request sequence of the preceding L consecutive time slots;

[0017] (B) The nth edge server inputs the historical task request sequence of the nth edge server in the current time slot t (normalized in step (A)) into the pre-trained LSTM prediction model to obtain the task request probability of the (t+1)th time slot. Based on this request probability, and through the 0-1 knapsack optimization algorithm, the optimal caching decision for the (t+1)th time slot is obtained, and the nth edge server... The task request probability of each time slot, the normalized input data volume and total CPU cycles required for the i-th subtask, the normalized location coordinates, and the remaining computing resources and normalized cache resources of the n-th edge server in the previous time slot t-1 are input into a pre-trained TD3 reinforcement learning model to obtain the optimal offloading ratio of the m-th user for the i-th subtask. ;

[0018] (C) The nth edge server obtains the optimal cache content in the t-th time slot from step (B) and the optimal offloading ratio of the ith subtask of the m-th mobile user. Get the local processing latency, local processing energy consumption, task uplink transmission latency, task uplink transmission energy consumption, edge processing latency, edge processing energy consumption, total execution latency, and earliest completion time for the m-th user on the i-th subtask.

[0019] (D) Based on the local processing energy consumption, uplink transmission energy consumption, edge processing energy consumption, and earliest completion time of the m-th mobile user for the i-th subtask obtained in step (C), the n-th edge server obtains the total response latency and total energy consumption of the edge computing system in the t-th time slot, sets t=t+1, and returns to step (2).

[0020] Fully local processing latency of the i-th subtask corresponding to the DAG graph of the m-th mobile user in the t-th time slot. equal:

[0021] ;

[0022] in The frequency of the local CPU of the device of the m-th mobile user;

[0023] Fully local processing energy consumption of the i-th subtask corresponding to the DAG graph of the m-th mobile user in the t-th time slot equal:

[0024] ;

[0025] in Let be the effective switching capacitor coefficient of the local CPU of the m-th mobile user's device.

[0026] Preferably, the input data volume of the i-th subtask corresponding to the subtask parameters of the DAG graph obtained in step (3) is... With the required total number of CPU cycles Normalization is performed to obtain the normalized input data volume for the i-th subtask. With the required total number of CPU cycles The formula is as follows:

[0027] ;

[0028] ;

[0029] in This indicates the maximum amount of input data allowed for a task in an edge computing system. This represents the maximum total number of CPU cycles required to complete a single subtask.

[0030] In step (A), the position coordinates of the m-th user in time slot t obtained in step (3) are normalized to obtain the normalized position coordinates. The following formula is used:

[0031] , ;

[0032] in , These represent the maximum x-axis coordinate and the maximum y-axis coordinate of the coverage area of ​​the edge computing system, respectively.

[0033] In step (A), the remaining computing resources and remaining cache resources of the N edge servers in the (t-1)th time slot are normalized to obtain the normalized remaining computing resources of the nth edge server in the (t-1)th time slot. and normalized remaining cache resources The following formula is used:

[0034] ;

[0035] ;

[0036] in , These represent the maximum computing resources and maximum cache resources of the nth edge server, respectively.

[0037] Preferably, the process in step (B) of obtaining the optimal caching decision for the (t+1)th time slot based on the task request probability of the (t+1)th time slot and solving it using the 0-1 knapsack optimization algorithm is as follows:

[0038] (A1) Based on the request probability distribution of the nth edge server in the (t+1)th time slot of the (nth)th subtask output by the LSTM prediction model, obtain the cache value of the i-th subtask. :

[0039] , ;

[0040] in, Let represent the request probability distribution of the i-th subtask in the (t+1)-th time slot of the n-th edge server. This represents the average execution latency saved after the i-th subtask's cache is hit. This represents the cache priority coefficient of the i-th subtask; As an indicator function, when all predecessor tasks of the DAG graph obtained by the i-th subtask in step (3) are cached or prioritized for scheduling, ;otherwise ;

[0041] (A2) The cache value of the i-th subtask obtained from step (A1). And the optimal cache decision for the (t+1)th time slot is obtained through a 0-1 knapsack dynamic programming algorithm, which is equal to maximizing the total cache value within a finite space:

[0042] ;

[0043] ;

[0044] in Indicates whether to cache the i-th subtask. This represents the storage space required to cache the i-th subtask. This represents the remaining cache resources of the nth edge server in the tth time slot.

[0045] Preferably, the LSTM time series prediction model is trained through the following steps:

[0046] (B1) Collect the historical task request sequence of the nth edge server for L consecutive time slots under different time periods and different load conditions, extract features from the collected historical task request sequence, obtain the correlation features between task periodicity, burstiness and DAG subtasks, construct a large-scale time series dataset based on the obtained feature samples, and randomly divide the large-scale time series dataset into training set and test set in a 9:1 ratio.

[0047] (B2) Normalize each sample in the training set obtained in step (B1) to obtain a normalized training set;

[0048] (B3) Initialize the hyperparameters of the LSTM model to obtain the initialized LSTM model;

[0049] (B4) For each sample in the training set obtained in step (B2), input the sample into the LSTM model initialized in step (B3), obtain the predicted task request probability distribution of the sample in the (t+1)th time slot through forward propagation, calculate the cross-entropy loss between the predicted request probability distribution and the true request probability label of the sample, update the network weights of the LSTM model based on the cross-entropy loss and through the backpropagation algorithm to obtain the updated LSTM model;

[0050] (B5) Repeat step (B4) above until the LSTM model converges, thus obtaining the trained LSTM model.

[0051] Preferably, the TD3 reinforcement learning model is trained through the following steps:

[0052] (C1) Construct a large-scale edge computing simulation environment based on the physical operation rules of the edge computing system, obtain the simulation configuration of the large-scale edge computing simulation environment, obtain the simulation environment after the simulation configuration is completed, and conduct multiple rounds of simulation simulation based on the simulation environment after the simulation configuration is completed, so as to obtain multiple full-scenario running samples as a verification dataset.

[0053] (C2) The hyperparameters and network structure of the TD3 reinforcement learning model are randomly initialized to obtain the initialized TD3 reinforcement learning model; the hyperparameters include the experience replay pool, learning rate, discount factor, soft update coefficient, exploration noise, preset batch, and network update interval steps; the network structure includes Actor network, dual Critic network, target Actor network, and target dual Critic network.

[0054] (C3) Based on the simulation environment after the simulation configuration obtained in step (C1), perform single-slot environment interaction and experience sample collection on the edge computing system to obtain the state of time slot t. Execution actions of time slot t Instant rewards for t-slots The state of time slot t+1 The quadruple formed And store the quadruple into a pre-established experience replay pool;

[0055] The TD3 reinforcement learning model uses a reward function r, and has the following:

[0056] ;

[0057] in Indicates normalization, These are the weighting coefficients. This represents the total response delay of the edge computing system in the t-th time slot. This represents the total energy consumption of the edge computing system in the t-th time slot. This represents the cache hit rate of the edge computing system in the t-th time slot. This represents the number of timed-out tasks in the edge computing system during the t-th time slot;

[0058] (C4) Randomly obtain a small batch of data from the experience replay pool obtained in step (C3), and use the small batch of data to update the dual Critic network in the TD3 reinforcement learning model to obtain the weight parameters of the two sets of main Critic networks after the update.

[0059] (C5) Repeat the above steps (C4) until the TD3 reinforcement learning model converges, thus obtaining the trained TD3 reinforcement learning model.

[0060] Preferably, step (C4) includes the following sub-steps:

[0061] (C4-1) Randomly select multiple time slots t from the experience replay pool obtained in step (C3). Execution actions of multiple time slots. Rewards for multiple t-slots and the status of multiple t+1 time slots. ;

[0062] (C4-2) Set the iteration counter b=0;

[0063] (C4-3) The states of the multiple t+1 time slots obtained in step (C4-1) Input the target Actor network of the initialized TD3 reinforcement learning model obtained in step (C2) to obtain candidate actions for time slot t+1. The state of time slot t+1 Candidate actions for slot t+1 The initialized TD3 reinforcement learning model, obtained through the common input step (C2), uses two sets of target Critic networks to obtain two sets of target Q-values. The smaller of the two target Q-values ​​is selected as the baseline target Q-value output, and the label Q-value is obtained by combining it with the single-step reward. The state of time slot t is then used to calculate the target Q-value. Execution action with time slot t The two sets of main Critic networks of the initialized TD3 reinforcement learning model obtained in step (C2) are used to obtain the predicted Q values ​​of the two sets of main Critic networks of the TD3 reinforcement learning model. The mean square error between the predicted Q value and the label Q value of each set is used as the loss function of the corresponding main Critic network, and the gradient descent method is used to perform backpropagation iteration on the main Critic network to obtain the updated main Critic network parameters.

[0064] (C4-4) Set the iteration counter b = b + 1;

[0065] (C4-5) Determine if b is an integer multiple of the network update interval step number obtained in step (C2). If it is, proceed to step (C4-6); otherwise, return to step (C4-3).

[0066] (C4-6) Take the average of the predicted Q values ​​of the two sets of main Critic networks obtained in step (C4-3), and obtain the Actor loss based on the average value as the parameters of the main Actor network after the back gradient update.

[0067] (C4-7) Perform target network soft update processing on the updated main Critic network parameters obtained in step (C4-3) and the main Actor network parameters updated by the reverse gradient obtained in step (C4-6) respectively, so as to obtain the updated target Actor network parameters and target Critic network parameters respectively.

[0068] The target network soft update is performed using the following formula:

[0069] ;

[0070] in This is the soft update coefficient, with a value ranging from 0.001 to 0.01. This represents the updated network parameters of the main Critic network and the main Actor network. This represents the network parameters of the target Critic network and the target Actor network before the update.

[0071] Preferably, the edge processing delay for the m-th mobile user on the i-th subtask Energy consumption of edge processing The calculation formulas are as follows:

[0072] ;

[0073] ;

[0074] in The CPU frequency allocated to the i-th subtask by the n-th edge server in time slot t is determined by the edge resource scheduling algorithm, satisfying the following conditions: ; The effective switched capacitor coefficient of the nth edge server CPU;

[0075] Uplink transmission delay of the m-th mobile user to the i-th subtask Uplink power consumption of the task They are respectively:

[0076] ;

[0077] ;

[0078] in Let represent the wireless transmission rate between the m-th mobile user in time slot t and the edge server, and we have:

[0079] ;

[0080] in For the channel bandwidth of the edge computing system, Let m be the transmit power of the m-th mobile user. This represents the power of additive white Gaussian noise.

[0081] Let be the channel gain between the m-th mobile user and the n-th edge server in time slot t, and equal to:

[0082] ;

[0083] in The reference channel gain is given when the distance between the mobile user and the edge server is 1 meter. Let be the Euclidean distance between the m-th mobile user and the n-th edge server. This is the path loss exponent, with a value between 2 and 4.

[0084] Local processing latency of the m-th mobile user for the i-th subtask and local processing energy consumption They are respectively:

[0085] ;

[0086] ;

[0087] Earliest start time of the i-th subtask for the m-th mobile user Let be the maximum value of the completion times of all predecessor tasks of the i-th subtask, and equal to:

[0088] ;

[0089] in Indicates cross-transmission delay;

[0090] The earliest completion time of the i-th subtask for the m-th mobile user is calculated using the following formula:

[0091] ;

[0092] in Let be the total execution latency of the i-th subtask for the m-th mobile user. Its value is the maximum time required for parallel execution both locally and at the edge, and is equal to:

[0093] ;

[0094] Preferably, the total response delay of the edge computing system in the t-th time slot is... equal:

[0095] ;

[0096] Total energy consumption of the edge computing system in the t-th time slot It equals the sum of local processing energy consumption, task uplink transmission energy consumption, and edge processing energy consumption:

[0097] ;

[0098] Step (4) includes the following sub-steps:

[0099] (4-1) The all-local processing delay of the i-th subtask of the m-th mobile user obtained in step (3) And all-local processing energy consumption Obtain the total latency for all K subtasks of all M users across all T time slots to be processed entirely locally. Total energy consumption :

[0100] ;

[0101] ;

[0102] (4-2) The total response delay of the edge computing system in the t-th time slot obtained from step (D) With total energy consumption Obtain the optimized total latency of the edge computing system. Total energy consumption :

[0103] ;

[0104] ;

[0105] (4-3) The total latency for all tasks to be processed entirely locally, as obtained from step (4-1). Total energy consumption and the optimized total latency of the edge computing system obtained in step (4-2). Total energy consumption To obtain the utility of edge computing systems :

[0106] ;

[0107] in , These are the time delay weighting coefficient and the energy consumption weighting coefficient, respectively, with values ​​ranging from [0,1], and satisfying the following conditions: .

[0108] According to another aspect of the present invention, a DAG-dependent task edge offloading system based on predictive dynamic caching and TD3 reinforcement learning is provided. This system is applied to an edge computing system comprising multiple edge servers and multiple mobile users, and includes the following modules:

[0109] The first module, located on the nth edge server, is used to obtain the total number T of time slots in the edge computing system and set the time slot number t=1.

[0110] The second module, which is set on the nth edge server, is used to determine whether t is greater than the total number of time slots T in the edge computing system obtained by the first module. If it is, it enters the eighth module; otherwise, it enters the third module. Here, n∈[1,N], and N represents the total number of edge servers in the edge computing system.

[0111] The third module, located on the edge server, is used to obtain the DAG graph of the m-th mobile user in the t-th time slot, the parameters of the i-th subtask corresponding to the DAG graph, the dependencies between subtasks, and the position coordinates of the m-th mobile user to which the i-th subtask belongs in time slot t, including the x-axis coordinates. and y-axis coordinates Obtain the all-local processing latency and all-local processing energy consumption of the i-th subtask corresponding to the DAG graph, and obtain the remaining computing resources of the n-th edge server in the (t-1)-th time slot. Remaining cache resources The sequence of historical task request counts, p(t-L+1), p(t-L+2), ..., p(t), consists of node load rate and the number of historical task requests over L consecutive time slots; where m∈[1,M], M represents the total number of mobile users in the edge computing system, i∈[1,K], K is the number of all subtasks corresponding to the DAG graph of the m-th mobile user in the t-th time slot, and the parameters of the i-th subtask include the amount of input data for the i-th subtask. The total number of CPU cycles required to complete the i-th subtask And the maximum tolerable delay of the i-th subtask. The value of L ranges from 3 to 20;

[0112] The fourth module, located on the nth edge server, is used to obtain the edge computing system utility based on the all-local processing latency and all-local processing energy consumption of the i-th subtask of the m-th mobile user obtained from the third module. ;

[0113] The fifth module, located on the nth edge server, is used to transfer the edge computing system utility obtained from the fourth module. Feedback will be sent to all mobile users.

[0114] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0115] (1) Since the present invention adopts steps (1) to (B), it introduces LSTM timing prediction to realize dynamic pre-caching, which can predict the distribution of future task requests and effectively improve the cache hit rate. Therefore, it can solve the technical problem of insufficient cache hit rate and large additional transmission delay overhead caused by the lack of future business traffic prediction capability of the matching static cache in the existing numerical programming heuristic optimization method. At the same time, since the present invention supports fine-grained partial unloading of DAG subtasks, it avoids the waste of resources of binary unloading and improves the overall resource utilization rate. Therefore, it can solve the technical problem of low resource utilization of terminals and edge nodes caused by the general use of coarse-grained binary unloading in the existing numerical programming heuristic optimization method, which cannot perform fine-grained partial unloading of subtasks.

[0116] (2) Since the present invention adopts steps (B) to (D), it sets up two independent value networks to fit the state action value through the TD3 dual-Critic-dual-valuation network architecture, and selects the network output with the smaller valuation as the target Q value, thereby actively suppressing the positive overestimation bias caused by the single network from the model structure level; at the same time, it weakens the training oscillation amplitude in the high-dimensional coupling environment by delaying the update strategy network and smoothing the target value gradient in small batches. Therefore, it can solve the technical problem that the optimization stability and scheduling accuracy of existing basic reinforcement learning methods are greatly reduced when facing the high-dimensional optimization space composed of DAG dependency constraints, multi-user competition, and dynamic cache coupling.

[0117] (3) Since the present invention adopts steps (A) to (C), it constructs priority parallel scheduling logic based on the predecessor and successor dependency relationship of DAG subtasks. Without destroying the task execution timing constraints, it explores the parallel execution capability between subtasks without dependency relationship, and inputs the cache replacement action and the subtask unloading ratio action into the TD3 reinforcement learning decision space to achieve joint solution. Therefore, it can solve the technical problem that the existing cache unloading discrete cooperative scheduling method cannot form a closed loop cooperative optimization because it severs the decision relationship between cache replacement and task unloading, and does not fully explore the parallel execution characteristics of DAG subtasks to perform joint resource allocation, resulting in limited global optimality and scheduling execution efficiency.

[0118] (4) The present invention constructs a cache-unload closed-loop joint optimization framework, and the reward function integrates multiple objective indicators such as latency, energy consumption, and cache hit rate to achieve global collaborative optimization and improve the overall effectiveness of edge computing system. Attached Figure Description

[0119] Figure 1 This is a flowchart of the DAG dependency task edge unloading method based on predictive dynamic caching and TD3 reinforcement learning, which is the basis of this invention. Detailed Implementation

[0120] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0121] The main technical concept of this invention is that, in a mobile edge computing system with multiple mobile users and multiple edge nodes, user tasks are modeled in the form of a Directed Acyclic Graph (DAG). It is assumed that each subtask is a data-parallel task (such as large matrix block computation), and its input data and computational load can be scaled arbitrarily. The system is divided into independent local and edge subsets. Edge servers utilize Long Short-Term Memory (LSTM) networks to learn from historical task request sequences, predicting the probability distribution of task requests in the (t+1)th time slot. Based on the prediction results and 0-1 knapsack dynamic programming, predictive dynamic caching is implemented. Normalized DAG task states, cache hit states, edge computing resources, cache resources, mobile user locations, and channel states are uniformly input into the gradient of a TwinDelayed Deep Deterministic Policy (TDP) gradient. The Gradient (TD3) reinforcement learning model outputs the optimal continuous offloading ratio for each subtask. Based on the offloading ratio, it calculates the local execution latency, uplink transmission latency, edge execution latency, and corresponding energy consumption. It calculates the earliest start time and earliest completion time for each subtask based on the DAG predecessor-successor dependency relationship to ensure the legality of the task execution order. After each time slot, the edge computing system latency, energy consumption, cache hit rate, and number of task timeouts are used as reward signals to update the LSTM prediction model and the TD3 policy model, forming a closed-loop iterative mechanism of "prediction-caching-offloading-feedback-optimization". Finally, under the premise of satisfying the maximum latency constraints of all tasks, it minimizes the average latency and total energy consumption of the edge computing system, and maximizes the cache hit rate, task completion rate, and overall utility of the edge computing system.

[0122] like Figure 1 As shown, this invention discloses a DAG-dependent task edge offloading method based on predictive dynamic caching and Twin Delayed Deep Deterministic Policy Gradient (TD3) reinforcement learning. It is applied to edge computing systems containing multiple edge servers and multiple mobile users, and includes the following steps:

[0123] (1) The nth edge server obtains the total number T of time slots in the edge computing system and sets the time slot number t=1.

[0124] This step discretizes the time dimension of the edge computing system, dividing the continuously running edge computing system into a fixed-length time slot sequence to achieve step-by-step decision-making, step-by-step execution, and step-by-step updates. Each time slot corresponds to a complete process of state acquisition, cache update, unloading decision, task execution, and model optimization. The time slot length can be flexibly configured according to the business scenario. In ultra-high reliability and low latency scenarios such as VR / AR and autonomous driving, it can be set to milliseconds, while in scenarios such as video analysis and data statistics, it can be appropriately relaxed.

[0125] The advantage of this step is that by discretizing the time slots, the difficulty of modeling dynamic environments can be effectively reduced, ensuring that the training and inference process of the reinforcement learning model is stable and reproducible. At the same time, it enables the edge computing system to quickly respond to changes in mobile user tasks, edge load, and channel quality in each time slot, meeting the real-time requirements of low-latency services.

[0126] (2) The nth edge server determines whether t is greater than the total number of time slots T in the edge computing system obtained in step (1). If it is, proceed to step (8); otherwise, proceed to step (3), where n∈[1, N] and N represents the total number of edge servers in the edge computing system.

[0127] This step controls the overall operating cycle of the edge computing system. When the current time slot number exceeds the preset total number of time slots, it indicates that all tasks have completed their execution process, and the edge computing system enters the result statistics and utility calculation stage. If there are still time slots that have not been executed, the system continues to the status acquisition and decision-making process of the next time slot. The total number of time slots T is set according to the task scale, the number of mobile users, the simulation scenario, or the actual deployment requirements. In a real edge computing system, it can run continuously in an infinite loop mode. For ease of explanation and model training, this invention uses a fixed number of time slots. Through the time slot judgment mechanism, the edge computing system can achieve automated periodic operation and periodic optimization, completing continuous iteration without manual intervention.

[0128] (3) The nth edge server obtains the DAG graph of the mth mobile user in the tth time slot and the parameters of the i-th subtask corresponding to the DAG graph (including the input data of the i-th subtask) from the m-th mobile user. The total number of CPU cycles required to complete the i-th subtask And the maximum tolerable delay of the i-th subtask. The dependencies between subtasks, and the location coordinates (including x-axis coordinates) of the m-th mobile user to which the i-th subtask belongs in time slot t. and y-axis coordinates Get the all-local processing latency and all-local processing energy consumption of the i-th subtask corresponding to the DAG graph, and get the remaining computing resources of the n-th edge server in the (t-1)-th time slot. Remaining cache resources The sequence of historical task request counts is p(t-L+1), p(t-L+2), ..., p(t), consisting of node load rate and the number of historical task requests over L consecutive time slots; where m∈[1,M], M represents the total number of mobile users in the edge computing system, i∈[1,K], K is the number of all subtasks corresponding to the DAG graph of the m-th mobile user in the t-th time slot, and L ranges from 3 to 20, preferably 8.

[0129] Historical request count sequences are used for subsequent time-series prediction and caching decisions. These historical request sequences are the core basis for predicting future task popularity, reflecting the periodicity, suddenness, and regularity of tasks.

[0130] In a practical edge computing system, mobile user devices report location information and task parameters in real time through the uplink control channel. The edge server completes information parsing and storage through the baseband processing unit and the core network interface. The DAG dependency relationship is constructed and uploaded by the mobile user application when the task is generated. It includes a set of predecessor subtasks prec(m,i) and a set of successor subtasks suc(m,i), which are used to constrain the execution order of the subtasks.

[0131] The fully local processing latency of the i-th subtask corresponding to the DAG graph of the m-th mobile user in the t-th time slot is used to characterize the time required for the i-th subtask to be executed without any offloading, relying solely on the local CPU of the mobile user equipment. Its calculation formula is as follows:

[0132] ;

[0133] in The frequency of the local CPU of the m-th mobile user's device, in GHz, is determined by the hardware configuration and is collected during the initialization of the edge computing system. Due to limited local CPU resources, the local execution latency of computationally intensive tasks is typically high, failing to meet latency constraints; therefore, edge offloading is necessary.

[0134] The energy consumption of the fully local processing of the i-th subtask corresponding to the DAG graph of the m-th mobile user in the t-th time slot is used to characterize the device power consumed by the i-th subtask when it is executed entirely locally. Its calculation formula is as follows:

[0135] ;

[0136] in The effective switched capacitor coefficient of the local CPU of the m-th mobile user's device, whose value ranges from... The energy consumption level is determined by the local CPU's chip architecture and semiconductor process, and is an inherent hardware parameter. Energy consumption is proportional to the square of the CPU frequency; therefore, high-frequency operation leads to a significant increase in energy consumption. This invention reduces local computing pressure by partially offloading some components, thereby effectively reducing terminal energy consumption.

[0137] The advantages of this step are that it obtains multi-dimensional raw state information such as DAG task topology, edge node resources, user movement location, and historical request time series, accurately quantifies the latency and energy consumption benchmark values ​​of local execution of subtasks, and provides a reliable comparative reference for obtaining fine-grained offloading ratios in the future; by obtaining historical request time series and location data, it supports the LSTM time series prediction module in predicting future task popularity and channel change trends; and by obtaining complete DAG dependencies, it ensures the execution order of subtasks.

[0138] (4) The nth edge server normalizes the input data volume of the i-th subtask corresponding to the DAG graph obtained in step (3), the total number of CPU cycles required to complete the i-th subtask, the location coordinates of the mobile user to which the i-th subtask belongs in time slot t, the remaining computing resources, remaining cache resources of the nth edge server in time slot t-1, and the historical task request sequence of the preceding L consecutive time slots, respectively, so as to obtain the normalized input data volume of the i-th subtask, the normalized total number of CPU cycles required, the normalized location coordinates, the normalized remaining computing resources, the normalized remaining cache resources of the nth edge server in time slot t-1, and the normalized historical task request sequence of the preceding L consecutive time slots.

[0139] Because the range of values, physical units, and numerical magnitudes of task parameters, location coordinates, and resource states vary greatly, directly inputting them into a deep reinforcement learning model can lead to problems such as vanishing gradients, exploding gradients, training oscillations, and slow convergence. Therefore, this invention performs linear normalization on all state parameters, mapping them uniformly to the [0,1] interval, eliminating interference from differences in units and numerical fluctuations, and improving the model's training stability, convergence speed, and decision accuracy.

[0140] Specifically, in this step, the amount of input data for the i-th subtask corresponding to the subtask parameters of the DAG graph obtained in step (3) is... With the required total number of CPU cycles Normalization is performed to obtain the normalized input data volume for the i-th subtask. With the required total number of CPU cycles The formula is as follows:

[0141] ;

[0142] ;

[0143] in This indicates the maximum amount of input data allowed for a task in an edge computing system. This represents the maximum total number of CPU cycles required to complete a single subtask; both are preset thresholds for edge computing systems. Normalized data preserves the relative magnitudes of the original parameters while eliminating absolute numerical differences, enabling reinforcement learning models to treat features of different dimensions equally and improving feature extraction capabilities.

[0144] In this step, the position coordinates of the m-th user in time slot t obtained in step (3) are normalized to obtain the normalized position coordinates. The following formula is used:

[0145] , ;

[0146] in , These represent the maximum x-axis and maximum y-axis coordinates of the edge computing system's coverage area, corresponding to the coverage radius of the base station or roadside unit. The normalized coordinates characterize the relative position of mobile users within the coverage area, used to calculate communication parameters such as channel gain, transmission distance, and path loss, while also adapting to the versatility of different coverage scenarios.

[0147] In this step, the remaining computing resources and remaining cache resources of the N edge servers in the (t-1)th time slot are normalized to obtain the normalized remaining computing resources of the nth edge server in the (t-1)th time slot. and normalized remaining cache resources The following formula is used:

[0148] ;

[0149] ;

[0150] in , These represent the maximum computing resources and maximum cache resources of the nth edge server, respectively. The normalized resource state can characterize the load level of the edge nodes; the closer the value is to 1, the more abundant the resources, and the closer it is to 0, the more strained the resources. It can be directly used for reinforcement learning state input and cache decision constraints.

[0151] The advantage of this step is that it unifies input parameters of different dimensions, ranges, and scales, simplifies the complexity of the state space in dynamic environments, enhances the correlation between various input parameters, improves the ability of deep reinforcement learning models to analyze states and extract features in complex dynamic environments, and solves the technical problem of difficult model training and insufficient decision-making accuracy caused by excessive parameter differences.

[0152] (5) The nth edge server inputs the historical task request sequence of the nth edge server in the current time slot t after normalization of the L consecutive time slots obtained in step (4) into the pre-trained LSTM prediction model to obtain the task request probability of the (t+1)th time slot. Based on the request probability, the optimal cache decision of the (t+1)th time slot is obtained through the 0-1 knapsack optimization algorithm. The task request probability of the (t+1)th time slot, the normalized input data volume of the i-th sub-task, the required total CPU cycles, the normalized location coordinates, and the remaining computing resources and normalized cache resources of the nth edge server in the previous time slot t-1 are input into the pre-trained TD3 reinforcement learning model to obtain the optimal offloading ratio of the m-th user for the i-th sub-task. .

[0153] This step is the core innovation of this invention. Its advantage lies in the deep integration of time-series prediction, dynamic caching, and reinforcement learning decision-making, realizing the transformation from passive response to active optimization.

[0154] Specifically, the process in step (5) of this invention, which involves determining the optimal cache decision for the (t+1)th time slot based on the task request probability of the (t+1)th time slot using a 0-1 knapsack optimization algorithm, is as follows:

[0155] (A1) Based on the request probability distribution of the nth edge server in the (t+1)th time slot of the (nth)th subtask output by the LSTM prediction model, obtain the cache value of the i-th subtask. .

[0156] Specifically, the cache value of the i-th subtask in this step The following formula is used for calculation:

[0157] , ;

[0158] in, Let represent the request probability distribution of the i-th subtask in the (t+1)-th time slot of the n-th edge server. This represents the average execution latency saved after the i-th subtask's cache is hit. This represents the cache priority coefficient of the i-th subtask. As an indicator function, when all predecessor tasks of the DAG graph obtained by the i-th subtask in step (3) are cached or prioritized for scheduling, ;otherwise .

[0159] (A2) The cache value of the i-th subtask obtained from step (A1). And the optimal cache decision for the (t+1)th time slot is obtained through a 0-1 knapsack dynamic programming algorithm, which is equal to maximizing the total cache value within a finite space:

[0160] ;

[0161] ;

[0162] in Indicates whether to cache the i-th subtask. This represents the storage space required to cache the i-th subtask. This represents the remaining cache resources of the nth edge server in the tth time slot. Dynamic programming is used to solve this problem, obtaining the optimal cache set for the tth time slot, and updating the cache hit flag of the i-th subtask of the m-th mobile user in the tth time slot. A hit is 1, and a miss is 0.

[0163] The advantages of the sub-steps (A1) to (A2) above are that by using the task request probability predicted by LSTM, combined with latency savings, priority coefficients and DAG predecessor task constraints, the actual cache benefit of each sub-task is accurately quantified, effectively avoiding the invalid cache value caused by incomplete task links; and the cache deployment problem is transformed into a 0-1 knapsack optimization model under storage capacity constraints, and the global optimal cache solution that maximizes the total cache value within the limited storage space is solved by dynamic programming, rather than a local greedy suboptimal solution.

[0164] Specifically, the LSTM time series prediction model in this invention is trained through the following steps:

[0165] (B1) Collect the historical task request sequence of the nth edge server for L consecutive time slots under different time periods and different load conditions. Extract features from the collected historical task request sequence to obtain the periodicity, burstiness and correlation features between DAG subtasks. Construct a large-scale time series dataset (including task periodicity, burstiness and correlation features) based on the obtained feature samples, and randomly divide the large-scale time series dataset into training set and test set in a 9:1 ratio.

[0166] (B2) Normalize each sample in the training set obtained in step (B1) to obtain a normalized training set.

[0167] (B3) Initialize the hyperparameters of the LSTM model (including learning rate, batch size, number of iterations, Dropout rate, etc.) to obtain the initialized LSTM model.

[0168] Specifically, this step involves setting the initial learning rate to 10. -3Set the batch size to 32, the number of iteration rounds to 150, and the dropout rate to 0.2.

[0169] (B4) For each sample in the training set obtained in step (B2), input the sample into the LSTM model initialized in step (B3), obtain the predicted task request probability distribution of the sample in the (t+1)th time slot through forward propagation, calculate the cross-entropy loss between the predicted request probability distribution and the true request probability label of the sample, update the network weights of the LSTM model based on the cross-entropy loss and through the backpropagation algorithm to obtain the updated LSTM model.

[0170] (B5) Repeat step (B4) above until the LSTM model converges, thus obtaining the trained LSTM model.

[0171] Specifically, the TD3 reinforcement learning model in this invention is trained through the following steps:

[0172] (C1) Construct a large-scale edge computing simulation environment based on the physical operation rules of the edge computing system, obtain the simulation configuration of the large-scale edge computing simulation environment (including mobile user trajectory, DAG task structure, sub-task parameters, channel status, edge server load, caching strategy, etc.), obtain the simulation environment after the simulation configuration is completed, and perform multiple rounds of simulation simulation based on the simulation environment after the simulation configuration is completed, so as to obtain multiple full-scenario running samples as verification datasets.

[0173] (C2) The hyperparameters (including experience replay pool, learning rate, discount factor, soft update coefficient, exploration noise, preset batch, network update interval steps, etc.) and network structure (including Actor network, dual Critic network, target Actor network, target dual Critic network) of the TD3 reinforcement learning model are randomly initialized to obtain the initialized TD3 reinforcement learning model.

[0174] (C3) Based on the simulation environment after the simulation configuration obtained in step (C1), perform single-slot environment interaction and experience sample collection on the edge computing system to obtain the state of time slot t. Execution actions of time slot t Instant rewards for t-slots The state of time slot t+1 The quadruple formed The quadruple is then stored in a pre-established experience replay pool.

[0175] The TD3 reinforcement learning model uses a reward function r, and has the following:

[0176] ;

[0177] in Indicates normalization, These are the weighting coefficients. This represents the total response delay of the edge computing system in the t-th time slot. This represents the total energy consumption of the edge computing system in the t-th time slot. This represents the cache hit rate of the edge computing system in the t-th time slot. This represents the number of timeout tasks in the edge computing system during the t-th time slot.

[0178] (C4) Randomly obtain a small batch of data from the experience replay pool obtained in step (C3) (the small batch specifically refers to the preset batch in the hyperparameters of the initialized TD3 reinforcement learning model obtained in step (C2)), and use the small batch of data to update the dual Critic network in the TD3 reinforcement learning model to obtain the weight parameters of the two sets of main Critic networks after the update (to reduce the mean square error between the predicted Q value and the label Q value).

[0179] This step includes the following sub-steps:

[0180] (C4-1) Randomly select multiple time slots t from the experience replay pool obtained in step (C3). Execution actions of multiple time slots. Rewards for multiple t-slots and the status of multiple t+1 time slots. ;

[0181] (C4-2) Set the iteration counter b=0;

[0182] (C4-3) The states of the multiple t+1 time slots obtained in step (C4-1) Input the target Actor network of the initialized TD3 reinforcement learning model obtained in step (C2) to obtain candidate actions for time slot t+1. The state of time slot t+1 Candidate actions for slot t+1 The initialized TD3 reinforcement learning model, obtained through the common input step (C2), uses two sets of target Critic networks to obtain two sets of target Q-values. The smaller of the two target Q-values ​​is selected as the baseline target Q-value output, and the label Q-value is obtained by combining it with the single-step reward. The state of time slot t is then used to calculate the target Q-value. Execution action with time slot t The two sets of main Critic networks of the initialized TD3 reinforcement learning model obtained in step (C2) are used to obtain the predicted Q values ​​of the two sets of main Critic networks of the TD3 reinforcement learning model. The mean square error between the predicted Q value and the label Q value of each set is used as the loss function of the corresponding main Critic network, and the gradient descent method is used to perform backpropagation iteration on the main Critic network to obtain the updated main Critic network parameters.

[0183] (C4-4) Set the iteration counter b = b + 1;

[0184] (C4-5) Determine if b is an integer multiple of the network update interval step number obtained in step (C2). If it is, proceed to step (C4-6); otherwise, return to step (C4-3).

[0185] (C4-6) Take the average of the predicted Q values ​​of the two sets of main Critic networks obtained in step (C4-3), and obtain the Actor loss based on the average value as the parameters of the main Actor network after the back gradient update.

[0186] (C4-7) Perform target network soft update processing on the updated main Critic network parameters obtained in step (C4-3) and the main Actor network parameters updated by the reverse gradient obtained in step (C4-6) respectively, so as to obtain the updated target Actor network parameters and target Critic network parameters respectively.

[0187] The target network soft update is performed using the following formula:

[0188] ;

[0189] in This is the soft update coefficient, with a value ranging from 0.001 to 0.01. This represents the updated network parameters of the main Critic network and the main Actor network. This represents the network parameters of the target Critic network and the target Actor network before the update.

[0190] (C5) Repeat the above steps (C4) until the TD3 reinforcement learning model converges, thus obtaining the trained TD3 reinforcement learning model.

[0191] (6) The nth edge server obtains the optimal cache content in the tth time slot and the optimal offloading ratio of the ith subtask of the mth mobile user based on step (5). Get the local processing latency, local processing energy consumption, task uplink transmission latency, task uplink transmission energy consumption, edge processing latency, edge processing energy consumption, total execution latency, and earliest completion time for the m-th user on the i-th subtask.

[0192] This step, based on cache status and unloading decisions, completes a detailed model of latency and energy consumption, and is the core link in the performance calculation of edge computing systems.

[0193] Specifically, the edge processing latency of the m-th mobile user for the i-th subtask Energy consumption of edge processing The calculation formulas are as follows:

[0194] ;

[0195] ;

[0196] in The CPU frequency allocated to the i-th subtask by the n-th edge server in time slot t is determined by the edge resource scheduling algorithm, satisfying the following conditions: . Let be the effective switched capacitor coefficient of the nth edge server CPU.

[0197] Uplink transmission delay of the m-th mobile user to the i-th subtask Uplink power consumption of the task They are respectively:

[0198] ;

[0199] ;

[0200] in Let represent the wireless transmission rate between the m-th mobile user in time slot t and the edge server, and we have:

[0201] ;

[0202] in For the channel bandwidth of the edge computing system, Let m be the transmit power of the m-th mobile user. This represents the power of additive white Gaussian noise.

[0203] Let be the channel gain between the m-th mobile user and the n-th edge server in time slot t, and equal to:

[0204] ;

[0205] in The reference channel gain is given when the distance between the mobile user and the edge server is 1 meter. Let be the Euclidean distance between the m-th mobile user and the n-th edge server. This is the path loss index, with a value between 2 and 4.

[0206] Local processing latency of the m-th mobile user for the i-th subtask and local processing energy consumption They are respectively:

[0207] ;

[0208] ;

[0209] The local execution ratio is determined by both the local and uninstallation ratios, and can be adjusted accordingly. It can balance the computing pressure, latency, and energy consumption between local and edge environments.

[0210] Based on the DAG dependency relationship, the earliest start time of the i-th subtask of the m-th mobile user. Let be the maximum completion time of all predecessor tasks of the i-th subtask, expressed as:

[0211] ;

[0212] Cross transmission delay This can be ignored. If the i-th subtask has no predecessor task, then... .

[0213] The earliest completion time of the i-th subtask for the m-th mobile user is calculated using the following formula:

[0214] ;

[0215] in Let be the total execution latency of the i-th subtask for the m-th mobile user, and let its value be the maximum time taken for parallel execution at both local and edge locations, expressed as:

[0216] ;

[0217] If the earliest completion time of the i-th subtask of the m-th mobile user Less than or equal to its maximum tolerable delay If the task is completed on time, it will be considered a timeout and will be included in the edge computing system penalty item.

[0218] The advantage of this step is that it fully considers the comprehensive impact of cache state, dependencies, communication conditions, and resource allocation on task execution, resulting in high modeling accuracy and close resemblance to real-world scenarios. This provides an accurate basis for optimizing reinforcement learning reward calculations and edge computing systems.

[0219] (7) Based on the local processing energy consumption, uplink transmission energy consumption, edge processing energy consumption, and earliest completion time of the m-th mobile user for the i-th subtask obtained in step (6), the n-th edge server obtains the total response latency and total energy consumption of the edge computing system in the t-th time slot, sets t=t+1, and returns to step (2).

[0220] Specifically, the total response delay of the edge computing system in the t-th time slot equal:

[0221] ;

[0222] Total energy consumption of the edge computing system in the t-th time slot It equals the sum of local processing energy consumption, task uplink transmission energy consumption, and edge processing energy consumption:

[0223] ;

[0224] (8) The nth edge server obtains the utility of the edge computing system based on the all-local processing latency and all-local processing energy consumption of the i-th subtask of the m-th mobile user obtained in step (3), and the total response latency and total energy consumption of the edge computing system in the t-th time slot obtained in step (7). .

[0225] Step (8) includes the following sub-steps:

[0226] (8-1) The all-local processing delay of the i-th subtask of the m-th mobile user obtained in step (3) And all-local processing energy consumption Obtain the total latency for all K subtasks of all M users across all T time slots to be processed entirely locally. Total energy consumption :

[0227] ;

[0228] ;

[0229] Specifically, this metric measures the upper limit of edge computing system performance without any offloading and caching optimizations.

[0230] (8-2) The total response delay of the edge computing system in the t-th time slot obtained from step (7) With total energy consumption Obtain the optimized total latency of the edge computing system. Total energy consumption :

[0231] ;

[0232] ;

[0233] (8-3) The total latency for all tasks to be processed entirely locally, as obtained from step (8-1). Total energy consumption and the optimized total latency of the edge computing system obtained in step (8-2). Total energy consumption To obtain the utility of edge computing systems (This indicator combines latency and energy consumption; a higher value indicates better optimization results.)

[0234] ;

[0235] in , These are the time delay weighting coefficient and the energy consumption weighting coefficient, respectively, with values ​​ranging from [0,1], and satisfying the following conditions: This can be adjusted according to business preferences. Latency-sensitive businesses can increase [the limit / size]. Energy-sensitive businesses can increase .

[0236] The advantage of step (8-3) is that, through benchmarking, it avoids the influence of... , The problem of underflow where the utility value approaches 0 due to excessively large values ​​can be clearly demonstrated, and the improvement ratio of the optimized solution relative to the fully local execution baseline can be intuitively shown.

[0237] (9) The nth edge server will use the edge computing system utility obtained in step (8) The total latency, total energy consumption, cache hit rate, number of timeout tasks, and utility of the edge computing system are fed back to all mobile users.

[0238] Specifically, the edge server feeds back the total latency, total energy consumption, cache hit rate, number of timed-out tasks, and edge computing system utility to the mobile user via the downlink control channel. The mobile user can adjust the task generation strategy and service request mode based on this feedback. The edge server can also use the feedback to further fine-tune the model parameters, achieving continuous evolution. At this point, the entire process of edge computing task unloading, caching, scheduling, and optimization is complete.

[0239] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for DAG-dependent task offloading based on predictive dynamic caching and TD3 reinforcement learning, applicable to an edge computing system containing multiple edge servers and multiple mobile users, characterized in that... The method includes the following steps: (1) The nth edge server obtains the total number T of time slots in the edge computing system and sets the time slot number t=1; (2) The nth edge server determines whether t is greater than the total number of time slots T in the edge computing system obtained in step (1). If it is, proceed to step (8); otherwise, proceed to step (3), where n∈[1, N] and N represents the total number of edge servers in the edge computing system. (3) The nth edge server obtains from the mth mobile user the DAG graph of the mobile user in the tth time slot, the parameters of the i-th subtask corresponding to the DAG graph, the dependencies between subtasks, and the position coordinates of the m-th mobile user to which the i-th subtask belongs in the time slot t, including the x-axis coordinates. and y-axis coordinates Obtain the all-local processing latency and all-local processing energy consumption of the i-th subtask corresponding to the DAG graph, and obtain the remaining computing resources of the n-th edge server in the (t-1)-th time slot. Remaining cache resources The sequence of historical task request counts, p(t-L+1), p(t-L+2), ..., p(t), consists of node load rate and the number of historical task requests over L consecutive time slots; where m∈[1,M], M represents the total number of mobile users in the edge computing system, i∈[1,K], K is the number of all subtasks corresponding to the DAG graph of the m-th mobile user in the t-th time slot, and the parameters of the i-th subtask include the amount of input data for the i-th subtask. The total number of CPU cycles required to complete the i-th subtask And the maximum tolerable delay of the i-th subtask. The value of L ranges from 3 to 20; (4) The nth edge server obtains the edge computing system utility based on the all-local processing latency and all-local processing energy consumption of the i-th subtask of the m-th mobile user obtained in step (3). ; (5) The nth edge server will use the edge computing system utility obtained in step (4) Feedback will be sent to all mobile users.

2. The DAG dependency task edge unloading method based on predictive dynamic caching and TD3 reinforcement learning according to claim 1, characterized in that, The DAG-dependent task edge unloading method further includes the following steps after step (3) and before step (4): (A) The nth edge server normalizes the input data volume of the i-th subtask corresponding to the DAG graph obtained in step (3), the total number of CPU cycles required to complete the i-th subtask, the location coordinates of the mobile user to which the i-th subtask belongs in time slot t, the remaining computing resources, remaining cache resources of the nth edge server in time slot t-1, and the historical task request sequence of the preceding L consecutive time slots, respectively, so as to obtain the normalized input data volume of the i-th subtask, the normalized total number of CPU cycles required, the normalized location coordinates, the normalized remaining computing resources, the normalized remaining cache resources of the nth edge server in time slot t-1, and the normalized historical task request sequence of the preceding L consecutive time slots; (B) The nth edge server inputs the historical task request sequence of the nth edge server in the current time slot t (normalized in step (A)) into the pre-trained LSTM prediction model to obtain the task request probability of the (t+1)th time slot. Based on this request probability, and through the 0-1 knapsack optimization algorithm, the optimal caching decision for the (t+1)th time slot is obtained, and the nth edge server... The task request probability of each time slot, the normalized input data volume and total CPU cycles required for the i-th subtask, the normalized location coordinates, and the remaining computing resources and normalized cache resources of the n-th edge server in the previous time slot t-1 are input into a pre-trained TD3 reinforcement learning model to obtain the optimal offloading ratio of the m-th user for the i-th subtask. ; (C) The nth edge server obtains the optimal cache content in the t-th time slot from step (B) and the optimal offloading ratio of the ith subtask of the m-th mobile user. Get the local processing latency, local processing energy consumption, task uplink transmission latency, task uplink transmission energy consumption, edge processing latency, edge processing energy consumption, total execution latency, and earliest completion time for the m-th user on the i-th subtask. (D) Based on the local processing energy consumption, uplink transmission energy consumption, edge processing energy consumption, and earliest completion time of the m-th mobile user for the i-th subtask obtained in step (C), the n-th edge server obtains the total response latency and total energy consumption of the edge computing system in the t-th time slot, sets t=t+1, and returns to step (2). Fully local processing latency of the i-th subtask corresponding to the DAG graph of the m-th mobile user in the t-th time slot. equal: ； in The frequency of the local CPU of the device of the m-th mobile user; Fully local processing energy consumption of the i-th subtask corresponding to the DAG graph of the m-th mobile user in the t-th time slot equal: ； in Let be the effective switching capacitor coefficient of the local CPU of the m-th mobile user's device.

3. The DAG dependency task edge unloading method based on predictive dynamic caching and TD3 reinforcement learning according to claim 2, characterized in that, The amount of input data for the i-th subtask corresponding to the subtask parameters of the DAG graph obtained in step (3). With the required total number of CPU cycles Normalization is performed to obtain the normalized input data volume for the i-th subtask. With the required total number of CPU cycles The formula is as follows: ； ； in This indicates the maximum amount of input data allowed for a task in an edge computing system. This represents the maximum total number of CPU cycles required to complete a single subtask. In step (A), the position coordinates of the m-th user in time slot t obtained in step (3) are normalized to obtain the normalized position coordinates. The following formula is used: ， ； in , These represent the maximum x-axis coordinate and the maximum y-axis coordinate of the coverage area of ​​the edge computing system, respectively. In step (A), the remaining computing resources and remaining cache resources of the N edge servers in the (t-1)th time slot are normalized to obtain the normalized remaining computing resources of the nth edge server in the (t-1)th time slot. and normalized remaining cache resources The following formula is used: ； ； in , These represent the maximum computing resources and maximum cache resources of the nth edge server, respectively.

4. The DAG dependency task edge unloading method based on predictive dynamic caching and TD3 reinforcement learning according to claim 3, characterized in that, Step (B) involves determining the optimal caching decision for the (t+1)th time slot based on the task request probability of the (t+1)th time slot using a 0-1 knapsack optimization algorithm. The specific steps are as follows: (A1) Based on the request probability distribution of the nth edge server in the (t+1)th time slot of the (nth)th subtask output by the LSTM prediction model, obtain the cache value of the i-th subtask. : ， ； in, Let represent the request probability distribution of the i-th subtask in the (t+1)-th time slot of the n-th edge server. This represents the average execution latency saved after the i-th subtask's cache is hit. This represents the cache priority coefficient of the i-th subtask; As an indicator function, when all predecessor tasks of the DAG graph obtained by the i-th subtask in step (3) are cached or prioritized for scheduling, ;otherwise ; (A2) The cache value of the i-th subtask obtained from step (A1). And the optimal cache decision for the (t+1)th time slot is obtained through a 0-1 knapsack dynamic programming algorithm, which is equal to maximizing the total cache value within a finite space: ； ； in Indicates whether to cache the i-th subtask. This represents the storage space required to cache the i-th subtask. This represents the remaining cache resources of the nth edge server in the tth time slot.

5. The DAG dependency task edge unloading method based on predictive dynamic caching and TD3 reinforcement learning according to claim 4, characterized in that, The LSTM time series prediction model is trained through the following steps: (B1) Collect the historical task request sequence of the nth edge server for L consecutive time slots under different time periods and different load conditions, extract features from the collected historical task request sequence, obtain the correlation features between task periodicity, burstiness and DAG subtasks, construct a large-scale time series dataset based on the obtained feature samples, and randomly divide the large-scale time series dataset into training set and test set in a 9:1 ratio. (B2) Normalize each sample in the training set obtained in step (B1) to obtain a normalized training set; (B3) Initialize the hyperparameters of the LSTM model to obtain the initialized LSTM model; (B4) For each sample in the training set obtained in step (B2), input the sample into the LSTM model initialized in step (B3), obtain the predicted task request probability distribution of the sample in the (t+1)th time slot through forward propagation, calculate the cross-entropy loss between the predicted request probability distribution and the true request probability label of the sample, update the network weights of the LSTM model based on the cross-entropy loss and through the backpropagation algorithm to obtain the updated LSTM model; (B5) Repeat step (B4) above until the LSTM model converges, thus obtaining the trained LSTM model.

6. The DAG dependency task edge unloading method based on predictive dynamic caching and TD3 reinforcement learning according to claim 5, characterized in that, The TD3 reinforcement learning model is trained through the following steps: (C1) Construct a large-scale edge computing simulation environment based on the physical operation rules of the edge computing system, obtain the simulation configuration of the large-scale edge computing simulation environment, obtain the simulation environment after the simulation configuration is completed, and conduct multiple rounds of simulation simulation based on the simulation environment after the simulation configuration is completed, so as to obtain multiple full-scenario running samples as a verification dataset. (C2) The hyperparameters and network structure of the TD3 reinforcement learning model are randomly initialized to obtain the initialized TD3 reinforcement learning model; the hyperparameters include the experience replay pool, learning rate, discount factor, soft update coefficient, exploration noise, preset batch, and network update interval steps; the network structure includes Actor network, dual Critic network, target Actor network, and target dual Critic network. (C3) Based on the simulation environment after the simulation configuration is completed in step (C1), the edge computing system is subjected to single-slot environment interaction and empirical sample collection in sequence to obtain the state of time slot t. Execution actions of time slot t Instant rewards for t-slots The state of time slot t+1 The quadruple formed And store the quadruple into a pre-established experience replay pool; The TD3 reinforcement learning model uses a reward function r, and has the following: ； Where ^ denotes normalization. These are the weighting coefficients. This represents the total response delay of the edge computing system in the t-th time slot. This represents the total energy consumption of the edge computing system in the t-th time slot. This represents the cache hit rate of the edge computing system in the t-th time slot. This represents the number of timed-out tasks in the edge computing system during the t-th time slot; (C4) Randomly obtain a small batch of data from the experience replay pool obtained in step (C3), and use the small batch of data to update the dual Critic network in the TD3 reinforcement learning model to obtain the weight parameters of the two sets of main Critic networks after the update. (C5) Repeat the above steps (C4) until the TD3 reinforcement learning model converges, thus obtaining the trained TD3 reinforcement learning model.

7. The DAG dependency task edge unloading method based on predictive dynamic caching and TD3 reinforcement learning according to claim 6, characterized in that, Step (C4) includes the following sub-steps: (C4-1) Randomly select multiple time slots t from the experience replay pool obtained in step (C3). Execution actions of multiple time slots. Rewards for multiple t-slots and the status of multiple t+1 time slots. ; (C4-2) Set the iteration counter b=0; (C4-3) The states of the multiple t+1 time slots obtained in step (C4-1) Input the target Actor network of the initialized TD3 reinforcement learning model obtained in step (C2) to obtain candidate actions for time slot t+1. The state of time slot t+1 Candidate actions for slot t+1 The initialized TD3 reinforcement learning model, obtained through the common input step (C2), uses two sets of target Critic networks to obtain two sets of target Q-values. The smaller of the two target Q-values ​​is selected as the baseline target Q-value output, and the label Q-value is obtained by combining it with the single-step reward. The state of time slot t is then used to calculate the target Q-value. Execution action with time slot t The two sets of main Critic networks of the initialized TD3 reinforcement learning model obtained in step (C2) are used to obtain the predicted Q values ​​of the two sets of main Critic networks of the TD3 reinforcement learning model. The mean square error between the predicted Q value and the label Q value of each set is used as the loss function of the corresponding main Critic network, and the gradient descent method is used to perform backpropagation iteration on the main Critic network to obtain the updated main Critic network parameters. (C4-4) Set the iteration counter b = b + 1; (C4-5) Determine if b is an integer multiple of the network update interval step number obtained in step (C2). If it is, proceed to step (C4-6); otherwise, return to step (C4-3). (C4-6) Take the average of the predicted Q values ​​of the two sets of main Critic networks obtained in step (C4-3), and obtain the Actor loss based on the average value as the parameters of the main Actor network after the back gradient update. (C4-7) Perform target network soft update processing on the updated main Critic network parameters obtained in step (C4-3) and the main Actor network parameters updated by the reverse gradient obtained in step (C4-6) respectively, so as to obtain the updated target Actor network parameters and target Critic network parameters respectively. The target network soft update is performed using the following formula: ； in This is the soft update coefficient, with a value ranging from 0.001 to 0.

01. This represents the updated network parameters of the main Critic network and the main Actor network. This represents the network parameters of the target Critic network and the target Actor network before the update.

8. The DAG dependency task edge unloading method based on predictive dynamic caching and TD3 reinforcement learning according to claim 7, characterized in that, Edge processing latency of the m-th mobile user for the i-th subtask Energy consumption of edge processing The calculation formulas are as follows: ； ； in The CPU frequency allocated to the i-th subtask by the n-th edge server in time slot t is determined by the edge resource scheduling algorithm, satisfying the following conditions: ; The effective switched capacitor coefficient of the nth edge server CPU; Uplink transmission delay of the m-th mobile user to the i-th subtask Uplink power consumption of the task They are respectively: ； ； in Let represent the wireless transmission rate between the m-th mobile user in time slot t and the edge server, and we have: ； in For the channel bandwidth of the edge computing system, Let m be the transmit power of the m-th mobile user. This represents the power of additive white Gaussian noise. Let be the channel gain between the m-th mobile user and the n-th edge server in time slot t, and equal to: ； in The reference channel gain is given when the distance between the mobile user and the edge server is 1 meter. Let be the Euclidean distance between the m-th mobile user and the n-th edge server. This is the path loss exponent, with a value between 2 and 4. Local processing latency of the m-th mobile user for the i-th subtask and local processing energy consumption They are respectively: ； ； Earliest start time of the i-th subtask for the m-th mobile user Let be the maximum value of the completion times of all predecessor tasks of the i-th subtask, and equal to: ； in Indicates cross-transmission delay; The earliest completion time of the i-th subtask for the m-th mobile user is calculated using the following formula: ； in Let be the total execution latency of the i-th subtask for the m-th mobile user. Its value is the maximum time required for parallel execution both locally and at the edge, and is equal to: 。 9. The DAG dependency task edge unloading method based on predictive dynamic caching and TD3 reinforcement learning according to claim 8, characterized in that, Total response delay of the edge computing system in the t-th time slot equal: ； Total energy consumption of the edge computing system in the t-th time slot It equals the sum of local processing energy consumption, task uplink transmission energy consumption, and edge processing energy consumption: ； Step (4) includes the following sub-steps: (4-1) The all-local processing delay of the i-th subtask of the m-th mobile user obtained in step (3) And all-local processing energy consumption Obtain the total latency for all K subtasks of all M users across all T time slots to be processed entirely locally. Total energy consumption : ； ； (4-2) The total response delay of the edge computing system in the t-th time slot obtained from step (D) With total energy consumption Obtain the optimized total latency of the edge computing system. Total energy consumption : ； ； (4-3) The total latency for all tasks to be processed entirely locally, as obtained from step (4-1). Total energy consumption and the optimized total latency of the edge computing system obtained in step (4-2). Total energy consumption To obtain the utility of edge computing systems : ； in , These are the time delay weighting coefficient and the energy consumption weighting coefficient, respectively, with values ​​ranging from [0,1], and satisfying the following conditions: .

10. A DAG-dependent task edge offloading system based on predictive dynamic caching and TD3 reinforcement learning, applied to an edge computing system containing multiple edge servers and multiple mobile users, characterized in that... The system includes the following modules: The first module, located on the nth edge server, is used to obtain the total number T of time slots in the edge computing system and set the time slot number t=1. The second module, which is set on the nth edge server, is used to determine whether t is greater than the total number of time slots T in the edge computing system obtained by the first module. If it is, it enters the eighth module; otherwise, it enters the third module. Here, n∈[1,N], and N represents the total number of edge servers in the edge computing system. The third module, located on the edge server, is used to obtain the DAG graph of the m-th mobile user in the t-th time slot, the parameters of the i-th subtask corresponding to the DAG graph, the dependencies between subtasks, and the position coordinates of the m-th mobile user to which the i-th subtask belongs in time slot t, including the x-axis coordinates. and y-axis coordinates Obtain the all-local processing latency and all-local processing energy consumption of the i-th subtask corresponding to the DAG graph, and obtain the remaining computing resources of the n-th edge server in the (t-1)-th time slot. Remaining cache resources The sequence of historical task request counts, p(t-L+1), p(t-L+2), ..., p(t), consists of node load rate and the number of historical task requests over L consecutive time slots; where m∈[1,M], M represents the total number of mobile users in the edge computing system, i∈[1,K], K is the number of all subtasks corresponding to the DAG graph of the m-th mobile user in the t-th time slot, and the parameters of the i-th subtask include the amount of input data for the i-th subtask. The total number of CPU cycles required to complete the i-th subtask And the maximum tolerable delay of the i-th subtask. The value of L ranges from 3 to 20; The fourth module, located on the nth edge server, is used to obtain the edge computing system utility based on the all-local processing latency and all-local processing energy consumption of the i-th subtask of the m-th mobile user obtained from the third module. ; The fifth module, located on the nth edge server, is used to transfer the edge computing system utility obtained from the fourth module. Feedback will be sent to all mobile users.

Images