An edge computing task dynamic scheduling method based on deep reinforcement learning

Abstract

The application relates to the technical field of edge computing, and particularly discloses an edge computing task dynamic scheduling method based on deep reinforcement learning. The method comprises the following steps: obtaining an environment state; using a RainbowLBO deep reinforcement learning model fusing a showdown network and a noise network to select a task allocation action; executing the action and obtaining a reward signal based on a task response time and a load imbalance degree; and updating the deep reinforcement learning model by using the environment state, the action and the reward signal. The application can effectively improve the load balancing effect and the task response efficiency in an edge heterogeneous resource environment.

Description

Technical Field

[0001] This application relates to the field of edge computing technology, specifically to a dynamic scheduling method for edge computing tasks based on deep reinforcement learning. Background Technology

[0002] The current edge computing environment is characterized by highly heterogeneous server resources. Different edge servers exhibit significant differences in CPU, memory, and disk I / O configurations, resulting in various server types, including general-purpose, compute-intensive, memory-intensive, and I / O-intensive servers. This heterogeneity makes it difficult for traditional load assessment methods to accurately quantify the true load status of servers, as fixed-weight load calculation models cannot adapt to the dynamic changes in the sensitivity of different server types to various resources. Furthermore, existing deep reinforcement learning-based task scheduling methods have significant shortcomings when applied to this environment: algorithms such as standard Deep Q-Networks (DQN) fail to fully consider the impact of server heterogeneity on state representation and reward function design; their action exploration strategies lack adaptive adjustment to task urgency; and reward functions typically only focus on immediate task response time while ignoring long-term load balancing effects. This leads to problems such as load imbalance and large fluctuations in task response time in dynamic edge environments. In addition, existing methods fail to effectively balance short-term rewards and long-term benefits during experience replay, further limiting the overall performance optimization of scheduling algorithms in heterogeneous resource environments. Summary of the Invention

[0003] To solve or at least partially solve the above-mentioned technical problems, embodiments of this application provide a dynamic scheduling method for edge computing tasks based on deep reinforcement learning.

[0004] This invention provides a dynamic scheduling method for edge computing tasks based on deep reinforcement learning, applicable to heterogeneous edge resource environments, comprising the following steps:

[0005] S1. Obtain the current environmental status, which includes the resource requirements of the computing task to be processed, the idle resources of each edge server in the edge server cluster, the normalized real-time processing speed of each edge server, and the load status of each edge server.

[0006] S2. Based on the environmental state, a deep reinforcement learning model is used to select an action, wherein the action is to assign the computing task to be processed to an edge server in the edge server cluster, wherein the deep reinforcement learning model is a RainbowLBO model that integrates a duel network and a noisy network.

[0007] S3. Perform the action to assign the computing task to be processed to the selected edge server and obtain a reward signal, the reward signal being calculated based on task response time and load imbalance.

[0008] S4. Update the deep reinforcement learning model using the environmental state, the action, the reward signal, and the environmental state at the next time step.

[0009] Optionally, updating the deep reinforcement learning model in S4 includes the following steps:

[0010] S4.11. Store the experience tuple into the experience replay buffer, wherein the experience tuple includes the environment state, the action, the reward signal, and the environment state at the next moment;

[0011] S4.12 Calculate the temporal difference error of the empirical tuple;

[0012] S4.13. Determine the sampling priority of the empirical tuple based on the time-series differential error;

[0013] S4.14. Sample experience tuples from the experience playback buffer according to the sampling priority;

[0014] S4.15. Update the parameters of the deep reinforcement learning model using the sampled empirical tuples.

[0015] Optionally, updating the deep reinforcement learning model in S4 employs a double-Q learning mechanism, including the following steps:

[0016] S4.21. Using the evaluation network in the deep reinforcement learning model, select the action with the largest Q value from the next time-step environment state;

[0017] S4.22. Using the target network in the deep reinforcement learning model, calculate the target Q value corresponding to the selected action;

[0018] S4.23. Update the parameters of the evaluation network based on the target Q value.

[0019] Optionally, the RainbowLBO model in S2 adopts a structure that integrates the adversarial network and the noise network, including the following processing:

[0020] S2.1 In the adversarial network, the state-action value function of the deep reinforcement learning model is decomposed into a state value function and an advantage function;

[0021] S2.2 Introduce learnable Gaussian noise into the fully connected layer parameters of the dominant function;

[0022] S2.3. Based on the state value function and the advantage function after adding Gaussian noise, calculate the state-action value function value.

[0023] Optionally, the specific method for obtaining the reward signal in step S3 includes:

[0024] S3.1 Calculate the task response time, wherein the task response time is the sum of the task waiting time and the task execution time;

[0025] S3.2 Calculate the load imbalance, where the load imbalance is the coefficient of variation of the load of each server in the edge server cluster;

[0026] S3.3. Based on the task response time and the load imbalance, calculate the reward signal by weighted summation.

[0027] Optionally, in step S3.3, the calculation of the reward signal is further based on the comparison between the task response time and the task deadline, and the calculation formula for the reward signal is:

[0028]

[0029] in, It is the task response time. LI represents the task deadline and the load imbalance level. , , It is the weighting coefficient.

[0030] Optionally, in S2.2, the noise scale of the learnable Gaussian noise is adjusted according to the urgency of the task, and the calculation formula for the noise scale is:

[0031]

[0032]

[0033] Where U represents the urgency of the task. , and This is the adjusted noise scale. and It is the basic noise scale. It is the task deadline. It is the moment to make a decision.

[0034] Optionally, the environmental state in S1 further includes the number of pending tasks for each edge server in the edge server cluster, and the environmental state is represented as follows:

[0035]

[0036] in, This represents the number of pending tasks on each edge server at time t. This indicates the resource requirements of the computational task to be processed. This indicates the idle resources of the edge server. This indicates the normalized real-time processing speed of the edge server. This indicates the load status of the edge server.

[0037] Optionally, in S4.22, the calculation of the target Q value adopts an N-step reward mechanism, and the formula for calculating the target Q value is:

[0038]

[0039] Where N is the number of steps, and γ is the discount factor. It is the reward signal for the k-th step in the future. The target network in the environmental state The target Q value estimation is as follows. These are the target network parameters. It is the selected action.

[0040] Optionally, when updating the parameters of the deep reinforcement learning model using sampled empirical tuples in S4.15, the following steps are further included:

[0041] S4.15.1 Calculate the importance sampling weights based on the sampling priority;

[0042] S4.15.2, Correct the gradient update using the aforementioned importance sampling weights;

[0043] Importance sampling weight The calculation formula is:

[0044]

[0045] Where A is the size of the experience playback buffer. It is the sampling probability of empirical tuple i, and β is the importance sampling correction parameter, which is adjusted from the initial value during training. It increases linearly to 1.0.

[0046] The method provided by this invention has the following beneficial effects:

[0047] The method provided by this invention can comprehensively perceive the real-time operating status of the system by acquiring environmental conditions including task resource requirements, server idle resources, processing speed, and load status, providing an accurate basis for scheduling decisions. It employs a deep reinforcement learning model that integrates adversarial networks and noisy networks for task allocation decisions, effectively distinguishing the state value and action advantages of different servers, and enhancing exploration capabilities through adaptive noise injection, thereby achieving more reasonable task scheduling in edge heterogeneous resource environments. Based on reward signals calculated from task response time and load imbalance, the model is guided to simultaneously optimize task execution efficiency and system load distribution, promoting improved resource utilization.

[0048] By employing an experience replay buffer and a priority sampling mechanism, data utilization efficiency is improved, accelerating the model convergence process. The introduction of a double-Q learning mechanism reduces value estimation bias and enhances learning stability. The inclusion of a task deadline violation penalty term in the reward function makes the model more focused on task completion timeliness, contributing to improved task success rate. The noisy network parameters are dynamically adjusted based on task urgency, achieving adaptive optimization of the exploration strategy and improving decision-making efficiency while maintaining exploration capabilities.

[0049] Introducing information about the number of tasks pending on the server into the environmental state enhances the system's ability to predict potential congestion, making scheduling decisions more forward-looking. The application of an N-step reward mechanism allows the model to better balance immediate rewards and long-term benefits, optimizing long-term load balancing performance. The use of importance sampling weights effectively corrects the bias caused by priority sampling, further improving training stability. The combined effect of these technical features enables this method to significantly reduce load imbalance and improve task response speed in edge heterogeneous resource environments, while maintaining a high task completion rate, thereby comprehensively improving the service quality and resource utilization efficiency of edge computing systems. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of a dynamic scheduling method for edge computing tasks based on deep reinforcement learning, provided by an embodiment of the present invention.

[0051] Figure 2 A schematic diagram of another dynamic scheduling method for edge computing tasks based on deep reinforcement learning provided in an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of another dynamic scheduling method for edge computing tasks based on deep reinforcement learning provided in an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of another dynamic scheduling method for edge computing tasks based on deep reinforcement learning provided in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of another dynamic scheduling method for edge computing tasks based on deep reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0056] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0057] Figure 1 This invention provides a flowchart illustrating a dynamic scheduling method for edge computing tasks based on deep reinforcement learning, applicable to heterogeneous edge resource environments. The method includes the following steps:

[0058] S1. Obtain the current environmental status, which includes the resource requirements of the computing task to be processed, the idle resources of each edge server in the edge server cluster, the normalized real-time processing speed of each edge server, and the load status of each edge server.

[0059] S2. Based on the environmental state, a deep reinforcement learning model is used to select an action, which is to assign the computation task to be processed to an edge server in the edge server cluster. The deep reinforcement learning model is a RainbowLBO model that integrates a duel network and a noisy network.

[0060] S3. Execute the action, assign the computing tasks to be processed to the selected edge server, and obtain the reward signal. The reward signal is calculated based on the task response time and load imbalance.

[0061] S4. Update the deep reinforcement learning model using the environment state, actions, reward signals, and the environment state at the next time step.

[0062] The heterogeneous edge resource environment consists of various types of edge servers, including general-purpose edge servers, compute-intensive edge servers, memory-intensive edge servers, and I / O-intensive edge servers. These servers differ in computing power, memory capacity, and disk I / O performance. Dynamic task scheduling aims to rationally allocate computing tasks to servers to optimize system performance.

[0063] First, the current environmental state is obtained. This environmental state includes the resource requirements of the computing tasks to be processed, the idle resources of each edge server in the edge server cluster, the normalized real-time processing speed of each edge server, and the load status of each edge server. The resource requirements of the computing tasks to be processed refer to the amount of CPU, memory, and disk I / O requested by the task. For example, a task might require a 2GHz CPU, 4GB of memory, and 100MB / s of I / O bandwidth. Edge server idle resources represent the currently available CPU cycles, memory space, and I / O bandwidth of the server; these resources change dynamically over time. The normalized real-time processing speed is a relative value of the server's processing speed, used to eliminate the impact of performance differences between different servers, and is usually calculated as the ratio of current processing capacity to maximum processing capacity. The load status reflects the current workload of the server and can be a combined indicator of CPU utilization, memory utilization, or I / O utilization. The environmental state is obtained in real-time through monitoring tools or sensors to ensure data accuracy. For example, in a cluster containing 10 edge servers, the environmental state might be a vector containing task requirements and the resource status of each server.

[0064] Action selection is performed using a deep reinforcement learning model based on the environment state. This deep reinforcement learning model is the RainbowLBO model, which combines the structures of a duel network and a noisy network. The duel network decomposes the state-action value function into a state value function and an advantage function. The state value function evaluates the long-term value of a given state, while the advantage function evaluates the advantage of a particular action relative to the average action. The noisy network introduces learnable Gaussian noise into the fully connected layer parameters of the advantage function, enhancing exploration capabilities by adding randomness and preventing the model from getting trapped in local optima. The action selection process involves inputting the environment state into the RainbowLBO model, which outputs an action value representing the assignment of the computational task to a specific edge server in the edge server cluster. For example, the model might choose to assign the task to a computationally intensive edge server because that server currently has high CPU idle resources. The RainbowLBO model is trained based on historical data, optimizing the policy through continuous interaction with the environment. The goal of action selection is to maximize cumulative reward, considering both immediate and future gains.

[0065] The selected action is executed, allocating the computational tasks to the chosen edge servers. After allocation, the system collects reward signals, calculated based on task response time and load imbalance. Task response time is the total time from task arrival in the system to completion, including waiting time and execution time. Waiting time refers to the time a task waits for resources in the queue, while execution time refers to the actual time the task runs on the server. Load imbalance is the coefficient of variation of the load across servers in the edge server cluster, used to measure the uniformity of load distribution; it is calculated as the ratio of the load standard deviation to the load mean. The reward signal is calculated using a weighted summation method, as shown in the formula: ,in It is the task response time. LI represents the task deadline and the load imbalance level. and These are weighting coefficients used to balance the importance of response time and load balancing. The negative sign indicates that the reward is a penalty; the smaller the value, the better. The reward signal reflects the immediate effect of the scheduling decision and is used to guide model learning.

[0066] The deep reinforcement learning model is updated using the environment state, actions, reward signals, and the next-time environment state. The update process involves storing empirical data in an experience replay buffer, and then sampling the data for model training. The empirical data includes the current state, the action taken, the reward received, and the next state, forming a four-tuple. Model updates optimize network parameters using gradient descent to reduce prediction errors. For example, a loss function is calculated using temporal difference error, and weights are adjusted via backpropagation. The update frequency depends on the task arrival rate and is typically performed after each task assignment or periodically. This process allows the model to adapt to environmental changes and progressively improve scheduling strategies. The entire implementation emphasizes real-time performance and adaptability, ensuring effective operation in dynamic edge environments.

[0067] This method improves load balancing and task processing efficiency in edge computing systems. Through intelligent scheduling, it reduces server overload or idle periods, optimizing resource utilization. Task response time is reduced, improving user experience and system throughput. Simultaneously, the self-learning capability of the deep reinforcement learning model enables the system to cope with sudden load or resource changes, enhancing robustness and stability.

[0068] Figure 2 This is a schematic flowchart of another edge computing task dynamic scheduling method based on deep reinforcement learning provided by an embodiment of the present invention. In some embodiments, updating the deep reinforcement learning model in S4 includes the following steps:

[0069] S4.11. Store the experience tuple into the experience replay buffer. The experience tuple includes the environment state, action, reward signal and the environment state at the next moment.

[0070] S4.12 Calculate the temporal difference error of empirical tuples;

[0071] S4.13 Determine the sampling priority of empirical tuples based on time-series differential error;

[0072] S4.14. Sample empirical tuples from the empirical replay buffer according to the sampling priority;

[0073] S4.15. Update the parameters of the deep reinforcement learning model using the sampled empirical tuples.

[0074] First, experience tuples are stored in an experience replay buffer. Each experience tuple includes the environment state, action, reward signal, and next-moment environment state, forming a complete interaction record. The environment state contains task resource requirements and server status information; the action is the task allocation decision; the reward signal is calculated based on task response time and load imbalance; and the next-moment environment state reflects the system's new state after the action is executed. The experience replay buffer is a fixed-size storage pool that uses a first-in, first-out (FIFO) strategy to manage data, ensuring that the buffer always contains the latest experience while retaining some historical data to provide diversity. For example, in a task scheduling cycle, after each decision, a four-tuple (current state, action, reward, next state) is added to the end of the buffer; when the buffer is full, the oldest experience is overwritten. This storage method allows the model to learn from past experience, avoiding over-reliance on immediate data and enhancing stability.

[0075] Calculate the temporal difference error (TD error) of the empirical tuple. TD error is a key metric in reinforcement learning, used to evaluate the difference between the current predicted Q-value and the target Q-value. The specific calculation uses the following formula:

[0076]

[0077] in, It is an instant reward signal, and γ is a discount factor (with a value of 0.9 to 0.99). It is the maximum Q-value estimate of the target network in the next state. It evaluates the network's Q-value prediction under the current state and action. and These are the parameters of the target network and the evaluation network, respectively. The TD error reflects the bias in predicting the long-term returns of the current decision; a positive value indicates underestimation, and a negative value indicates overestimation. The calculation process is performed after each empirical tuple is stored, or in batches periodically to maintain real-time performance.

[0078] The sampling priority of empirical tuples is determined based on the timing difference error. The priority is calculated using the following formula:

[0079]

[0080] in, It is the priority of empirical tuple i. It is the absolute value of the TD error. It is a small constant used to prevent the sampling probability from disappearing due to a priority of zero. The higher the priority, the more information the empirical tuple contains, and the more valuable it is for model correction. For example, an empirical tuple with a high TD error may correspond to a state-action pair with a large prediction error in the model, and therefore is given a high priority to be used preferentially for training. After the priority is calculated, it is stored in association with the empirical tuple and dynamically adjusted when the buffer is updated.

[0081] Experience tuples are sampled from the experience replay buffer according to sampling priority. The sampling probability is calculated based on the priority distribution using the formula:

[0082]

[0083] in, Let be the probability that experience tuple i is sampled, and N be the total number of experience tuples in the buffer. The sampling process uses weighted random sampling, where higher-priority experiences have a higher probability of being selected. For example, in a single training iteration, a batch of experiences (typically 32 or 64) is randomly drawn from the buffer, and the sampling probability ensures that the model focuses on samples with learning value. This priority sampling mechanism improves data utilization efficiency, avoids the blindness of uniform sampling, and makes training more targeted at the model's weak points.

[0084] The parameters of the deep reinforcement learning model are updated using sampled empirical tuples. The update process is based on the principle of temporal difference learning, optimizing the evaluation network parameters by minimizing the squared loss of the TD error. The loss function is defined as:

[0085]

[0086] Where B is the size of the sampling batch. This represents the TD error of the i-th sample. Parameter updates employ a gradient descent algorithm, such as the Adam optimizer, calculating the gradient of the loss function with respect to the network parameters and adjusting the weights along the negative gradient direction. For example, with the learning rate set to 0.001, after each update step, the target network parameters are periodically synchronized with the evaluation network (e.g., every 1000 steps) to stabilize training. The update process ensures that the model gradually converges to the optimal policy, adapting to dynamic changes in the marginal environment.

[0087] This implementation improves model learning efficiency through an experience replay mechanism. Priority sampling enables the model to quickly correct erroneous predictions, accelerating the convergence process. Buffer management ensures experience diversity and avoids overfitting. Overall, this method enhances the robustness of the scheduling strategy, achieving more stable load balancing and task processing in edge heterogeneous resource environments.

[0088] Figure 3 This is a schematic flowchart illustrating another edge computing task dynamic scheduling method based on deep reinforcement learning provided by an embodiment of the present invention. In some implementations, updating the deep reinforcement learning model in step S4 employs a double-Q learning mechanism, including the following steps:

[0089] S4.21. Use the evaluation network in the deep reinforcement learning model to select the action with the largest Q value from the environment state at the next time step;

[0090] S4.22. Using the target network in the deep reinforcement learning model, calculate the target Q value corresponding to the selected action;

[0091] S4.23. Based on the target Q value, update the parameters of the evaluation network.

[0092] In the dynamic scheduling method for edge computing tasks based on deep reinforcement learning, the deep reinforcement learning model is updated using a double Q-learning mechanism. This mechanism reduces the problem of Q-value overestimation and improves the stability of the learning process by separating action selection and target value calculation.

[0093] The double-Q learning mechanism relies on two independent neural networks: an evaluation network and a target network. The evaluation network is responsible for online learning and parameter updates, with its parameters adjusted in each training iteration. The target network provides a stable estimate of the target value, with its parameters periodically copied from the evaluation network. The update frequency is typically set to synchronize every certain number of steps, such as once every 1000 training steps. This design makes the calculation of the target value unaffected by frequent parameter changes, reducing fluctuations during the learning process.

[0094] The evaluation network is used to select the action with the largest Q-value from the next-time environment state. Specifically, when the system obtains the next-time environment state s... t+1 Then, this state is input into the evaluation network, which outputs Q-value estimates for all possible actions. Action selection is achieved through the argmax operation, with the formula:

[0095]

[0096] in, It is the action of selection. It is to evaluate the network in state s t+1 Next action The Q-value prediction is performed, where θ is a parameter of the evaluation network. For example, in an edge server cluster, the action space corresponds to the indexes that assign tasks to different servers. The evaluation network calculates the expected reward assigned to each server and selects the server with the highest reward. This step ensures that action selection is based on the optimization direction of the current strategy, but by performing this independently through the evaluation network, it avoids the bias that might be introduced by the target network's involvement in the selection.

[0097] The target network is used to calculate the target Q-value for the selected action. The target Q-value serves as the baseline for model updates, combining immediate reward and discounted future reward. The calculation formula is as follows:

[0098]

[0099] in, It is an immediate reward signal, and γ is a discount factor (usually between 0.9 and 0.99). Is the target network in state? and actions Q-value estimation under the following conditions These are the parameters of the target network. The discount factor γ balances the importance of immediate and future rewards; a larger value indicates that the model focuses more on long-term returns. Target network parameters Maintaining relative stability ensures that the target value does not change rapidly as the network is evaluated, thus providing a consistent learning objective. For example, in task scheduling scenarios, the target Q-value reflects the long-term impact of current decisions, including trade-offs between load balancing and task latency.

[0100] The parameters of the evaluation network are updated based on the target Q-value. The update process is achieved by minimizing the temporal difference error, which is the difference between the target Q-value and the Q-value predicted by the evaluation network. The loss function is defined as the mean squared error:

[0101]

[0102] in, This involves evaluating the network's Q-value prediction under the current state and actions. Parameter optimization employs a gradient descent algorithm, such as the Adam optimizer, to calculate the gradient of the loss function with respect to θ and update the weights along the negative gradient direction. The learning rate is set to a small value (e.g., 0.001) to ensure convergence stability. After updating, the network parameters are progressively adjusted to make the Q-value estimate closer to the target value, thereby improving the scheduling strategy.

[0103] The double-Q learning mechanism demonstrates good adaptability in edge heterogeneous resource environments. By separating action selection and target computation, it reduces the optimism bias in value estimation, enabling the model to more accurately assess the long-term effects of server allocation decisions. For example, in scenarios with large load fluctuations, this mechanism can prevent the model from over-favoring certain servers, thereby improving load balancing. The learning process is smoother, the convergence speed is faster, and the model can quickly adapt to changes in task arrival rates. Overall, double-Q learning enhances the robustness of scheduling strategies, improving system resource utilization while maintaining low latency.

[0104] Figure 4 This is a schematic flowchart of another edge computing task dynamic scheduling method based on deep reinforcement learning provided by an embodiment of the present invention. In some implementations, the RainbowLBO model in S2 adopts a structure that integrates a duel network and a noisy network, including the following processing:

[0105] S2.1 In the adversarial network, the state-action value function of the deep reinforcement learning model is decomposed into a state value function and an advantage function;

[0106] S2.2 Introduce learnable Gaussian noise into the fully connected layer parameters of the dominance function;

[0107] S2.3. Calculate the state-action value function based on the state value function and the advantage function after adding Gaussian noise.

[0108] In the dynamic scheduling method for edge computing tasks based on deep reinforcement learning, the RainbowLBO model adopts a structure that integrates adversarial networks and noisy networks. This design enhances the model's expressive power and exploration efficiency in heterogeneous edge resource environments. This model structure is specifically optimized for task scheduling scenarios and can better handle the challenges brought about by server heterogeneity.

[0109] The Rainbow LBO model first decomposes the state-action value function into two independent branches in the adversarial network part. The state value function branch evaluates the overall value of a given state, regardless of the specific action choice, reflecting the inherent value of the edge server cluster. The advantage function branch evaluates the advantage of each action relative to the average action, which captures the relative value of different server allocation decisions. The specific decomposition formula is as follows:

[0110]

[0111] in, Let V(s) represent the state-action value function, and A(s,a) represent the advantage function. It is the action space, where a' represents the action space. This decomposition method allows the model to learn state value and action advantage separately, improving the accuracy of value estimation. In practice, the two branches share the underlying feature extraction layer, and then each generates its corresponding value output through a fully connected layer.

[0112] The core innovation of noisy networks lies in introducing learnable Gaussian noise into the fully connected layer parameters of the advantage function. The linear layer parameters ω and bias b are calculated as follows:

[0113]

[0114]

[0115] in, and These are the trainable basic parameters. and It is a trainable noise scale parameter. and The noise variable is sampled from a standard normal distribution, and ⊙ denotes element-wise multiplication. The introduction of noise enables the model to adaptively explore during training, rather than relying on a traditional ε-greedy strategy. This mechanism is particularly well-suited for edge computing environments because the heterogeneity of different servers requires varying degrees of exploration intensity.

[0116] The final state-action value function is calculated based on the state value function and the advantage function with added Gaussian noise. The calculation process first obtains the outputs of the state value function and the advantage function through forward propagation, and then fuses them according to the combination formula of the adversarial network. During actual inference, the model receives the environmental state as input, processes it through multiple layers of neural networks, and finally outputs the Q-value for each possible action. These Q-values ​​represent the expected long-term reward of assigning the task to each edge server, and the action corresponding to the largest Q-value is selected as the final decision.

[0117] The entire processing flow demonstrates the model's excellent adaptability to heterogeneous edge environments. The adversarial network structure improves the stability of value estimation, enabling the model to better distinguish the inherent value and relative advantages of different servers. The noisy network mechanism provides an intelligent exploration strategy, reducing unnecessary randomness while ensuring exploration efficiency. This combined design allows the RainbowLBO model to quickly adapt to dynamically changing edge environments and make more accurate task scheduling decisions.

[0118] This implementation improves the model's performance in complex edge environments. Value function decomposition enhances learning efficiency, and noise injection strengthens exploration capabilities, ultimately enabling the model to better balance exploration and exploitation. In practical deployments, this design effectively addresses the challenges posed by server heterogeneity, achieving superior load balancing.

[0119] Figure 5 This is a schematic flowchart illustrating another edge computing task dynamic scheduling method based on deep reinforcement learning provided in an embodiment of the present invention. In some implementations, the specific method for obtaining the reward signal in S3 includes:

[0120] S3.1 Calculate the task response time, where the task response time is the sum of the task waiting time and the task execution time;

[0121] S3.2 Calculate the load imbalance, where the load imbalance is the coefficient of variation of the load of each server in the edge server cluster;

[0122] S3.3. Based on task response time and load imbalance, calculate the reward signal by weighted summation.

[0123] In edge computing task dynamic scheduling methods based on deep reinforcement learning, the calculation of reward signals is a core step in training deep reinforcement learning models, directly affecting the quality of the scheduling strategies learned by the model. Reward signals provide immediate feedback to the model, guiding it to adjust its strategies in a way that optimizes system performance. In edge heterogeneous resource environments, reward signals need to comprehensively consider task execution efficiency and system resource balance to ensure that scheduling decisions can both process tasks quickly and maintain server load balance.

[0124] Calculating task response time is a fundamental component of reward signals. Task response time refers to the total time a task takes from arriving in the system to completing execution, including task waiting time and task execution time. Task waiting time is the time a task waits for resource allocation in the edge server queue, depending on the server's current load and task priority. Task execution time is the actual time a task runs on the server, related to the task's computational load, server processing capacity, and real-time load. The specific calculation uses the following formula:

[0125]

[0126] in, Indicates the task response time. Indicates the task waiting time. This represents the task execution time. Waiting time is obtained by monitoring the time difference between when a task enters the queue and when it begins execution, while execution time is calculated from the time difference between when the task completes and when it begins execution. For example, when a computationally intensive task arrives at an edge server, if the server is currently under high load, the task may need to wait in the queue for other tasks to complete, leading to increased waiting time. Once execution begins, the execution time depends on the server's CPU speed and real-time performance. Accurate calculation of response time helps assess the impact of scheduling decisions on task latency and is a key indicator of latency penalty in reward signals.

[0127] Calculating load imbalance is another key element of the reward signal. Load imbalance measures the uniformity of load distribution across servers in an edge server cluster, using the coefficient of variation (CV) to eliminate the impact of absolute performance differences among servers. The CV is the ratio of the standard deviation of the load to the mean load, reflecting the relative fluctuation of the load. The specific calculation uses the following formula:

[0128]

[0129] Here, LI represents the load imbalance, σ is the standard deviation of the load for each server, and μ is the mean of the load for each server. Server load can be a combined value of CPU utilization, memory utilization, or disk I / O utilization, usually obtained by weighted summation. For example, in a cluster of multiple heterogeneous servers, compute servers may be primarily CPU-loaded, while memory-loaded servers may be primarily memory-loaded, but the coefficient of variation can uniformly measure the load fluctuation of different server types. Load data is collected through real-time monitoring tools, and the load value of each server is updated periodically. A low load imbalance indicates that the load distribution among servers is even and resource utilization is high; a high imbalance indicates that some servers are overloaded while others are idle. By incorporating this indicator into the reward, the model can learn to avoid load imbalance and improve system stability.

[0130] The reward signal is calculated using a weighted summation based on task response time and load imbalance. The weighted summation allows for adjusting the relative importance of different metrics to suit specific application scenarios. The reward function is designed to be negative because the scheduling objective is to minimize response time and load imbalance. The specific formula is:

[0131]

[0132] Task response time is normalized by dividing by the deadline to ensure consistent timescales across different tasks and prevent large values ​​from dominating rewards. Weighting coefficients. and Configure according to system requirements; for example, in latency-sensitive scenarios, it can improve... Weights are used to emphasize response time optimization; in resource-constrained environments, this can improve... Weighted load balancing is performed based on priority. After the reward signal is calculated, it is input into the deep reinforcement learning model to update the Q-value estimate. The model learns and adjusts the influence of the weight coefficients by continuously interacting with the environment, gradually optimizing the scheduling strategy.

[0133] The calculation of the reward signal is performed in real time during system operation. After each task assignment, the system collects task response time and server load data, calculates the current reward, and stores it as part of the empirical data. The reward value is typically controlled within the range of [-1, 0], and normalization is used to ensure training stability. For example, when a task is assigned to a server with low load, the response time may be shorter, the load imbalance may be improved, and the reward value is close to zero, indicating a good decision; conversely, if the task causes server overload or the response time exceeds the limit, the reward value approaches -1, prompting the model to avoid similar decisions. This design allows the reward signal to clearly reflect the scheduling effect and accelerates model convergence.

[0134] This method comprehensively considers task response time and load imbalance, ensuring that the reward signal fully reflects system performance. Task response time optimization guarantees efficient task processing and improves user experience; load imbalance control ensures rational resource utilization and reduces server bottlenecks. The weighted summation mechanism provides flexibility to adapt to the needs of different edge computing scenarios. Overall, the reward signal calculation enhances the model's learning orientation, drives scheduling strategies towards efficiency and balance, and ultimately improves the overall performance and robustness of the edge computing system.

[0135] In some implementations, in S3.3, the calculation of the reward signal is also based on the comparison between the task response time and the task deadline, and the formula for calculating the reward signal is:

[0136]

[0137] in, It is the task response time. LI represents the task deadline and the load imbalance level. , , It is the weighting coefficient.

[0138] This implementation extends the calculation of the reward signal by incorporating a comparison between task response time and task deadline, thereby enhancing the reward function's ability to constrain task timeliness. This improvement allows the reward signal to not only reflect task execution efficiency and load balancing but also directly correlate with task completion time requirements, thus guiding the model to prioritize the scheduling of urgent tasks.

[0139] The calculation formula for the reward signal has been updated to include a third penalty term, as shown above.

[0140] in, It is the task response time, which refers to the total time it takes for a task to go from arriving at the edge server cluster to completion, including waiting time and execution time; LI is the task deadline, which is the maximum time limit that the task must be completed. It is predefined by the task characteristics or user requirements. LI is the load imbalance, which is calculated by the coefficient of variation of the load of each server in the edge server cluster and reflects the uniformity of the load distribution. , , These are weighting coefficients, used to adjust the relative importance of various indicators, and typically satisfy the following conditions: + + =1, the coefficient value can be set according to system preferences, for example, increasing it in scenarios with high real-time requirements. and The weighting. Newly added penalty item. This feature is specifically designed to handle task timeouts. When the task response time does not exceed the deadline, this value is zero, and no penalty is incurred. Once a timeout occurs, the penalty value is proportional to the severity of the timeout and is normalized by dividing by the deadline to ensure a consistent value range.

[0141] When calculating the reward signal, the system first monitors the task response time in real time. The load imbalance score is obtained by recording the task arrival time, start time, and completion time. The load imbalance score (LI) is calculated based on the current load data of each server; the load value can be a weighted average of CPU, memory, and I / O utilization. (Weighting coefficients are also mentioned.) , , The penalty term is set during model initialization and can be fine-tuned during training to adapt to environmental changes. The calculation of the penalty term depends on the task deadline. This parameter is typically part of the task's metadata and is known when the task arrives. For example, a computation task might require completion within 0.5 seconds. If the actual response time is 0.6 seconds, the timeout is 0.1 seconds, and the penalty is 0.1 / 0.5 = 0.2. The reward signal is ultimately obtained through a weighted summation; the negative sign indicates the penalty, and the closer the value is to zero, the better the scheduling performance.

[0142] The inclusion of a deadline violation penalty addresses the high timeliness requirements of tasks in edge computing environments. In heterogeneous resource environments, different tasks may have different deadline constraints; ignoring this factor can lead to increased task failure rates. By penalizing timeout tasks, the reward signal can more accurately reflect the quality of scheduling decisions, allowing the model to avoid selecting server allocation schemes that may cause latency during the exploration process. This mechanism, combined with load imbalance optimization, ensures that the scheduling strategy pursues low latency and load balancing without sacrificing the guarantee of task completion.

[0143] In the process simulation, the calculation of the reward signal is integrated into the feedback loop after each task assignment. Once a task is assigned to an edge server and completed, the system collects response time data and server load status, calculates various metric values, and substitutes them into the formula to generate the reward. The reward value is then used to update the Q-score estimate of the deep reinforcement learning model, guiding subsequent decisions. For example, if the model assigns a task to a heavily loaded but powerful server, potentially exceeding response time limits, the reward signal will provide negative feedback through a penalty term, prompting the model to choose a more conservative assignment strategy in similar future scenarios.

[0144] This implementation effectively reduces task timeout rates, improves system reliability in real-time scenarios, and maintains load balancing, avoiding resource waste caused by over-optimizing a single objective. The model learns the importance of time constraints through reward signals, generating smarter scheduling strategies to adapt to the dynamic changes in the edge environment. Overall, this method improves task success rates and resource utilization efficiency, providing more robust support for edge computing applications.

[0145] In some implementations, in S2.2, the noise scale of the learnable Gaussian noise is adjusted according to the urgency of the task, and the formula for calculating the noise scale is:

[0146]

[0147]

[0148] Where U represents the urgency of the task. , and This is the adjusted noise scale. and It is the basic noise scale. It is the task deadline. It is the moment to make a decision.

[0149] This implementation achieves adaptive control of the exploration strategy by dynamically adjusting the noise scale based on task urgency. This improvement enables the noise network to intelligently adjust the exploration intensity according to task timeliness requirements, ensuring sufficient exploration while avoiding unnecessary randomness, making it particularly suitable for real-time scheduling needs in edge computing environments.

[0150] The core formula for noise scale adjustment is shown above. The urgency level U ranges from [0,1], with a smaller value indicating a more urgent task. As the task approaches its deadline, the U value approaches 0, and the noise scale decreases accordingly; when the task has just arrived at the system, the U value approaches 1, and the noise scale remains relatively large.

[0151] In actual implementation, the system first needs to obtain the task deadline in real time. and the current moment Task deadlines are typically part of the task metadata and are known when the task arrives at the edge server cluster. The current time can be obtained from the system clock, and accurate time synchronization must be ensured. The urgency level U is calculated at each decision point to guarantee the real-time value. For example, if a task needs to be completed within 0.5 seconds, and there are 0.4 seconds left until the deadline, then U = 0.4 / 0.5 = 0.8; if there are only 0.1 seconds left, then U = 0.1 / 0.5 = 0.2.

[0152] Basic noise scale and These are the trainable parameters of the noisy network, randomly set during model initialization and optimized during training using the gradient descent algorithm. These parameters determine the base noise intensity and need to be adjusted according to the specific task scenario. In the early stages of training, a larger base noise scale can be set to enhance exploration; as training progresses and the model gradually converges, the base noise scale will automatically adjust to an appropriate range.

[0153] The noise scaling mechanism directly impacts the effectiveness of the exploration strategy. When the task urgency is high (smaller U-value), the noise scale decreases, making the network output more deterministic. The model tends to utilize existing knowledge to make decisions, avoiding task timeouts due to overexploration. When the task urgency is low (larger U-value), the noise scale increases, enhancing exploration capabilities. The model can try new scheduling strategies and discover better server allocation schemes. This adaptive mechanism balances the conflict between exploration and utilization, enabling the model to maintain good performance in different scenarios.

[0154] At the algorithm implementation level, noise scaling needs to be integrated into the forward propagation process of the neural network. Specifically, when calculating the dominance function, the parameters for each fully connected layer are calculated as follows:

[0155]

[0156]

[0157] Dynamically adjusted noise scale and This allows the noise intensity to adapt to the urgency of the task.

[0158] This noise scaling method based on task urgency offers significant advantages in edge computing task scheduling. First, it improves the reliability of model decisions in urgent task scenarios, ensuring timely task completion by reducing randomness. Second, in non-urgent task scenarios, it maintains sufficient exploratory capabilities, helping to discover new optimization opportunities. Finally, the adaptive mechanism enables the model to better adapt to dynamically changing edge environments, improving overall system performance.

[0159] From a training perspective, this improvement accelerated the model convergence process and reduced fluctuations during training. During the testing phase, the scheduling strategy demonstrated better stability and adaptability, with significant improvements in task success rate and load balancing metrics. Particularly in high-load scenarios, the dynamic noise adjustment mechanism effectively prevented performance degradation due to overexploration, ensuring the quality of system service.

[0160] This implementation method, through intelligent exploration strategy management, enables deep reinforcement learning models to achieve greater efficiency in edge computing task scheduling. Dynamic adjustment of the noise scale not only improves algorithm performance but also enhances the system's adaptability to complex environments, providing a more reliable scheduling solution for edge computing applications.

[0161] In some implementations, the environment state in S1 also includes the number of pending tasks for each edge server in the edge server cluster, and the environment state is represented as follows:

[0162]

[0163] in, This represents the number of pending tasks on each edge server at time t. This indicates the resource requirements of the computational task to be processed. This indicates the idle resources of the edge server. This indicates the normalized real-time processing speed of the edge server. This indicates the load status of the edge server.

[0164] This implementation enhances the model's ability to perceive real-time system congestion by incorporating the number of pending tasks on each edge server in the edge server cluster into the state space. This improvement allows the environment state to not only reflect task resource requirements and the current resource status of servers, but also capture task queue information, providing a more comprehensive context for scheduling decisions.

[0165] The specific formula for environmental conditions is as shown above. Among them, It represents the resource requirements of the computing task to be processed, including the amount of CPU, memory, and disk I / O requested. It represents the idle resources of the edge server cluster at time t, including the CPU, memory, and I / O availability of each server; It is the normalized real-time processing speed of each server, reflecting the relative level of the server's current computing power. It represents the load status of each server, typically calculated based on a weighted average of resource utilization. This represents the number of newly added pending tasks, indicating the number of tasks waiting to be executed in each server's queue at time t. This state representation is organized in vector form, which is convenient for deep reinforcement learning models to process. For example, in a cluster with N servers, the state vector may contain task requirements (such as the number of CPU cores and memory size), an N-dimensional vector of idle resources, an N-dimensional vector of processing speed, an N-dimensional vector of load, and an N-dimensional vector of queue length.

[0166] Number of tasks to be processed The acquisition is based on the queue management mechanism in the task scheduling system model. In an edge computing environment, each edge server maintains a task queue to store assigned but not yet executed tasks. The system monitors the queue status in real time to count the number of pending tasks for each server. Specifically, when a task is assigned to a server, it enters that server's waiting queue; when the server begins executing the task, the task is removed from the queue. The value is obtained by periodically querying the queue length, with the update frequency matching the task arrival rate to ensure data real-time performance. For example, under high load, the system may update the queue length once per second, while the update interval can be appropriately extended under low load. This design allows the state to accurately reflect the system's congestion level, providing dynamic information for the model.

[0167] The expansion of the environmental state stems from a deep analysis of the scheduling complexity in edge heterogeneous resource environments. Traditional state spaces only focus on instantaneous resource states, ignoring the cumulative effect of tasks, which may lead to the model's inability to predict queuing delays. By incorporating the number of pending tasks, the state space can more comprehensively characterize system behavior, enabling the model to consider not only current resource availability but also future server load trends when making decisions. For example, a server may have high idle resources, but if its queue is long, new task allocation may cause a surge in waiting time; while another server with limited resources but a short queue may be more suitable for handling urgent tasks. This multidimensional state representation enhances the model's adaptability to the dynamic characteristics of edge environments.

[0168] In the process implementation, the construction of the environment state is integrated into every scheduling decision. When a new task arrives, the system collects the task's resource requirements. And obtain real-time data from each server through a monitoring agent: idle resources Reading from File Explorer, processing speed Based on server performance benchmarks and current load calculations, the load status... The number of tasks to be processed is obtained through the weighted resource utilization formula. Extracted from the queue management module. All data is combined into a state vector. This serves as input to deep reinforcement learning models. The state update process must ensure low latency to avoid outdated information affecting decision accuracy. For example, in distributed edge systems, state collection might be accomplished through lightweight communication protocols to ensure efficiency.

[0169] This approach improves the predictability of scheduling decisions, enabling the model to avoid assigning tasks to servers with excessively long queues, thereby reducing task waiting time. Secondly, by reflecting system congestion, it helps to more accurately optimize load balancing, preventing some servers from becoming overloaded due to queue accumulation. Finally, rich state information enhances the model's robustness in dynamic environments, allowing it to adapt to fluctuations in task arrival rates. Overall, this method provides a more intelligent solution for edge computing task scheduling by enhancing environmental awareness.

[0170] In some implementations, in S4.22, the target Q value is calculated using an N-step reward mechanism, and the formula for calculating the target Q value is:

[0171]

[0172] Where N is the number of steps, and γ is the discount factor. It is the reward signal for the k-th step in the future. The target network in the environmental state The target Q value estimation is as follows. These are the target network parameters. It is the selected action.

[0173] This implementation introduces an N-step reward mechanism, enabling the model to better balance immediate and future rewards, thus improving the accuracy of long-term decision-making. The formula for calculating the target Q value is as shown above.

[0174] in, γ represents the target Q value, which is the baseline for model updates; N is the step parameter, a positive integer that determines the range of steps considered for future rewards. It is usually set according to the characteristics of the task, such as 3 to 5 steps in edge computing to balance computational complexity and forward-looking nature; γ is the discount factor, ranging from 0.9 to 0.99, used to adjust the present value of future rewards. The larger the value, the more the model focuses on long-term returns. It is the reward signal for the k-th step in the future, calculated based on task response time and load imbalance. Is the target network in state? The maximum Q-value estimate for all possible actions is given below. These are the parameters of the target network. The first term of the formula, the summation part, represents the cumulative reward with discounts over the next N steps. The second term is the state value estimate after N steps, which is weighted by a discount factor so that the target value includes both recent feedback and long-term impact.

[0175] The N-step reward mechanism is an extension of traditional temporal difference learning. In edge heterogeneous resource environments, the impact of task scheduling decisions is often delayed; for example, assigning a task to a server may lead to subsequent load accumulation, and single-step rewards cannot capture this multi-step correlation. By introducing N-step rewards, the model can consider a longer reward sequence during updates, reducing estimation bias. Specifically, the choice of the step number parameter N requires a trade-off: a smaller N value (such as 1 or 2) is computationally simple but has a shorter field of view; a larger N value can cover longer-term effects but increases computational overhead and estimation variance. In implementation, the N value is usually fine-tuned experimentally to adapt to the dynamics of the edge environment. The discount factor γ controls the decay rate of future rewards; a smaller value can be set in scenarios with high real-time requirements to emphasize immediate gains, while a larger value can be set in resource optimization scenarios to encourage long-term equilibrium.

[0176] After the system executes an action and the environment transitions, the experience tuple is stored, but the target value calculation is no longer limited to single-step rewards. The specific steps include: first, sampling an experience from the experience replay buffer to obtain the current state. ,action ,award and the next state Then, based on the current strategy or experience trajectory, simulate or backtrack the reward sequence for the next N steps, for example, by obtaining r through model prediction or actual interaction records. t+1 ,r t+2 ,…,r t+N-1 Next, the target network is used to calculate the state s. t+NThe maximum Q-value is calculated; finally, the target Q-value is calculated by substituting the values ​​into the formula. For example, in a task scheduling decision, if N=3, the model will consider the rewards of the next three steps (such as load changes and latency effects after task execution) and combine them with the state value after the third step to form a comprehensive objective. This calculation requires an environmental model or sufficient historical data support. In online learning scenarios, this can be achieved through multi-step experience replay, i.e., storing continuous multi-step interaction sequences.

[0177] The implementation of the N-step reward mechanism also involves collaboration with the double-Q learning mechanism. In the double-Q learning framework, action selection is performed by the evaluation network, while target value calculation is handled by the target network. With the integration of N-step rewards, target value calculation not only relies on the target network's estimation after a single step but also extends to multiple steps. For example, during updates, the evaluation network is used to select actions for future states (such as...). However, the Q-value portion of the objective value is still calculated by the target network (e.g., This combination ensures stability. It reduces overestimation problems and improves the accuracy of values ​​over a long-term perspective.

[0178] This approach improves the model's adaptability to dynamic edge environments. Through multi-step reward balancing, the scheduling strategy is better able to anticipate load fluctuations and task queue changes, reducing short-sighted decision-making. Secondly, the target value estimation is more stable, and the convergence process is smoother because N-step rewards reduce the noise impact of single-step rewards. Finally, in the trade-off between load balancing and task latency, the model can better optimize long-term resource utilization and improve overall system performance.

[0179] In some implementations, when updating the parameters of the deep reinforcement learning model using sampled empirical tuples in S4.15, the following steps are also included:

[0180] S4.15.1 Calculate importance sampling weights based on sampling priority;

[0181] S4.15.2, Correct gradient updates using importance sampling weights;

[0182] Importance sampling weight The calculation formula is:

[0183]

[0184] Where A is the size of the experience playback buffer. It is the sampling probability of empirical tuple i, and β is the importance sampling correction parameter, which is adjusted from the initial value during training. It increases linearly to 1.0.

[0185] By introducing importance sampling weights to correct biases in the gradient update process, the stability and convergence efficiency of model training are improved. This mechanism is an important component of priority experience replay, specifically designed to address estimation bias caused by non-uniform sampling.

[0186] The formula for calculating the importance sampling weights is as shown above. Wherein, The importance sampling weight represents the importance of empirical tuple i. N is the size of the empirical replay buffer, i.e., the maximum number of empirical tuples that the buffer can store. P(i) is the sampling probability of empirical tuple i, calculated based on the priority of temporal difference error (TD error). β is the importance sampling correction parameter, used to control the strength of weight correction, which is usually gradually increased from an initial value (e.g., 0.5) to 1.0 as training progresses. The core function of the importance sampling weight is to adjust the contribution of each sample in gradient update, so that the update amplitude of high-priority samples is appropriately suppressed, thereby maintaining the unbiasedness of training.

[0187] In practice, the calculation of importance sampling weights is integrated into the sampling and updating process of the experience replay buffer. When the system needs to sample a batch of experiences from the buffer for training, it first calculates the sampling probability P(i) based on the priority of each experience, which is proportional to the priority. Subsequently, for each sampled experience tuple i, the system calculates its importance sampling weight in real time. Of the parameters required for calculation, the buffer size N is a fixed value set during system initialization; the sampling probability P(i) is dynamically calculated with each sampling; and the correction parameter β is linearly adjusted with the number of training steps, for example, gradually increasing from 0.5 at step 0 to 1.0 at step 10000. This design causes the weight correction strength to gradually weaken as the training progresses, emphasizing stability in the early stages of training and pursuing efficiency in the later stages.

[0188] The application of importance sampling weights is seen in the gradient update phase. When calculating the loss function, the loss for each sample is multiplied by its importance sampling weight, resulting in the corrected loss function:

[0189]

[0190] Where B is the batch size. This is the TD error of empirical tuple i. By introducing... The gradient update magnitude of high-probability samples (i.e., high-priority samples) is appropriately reduced, while the update magnitude of low-probability samples is increased, thereby offsetting the bias caused by non-uniform sampling. For example, a high-priority empirical tuple may be sampled frequently, but its importance sampling weight is small, so that its influence on gradient updates is not too large; conversely, rare but important samples receive sufficient attention through larger weights.

[0191] The implementation of this mechanism requires coordination with other components that prioritize experience replay. Regarding buffer management, the system needs to maintain the priority and latest sampling probability of each experience. During training, sampling, weight calculation, and gradient updates occur: after each sampling, the sampling probability record is updated, weights are calculated and used for gradient updates, and priorities are adjusted based on the new TD error. This approach ensures the continuity and consistency of the training process.

[0192] The introduction of importance sampling weights significantly improves model training quality. First, it effectively prevents overfitting to high-priority samples, allowing the model to maintain its ability to learn from diverse experiences. Second, through bias correction, it improves the accuracy of parameter estimation and accelerates model convergence. Experiments show that after using importance sampling weights, the fluctuations in the training process are significantly reduced, and the policy performance is more stable. In edge computing task scheduling scenarios, this means that the model can adapt to environmental changes more quickly and generate more reliable scheduling policies.

[0193] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. A dynamic scheduling method for edge computing tasks based on deep reinforcement learning, applied to edge heterogeneous resource environments, characterized in that, Includes the following steps: S1. Obtain the current environmental status, which includes the resource requirements of the computing task to be processed, the idle resources of each edge server in the edge server cluster, the normalized real-time processing speed of each edge server, and the load status of each edge server. S2. Based on the environmental state, a deep reinforcement learning model is used to select an action, wherein the action is to assign the computing task to be processed to an edge server in the edge server cluster, wherein the deep reinforcement learning model is a RainbowLBO model that integrates a duel network and a noisy network. S3. Perform the action to assign the computing task to be processed to the selected edge server and obtain a reward signal. The reward signal is calculated based on the task response time and load imbalance. The load imbalance is the coefficient of variation of the load of each server in the edge server cluster, which is used to measure the uniformity of load distribution. The calculation method is the ratio of the load standard deviation to the load mean. S4. Update the deep reinforcement learning model using the environmental state, the action, the reward signal, and the environmental state at the next time step.

2. The method according to claim 1, characterized in that, Updating the deep reinforcement learning model in step S4 includes the following steps: S4.

11. Store the experience tuple into the experience replay buffer, wherein the experience tuple includes the environment state, the action, the reward signal, and the environment state at the next moment; S4.12 Calculate the temporal difference error of the empirical tuple; S4.

13. Determine the sampling priority of the empirical tuple based on the time-series differential error; S4.

14. Sample experience tuples from the experience playback buffer according to the sampling priority; S4.

15. Update the parameters of the deep reinforcement learning model using the sampled empirical tuples.

3. The method according to claim 1, characterized in that, The deep reinforcement learning model is updated in S4 using a double-Q learning mechanism, including the following steps: S4.

21. Using the evaluation network in the deep reinforcement learning model, select the action with the largest Q value from the next time-step environment state; S4.

22. Using the target network in the deep reinforcement learning model, calculate the target Q value corresponding to the selected action; S4.

23. Update the parameters of the evaluation network based on the target Q value.

4. The method according to claim 1, characterized in that, The RainbowLBO model in S2 adopts a structure that integrates a duel network and a noisy network, including the following processing: S2.1 In the adversarial network, the state-action value function of the deep reinforcement learning model is decomposed into a state value function and an advantage function; S2.2 Introduce learnable Gaussian noise into the fully connected layer parameters of the dominant function; S2.

3. Based on the state value function and the advantage function after adding Gaussian noise, calculate the state-action value function value.

5. The method according to claim 1, characterized in that, The specific methods for obtaining the reward signal in S3 include: S3.1 Calculate the task response time, wherein the task response time is the sum of the task waiting time and the task execution time; S3.2 Calculate the load imbalance, where the load imbalance is the coefficient of variation of the load of each server in the edge server cluster; S3.

3. Based on the task response time and the load imbalance, calculate the reward signal by weighted summation.

6. The method according to claim 5, characterized in that, In step S3.3, the calculation of the reward signal is further based on the comparison between the task response time and the task deadline, and the calculation formula for the reward signal is as follows: in, It is the task response time. LI represents the task deadline and the load imbalance level. , , It is the weighting coefficient.

7. The method according to claim 4, characterized in that, In step S2.2, the noise scale of the learnable Gaussian noise is adjusted according to the urgency of the task, and the calculation formula for the noise scale is as follows: Where U represents the urgency of the task. , and This is the adjusted noise scale. and It is the basic noise scale. It is the task deadline. It is the moment to make a decision.

8. The method according to claim 1, characterized in that, The environmental state in S1 also includes the number of pending tasks on each edge server in the edge server cluster, and the environmental state is represented as follows: in, This represents the number of pending tasks on each edge server at time t. This indicates the resource requirements of the computational task to be processed. This indicates the idle resources of the edge server. This indicates the normalized real-time processing speed of the edge server. This indicates the load status of the edge server.

9. The method according to claim 3, characterized in that, In S4.22, the target Q value is calculated using an N-step reward mechanism, and the formula for calculating the target Q value is as follows: Where N is the number of steps, and γ is the discount factor. It is the reward signal for the k-th step in the future. The target network in the environmental state The target Q value estimation is as follows. These are the target network parameters. It is the selected action.

10. The method according to claim 2, characterized in that, When updating the parameters of the deep reinforcement learning model using sampled empirical tuples in S4.15, the following steps are also included: S4.15.1 Calculate the importance sampling weights based on the sampling priority; S4.15.2, Correct the gradient update using the aforementioned importance sampling weights; Importance sampling weight The calculation formula is: Where A is the size of the experience playback buffer. It is the sampling probability of empirical tuple i, and β is the importance sampling correction parameter, which is adjusted from the initial value during training. It increases linearly to 1.0.

Images