An edge side task scheduling method and system based on heuristic reinforcement learning

Abstract

The application discloses an edge side task scheduling method and system based on heuristic reinforcement learning. The method comprises the following steps: collecting tasks to be scheduled in a queue and system running state features, and constructing a state vector for scheduling decision; calculating a heuristic score of each candidate task; calculating a reinforcement learning Q value of each candidate scheduling action based on a deep Q network; calculating a final scheduling score; selecting a task with the maximum final scheduling score as a current scheduling task for execution, and recording task result information; calculating a reward R according to the task result information, and updating deep Q network parameters according to the reward R. The method combines heuristic rules and reinforcement learning, improves the efficiency and stability of scheduling decisions, and realizes multi-objective collaborative optimization of task priority, waiting time, resource efficiency and timeliness. Meanwhile, the scheduling method has the parameter self-adaptive adjustment capability, so that the system can dynamically respond to the edge environment changes.

Description

Technical Field

[0001] This invention belongs to the field of smart grid control, specifically relating to an edge-side task scheduling method and system based on heuristic reinforcement learning. Background Technology

[0002] In the edge computing environment of power systems, task scheduling is crucial for ensuring system real-time performance and resource efficiency. Traditional scheduling methods, such as First-Come, First-Served (FCFS), Shortest Job First (SJF), and priority scheduling, are simple to implement but lack dynamic adaptability and are difficult to cope with the suddenness and heterogeneity of tasks on the edge side.

[0003] In recent years, reinforcement learning (RL) has been introduced into the field of task scheduling. The most representative method is the scheduling strategy based on deep Q-networks (DQNs), which, by constructing a state space and reward function, enables the system to learn optimal decisions from historical scheduling experience. While this type of method improves the adaptability of scheduling to some extent, its purely data-driven nature leads to low exploration efficiency and a prominent cold-start problem in the early stages of training. Furthermore, it fails to effectively integrate domain knowledge (such as task priority and resource efficiency) and lacks explicit modeling of key factors such as task priority, waiting time, and resource efficiency, resulting in limited scheduling effectiveness in complex power business scenarios. In addition, reinforcement learning-based scheduling methods, due to their fixed parameters, cannot dynamically adjust according to the system's operating state, thus limiting their adaptability. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide an edge task scheduling method based on heuristic reinforcement learning. By integrating heuristic rules and reinforcement learning, the method improves the efficiency and stability of scheduling decisions, and achieves multi-objective collaborative optimization of task priority, waiting time, resource efficiency, and timeliness. At the same time, the scheduling method has the ability to adaptively adjust parameters, enabling the system to dynamically respond to changes in the edge environment.

[0005] The second objective of this invention is to provide a system for implementing the edge task scheduling method based on heuristic reinforcement learning.

[0006] This invention provides an edge task scheduling method based on heuristic reinforcement learning, comprising the following steps:

[0007] Collect the features of tasks to be scheduled in the queue and the system's operating status, and construct a state vector for scheduling decisions;

[0008] Calculate a heuristic score for each candidate task; the heuristic score includes a priority inversion compensation term, a logarithmic waiting time compensation term, and a resource efficiency factor;

[0009] Based on a deep Q-network, the reinforcement learning Q-value for each candidate scheduling action is calculated;

[0010] The final scheduling score is calculated based on heuristic scoring and reinforcement learning Q-value.

[0011] Based on the final scheduling score, all candidate tasks are sorted from largest to smallest, the task with the largest score is selected as the current scheduling task for execution, and the task result information is recorded.

[0012] Calculate the reward R based on the task result information, and update the parameters of the depth Q network based on the reward R;

[0013] After the preset task batch is completed, the parameters are adaptively adjusted based on the performance metrics of the batch of tasks.

[0014] The characteristics of the tasks to be scheduled and the system operating status include task load, power business task information, edge computing resource usage, and task timeliness information; the task load includes queue load rate, total queue resource requirement, average single task resource requirement, and system CPU utilization; the power business task information includes the priority, waiting time, normalized resource requirement, and remaining time of the current candidate task; the edge computing resource usage includes CPU demand distribution statistics, memory demand distribution statistics, I / O demand distribution statistics, and resource overrun flags; the task timeliness information includes the average remaining time of the task, the proportion of urgent tasks with remaining time below the threshold, the proportion of timed-out tasks, and the average queue waiting time;

[0015] The state vector is a 16-dimensional vector, including 0-11 dimensional system-level features and 12-15 dimensional task-level features; the system-level features include 0-3 dimensional load features, 4-7 dimensional resource features, and 8-11 dimensional timeliness features.

[0016] The 4-dimensional load characteristics include queue load rate, total queue resource requirements, average single task resource requirements, and system CPU utilization; the 4-dimensional resource characteristics include CPU demand distribution statistics, memory demand distribution statistics, I / O demand distribution statistics, and resource over-limit flags; the 4-dimensional timeliness characteristics include average remaining time of tasks, the proportion of urgent tasks with remaining time below the threshold, the proportion of timed-out tasks, and average queue waiting time; the 4-dimensional task-level characteristics include the priority of the current candidate task, waiting time, normalized resource requirements, and remaining time.

[0017] The state vector uses 16-dimensional features to encode the system's operating state, thereby improving the deep Q-network's ability to perceive global queue pressure and task distribution characteristics.

[0018] The heuristic scoring H includes a priority inversion compensation term. Logarithmic compensation term for waiting time and resource efficiency factor It can be expressed using the following formula: The priority inversion compensation term Calculate using the following formula: ;in, As the basic priority of the task; The priority weight parameter; the logarithmic compensation term for the waiting time Calculate using the following formula: Where WH represents the task waiting time, in hours; The waiting time weight parameter; the resource efficiency factor Calculate using the following formula: Where RN represents the task normalization resource requirement; The resource discount factor is used; the normalized resource requirement RN of the task is obtained by normalizing the three types of resources, namely CPU, memory and I / O, and then performing a weighted summation, wherein the weights are manually preset based on historical operation statistics.

[0019] The state vector is input into the deep Q-network, which outputs the Q-value for each candidate scheduling action. The deep Q-network includes a policy network and a target network. Both the policy network and the target network use a 3-layer fully connected neural network, with 16 nodes in the input layer, 64 nodes in the hidden layer, and N nodes in the output layer, where N is the number of candidate tasks. The policy network and the target network have the same neural network structure but different parameters, and therefore different functions. The policy network is used to output the Q-value of each candidate task in real time and is used for online decision-making and experience generation. The target network is used to generate a stable Q-value estimate in the TD target value y, and only participates in the target calculation during the training process, without participating in the actual scheduling decision. The policy network uses... - A greedy strategy is used for searching, where the exploration rate is... The exponential decay occurs as training progresses, as expressed by the following formula: ;in, This is the initial exploration rate; The minimum exploration rate is given by k; the decay coefficient is given by t; and the number of iterations in the training process is given by t.

[0020] A soft update mechanism is used to update the target network parameters, fusing historical parameters with new parameters according to a preset ratio, expressed by the following formula: ;in, To update the coefficients, control the update step size; For target network parameters; These are the parameters of the policy network; soft updates ensure training stability by slowly copying the parameters of the policy network to the target network.

[0021] Based on heuristic scoring and reinforcement learning Q-values, the final scheduling score S is calculated using the following formula: Where Q is the Q-value of the candidate scheduling action corresponding to the task; H is the heuristic score; Let be the decision weight parameters, and .

[0022] The decision weight parameters An online adaptive adjustment mechanism is adopted. When the queue load rate is greater than 80% and the CPU utilization is greater than 90%, the system is determined to be in a high-load state, and the decision weight parameters are increased. Preset the step size and re-detect until the system is no longer under high load or the decision weight parameters are adjusted. This is the preset maximum value.

[0023] In each scheduling cycle, S is calculated for all candidate tasks. The task with the largest S is selected as the current scheduled task and put into execution. The execution start time, execution end time, whether timeout occurs, and actual resource usage are recorded.

[0024] The reward R includes positive reward items related to task priority, positive reward items related to resource efficiency, and negative reward items related to timeout penalties; the positive reward items related to task priority are... Positive rewards related to resource efficiency are: The negative rewards associated with timeout penalties are: Where TP is a segmented penalty coefficient based on the task timeout duration. The segmented penalty coefficient TP is based on the timeout task duration. Define segments: when When TP is 1, its value is 1; when When TP is 2, its value is 2; when When TP is 3, the value of TP is 3; where, This is the first timeout threshold; This is the second timeout threshold.

[0025] A temporal difference learning mechanism is adopted to update the parameters of the deep Q network according to the reward R. Specifically, this includes calculating the temporal difference TD target value y using the following formula:

[0026] ;in, This indicates the subsequent state after the current task is executed. Indicates the state in the successor state The candidate scheduling action variables are used to perform a Q-value maximization search across all possible action sets. This is the discount factor.

[0027] The timing difference (TD) error L is calculated using the following formula: ;in, s represents the Q-value prediction of the policy network for the current state-action pair; s represents the current system state, which is a state vector composed of multi-dimensional features such as queue load, CPU / memory / I / O resource utilization, task priority, waiting time, and remaining time limit; a represents the candidate scheduling action under the state s, which represents the specific task decision to be executed from the current set of tasks to be scheduled.

[0028] The policy network parameters are updated by minimizing the TD error through gradient descent. .

[0029] After the preset task batch is completed, the parameters are adjusted based on the statistically obtained performance indicators such as on-time completion rate, CPU utilization, and high-priority task response time. Adaptive adjustments are made, with parameter adjustments using a combination of grid search and performance evaluation, to ensure that the system performance indicators reach the preset optimization goals.

[0030] The adaptive parameter adjustment includes: 1) generating parameter combinations within a preset range; 2) evaluating the comprehensive score of each combination within a sliding window; 3) selecting the parameter combination with the highest score and applying it to the next stage of scheduling; the sliding window includes the most recent M batches; in the adaptive parameter adjustment step: priority weight Search within the range [0.8, 1.2]; waiting time weight. Search within the range [0.4, 0.8]; resource efficiency discount factor Search within the range [0.1, 0.49]; decision weights Search within the range [0.3, 0.7]; and use the on-time completion rate as the benchmark. Maximize CPU utilization Maximize the average response time of high-priority tasks The comprehensive objective scoring function F is minimized to achieve joint optimization of multi-objective performance; the comprehensive objective scoring function F is expressed by the following formula: ;in Weighted by on-time completion rate; As a weight for CPU utilization; Weight the average response time of high-priority tasks;

[0031] The parameter adjustment trigger condition after each batch of tasks is completed is as follows: when the cumulative number of completed tasks reaches N, a parameter search and evaluation is triggered once, where N is preset in the range of [50, 200]; during the parameter evaluation process, the performance indicators of the most recent M batches of tasks are statistically analyzed in a sliding window, where M is in the range of [3, 10].

[0032] The present invention also provides a system for implementing the aforementioned edge task scheduling method based on heuristic reinforcement learning, comprising:

[0033] Task feature input module: used to collect multi-dimensional features of tasks to be scheduled in the queue and the system running status in real time, and construct a 16-dimensional state vector S;

[0034] Heuristic decision-making module: used to calculate the heuristic score H of each candidate task according to the preset heuristic rules, including at least the priority inversion compensation term, the logarithmic compensation term of the waiting time and the resource efficiency factor;

[0035] The reinforcement learning engine module includes a policy network and a target network, which are used to calculate the Q-values ​​of each candidate task based on the state vector S, and train the network parameters using an experience replay and soft update mechanism.

[0036] Strategy fusion module: performs weighted fusion of Q-value and heuristic score H to obtain the final scheduling score of candidate tasks;

[0037] Task execution and monitoring module: used to select and execute target tasks based on the final scheduling score, while monitoring resource usage, execution time and timeout during task execution;

[0038] Reward calculation module: used to calculate reward R based on task execution results and feed reward R back to reinforcement learning engine module;

[0039] Parameter self-optimization module: used to optimize parameters based on system statistical performance indicators. The parameters are adaptively adjusted.

[0040] This invention discloses an edge task scheduling method and system based on heuristic reinforcement learning, which can quickly respond to high real-time tasks in edge computing scenarios with multiple concurrent tasks and significant load fluctuations. Through state representation and online parameter update mechanism, it can improve the overall task completion rate, system throughput and resource utilization. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the method in an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of the system structure in an embodiment of the present invention;

[0043] Figure 3 This is a comparison chart of simulation test performance in an embodiment of the present invention. Detailed Implementation

[0044] This invention provides an edge task scheduling method based on heuristic reinforcement learning, the flowchart of which is shown below. Figure 1 As shown, it includes the following steps:

[0045] Collect the features of tasks to be scheduled in the queue and the system's operating status, and construct a state vector for scheduling decisions;

[0046] Calculate a heuristic score for each candidate task; the heuristic score includes a priority inversion compensation term, a logarithmic waiting time compensation term, and a resource efficiency factor;

[0047] Based on a deep Q-network, the reinforcement learning Q-value for each candidate scheduling action is calculated;

[0048] The final scheduling score is calculated based on heuristic scoring and reinforcement learning Q-value.

[0049] Based on the final scheduling score, all candidate tasks are sorted from largest to smallest, the task with the largest score is selected as the current scheduling task for execution, and the task result information is recorded.

[0050] Calculate the reward R based on the task result information, and update the parameters of the depth Q network based on the reward R;

[0051] After the preset task batch is completed, the parameters are adaptively adjusted based on the performance metrics of the batch of tasks.

[0052] The characteristics of the tasks to be scheduled and the system operating status include task load, power business task information, edge computing resource usage, and task timeliness information; the task load includes queue load rate, total queue resource requirement, average single task resource requirement, and system CPU utilization; the power business task information includes the priority, waiting time, normalized resource requirement, and remaining time of the current candidate task; the edge computing resource usage includes CPU demand distribution statistics, memory demand distribution statistics, I / O demand distribution statistics, and resource overrun flags; the task timeliness information includes the average remaining time of the task, the proportion of urgent tasks with remaining time below the threshold, the proportion of timed-out tasks, and the average queue waiting time;

[0053] The state vector is a 16-dimensional vector, including 0-11 dimensional system-level features and 12-15 dimensional task-level features; the system-level features include 0-3 dimensional load features, 4-7 dimensional resource features, and 8-11 dimensional timeliness features.

[0054] The 4-dimensional load characteristics include queue load rate, total queue resource requirements, average single task resource requirements, and system CPU utilization; the 4-dimensional resource characteristics include CPU demand distribution statistics, memory demand distribution statistics, I / O demand distribution statistics, and resource over-limit flags; the 4-dimensional timeliness characteristics include average remaining time of tasks, the proportion of urgent tasks with remaining time below the threshold, the proportion of timed-out tasks, and average queue waiting time; the 4-dimensional task-level characteristics include the priority of the current candidate task, waiting time, normalized resource requirements, and remaining time.

[0055] The state vector uses 16-dimensional features to encode the system's operating state, thereby improving the deep Q-network's ability to perceive global queue pressure and task distribution characteristics.

[0056] The heuristic scoring H includes a priority inversion compensation term. Logarithmic compensation term for waiting time and resource efficiency factor It can be expressed using the following formula: The priority inversion compensation term Calculate using the following formula: ;in, As the basic priority of the task; The priority weight parameter; the logarithmic compensation term for the waiting time Calculate using the following formula: Where WH represents the task waiting time, in hours; The waiting time weight parameter; the resource efficiency factor Calculate using the following formula: Where RN represents the task normalization resource requirement; The resource discount factor is used; the normalized resource requirement RN of the task is obtained by normalizing the three types of resources, namely CPU, memory and I / O, and then performing a weighted summation, wherein the weights are manually preset based on historical operation statistics.

[0057] The state vector is input into the deep Q-network, which outputs the Q-value for each candidate scheduling action. The deep Q-network includes a policy network and a target network. Both the policy network and the target network use a 3-layer fully connected neural network, with 16 nodes in the input layer, 64 nodes in the hidden layer, and N nodes in the output layer, where N is the number of candidate tasks. The policy network and the target network have the same neural network structure but different parameters, and therefore different functions. The policy network is used to output the Q-value of each candidate task in real time and is used for online decision-making and experience generation. The target network is used to generate a stable Q-value estimate in the TD target value y, and only participates in the target calculation during the training process, without participating in the actual scheduling decision. The policy network uses... - A greedy strategy is used for searching, where the exploration rate is... The exponential decay occurs as training progresses, as expressed by the following formula: ;in, This is the initial exploration rate; The minimum exploration rate is given by k; the decay coefficient is given by t; and the number of iterations in the training process is given by t.

[0058] A soft update mechanism is used to update the target network parameters, fusing historical parameters with new parameters according to a preset ratio, expressed by the following formula: ;in, To update the coefficients, control the update step size; For target network parameters; These are the parameters of the policy network; soft updates ensure training stability by slowly copying the parameters of the policy network to the target network.

[0059] Specifically, the policy network is responsible for real-time decision-making. It takes the state S as input and outputs the Q-value of each candidate action. Its output is used to select actions (which task to schedule), interact with the environment, and generate experience samples. It is the network that truly "makes decisions".

[0060] The target network is responsible for providing a stable training objective. Its parameters are "delayed copies" of the policy network, used to calculate the TD objective value y. It does not participate in action selection; it is only used to calculate the learning objective. After introducing the target network, the TD objective becomes more stable, convergence is smoother, and training is more reliable.

[0061] Based on heuristic scoring and reinforcement learning Q-values, the final scheduling score S is calculated using the following formula: Where Q is the Q-value of the candidate scheduling action corresponding to the task; H is the heuristic score; Let be the decision weight parameters, and .

[0062] The decision weight parameters An online adaptive adjustment mechanism is adopted. When the queue load rate is greater than 80% and the CPU utilization is greater than 90%, the system is determined to be in a high-load state, and the decision weight parameters are increased. Preset the step size and re-detect until the system is no longer under high load or the decision weight parameters are adjusted. This is the preset maximum value.

[0063] In each scheduling cycle, S is calculated for all candidate tasks. The task with the largest S is selected as the current scheduled task and put into execution. The execution start time, execution end time, whether timeout occurs, and actual resource usage are recorded.

[0064] The reward R includes positive reward items related to task priority, positive reward items related to resource efficiency, and negative reward items related to timeout penalties; the positive reward items related to task priority are... Positive rewards related to resource efficiency are: The negative rewards associated with timeout penalties are: Where TP is a segmented penalty coefficient based on the task timeout duration. The segmented penalty coefficient TP is based on the timeout task duration. Define segments: when When TP is 1, its value is 1; when When TP is 2, its value is 2; when When TP is 3, the value of TP is 3; where, This is the first timeout threshold; This is the second timeout threshold.

[0065] A temporal difference learning mechanism is adopted to update the parameters of the deep Q-network based on the reward R, specifically including:

[0066] The target value y of the time-difference time (TD) is calculated using the following formula:

[0067] ;in, This indicates the subsequent state after the current task is executed. Indicates the state in the successor state The candidate scheduling action variables are used to perform a Q-value maximization search across all possible action sets. This is the discount factor.

[0068] The timing difference (TD) error L is calculated using the following formula: ;in, s represents the Q-value prediction of the policy network for the current state-action pair; s represents the current system state, which is a state vector composed of multi-dimensional features such as queue load, CPU / memory / I / O resource utilization, task priority, waiting time, and remaining time limit; a represents the candidate scheduling action under the state s, which represents the specific task decision to be executed from the current set of tasks to be scheduled.

[0069] The policy network parameters are updated by minimizing the TD error through gradient descent. .

[0070] After the preset task batch is completed, based on the statistically obtained performance indicators such as on-time completion rate, CPU utilization, and high-priority task response time, the parameters are adjusted. Adaptive adjustments are made, with parameter adjustments using a combination of grid search and performance evaluation, to ensure that the system performance indicators reach the preset optimization goals.

[0071] The adaptive parameter adjustment includes: 1) generating parameter combinations within a preset range; 2) evaluating the comprehensive score of each combination within a sliding window; 3) selecting the parameter combination with the highest score and applying it to the next stage of scheduling; the sliding window includes the most recent M batches; in the adaptive parameter adjustment step: priority weight Search within the range [0.8, 1.2]; waiting time weight. Search within the range [0.4, 0.8]; resource efficiency discount factor Search within the range [0.1, 0.49]; decision weights Search within the range [0.3, 0.7]; and use the on-time completion rate as the benchmark. Maximize CPU utilization Maximize the average response time of high-priority tasks The comprehensive objective scoring function F is minimized to achieve joint optimization of multi-objective performance; the comprehensive objective scoring function F is expressed by the following formula: ;in Weighted by on-time completion rate; As a weight for CPU utilization; Weight the average response time of high-priority tasks;

[0072] The parameter adjustment trigger condition after each batch of tasks is completed is as follows: when the cumulative number of completed tasks reaches N, a parameter search and evaluation is triggered once, where N is preset in the range of [50, 200]; during the parameter evaluation process, the performance indicators of the most recent M batches of tasks are statistically analyzed in a sliding window, where M is in the range of [3, 10].

[0073] The present invention also provides a system for implementing the aforementioned edge task scheduling method based on heuristic reinforcement learning, the schematic diagram of which is shown below. Figure 2 As shown, the system includes: a task feature input module for real-time collection of multi-dimensional features of tasks to be scheduled in the queue and the system's running status, and construction of a 16-dimensional state vector S; a heuristic decision module for calculating the heuristic score H of each candidate task according to preset heuristic rules, including at least a priority inversion compensation term, a logarithmic waiting time compensation term, and a resource efficiency factor; a reinforcement learning engine module including a policy network and a target network for calculating the Q-value corresponding to each candidate task based on the state vector S, and training the network parameters using an experience replay and soft update mechanism; a policy fusion module for weighted fusion of the Q-value and the heuristic score H to obtain the final scheduling score of the candidate task; a task execution and monitoring module for selecting and executing the target task based on the final scheduling score, while monitoring the resource usage, execution time, and timeout during task execution; a reward calculation module for calculating the reward R based on the task execution result and feeding the reward R back to the reinforcement learning engine module; and a parameter self-optimization module for optimizing parameters based on system statistical performance indicators. The parameters are adaptively adjusted.

[0074] The method of the present invention will be further described below with reference to an embodiment:

[0075] The method was tested in a simulated high-load (continuous task flow > 50 tasks / minute) power system edge computing environment. The performance of the method described in this invention (hereinafter referred to as "this technical solution") was compared with that of the traditional priority queue scheduling method. The comparison of key performance indicators is shown in Table 1 below. Figure 3 As shown;

[0076] Table 1 Comparison of performance indicators under high load conditions

[0077] The experimental data above fully demonstrates that, compared with existing technologies, the task scheduling method based on heuristic reinforcement learning provided by this invention achieves significant and simultaneous improvements in multiple core performance indicators, including system reliability, resource utilization efficiency, critical business response speed, and congestion control capabilities. In particular, it solves the problem of drastic performance degradation of traditional methods under high loads, providing an effective technical solution for achieving efficient and adaptive edge computing scheduling in power systems.

Claims

1. An edge-side task scheduling method based on heuristic reinforcement learning, characterized in that, Includes the following steps: Collect the features of tasks to be scheduled in the queue and the system's operating status, and construct a state vector for scheduling decisions; Calculate a heuristic score for each candidate task; the heuristic score includes a priority inversion compensation term, a logarithmic waiting time compensation term, and a resource efficiency factor; Based on a deep Q-network, the reinforcement learning Q-value for each candidate scheduling action is calculated; The final scheduling score is calculated based on heuristic scoring and reinforcement learning Q-value. Based on the final scheduling score, all candidate tasks are sorted from largest to smallest, the task with the largest score is selected as the current scheduling task for execution, and the task result information is recorded. Calculate the reward R based on the task result information, and update the parameters of the depth Q network based on the reward R; After the preset task batch is completed, the parameters are adaptively adjusted according to the performance indicators of the batch of tasks. The heuristic scoring H includes a priority inversion compensation term. Logarithmic compensation term for waiting time and resource efficiency factor It can be expressed using the following formula: ; The priority inversion compensation item Calculate using the following formula: ;in, As the basic priority of the task; These are priority weight parameters; The waiting time logarithmic compensation term Calculate using the following formula: Where WH represents the task waiting time, in hours; For waiting time weighting parameters; The resource efficiency factor Calculate using the following formula: Where RN represents the task normalization resource requirement; The resource discount factor is used; the normalized resource requirement RN of the task is obtained by normalizing the three types of resources, namely CPU, memory and I / O, and then performing a weighted summation, wherein the weights are manually preset based on historical operation statistics. The reward R is the sum of positive reward items related to task priority, positive reward items related to resource efficiency, and negative reward items related to timeout penalties; The positive reward items related to task priority are: Positive rewards related to resource efficiency are: The negative rewards associated with timeout penalties are: TP is the segmented penalty coefficient based on the task timeout duration; The basic priority of the task is RN; the normalized resource requirement of the task is RN; the segmented penalty coefficient TP is based on the timeout duration of the task. Define segments: when When TP is 1, its value is 1; when When TP is 2, its value is 2; when When TP is 3, the value of TP is 3; where, This is the first timeout threshold; This is the second timeout threshold; A temporal difference learning mechanism is adopted to update the parameters of the deep Q-network based on the reward R, specifically including: The target value y of the time-difference time (TD) is calculated using the following formula: ;in, This indicates the subsequent state after the current task is executed. Indicates the state in the successor state The candidate scheduling action variables are used to perform a Q-value maximization search across the entire action set. Discount factor; For target network parameters; The timing difference (TD) error L is calculated using the following formula: ;in, s represents the Q-value prediction of the policy network for the current state-action pair; s represents the current system state, which is a state vector composed of multi-dimensional features such as queue load, CPU / memory / I / O resource utilization, task priority, waiting time, and remaining time limit; a represents the candidate scheduling action under the state s, which represents the specific task decision to be executed from the current set of tasks to be scheduled. For policy network parameters; The policy network parameters are updated by minimizing the TD error through gradient descent. .

2. The edge task scheduling method based on heuristic reinforcement learning according to claim 1, characterized in that, The state vector is a 16-dimensional vector, including 0-11 dimensional system-level features and 12-15 dimensional task-level features; the system-level features include 0-3 dimensional load features, 4-7 dimensional resource features, and 8-11 dimensional timeliness features. The state vector uses 16-dimensional features to encode the system's operating state, thereby improving the deep Q-network's ability to perceive global queue pressure and task distribution characteristics.

3. The edge task scheduling method based on heuristic reinforcement learning according to claim 2, characterized in that, The 4-dimensional load characteristics include queue load rate, total queue resource requirements, average single task resource requirements, and system CPU utilization; the 4-dimensional resource characteristics include CPU demand distribution statistics, memory demand distribution statistics, I / O demand distribution statistics, and resource over-limit flags; the 4-dimensional timeliness characteristics include average remaining time of tasks, the proportion of urgent tasks with remaining time below the threshold, the proportion of timed-out tasks, and average queue waiting time; the 4-dimensional task-level characteristics include the priority of the current candidate task, waiting time, normalized resource requirements, and remaining time.

4. The edge task scheduling method based on heuristic reinforcement learning according to claim 1, characterized in that, The state vector is input into the deep Q-network, which outputs the Q-value for each candidate scheduling action. The deep Q-network includes a policy network and a target network. Both the policy network and the target network use a 3-layer fully connected neural network, with 16 nodes in the input layer, 64 nodes in the hidden layer, and N nodes in the output layer, where N is the number of candidate tasks. The policy network and the target network have the same neural network structure but different parameters, and therefore different functions. The policy network is used to output the Q-value of each candidate task in real time and is used for online decision-making and experience generation. The target network is used to generate a stable Q-value estimate in the TD target value y, and only participates in the target calculation during the training process, without participating in the actual scheduling decision. The policy network uses... - A greedy strategy is used for searching, where the exploration rate is... The exponential decay occurs as training progresses, as expressed by the following formula: ;in, This is the initial exploration rate; The minimum exploration rate is given by k; the decay coefficient is given by t; and the number of iterations in the training process is given by t. A soft update mechanism is used to update the target network parameters, fusing historical parameters with new parameters according to a preset ratio, expressed by the following formula: ;in, To update the coefficients, control the update step size; For target network parameters; These are the parameters of the policy network; soft updates ensure training stability by slowly copying the parameters of the policy network to the target network.

5. The edge task scheduling method based on heuristic reinforcement learning according to claim 1, characterized in that, Based on heuristic scoring and reinforcement learning Q-values, the final scheduling score S is calculated using the following formula: Where Q is the Q-value of the candidate scheduling action corresponding to the task; H is the heuristic score; Let be the decision weight parameters, and .

6. The edge task scheduling method based on heuristic reinforcement learning according to claim 5, characterized in that, The decision weight parameters An online adaptive adjustment mechanism is adopted. When the queue load rate is greater than 80% and the CPU utilization is greater than 90%, the system is determined to be in a high-load state, and the decision weight parameters are increased. Preset the step size and re-detect until the system is no longer under high load or the decision weight parameters are adjusted. This is the preset maximum value.

7. The edge task scheduling method based on heuristic reinforcement learning according to claim 1, characterized in that, After the preset task batch is completed, based on the statistically obtained performance indicators including on-time completion rate, CPU utilization, and response time of high-priority tasks, the parameters are adjusted. Adaptive adjustments are made, with parameter adjustments using a combination of grid search and performance evaluation, to ensure that the system performance indicators reach the preset optimization goals. The adaptive parameter adjustment includes: 1) generating parameter combinations within a preset range; 2) evaluating the comprehensive score of each combination within a sliding window; 3) selecting the parameter combination with the highest score and applying it to the next stage of scheduling; the sliding window includes the most recent M batches; In the adaptive parameter tuning step: priority weights Search within the range [0.8, 1.2]; waiting time weight. Search within the range [0.4, 0.8]; resource efficiency discount factor Search within the range [0.1, 0.49]; decision weights Search within the range [0.3, 0.7]; and use the on-time completion rate as the benchmark. Maximize CPU utilization Maximize the average response time of high-priority tasks The comprehensive objective scoring function F is minimized to achieve joint optimization of multi-objective performance; the comprehensive objective scoring function F is expressed by the following formula: ;in Weighted by on-time completion rate; As a weight for CPU utilization; Weight the average response time of high-priority tasks; The parameter adjustment trigger condition after each batch of tasks is completed is as follows: when the cumulative number of completed tasks reaches N, a parameter search and evaluation is triggered once, where N is preset in the range of [50, 200]; during the parameter evaluation process, the performance indicators of the most recent M batches of tasks are statistically analyzed in a sliding window, where M is in the range of [3, 10].

8. A system for implementing the edge task scheduling method based on heuristic reinforcement learning as described in any one of claims 1 to 7, characterized in that, include: Task feature input module: used to collect multi-dimensional features of tasks to be scheduled in the queue and the system running status in real time, and construct a 16-dimensional state vector S; Heuristic decision-making module: used to calculate the heuristic score H of each candidate task according to the preset heuristic rules, including at least the priority inversion compensation term, the logarithmic compensation term of the waiting time and the resource efficiency factor; The reinforcement learning engine module includes a policy network and a target network, which are used to calculate the Q-values ​​of each candidate task based on the state vector S, and train the network parameters using an experience replay and soft update mechanism; the policy fusion module performs weighted fusion of the Q-values ​​and the heuristic score H to obtain the final scheduling score of the candidate task. The task execution and monitoring module is used to select and execute target tasks based on the final scheduling score, while monitoring resource usage, execution duration, and timeout status during task execution. The reward calculation module calculates the reward R based on the task execution results and feeds it back to the reinforcement learning engine module. The parameter self-optimization module optimizes parameters based on system statistical performance metrics. The parameters are adaptively adjusted.

Images