An adaptive task scheduling method based on node real-time feedback

Abstract

The application discloses a kind of adaptive task scheduling methods based on node real-time feedback, it is related to task scheduling technical field, the method includes the following steps: performance data is periodically collected and reported to database by calculation node;Data is normalized and load instantaneous change trend is calculated;Combining task resource demand, the comprehensive priority score including static resource availability score, dynamic trend penalty term and historical performance trust term is calculated;According to the score sorting screening node, the optimal target node is selected to allocate task;Real-time update task state and monitor execution process, trigger rescheduling when abnormal;With system performance index as objective function, automatically optimize scheduling parameter.The application realizes adaptive, efficient load balancing by real-time monitoring the performance index of calculation node, dynamically calculating node priority, and preferentially allocating task to the most suitable node.

Description

Technical Field

[0001] This invention relates to the field of task scheduling technology, and specifically to an adaptive task scheduling method based on real-time node feedback. Background Technology

[0002] Task scheduling technology is a core component of distributed computing and cloud computing. Its core objective is to rationally allocate tasks among multiple computing nodes to optimize overall system performance, improve resource utilization, and ensure service quality. With the widespread deployment of large data centers and cloud computing platforms, the importance of task scheduling is increasingly prominent. Traditional scheduling strategies, such as First-Come, First-Served, Shortest Job First, and Minimum Resource First, have achieved orderly task execution to a certain extent. In recent years, scheduling methods based on resource reservation and budget constraints have also been applied in cluster management and cloud services. They provide basic resource guarantees for specific types of applications through static or semi-static resource allocation strategies.

[0003] However, existing task scheduling methods still have significant shortcomings. On the one hand, most traditional strategies are static or offline scheduling methods, unable to perceive real-time changes in the state of nodes during runtime, such as instantaneous CPU load, memory pressure, and network bandwidth fluctuations. This leads to scheduling decisions that may rely on outdated or macroscopic average metrics, making it difficult to cope with dynamic and heterogeneous computing environments. On the other hand, although some methods introduce simple load balancing mechanisms, they typically lack a comprehensive evaluation of historical node performance and multi-dimensional performance metrics, failing to deeply match task characteristics with real-time node states. Furthermore, existing methods generally lack parameter self-optimization capabilities, with strategy parameters largely relying on manual experience settings, making it difficult to maintain efficiency and adaptability under different load conditions. Therefore, there is an urgent need for an adaptive task scheduling method that can dynamically adjust based on real-time node feedback, support multi-metric fusion evaluation, and possess self-optimization capabilities.

[0004] For example, Chinese patent CN113934525B discloses a Hadoop cluster task scheduling method based on a positive and negative feedback load scheduling algorithm, applied to the Hadoop resource allocation center. The cluster task scheduling method includes: idle computing nodes requesting to execute tasks from the management node; the management node receiving the task request and querying whether there is a history of running the task request; comparing and analyzing the history, if it matches the resource requirements of the task configuration, the task is computed on this computing node; otherwise, the computing node re-matches a new task request; it can dynamically allocate computing tasks in cluster task scheduling according to the resource usage and performance differences of different servers, thereby achieving efficient completion of computing tasks and improving CPU and resource utilization efficiency.

[0005] For example, Chinese Patent No. CN111897638B discloses a distributed task scheduling method and system. The method is executed by worker nodes in a distributed scheduling system, which includes multiple master nodes. The method includes: periodically sending parameter acquisition requests to at least one master node and receiving and locally storing the task type and total task load from the master node; upon detecting a new task processing condition, calculating the number of requested tasks based on the number of tasks in processing and the total task load, and constructing a task acquisition request based on the number of requested tasks and the task type; and sending a task acquisition request to a first target master node determined among the multiple master nodes, the task acquisition request instructing the first target master node to acquire tasks matching the requested task number and task type and provide feedback. This embodiment of the invention reduces the computational load on the master nodes and improves the system's supported task concurrency by having worker nodes actively request task types and task quantities from master nodes. Summary of the Invention

[0006] The technical problem this invention aims to solve is to address the shortcomings of existing technologies by providing an adaptive task scheduling method based on real-time node feedback. This method achieves adaptive and efficient load balancing by dynamically calculating node priorities through real-time monitoring of computing node performance metrics and prioritizing task allocation to the most suitable node.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] An adaptive task scheduling method based on real-time node feedback includes the following steps:

[0009] Step S1: Each node in the computing node cluster periodically collects real-time performance data according to a preset collection frequency, and reports the collected performance data to the database for centralized storage and management through the network communication interface.

[0010] Step S2: Normalize the collected performance data to convert it into dimensionless normalized data, and calculate the instantaneous load change trend based on load at different time scales.

[0011] Step S3: Based on the normalized data and the instantaneous load change trend, and combined with the resource requirement characteristics of the task to be scheduled, calculate a comprehensive priority score for each computing node. The priority score includes three parts: static resource availability score, dynamic trend penalty item, and historical performance trust item.

[0012] Step S4: Sort and filter all computing nodes according to the comprehensive priority score, determine the optimal target node from all computing nodes according to the predefined node selection strategy, and dynamically assign the task to be scheduled to the target node for execution.

[0013] Step S5: Update the status information of the assigned tasks in the database in real time and continuously monitor the execution process of the tasks. When a task execution timeout or failure is detected, the rescheduling mechanism is automatically triggered.

[0014] Step S6: Construct an objective function with system-level performance indicators as the optimization goal, and automatically optimize the parameters involved in the comprehensive priority score calculation process based on historical task execution data.

[0015] Furthermore, in step S1, the real-time performance data includes: CPU utilization, memory utilization, one-minute average load, five-minute average load, network input bandwidth utilization, network output bandwidth utilization, disk input / output utilization, and node local task queue length.

[0016] Further, in step S2, the normalization process includes: using a linear normalization method to map the utilization rate index to the [0, 1] interval, and using a normalization process based on the improved Sigmoid function to normalize the load index;

[0017] The utilization metrics include: CPU utilization, memory utilization, network input bandwidth utilization, network output bandwidth utilization, and disk input / output utilization.

[0018] The load metrics include: one-minute average load and five-minute average load.

[0019] Furthermore, in step S2, the instantaneous load change trend is calculated by calculating the relative change rate between the one-minute average load and the five-minute average load.

[0020] Furthermore, in step S3, the static resource availability score is calculated by multiplying the resource demand weight and the resource availability; the dynamic trend penalty item is composed of three weighted components: the load increase trend penalty item, the node local queue length penalty item, and the node overload state penalty item; the historical performance trust item is calculated based on two indicators: the node's historical task success rate and task completion efficiency, and the historical data is weighted using an exponential decay model.

[0021] Furthermore, in step S4, the predefined node selection strategy is a step-by-step filtering strategy, specifically including:

[0022] Step S401: Perform health screening and exclude calculation nodes whose comprehensive priority score is less than the health threshold;

[0023] Step S402: Perform resource matching degree filtering and select nodes that meet the minimum resource requirements of the task to form a set of selectable nodes;

[0024] Step S403: Select the node with the highest overall priority score from the set of selectable nodes as the target node.

[0025] Furthermore, step S5 specifically includes: if a task does not receive a task completion confirmation signal from the target node within a preset timeout threshold after being successfully assigned, the status of the task is rolled back to the schedulable state in the database, the rescheduling process is automatically triggered, and the historical performance trust value of the target node is reduced simultaneously.

[0026] Furthermore, step S6 specifically includes the following steps:

[0027] Construct a system-level multi-objective optimization function, which is a weighted combination of system throughput, average resource utilization, and average task response time;

[0028] Collect historical task execution data, including time-series data of node performance indicators, task execution results, and system operation indicators;

[0029] Initialize the parameters in the priority calculation model, including resource weight coefficient, dynamic trend penalty coefficient, and historical trust decay coefficient;

[0030] An iterative optimization algorithm based on gradient descent is adopted to automatically adjust various parameters with the goal of maximizing the system-level multi-objective optimization function.

[0031] The optimized parameter combination is updated to the task scheduling system, and the parameter optimization process is periodically retried.

[0032] A storage medium, characterized in that the storage medium stores instructions, which, when read by a computer, cause the computer to execute an adaptive task scheduling method based on real-time node feedback.

[0033] An electronic device, characterized in that it includes a processor and a storage medium, wherein the processor executes instructions in the storage medium.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] 1. This invention can dynamically perceive changes in system status by collecting and feeding back multi-dimensional performance data of nodes in real time, enabling scheduling strategies to adapt to node load fluctuations in real time and significantly improving scheduling effectiveness in complex environments.

[0036] 2. Based on a comprehensive priority scoring mechanism, this invention accurately allocates tasks to the most suitable nodes, reduces resource fragmentation and idle time, optimizes the overall resource utilization of the cluster, and increases the number of tasks completed per unit time.

[0037] 3. By introducing historical performance trust items and a real-time monitoring mechanism, this invention can quickly identify faulty nodes or nodes with abnormal performance, and automatically trigger task rescheduling, thereby ensuring the reliable execution of tasks and improving the availability of the system.

[0038] 4. This invention comprehensively considers the usage of various resources such as CPU, memory, network, and I / O, as well as load change trends, to achieve fine-grained load balancing, avoid local resource overheating, and improve the overall stability of the system.

[0039] 5. Adopt a parameter self-optimization framework with system indicators as the target, reduce the reliance on manual parameter tuning, enable the system to autonomously and continuously improve the scheduling strategy, reduce the complexity of operation and maintenance, and adapt to long-term evolving workload changes. Attached Figure Description

[0040] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0041] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of the system architecture according to an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] like Figure 1 As shown, an adaptive task scheduling method based on real-time node feedback includes the following steps:

[0045] Step S1: Each node in the computing node cluster periodically collects real-time performance data according to a preset collection frequency, and reports the collected performance data to the database for centralized storage and management through the network communication interface.

[0046] Step S2: Normalize the collected performance data to convert it into dimensionless normalized data, and calculate the instantaneous load change trend based on load at different time scales.

[0047] Step S3: Based on the normalized data and the instantaneous load change trend, and combined with the resource requirement characteristics of the task to be scheduled, calculate a comprehensive priority score for each computing node. The priority score includes three parts: static resource availability score, dynamic trend penalty item, and historical performance trust item.

[0048] Step S4: Sort and filter all computing nodes according to the comprehensive priority score, determine the optimal target node from all computing nodes according to the predefined node selection strategy, and dynamically assign the task to be scheduled to the target node for execution.

[0049] Step S5: Update the status information of the assigned tasks in the database in real time and continuously monitor the execution process of the tasks. When a task execution timeout or failure is detected, the rescheduling mechanism is automatically triggered.

[0050] Step S6: Construct an objective function with system-level performance indicators as the optimization goal, and automatically optimize the parameters involved in the comprehensive priority score calculation process based on historical task execution data.

[0051] In step S1, the real-time performance data includes: CPU utilization, memory utilization, one-minute average load, five-minute average load, network input bandwidth utilization, network output bandwidth utilization, disk input / output utilization, and node local task queue length.

[0052] In step S2, the normalization process includes: using a linear normalization method to map the utilization rate index to the [0, 1] interval, and using a normalization process based on the improved Sigmoid function to normalize the load index.

[0053] The formula for calculating the normalized load index is as follows:

[0054] ;

[0055] Where L represents the normalized load index, l represents the load index, and k represents the curve steepness coefficient, with a default value of k=0.8. This represents the intermediate load threshold, set to 1.5 times the number of CPU cores.

[0056] The utilization metrics include: CPU utilization, memory utilization, network input bandwidth utilization, network output bandwidth utilization, and disk input / output utilization.

[0057] The load metrics include: one-minute average load and five-minute average load.

[0058] In step S2, the instantaneous load change trend is calculated by calculating the relative change rate between the one-minute average load and the five-minute average load.

[0059] The formula for calculating the instantaneous change trend of the load is:

[0060] ;

[0061] in, This represents the instantaneous load change trend of node i at time t. This represents the normalized one-minute average load. This represents the normalized five-minute average load. This represents the smoothing factor, set to 10. -5 This is used to prevent the denominator from being zero. Indicates the index of the node. Indicates time.

[0062] In step S3, the static resource availability score is calculated by multiplying the resource demand weight and the resource availability. The dynamic trend penalty item is composed of three weighted components: the load increase trend penalty item, the node local queue length penalty item, and the node overload state penalty item. The historical performance trust item is calculated based on two indicators: the node's historical task success rate and task completion efficiency, and the historical data is weighted using an exponential decay model.

[0063] The formula for calculating the static resource availability score is as follows:

[0064] ;

[0065] in, This represents the static resource availability score of node i at time t. This represents the weight of resource j. This represents the normalized utilization rate of resource j at time t. Index representing a resource This indicates the total number of resources.

[0066] The formula for calculating the dynamic trend penalty term is:

[0067] ;

[0068] in, This represents the dynamic trend penalty for node i at time t. This represents the function that takes the maximum value. This represents the logarithmic function. This represents the queue length of node i at time t. This indicates an indicator function that takes the value 1 if the expression within the parentheses is true, and 0 otherwise. The load threshold is represented by α, β, and γ, which represent the penalty coefficients for the load increase trend penalty, the node local queue length penalty, and the node overload state penalty, respectively.

[0069] Load threshold Set to 1.5 times the number of node CPU cores. When the node load exceeds this threshold, the overload penalty sub-item will be triggered.

[0070] The formula for calculating the historical performance trust item is as follows:

[0071] ;

[0072] in, This indicates the trust level of node i's historical performance at time t. Represents the global gain coefficient. This represents the number of successful tasks performed by node i within the time window t. η represents the total number of tasks at node i within the time window t, and η represents the delay penalty coefficient. This represents the average task completion delay for node i within the time window t.

[0073] The global gain coefficient λ is set to 0.2 to avoid the problem that new nodes (with little historical data) will never be selected due to their low score in this item, while also ensuring that nodes with consistently excellent performance can receive stable bonuses.

[0074] In step S3, the formula for calculating the comprehensive priority score is as follows:

[0075] ;

[0076] in, This represents the overall priority score of node i at time t.

[0077] In step S4, the predefined node selection strategy is a step-by-step filtering strategy, specifically including:

[0078] Step S401: Perform health screening and exclude calculation nodes whose comprehensive priority score is less than the health threshold;

[0079] The health threshold is set to 0.4. This threshold is derived from statistical analysis of the cluster's historical operating data and can effectively distinguish between healthy and abnormal nodes.

[0080] When the health threshold is less than 0.4, a node is considered unhealthy and excluded from the candidate node set; when the health threshold is greater than or equal to 0.4, the node can participate in task scheduling.

[0081] Step S402: Perform resource matching degree filtering and select nodes that meet the minimum resource requirements of the task to form a set of selectable nodes;

[0082] Step S403: Select the node with the highest overall priority score from the set of selectable nodes as the target node.

[0083] The allocation algorithm uses a two-phase commit:

[0084] Pre-allocation: Locking node resources in the database;

[0085] Confirm Allocation: The node updates its status after confirming receipt of the task.

[0086] Step S5 specifically includes: if a task does not receive a task completion confirmation signal from the target node within a preset timeout threshold after it has been successfully assigned, the task's status is rolled back to the schedulable state in the database, a rescheduling process is automatically triggered, and the historical performance trust value of the target node is reduced simultaneously.

[0087] Each time a node fails or times out, its trust value is reduced to 80% of its original value.

[0088] Status monitoring specifically includes the following monitoring mechanisms:

[0089] Heartbeat timeout detection: If no report is submitted for more than 3 heartbeat cycles, the node is considered abnormal;

[0090] Task timeout detection: If the task execution time exceeds twice the expected time, a rescheduling will be triggered;

[0091] Resource conflict detection: When resource contention is detected, related tasks are rescheduled.

[0092] The rescheduling strategy employs a priority inheritance mechanism to ensure that important tasks are scheduled first.

[0093] Step S6 specifically includes the following steps:

[0094] Construct a system-level multi-objective optimization function, which is a weighted combination of system throughput, average resource utilization, and average task response time;

[0095] Collect historical task execution data, including time-series data of node performance indicators, task execution results, and system operation indicators;

[0096] Initialize the parameters in the priority calculation model, including resource weight coefficient, dynamic trend penalty coefficient, and historical trust decay coefficient;

[0097] An iterative optimization algorithm based on gradient descent is adopted to automatically adjust various parameters with the goal of maximizing the system-level multi-objective optimization function.

[0098] The optimized parameter combination is updated to the task scheduling system, and the parameter optimization process is periodically retried.

[0099] The specific formula for the system-level multi-objective optimization function is as follows:

[0100] ;

[0101] in, This represents a system-level multi-objective optimization function. This represents the set of parameters to be optimized, including resource weight coefficients, dynamic trend penalty coefficients, and historical trust decay coefficients. This represents the standardized system throughput, i.e., the number of tasks completed per hour. This represents the standardized average resource utilization rate. This represents the standardized average task response time. , and These represent the corresponding weighting coefficients.

[0102] The weight coefficients in the objective optimization function are: system throughput weights = 0.4; Resource utilization rate weight = 0.3; Response time weight = 0.3.

[0103] The learning rate of the gradient descent iterative optimization algorithm is set to 0.01, and the optimization process is terminated early when the change in the objective function value is less than a preset threshold. The constraints of the algorithm include: the value range of each parameter is limited to the interval [0, 1], and the sum of the sub-weights of each coefficient is equal to 1; a parameter re-optimization process is automatically triggered every 24 hours to ensure that the system continuously adapts to load changes. The preset threshold is set to 0.001.

[0104] like Figure 2 As shown, the system architecture used in this application includes:

[0105] The compute node cluster is responsible for executing specific tasks.

[0106] Each node periodically collects local performance data (CPU, memory, load, etc.) and reports it to the metadata database via the network interface. Simultaneously, it pulls task input data from shared storage and writes the results back.

[0107] The database serves as the "brain" of the system and a persistent storage center.

[0108] The stored content includes: historical and real-time performance data of each node, the status of all tasks (queued, running, completed, failed), assigned node ID, creation / start / end time, historical trust level of each node, success rate, average completion time, and storage-optimized scheduling parameters.

[0109] Shared storage, acting as the system's "data lake," decouples computing from storage.

[0110] It stores the task's input data, execution program, and output results. Any computing node can access it through a standard interface, ensuring the task's portability across different nodes.

[0111] The scheduling decision module serves as the "decision center" of the system.

[0112] Listen for new tasks or task completion events in the database.

[0113] When triggered, the latest state of all nodes is retrieved from the database.

[0114] Calculate normalized data, trends, and overall priority scores, and make scheduling decisions.

[0115] The task allocation decision is written to the database and the target computing node is notified to execute it.

[0116] Execute parameter optimization logic at a specific time or when triggered by an event.

[0117] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0118] The examples described herein are merely preferred embodiments of the invention and are not intended to limit the concept and scope of the invention. Any modifications and improvements made by those skilled in the art to the technical solutions of the invention without departing from the design concept of the invention should fall within the protection scope of the invention.

Claims

1. A method for adaptive task scheduling based on node real-time feedback, characterized in that, The method comprises the following steps: Step S1, each node in the node cluster periodically collects real-time performance data according to a preset collection frequency, and reports the collected performance data to a database through a network communication interface for centralized storage and management; Step S2, the collected performance data is normalized to convert it into dimensionless standardized data, and a load instantaneous change trend is calculated based on different time scales; Step S3, according to the standardized data and the load instantaneous change trend, and in combination with the resource demand characteristics of a to-be-scheduled task, a comprehensive priority score is calculated for each computing node, wherein the priority score comprises a static resource availability score, a dynamic trend penalty term and a historical performance trust term; The calculation formula of the static resource availability score is: ； in, Represents a node In time Static resource availability score, Representing resources The weight, Representing resources In time Normalized utilization rate Index representing a resource Indicates the total number of resources; The calculation formula of the dynamic trend penalty term is: ； wherein, represents a node in time a dynamic trend penalty term, represents a max function, represents a node in time a load transient change trend, represents a log function, represents a node in time a queue length, represents an indicator function, taking 1 when the expression inside the parentheses is true, and 0 otherwise, represents a normalized one-minute average load, represents a load threshold, , and represent the penalty coefficients of the load rising trend penalty sub-term, the node local queue length penalty sub-term and the node overload state penalty sub-term, respectively. The calculation formula of the historical performance trust term is: ； wherein, represents a node at time a history performance trust item, represents a global gain coefficient, represents a node a number of task successes within a time window about represents a node a total number of tasks within a time window about represents a delay penalty coefficient, represents a node an average task completion delay within a time window about ​​​ Step S4, all computing nodes are sorted and filtered according to the comprehensive priority score, an optimal target node is determined from all computing nodes according to a predefined node selection strategy, and the to-be-scheduled task is dynamically assigned to the target node for execution; Step S5, the state information of the assigned task is updated in real time in the database, and the task execution process is continuously monitored, and a re-scheduling mechanism is automatically triggered when a task execution timeout or failure exception is detected; Step S6, a target function with a system-level performance index as an optimization target is constructed, and each parameter involved in the comprehensive priority score calculation process is automatically optimized based on historical task execution data; The specific formula of the constructed target function is: ； wherein, represents the objective function, represents the set of parameters to be optimized, including the resource weight coefficient, the dynamic trend penalty coefficient and the historical trust decay coefficient, represents the normalized system throughput, i.e. the number of tasks completed per hour, represents the normalized average resource utilization, represents the normalized average task response time, , and respectively represent the corresponding weight coefficients.

2. The method of claim 1, wherein, In the step S1, the real-time performance data includes: central processing unit usage rate, memory usage rate, one-minute average load, five-minute average load, network input bandwidth usage rate, network output bandwidth usage rate, disk input / output usage rate and node local task queue length.

3. The method of claim 2, wherein, In the step S2, the normalization processing includes: for the usage rate index, a linear normalization method is adopted to map it to the [0, 1] interval, and for the load index, an improved Sigmoid function is adopted for normalization processing; The usage rate index includes: central processing unit usage rate, memory usage rate, network input bandwidth usage rate, network output bandwidth usage rate and disk input / output usage rate; The load index includes: one-minute average load and five-minute average load.

4. The method of claim 3, wherein, In the step S2, the calculation method of the load instantaneous change trend is: the relative change rate of the one-minute average load and the five-minute average load is calculated.

5. The method of claim 4, wherein, In the step S4, the predefined node selection strategy is a step-by-step screening strategy, which specifically includes: Step S401, health degree screening is performed to exclude computing nodes with a comprehensive priority score less than a health threshold; Step S402, resource matching degree screening is performed to select nodes that meet the minimum resource requirements of the task to form a selectable node set; Step S403, the node with the highest comprehensive priority score in the selectable node set is selected as the target node.

6. The method of claim 5, wherein, The step S5 specifically comprises: if a task does not receive a task completion confirmation signal from a target node within a preset timeout threshold after being successfully assigned, the state of the task is rolled back to a schedulable state in the database, a re-scheduling process is automatically triggered, and the historical performance trust item value of the target node is synchronously reduced.

7. A storage medium, characterized by The storage medium has instructions stored therein, and when a computer reads the instructions, the computer is caused to perform the method of any one of claims 1-6.

8. An electronic device, comprising: A computer system comprising a processor and the storage medium of claim 7, wherein the processor executes the instructions in the storage medium.

Images