Distributed integrated task scheduling method, system, and media based on load balancing
By constructing a directed acyclic graph and predicting load changes, the distributed task scheduling is optimized, solving the problems of inter-task communication dependencies and collaborative perception of node load changes. This achieves communication minimization and load balancing, improving the accuracy of scheduling decisions and the efficiency of cluster execution.
Patent Information
- Application Number
- CN202611491926.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-23
- Publication Date
- 2026-11-17
AI Technical Summary
Existing distributed task scheduling methods lack the ability to collaboratively perceive inter-task communication dependencies and node load changes, resulting in high communication overhead and uneven resource load.
By constructing a directed acyclic graph to compute the communication sensitivity of tasks, we divide tasks into groups and predict load changes. We then combine the inter-group communication matrix and node load data to optimize scheduling and generate a group-node mapping table.
It achieves joint optimization of minimizing cross-node communication and balancing node load, improving the accuracy of scheduling decisions and the overall execution efficiency of the cluster.
Smart Images

Figure CN123114698A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of task scheduling technology, specifically to a distributed integrated task scheduling method, system, and medium based on load balancing. Background Technology
[0002] In distributed computing scenarios, such as big data analytics, machine learning training, and real-time stream processing, the task scheduler is responsible for distributing a large number of data-dependent computing tasks across the nodes of the cluster for execution. Existing distributed task scheduling methods mainly rely on resource load status for scheduling. By collecting real-time utilization data such as CPU and memory on each node, tasks are preferentially allocated to nodes with lower current loads to achieve load balancing. Alternatively, scheduling may be based on data locality, allocating tasks to nodes where the required data resides to reduce data network transmission overhead.
[0003] However, the two types of methods mentioned above are usually independent of each other and lack the ability to collaboratively perceive the communication dependencies between tasks and the dynamic load changes of nodes. The scheduling method based on resource load status ignores the cross-node transmission cost when there is frequent data interaction between tasks, which may lead to communication-intensive tasks being distributed to different nodes. A large number of intermediate results need to be transmitted over the network, resulting in high communication overhead. The scheduling method based on data locality ignores the dynamic fluctuation trend of node load, which may lead to some nodes being overloaded due to continuously receiving new tasks, while other nodes are in an idle state, resulting in uneven resource load.
[0004] In summary, existing distributed task scheduling technologies lack the ability to collaboratively perceive inter-task communication dependencies and node load changes, resulting in technical problems such as high communication overhead and uneven resource load. Summary of the Invention
[0005] The purpose of this application is to provide a distributed integrated task scheduling method, system, and medium based on load balancing, which solves the technical problems in the prior art of distributed task scheduling, such as the lack of collaborative perception of inter-task communication dependencies and node load changes, resulting in high communication overhead and uneven resource load.
[0006] In view of the above problems, this application provides a distributed integrated task scheduling method, system and medium based on load balancing.
[0007] The first aspect of this application provides a load-balancing-based distributed integrated task scheduling method, which includes: acquiring an original task set to be scheduled; constructing a directed acyclic graph (DAG) based on the data dependencies between tasks and calculating the communication sensitivity of each task; based on the DAG and the communication sensitivity of each task, dividing the task set into K task groups with the goal of minimizing the sum of the total cross-group communication volume and the variance of resource demand within the group, and calculating the inter-group communication matrix; acquiring real-time load data of each node in the distributed processing system, predicting the global total load and load change rate within a future time window, and calculating the minimum number of active nodes and the dynamic overload threshold based on the prediction results; assigning the K task groups to active nodes according to the minimum number of active nodes and the dynamic overload threshold, optimizing the scheduling based on the inter-group communication matrix, generating a group-node mapping table, and distributing each task group to the corresponding node for execution.
[0008] Optionally, the directed acyclic graph is constructed based on the data dependencies between tasks, wherein each directed edge marks the total amount of communication data and the communication frequency between the two task nodes connected by the directed edge, and the weight of each directed edge is determined by normalized weighting of the total amount of communication data and the communication frequency between the two tasks connected by the directed edge; the communication sensitivity of each task is obtained by calculating the sum of the edge weights between each task node and all its adjacent task nodes.
[0009] Optionally, using each task node in the directed acyclic graph as a clustering object and the edge weights between tasks as the clustering distance metric, tasks with communication sensitivity higher than a preset sensitivity threshold are prioritized for aggregation to obtain an initial aggregation result. Based on the initial aggregation result, a grouping operation is iteratively performed with the objective of minimizing the sum of the total cross-group communication volume and the variance of resource demand within the group, generating the K task groups. The total cross-group communication volume between any two task groups is calculated for the K task groups to obtain the inter-group communication matrix.
[0010] Optionally, when performing grouping operations iteratively with the goal of minimizing the sum of cross-group communication volume and the variance of resource demand within a group, the total resource demand of each group is calculated in each iteration. Groups whose total resource demand exceeds the single-node capacity limit are split up, and the two groups with the largest cross-group communication volume are merged. The iteration is terminated when merging any two groups would cause the resource demand of a group to exceed the single-node capacity limit or the decrease in the total cross-group communication volume is lower than a preset threshold.
[0011] Optionally, the CPU utilization, memory utilization, and network bandwidth utilization of each node are collected at fixed time intervals as real-time load data; based on the real-time load data, time-series prediction is performed based on historical load sequences, and the predicted global total load and the predicted load change rate within the future time window are output; the basic number of active nodes is determined based on the ratio of the predicted global total load to the capacity limit of a single node; the basic number of active nodes is corrected based on the predicted load change rate to obtain the minimum number of active nodes; the initial value is dynamically adjusted based on the predicted load change rate to set a dynamic overload threshold.
[0012] Optionally, when the predicted load change rate indicates that the future load is on an upward trend, the number of redundant nodes is increased based on the ratio of the total upward load to the upper limit of the capacity of a single node, calculated according to the rate of increase, on the basis of the basic number of active nodes. When the predicted load change rate indicates that the future load is on a downward trend, the number of recyclable nodes is subtracted from the ratio of the total downward load to the upper limit of the capacity of a single node, calculated according to the rate of decrease, based on the basic number of active nodes, but the minimum number of active nodes is kept not lower than a preset safety lower limit, which is preset according to the system fault tolerance requirements and the number of critical tasks.
[0013] Optionally, the total resource requirements and inter-group communication matrix of each task group are extracted; based on the minimum number of active nodes, a set of candidate nodes to be activated is selected from all nodes, and the nodes in the candidate node set are sorted from low to high according to their real-time load status; based on the inter-group communication matrix, each task group is treated as a whole and allocated to the candidate node with the lowest current load and remaining capacity that can meet the total resource requirements of the task group, in descending order of inter-group communication volume; after each task group is allocated, the allocated load and remaining capacity of the target node are updated, and the inter-node load variance and cross-node communication volume are recalculated; when all task groups are allocated, if the cross-node communication volume exceeds a preset threshold, an inter-group exchange optimization step is triggered, and the allocated task groups are partially exchanged. If the cross-node communication volume still exceeds the preset threshold after the exchange, the partial exchange is iteratively executed until the cross-node communication volume does not exceed the preset threshold, and the optimized group-node mapping table is output.
[0014] Optionally, calculate the total cross-group communication volume between any two different nodes and identify the node pair with the largest cross-node communication volume; among the task groups contained in the node pair with the largest cross-node communication volume, select the group with the highest communication volume with other nodes as the group to be migrated; evaluate the joint change in the load variance between nodes and the total cross-node communication volume after swapping the group to be migrated to the node of the communication counterpart; if the joint change is better than the current scheme, then perform the swap operation.
[0015] A second aspect of this application provides a load-balanced distributed integrated task scheduling system, comprising: a task acquisition module, used to acquire the original task set to be scheduled, construct a directed acyclic graph based on the data dependencies between tasks, and calculate the communication sensitivity of each task; a task partitioning module, used to partition the task set into K task groups based on the directed acyclic graph and the communication sensitivity of each task, with the goal of minimizing the sum of the total cross-group communication volume and the variance of resource demand within the group, and calculate the inter-group communication matrix; a parameter calculation module, used to acquire real-time load data of each node in the distributed processing system, predict the global total load and load change rate within a future time window, and calculate the minimum number of active nodes and the dynamic overload threshold based on the prediction results; and a task allocation module, used to allocate the K task groups to active nodes according to the minimum number of active nodes and the dynamic overload threshold, using the group as the scheduling unit, perform scheduling optimization in conjunction with the inter-group communication matrix, generate a group-node mapping table, and distribute each task group to the corresponding node for execution.
[0016] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed, implements the steps of the above-described load-balanced distributed integrated task scheduling method.
[0017] One or more technical solutions provided in this application have at least the following technical effects or advantages: The method provided in this application obtains the original task set to be scheduled, constructs a directed acyclic graph (DAG) based on the data dependencies between tasks, and calculates the communication sensitivity of each task. Based on the DAG and the communication sensitivity of each task, the task set is divided into K task groups with the goal of minimizing the sum of the total cross-group communication volume and the variance of resource demand within the group, and an inter-group communication matrix is calculated. Real-time load data of each node in the distributed processing system is obtained, and the global total load and load change rate within a future time window are predicted. The minimum number of active nodes and the dynamic overload threshold are calculated based on the prediction results. The K task groups are used as scheduling units, and each task group is assigned to an active node according to the minimum number of active nodes and the dynamic overload threshold. Scheduling optimization is performed in conjunction with the inter-group communication matrix to generate a group-node mapping table, and each task group is sent to the corresponding node for execution. This achieves the technical effect of improving the accuracy of scheduling decisions and the overall execution efficiency of the cluster by integrating communication awareness and load awareness to jointly optimize cross-node communication minimization and node load balancing.
[0018] The above description is merely an overview of the technical solution of this application. To better understand the technical means of this application and to facilitate its implementation according to the description, and to make the above and other objects, features, and advantages of this application more apparent, specific embodiments of this application are described below. It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent through the following description. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0020] Figure 1 A flowchart illustrating the load-balancing-based distributed integrated task scheduling method provided in this application.
[0021] Figure 2 This is a schematic diagram of the structure of the distributed integrated task scheduling system based on load balancing provided in this application.
[0022] Figure labeling: Task acquisition module 11, Task division module 12, Parameter calculation module 13, Task allocation module 14. Detailed Implementation
[0023] This application provides a load-balanced distributed integrated task scheduling method, system, and medium to address the technical problems in existing distributed task scheduling technologies, such as the lack of collaborative awareness of inter-task communication dependencies and node load changes, resulting in high communication overhead and uneven resource load. It achieves the technical effect of improving scheduling decision accuracy and overall cluster execution efficiency by fusing communication awareness and load awareness to minimize cross-node communication and balance node load.
[0024] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be understood that the present invention is not limited to the exemplary embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. It should also be noted that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, not all of them.
[0025] Example 1, as Figure 1As shown, this application provides a distributed integrated task scheduling method based on load balancing, the distributed integrated task scheduling method based on load balancing includes: Obtain the original set of tasks to be scheduled, construct a directed acyclic graph based on the data dependencies between tasks, and calculate the communication sensitivity of each task.
[0026] Furthermore, a directed acyclic graph is constructed based on the data dependencies between tasks, and the communication sensitivity of each task is calculated. This includes: constructing the directed acyclic graph based on the data dependencies between tasks, wherein each directed edge marks the total amount of communication data and the communication frequency between the two task nodes connected by the directed edge, wherein the weight of each directed edge is determined by normalized weighting of the total amount of communication data and the communication frequency between the two tasks connected by the directed edge; and calculating the sum of the edge weights between each task node and all its adjacent task nodes to obtain the communication sensitivity of each task.
[0027] Specifically, the process involves obtaining a raw set of tasks to be scheduled from the task queue or job submission interface. This raw set contains multiple tasks to be executed. Data dependencies between tasks in the raw set are analyzed through static code analysis, execution plan parsing, or historical log tracking. For structured computing frameworks such as Spark and Tez, the parent task identifiers and data partition mappings for each task are parsed from the logical execution plan carried during task submission, such as RDD dependency chains or DAG execution plans. For custom business tasks, the process involves scanning the inputs / outputs field declarations in the task configuration file or using bytecode instrumentation to extract variable read / write dependencies during task compilation, thereby determining which preceding tasks each task needs to wait for to complete and receive their output data. Each independent task instance is treated as a vertex in the graph and assigned a globally unique number. For each identified dependency relationship—the output of task A being the input of task B—a directed edge is added from vertex A to vertex B, constructing a directed acyclic graph G=(V,E), where each vertex in V represents an independent task node, and each directed edge in E represents a task node. ij Indicates task v i The execution output is task v j Input dependency, i.e., v j We must wait for v i It can only be started after the execution is complete and the data is generated.
[0028] Simultaneously, for each directed edge, the total amount of communication data and the communication frequency between the two task nodes connected by the directed edge are marked, wherein the total amount of communication data D ij Indicates v i Passed to task v jThe number of data bytes is obtained from the average number of bytes transmitted in the historical execution log, or specified by the estimated output size field in the task resource configuration. The communication frequency F ij This indicates the number of tasks per second (v) within a unit time window. i To task v j The number of data transmissions initiated reflects the intensity of interaction, obtained from data stream sampling between tasks or throughput monitoring of message queues. Based on the total amount of communication data and communication frequency, a corresponding weight is calculated for each directed edge to characterize the impact of this dependency on communication performance. Each directed edge e... ij weight W ij The total amount of communication data D ij and communication frequency F ij After normalization and weighting, its expression is: Where α and β are the weighting coefficients of the total data volume and communication frequency in the edge weights, respectively, both ranging from 0 to 1. They can be optimized based on the sensitivity to network bandwidth and I / O latency, and satisfy α + β = 1, such as α = 0.6, β = 0.4, D max F represents the maximum total amount of communication data across all directed edges in the current task set. max This represents the maximum communication frequency across all directed edges in the current task set. After completing graph construction and edge weight assignment, for each task node v... i The communication sensitivity C of each task is obtained by summing the edge weights of all adjacent task nodes. i And store it as an attribute of the task node.
[0029] By transforming the original task set into a quantifiable computational graph structure and assigning each task a numerical label that clearly reflects the strength of its interaction through communication sensitivity, the joint perception of communication overhead and computational load is achieved, thereby improving the load balancing accuracy and execution efficiency of task scheduling.
[0030] Based on the directed acyclic graph and the communication sensitivity of each task, with the goal of minimizing the sum of the total cross-group communication volume and the variance of resource demand within the group, the task set is divided into K task groups, and the inter-group communication matrix is calculated.
[0031] Furthermore, based on the directed acyclic graph and the communication sensitivity of each task, with the objective of minimizing the sum of the total cross-group communication volume and the variance of resource demand within the group, the task set is divided into K task groups, and the inter-group communication matrix is calculated. This includes: using each task node in the directed acyclic graph as a clustering object, using the edge weights between tasks as the clustering distance metric, prioritizing the aggregation of tasks with communication sensitivity higher than a preset sensitivity threshold to obtain an initial aggregation result; based on the initial aggregation result, iteratively performing grouping operations with the objective of minimizing the sum of the total cross-group communication volume and the variance of resource demand within the group to generate the K task groups; and calculating the total cross-group communication volume between any two task groups for the K task groups to obtain the inter-group communication matrix.
[0032] Furthermore, when iteratively performing grouping operations with the goal of minimizing the sum of cross-group communication volume and the variance of resource demand within a group, the total resource demand of each group is calculated in each iteration. Groups whose total resource demand exceeds the single-node capacity limit are split up, and the two groups with the largest cross-group communication volume are merged. The iteration is terminated when merging any two groups would cause the resource demand of a group to exceed the single-node capacity limit or the decrease in the total cross-group communication volume is lower than a preset threshold.
[0033] Specifically, each task node V in the constructed directed acyclic graph G=(V,E) is used as a clustering object, and the edge weights W between tasks are used as the clustering criteria. ij As a clustering distance metric, a larger edge weight indicates closer communication between two tasks, resulting in a smaller clustering distance. The average communication sensitivity of all task nodes is used as a preset sensitivity threshold. Task nodes with communication sensitivity higher than this threshold are marked as high-communication-sensitive nodes and prioritized for aggregation. For each high-communication-sensitive node, a breadth-first search strategy is used to traverse its neighboring nodes. If the edge weight between a neighboring node and the task node is greater than the median of all current edge weights, the neighboring task node is included in the same initial cluster; otherwise, it is not merged. After the traversal, task nodes with communication sensitivity less than or equal to the preset sensitivity threshold are assigned to the nearest cluster based on proximity, i.e., the existing cluster with the largest sum of edge weights is selected, forming the initial aggregation result. Each initial cluster in the initial aggregation result is a preliminary task group, recording the set of all task nodes contained within each task group.
[0034] Based on the initial aggregation results, with the joint optimization objective of minimizing the sum of cross-group communication volume and intra-group resource demand variance, an iterative grouping operation is performed. In each iteration, the total resource demand R of each task group g is first calculated. g , , where r i For task node v iThe resource requirement vector, encompassing multiple dimensions such as CPU core count, memory capacity, and disk I / O bandwidth, is calculated using a weighted scalar merging method. The normalized weight coefficients for each dimension satisfy the constraint that the sum of the weights is 1. Simultaneously, the maximum capacity U of a single node is determined by multiplying the total resources of the lowest-performing node in the distributed system by a safety factor. The safety factor ranges from 0 to 1, with a preferred value of 0.8, and can be dynamically set according to actual needs. The sum R of the resource requirements of each group is then calculated. g The resource requirements of a task group are compared with the single-node capacity limit U. If the total resource requirement of a task group exceeds the single-node capacity limit, the group is marked as an overloaded group and split. The splitting strategy adopts the Kernighan-Lin graph partitioning algorithm, which performs a bisection within the group, such that the total resource requirement of the two subgroups after the split is less than or equal to the single-node capacity limit U, and the sum of the weights on the split edges is minimized.
[0035] After splitting all overload groups, calculate g for any two current task groups. p and g q Total cross-group communication T pq : Among them, W ij For directed edge v i to v j The weight, W ji For directed edge v j to v i The weighting is determined based on the calculated total cross-group communication volume. The pair with the largest total cross-group communication volume is identified, and the two pairs with the largest cross-group communication volumes are merged. Before merging, the total resource requirement after merging is pre-calculated. If the total resource requirement is less than or equal to the single-node capacity limit, the merge operation is performed; otherwise, the pair is discarded, and the pair with the second largest cross-group communication volume is selected for analysis and merging. This splitting and merging iteration continues. The iteration terminates when merging any two task groups would cause the resource requirement of one group to exceed the single-node capacity limit, or when the decrease in the total cross-group communication volume compared to the previous round is less than a preset threshold. The preset threshold is 0.01 by default.
[0036] After terminating the iteration, K task groups are generated. The total cross-group communication between any two task groups is calculated for each of the K task groups, and a K×K inter-group communication matrix M is constructed, with matrix elements M. pq Indicates task group G p With task group G q The total amount of cross-group communication is calculated by setting the diagonal elements of the matrix to zero, indicating that intra-group communication does not generate cross-node network overhead.
[0037] Through an iterative grouping mechanism based on dual-objective joint optimization, the discrete and disordered task set is reconstructed into several task groups with dense internal communication, less external cross-group communication, and balanced resource requirements, thereby reducing the amount of cross-node data transmission and the risk of node overload.
[0038] Obtain real-time load data of each node in the distributed processing system, predict the global total load and load change rate within a future time window, and calculate the minimum number of active nodes and dynamic overload threshold based on the prediction results.
[0039] Furthermore, real-time load data of each node in the distributed processing system is acquired, and the global total load and load change rate within a future time window are predicted. Based on the prediction results, the minimum number of active nodes and the dynamic overload threshold are calculated. This includes: collecting the CPU utilization, memory utilization, and network bandwidth utilization of each node at fixed time intervals as real-time load data; performing time-series prediction based on historical load sequences using the real-time load data, and outputting the predicted global total load and load change rate within the future time window; determining the basic number of active nodes based on the ratio of the predicted global total load to the capacity limit of a single node; correcting the basic number of active nodes based on the predicted load change rate to obtain the minimum number of active nodes; and dynamically adjusting the initial value based on the predicted load change rate to set the dynamic overload threshold.
[0040] Furthermore, the basic number of active nodes is adjusted based on the predicted load change rate to obtain the minimum number of active nodes. This includes: when the predicted load change rate indicates that the future load is trending upward, the number of redundant nodes is increased based on the ratio of the total increasing load to the upper limit of the capacity of a single node, calculated according to the rate of increase. When the predicted load change rate indicates that the future load is trending downward, the number of recyclable nodes is subtracted from the ratio of the total decreasing load to the upper limit of the capacity of a single node, calculated according to the rate of decrease, based on the basic number of active nodes, while maintaining the minimum number of active nodes at no less than a preset safety lower limit, which is preset based on system fault tolerance requirements and the number of critical tasks.
[0041] Specifically, operations and maintenance personnel dynamically configure fixed time intervals, such as 5 seconds, based on the cluster size. At these fixed intervals, they collect CPU utilization, memory utilization, and network bandwidth utilization from each node in the distributed processing system. Specifically, CPU time slice allocation statistics are read using system calls provided by the node's operating system kernel, such as the ` / proc / stat` file or the `getrusage` interface in Linux. The percentage of non-idle time within a unit time interval is calculated to obtain CPU utilization. The ratio of currently used physical memory to total physical memory capacity is obtained by reading interfaces provided by the operating system's memory management subsystem, such as the ` / proc / meminfo` file in Linux, and expressed as a percentage. Simultaneously, the total number of bytes sent and received by the network interface controller (NIC) within a unit time interval is obtained using statistical counters provided by the NIC driver, such as the ` / proc / net / dev` file or the `ethtool` command in Linux. This number is then divided by the NIC's nominal maximum bandwidth rate to calculate network bandwidth utilization. Finally, the CPU utilization of each node is calculated. Memory utilization and network bandwidth utilization This is real-time load data, where j is the node number and t is the collection time.
[0042] Based on real-time load data, an autoregressive integral moving average (ARIMA) model is used to perform time-series forecasting of the historical load sequence, obtaining the predicted global total load and load change rate within the future time window. The historical load sequence refers to the historical window length m, such as the data set of the most recent 5 minutes. The future time window ΔT is 30 seconds by default. The ARIMA model is used to perform time-series forecasting of the historical load sequence. First, the CPU utilization, memory utilization, and network bandwidth utilization of each node are combined into a single-dimensional load value through a weighted scalar: ,in, , , These are the weighting coefficients corresponding to CPU utilization, memory utilization, and network bandwidth utilization, respectively. Preferably, the values are 0.5, 0.3, and 0.2, respectively. Then, ARIMA prediction is performed independently for each node. During prediction, the unit root test is first used to check the stationarity of the historical load sequence for each node. If the historical load sequence is non-stationary, it is converted into a stationary sequence through differencing. Based on the Bayesian information criterion, the optimal parameter combination is searched within a preset range of autoregressive order ∈ [0,5] and moving average order ∈ [0,5] to minimize the Akaike information criterion value of the ARIMA model fitting residuals. The determined optimal ARIMA model is used to predict the stationary historical load sequence forward. The prediction results are then subjected to an inverse summation transformation to obtain the predicted load value at time ΔT in the future. The predicted load values of all nodes are summed to obtain the global total load prediction. Simultaneously, load data from the most recent m sampling times are extracted from the historical load queue, forming a set of discrete data points with time as the independent variable and load as the dependent variable. A least squares method is used to fit an optimal straight line that minimizes the sum of the squared vertical distances from each data point to this line. The slope of this line represents the predicted load change rate on the time axis. A predicted load change rate greater than 0 indicates an increasing load trend, while a value less than 0 indicates a decreasing load trend. The absolute value indicates the rate of change. The basic active node number is determined by rounding up the ratio of the global total load prediction to the capacity limit of a single node, ensuring that the calculated number of basic active nodes can support the predicted load.
[0043] The basic active node count is adjusted based on the predicted load change rate. When the predicted load change rate indicates an upward trend in future load, the total additional load within the future time window ΔT is calculated based on the rate of increase: Total load ΔL = Predicted load change rate × ΔT. The ratio of ΔL to the upper limit of a single node's capacity is calculated and rounded up to determine the number of redundant nodes to absorb sudden load increases within the prediction window and prevent node overload. The minimum active node count is obtained by summing the basic active node count and the number of redundant nodes. When the predicted load change rate indicates a downward trend in future load, the total load that can be released in the future is calculated based on the rate of decrease: Total load ΔL = |Predicted load change rate| × ΔT. The ratio of ΔL to the upper limit of a single node's capacity is calculated and rounded down to determine the number of reclaimable nodes. The number of reclaimable nodes is subtracted from the basic active node count and compared with a safety lower limit to ensure that the minimum active node count is not lower than the preset safety lower limit, allowing for the reclamation of some nodes to save energy and resources. The safety lower limit is determined and preset based on system fault tolerance requirements, such as the requirement of at least 3 live nodes for HDFS replicas and the number of critical tasks. The default value is 10% of the total number of nodes in the cluster and no less than 3.
[0044] The initial value of the dynamic overload threshold is the rated load limit of the physical node, i.e., the nominal capacity of the node, multiplied by a preset safety factor. The safety factor is 0.8 by default and can be dynamically set according to actual needs. The system monitors the predicted load change rate in real time. When the predicted load change rate is consistently positive and exceeds the first threshold, the dynamic overload threshold is gradually reduced to reserve buffer capacity. When the predicted load change rate is consistently negative and below the second threshold, the dynamic overload threshold is gradually increased to allow more nodes to enter a dormant state. The adjustment range of the dynamic overload threshold is proportional to the absolute value of the load change rate, and the adjusted dynamic overload threshold does not exceed the preset upper and lower limits. The first threshold is greater than the second threshold. The default value of the first threshold is 0.05, meaning that when the load growth rate exceeds 5% / second, it is determined that the load is about to enter a peak period, and the dynamic overload threshold is gradually reduced to reserve buffer capacity. The second threshold is -0.05, meaning that when the load decrease rate exceeds 5% / second, it is determined that the load is about to enter a low period, and the dynamic overload threshold is gradually increased to allow more nodes to enter a dormant state, thereby reducing overall energy consumption. The preset upper limit is 0.95 times the rated load limit to prevent nodes from crashing due to overload, and the preset lower limit is 0.6 times the rated load limit to ensure that nodes are not overworked.
[0045] By predicting and proactively planning resource capacity, the required number of active nodes is dynamically determined and the overload judgment criteria are adaptively adjusted, enabling the scheduling system to proactively respond to load fluctuations. This ensures that nodes are not overloaded, avoids excessive or insufficient resource reservation, and improves the cluster's resource utilization efficiency and scheduling stability.
[0046] The K task groups are used as scheduling units. Based on the minimum number of active nodes and the dynamic overload threshold, each task group is assigned to an active node. The scheduling is optimized by combining the inter-group communication matrix, a group-node mapping table is generated, and each task group is sent to the corresponding node for execution.
[0047] Furthermore, the K task groups are scheduled as groups, and each task group is assigned to an active node according to the minimum number of active nodes and the dynamic overload threshold. Scheduling optimization is performed using the inter-group communication matrix to generate a group-node mapping table, including: extracting the total resource requirements of each task group and the inter-group communication matrix; selecting a set of candidate nodes to be activated from all nodes based on the minimum number of active nodes, and sorting the nodes in the candidate node set from low to high according to their real-time load status; and, based on the inter-group communication matrix, sequentially assigning each task group to an active node according to the inter-group communication volume from high to low. As a whole, tasks are assigned to candidate nodes with the lowest current load and sufficient remaining capacity to meet the total resource requirements of the task group. After each task group is assigned, the assigned load and remaining capacity of the target node are updated, and the load variance between nodes and the total cross-node communication volume are recalculated. When all task groups are assigned, if the total cross-node communication volume exceeds a preset threshold, the inter-group exchange optimization step is triggered to perform a partial exchange on the assigned task groups. If the total cross-node communication volume still exceeds the preset threshold after the exchange, the partial exchange is iterated until the total cross-node communication volume does not exceed the preset threshold, and the optimized group-node mapping table is output.
[0048] Furthermore, trigger the inter-group exchange optimization step to perform local exchange on the assigned task groups, including: calculating the total cross-group communication volume between any two different nodes, identifying the node pair with the largest cross-node communication volume; among the task groups contained in the node pair with the largest cross-node communication volume, selecting the group with the highest communication volume with other nodes as the group to be migrated, evaluating the joint change in the inter-node load variance and the total cross-node communication volume after exchanging the group to be migrated to the node of the communication counterpart, and if the joint change is better than the current scheme, then execute the exchange operation.
[0049] Specifically, the K task groups are organized into groups as scheduling units. The total resource requirements and inter-group communication matrix M of each of the K task groups are extracted, and the minimum number of active nodes N is used as the scheduling unit. min The process involves selecting a set of candidate nodes to be activated from all nodes in the distributed system. The selection logic is as follows: sort all current nodes from low to high according to their real-time load status, and select the N nodes with the lowest load. min One node is selected as the candidate node set, and the remaining nodes are marked as waiting to sleep. This ensures that the activated nodes are the nodes with the lightest current load, and the nodes in the candidate node set are sorted from low to high according to their real-time load status.
[0050] Based on the inter-group communication matrix M, the task groups are sorted in descending order of total cross-group communication volume to generate an allocation queue. One task group is sequentially selected from the allocation queue as the current task group to be allocated, and treated as an indivisible whole. For the current task group to be allocated, all nodes in the candidate node set are traversed. For each node, nodes with remaining capacity sufficient to meet the total resource requirements of the group are selected as candidate nodes. From the candidate node set, the node with the lowest current load is selected as the target node. When multiple candidate nodes with the same load exist, the increase in cross-node communication volume after allocating the group to each candidate node is compared, and the node with the smallest increase in communication volume is selected as the final target node. That is, communication locality is prioritized under the premise of load balancing.
[0051] After each task group is assigned, the allocated load and remaining capacity of the target node are updated. Simultaneously, the inter-node load variance and total cross-node communication volume of the current allocation scheme are recalculated. Once all K task groups have been assigned, it is checked whether the total cross-node communication volume exceeds a preset threshold. If the total cross-node communication volume is less than or equal to the preset threshold, the current group-node mapping table is directly output. If the total cross-node communication volume exceeds the preset threshold, the inter-group exchange optimization step is triggered: Calculate the n values of every two distinct nodes in the candidate node set. p and n q The total amount of cross-group communication between them, and find the node pair with the largest cross-node communication volume (n a n b ), in the node pair with the largest cross-node communication volume n a and n b Among the included task groups, the group with the highest communication volume with other nodes is identified as the group to be migrated. That is, this group communicates most intensively with other nodes besides the hot node pairs, and migrating it will help reduce the global communication volume.
[0052] The assessment evaluates the joint change after the group to be migrated is swapped from its current node to another node in the hot spot node pair. : ,in, The change in load variance is calculated by subtracting the load variance between nodes before the swap from the load variance between nodes after the swap. This represents the change in total cross-node communication volume, calculated by subtracting the total cross-node communication volume before the exchange from the total cross-node communication volume after the exchange. λ is a preset tradeoff coefficient, defaulting to 0.01, used to unify load variance and communication volume to the same order of magnitude. If... A value less than 0 indicates that the joint objective after the exchange is better than the current scheme, so the exchange operation is executed; otherwise, the exchange is abandoned, the current mapping scheme remains unchanged, and the next candidate migration group or the next pair of hot nodes is evaluated. After each exchange, the total cross-node communication volume is recalculated. If the total cross-node communication volume after the exchange still exceeds a preset threshold, the local exchange step is iteratively executed until the total cross-node communication volume after the exchange is less than the preset threshold or the maximum number of iterations is reached. The maximum number of iterations is 5×K times. After each task group is assigned to a target node, the correspondence between the task group identifier and the target node identifier is written into the mapping table until all K task groups are assigned, resulting in an optimized group-node mapping table. The group-node mapping table is stored in key-value pair format, where the key is the task group identifier and the value is the physical node identifier assigned to the task group.
[0053] Based on the final generated group-node mapping table, the complete task description of each task group is sent to the execution engine of the corresponding node, such as the Kubernetes Pod controller, via Remote Procedure Call (RPC) or message queue. After receiving the instruction, each node starts task execution and completes the overall scheduling process.
[0054] By optimizing the allocation and exchange based on communication awareness, the cross-node communication overhead is minimized while ensuring node load balancing and resource capacity constraints. Ultimately, tasks are distributed at the group level, achieving a complete closed loop from scheduling decision to actual cluster execution. This improves the accuracy of distributed integrated task scheduling decisions, the overall cluster execution efficiency, and resource utilization.
[0055] Example 2, based on the same inventive concept as the load-balancing-based distributed integrated task scheduling method in the previous examples, such as... Figure 2 As shown, this application provides a load-balanced distributed integrated task scheduling system, wherein the load-balanced distributed integrated task scheduling system includes: The task acquisition module 11 is used to acquire the original task set to be scheduled, construct a directed acyclic graph based on the data dependencies between tasks, and calculate the communication sensitivity of each task. The task partitioning module 12 is used to divide the task set into K task groups based on the directed acyclic graph and the communication sensitivity of each task, with the goal of minimizing the sum of the total cross-group communication volume and the variance of resource demand within the group, and calculate the inter-group communication matrix. The parameter calculation module 13 is used to acquire the real-time load data of each node in the distributed processing system, predict the global total load and load change rate within a future time window, and calculate the minimum number of active nodes and the dynamic overload threshold based on the prediction results. The task allocation module 14 is used to allocate the K task groups to active nodes according to the minimum number of active nodes and the dynamic overload threshold, using the group as the scheduling unit, and perform scheduling optimization in conjunction with the inter-group communication matrix to generate a group-node mapping table and send each task group to the corresponding node for execution.
[0056] Furthermore, the task acquisition module 11 is also used to: construct the directed acyclic graph based on the data dependencies between tasks, wherein each directed edge marks the total amount of communication data and the communication frequency between the two task nodes connected by the directed edge, wherein the weight of each directed edge is determined by normalized weighting based on the total amount of communication data and the communication frequency between the two tasks connected by the directed edge; calculate the sum of the edge weights between each task node and all adjacent task nodes to obtain the communication sensitivity of each task.
[0057] Furthermore, the task partitioning module 12 is also used to: use each task node in the directed acyclic graph as a clustering object, use the edge weight between tasks as a clustering distance metric, prioritize the aggregation of tasks with communication sensitivity higher than a preset sensitivity threshold, and obtain an initial aggregation result; based on the initial aggregation result, iteratively perform grouping operations with the objective of minimizing the sum of the total cross-group communication and the variance of resource demand within the group, and generate the K task groups; calculate the total cross-group communication between any two task groups for the K task groups, and obtain the inter-group communication matrix.
[0058] Furthermore, the task partitioning module 12 is also used to: when iteratively performing grouping operations with the goal of minimizing the sum of cross-group communication volume and the variance of resource demand within a group, calculate the total resource demand of each group in each iteration, split the group whose total resource demand exceeds the single node capacity limit, merge the two groups with the largest cross-group communication volume, and terminate the iteration when merging any two groups would cause the resource demand of a group to exceed the single node capacity limit or the decrease in the total cross-group communication volume is lower than a preset threshold.
[0059] Furthermore, the parameter calculation module 13 is also used to: collect the CPU utilization, memory utilization, and network bandwidth utilization of each node at fixed time intervals as real-time load data; perform time-series prediction based on the real-time load data and historical load sequences, and output the global total load prediction value and load change rate prediction value within the future time window; determine the basic number of active nodes based on the ratio of the global total load prediction value to the capacity limit of a single node; correct the basic number of active nodes based on the load change rate prediction value to obtain the minimum number of active nodes; and dynamically adjust the initial value based on the load change rate prediction value to set a dynamic overload threshold.
[0060] Furthermore, the parameter calculation module 13 is also used to: when the predicted load change rate indicates that the future load is on an upward trend, calculate the ratio of the total upward load to the upper limit of the capacity of a single node based on the rate of increase, and increase the number of redundant nodes; when the predicted load change rate indicates that the future load is on a downward trend, calculate the ratio of the total downward load to the upper limit of the capacity of a single node based on the rate of decrease, and subtract the number of recyclable nodes, but keep the minimum number of active nodes not lower than a preset safety lower limit value, which is preset according to the system fault tolerance requirements and the number of critical tasks.
[0061] Furthermore, the task allocation module 14 is also used to: extract the total resource requirements of each task group and the inter-group communication matrix; based on the minimum number of active nodes, select a set of candidate nodes to be activated from all nodes, and sort the nodes in the candidate node set from low to high according to their real-time load status; based on the inter-group communication matrix, in order of high to low inter-group communication volume between each task group, allocate each task group as a whole to the candidate node with the lowest current load and remaining capacity that can meet the total resource requirements of the task group; after each task group is allocated, update the allocated load and remaining capacity of the target node, and recalculate the load variance between nodes and the total cross-node communication volume; when all task groups are allocated, if the total cross-node communication volume exceeds a preset threshold, trigger the inter-group exchange optimization step, perform a partial exchange on the allocated task groups, and if the total cross-node communication volume still exceeds the preset threshold after the exchange, iterate the partial exchange until the total cross-node communication volume does not exceed the preset threshold, and output the optimized group-node mapping table.
[0062] Furthermore, the task allocation module 14 is also used to: calculate the total cross-group communication volume between any two different nodes, identify the node pair with the largest cross-node communication volume; among the task groups contained in the node pair with the largest cross-node communication volume, select the group with the highest communication volume with other nodes as the group to be migrated, evaluate the joint change in the load variance between nodes and the total cross-node communication volume after the group to be migrated is swapped to the node of the communication counterpart, and if the joint change is better than the current scheme, then the swap operation is performed.
[0063] In Embodiment 3, based on the same inventive concept as the load-balancing-based distributed integrated task scheduling method in the foregoing embodiments, this application also provides a computer-readable storage medium storing a computer program, which, when executed, implements the steps of the load-balancing-based distributed integrated task scheduling method described in any one of Embodiment 1 above.
[0064] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[0065] Obviously, those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.
Claims
1. A distributed integrated task scheduling method based on load balancing, characterized in that, include: Obtain the original set of tasks to be scheduled, construct a directed acyclic graph based on the data dependencies between tasks, and calculate the communication sensitivity of each task. Based on the directed acyclic graph and the communication sensitivity of each task, with the goal of minimizing the sum of the total cross-group communication volume and the variance of resource demand within the group, the task set is divided into K task groups, and the inter-group communication matrix is calculated. Obtain real-time load data of each node in the distributed processing system, predict the global total load and load change rate within a future time window, and calculate the minimum number of active nodes and dynamic overload threshold based on the prediction results. The K task groups are used as scheduling units. Based on the minimum number of active nodes and the dynamic overload threshold, each task group is assigned to an active node. The scheduling is optimized by combining the inter-group communication matrix, a group-node mapping table is generated, and each task group is sent to the corresponding node for execution.
2. The distributed integrated task scheduling method based on load balancing as described in claim 1, characterized in that, Construct a directed acyclic graph based on the data dependencies between tasks, and calculate the communication sensitivity of each task, including: The directed acyclic graph is constructed based on the data dependencies between tasks. Each directed edge marks the total amount of communication data and the communication frequency between the two task nodes connected by the directed edge. The weight of each directed edge is determined by normalized weighting of the total amount of communication data and the communication frequency between the two tasks connected by the directed edge. The communication sensitivity of each task is obtained by summing the edge weights between each task node and all its neighboring task nodes.
3. The distributed integrated task scheduling method based on load balancing as described in claim 1, characterized in that, Based on the directed acyclic graph and the communication sensitivity of each task, with the objective of minimizing the sum of the total cross-group communication volume and the variance of resource demand within the group, the task set is divided into K task groups, and the inter-group communication matrix is calculated, including: Using each task node in the directed acyclic graph as a clustering object and the edge weight between tasks as a clustering distance metric, tasks with communication sensitivity higher than a preset sensitivity threshold are prioritized for aggregation to obtain the initial aggregation result. Based on the initial aggregation results, the grouping operation is iteratively executed with the goal of minimizing the sum of the total cross-group communication volume and the variance of resource demand within the group, thereby generating the K task groups; The total amount of cross-group communication between any two task groups is calculated for the K task groups to obtain the inter-group communication matrix.
4. The distributed integrated task scheduling method based on load balancing as described in claim 3, characterized in that, When performing grouping operations iteratively with the goal of minimizing the sum of cross-group communication volume and the variance of resource demand within a group, the total resource demand of each group is calculated in each iteration. Groups whose total resource demand exceeds the single-node capacity limit are split up, and the two groups with the largest cross-group communication volume are merged. The iteration ends when merging any two groups would cause the resource demand of a group to exceed the single-node capacity limit or the decrease in the total cross-group communication volume is lower than a preset threshold.
5. The distributed integrated task scheduling method based on load balancing as described in claim 1, characterized in that, Acquire real-time load data for each node in the distributed processing system, predict the total global load and load change rate within a future time window, and calculate the minimum number of active nodes and dynamic overload threshold based on the prediction results, including: The CPU utilization, memory utilization, and network bandwidth utilization of each node are collected at fixed time intervals as real-time load data. Based on the real-time load data, time-series prediction is performed based on historical load sequences, and the global total load prediction value and load change rate prediction value within the future time window are output. The basic number of active nodes is determined based on the ratio of the predicted global total load to the capacity limit of a single node; The minimum number of active nodes is obtained by correcting the basic number of active nodes based on the predicted load change rate. The initial value is dynamically adjusted based on the predicted load change rate, and a dynamic overload threshold is set.
6. The distributed integrated task scheduling method based on load balancing as described in claim 5, characterized in that, The minimum number of active nodes is obtained by correcting the basic active node count based on the predicted load change rate, including: When the predicted load change rate indicates that the future load is on the rise, the number of redundant nodes is increased based on the ratio of the total increase load to the capacity limit of a single node, according to the increase rate on the basis of the basic number of active nodes. When the predicted load change rate indicates that the future load is trending downward, the ratio of the total load decrease to the upper capacity limit of a single node is calculated based on the basic number of active nodes, minus the number of recyclable nodes, while keeping the minimum number of active nodes no lower than a preset safety lower limit. The safety lower limit is preset according to the system fault tolerance requirements and the number of critical tasks.
7. The distributed integrated task scheduling method based on load balancing as described in claim 1, characterized in that, The K task groups are organized into groups as scheduling units. Based on the minimum number of active nodes and the dynamic overload threshold, each task group is assigned to an active node. Scheduling optimization is performed using the inter-group communication matrix, generating a group-node mapping table, including: Extract the total resource requirements of each task group and the inter-group communication matrix; Based on the minimum number of active nodes, a set of candidate nodes to be activated is selected from all nodes, and the nodes in the candidate node set are sorted from low to high according to their real-time load status. Based on the inter-group communication matrix, each task group is treated as a whole and assigned to the candidate node with the lowest current load and the remaining capacity to meet the total resource requirements of the task group, in descending order of inter-group communication volume. After each task group is assigned, update the assigned load and remaining capacity of the target node, and recalculate the load variance between nodes and the total amount of cross-node communication. Once all task groups have been assigned, if the total amount of cross-node communication exceeds a preset threshold, an inter-group exchange optimization step is triggered. The assigned task groups are partially exchanged. If the total amount of cross-node communication still exceeds the preset threshold after the exchange, the partial exchange is iteratively executed until the total amount of cross-node communication does not exceed the preset threshold, and the optimized group-node mapping table is output.
8. The distributed integrated task scheduling method based on load balancing as described in claim 7, characterized in that, Trigger the inter-group swap optimization step to perform a partial swap of the assigned task groups, including: Calculate the total cross-group communication volume between any two different nodes and identify the node pairs with the largest cross-node communication volume; Among the task groups contained in the node pair with the largest cross-node communication volume, the group with the highest communication volume with other nodes is selected as the migration group. The joint change in the load variance between nodes and the total cross-node communication volume after the migration group is swapped to the node of the communication counterpart is evaluated. If the joint change is better than the current scheme, the swap operation is performed.
9. A distributed integrated task scheduling system based on load balancing, characterized in that, The steps for implementing the load-balancing-based distributed integrated task scheduling method according to any one of claims 1 to 8 include: The task acquisition module is used to acquire the original set of tasks to be scheduled, construct a directed acyclic graph based on the data dependencies between tasks, and calculate the communication sensitivity of each task. The task partitioning module is used to divide the task set into K task groups based on the directed acyclic graph and the communication sensitivity of each task, with the goal of minimizing the sum of the total cross-group communication and the variance of resource demand within the group, and to calculate the inter-group communication matrix. The parameter calculation module is used to obtain real-time load data of each node in the distributed processing system, predict the global total load and load change rate within a future time window, and calculate the minimum number of active nodes and dynamic overload threshold based on the prediction results. The task allocation module is used to allocate the K task groups to active nodes according to the minimum number of active nodes and the dynamic overload threshold, using the group as the scheduling unit. It also performs scheduling optimization in conjunction with the inter-group communication matrix, generates a group-node mapping table, and sends each task group to the corresponding node for execution.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the steps of the load-balancing-based distributed integrated task scheduling method according to any one of claims 1 to 8.
