A scheduling method

Abstract

This invention provides a scheduling method, relating to the field of data processing technology. The method includes: acquiring metadata of a task set, dividing and identifying long tasks and short tasks to form classified data; calculating a minimum continuous resource window for each long task, locking a window set, and obtaining mapping data from long tasks to windows; identifying time resource gaps outside the windows based on available resource status, aggregating short tasks by resource and duration, and generating mapping data from gaps to short task packages; matching and dispatching tasks, triggering the corresponding long task if the window meets the conditions, and filling short task packages in the order of gaps if the window is not ready, generating a scheduling plan; performing online monitoring and rebalancing based on the execution metrics of the scheduling plan, releasing unstarted short task packages and rearranging gaps and window boundaries when the window is disturbed or a long task times out, generating an updated mapping and an updated plan; generating execution results based on the scheduling plan or the updated plan. This invention improves the accuracy of the scheduling method.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a scheduling method. Background Technology

[0002] In existing technologies, commonly used scheduling methods primarily employ priority queue-based allocation mechanisms. For example, in a multitasking operating system, the system inserts tasks into the scheduling queue sequentially based on their priorities, and the scheduler allocates CPU resources according to priority order. This scheduling method typically utilizes mechanisms such as time-slice round-robin, preemption, or non-preemption to ensure that high-priority tasks can obtain processing rights first, thereby satisfying the response needs of different tasks.

[0003] However, in large-scale distributed computing scenarios, such as batch job scheduling in data centers, existing priority queue mechanisms suffer from resource fragmentation. Specifically, if the scheduling queue contains a large number of short tasks and a small number of long tasks simultaneously, the system tends to prioritize short tasks when allocating resources, causing long tasks to be unable to obtain the continuous resources they need for a long time, resulting in a "task starvation" phenomenon. Taking a certain cloud computing platform as an example, the scheduler repeatedly assigned small data processing tasks, causing a model training job that required a large number of nodes to be in a waiting state for a long time, seriously affecting the timely completion of the job. Summary of the Invention

[0004] The purpose of this invention is to provide a scheduling method that aims to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] A scheduling method, the method comprising:

[0007] Obtain the first set of metadata for the task set, including resource requirements, duration, and priority; classify and identify long tasks and short tasks based on the first set of metadata to form the first set of category data;

[0008] Based on the classification data one, calculate the minimum continuous resource window for each long task, lock the window set, and obtain the mapping data two from long tasks to windows;

[0009] Based on the mapping data 2 and the available resource status, identify time resource gaps outside the window, aggregate short tasks by resource and duration, and generate mapping data 3 of gaps and short task packages;

[0010] Based on the matching and dispatching of mapping data 2 and mapping data 3, if the window meets the conditions, the corresponding long task is triggered; if the window is not ready, the short task package is filled in the gap order to generate a scheduling plan.

[0011] Online monitoring and rebalancing are performed based on the execution metrics of the scheduling plan. When a window is disturbed or a long task times out, the unstarted short task packages are released and the gaps and window boundaries are rearranged to generate update mappings and update plans.

[0012] The execution results are generated based on the scheduling plan or update plan, and the execution results are written back to metadata 1 as input for the next round of windowing and matching.

[0013] Preferably, based on classification data one, a minimum continuous resource window is calculated for each long task, the window set is locked, and mapping data two from long tasks to windows is obtained, including:

[0014] Based on the classification data and available resource status, candidate draft windows are determined for each long task according to resource requirements and continuous duration, and then summarized to form a candidate window set.

[0015] Based on the candidate window set, the node location affinity, number of racks spanned, data locality and interconnect bandwidth are verified item by item to generate a feasible window set;

[0016] Based on the set of feasible windows, select the smallest window that satisfies the resource quantity and continuous duration at one time. If there are multiple windows, prioritize the window with a smaller number of racks and fewer average hops between nodes to generate the smallest continuous resource window.

[0017] Based on the minimum continuous resource time window, set the protection period and boundary attributes, and mark whether backfilling is allowed and the minimum start unit to generate a set of labeled windows. The protection period is the time period within the time window during which tasks that conflict with long task resources are prohibited from being assigned. The boundary attributes are the start and end times of the window and the range of nodes. The minimum start unit is a combination constraint of the minimum number of nodes and the minimum duration.

[0018] Based on the set of labeled windows, the long task identifier is mapped to the corresponding minimum continuous resource window, protection period, and boundary attributes to obtain the second mapping data from long task to window.

[0019] Preferably, based on mapping data two and the available resource status, time resource gaps outside the window are identified, short tasks are aggregated by resource and duration, and mapping data three of gaps and short task packages is generated, including:

[0020] Based on the mapping data and available resource status, the nodes and time slices not covered by the minimum continuous resource time window are scanned to generate an initial gap set;

[0021] Based on the initial set of gaps, each gap is measured by the number of available nodes and the duration of continuous intervals. Gaps that are insufficient to support long tasks are marked as candidate regions for short tasks, and a list of candidate gaps is generated.

[0022] Based on the candidate gap list, the resource requirements and expected duration of short tasks are aggregated to form a draft of the short task package;

[0023] Based on the initial draft of the short task package, the node location distribution and network bandwidth of the aggregated short task package are checked one by one to see if they meet the preset threshold. If there are multiple feasible combinations, the combination with larger bandwidth margin and better data locality is selected first to generate a set of short task packages.

[0024] Based on the candidate gaps and the set of short task packets, a mapping relationship is established between each short task packet and the corresponding gap, resulting in the mapping data between gaps and short task packets.

[0025] Preferably, dispatch is performed based on matching mapping data two and mapping data three. If the window meets the conditions, the corresponding long task is triggered; if the window is not ready, short task packets are filled in the gap sequence to generate a scheduling plan, including:

[0026] Based on mapping data 2, mapping data 3 and available resource status, a readiness check is performed according to the following criteria: resource quantity meets requirements, node location matches, network bandwidth is not lower than a preset threshold, and protection period integrity is ensured, and a ready window list is generated.

[0027] Based on the ready window list, generate the corresponding long task start command and register the start time, resource location and protection period occupation to form a long task scheduling record;

[0028] Based on the incomplete window and mapping data 3, short task packages are selected for filling according to the time resource gap from early to late and the principle of small to large withdrawal cost, and a backfill list is generated.

[0029] Based on the long task scheduling record and the backfill list, identify conflicting short task packages that have been dispatched and whose startup has been paused, move them into the queue to be reclaimed and retain the task parameters and allocated resource identifiers to form a list to be reclaimed.

[0030] Based on the long task scheduling records, the backfill list, and the pending recycling list, a scheduling plan containing trigger records and the pending recycling list is formed.

[0031] Preferably, online monitoring and rebalancing are performed based on the execution metrics of the scheduling plan. When a window is intruded upon or a long task times out, unstarted short task packages are released and gaps and window boundaries are rearranged, generating update mappings and update plans, including:

[0032] Based on the execution metrics of the scheduling plan, the deviation between the planned start and the actual start is calculated according to the preset monitoring cycle, the number of times the protection period is violated is counted, and the resource occupation continuity of adjacent time slices is checked to generate monitoring results;

[0033] Based on the monitoring results, if the soft threshold condition is met, the backfilling order is adjusted from small to large according to the withdrawal cost, and the affected window boundary is adjusted at the granularity of one time slice or one node to restore readiness and generate the first update plan. The soft threshold condition is that the start-up deviation does not exceed one time slice and the protection period is not violated more than once.

[0034] Based on the monitoring results meeting the hard threshold conditions, release the unstarted short task packages and rearrange the time resource gaps according to the principle of prioritizing those closer to the target window under the same conditions, starting from the earliest to the latest time, to generate a second update scheme. The hard threshold conditions are that the start deviation exceeds one time slice for two consecutive monitoring cycles, or the protection period is violated twice or more in a single cycle, or there is a long task expected to time out.

[0035] Based on the first and second update schemes, simulate the layout and record the expected start time and average resource consumption to generate candidate scheme evaluation results;

[0036] Based on the evaluation results of the candidate schemes, the target scheme that can meet the long task startup requirements earliest and whose expected average resource consumption is not lower than the actual average resource consumption of the previous monitoring period is selected, and mapping data two and mapping data three are updated and an update plan is generated.

[0037] Preferably, based on the classification data and available resource status, candidate draft windows are determined for each long task according to resource requirements and continuous duration, and these are aggregated to form a candidate window set, including:

[0038] Based on the first classification data, extract the number of nodes required for long tasks and the required continuous duration, divide the available resource status into multiple continuous time slices, and obtain time slice resource data.

[0039] Based on the time slice resource data, the resource segments that can meet the node requirements are retrieved within each time slice interval, and then filtered according to the continuous duration constraint to obtain candidate resource segments that meet the conditions.

[0040] Based on the candidate resource fragments, the start time, end time, and node range are combined to determine the corresponding window candidate drafts;

[0041] Based on multiple initial drafts of candidate windows, they are summarized according to task identifiers to form a candidate window set.

[0042] Preferably, based on the minimum continuous resource time window, the protection period and boundary attributes are set, and whether backfilling is allowed and the minimum start unit are marked, generating a set of labeled windows, including:

[0043] Based on the minimum continuous resource time window and historical execution results, determine the initial protection period, minimum start unit, and boundary attributes, and generate an initial value label set;

[0044] Based on the number of times the protection period was violated and the start-up deviation in the previous monitoring cycle, if the number of violations is one and the start-up deviation does not exceed one time slice, the protection period will be extended by one time slice. If the number of violations is zero, the protection period will be shortened by one time slice and the protection period will not be less than one time slice. At the same time, the minimum start-up unit will not be less than the minimum number of nodes and the minimum duration, and an adjustment tag set will be generated.

[0045] Based on the adjusted label set, indicate whether backfilling is allowed: if the protection period was not breached in the previous monitoring cycle and the start-up deviation is zero, mark backfilling as allowed and limit backfilling to not occupy the minimum start-up unit; otherwise, mark backfilling as not allowed and generate a set of labeled windows.

[0046] Preferably, based on the candidate gap list, the resource requirements and expected duration of short tasks are aggregated to form a draft short task package, including:

[0047] Based on the candidate gap list, extract the number of available nodes and the continuous duration of each gap to obtain the gap attribute record;

[0048] Based on the gap attribute records, the resource requirements of short tasks are compared one by one, and short tasks with no more than the gap attribute number and duration are selected to form a candidate short task set.

[0049] Based on the candidate short task set, the tasks are aggregated according to a priority rule. The priority rule is to select the combination whose cumulative execution time is closest to the gap length, without exceeding the gap duration, to generate the aggregated short task combination.

[0050] Based on the aggregated short task combinations, generate a draft short task package and record the number of nodes occupied, the estimated total execution time, and the corresponding gap identifier.

[0051] Preferably, based on the window ready list and the execution metrics of the scheduling plan, the deviation between planned startup and actual startup is calculated according to a preset monitoring cycle, the number of times the protection period is violated is counted, and the resource usage continuity of adjacent time slices is checked, generating monitoring results, including:

[0052] Based on the execution metrics of the scheduling plan, extract the planned start time and actual start time of each long task, calculate the difference between the two and record it as the start deviation;

[0053] Based on the execution metrics, the number of times the protection period was interrupted by short tasks is counted, and the protection period violation record is obtained.

[0054] Based on the execution metrics, the resource usage ratio of adjacent time slices is compared one by one. If the usage ratio continuously decreases or fluctuates beyond a preset threshold, it is marked as a continuity anomaly.

[0055] Monitoring results are generated by summarizing the start-up deviation, protection period violation records, and continuous abnormal situations.

[0056] Preferably, based on the first update scheme and the second update scheme, a simulated layout is performed and the expected start time and average resource consumption are recorded to generate candidate scheme evaluation results, including:

[0057] Based on the first update scheme, the second update scheme, and the available resource status, a simulated resource view is constructed as input, wherein the simulated resource view is a simulated environment copied from the current available resource status;

[0058] Based on the simulated resource view, long tasks in the ready window are placed first in the order of time slices from early to late, followed by short task packages for backfilling. When placing each item, the resource quantity, node position and correspondence with mapping data two and mapping data three are checked one by one, the network bandwidth is not lower than the preset threshold and the integrity of the protection period is verified, the expected start time of each long task and the resource occupancy ratio of each time slice are recorded, and simulated layout records of the two schemes are generated.

[0059] Based on the simulated layout records, the resource usage ratio of each scheme is accumulated for each time slice, and then the average resource usage value is generated by dividing it equally according to the number of time slices. At the same time, it is calculated whether all verification items have passed and the data locality index is summarized to generate candidate scheme evaluation data.

[0060] Based on the candidate solution evaluation data, an evaluation item is generated for each solution, including the expected start time, average resource consumption, verification conclusions, and data locality summary, thus generating the candidate solution evaluation results.

[0061] The above-described solution of the present invention has at least the following beneficial effects:

[0062] First, by acquiring the metadata information of the task set and classifying long and short tasks, the different resource and duration requirements of tasks can be clearly defined at the initial stage of scheduling. This avoids the scheduling blind spots caused by the traditional priority queue's "one-size-fits-all" reliance on priority sorting. By distinguishing between task identifiers, resource requirements, and priority parameters, the system has a more granular scheduling basis when facing diverse tasks.

[0063] Secondly, by calculating the minimum continuous resource window for long tasks and generating a window set, it is possible to ensure that long tasks have sufficient continuous resources to meet their needs during scheduling, avoiding the "task starvation" problem where long tasks cannot start due to repeated insertion of short tasks. For example, in scenarios where model training tasks require the continuous use of multiple nodes, this solution can lock appropriate time and node resources in advance to ensure that long tasks are executed promptly once resources are available.

[0064] Furthermore, by identifying time resource gaps outside the window and aggregating and filling them with short tasks, fragmented resources can be effectively utilized, improving the overall resource utilization of the system. Even if long tasks fail to start immediately, short tasks can run efficiently during the gaps, thus balancing the protection of long tasks with the response speed of short tasks.

[0065] Furthermore, by introducing execution metrics and performing online monitoring and rebalancing during the scheduling process, unstarted short tasks can be released promptly when window intrusion or long task timeouts are detected, and resource boundaries can be dynamically adjusted to ensure the start conditions of long tasks. This adaptive mechanism enhances the robustness of system scheduling, enabling it to maintain stable scheduling performance even in complex distributed environments.

[0066] Finally, by writing the execution results of the scheduling plan or update plan back to the metadata as input for the next round of scheduling, a closed-loop optimization mechanism is formed, enabling the system to improve its scheduling strategy through continuous iteration. For example, when the system detects that a certain type of task is repeatedly blocked due to insufficient protection time, it can automatically adjust the protection time in the next round, thereby reducing conflicts and waiting.

[0067] In summary, this invention can ensure rapid response for short tasks while avoiding long-term blocking of long tasks due to resource fragmentation, significantly improving resource utilization and overall throughput in large-scale distributed computing environments. It is particularly suitable for batch job scheduling in data centers and large-scale model training scenarios in cloud platforms. Attached Figure Description

[0068] Figure 1 This is a flowchart of a scheduling method provided by an embodiment of the present invention. Detailed Implementation

[0069] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0070] like Figure 1 As shown, an embodiment of the present invention proposes a scheduling method, the method comprising:

[0071] Obtain the first set of metadata for the task set, including resource requirements, duration, and priority; classify and identify long tasks and short tasks based on the first set of metadata to form the first set of category data;

[0072] Based on the classification data one, calculate the minimum continuous resource window for each long task, lock the window set, and obtain the mapping data two from long tasks to windows;

[0073] Based on the mapping data 2 and the available resource status, identify time resource gaps outside the window, aggregate short tasks by resource and duration, and generate mapping data 3 of gaps and short task packages;

[0074] Based on the matching and dispatching of mapping data 2 and mapping data 3, if the window meets the conditions, the corresponding long task is triggered; if the window is not ready, the short task package is filled in the gap order to generate a scheduling plan.

[0075] Online monitoring and rebalancing are performed based on the execution metrics of the scheduling plan. When a window is disturbed or a long task times out, the unstarted short task packages are released and the gaps and window boundaries are rearranged to generate update mappings and update plans.

[0076] The execution results are generated based on the scheduling plan or update plan, and the execution results are written back to metadata 1 as input for the next round of windowing and matching, so as to maintain the response of short tasks and avoid the starvation of long tasks, reduce resource fragmentation and improve throughput.

[0077] In this embodiment of the invention, by acquiring the metadata of the task set and classifying and identifying tasks as long and short tasks based on resource requirements, duration, and priority, the system can have a clear profile of task attributes at the initial scheduling stage, thereby providing an accurate basis for subsequent resource allocation. For example, in a data center environment, if a task requires ten nodes and lasts for more than one hour, it is classified as a long task, while a task requiring only two nodes and with an execution time of five minutes is classified as a short task. Such classification can effectively avoid blindness in resource scheduling.

[0078] By calculating the minimum continuous resource window based on the first set of categorized data, locking the window set, and generating the second set of mapping data from long tasks to windows, it can be ensured that long tasks can obtain continuous resources to meet their execution requirements during resource allocation. This avoids situations where resources are split among a large number of short tasks, preventing timely startup. For example, in model training tasks, if the system can lock a window of at least ten consecutive nodes in advance, it can ensure that the task can be quickly put into execution when resources are idle, avoiding long waiting times.

[0079] By identifying time resource gaps outside the window based on mapping data two and available resource status, and aggregating short tasks by resource and duration to generate mapping data three between gaps and short task packages, fragmented computing resources can be fully utilized, improving overall resource utilization. For example, when the available time for a batch of nodes is only three minutes, combining multiple short tasks into a short task package to fill the gap can prevent the gap from being wasted.

[0080] By matching and dispatching tasks based on mapping data two and mapping data three, long tasks are triggered when the window meets the conditions, and short task packets are filled in the order of gaps when the window is not ready. This generates a scheduling plan and achieves a dynamic balance between long and short tasks, ensuring that long tasks are not starved while short tasks receive timely responses. For example, when a locked long task window is detected to be ready, the system immediately triggers the execution of the long task; while when the window does not yet meet the conditions, short tasks are scheduled to execute first to avoid resource idleness.

[0081] By performing online monitoring and rebalancing based on the execution metrics of the scheduling plan, the system can release unstarted short task packages and rearrange gaps and window boundaries when windows are intruded or long tasks time out, generating update maps and update plans. This ensures the robustness and flexibility of scheduling. For example, when a short task timeout causes it to occupy the protection period of a long task, the system can immediately release the short task and adjust its boundaries to ensure that the long task starts on schedule.

[0082] By generating execution results based on the scheduling plan or update plan and writing these results back to metadata 1 as input for the next round of windowing and matching, a closed-loop optimization mechanism can be formed, enabling the system to gradually improve its strategy in each iteration of scheduling. For example, when it is found that a certain type of task repeatedly triggers the protection period violation, the system can automatically extend the protection period in the next round of windowing, thereby reducing the occurrence of conflicts.

[0083] In summary, the embodiments of the present invention can ensure rapid response for short tasks while avoiding long tasks from starving due to resource fragmentation, significantly improving resource utilization and overall throughput. They are particularly suitable for scenarios with high requirements for task fairness and system efficiency, such as batch job scheduling in data centers and model training on cloud platforms.

[0084] In one optional implementation, the metadata includes at least the following fields: task identifier, resource requirements, expected duration, and priority parameters; whereby resource requirements can be further refined into the number of compute nodes, CPU / GPU cores, memory, and bandwidth requirements. Execution metrics include at least startup skew, number of protection period violations, resource utilization continuity, and throughput, used to measure the execution quality of the scheduling plan. These parameters can be submitted by the user, set by the administrator, or automatically generated by the system based on historical data.

[0085] In a preferred embodiment of the present invention, the minimum continuous resource window is calculated for each long task based on classification data one, the window set is locked, and mapping data two from long tasks to windows is obtained, including:

[0086] Based on the classification data and available resource status, candidate draft windows are determined for each long task according to resource requirements and continuous duration, and then summarized to form a candidate window set.

[0087] Based on the candidate window set, the node location affinity, number of racks spanned, data locality and interconnect bandwidth are verified item by item to generate a feasible window set;

[0088] Based on the set of feasible windows, select the smallest window that satisfies the resource quantity and continuous duration at one time. If there are multiple windows, prioritize the window with a smaller number of racks and a smaller average number of hops between nodes to generate the smallest continuous resource window. The average number of hops between nodes is the average number of forwarding times according to the switching device level.

[0089] Based on the minimum continuous resource time window, set the protection period and boundary attributes, and mark whether backfilling is allowed and the minimum start unit to generate a set of labeled windows. The protection period is the time period within the time window during which tasks that conflict with long task resources are prohibited from being assigned. The boundary attributes are the start and end times of the window and the range of nodes. The minimum start unit is a combination constraint of the minimum number of nodes and the minimum duration.

[0090] Based on the set of labeled windows, the long task identifier is mapped to the corresponding minimum continuous resource window, protection period, and boundary attributes to obtain the second mapping data from long task to window.

[0091] In this embodiment of the invention, a minimum continuous resource window is calculated for each long task based on classification data one, a window set is locked, and mapping data two from long tasks to windows is generated. Through step-by-step filtering and comparison, it can be effectively ensured that long tasks obtain the required continuous resources. By establishing a candidate window set, the range of available resources for different tasks can be clearly defined before task scheduling. On this basis, the node location affinity, number of cross-racks, data locality, and interconnection bandwidth are verified item by item, which can prevent long tasks from being incorrectly scheduled under conditions of cross-racks, insufficient bandwidth, or poor data locality. For example, a deep learning training task that requires twenty nodes will lead to communication latency and efficiency reduction if there are too many cross-racks. This verification process can eliminate unsuitable candidate windows, thereby obtaining feasible windows that meet the execution requirements. Furthermore, by selecting the minimum window that meets the resource quantity and continuous duration and configuring a protection period and minimum start unit for it, it can be ensured that the window is not occupied by short tasks during task execution. For example, in a multi-rack cluster, the system can lock twenty nodes within the same rack and set a twenty-minute protection period for them, thereby ensuring that the task can be started and executed continuously. Finally, the mapping relationship between long tasks and their corresponding windows is saved as Data 2, providing a reliable basis for subsequent scheduling and dispatch, thereby improving the success rate and execution stability of long task scheduling.

[0092] The method for generating candidate draft windows is as follows: based on the resource requirements and duration in Classification Data 1, the available resource status is divided into time slices, and candidate resource segments that can continuously provide the required number of nodes within the target duration are selected and combined to form candidate draft windows.

[0093] The method for calculating the average hop count between nodes is as follows: In the network topology, all nodes used by the task are paired up, and the number of times the communication path is forwarded by the switching devices is calculated. The average number of hops between nodes is then taken as the average number of hops. The smaller the value, the lower the communication latency.

[0094] Protection period setting: The initial protection period is set to at least one full execution cycle (e.g., 10 minutes or 1 hour) before the expected task starts, to prevent short tasks from occupying the resources of long tasks. It can be dynamically adjusted based on historical scheduling feedback.

[0095] Minimum start unit setting: determined by the minimum resource requirements of the task, and is "number of nodes × minimum duration". For example, if a task requires at least 8 nodes and 15 minutes, then the minimum start unit is "8 nodes × 15 minutes".

[0096] The initial value of the protection period can be set by the system policy to 0.5 times the estimated task duration, but not less than 10 minutes, and then dynamically adjusted based on historical running data. The minimum startup unit must always be satisfied when the window shrinks or merges to ensure that long tasks will not fail to start due to splitting.

[0097] In a preferred embodiment of the present invention, based on the mapping data two and the available resource status, time resource gaps outside the window are identified, short tasks are aggregated according to resources and duration, and mapping data three of gaps and short task packages is generated, including:

[0098] Based on the mapping data and available resource status, the nodes and time slices not covered by the minimum continuous resource time window are scanned to generate an initial gap set;

[0099] Based on the initial set of gaps, each gap is measured by the number of available nodes and the duration of continuous intervals. Gaps that are insufficient to support long tasks are marked as candidate regions for short tasks, and a list of candidate gaps is generated.

[0100] Based on the candidate gap list, the resource requirements and expected duration of short tasks are aggregated to form a draft of the short task package;

[0101] Based on the initial draft of the short task package, the node location distribution and network bandwidth of the aggregated short task package are checked one by one to see if they meet the preset threshold. If there are multiple feasible combinations, the combination with larger bandwidth margin and better data locality is selected first to generate a set of short task packages. Better data locality means fewer average hops between nodes and required data or less cross-node transmission volume.

[0102] Based on the candidate gaps and the set of short task packets, a mapping relationship is established between each short task packet and the corresponding gap, resulting in the mapping data between gaps and short task packets.

[0103] In this embodiment of the invention, by identifying time resource gaps outside the available resource status window based on mapping data two, and aggregating short tasks according to resource and duration, mapping data three of gaps and short task packages is generated, which can avoid the idle waste of scattered resources. By scanning nodes and time slices not covered by the minimum continuous resource time window, the system can accurately identify gaps that long tasks cannot use; furthermore, by marking gaps that are insufficient to support long tasks as short task candidate areas, it can ensure that these fragmented resources are used reasonably without affecting the execution space of long tasks. By aggregating short tasks within candidate gaps and prioritizing the combination of tasks with a cumulative execution duration close to the gap length, the residual time at the tail of the gap can be reduced, improving resource utilization. For example, when the gap of a node is 5 minutes, the system will prioritize selecting two short tasks with a cumulative duration of 4 minutes and 50 seconds, rather than selecting only one short task with a duration of 2 minutes, thereby making fuller use of the gap. The resulting gap and the mapping data of the short task packets provide a precise basis for subsequent dispatch and scheduling, enabling the system to improve the response speed and overall throughput of short tasks while ensuring the fairness of long tasks.

[0104] Nodes and time slices not covered by the minimum continuous resource window are identified through a slice-by-slice scanning method to form an initial set of gaps. The system first divides the entire time axis into time slices with fixed granularity, such as 5 minutes per slice; then, within each time slice, it marks the set of nodes not assigned to long tasks and merges consecutive adjacent time slices to obtain the initial set of gaps.

[0105] For each gap, the system measures based on the number of available nodes and the continuous duration. When the number of nodes is less than the requirements of long tasks or the duration is insufficient to cover the requirements of long tasks, the gap is marked as a candidate area for short tasks and added to the candidate gap list.

[0106] During the short task aggregation phase, the system extracts a set of candidate short tasks and aggregates them according to the following priority rules:

[0107] Prioritize combinations whose cumulative execution time is closest to the gap duration;

[0108] When the cumulative execution time is the same, the combination with better data locality should be selected first;

[0109] If they are still the same, then the combination with the larger bandwidth margin should be selected first.

[0110] For example, in a given gap with 4 nodes and a continuous duration of 10 minutes, the candidate short tasks include two 2-minute tasks and one 6-minute task. The system will prioritize aggregating these into "2 minutes + 2 minutes + 6 minutes", accumulating to 10 minutes, to maximize the utilization of the gap. Each record in the final generated short task package contains: gap ID, short task ID set, cumulative duration, node range, and data locality parameters.

[0111] In a preferred embodiment of the present invention, dispatch is performed based on the matching of mapping data two and mapping data three. If a window meets the conditions, the corresponding long task is triggered; if a window is not ready, short task packets are filled in the order of gaps to generate a scheduling plan, including:

[0112] Based on mapping data 2, mapping data 3 and available resource status, a readiness check is performed according to the following criteria: resource quantity meets requirements, node location matches, network bandwidth is not lower than a preset threshold, and protection period integrity is ensured, and a ready window list is generated.

[0113] Based on the ready window list, generate the corresponding long task start command and register the start time, resource location and protection period occupation to form a long task scheduling record;

[0114] Based on the incomplete window and mapping data 3, short task packages are selected for filling according to the time resource gap from early to late and the principle of small to large withdrawal cost, and a backfill list is generated.

[0115] Based on the long task scheduling record and the backfill list, identify conflicting short task packages that have been dispatched and whose startup has been paused, move them into the queue to be reclaimed and retain the task parameters and allocated resource identifiers to form a list to be reclaimed.

[0116] Based on the long task scheduling records, the backfill list, and the pending recycling list, a scheduling plan containing trigger records and the pending recycling list is formed.

[0117] In this embodiment of the invention, matching and dispatching are performed based on mapping data two and mapping data three. Long tasks are triggered when the window meets the conditions; when the window is not ready, short task packets are filled in the gap order. This maximizes resource utilization and balances fairness in task scheduling. The readiness check mechanism ensures that long tasks start with sufficient resources, reasonable node location matching, and network bandwidth meeting preset thresholds, while maintaining a complete protection period, thus preventing long tasks from being incorrectly started under suboptimal conditions. For example, in a cluster environment, if the bandwidth is insufficient to support distributed training, the system will delay the start of the long task to avoid failure. For short tasks, filling is selected based on the time resource gap order and the size of the withdrawal cost, fully utilizing fragmented resources while reducing the cost of future withdrawals. For example, filling two short task packets, each requiring only two minutes, in a five-minute gap ensures that the gap time is fully utilized without affecting the start of subsequent long tasks. When a conflict occurs between a long task and a short task, the system pauses the conflicting short task and moves it to a queue for recycling, while saving its task parameters and allocated resource identifiers, thus supporting rapid recovery later. This mechanism not only avoids the problem of long tasks being starved, but also ensures that short tasks are not completely discarded due to interruption. Finally, a complete scheduling plan is generated by merging the long task scheduling records, the backfill list, and the list to be recycled, ensuring the efficient operation and stability of the system in dynamic environments.

[0118] Among them, the readiness check details are as follows:

[0119] Resource quantity: The number of nodes required to meet the task requirements.

[0120] Node location: Whether affinity or topological constraints are satisfied.

[0121] Network bandwidth: Is it not lower than the preset threshold (e.g., 10Gbps)?

[0122] Protection period: Whether it is intact and undamaged.

[0123] Trigger record fields include: task ID, window ID, trigger time, trigger reason (e.g., window ready, short task withdrawal), and expected execution duration.

[0124] The queue to be recycled is implemented by using a task queue structure to store the metadata of the paused tasks, including task parameters (number of nodes, estimated duration), allocated resource identifiers (node ​​ID, time slice), and interruption reasons, for subsequent recovery.

[0125] The queue to be recycled is stored in the central task management module of the distributed scheduling system, supporting task pause, resumption, and migration. The queue adopts a first-in, first-out (FIFO) structure, and each record includes the task ID, allocation node, pause time, and resumption strategy.

[0126] In a preferred embodiment of the present invention, online monitoring and rebalancing are performed based on the execution metrics of the scheduling plan. When a window is disturbed or a long task times out, unstarted short task packages are released and gaps and window boundaries are rearranged, generating update mappings and update plans, including:

[0127] Based on the execution metrics of the scheduling plan, the deviation between the planned start and the actual start is calculated according to the preset monitoring cycle, the number of times the protection period is violated is counted, and the resource occupation continuity of adjacent time slices is checked to generate monitoring results;

[0128] Based on the monitoring results, if the soft threshold condition is met, the backfilling order is adjusted from small to large according to the withdrawal cost, and the affected window boundary is adjusted at the granularity of one time slice or one node to restore readiness and generate the first update plan. The soft threshold condition is that the start-up deviation does not exceed one time slice and the protection period is not violated more than once.

[0129] Based on the monitoring results meeting the hard threshold conditions, release the unstarted short task packages and rearrange the time resource gaps according to the principle of prioritizing those closer to the target window under the same conditions, starting from the earliest to the latest time, to generate a second update scheme. The hard threshold conditions are that the start deviation exceeds one time slice for two consecutive monitoring cycles, or the protection period is violated twice or more in a single cycle, or there is a long task expected to time out.

[0130] Based on the first and second update schemes, simulate the layout and record the expected start time and average resource consumption to generate candidate scheme evaluation results;

[0131] Based on the evaluation results of the candidate schemes, the target scheme that can meet the long task startup requirements earliest and whose expected average resource consumption is not lower than the actual average resource consumption of the previous monitoring period is selected, and mapping data two and mapping data three are updated and an update plan is generated.

[0132] In this embodiment of the invention, online monitoring and rebalancing are performed based on the execution metrics of the scheduling plan. When a window is disturbed or a long task times out, the system releases unstarted short task packages and rearranges the gaps and window boundaries, which can significantly improve the stability and adaptability of scheduling. By calculating the deviation between planned and actual startup according to a preset monitoring period, counting the number of times the protection period is violated, and checking the continuity of resource occupation in adjacent time slices, the system can monitor abnormal situations in the task execution process in real time. For example, when the actual startup time of a long task deviates from the plan by more than one time slice twice consecutively, the system immediately determines that there is a resource scheduling risk. Furthermore, when the monitoring results meet the soft threshold condition, the system can restore the startability of long tasks without significantly disturbing the existing scheduling by adjusting the backfilling order and making minor adjustments to the window boundaries, thereby reducing the impact on the overall throughput. When the monitoring results meet the hard threshold condition, the system can ensure that long tasks have priority execution by releasing unstarted short task packages and reordering time resource gaps. For example, if a task requiring one hour of training cannot start due to short tasks continuously consuming resources, the system can immediately cancel some short task packages to restore the long task window. Finally, by simulating and evaluating the update schemes, the scheme that can start the long task earliest and whose average resource utilization rate is not lower than that of the previous cycle is selected, achieving dual optimization of resource utilization and task fairness.

[0133] The monitoring period is typically set to a time slice or an integer multiple thereof, such as 5 minutes. The period can be automatically adjusted according to the system load.

[0134] Soft and hard threshold triggering logic:

[0135] Soft threshold: Start-up deviation ≤ 1 time slice, protection period violation ≤ 1 time.

[0136] Hard threshold: Startup deviation > 2 cycles, or protection period violation ≥ 2 times, or long task timeout.

[0137] Simulated layout execution rules:

[0138] Arrange long tasks and short tasks in chronological order;

[0139] Each deployment requires verification of resource quantity, location, bandwidth, and protection period;

[0140] Record the estimated startup time and average resource usage;

[0141] If placement fails, roll back and try another solution.

[0142] The rollback strategy for the simulated task arrangement is as follows: first, the most recently placed short task packet is canceled; if the conditions are still not met, the most recent long task boundary adjustment operation is rolled back. The entire rollback process follows the principle of "prioritizing the startability of long tasks and secondarily maintaining bandwidth integrity".

[0143] In a preferred embodiment of the present invention, based on classification data and available resource status, candidate draft windows are determined for each long task according to resource requirements and continuous duration, and these are aggregated to form a candidate window set, including:

[0144] Based on the first classification data, extract the number of nodes required for long tasks and the required continuous duration, divide the available resource status into multiple continuous time slices, and obtain time slice resource data.

[0145] Based on the time slice resource data, the resource segments that can meet the node requirements are retrieved within each time slice interval, and then filtered according to the continuous duration constraint to obtain candidate resource segments that meet the conditions.

[0146] Based on the candidate resource fragments, the start time, end time, and node range are combined to determine the corresponding window candidate drafts;

[0147] Based on multiple initial drafts of candidate windows, they are summarized according to task identifiers to form a candidate window set.

[0148] In this embodiment of the invention, based on classification data and available resource status, a preliminary draft of candidate windows is determined for each long task according to resource requirements and continuous duration, and these drafts are aggregated to form a candidate window set. This allows for the establishment of clear resource availability boundaries before scheduling. By dividing the available resource status into multiple continuous time slices, the system can intuitively display the node occupancy in the time dimension, thereby quickly retrieving candidate segments that meet the resource quantity and continuous duration requirements within the time slice interval. Furthermore, by combining the start time, end time, and node range to form the preliminary draft of candidate windows, multiple potential execution options can be provided for long tasks, avoiding long-term task waiting due to insufficient single windows during scheduling. For example, in a long task scenario requiring 10 nodes and lasting 30 minutes, if the system generates two candidate windows for different racks in advance, the scheduler can flexibly choose when resources fluctuate, thereby improving the scheduling success rate. The final candidate window set provides high-quality input for subsequent verification and screening, enabling long tasks to quickly obtain feasible scheduling paths under complex resource conditions.

[0149] The available resource status is divided into consecutive time slices to form time-slice resource data. For example, the cluster scheduling cycle is divided into 5-minute slices, and the number of available nodes, node distribution, and bandwidth are recorded for each time slice.

[0150] When retrieving candidate resource segments, the system performs a sliding window search for each time slice interval to check whether the interval meets the node requirements and continuous duration requirements of a long task. If it does, the start time, end time, and node range are combined to form a window of candidate drafts.

[0151] Multiple candidate draft windows will be aggregated by task identifier to form a candidate window set. The candidate window set is stored in a table structure, with each row containing: task ID, start time, end time, node range, number of available nodes, and continuous duration.

[0152] For example, for a task requiring 10 nodes and lasting 30 minutes, two candidate segments are found in the available resource state:

[0153] Candidate 1: Start time is 10:00, end time is 10:30, and the node range is 10 nodes within rack 1;

[0154] Candidate 2: Start time is 10:15, end time is 10:45, and the node range is 12 nodes across the rack.

[0155] Both are stored in the candidate window set for subsequent verification and screening.

[0156] In a preferred embodiment of the present invention, based on the candidate window set, node location affinity, number of racks spanned, data locality, and interconnect bandwidth are verified item by item to generate a feasible window set, including:

[0157] Based on the candidate window set and available resource status, each candidate is verified for affinity according to the node position of the task, and an affinity pass list is generated;

[0158] Based on the affinity pass list, the topology pass list is generated by comparing the nodes in order of the number of racks crossed from least to most, and in case of a tie, by the number of average hops between nodes from least to most.

[0159] Based on the topology list, the data locality priority and interconnection bandwidth are verified sequentially to ensure they are not lower than the preset threshold. In case of conflict, the data with higher locality is retained first. If they are still tied, the candidate with the larger available bandwidth margin is selected to generate a feasible window set.

[0160] In this embodiment of the invention, a feasible window set is generated by verifying node location affinity, number of racks spanned, data locality, and interconnect bandwidth item by item based on the candidate window set. This significantly improves the rationality of resource selection and the reliability of execution for long tasks. By first generating an affinity pass list, it ensures that tasks are preferentially run on node sets related to their data or computational logic, avoiding increased latency due to overly scattered distribution. For example, a task that needs to access local storage will experience significantly increased I / O latency if it is assigned to resources far from data nodes. Furthermore, by comparing the number of racks spanned and the average number of hops between nodes, windows with lower communication latency can be selected from multiple candidate windows. For example, when two candidate windows both meet the resource requirements, the system will select the window with fewer racks spanned and a lower number of hops between switching devices, thereby reducing communication overhead. By verifying data locality and interconnect bandwidth, long tasks can be prevented from being scheduled and started under conditions of unreasonable data distribution or insufficient bandwidth, thereby improving execution efficiency and success rate. The final feasible window set provides a high-quality candidate resource pool for long task mapping, ensuring the stability of subsequent mapping and execution.

[0161] The inventory data structure consists of task IDs and verification results, divided into affinity-passed, topology-passed, and final feasible window lists. It can be stored in a table structure, with each row corresponding to a candidate window, and columns including the number of resources, number of racks spanned, average hop count, and bandwidth margin.

[0162] Bandwidth margin measurement method: Bandwidth margin = Available bandwidth − Bandwidth required by the task. If the bandwidth of multiple candidate windows is satisfied, the window with the larger bandwidth margin is selected.

[0163] In a preferred embodiment of the present invention, based on the minimum continuous resource window, a protection period and boundary attributes are set, and whether backfilling is allowed and the minimum start unit are marked, generating a set of labeled windows, including:

[0164] Based on the minimum continuous resource time window and historical execution results, determine the initial protection period, minimum start unit, and boundary attributes, and generate an initial value label set;

[0165] Based on the number of times the protection period was violated and the start-up deviation in the previous monitoring cycle, if the number of violations is one and the start-up deviation does not exceed one time slice, the protection period will be extended by one time slice. If the number of violations is zero, the protection period will be shortened by one time slice and the protection period will not be less than one time slice. At the same time, the minimum start-up unit will not be less than the minimum number of nodes and the minimum duration, and an adjustment tag set will be generated.

[0166] Based on the adjusted label set, indicate whether backfilling is allowed: if the protection period was not breached in the previous monitoring cycle and the start-up deviation is zero, mark backfilling as allowed and limit backfilling to not occupy the minimum start-up unit; otherwise, mark backfilling as not allowed and generate a set of labeled windows.

[0167] In this embodiment of the invention, a protection period and boundary attributes are set according to the minimum continuous resource time window, and backfilling and the minimum start unit are marked to generate a set of labeled windows. This enables the execution guarantee of long tasks and the dynamic utilization of resources during resource allocation. By setting the initial protection period, minimum start unit, and boundary attributes based on historical execution results, the system can perform personalized protection configurations for different task characteristics. For example, a computing task requiring eight nodes can have its minimum start unit set to a continuous time slice of eight nodes, thereby avoiding excessive segmentation. Furthermore, by expanding or shortening the protection period based on feedback from the monitoring cycle, the protection strategy can be adaptively adjusted, ensuring the security of long task startup while preventing resources from being locked for a long time and affecting overall utilization. For example, if the protection period of a long task is not broken within two consecutive cycles, the system automatically shortens the protection period, thereby increasing available resources for short tasks to fill. Through backfilling labeling rules, the system can clearly identify which windows are allowed for short tasks to enter, and the minimum start unit that they must not occupy after entering, preventing short tasks from interfering with the core resources of long task windows. The resulting set of labeled windows provides clear, dynamic, and executable resource allocation constraints for subsequent task scheduling, which helps improve the fairness of scheduling long and short tasks and the overall throughput efficiency of the system.

[0168] The tag set consists of:

[0169] The tags include: window ID, protection period, boundary attributes (start and end time and node range), whether backfilling is allowed, and minimum starting unit.

[0170] Adjustment logic:

[0171] Based on feedback from the previous monitoring period:

[0172] If the protection period is breached once, the protection period will be extended by one time slice;

[0173] If the protection period is not breached, the protection period will be shortened by one time slice (≥1 slice).

[0174] At the same time, it is verified that the minimum startup unit is always no less than the task requirements.

[0175] Backfilling allowed implementation: When backfilling is allowed, the system schedules short tasks at the outer edge of the window boundary, but must not encroach on the minimum start unit range marked.

[0176] The system records the task ID and scope of use of short tasks. When the protection period is about to expire, the system automatically releases all short tasks that have intruded into the boundary of the protection zone to ensure that long tasks can be started smoothly.

[0177] In a preferred embodiment of the present invention, the resource requirements and expected duration of short tasks are aggregated to form a draft short task package, including:

[0178] Based on the candidate gap list, extract the number of available nodes and the continuous duration of each gap to obtain the gap attribute record;

[0179] Based on the gap attribute records, the resource requirements of short tasks are compared one by one, and short tasks with no more than the gap attribute number and duration are selected to form a candidate short task set.

[0180] Based on the candidate short task set, the tasks are aggregated according to a priority rule. The priority rule is to select the combination whose cumulative execution time is closest to the gap length, without exceeding the gap duration, to generate the aggregated short task combination.

[0181] Based on the aggregated short task combinations, generate a draft short task package and record the number of nodes occupied, the estimated total execution time, and the corresponding gap identifier.

[0182] In this embodiment of the invention, the resource requirements and expected duration of short tasks are aggregated based on a candidate gap list to form a draft short task package, which effectively improves the utilization efficiency of fragmented resources. By extracting the number of available nodes and continuous duration of each gap and comparing them one by one with the short task requirements, the system can ensure that the selected short tasks do not exceed the gap's carrying capacity, avoiding resource allocation failure. Furthermore, by aggregating short tasks according to priority rules, the system prioritizes the combination with the cumulative execution time closest to the gap length, provided it does not exceed the gap duration, thus minimizing gap waste. For example, if the available node gap length is 20 minutes, the system will prioritize combining a short task package with a cumulative execution time of 19 minutes and 30 seconds, rather than executing only a 10-minute short task. The final draft short task package clearly records the number of occupied nodes, the expected execution time, and the corresponding gap identifier, which not only facilitates subsequent verification but also provides a basis for rapid matching during the dispatch phase. In this way, short tasks can be batch-scheduled to suitable gaps, improving overall scheduling efficiency and resource utilization.

[0183] In this process, each gap in the candidate gap list will have its attribute records extracted, including the gap ID, the number of available nodes, and the continuous duration. The system will then compare each short task request with the others, retaining only those short tasks whose number of nodes and duration do not exceed the gap attributes, thus forming a candidate short task set.

[0184] Subsequently, the system aggregates the candidate short task set, with the following priority rules:

[0185] Provided that the execution time does not exceed the gap length, priority should be given to the combination whose cumulative execution time is closest to the gap length;

[0186] If the cumulative duration of the combined tasks is the same, the shorter task with the higher priority parameter will be selected first.

[0187] If they are still the same, choose the combination with better data locality.

[0188] After aggregation is complete, a draft of the short task package is generated, and the following fields are recorded:

[0189] Gap ID;

[0190] Short task ID set;

[0191] Number of nodes occupied;

[0192] Total estimated execution time;

[0193] Data locality identifier.

[0194] For example, a gap might have the attribute "3 nodes, duration 20 minutes". Candidate short tasks could be 5-minute tasks (requiring 2 nodes), 10-minute tasks (requiring 1 node), and 15-minute tasks (requiring 3 nodes). The system will prioritize aggregating combinations of "10 minutes + 10 minutes", accumulating a total duration close to 20 minutes, to generate a draft short task package.

[0195] In a preferred embodiment of the present invention, based on the execution metrics of the scheduling plan, the deviation between planned startup and actual startup is calculated according to a preset monitoring period, the number of times the protection period is violated is counted, and the resource occupancy continuity of adjacent time slices is checked, generating monitoring results, including:

[0196] Based on the execution metrics of the scheduling plan, extract the planned start time and actual start time of each long task, calculate the difference between the two and record it as the start deviation;

[0197] Based on the execution metrics, the number of times the protection period was interrupted by short tasks is counted, and the protection period violation record is obtained.

[0198] Based on the execution metrics, the resource usage ratio of adjacent time slices is compared one by one. If the usage ratio continuously decreases or fluctuates beyond a preset threshold, it is marked as a continuity anomaly.

[0199] Monitoring results are generated by summarizing the start-up deviation, protection period violation records, and continuous abnormal situations.

[0200] In this embodiment of the invention, the deviation between planned startup and actual startup is calculated according to the execution metric of the scheduling plan and a preset monitoring cycle. The number of times the protection period is violated is also counted, and the continuity of resource occupancy in adjacent time slices is verified, thereby generating monitoring results. This enables real-time monitoring and dynamic feedback of the scheduling execution process. By calculating the startup deviation, the system can intuitively determine whether the startup of long tasks is delayed and infer the accuracy of the scheduler's resource allocation accordingly. By counting the number of times the protection period is violated, it can be found whether short tasks have violated the resource isolation of long tasks, thus ensuring the execution security of long tasks. By comparing the resource occupancy ratio of adjacent time slices, it can promptly identify whether there is a continuous decline or abnormal fluctuation in resource utilization, avoiding task failure due to sudden resource jitter. For example, when a long task is delayed in startup for two consecutive monitoring cycles, the system can trigger an update scheme in advance to release short task packages, thereby ensuring that the long task starts on time. The final monitoring results provide a quantitative basis for subsequent online rebalancing, enabling the scheduling system to have adaptive adjustment capabilities and improving the overall stability and robustness of the scheduling.

[0201] The system calculates and outputs monitoring results based on performance metrics and a preset monitoring cycle. The default monitoring cycle is 5 minutes, which can be dynamically adjusted according to system load.

[0202] The monitoring indicators include:

[0203] Startup Deviation: Extract the planned start time from the actual start time and calculate the difference as the startup deviation. For example, if a task is planned to start at 10:00 but actually starts at 10:05, the deviation is 5 minutes.

[0204] Protection period violation count: When a short task occupies a protection period within a long task window, it is counted as one protection period violation. The system automatically records the task ID, the time of violation, and the number of nodes occupied.

[0205] Resource usage continuity: Calculate the resource usage ratio of adjacent time slices. If the usage ratio decreases by more than 20% continuously, or fluctuates by more than 30% within three consecutive periods, it is marked as a continuity anomaly.

[0206] The final monitoring results include: task ID, startup deviation, number of protection period violations, continuous anomaly markers, and monitoring cycle duration.

[0207] For example, if a long task starts with a 5-minute delay in both cycles, and the protection period is violated once in the second cycle, the monitoring result is recorded as "Startup deviation: 5 minutes, number of protection period violations: 1, continuity is normal". This result serves as input, triggering the system to enter the rebalancing phase.

[0208] In a preferred embodiment of the present invention, based on the first update scheme and the second update scheme, a simulated layout is performed and the expected start time and average resource consumption are recorded to generate candidate scheme evaluation results, including:

[0209] Based on the first update scheme and the second update scheme, as well as the available resource status, a simulated resource view is constructed as input. The simulated resource view is a simulated environment copied from the current available resource status.

[0210] Based on the simulated resource view, long tasks in the ready window are placed first in the order of time slices from early to late, followed by short task packages for backfilling. When placing each item, the resource quantity, node position and correspondence with mapping data two and mapping data three are checked one by one, the network bandwidth is not lower than the preset threshold and the integrity of the protection period is verified, the expected start time of each long task and the resource occupancy ratio of each time slice are recorded, and simulated layout records of the two schemes are generated.

[0211] Based on the simulated layout records, the resource usage ratio of each scheme is accumulated for each time slice, and then the average resource usage value is generated by dividing it equally according to the number of time slices. At the same time, it is calculated whether all verification items have passed and the data locality index is summarized to generate candidate scheme evaluation data.

[0212] Based on the candidate solution evaluation data, an evaluation item is generated for each solution, including the expected start time, average resource consumption, verification conclusions, and data locality summary, thus generating the candidate solution evaluation results.

[0213] In this embodiment of the invention, simulated scheduling based on a first update scheme and a second update scheme, along with recording the expected start time and average resource usage, generates candidate scheme evaluation results, providing a scientific basis for the scheduler's decision-making. By constructing a simulated resource view generated from the current available resource state, the system can predict the execution effects of different scheduling schemes without actually executing tasks. For example, in the first scheme, the system might achieve long task startup by slightly adjusting window boundaries, while in the second scheme, it might need to release multiple short task packets and reorder resource gaps; simulated scheduling allows for comparison of the execution effects of the two. By verifying the resource quantity, node position, mapping relationship, bandwidth threshold, and protection period integrity one by one, the system can record the expected start time of long tasks and the resource usage ratio of each time slice, thereby quantifying the merits of each scheme. Furthermore, by accumulating and averaging the time slice resource usage ratios, the system obtains an average resource usage value, and combines this with data locality indicators and verification conclusions to generate candidate scheme evaluation results. For example, when both schemes meet the long task startup conditions, the system will prioritize the scheme with a higher average resource usage value and better data locality indicators. The final generated candidate solution evaluation entries can provide transparent and traceable decision support for the scheduler, ensuring that the final update plan guarantees fairness for long tasks while taking into account overall resource utilization and system throughput.

[0214] The simulated resource view data structure includes a node table, a time slice table, and a bandwidth matrix, reflecting the distribution of available resources in time and space.

[0215] Average resource usage is calculated by summing the resource usage ratio of each time slice (number of occupied nodes / total number of nodes) and then dividing by the number of time slices to obtain the average value.

[0216] Data locality metric calculation: Calculate the proportion of each task executed on the local node. A high local proportion indicates good data locality. If cross-node operations are involved, calculate the cross-node traffic and apply weights.

[0217] Evaluation item fields include: scheme ID, expected start time, average resource consumption, verification result (pass / fail), data locality index, and recommendation priority.

[0218] The recommendation priority is calculated by the system based on a combination of resource usage and startup time. For example, the system prioritizes the earliest startup time and the average utilization rate that is not lower than the historical value. This evaluation item is stored in the system database and serves as the input for the next round of scheduling decisions.

[0219] Embodiments of the present invention also provide a scheduling system, the system comprising:

[0220] Metadata acquisition unit is used to acquire metadata one of the task set, the metadata one including resource requirements, continuous duration and priority, and to divide and identify long tasks and short tasks according to the metadata one to form classification data one;

[0221] The window calculation unit is used to calculate the minimum continuous resource window for each long task based on the classification data one, lock the window set, and obtain the mapping data two from the long task to the window;

[0222] The gap identification and aggregation unit is used to identify time resource gaps outside the window based on the mapping data 2 and the available resource status, and to aggregate short tasks according to resources and duration to generate the mapping data 3 of gap and short task packages;

[0223] The task dispatch unit is used to match and dispatch tasks according to mapping data 2 and mapping data 3. When the window meets the conditions, the corresponding long task is triggered. When the window is not ready, the short task package is filled in the gap order to generate a scheduling plan.

[0224] The monitoring and rebalancing unit is used to perform online monitoring and rebalancing based on the execution metrics of the scheduling plan. When the window is disturbed or a long task times out, it releases the unstarted short task packages and rearranges the gaps and window boundaries, generating update mappings and update plans.

[0225] The execution result generation unit is used to generate execution results according to the scheduling plan or update plan, and write the execution results back to metadata 1 as input for the next round of windowing and matching.

[0226] It should be noted that this system is a system corresponding to the above method. All implementation methods in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.

[0227] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A scheduling method, characterized by, The method comprises: Obtaining metadata I of a task set, including resource requirements, continuous time length and priority; dividing long tasks and short tasks and identifying according to the metadata I, forming classification data I; According to the classification data I, calculating the minimum continuous resource time window for each long task, locking the window set, and obtaining the mapping data II of the long task to the window, comprising: According to the classification data I and the available resource state, determining the window candidate draft for each long task according to the resource requirement and the continuous time length, and summarizing to form a candidate window set; According to the candidate window set, verifying node position affinity, cross-rack quantity, data locality and interconnection bandwidth one by one, and generating a feasible window set; According to the feasible window set, selecting the minimum window that satisfies the resource quantity and the continuous time length at one time, if there are multiple, preferentially selecting the window with smaller cross-rack quantity and fewer average hops between nodes, and generating the minimum continuous resource time window; According to the minimum continuous resource time window, setting the protection period and the boundary attribute and marking whether backfilling is allowed and the minimum starting unit, generating the window set with labels, wherein the protection period is a time period in which the window is prohibited from allocating and long task resource conflict tasks, the boundary attribute is the start and end time of the window and the node range, and the minimum starting unit is the combined constraint of the minimum node number and the minimum time length; According to the window set with labels, the long task identifier is matched with the corresponding minimum continuous resource time window, protection period and boundary attribute, and the mapping data II of the long task to the window is obtained. According to the mapping data II and the available resource state, identifying the time resource gap outside the window, aggregating the short tasks according to the resource and the time length, and generating the mapping data III of the gap and the short task package, comprising: According to the mapping data II and the available resource state, scanning the nodes and time slices not covered by the minimum continuous resource time window, and generating an initial gap set; According to the initial gap set, measuring the available node number and the continuous time length for each gap, marking the gap that is not enough to support the long task as a short task candidate area, and generating a candidate gap list; According to the candidate gap list, aggregating the resource requirements and the expected time length of the short tasks to form a short task package draft; According to the scheduling plan, generating an execution result, and writing the execution result back to the metadata I as the input of the next round of windowing and matching; 2. The scheduling method of claim 1, wherein, According to the scheduling plan or the updated plan, generating an execution result, and writing the execution result back to the metadata I as the input of the next round of windowing and matching; According to the classification data I, calculating the minimum continuous resource time window for each long task, locking the window set, and obtaining the mapping data II of the long task to the window, comprising: According to the classification data I and the available resource state, determining the window candidate draft for each long task according to the resource requirement and the continuous time length, and summarizing to form a candidate window set; According to the candidate window set, verifying node position affinity, cross-rack quantity, data locality and interconnection bandwidth one by one, and generating a feasible window set; According to the feasible window set, selecting the minimum window that satisfies the resource quantity and the continuous time length at one time, if there are multiple, preferentially selecting the window with smaller cross-rack quantity and fewer average hops between nodes, and generating the minimum continuous resource time window; According to the minimum continuous resource time window, setting the protection period and the boundary attribute and marking whether backfilling is allowed and the minimum starting unit, generating the window set with labels, wherein the protection period is a time period in which the window is prohibited from allocating and long task resource conflict tasks, the boundary attribute is the start and end time of the window and the node range, and the minimum starting unit is the combined constraint of the minimum node number and the minimum time length; According to the window set with labels, the long task identifier is matched with the corresponding minimum continuous resource time window, protection period and boundary attribute, and the mapping data II of the long task to the window is obtained. According to the short task package preliminary draft, the node position distribution and network bandwidth of the aggregated short task package are checked one by one to see whether they meet the preset threshold value, if there are multiple feasible combinations, the combination with larger bandwidth margin and better data locality is selected preferentially, and a short task package set is generated; According to the candidate gap and the short task package set, a mapping relationship between each short task package and the corresponding gap is established, and mapping data three of the gap and the short task package is obtained.

3. The scheduling method of claim 1, wherein, According to the mapping data two and the mapping data three, if the window meets the condition, the corresponding long task is triggered, if the window is not ready, the short task package is filled according to the gap order, a scheduling plan is generated, including: According to the mapping data two, the mapping data three and the available resource state, the readiness check is carried out according to the resource quantity meeting, the node position matching, the network bandwidth not being lower than the preset threshold value and the integrity of the protection period, and a ready window list is generated; According to the ready window list, the start instruction of the corresponding long task is generated, and the start time, resource position and protection period occupation are registered, forming a long task scheduling record; According to the unready window and the mapping data three, the short task package is selected for filling according to the time resource gap from early to late and the principle of withdrawal cost from small to large, and a backfill list is generated; According to the long task scheduling record and the backfill list, the dispatch conflict is identified, and the conflict short task package is suspended and moved into the recycling queue, and the task parameters and allocated resource identifiers are retained, forming a recycling list; According to the long task scheduling record, the backfill list and the recycling list, the scheduling plan containing the trigger record and the recycling list is formed.

4. The scheduling method of claim 1, wherein, According to the execution measurement of the scheduling plan, online monitoring and rebalancing are carried out, when the window is disturbed or the long task is overdue, the unstarted short task package is released and the gap and the window boundary are rearranged, the update mapping and the update plan are generated, including: According to the execution measurement of the scheduling plan, the deviation between the planned start and the actual start is calculated according to the preset monitoring period, the number of times the protection period is destroyed is counted, and the resource occupation continuity of adjacent time slices is verified, and a monitoring result is generated; According to the monitoring result meeting the soft threshold condition, the backfill order is adjusted according to the withdrawal cost from small to large, and the affected window boundary is adjusted in the granularity of a time slice or a node to restore readiness, and a first update scheme is generated, wherein the soft threshold condition is that the start deviation does not exceed a time slice and the protection period is destroyed not more than once; According to the monitoring result meeting the hard threshold condition, the unstarted short task package is released, and the time resource gap is rearranged according to the principle of starting time from early to late and under the same condition, the target window is preferentially close, and a second update scheme is generated, wherein the hard threshold condition is that the start deviation exceeds a time slice for two consecutive monitoring periods, or the protection period is destroyed twice or more in a single period, or there is a long task expected to be overdue; According to the first update scheme and the second update scheme, the simulation arrangement is carried out and the expected start time and the average resource occupation are recorded, and a candidate scheme evaluation result is generated; According to the candidate scheme evaluation result, the target scheme that can meet the long task start earliest and the expected average resource occupation is not lower than the actual average resource occupation of the last monitoring period is selected, the mapping data two and the mapping data three are updated, and an update plan is generated.

5. The scheduling method of claim 1, wherein, According to the classification data and the available resource state, a window candidate preliminary draft is determined for each long task according to resource demand and continuous time length, and a candidate window set is formed by summarizing, including: According to the classification data, the node demand quantity and the required continuous time length of the long task are extracted, the available resource state is divided into multiple continuous time slices, and time slice resource data is obtained; According to the time slice resource data, the resource segment that can meet the node demand quantity is searched in each time slice interval, and the continuous time length constraint is filtered to obtain the candidate resource segment that meets the condition; According to the candidate resource segment, the start time, end time and node range are combined to determine the corresponding window candidate preliminary draft; According to the multiple window candidate preliminary drafts, the task identification is summarized to form a candidate window set.

6. The scheduling method of claim 1, wherein, According to the minimum continuous resource time window, the protection period and the boundary attribute are set and whether backfilling is allowed and the minimum starting unit are marked to generate a labeled window set, including: According to the minimum continuous resource time window and the historical execution result, the initial protection period, the minimum starting unit and the boundary attribute are determined to generate an initial value label set; According to the number of times the protection period is destroyed in the last monitoring period and the starting deviation, if the number of times is one and the starting deviation does not exceed one time slice, the protection period is expanded by one time slice, if the number of times is zero, the protection period is shortened by one time slice and the protection period is not less than one time slice, while the minimum starting unit is not less than the minimum node number and the minimum time length, to generate an adjustment label set; According to the adjustment label set, whether backfilling is allowed is marked: when the protection period is not destroyed and the starting deviation is zero in the last monitoring period, backfilling is allowed and backfilling is limited to not occupy the minimum starting unit, otherwise backfilling is not allowed, to generate a labeled window set.

7. The scheduling method of claim 2, wherein, According to the candidate gap list, the resource demand and the estimated length of the short task are aggregated to form a short task package preliminary draft, including: According to the candidate gap list, the available node number and the continuous time length of each gap are extracted to obtain a gap attribute record; According to the gap attribute record, the resource demand of the short task is compared one by one, and the short task whose node number and time length do not exceed the gap attribute is selected to form a candidate short task set; According to the candidate short task set, aggregation is performed according to the priority rule, and the priority rule is to preferentially select the combination whose cumulative execution time length is closest to the gap length under the premise of not exceeding the gap length, to generate an aggregated short task combination; According to the aggregated short task combination, a short task package preliminary draft is generated, and the occupied node number, the estimated total execution time and the identifier of the corresponding gap are recorded.

8. The scheduling method of claim 4, wherein, According to the window readiness list, according to the execution metric of the scheduling plan, the deviation between the planned start and the actual start is calculated according to the preset monitoring period, the number of times the protection period is destroyed is counted and the resource occupation continuity of adjacent time slices is verified to generate a monitoring result, including: According to the execution metric of the scheduling plan, the planned start time and the actual start time of each long task are extracted, the difference between the two is calculated and recorded as the starting deviation; According to the execution metric, the number of times each protection period is occupied by the short task is counted to obtain a protection period destruction record; According to the execution metric, the resource occupation proportions of adjacent time slices are compared one by one, if the occupation proportions continuously decrease or fluctuate more than a preset threshold, the continuity anomaly is marked; According to the start bias, the protection period damage record and the continuity anomaly, the monitoring result is generated.

9. The scheduling method of claim 4, wherein, According to the first update scheme and the second update scheme, the simulation arrangement is performed and the predicted start time and the average resource occupation are recorded to generate the candidate scheme evaluation result, including: According to the first update scheme, the second update scheme and the available resource state, a simulation resource view is constructed as input, which is a simulation environment generated by copying the current available resource state; According to the simulation resource view, the long tasks in the ready window are placed first, and then the backfill short task package is placed in the order of time slices from early to late, and the resource quantity, the node position and the corresponding relationship of mapping data two and mapping data three, the network bandwidth not less than the preset threshold and the protection period integrity are checked one by one for each item, the predicted start time of each long task and the resource occupation proportion of each time slice are recorded, and the simulation arrangement record of the two schemes is generated; According to the simulation arrangement record, for each scheme, the resource occupation proportions are accumulated one by one for each time slice, then the average resource occupation value is generated by equal division according to the number of time slices, and whether all the check items pass is counted and the data locality index is summarized to generate the candidate scheme evaluation data; According to the candidate scheme evaluation data, for each scheme, the evaluation entry including the predicted start time, the average resource occupation, the check conclusion and the data locality summary is generated to generate the candidate scheme evaluation result.

Images