Server cluster task scheduling optimization system adaptive to large model training scenarios

Abstract

This invention discloses a server cluster task scheduling optimization system adapted to large model training scenarios, belonging to the field of server cluster task scheduling technology. The invention includes a priori topology parsing module: used to parse the parallel strategy and model size of the training task at the task submission node, construct a theoretical iteration cycle model, and identify and mark a subset of non-critical path computers from the single iteration task execution flow of the training task; and a phase prediction module: used to capture the standard output log stream of the training framework in real time during the task execution phase, extract periodic progress anchors, and perform time-series extrapolation in conjunction with the theoretical iteration cycle model to predict the theoretical end time of the gradient synchronization phase within the current iteration cycle. This invention solves the problem of effective computing power loss caused by interrupted computation in existing scheduling systems by executing scheduling operations and non-critical path computation tasks in parallel within a window period, ensuring that the time consumed by scheduling is fully covered by effective computation.

Description

Technical Field

[0001] This invention belongs to the field of server cluster task scheduling technology, and in particular relates to a server cluster task scheduling optimization system adapted to large model training scenarios. Background Technology

[0002] As the scale of large model parameters exceeds hundreds of billions, training tasks require supercomputing clusters consisting of thousands of GPUs and employ complex hybrid strategies such as tensor parallelism and pipeline parallelism. Although existing intelligent scheduling systems can dynamically optimize resource allocation through algorithms such as reinforcement learning, the computing power and I / O overhead they generate have become significant bottlenecks restricting cluster efficiency.

[0003] To maintain a global view, the system needs to poll the status of a massive number of GPU nodes every second, causing continuous pressure on the CPU and I / O of the central node. During the decision-making phase, the optimal GPU placement scheme under hybrid parallel computing involves a large number of matrix operations and topology searches. A single decision takes several to tens of seconds and consumes computing power equivalent to the computing power of dozens of GPUs. However, during the execution phase, frequent priority adjustments and context switching cause GPU utilization to fluctuate, further exacerbating the loss of computing power. Moreover, as the cluster size increases, it grows superlinearly. Existing technical solutions alleviate the overhead by reducing the sampling frequency, adopting lightweight algorithms or distributed architectures, but at the expense of real-time performance, decision quality or introducing consistency costs, failing to fundamentally solve the contradiction between scheduling efficiency and effectiveness. Summary of the Invention

[0004] The purpose of this invention is to provide a server cluster task scheduling optimization system adapted to large model training scenarios, which solves the problem that the existing intelligent scheduling system has excessive computing power and I / O overhead in the monitoring and collection, decision calculation and task orchestration stages, and that the overhead increases superlinearly with the expansion of the cluster size, resulting in a large amount of effective computing power being squeezed out.

[0005] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: This invention is a server cluster task scheduling optimization system adapted to large model training scenarios, comprising: Prior topology parsing module: used to parse the parallel strategy and model size of the training task at the task submission node, build a theoretical iteration cycle model, and identify and mark the non-critical path computer subset from the single iteration task execution flow of the training task; Phase prediction module: used to capture the standard output log stream of the training framework in real time during the task execution phase, extract periodic progress anchors, and perform time-series extrapolation in combination with the theoretical iteration cycle model to predict the theoretical end time of the gradient synchronization phase in the current iteration cycle. Window demand assessment module: used to determine the minimum continuous time threshold required to execute the scheduling operation based on the type of scheduling operation to be executed, as the target window duration; Active window construction module: When the theoretical end time of the gradient synchronization phase is detected, a suspension instruction is sent to the training process to freeze the execution state of the training process in memory, release the central processing unit computing resources, dynamically select target tasks from the non-critical path computing subset based on the target window duration, assemble and generate a window filling queue, so that the estimated total time of the queue covers the target window duration, and schedule the scheduling operation and the target tasks in the window filling queue are executed in parallel on the released central processing unit computing resources; Timing Trigger Module: Used to monitor the window period filling queue and the execution status of scheduling operations. After the task is completed, it sends a recovery command to the training process to unfreeze the execution status and allow the training process to continue to execute the next iteration cycle.

[0006] Furthermore, the prior topology parsing module includes a dependency analysis unit and a path classification unit; The dependency analysis unit is used to obtain the computation graph representation of the training task, traverse each computation node in the computation graph, and determine the execution level of each computation node and the cross-node communication dependency between each computation node based on the input-output dependency edges between each computation node. The path classification unit is used to classify and label each computing node according to a preset non-critical path determination rule. The non-critical path determination rule includes: First criterion: All input tensors of the computing node come from the local graphics processor and do not depend on the output results of cross-node set communication operations; Second criterion: The output tensor of a computation node is dependent on the parameter update operation, but not on other cross-node set communication operations within a single iteration; The third criterion is that the topological position of the computing node in the computation graph is after all cross-node set communication operations are completed and before the parameter update operation is executed. After completing the path classification and labeling, the prior topology parsing module will identify the set of computing nodes that simultaneously satisfy the first, second, and third decision rules as the non-critical path computing subset and output it to the active window construction module.

[0007] Furthermore, the phase prediction module includes a real-time feature extraction unit and a time-series extrapolation unit; The real-time feature extraction unit is used to listen to the standard output stream of the training task container in a non-blocking manner during the task execution phase. It matches the iteration progress identifier text in the standard output stream through a pre-configured regular expression pattern library, extracts the iteration round number corresponding to the iteration progress identifier text and the timestamp of the text output, and uses them as periodic progress anchors. The timing extrapolation unit is used to accept periodic progress anchors and obtain the relative cost relationship from the preset anchor to the end of the gradient synchronization stage from the theoretical iteration cycle model. The relative cost relationship describes the proportion of the estimated time interval from the occurrence of the periodic progress anchor to the end of the gradient synchronization stage in the total duration of a single iteration.

[0008] Furthermore, the timing extrapolation unit is also used to superimpose the timestamp of the most recently captured periodic progress anchor point with the relative offset relationship to obtain the theoretical end time of the gradient synchronization phase. When two or more consecutive periodic progress anchor points are captured, the timing extrapolation unit updates the measured duration of the current iteration cycle according to the actual time interval between each periodic anchor point, and corrects the absolute time offset corresponding to the relative offset relationship with the measured duration to obtain the corrected theoretical end time of the gradient synchronization phase.

[0009] Furthermore, the window demand assessment module includes a scheduling operation time statistics unit and a window duration decision unit; The scheduling operation time consumption statistics unit is used to maintain the actual time consumption sample set of each type of scheduling operation during the historical execution period, and to perform distribution fitting on the time consumption sample of each type of scheduling operation to obtain the corresponding probability distribution model. The types of scheduling operations include task migration, resource scaling up and down, and priority adjustment. The window duration decision unit is used to accept the type of scheduling operation to be executed this time, calculate the upper quantile value of the time consumption under the preset confidence level according to the probability distribution model corresponding to the scheduling operation type, and output the upper quantile value of the time consumption as the minimum continuous time threshold. The minimum continuous time threshold is dynamically adjusted as the execution record of the scheduling operation type is updated.

[0010] Furthermore, the proactive window building module includes a suspend control unit, a queue assembly unit, and a parallel execution unit; The suspend control unit sends a computation flow suspension signal to the graphics processor driver layer to which the training process belongs when the theoretical end time of the gradient synchronization phase is detected. By inserting unmet event synchronization points into the command queue of the graphics processor, subsequent computation kernels are queued and wait at the event synchronization points. The graphics processor memory state, unified computing device architecture context, and tensor data of the training process are completely preserved during the suspension. At the same time, the suspend control unit marks the CPU cores occupied by the training process as preemptible for scheduling operations. The queue assembly unit is used to read the target window duration after the training process is suspended and to execute a multi-level progressive selection strategy: First priority selection layer: Select all non-critical path computation tasks that have not yet been executed within the current iteration cycle from the non-critical path computation subset, add them to the window period filling queue as basic filling items, and calculate the estimated total time of the basic filling items; Second priority selection layer: If the estimated total time of the basic filling items is less than the target window duration, calculate the time gap value, and select pre-execution tasks from the subset of non-critical paths in the next iteration cycle in descending order of estimated time, until the total estimated time of the accumulated pre-execution tasks is greater than or equal to the time gap value, and add the selected pre-execution tasks as supplementary filling items to the window filling queue. Third priority selection layer: If the total estimated time of all available tasks in the non-critical path computation subset is still less than the target window duration, then a placeholder computation task matching the remaining gap duration is generated. The placeholder computation task is a lightweight graphics processor computation kernel sequence with a precisely configurable execution duration. The placeholder computation task is added to the window filling queue as a fallback filling item. The parallel execution unit is used to distribute the computational tasks in the window filling queue to the graphics processor occupied by the training process after the window filling queue is assembled. At the same time, the scheduling operation is distributed to the central processing unit core released by the suspend control unit for execution, so that the effective computation and scheduling operations can proceed in parallel within the window period.

[0011] Furthermore, the timing-triggered module includes a runtime monitoring unit and a recovery control unit; The runtime monitoring unit is used to continuously monitor the graphics processor execution completion signals of each computing task in the window filling queue, as well as the execution completion signals of the scheduling operation, within the window period. The graphics processor execution completion signal is obtained by inserting an event record node at the end of the graphics processor command queue of each computing task in the window filling queue. When the event record node is executed by the graphics processor, the corresponding execution completion callback is triggered. The recovery control unit is used to send a recovery command to the training process when the execution conditions are met. The execution conditions include the following: the graphics processor completion signals of all computation tasks in the window filling queue have been triggered, and the execution completion signal of the scheduling operation has been triggered. The recovery command is used to unfreeze the execution state of the training process, sending a satisfaction signal to the unsatisfied event synchronization point inserted by the suspended control unit in the graphics processor command queue, so that the event synchronization point can be released and the subsequent computing kernel can switch from the queued state to the execution state.

[0012] Furthermore, the recovery control unit is also used to perform timeout processing when all computational tasks in the window period filling queue have been completed, but the scheduling operation has not returned an execution completion signal within the preset maximum waiting time limit. The timeout processing is used to send a recovery instruction to the training process, and at the same time mark the unfinished scheduling operation as an asynchronous compensation task, which will continue to be executed within the next safe window period determined in consultation with the active window building module.

[0013] The present invention has the following beneficial effects: 1. This invention solves the problem of effective computing power loss caused by interruption of calculation in existing scheduling systems by executing scheduling operations and non-critical path calculation tasks in parallel within a window period. The time consumed by scheduling is fully covered by effective calculation, and no additional time overhead is generated from the macro training timeline.

[0014] 2. This invention solves the problem of scheduling failure or window waste caused by prediction errors by progressively selecting target tasks from a subset of non-critical paths to fill the window, and dynamically assembling the window width according to the scheduling operation time requirements, rather than passively relying on unstable external predictions.

[0015] 3. This invention achieves suspension by inserting event synchronization points into the GPU command queue, thus preserving the complete state of the video memory. When resuming, there is no need to rebuild from the checkpoint. When the window requirement time exceeds the total time of the current non-critical tasks, the system pre-selects subsequent tasks across iterations to execute them in advance, folding future computation time into the current window, thereby shortening the overall training time.

[0016] 4. This invention utilizes the CPU cores released during the training process suspension to perform scheduling operations, eliminating the need to reserve dedicated computing nodes for the scheduler. Monitoring data only requires the standard output log stream and the GPU standard telemetry interface, without introducing additional acquisition burden, and significantly reducing the system deployment threshold. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a system block diagram of the present invention. Detailed Implementation

[0019] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0020] See Figure 1 This invention is a server cluster task scheduling optimization system adapted to large model training scenarios, comprising: Prior topology parsing module: used to parse the parallel strategy and model size of the training task at the task submission node, build a theoretical iteration cycle model, and identify and mark the non-critical path computer subset from the single iteration task execution flow of the training task; The prior topology parsing module includes a dependency analysis unit and a path classification unit; The dependency analysis unit is used to obtain the computation graph representation of the training task, traverse each computation node in the computation graph, and determine the execution level of each computation node and the cross-node communication dependency between each computation node based on the input-output dependency edges between each computation node. The path classification unit is used to classify and label each computing node according to a preset non-critical path determination rule. The non-critical path determination rule includes: First criterion: All input tensors of the computing node come from the local graphics processor and do not depend on the output results of cross-node set communication operations; Second criterion: The output tensor of a computation node is dependent on the parameter update operation, but not on other cross-node set communication operations within a single iteration; The third criterion is that the topological position of the computing node in the computation graph is after all cross-node set communication operations are completed and before the parameter update operation is executed. After completing the path classification and labeling, the prior topology parsing module will identify the set of computing nodes that simultaneously satisfy the first, second, and third decision rules as the non-critical path computing subset and output it to the active window construction module.

[0021] Phase prediction module: used to capture the standard output log stream of the training framework in real time during the task execution phase, extract periodic progress anchors, and perform time-series extrapolation in combination with the theoretical iteration cycle model to predict the theoretical end time of the gradient synchronization phase in the current iteration cycle. The phase prediction module includes a real-time feature extraction unit and a temporal extrapolation unit; The real-time feature extraction unit is used to listen to the standard output stream of the training task container in a non-blocking manner during the task execution phase. It matches the iteration progress identifier text in the standard output stream through a pre-configured regular expression pattern library, extracts the iteration round number corresponding to the iteration progress identifier text and the timestamp of the text output, and uses them as periodic progress anchors. The timing extrapolation unit is used to accept periodic progress anchors and obtain the relative cost relationship from the preset anchor to the end of the gradient synchronization stage from the theoretical iteration cycle model. The relative cost relationship describes the proportion of the estimated time interval from the occurrence of the periodic progress anchor to the end of the gradient synchronization stage in the total duration of a single iteration.

[0022] The timing extrapolation unit is also used to superimpose the timestamp of the most recently captured periodic progress anchor point with the relative offset relationship to obtain the theoretical end time of the gradient synchronization phase. When two or more consecutive periodic progress anchor points are captured, the timing extrapolation unit updates the measured duration of the current iteration cycle according to the actual time interval between each periodic anchor point, and corrects the absolute time offset corresponding to the relative offset relationship with the measured duration to obtain the corrected theoretical end time of the gradient synchronization phase.

[0023] Window demand assessment module: used to determine the minimum continuous time threshold required to execute the scheduling operation based on the type of scheduling operation to be executed, as the target window duration; The window demand assessment module includes a scheduling operation time statistics unit and a window duration decision unit; The scheduling operation time consumption statistics unit is used to maintain the actual time consumption sample set of each type of scheduling operation during the historical execution period, and to perform distribution fitting on the time consumption sample of each type of scheduling operation to obtain the corresponding probability distribution model. The types of scheduling operations include task migration, resource scaling up and down, and priority adjustment. The window duration decision unit is used to accept the type of scheduling operation to be executed this time, calculate the upper quantile value of the time consumption under the preset confidence level according to the probability distribution model corresponding to the scheduling operation type, and output the upper quantile value of the time consumption as the minimum continuous time threshold. The minimum continuous time threshold is dynamically adjusted as the execution record of the scheduling operation type is updated.

[0024] Active window construction module: When the theoretical end time of the gradient synchronization phase is detected, a suspension instruction is sent to the training process to freeze the execution state of the training process in memory, release the central processing unit computing resources, dynamically select target tasks from the non-critical path computing subset based on the target window duration, assemble and generate a window filling queue, so that the estimated total time of the queue covers the target window duration, and schedule the scheduling operation and the target tasks in the window filling queue are executed in parallel on the released central processing unit computing resources; The proactive window building module includes a suspend control unit, a queue assembly unit, and a parallel execution unit; The suspend control unit sends a computation flow suspension signal to the graphics processor driver layer to which the training process belongs when the theoretical end time of the gradient synchronization phase is detected. By inserting unmet event synchronization points into the command queue of the graphics processor, subsequent computation kernels are queued and wait at the event synchronization points. The graphics processor memory state, unified computing device architecture context, and tensor data of the training process are completely preserved during the suspension. At the same time, the suspend control unit marks the CPU cores occupied by the training process as preemptible for scheduling operations. The queue assembly unit is used to read the target window duration after the training process is suspended and to execute a multi-level progressive selection strategy: First priority selection layer: Select all non-critical path computation tasks that have not yet been executed within the current iteration cycle from the non-critical path computation subset, add them to the window period filling queue as basic filling items, and calculate the estimated total time of the basic filling items; Second priority selection layer: If the estimated total time of the basic filling items is less than the target window duration, calculate the time gap value, and select pre-execution tasks from the subset of non-critical paths in the next iteration cycle in descending order of estimated time, until the total estimated time of the accumulated pre-execution tasks is greater than or equal to the time gap value, and add the selected pre-execution tasks as supplementary filling items to the window filling queue. Third priority selection layer: If the total estimated time of all available tasks in the non-critical path computation subset is still less than the target window duration, then a placeholder computation task matching the remaining gap duration is generated. The placeholder computation task is a lightweight graphics processor computation kernel sequence with a precisely configurable execution duration. The placeholder computation task is added to the window filling queue as a fallback filling item. The parallel execution unit is used to distribute the computational tasks in the window filling queue to the graphics processor occupied by the training process after the window filling queue is assembled. At the same time, the scheduling operation is distributed to the central processing unit core released by the suspend control unit for execution, so that the effective computation and scheduling operations can proceed in parallel within the window period.

[0025] Timing Trigger Module: Used to monitor the window period filling queue and the execution status of scheduling operations. After the task is completed, it sends a recovery command to the training process to unfreeze the execution status and allow the training process to continue to execute the next iteration cycle.

[0026] The timing-triggered module includes a runtime monitoring unit and a recovery control unit; The runtime monitoring unit is used to continuously monitor the graphics processor execution completion signals of each computing task in the window filling queue, as well as the execution completion signals of the scheduling operation, within the window period. The graphics processor execution completion signal is obtained by inserting an event record node at the end of the graphics processor command queue of each computing task in the window filling queue. When the event record node is executed by the graphics processor, the corresponding execution completion callback is triggered. The recovery control unit is used to send a recovery command to the training process when the execution conditions are met. The execution conditions include the following: the graphics processor completion signals of all computation tasks in the window filling queue have been triggered, and the execution completion signal of the scheduling operation has been triggered. The recovery command is used to unfreeze the execution state of the training process, sending a satisfaction signal to the unsatisfied event synchronization point inserted by the suspended control unit in the graphics processor command queue, so that the event synchronization point can be released and the subsequent computing kernel can switch from the queued state to the execution state.

[0027] The recovery control unit is also used to perform timeout processing when all computational tasks in the queue during the window period have been completed, but the scheduling operation has not returned an execution completion signal within the preset maximum waiting time limit. The timeout processing is used to send a recovery instruction to the training process, and at the same time mark the unfinished scheduling operation as an asynchronous compensation task, which will continue to be executed within the next safe window period determined in consultation with the active window building module.

[0028] The working principle of this invention is as follows: When a large model training task is submitted, the prior topology parsing module parses the parallel strategy and model parameter quantity of the task, constructs a theoretical iteration cycle model, and identifies a subset of non-critical path computations that only rely on the local GPU and do not involve cross-node communication from the computation graph. The phase prediction module listens to the standard output log of the training framework during the task execution phase, captures periodic progress anchors, and performs time-series extrapolation in combination with the theoretical iteration cycle model to predict the theoretical end time of the current iteration gradient synchronization phase. The window requirement evaluation module determines the required minimum continuous time threshold based on the type of scheduling operation to be executed and historical time consumption statistics. When the theoretical end time of gradient synchronization is reached, the proactive window construction module sends a request to the GPU. The U-driver layer sends a suspension signal, inserts an event synchronization point in the GPU command queue to make subsequent computation kernels queue up and wait, and maintains the complete memory state and CUDA context of the training process. The released CPU cores are used for scheduling operations. At the same time, target tasks are progressively selected from the non-critical path computation subset to fill the window period. If the non-critical tasks in the current iteration are insufficient, they are pre-selected across iterations to ensure that the total time spent filling the queue covers the required window duration. During the window period, the GPU executes the filling computation tasks and the CPU executes the scheduling operations. The two proceed in parallel. The timing trigger module monitors the completion status of the computation tasks and scheduling operations. When both conditions are met, a satisfaction signal is sent to the event synchronization point to release the suspension, and the training process seamlessly resumes to the next iteration.

[0029] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A server cluster task scheduling optimization system adapted to large model training scenarios, characterized in that, include: Prior topology parsing module: used to parse the parallel strategy and model size of the training task at the task submission node, build a theoretical iteration cycle model, and identify and mark the non-critical path computer subset from the single iteration task execution flow of the training task; Phase prediction module: used to capture the standard output log stream of the training framework in real time during the task execution phase, extract periodic progress anchors, and perform time-series extrapolation in combination with the theoretical iteration cycle model to predict the theoretical end time of the gradient synchronization phase in the current iteration cycle. Window demand assessment module: used to determine the minimum continuous time threshold required to execute the scheduling operation based on the type of scheduling operation to be executed, as the target window duration; Active window construction module: When the theoretical end time of the gradient synchronization phase is detected, a suspension instruction is sent to the training process to freeze the execution state of the training process in memory, release the central processing unit computing resources, dynamically select target tasks from the non-critical path computing subset based on the target window duration, assemble and generate a window filling queue, so that the estimated total time of the queue covers the target window duration, and schedule the scheduling operation and the target tasks in the window filling queue to be executed in parallel on the released central processing unit computing resources; Timing Trigger Module: Used to monitor the window period filling queue and the execution status of scheduling operations. After the task is completed, it sends a recovery command to the training process to unfreeze the execution status and allow the training process to continue to execute the next iteration cycle.

2. The server cluster task scheduling optimization system adapted to large model training scenarios according to claim 1, characterized in that, The prior topology parsing module includes a dependency analysis unit and a path classification unit; The dependency analysis unit is used to obtain the computation graph representation of the training task, traverse each computation node in the computation graph, and determine the execution level of each computation node and the cross-node communication dependency between each computation node based on the input-output dependency edges between each computation node. The path classification unit is used to perform path classification and labeling on a node-by-node basis for each computing node according to preset non-critical path determination rules, wherein the non-critical path determination rules include: First criterion: All input tensors of the computing node come from the local graphics processor and do not depend on the output results of cross-node set communication operations; Second criterion: The output tensor of a computation node is dependent on the parameter update operation, but not on other cross-node set communication operations within a single iteration; The third criterion is that the topological position of the computing node in the computation graph is after all cross-node set communication operations are completed and before the parameter update operation is executed. After completing the path classification and labeling, the prior topology parsing module determines the set of computing nodes that simultaneously satisfy the first, second, and third judgment rules as the non-critical path computing subset and outputs it to the active window construction module.

3. The server cluster task scheduling optimization system adapted to large model training scenarios according to claim 1, characterized in that, The phase prediction module includes a real-time feature extraction unit and a time-series extrapolation unit; The real-time feature extraction unit is used to listen to the standard output stream of the training task container in a non-blocking manner during the task execution phase, and to match the iteration progress identifier text in the standard output stream through a pre-configured regular expression pattern library, and extract the iteration round number and the timestamp of the text output corresponding to the iteration progress identifier text as periodic progress anchor points. The timing extrapolation unit is used to accept periodic progress anchors and obtain the relative cost relationship from the preset anchor to the end of the gradient synchronization stage from the theoretical iteration cycle model; the relative cost relationship describes the proportion of the estimated time interval from the occurrence of the periodic progress anchor to the end of the gradient synchronization stage in the total duration of a single iteration.

4. The server cluster task scheduling optimization system adapted to large model training scenarios according to claim 3, characterized in that, The timing extrapolation unit is also used to superimpose the timestamp of the most recently captured periodic progress anchor point with the relative offset relationship to obtain the theoretical end time of the gradient synchronization phase; when two or more consecutive periodic progress anchor points are captured, the timing extrapolation unit updates the measured duration of the current iteration cycle according to the actual time interval between each periodic anchor point, and corrects the absolute time offset corresponding to the relative offset relationship with the measured duration to obtain the corrected theoretical end time of the gradient synchronization phase.

5. The server cluster task scheduling optimization system adapted to large model training scenarios according to claim 1, characterized in that, The window demand assessment module includes a scheduling operation time statistics unit and a window duration decision unit; The scheduling operation time consumption statistics unit is used to maintain the actual time consumption sample set of each type of scheduling operation during the historical execution period, and to perform distribution fitting on the time consumption sample of each type of scheduling operation to obtain the corresponding probability distribution model. The types of scheduling operations include task migration, resource scaling up and down, and priority adjustment. The window duration decision unit is used to accept the type of scheduling operation to be executed this time, calculate the upper quantile value of the time consumption under the preset confidence level according to the probability distribution model corresponding to the scheduling operation type, and output the upper quantile value of the time consumption as the minimum continuous time threshold. The minimum continuous time threshold is dynamically adjusted as the execution record of the scheduling operation type is updated.

6. The server cluster task scheduling optimization system adapted to large model training scenarios according to claim 1, characterized in that, The active window construction module includes a suspension control unit, a queue assembly unit, and a parallel execution unit; The suspension control unit is used to send a computation flow suspension signal to the graphics processor driver layer to which the training process belongs when the theoretical end time of the gradient synchronization phase is detected. By inserting unmet event synchronization points into the command queue of the graphics processor, subsequent computation kernels are queued and wait at the event synchronization points. The graphics processor memory state, unified computing device architecture context, and tensor data of the training process are completely preserved during the suspension. At the same time, the suspension control unit marks the central processing unit core occupied by the training process as preemptible for scheduling operations. The queue assembly unit is used to read the target window duration after the training process is suspended and to execute a multi-level progressive selection strategy: First priority selection layer: Select all non-critical path computation tasks that have not yet been executed within the current iteration cycle from the non-critical path computation subset, add them to the window period filling queue as basic filling items, and calculate the estimated total time of the basic filling items; Second priority selection layer: If the estimated total time of the basic filling items is less than the target window duration, calculate the time gap value, and select pre-execution tasks from the subset of non-critical paths in the next iteration cycle in descending order of estimated time, until the total estimated time of the accumulated pre-execution tasks is greater than or equal to the time gap value, and add the selected pre-execution tasks as supplementary filling items to the window filling queue. Third priority selection layer: If the total estimated time of all available tasks in the non-critical path computation subset is still less than the target window duration, then a placeholder computation task matching the remaining gap duration is generated. The placeholder computation task is a lightweight graphics processor computation kernel sequence with a precisely configurable execution duration. The placeholder computation task is added to the window period filling queue as a fallback filling item. The parallel execution unit is used to distribute each computation task in the window period filling queue to the graphics processor occupied by the training process for execution after the window period filling queue is assembled, and at the same time distribute the scheduling operation to the central processing unit core released by the suspend control unit for execution, so that the effective computation and scheduling operation can proceed in parallel within the window period.

7. The server cluster task scheduling optimization system adapted to large model training scenarios according to claim 1, characterized in that, The timing triggering module includes a runtime monitoring unit and a recovery control unit; The runtime monitoring unit is used to continuously monitor the graphics processor execution completion signal of each computing task in the window period filling queue, as well as the execution completion signal of the scheduling operation. The graphics processor execution completion signal is obtained by inserting an event record node at the end of the graphics processor command queue of each computing task in the window period filling queue. When the event record node is executed by the graphics processor, the corresponding execution completion callback is triggered. The recovery control unit is used to send a recovery instruction to the training process when the execution conditions are met. The execution conditions include the following: the graphics processor completion signals of all computing tasks in the window filling queue have been triggered, and the execution completion signal of the scheduling operation has been triggered. The recovery instruction is used to unfreeze the execution state of the training process and send a satisfaction signal to the unsatisfied event synchronization point inserted by the suspended control unit in the graphics processor command queue, so that the event synchronization point can be released and the subsequent computing kernel can switch from the queued state to the execution state.

8. The server cluster task scheduling optimization system adapted to large model training scenarios according to claim 7, characterized in that, The recovery control unit is also used to perform timeout processing when all computational tasks in the window period filling queue have been completed, but the scheduling operation has not returned an execution completion signal within the preset maximum waiting time limit. The timeout processing is used to send a recovery instruction to the training process, and at the same time mark the unfinished scheduling operation as an asynchronous compensation task, which will continue to be executed within the next safe window period determined in consultation with the active window construction module.

Images