Large model batch scheduling method and system based on shortest job priority

Abstract

The invention discloses a large-model batch scheduling method based on shortest job priority. The method comprises the steps that key information obtaining and preprocessing are conducted on a model processing task request; storing the tasks subjected to preliminary information processing into a uniformly managed to-be-scheduled task queue, and performing preliminary sorting based on a shortest job priority principle; a long-task anti-hunger mechanism is introduced, a scheduling sequence is intervened under specific conditions, and priority dynamic adjustment is carried out; according to the evaluated available resources and batch capacity, starting to select tasks from the task queue to be scheduled to form one or more new execution batches; the scheduler officially submits the constructed batch to a rear-end calculation engine where the large model is located for actual model processing; the processing result is returned to the user or the upstream system which initially submits the request according to a preset mode. The invention further discloses a large model batch scheduling system based on the shortest job priority. According to the invention, the overall service quality and user satisfaction of the large model service system are improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically a large-scale batch scheduling method and system based on the shortest job first principle. Background Technology

[0002] In recent years, the field of artificial intelligence has made groundbreaking progress. Among them, large-scale AI models, with Large Language Models (LLMs) as a prominent example, are rapidly penetrating and empowering various industries with their outstanding capabilities in natural language understanding, text generation, knowledge-based question answering, coding, and logical reasoning. These include areas such as intelligent customer service, content creation, scientific research assistance, educational tutoring, and software development. These advanced models share the common characteristic of exceptionally complex structures, with parameter scales often reaching billions, tens of billions, or even trillions. Such a massive number of parameters means that training these models to learn knowledge or performing inference to apply their capabilities in response to user requests requires extremely large computational resources. These resources mainly include the computation time of high-end graphics processing units (GPUs) or tensor processing units (TPUs), a large amount of video memory (VRAM) or system memory, and corresponding energy consumption. Therefore, the execution time of a single model inference or training iteration is usually relatively long, ranging from seconds to minutes or even hours, posing a severe challenge to the carrying capacity of computing infrastructure and the response efficiency of service systems.

[0003] Currently, several other scheduling strategies have been tried or applied in similar compute-intensive service systems. For example, the First-In, First-Out (FIFO) strategy is the simplest and most intuitive, strictly scheduling tasks according to their arrival order. This strategy guarantees absolute fairness, as the waiting time of each task is directly related to its arrival order. However, FIFO also has significant drawbacks: if a very long-running task arrives first and occupies processing resources, then even if a large number of short-running tasks arrive later, they must wait in the queue. This leads to a significant increase in the average response time of short tasks, thus lowering the average performance of the entire system. Other systems may employ priority-based scheduling strategies, where tasks are assigned different priorities, and the scheduler prioritizes tasks with higher priorities. While this strategy can alleviate some problems to some extent, such as providing priority service to VIP users or urgent tasks, the static setting or dynamic adjustment mechanism of priorities is often complex to design and requires careful consideration. More importantly, if the priority is still primarily based on task length (implicitly or explicitly), it may still fail to fundamentally solve the starvation problem inherent in long tasks in the SJF algorithm, unless the priority adjustment mechanism specifically considers the task's waiting time or history of being bypassed.

[0004] Therefore, how to design an effective and robust mechanism to avoid or significantly alleviate the starvation phenomenon of long tasks while fully leveraging the significant advantages of the SJF algorithm in high throughput and low average waiting time, and to ensure that all tasks (regardless of length) can be served within an acceptable time range, has become a technical problem that urgently needs to be solved and has important practical significance in the field of large-scale batch scheduling.

[0005] Therefore, a large-scale batch scheduling method and system based on the shortest job priority model is provided. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a large-scale model batch scheduling method and system based on shortest job priority, achieving a better balance between efficiency and fairness, and improving the overall service quality and user satisfaction of the large-scale model service system.

[0007] The technical solution to achieve the above objectives is: One of the present inventions is a large-scale batch scheduling method based on the shortest job first model, comprising: Step S1: Receive the model processing task request and acquire and preprocess its key information; Step S2: The tasks that have been received and processed are stored in a unified management queue of tasks to be scheduled, and are initially sorted based on the principle of shortest job priority. Step S3: Introduce a long-task anti-starvation mechanism to intervene in the scheduling order under specific conditions and dynamically adjust the priority. Step S4: At the beginning of each scheduling cycle, assess the current available computing resources and, based on the assessed available resources and batch capacity, begin selecting tasks from the queue of tasks to be scheduled to form one or more new execution batches. Step S5: After one or more batches are built, the scheduler will formally submit the completed batches to the backend computing engine where the large model resides for actual model processing. Step S6: Track the processing status of the submitted batch. Once the task or the entire batch is confirmed to be completed, the computing resources previously occupied are immediately released, and the processing results are returned to the user who originally submitted the request or the upstream system in a predetermined manner.

[0008] Preferably, in step S1, the application programming interface (API) for end users, the upstream business system within the enterprise, or other service modules that require model computing capabilities receive model processing task requests. Each task request contains core task data and metadata; The core task data includes the input text sequence, image data, and the parameter set to be processed; Meta information includes user identification, task submission timestamp, task type, user-specified priority, expected output format or length limit, and other business-related attributes that may affect scheduling.

[0009] Preferably, in step S1, the preprocessing operation includes: estimating the expected execution time for each newly received task; wherein, Estimation methods include: Task-feature-based rules or models: Explicit features such as task type, input data size, and output requirement complexity are used to estimate execution time through predefined rules, lookup tables, or a lightweight predictive model. Statistical analysis and machine learning prediction based on historical execution data: Record and analyze statistical data on the actual execution time of similar tasks in the past, or use more complex machine learning models to learn the mapping relationship between task characteristics and execution time from historical data, so as to make more accurate predictions of the execution time of new tasks. Hybrid strategy: Combining the two methods above, use precise predictions based on historical data for common task types, and use coarse estimations based on features for novel or rare task types.

[0010] Preferably, in step S2, the queue of tasks to be scheduled is the core data structure for scheduling decisions, and is implemented using a data structure suitable for frequent insertion, deletion and sorting operations; Based on the principle of shortest job first, under normal circumstances without triggering any special mechanisms, all tasks in the queue are sorted in ascending order according to their estimated execution time. in, The queue of tasks to be scheduled is not static; it grows dynamically as new tasks arrive and shrinks dynamically as tasks are scheduled, executed, and completed.

[0011] Preferably, in step S3, to prevent starvation, the status parameters of each task in the queue, or at least those potentially long tasks identified as possibly facing starvation, are recorded and continuously updated, including: Arrival timestamp: the exact time when the task initially arrived or entered the scheduling queue; Cumulative waiting time: how long the task has been waiting since it entered the scheduling queue until the current moment without being scheduled for execution; Skipped Count: Records the number of times a task is "skipped" by the Shortest Job First scheduler when a subsequent task arrives that is shorter than itself. Set threshold parameters for time and number of skips: Maximum waiting time threshold The maximum waiting time a task can tolerate in the queue. Once the actual waiting time of a task exceeds this threshold, it is considered to be in a state of starvation. Maximum number of skips threshold The maximum number of times a task can be skipped due to shorter tasks cutting in line; exceeding this number indicates that the task is being treated unfairly. When the anti-starvation mechanism detects that a task has reached the preset trigger conditions, it will activate the corresponding dynamic priority enhancement strategy, including: Strategy A, based on forced promotion according to absolute waiting time: Before each scheduling decision, iterate through and check the current waiting time of all tasks in the queue to be scheduled. If a task is found to have a certain waiting time, the strategy will be implemented. Waiting time The preset maximum waiting time threshold has been reached or exceeded. If so, the task's scheduling priority will be immediately raised to the highest level, or it will be given a special "priority scheduling" flag; during the subsequent batch build process, It will be selected unconditionally, and its high priority status can be reset once it is scheduled, even if there are other tasks in the queue with shorter estimated execution times. Strategy B, intervention based on cumulative skipped counts: This strategy focuses on the degree of unfair treatment of tasks. A counter is maintained for each long task, recording the number of times it has been "jumped" or "skipped" by newly arrived, shorter tasks. When a task... Recorded number of skips The maximum number of skips has been reached or exceeded. Similarly, the priority of this task will be significantly increased, and this counter can be cleared or reduced after each successful scheduling. Strategy C, based on aging-based incremental priority boosting: This is a smoother and more dynamic priority adjustment method. The final scheduling priority of a task comprehensively considers its base priority and the time it has been waiting, with a dynamic priority... It is calculated using a formula. ,in, It is the basic priority of the task. This is the current cumulative waiting time for the task. It is a positive "aging factor" or "weighting coefficient". The contribution of waiting time to the overall priority was controlled; As task waiting time increases, As it grows, its dynamic priority increases. As the number of tasks increases linearly or non-linearly, the scheduler will select tasks based on their dynamic priority within the current queue. The highest task.

[0012] Preferably, in step S4, the order of task selection follows the following logic: Prioritize tasks that prevent starvation: First, check if there are any long tasks that have been given higher priority because they triggered the starvation prevention mechanism in step S3. If such tasks exist and the current resources are sufficient to accommodate them, these tasks will be added to the current batch being built with the highest priority. The remaining capacity is filled according to the shortest job priority principle: if there is still space in the batch after all the tasks that have been upgraded in priority have been processed, or if no task is currently triggering the anti-starvation mechanism, then the scheduler will continue to select tasks to fill the batch according to the order of the tasks at the head of the queue, until the batch size limit is reached or resources are exhausted.

[0013] A second invention relates to a large-scale batch scheduling system based on the shortest job first model, comprising: The task receiving module is responsible for receiving externally received task requests and performing preliminary parsing and verification. The execution time estimation module is used to estimate the expected execution time for each newly received task; The task queue management module has a built-in core SJF sorting logic, which is responsible for maintaining the order of the queue of tasks to be scheduled and supports dynamic addition, deletion and sorting of tasks. The long-task anti-starvation module is responsible for implementing various dynamic priority promotion strategies, monitoring the waiting status of tasks, and adjusting the effective priority of the tasks in the queue management module when conditions are met. The batch building module is used to intelligently select tasks from the queue to form execution batches based on the current resource status, queue status, and batch constraints. The task execution and monitoring module is responsible for submitting the constructed batches to the actual large model calculation backend, tracking their execution progress, releasing resources after processing, and providing status feedback.

[0014] Preferably, the estimation method in the execution time estimation module includes: Task-feature-based rules or models: Explicit features such as task type, input data size, and output requirement complexity are used to estimate execution time through predefined rules, lookup tables, or a lightweight predictive model. Statistical analysis and machine learning prediction based on historical execution data: Record and analyze statistical data on the actual execution time of similar tasks in the past, or use more complex machine learning models to learn the mapping relationship between task characteristics and execution time from historical data, so as to make more accurate predictions of the execution time of new tasks. Hybrid strategy: Combining the two methods above, use precise predictions based on historical data for common task types, and use coarse estimations based on features for novel or rare task types.

[0015] Preferably, in the long-task anti-starvation module, to achieve starvation prevention, the status parameters of each task in the queue, or at least those potentially long tasks identified as possibly encountering starvation, are recorded and continuously updated, including: Arrival timestamp: the exact time when the task initially arrived or entered the scheduling queue; Cumulative waiting time: how long the task has been waiting since it entered the scheduling queue until the current moment without being scheduled for execution; Skipped Count: Records the number of times a task is "skipped" by the Shortest Job First scheduler when a subsequent task arrives that is shorter than itself. Set threshold parameters for time and number of skips: Maximum waiting time threshold The maximum waiting time a task can tolerate in the queue. Once the actual waiting time of a task exceeds this threshold, it is considered to be in a state of starvation. Maximum number of skips threshold The maximum number of times a task can be skipped due to shorter tasks cutting in line; exceeding this number indicates that the task is being treated unfairly. When the anti-starvation mechanism detects that a task has reached the preset trigger conditions, it will activate the corresponding dynamic priority enhancement strategy, including: Strategy A, based on forced promotion according to absolute waiting time: Before each scheduling decision, iterate through and check the current waiting time of all tasks in the queue to be scheduled. If a task is found to have a certain waiting time, the strategy will be implemented. Waiting time The preset maximum waiting time threshold has been reached or exceeded. If so, the task's scheduling priority will be immediately raised to the highest level, or it will be given a special "priority scheduling" flag; during the subsequent batch build process, It will be selected unconditionally, and its high priority status can be reset once it is scheduled, even if there are other tasks in the queue with shorter estimated execution times. Strategy B, intervention based on cumulative skipped counts: This strategy focuses on the degree of unfair treatment of tasks. A counter is maintained for each long task, recording the number of times it has been "jumped" or "skipped" by newly arrived, shorter tasks. When a task... Recorded number of skips The maximum number of skips has been reached or exceeded. Similarly, the priority of this task will be significantly increased, and this counter can be cleared or reduced after each successful scheduling. Strategy C, based on aging-based incremental priority boosting: This is a smoother and more dynamic priority adjustment method. The final scheduling priority of a task comprehensively considers its base priority and the time it has been waiting, with a dynamic priority... It is calculated using a formula. ,in, It is the basic priority of the task. This is the current cumulative waiting time for the task. It is a positive "aging factor" or "weighting coefficient". The contribution of waiting time to the overall priority was controlled; As task waiting time increases, As it grows, its dynamic priority increases. As the number of tasks increases linearly or non-linearly, the scheduler will select tasks based on their dynamic priority within the current queue. The highest task.

[0016] Compared with the prior art, the beneficial effects of the present invention are: The basic scheduling logic of this invention still inherits the core advantages of the Shortest Job First (SJF) algorithm. By prioritizing the processing of tasks with shorter estimated execution times, SJF enables a large number of tasks to be completed quickly and resources to be released. This strategy effectively reduces the average waiting time and average turnaround time of tasks in the queue. A shorter average turnaround time means that the system can process more tasks per unit of time, thereby directly improving the overall throughput of the entire large model service system. This is especially important for online inference services that need to respond quickly to a large number of concurrent requests, ensuring that the system maintains high processing efficiency under high load. The core improvement and contribution of this invention compared to the traditional SJF algorithm is the introduction of a specially designed long-task anti-starvation mechanism (e.g., dynamically increasing priority based on waiting time thresholds, skipped count thresholds, or aging mechanisms). This allows the system to proactively identify and intervene in long tasks that may be unable to receive service for extended periods due to being continuously "jumped in line" by shorter tasks. When the waiting time or unfair treatment of a long task reaches a certain threshold, its scheduling priority is dynamically and significantly increased, ensuring that it can obtain valuable computing resources and begin execution within the expected and reasonable time window. This fundamentally solves the "starvation" problem that long tasks may be delayed indefinitely or even never be able to execute under the pure SJF strategy, thereby greatly improving the fairness and robustness of the scheduling strategy. The anti-starvation mechanism provided by this invention is an important safeguard. It can be expected that although the task may need to wait for a period of time, it will not be ignored or forgotten by the system indefinitely. This predictable and lower-limited waiting time significantly improves the service quality perception and overall user satisfaction of such users, and avoids the frustration and doubts about service reliability caused by the task disappearing without a trace. The anti-starvation mechanism proposed in this invention is not a rigid, unchanging setting, but rather possesses high flexibility and adaptability. By carefully configuring key parameters such as the maximum waiting time threshold, the maximum number of skips threshold, or the aging factor in the aging mechanism, the scheduling strategy can be finely adjusted and optimized according to the actual application scenario characteristics, the current system load, and specific business objectives. This configurability allows the method of this invention to better adapt to diverse and dynamically changing workloads and maintain its effectiveness. This invention constructs batches by intelligently combining short tasks and timely inserting long tasks with increased priority. The system strives to keep valuable computing resources in a busy and effective working state for as long as possible, reducing resource idleness or waste caused by unreasonable scheduling. Therefore, while significantly improving fairness, this method is still committed to maintaining and even further optimizing the overall resource utilization through more refined batch construction, achieving a win-win situation for performance and fairness. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a large-scale batch scheduling method based on the shortest job first principle according to the present invention; Figure 2 This is a block diagram of a large-scale batch scheduling system based on the shortest job priority model of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, a large-scale batch scheduling method based on the shortest job first model includes: Step S1: Receive the model processing task request and acquire and preprocess its key information.

[0020] In this embodiment, the application programming interface (API) for end users, the upstream business system within the enterprise, or other service modules that require model computing capabilities receive model processing task requests; each received request represents an independent computing task, such as a text generation of a large language model, an image recognition, or a complex logical reasoning. Each task request contains core task data and metadata; The core task data includes the input text sequence, image data, and the parameter set to be processed; Meta information includes user identification, task submission timestamp, task type, user-specified priority, expected output format or length limit, and other business-related attributes that may affect scheduling.

[0021] In this embodiment, the preprocessing operation includes: estimating the expected execution time for each newly received task; wherein, Estimation methods include: Task-feature-based rules or models: Based on explicit features such as the type of task (e.g., text generation tasks are typically more time-consuming than simple classification tasks), the scale of the input data (e.g., the number of characters or tokens in the input text, the resolution and size of the image to be processed), and the complexity of the output requirements (e.g., the expected length of the generated text, the level of detail in the image), the execution time is estimated using predefined rules, lookup tables, or a lightweight predictive model (e.g., linear regression, decision trees). Statistical analysis and machine learning prediction based on historical execution data: The system can record and analyze statistical data (such as average and median) of the actual execution time of similar tasks (based on the similarity defined above), or use more complex machine learning models (such as gradient boosting trees and neural networks) to learn the mapping relationship between task features and execution time from historical data, thereby making more accurate predictions of the execution time of new tasks. Hybrid strategy: Combining the two methods above, use precise predictions based on historical data for common task types, and use coarse estimations based on features for novel or rare task types.

[0022] In step S2, the tasks that have been received and processed are stored in a unified management queue of tasks to be scheduled, and are initially sorted based on the principle of shortest job priority.

[0023] In this embodiment, the queue of tasks to be scheduled is the core data structure for scheduling decisions, and it is implemented using a data structure suitable for frequent insertion, deletion and sorting operations; Based on the principle of shortest job first, under normal circumstances without triggering any special mechanisms (such as anti-starvation mechanisms), all tasks in the queue are sorted in ascending order according to their estimated execution time. This means that the task with the shortest estimated execution time will be placed at the front of the queue (or has the highest natural priority), while the task with the longest estimated execution time will be placed at the end of the queue. When the scheduler builds the next execution batch, it will prioritize selecting tasks from the head of the queue. in, The queue of tasks to be scheduled is not static; it grows dynamically as new tasks arrive and shrinks dynamically as tasks are scheduled, executed, and completed. Therefore, the sorting logic needs to be able to efficiently handle these dynamic changes and ensure that the queue always (or before each scheduling decision) maintains the SJF order.

[0024] Step S3 introduces a long-task anti-starvation mechanism, which intervenes in the scheduling order under specific conditions and dynamically adjusts the priority.

[0025] In this embodiment, to prevent starvation, status parameters are recorded and continuously updated for each task in the queue, or at least those potentially long tasks identified as possibly facing starvation, including: Arrival timestamp: the exact time when the task initially arrived or entered the scheduling queue; Cumulative waiting time: how long the task has been waiting since it entered the scheduling queue until the current moment without being scheduled for execution; Skipped Count: Records the number of times a task is "skipped" by the Shortest Job First scheduler when a subsequent task arrives that is shorter than itself. Set threshold parameters for time and number of skips: Maximum waiting time threshold The maximum waiting time a task can tolerate in the queue. Once the actual waiting time of a task exceeds this threshold, it is considered to be in a state of starvation. Maximum number of skips threshold The maximum number of times a task can be skipped due to shorter tasks cutting in line; exceeding this number indicates that the task is being treated unfairly. When the anti-starvation mechanism detects that a task has reached the preset trigger conditions, it will activate the corresponding dynamic priority enhancement strategy, including: Strategy A, based on forced promotion according to absolute waiting time: Before each scheduling decision, iterate through and check the current waiting time of all tasks in the queue to be scheduled. If a task is found to have a certain waiting time, the strategy will be implemented. Waiting time The preset maximum waiting time threshold has been reached or exceeded. If so, the task's scheduling priority will be immediately raised to the highest level, or it will be given a special "priority scheduling" flag; during the subsequent batch build process, It will be selected unconditionally, and its high priority status can be reset once it is scheduled, even if there are other tasks in the queue with shorter estimated execution times. Strategy B, intervention based on cumulative skipped counts: This strategy focuses on the degree of unfair treatment of tasks. A counter is maintained for each long task, recording the number of times it has been "jumped" or "skipped" by newly arrived, shorter tasks. When a task... Recorded number of skips The maximum number of skips has been reached or exceeded. Similarly, the priority of this task will be significantly increased, and this counter can be cleared or reduced after each successful scheduling. Strategy C, based on aging-based incremental priority boosting: This is a smoother and more dynamic priority adjustment method. The final scheduling priority of a task comprehensively considers its base priority and the time it has been waiting, with a dynamic priority... It is calculated using a formula. ,in, It is the basic priority of the task. This is the current cumulative waiting time for the task. It is a positive "aging factor" or "weighting coefficient". The contribution of waiting time to the overall priority was controlled; As task waiting time increases, As it grows, its dynamic priority increases. As the number of tasks increases linearly or non-linearly, the scheduler will select tasks based on their dynamic priority within the current queue. The highest task.

[0026] Step S4: At the beginning of each scheduling cycle, assess the current available computing resources and, based on the assessed available resources and batch capacity, begin selecting tasks from the queue of tasks to be scheduled to form one or more new execution batches.

[0027] In this embodiment, the order of task selection follows the following logic: Prioritize tasks that prevent starvation: First, check if there are any long tasks that have been given higher priority because they triggered the starvation prevention mechanism in step S3. If such tasks exist and the current resources are sufficient to accommodate them, these tasks will be added to the current batch being built with the highest priority. The remaining capacity is filled according to the shortest job priority principle: if there is still space in the batch after all the tasks that have been upgraded in priority have been processed, or if no task is currently triggering the anti-starvation mechanism, then the scheduler will continue to select tasks to fill the batch according to the order of the tasks at the head of the queue, until the batch size limit is reached or resources are exhausted.

[0028] In step S5, after one or more batches are built, the scheduler formally submits the completed batches to the backend computing engine where the large model resides for actual model processing; when all tasks in a batch have been processed by the large model, the execution engine will notify the scheduling system.

[0029] Step S6: Track the processing status of submitted batches. Once a task or the entire batch is confirmed to be completed, the computing resources previously occupied (such as GPU time slices and video memory space) are immediately released so that they can be used to schedule and execute subsequent tasks, ensuring efficient resource flow. The processing results are returned to the user who initially submitted the request or the upstream system in a predetermined manner.

[0030] In this embodiment, the processing results are returned to the user who initially submitted the request or the upstream system in a predetermined manner. Simultaneously, the scheduling system updates the status information of these completed tasks (e.g., marking them from "in execution" to "completed," recording their actual execution time, resource consumption, etc.). This information can be very useful for subsequent performance analysis, billing, and optimization of execution time estimation algorithms. The task queue management module 3 also removes completed tasks accordingly.

[0031] like Figure 2 As shown, a large-scale batch scheduling system based on the shortest job first model includes: a task receiving module 1, an execution time estimation module 2, a task queue management module 3, a long task starvation prevention module 4, a batch construction module 5, and a task execution and monitoring module 6.

[0032] Task receiving module 1 is responsible for receiving externally received task requests and performing preliminary parsing and verification.

[0033] Execution time estimation module 2 is used to estimate the expected execution time of each newly received task.

[0034] In this embodiment, the estimation method includes: Task-feature-based rules or models: Explicit features such as task type, input data size, and output requirement complexity are used to estimate execution time through predefined rules, lookup tables, or a lightweight predictive model. Statistical analysis and machine learning prediction based on historical execution data: Record and analyze statistical data on the actual execution time of similar tasks in the past, or use more complex machine learning models to learn the mapping relationship between task characteristics and execution time from historical data, so as to make more accurate predictions of the execution time of new tasks. Hybrid strategy: Combining the two methods above, use precise predictions based on historical data for common task types, and use coarse estimations based on features for novel or rare task types.

[0035] Task queue management module 3 has a built-in core SJF sorting logic, which is responsible for maintaining the order of the queue of tasks to be scheduled and supports dynamic addition, deletion and sorting of tasks.

[0036] The long-task anti-starvation module 4 is responsible for implementing various dynamic priority promotion strategies, monitoring the waiting status of tasks, and adjusting the effective priority of tasks in the queue management module 3 when conditions are met.

[0037] In this embodiment, to prevent starvation, status parameters are recorded and continuously updated for each task in the queue, or at least those potentially long tasks identified as possibly facing starvation, including: Arrival timestamp: the exact time when the task initially arrived or entered the scheduling queue; Cumulative waiting time: how long the task has been waiting since it entered the scheduling queue until the current moment without being scheduled for execution; Skipped Count: Records the number of times a task is "skipped" by the Shortest Job First scheduler when a subsequent task arrives that is shorter than itself. Set threshold parameters for time and number of skips: Maximum waiting time threshold The maximum waiting time a task can tolerate in the queue. Once the actual waiting time of a task exceeds this threshold, it is considered to be in a state of starvation. Maximum number of skips threshold The maximum number of times a task can be skipped due to shorter tasks cutting in line; exceeding this number indicates that the task is being treated unfairly. When the anti-starvation mechanism detects that a task has reached the preset trigger conditions, it will activate the corresponding dynamic priority enhancement strategy, including: Strategy A, based on forced promotion according to absolute waiting time: Before each scheduling decision, iterate through and check the current waiting time of all tasks in the queue to be scheduled. If a task is found to have a certain waiting time, the strategy will be implemented. Waiting time The preset maximum waiting time threshold has been reached or exceeded. If so, the task's scheduling priority will be immediately raised to the highest level, or it will be given a special "priority scheduling" flag; during the subsequent batch build process, It will be selected unconditionally, and its high priority status can be reset once it is scheduled, even if there are other tasks in the queue with shorter estimated execution times. Strategy B, intervention based on cumulative skipped counts: This strategy focuses on the degree of unfair treatment of tasks. A counter is maintained for each long task, recording the number of times it has been "jumped" or "skipped" by newly arrived, shorter tasks. When a task... Recorded number of skips The maximum number of skips has been reached or exceeded. Similarly, the priority of this task will be significantly increased, and this counter can be cleared or reduced after each successful scheduling. Strategy C, based on aging-based incremental priority boosting: This is a smoother and more dynamic priority adjustment method. The final scheduling priority of a task comprehensively considers its base priority and the time it has been waiting, with a dynamic priority... It is calculated using a formula. ,in, It is the basic priority of the task. This is the current cumulative waiting time for the task. It is a positive "aging factor" or "weighting coefficient". The contribution of waiting time to the overall priority was controlled; As task waiting time increases, As it grows, its dynamic priority increases. As the number of tasks increases linearly or non-linearly, the scheduler will select tasks based on their dynamic priority within the current queue. The highest task.

[0038] Batch building module 5 is used to intelligently select tasks from the queue to form execution batches based on the current resource status, queue status, and batch constraints.

[0039] Task execution and monitoring module 6 is responsible for submitting the constructed batches to the actual large model calculation backend, tracking their execution progress, and releasing resources and providing status feedback after completion.

[0040] These modules interact and work together through well-defined interfaces to achieve the efficient and fair batch scheduling method described in this invention.

[0041] To more clearly and specifically illustrate the actual operation of the "shortest job priority-based large model batch scheduling method" proposed in this invention and its beneficial effects, the following will be explained through two detailed embodiments with different focuses.

[0042] Example 1: Operation process of a long-task starvation prevention mechanism based on waiting time threshold Suppose a large model inference service system (e.g., an LLM cluster providing text generation services) adopts the batch scheduling method described in this invention and is specifically configured with an anti-starvation strategy based on a waiting time threshold. The key scheduling parameters of the system are set as follows: Maximum waiting time threshold The time limit is set to 60 seconds. This means that any task waiting in the scheduling queue for 60 seconds or more will have its priority forcibly increased.

[0043] Maximum number of tasks per batch: Set to 4 tasks. This is determined by hardware or system configuration to balance parallelism with the performance of individual tasks.

[0044] Execution time estimation: Assume that the system is equipped with a relatively accurate task execution time estimation module.

[0045] Batch processing overhead: It is assumed that the actual execution time of a batch is mainly determined by the execution time of the longest task in the batch, and some inherent batch processing overhead (such as data aggregation, result distribution, etc.) is also considered.

[0046] The detailed scheduling process is simulated as follows: Tasks arrive at the system in sequence, and execution time is estimated: Task T1: At the initial reference time The estimated execution time of the system. Second.

[0047] Task T2: At time The estimated execution time is seconds until it reaches the system. Second.

[0048] Task T3 (a relatively long task): at time The estimated execution time is seconds until it reaches the system. Second.

[0049] Task T4: At time The estimated execution time is seconds until it reaches the system. Second.

[0050] Task T5: At time The system arrives in seconds, and its estimated execution time is seconds.

[0051] The first scheduling decision (approximately at time) (Seconds later, when T5 arrives and the system prepares to build the first batch): Scheduled queue status: At this time, all arrived tasks T1, T2, T3, T4, and T5 are in the scheduled queue. According to the SJF principle (arranged in ascending order of estimated execution time), the queue order should be: T5 (4s), T1 (5s), T4 (6s), T2 (8s), T3 (30s).

[0052] Anti-starvation mechanism check: The system checks the waiting time of each task in the queue.

[0053] The waiting time for T5 is approximately 0 seconds (upon arrival).

[0054] The waiting time for T1 is approximately ( )- =4s.

[0055] The waiting time for T4 is approximately ( )-( =1s.

[0056] The waiting time for T2 is approximately ( ) - ( = 3s.

[0057] The waiting time for T3 is ( ) - ( = 2s. Since the waiting time for all tasks (especially T3's 2 seconds) is much smaller than the preset maximum waiting time threshold... =60s, therefore, no task triggers the priority increase of the starvation prevention mechanism at this time.

[0058] Constructing the first batch P1: The scheduler selects tasks from the head of the queue in SJF order until the maximum number of tasks in a batch (4) is reached. Therefore, T5, T1, T4, and T2 are selected to form batch P1. Task T3, because it has the longest execution time, was not selected in this round and continues to wait in the queue.

[0059] Estimated execution time of batch P1: Assuming that the execution time of batch P1 is determined by the execution time of the longest task T2 (8 seconds), plus a certain batch processing overhead (e.g., 2 seconds), the total execution time of batch P1 is approximately 8s + 2s = 10s.

[0060] The second scheduling decision (approximately at time ( +4s + 10s) = Around +14s, i.e., after batch P1 has finished executing): New task arriving during the period: Suppose that within 10 seconds of batch P1 being executed, a new short task T6 arrives.

[0061] Task T6: At time It arrives at the system in +10s, and its estimated execution time is... =7s.

[0062] Scheduled queue status: At this time, the remaining tasks in the queue are T3 and the newly arrived T6. Sorted by SJF, the queue order is: T6 (7s), T3 (30s).

[0063] Checking the anti-starvation mechanism: The waiting time for T6 is approximately ( +14) - ( +10) = 4s.

[0064] The cumulative waiting time for T3 is updated to ( +14) - ( +2) = 12s. At this point, the waiting time of T3, 12 seconds, is still less than =60s, the starvation prevention mechanism was not triggered.

[0065] To construct the second batch P2: the scheduler selects the shortest task T6 in the current queue. Since the maximum number of tasks in a batch is 4, but there are only two tasks in the queue, and T3 is relatively long, if there are no other strategies (such as executing even when the batch is not full), it may only select T6 to form batch P2. Assume that the execution time of batch P2 (containing only T6) is 7s + (batch processing overhead, such as 1s) = 8s. Task T3 then continues to wait in the queue.

[0066] Subsequent scheduling process and triggering of starvation handling for long tasks: Simulate the arrival of continuous short tasks: Assume that during the execution of batch P2 and for a period of time thereafter, the system continuously receives a series of new tasks with relatively short estimated execution times (e.g., T7, T8, T9, ..., whose execution times are all less than 10 seconds, or even much less than the 30 seconds of T3).

[0067] T3 continues to wait: Due to the principle of SJF, these newly arrived short tasks will be prioritized and executed before T3, causing the waiting time of T3 to accumulate continuously.

[0068] Anti-starvation mechanism triggered: As time passes, the cumulative waiting time of task T3 finally reaches a preset threshold. For example, at a certain scheduling decision point, assuming the current time is... +62s (At this point, the arrival time of T3 is...) +2s), then the actual waiting time W(T3) = ( +62s)−( +2s)=60s. Since W(T3)=60s≥ =60s, the long-duration task starvation prevention mechanism is officially triggered.

[0069] T3 Priority Promotion and Scheduling: In the next scheduling decision (e.g., immediately following time...) After +62 seconds, when a batch has finished executing and the system is preparing to build a new batch Px, the anti-starvation module will detect that T3 has reached its waiting time limit. Therefore, the scheduling priority of T3 will be dynamically increased to the highest (or it will be given a special flag to ensure its priority execution).

[0070] Scheduling Result: At this point, even if other newly arrived tasks with very short estimated execution times (e.g., task T10, estimated execution time 3 seconds; task T11, estimated execution time 5 seconds) exist simultaneously in the scheduling queue, the system will prioritize adding T3 to the next batch Px to be built (assuming sufficient resources to accommodate T3) because T3's priority has been forcibly increased by the anti-starvation mechanism. If batch Px still has remaining capacity, it will then select from tasks such as T10 and T11 to fill the remaining capacity according to the SJF principle.

[0071] This example clearly demonstrates that the anti-starvation mechanism based on a waiting time threshold ensures that even with a continuous influx of short tasks, long task T3 will not be postponed indefinitely. Once its waiting time reaches the preset upper limit, it will have the opportunity to be executed first.

[0072] Example 2: Dynamic Priority Adjustment and Starvation Prevention Based on Aging Mechanisms In this embodiment, the scheduling system employs strategy C, the aging mechanism described in this invention, to dynamically adjust task priorities, thereby indirectly preventing starvation of long-running tasks. Dynamic task priorities The following simplified formula can be used for calculation (the formula may be more complex in practical applications and will be normalized): =(Estimated execution time 1)+ ; in: The base priority is simplified to the reciprocal of the estimated execution time, which means that the shorter the execution time, the higher the base priority.

[0073] Scheduling decision-making scenario analysis: Suppose that at a specific scheduling decision point in time, there are three tasks A, B, and C in the queue to be scheduled, and their status information and dynamic priorities are calculated as follows: Task A: Estimated execution time =5s.

[0074] Time already waiting in the queue =10s.

[0075] Dynamic priority calculation: =(1 / 5)+(0.1 10) = 0.20 + 1.0 = 1.20.

[0076] Task B (a relatively long task): Estimated execution time =20s.

[0077] Time already waiting in the queue =50s (indicating that it has been waiting for a relatively long time).

[0078] Dynamic priority calculation: =(1 / 20)+(0.1 50) = 0.05 + 5.0 = 5.05.

[0079] Task C (a newly arrived short task): Estimated execution time =3s.

[0080] The time it has been waiting in the queue since arriving at the system =0s.

[0081] Dynamic priority calculation: =(1 / 3)+(0.1⋅0)≈0.33+0.0=0.33.

[0082] Scheduling decision-making process: When making scheduling decisions, the system no longer simply compares the estimated execution times of tasks, but rather their dynamic priorities calculated through an aging mechanism. : =1.20 =5.05 =0.33 Compare the dynamic priority values ​​of the three: (5.05)> (1.20)> (0.33).

[0083] Scheduling result analysis: Based on the comparison of dynamic priorities, task B currently has the highest dynamic priority (5.05). Therefore, although task C has the shortest estimated execution time among all tasks (3 seconds, which pure SJF would choose), and task A is also much shorter than task B, task B has accumulated a very long waiting time (50 seconds). This waiting time, under the influence of the aging factor, significantly increases its overall priority, making it surpass other tasks. Therefore, the scheduling system will prioritize and schedule task B for execution.

[0084] This embodiment demonstrates that, through an aging mechanism, the priority of a task dynamically increases as its waiting time grows. This mechanism gradually enhances the "competitiveness" of long-waiting tasks (typically long tasks, but also potentially short tasks blocked by consecutive short tasks), eventually enabling them to surpass newly arriving short tasks, thereby effectively preventing starvation and achieving smoother fairness.

[0085] Conclusions and further explanations Through the two carefully designed embodiments described above, we can clearly observe that the shortest job first (SJF) based large-scale batch scheduling method proposed in this invention, by combining specific long-task anti-starvation mechanisms (whether it's forced boosting based on waiting time thresholds or dynamic priority adjustment based on aging mechanisms), can effectively ensure that tasks with long execution times or excessively long waiting times are not indefinitely delayed by the system, while maintaining the core advantages of high average efficiency and high throughput brought by the SJF algorithm. This achieves a very practical and important balance between pursuing ultimate computational efficiency and ensuring service fairness.

[0086] In practical industrial applications, several factors are crucial to the final performance of the method described in this invention and can serve as directions for further optimization and adjustment: Accuracy of Execution Time Estimation: The effectiveness of SJF and its improved strategies heavily relies on accurate predictions of task execution times. Excessive estimation errors can lead to flawed scheduling decisions, reducing system efficiency or affecting the effectiveness of anti-starvation mechanisms. Therefore, continuously optimizing the execution time prediction model (e.g., by collecting richer features, employing more advanced machine learning techniques, or conducting more refined analysis of historical data) is crucial for improving overall scheduling performance.

[0087] Dynamic tuning of threshold or aging factor: used in the examples Threshold or aging factor These parameters are not static. In complex production environments, they can be dynamically adjusted based on the system's real-time load, task arrival patterns (e.g., a sudden increase in short tasks or a rise in the proportion of long tasks), and the business's varying priorities regarding fairness and efficiency. For example, adaptive algorithms can be designed to automatically adjust these thresholds based on the frequency of historical starvation events or trends in average waiting times to achieve optimal performance.

[0088] Further refinement of batch building strategies: In addition to basic SJF and starvation prevention logic, the batch building module can integrate more complex strategies. For example, whether to allow dynamic batching (i.e., flexibly adjusting the size and composition of each batch based on task characteristics and current resource availability, rather than a fixed batch size), whether to consider task affinity (where processing certain tasks together may be more efficient), or how to handle extremely long tasks with increased priority when resources are scarce (e.g., whether to allow partial execution or require waiting for a larger resource window). These are all aspects that can be further studied and optimized to maximize resource utilization and system performance.

[0089] In summary, this invention provides a powerful and flexible framework, offering a practical technical approach to solving the task scheduling challenges in the era of large models, particularly the starvation problem in SJF (Survival of the Fittest) models. Through understanding the core mechanisms and carefully configuring and optimizing relevant parameters, the overall service quality and operational efficiency of large model service systems can be significantly improved.

[0090] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A shortest job first-based large model batch scheduling method, characterized in that, Comprise: Step S1, receiving model processing task requests and obtaining key information and preprocessing; Step S2, storing the tasks received and preliminarily processed in the unified management of the to-be-scheduled task queue, and preliminarily sorting based on the shortest job first principle; Step S3, introducing a long task anti-starvation mechanism to intervene in the scheduling order under certain conditions and dynamically adjust the priority; Step S4, at the beginning of each scheduling cycle, evaluate the current available computing resource status, and according to the evaluated available resources and batch capacity, start selecting tasks from the to-be-scheduled task queue to form one or more new execution batches; Step S5, after the completion of one or more batches, the scheduler will formally submit the completed batch to the backend computing engine where the large model is located for actual model processing; Step S6, track the processing status of the submitted batch, and once the task or the entire batch is confirmed to be completed, the computing resources previously occupied by it are immediately released, and the processing results are returned to the user or the upstream system that originally submitted the request in the predetermined manner.

2. The shortest job first based large model batch scheduling method according to claim 1, characterized in that, In step S1, the application program interface API facing the end user, the upstream business system within the enterprise, or other service modules requiring model computing capacity receive model processing task requests; Each task request contains core task data and meta information; The core task data includes the input text sequence, image data, and the parameter set to be processed; The meta information includes user identity, task submission timestamp, task type, user-specified priority, expected output format or length limit, and other business-related attributes that may affect scheduling.

3. The shortest job first based large model batch scheduling method according to claim 1, wherein, In step S1, the preprocessing operation includes estimating the expected execution time of each newly received task; wherein, The estimation methods include: Based on task features: according to the type of the task, the size of the input data, and the complexity of the output requirement, estimate the execution time through predefined rules, lookup tables, or a lightweight prediction model; Statistical analysis and machine learning prediction based on historical execution data: record and analyze statistical data of similar tasks' actual execution time in the past, or use more complex machine learning models to learn the mapping relationship between task features and execution time from historical data, thereby making more accurate predictions of the execution time of new tasks; Hybrid strategy: combine the above two methods, use precise prediction based on historical data for common task types, and use rough estimation based on features for novel or rare task types.

4. The shortest job first based large model batch scheduling method according to claim 1, wherein, In step S2, the to-be-scheduled task queue is the core data structure for scheduling decisions, and is implemented using a data structure suitable for frequent insertion, deletion, and sorting operations; Based on the shortest job first principle, in the regular case without triggering any special mechanism, all tasks in the queue are sorted in ascending order according to their estimated execution time; Wherein, The to-be-scheduled task queue is not static, dynamically growing with the arrival of new tasks, and dynamically shrinking as tasks are scheduled and completed.

5. The shortest job first based large model batch scheduling method according to claim 1, wherein, In step S3, in order to achieve starvation prevention, a state parameter is recorded and continuously updated for each task in the queue or at least those potential long tasks identified as likely to encounter starvation, including: Arrival timestamp, the exact time point when the task first arrives or enters the pending queue; Cumulative waiting duration, how long the task has been waiting for scheduling since it entered the pending queue; Number of times of being skipped, the number of times a task is "skipped" by the shortest job first scheduler after the arrival of a subsequent task shorter than it; Threshold parameters are set for time and number of times of being skipped: maximum latency threshold the maximum latency a task can tolerate in the queue, beyond which the task is considered to be in a starvation state; maximum skipped count threshold the maximum number of times a task can be skipped due to shorter tasks pre-empting it, beyond which it is deemed to be suffering from unfairness When the monitoring of the starvation prevention mechanism finds that a task has reached the preset triggering condition, the corresponding priority dynamic promotion strategy is started, including: Strategy A, absolute waiting time based forced promotion: before each scheduling decision, check the current waiting time of all tasks in the pending queue, if a task is found whose waiting time has reached or exceeded the preset maximum waiting time threshold, immediately promote the scheduling priority of the task to the highest level, or give it a special "priority scheduling" mark; in the next batch construction process, will be selected unconditionally, even if there are other tasks in the queue with shorter estimated execution time, once scheduled, its high priority state can be reset;​​​ Strategy B, intervention based on accumulated skipped times: this strategy focuses on the degree of unfairness of the task, maintaining a counter for each long task, recording the number of times it is "jumped" or "skipped" because of the arrival of new tasks with shorter execution times, when a certain task the recorded skipped times reach or exceed a preset maximum skipped times threshold , likewise, the priority of this task will be significantly improved, and this counter can be cleared or reduced after each successful scheduling; Strategy C, Aging-based Progressive Priority Promotion: for a more smooth and dynamic priority adjustment method, the final scheduling priority of a task is determined by considering both its base priority and the time it has been waiting for, its dynamic priority is calculated by a formula, where, is the base priority of the task, is the current accumulated waiting time of the task, is a positive "aging factor" or "weighting factor", the aging factor controls the degree of contribution of the waiting time to the total priority; As task waiting time increases, As it grows, its dynamic priority increases. As the number of tasks increases linearly or non-linearly, the scheduler will select tasks based on their dynamic priority within the current queue. The highest task.

6. The shortest job first based large model batch scheduling method according to claim 1, wherein, In step S4, the selection order of tasks follows the following logic: Prioritize processing starvation prevention tasks: first check if there are long tasks whose priority has been raised due to the triggering of the starvation prevention mechanism in step S3, if such tasks exist and the current resources are sufficient to accommodate them, these tasks will be prioritized to be added to the currently constructed batch; Fill the remaining capacity according to the shortest job first principle: if there is still space in the batch after processing all the tasks whose priority has been raised, or if no task triggers the starvation prevention mechanism, the scheduler will continue to select tasks to fill the batch according to the order of the tasks at the head of the queue until the batch size limit is reached or the resources are exhausted.

7. A scheduling system based on the shortest job first-based large model batch scheduling method of claim 1, characterized in that, It includes: Task receiving module, responsible for receiving external incoming task requests and performing preliminary parsing and verification; Execution time estimation module, used to estimate the expected execution time of each newly received task; Task queue management module, with built-in core SJF sorting logic, responsible for maintaining the order of the pending task queue, supporting dynamic addition and deletion and sorting of tasks; Long task starvation prevention module, responsible for implementing various priority dynamic promotion strategies, monitoring the waiting state of tasks, and adjusting the effective priority of the tasks in the queue management module when the conditions are met; Batch construction module, used to intelligently select tasks from the queue to form an execution batch according to the current resource status, queue state, and batch constraints; Task execution and monitoring module, responsible for submitting the constructed batch to the actual large model computing backend, tracking its execution progress, and releasing resources and providing state feedback after completion.

8. The dispatch system of claim 7, wherein, In the execution time estimation module, the estimation methods include: Rules or models based on task features: according to the type of task, the size of input data, and the complexity of output requirements, the execution time is estimated through predefined rules, lookup tables, or a lightweight prediction model; Statistical analysis and machine learning prediction based on historical execution data: record and analyze statistical data of similar tasks' actual execution time in the past, or use more complex machine learning models to learn the mapping relationship between task features and execution time from historical data, so as to make more accurate predictions of the execution time of new tasks; Hybrid strategy: combine the above two methods, use precise prediction based on historical data for common task types, and use rough estimation based on features for novel or rare task types.

9. The dispatch system of claim 7, wherein, In the long task anti-starvation module, in order to realize anti-starvation, a state parameter is recorded and continuously updated for each task in the queue or at least those potential long tasks identified as likely to encounter starvation, including: Arrival timestamp: the exact time point when the task first arrives or enters the pending queue; Cumulative waiting time: how long has the task been waiting for scheduling execution since it entered the pending queue until the current time; Number of times of being skipped: record the number of times a task is "skipped" by the shortest job first scheduler after the arrival of a subsequent task shorter than it; Threshold parameters are set for time and number of times of being skipped: maximum latency threshold the maximum latency a task can tolerate in the queue, beyond which the task is considered to be in a starvation state; maximum skipped count threshold the maximum number of times a task can be skipped due to shorter tasks pre-empting it, beyond which it is deemed to be suffering from unfairness When the monitoring of the anti-starvation mechanism finds that a task has reached the preset trigger condition, the corresponding priority dynamic promotion strategy will be started, including: Strategy A, absolute waiting time based forced promotion: before each scheduling decision, check the current waiting time of all tasks in the pending queue, if a task is found whose waiting time has reached or exceeded the preset maximum waiting time threshold, immediately promote the scheduling priority of the task to the highest level, or give it a special "priority scheduling" mark; in the next batch construction process, will be selected unconditionally, even if there are other tasks in the queue that are estimated to be shorter in execution time, once scheduled, its high priority state can be reset;​​​ Strategy B, intervention based on accumulated skipped times: this strategy focuses on the degree of unfairness of the task, maintaining a counter for each long task, recording the number of times it is "jumped" or "skipped" because of the arrival of new tasks with shorter execution times, when a certain task the recorded skipped times reach or exceed a preset maximum skipped times threshold , likewise, the priority of this task will be significantly increased, and this counter can be reset or reduced each time it is successfully scheduled; Strategy C, Aging-based Progressive Priority Promotion: for a more smooth and dynamic priority adjustment method, the final scheduling priority of a task is determined by considering both its base priority and the time it has been waiting for, its dynamic priority is calculated by a formula, where, is the base priority of the task, is the current accumulated waiting time of the task, is a positive "aging factor" or "weighting factor", the aging factor controls the degree of contribution of the waiting time to the total priority. As task waiting time increases, As it grows, its dynamic priority increases. As the number of tasks increases linearly or non-linearly, the scheduler will select tasks based on their dynamic priority within the current queue. The highest task.

Images