A multi-task flow scheduling method and system based on SLO constraint

Abstract

A multi-task flow scheduling method and system based on SLO constraints is disclosed. The method reads workflow templates and the performance results of each model for different tasks; it filters the candidate model list for each task based on an accuracy threshold, and rolls back to all available models when the set is empty; it schedules all workflows uniformly in an event-driven manner, selecting the model expected to complete the task earliest from the candidate list for each ready task, until all tasks are scheduled, obtaining a basic execution plan and its completion time; with the completion time budget as a constraint, tasks are replaced one by one with more accurate models according to the ratio of accuracy gain to time increment from high to low; after feasibility verification, valid replacements are accepted, and the upgraded execution plan is output. This invention coordinates the queuing competition of heterogeneous models from a global perspective, using the completion time budget as a continuous knob to achieve a flexible trade-off between time cost and accuracy, effectively improving the overall efficiency and service quality assurance capability of heterogeneous model clusters.

Description

Technical Field

[0001] This invention relates to large model inference services and workflow scheduling technology, and in particular to a multi-task flow scheduling method and system based on SLO constraints. Background Technology

[0002] With the rapid popularization of large language model inference services, concurrent processing of multi-task workflows has become a typical feature of large model applications and cloud platforms. From multi-turn dialogue generation in intelligent question answering to static checking, unit testing, and code completion in code generation pipelines, and multi-step inference in knowledge retrieval workflows, various scenarios exhibit significant characteristics such as heterogeneous task types, diverse model selections, and closely related processes. Different models show significant differences in accuracy, response latency, and computational cost: small-parameter specialized models (such as mathematical models and code-specific models) offer considerable accuracy, speed, and low cost for specific tasks, while general-purpose large models achieve higher accuracy but experience significantly increased latency and cost. Therefore, how to allocate appropriate models to each task while meeting the service quality constraints of each workflow has become a key issue in improving the overall efficiency of inference service clusters.

[0003] Currently, most mainstream multi-task workflow scheduling methods employ either a single-model fixed execution or a task-level greedy selection strategy. Single-model fixed execution means that each task in the workflow is pre-bound to a unique model, and the scheduling system distributes tasks sequentially according to their execution order, without considering the matching relationship between task type and model capabilities. Task-level greedy selection means that for each task encountered, the model with the highest accuracy or lowest latency is selected for execution, without considering the overall model load and execution progress from a multi-workflow global perspective.

[0004] However, the above scheduling method has some drawbacks that cannot be ignored: First, because the fixed execution method of a single model does not consider the matching relationship between task type and model capability, the utilization rate of heterogeneous models is low: specialized models that are good at coding may be assigned to language tasks, while general large models may be assigned to simple tasks, resulting in wasted computing power and loss of accuracy. Second, because the task-level greedy selection method only selects the best solution for a single task at the moment, it does not coordinate the queuing competition of shared model services from the global perspective of multiple workflows. During peak periods, multiple workflows compete for the same model at the same time, resulting in a surge in latency and SLO default. Third, while existing scheduling methods based on accuracy thresholds can filter out low-accuracy models, the threshold is a scalar knob that applies to each problem in a "one-size-fits-all" manner: the threshold value is discrete and cannot guarantee that a feasible solution will be provided within a given deadline. Furthermore, when the threshold value is too high, the candidate model set becomes too narrow, causing congestion of slow models and resulting in the failure phenomenon of "the more conservative, the worse". Fourth, existing methods lack a mechanism to use completion time as an explicit budget and systematically improve overall accuracy within the budget. The accuracy margin between fast and slow models (fast models execute quickly but get some questions wrong, while slow models can answer questions correctly but are idle) cannot be automatically utilized, and there is a lack of continuous and controllable means to balance time cost and accuracy.

[0005] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a multi-task flow scheduling method and system based on SLO constraints.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A multi-task flow scheduling method based on SLO constraints includes the following steps: (1) Read the configured workflow template from the configuration file and read the performance results of different models for tasks in different workflows from the performance test results; (2) Read the current workflow queue, filter the list of available candidate models for each task in each workflow according to the accuracy threshold, and when the list of available candidate models is empty, roll back the available models of the task to all available models. (3) Unified scheduling of all workflows: In an event-driven manner, ready tasks are retrieved one by one according to their ready time. For each ready task, the model expected to complete the task earliest is selected from its candidate model list for execution. The latest busy time of the model and the ready time of subsequent tasks in the workflow are updated until all tasks are scheduled and the basic task execution plan and its completion time are obtained. (4) Upgrade the accuracy within budget: Set a completion time budget, and take the completion time after replacement as a constraint that does not exceed the completion time budget. In order of the ratio of accuracy gain to time increment from high to low, replace the tasks in the basic task execution plan one by one with the model with higher accuracy. After each replacement, verify the feasibility. Accept the replacement that meets the constraint and improves the overall accuracy until there is no acceptable replacement. The upgraded task execution plan is obtained and issued for execution.

[0008] A multi-task flow scheduling system based on SLO constraints includes: The data acquisition module is used to acquire the configured workflow templates and their task dependencies and service level agreement constraints, the performance profiles of each model for tasks in different workflows, and the current workflow queue to be executed and its execution status. The candidate model filtering module is used to filter the list of available candidate models for each task based on an accuracy threshold, and to revert the available models for the task to all available models when the list of candidate models is empty. The unified scheduling module is used to retrieve ready tasks one by one according to their ready time in an event-driven manner. For each ready task, it selects the model that is expected to complete the task earliest from its candidate model list and executes it. It updates the latest busy time of the model and the ready time of subsequent tasks in the workflow until all tasks are scheduled. It then outputs the basic task execution plan and its completion time. The budget-inclusive upgrade module is used to replace each task in the basic task execution plan with a higher-accuracy model in descending order of the ratio of accuracy gain to time increment, within the constraint of not exceeding the completion time budget. It verifies whether the completion time constraint is met and the overall accuracy is improved by replaying all the replaced tasks. Replacements that meet the conditions are accepted and the upgraded task execution plan is output. The execution module is used to distribute the upgraded task execution plan to the heterogeneous model service cluster for execution.

[0009] The multi-task flow scheduling method and system based on SLO constraints provided by this invention can achieve the following beneficial effects: First, this invention manages workflow templates uniformly through configuration files and constructs a model performance profile library based on performance test results. Quantile statistics characterize the distribution characteristics of model latency, providing accurate data support for global scheduling. Furthermore, it filters the candidate model list based on an accuracy threshold and sets an empty set fallback mechanism. This ensures both the lower limit of the scheduling result's accuracy and avoids scheduling interruptions due to a lack of available models caused by strict filtering, effectively improving the robustness of scheduling.

[0010] Secondly, this invention uses an event-driven heap to uniformly schedule ready tasks across all workflows. For each ready task, it selects the model expected to complete the task earliest from its candidate model list. This coordinates the queuing competition of shared model services from a global perspective across multiple workflows. At the same time, it explicitly characterizes the serial constraints of the task chain by advancing the ready time of dependencies within the workflow. This achieves a globally feasible basic execution scheme with low computational complexity, fundamentally avoiding delay defaults caused by local optimal choices and peak queuing.

[0011] Furthermore, this invention uses completion time as an explicit budget and the ratio of accuracy gain to time increment as a selection key. It replaces tasks in the basic execution plan one by one with a more accurate model, which is then verified through replay before acceptance, directly optimizing the accuracy gained per unit time. This mechanism achieves accuracy improvement with zero time cost even when the budget equals the completion time of the basic plan. Simultaneously, the budget, as a continuously adjustable knob, provides a continuously controllable trade-off between time cost and accuracy, overcoming the shortcomings of traditional methods such as discrete adjustment of accuracy thresholds and congestion failure due to overly narrow candidate pools at high thresholds. In addition, serial chain length pre-screening and cost-effectiveness-priority ranking significantly reduce the number of candidates requiring replay verification, ensuring good real-time performance of the overall upgrade process.

[0012] This invention also generates a time-accuracy trade-off curve composed of non-dominated solutions by adjusting the value of the completion time budget. Each budget level on this curve produces an effective solution. The scheduler can select the optimal execution solution that satisfies the deadline and accuracy lower limit based on the workflow-level SLO constraints. This achieves the goal of providing the solution with the highest accuracy within any deadline constraint, overcoming the shortcomings of existing methods that can only choose between a fixed optimal accuracy and a fixed minimum delay, or that threshold scanning produces plateau segments and failure points in some intervals.

[0013] Furthermore, this invention ensures the reliability of accuracy quantification as the data source for upgrade decisions through mechanisms such as independent deployment of the referee model, automatic execution of code tasks for verification as the main scoring method, manual annotation and calibration verification and median of repeated scores, randomization of presentation order, and style-independent prompts. Through SLO failure degradation processing, the system can still provide a clear output strategy (fastest feasible solution plus default flag) in boundary scenarios where the deadline is stricter than any feasible solution, rather than suspending without a solution. Through joint search of accuracy threshold and completion time budget, feasible and optimal parameter pairs can be located under any combination of SLO constraints, enabling the solution to have complete engineering implementation capability.

[0014] Experimental results show that, compared with the scheme that binds all tasks to a single model, this invention reduces completion time by approximately 97% while achieving higher accuracy; compared with the pure speed-based greedy scheme, it only increases completion time by about 8.7% while improving accuracy by about 14 percentage points; compared with the single-question maximum accuracy greedy scheme, it reduces completion time by about 61% while only reducing accuracy by 0.3 percentage points. This invention significantly outperforms baseline schemes such as random assignment and minimum load priority in both completion time and accuracy, and can provide any feasible solution with a deadline between two threshold levels, fully verifying the significant superiority of this invention in balancing inference service quality constraints and the overall efficiency of heterogeneous model clusters.

[0015] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0016] Figure 1 This is a diagram of the scheduling system architecture according to an embodiment of the present invention.

[0017] Figure 2 This is a flowchart of a multi-task flow scheduling method based on SLO constraints according to an embodiment of the present invention.

[0018] Figure 3 This is a detailed flowchart of the event-driven earliest completion greedy scheduling in this embodiment of the invention.

[0019] Figure 4 This is a detailed flowchart of the on-budget accuracy upgrade in this embodiment of the invention.

[0020] Figure 5 This is a schematic diagram of the time-accuracy trade-off curve for adjusting the completion time budget B in an embodiment of the present invention. Detailed Implementation

[0021] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope and application of the present invention.

[0022] This invention aims to overcome the technical shortcomings of existing scheduling methods, such as single-task local optima, fixed model binding, lack of a global perspective, resulting in low utilization of heterogeneous models, latency spikes and SLO (Solution Time Limit) defaults caused by peak queuing, and the lack of a continuous trade-off mechanism between time cost and accuracy. It provides a multi-task flow scheduling method and system based on SLO constraints. The basic idea is as follows: using pre-calibrated model performance profiles as data foundation, accuracy thresholds as filtering knobs, and an event-driven heap to uniformly schedule ready tasks across all workflows. For each ready task, the earliest expected completion model is selected from the candidate model list, resulting in a basic execution plan that balances speed and accuracy minimums. Then, using completion time as an explicit budget, tasks are replaced one by one with more accurate models according to the ratio of accuracy gain to time increment, and accepted after replay verification. Within the budget, the accuracy margin between fast and slow models is systematically extracted, thus providing a continuously adjustable trade-off between time cost and accuracy.

[0023] See Figures 1 to 5 This invention provides a multi-task flow scheduling method based on SLO constraints, comprising the following steps: Step (1) Read the configured workflow template from the configuration file and read the performance results of different models for tasks in different workflows from the performance test results.

[0024] In some embodiments, step (1) includes: reading a workflow template from a configuration file, the workflow template containing a set of tasks, dependencies between tasks, and workflow-level service level protocol constraints; reading the accuracy and latency statistics of each model-task combination from the performance test results, the latency statistics being obtained by repeatedly sampling each model-task combination and statistically analyzing its latency in multiple quantile values, and all performance results constituting a model performance profile library.

[0025] In some embodiments, in step (1), the performance result read includes accuracy, which is obtained by the referee model scoring the model's generated results for the task: for language tasks, continuous scoring is performed based on the semantic fit between the answer and the reference answer; for code tasks, continuous scoring is performed based on the correctness of the code's functional implementation; and the scoring result is used as the accuracy value of the model on the task.

[0026] In some embodiments, the referee model is a general language model with capabilities superior to all candidate models, deployed independently of the scheduling system, and not included in the set of candidate models being evaluated, in order to avoid any conflict of interest between the evaluator and the evaluated.

[0027] In some embodiments, the scoring method of the referee model includes: for code-related tasks, priority is given to automatic execution of objective verification—the code generated by the model is injected into a preset unit test framework, and the proportion of passed test cases is used as the functional correctness score; the functional correctness is scored by the referee model only for code that cannot be automatically executed; for language-related tasks, the referee model outputs a continuous score of 0 to 1 according to the degree of semantic fit between the answer and the reference answer based on preset scoring criteria.

[0028] In some embodiments, the accuracy assurance mechanism of the referee model includes: First, calibration verification—constructing a calibration subset annotated by human experts, calculating the rank correlation coefficient between the referee model score and the human annotation, and only investing in large-scale scoring when the correlation coefficient reaches a preset level (e.g., not less than 0.85); otherwise, iteratively modifying the scoring criteria or replacing the referee model; Second, consistency of multiple scoring—repeatedly scoring the same generated result multiple times and taking the median score to suppress the variance of a single score; Third, suppression of systematic bias—randomizing the presentation order of candidate answers to eliminate positional bias, and using scoring prompts that are explicitly declared to be length-independent and style-independent to weaken the referee model's preference for lengthy outputs or specific expression styles; Fourth, robustness of scheduling caliber—scheduling decisions only depend on the relative difference in accuracy of different models on the same task, the impact of systematic bias of the referee model on the upgrade ranking on the same task partially cancels each other out, and the time delay profile uses conservative quantiles to further reserve a safety margin.

[0029] Step (2) Read the current workflow queue, filter the list of available candidate models for each task in each workflow according to the accuracy threshold, and when the list of candidate models is empty, roll back the available models of the task to all available models.

[0030] In some embodiments, in step (2), the selection method of the candidate model list is as follows: determine whether the accuracy of all available models on the task reaches the accuracy threshold, and include the models that reach the threshold into the candidate model list of the task; when no model has an accuracy that reaches the accuracy threshold, the candidate model list of the task is reverted to all available models.

[0031] Step (3) Perform unified scheduling for all workflows: In an event-driven manner, retrieve ready tasks one by one according to their ready time, select the model expected to complete the task earliest from its candidate model list for each ready task, and update the latest busy time of the model and the ready time of subsequent tasks in the workflow until all tasks are scheduled, and obtain the basic task execution plan and its completion time.

[0032] In some embodiments, in step (3), the criteria for selecting a model for each ready task are as follows: obtain the latest busy time of each model in the candidate model list of the task and the ready time of the task; for each candidate model, take the larger value between its latest busy time and the task ready time as the start time of the model to start processing the task, and then add the start time to the execution time of the task on the model to obtain the expected completion time of the task on the model; select the model with the smallest expected completion time as the execution model of the task; after the task is assigned, update the latest busy time of the execution model to the expected completion time.

[0033] like Figure 3 As shown, in some embodiments, in step (3), an event-driven heap is used to implement unified scheduling: the first task of all workflows is pushed into the min-heap according to the ready time; the following process is executed in a loop: the ready task with the earliest ready time is popped from the min-heap, and a model is selected and assigned to the task according to the criterion of selecting the model with the smallest expected completion time for each ready task; after the task is assigned, the ready time of the next task in the workflow is updated to the expected completion time of the task, and the next task is pushed into the min-heap; when the min-heap is empty, the scheduling ends, the completion time of all workflows is taken as the maximum value of the latest busy time of all models, and the basic task execution scheme is output.

[0034] Step (4) Upgrade within budget accuracy: such as Figure 4As shown, a completion time budget is set, and the completion time after replacement does not exceed the completion time budget as a constraint. In order of accuracy gain to time increment from high to low, the tasks in the basic task execution plan are replaced one by one with a model with higher accuracy. After each replacement, the feasibility is verified. Replacements that meet the constraints and improve the overall accuracy are accepted until there are no acceptable replacements. The upgraded task execution plan is then obtained and issued for execution.

[0035] In some embodiments, step (4) of the on-budget accuracy upgrade includes: for each assigned task and its current model in the basic task execution scheme, traversing all available models, constructing a candidate replacement set, wherein the candidate replacement satisfies that the accuracy of the task after replacement is strictly higher than the accuracy of the current model on the task; calculating the cost-effectiveness of each candidate replacement, wherein the cost-effectiveness is defined as the ratio of the accuracy gain after replacement to the time increment, and arranging all candidate replacements in descending order of the ratio; trying candidate replacements one by one in the order of arrangement; after executing the replacement, resimulating the execution process of all tasks in the event-driven manner in step (3) to obtain a new completion time and a new overall accuracy; determining whether the new completion time does not exceed the completion time budget and whether the new overall accuracy is higher than the overall accuracy before replacement, and accepting the replacement and updating the current execution scheme and the corresponding completion time and overall accuracy when both conditions are met, otherwise canceling the replacement and continuing to try the next candidate replacement in the order of arrangement; when all candidate replacements have been tried in a certain round of traversal and no replacement has been accepted, ending the upgrade process and outputting the upgraded task execution scheme.

[0036] In some embodiments, in step (4), when constructing the candidate replacement set, for each replacement to be evaluated, the serial chain length of the workflow in which the task is located after the replacement is first calculated. The serial chain length is the sum of the execution time of all tasks in the workflow on the allocated model. The replacement is included in the candidate replacement set only if the serial chain length does not exceed the completion time budget; otherwise, it is directly excluded.

[0037] In some embodiments, the method further includes, after step (4), selecting multiple different candidate budget values ​​within the range of the completion time budget, and independently executing step (4) to obtain the actual completion time and overall accuracy corresponding to each candidate budget value; filtering out non-dominated solutions from all the solutions corresponding to the candidate budget values, wherein the non-dominated solution is a solution for which there is no other candidate solution whose actual completion time is not greater than that solution and whose overall accuracy is not lower than that solution, and at least one of these conditions is strictly met; sorting and connecting the non-dominated solutions according to their actual completion time to form a time-accuracy trade-off curve; and selecting the corresponding task execution plan that simultaneously satisfies both constraints from the trade-off curve and issuing it for execution, based on the deadline and accuracy lower limit in the workflow-level service level agreement constraints.

[0038] In some embodiments, the method further includes an SLO failure handling step: when the completion time of the basic task execution plan exceeds the deadline in the service level agreement constraint, or the completion time of the upgraded plan under the completion time budget constraint still exceeds the deadline, a degraded scheduling is automatically triggered—first, the accuracy threshold is lowered and step (3) is re-executed to obtain a speed-priority basic plan, striving to meet the deadline with minimal accuracy loss; if a plan that meets the deadline still cannot be obtained, the fastest feasible plan in the candidate model space is output and the service level agreement is marked as defaulted, and / or new workflow submissions are rejected according to a preset strategy, and / or differentiated processing is performed according to workflow priority, prioritizing the guarantee that high-priority workflows meet the deadline. Thus, the scheduling system still has a clear output strategy in boundary scenarios where the deadline is stricter than any feasible plan, rather than suspending without a solution.

[0039] In some embodiments, the accuracy threshold and completion time budget are jointly adjusted as a parameter combination: the accuracy threshold determines the size of the candidate model set of the basic scheme, i.e., the starting point of the quality lower limit and the scheduling search space; the completion time budget determines the upper limit of the upgrade time and the upper limit of the achievable accuracy. Multiple candidate values ​​are taken for the accuracy threshold, and steps (2) to (4) are executed respectively to generate multiple trade-off curves. The combination that simultaneously satisfies the SLO constraint and has the smallest accuracy threshold is selected as the final parameter pair to avoid the narrowing of the candidate set and the congestion failure caused by the concentrated competition for a few fast models when the accuracy threshold is too high; and to avoid the insufficient accuracy of the basic scheme and the consumption of the budget by a large number of low-gain upgrades when the accuracy threshold is too low. The joint search of the two parameters ensures that the scheduling space is controllable and avoids the one-sided failure of adjusting any parameter alone.

[0040] See again Figure 1 This invention also provides a multi-task flow scheduling system based on SLO constraints, which includes a data acquisition module, a candidate model screening module, a unified scheduling module, an on-budget upgrade module, and an execution module.

[0041] The data acquisition module is used to acquire the configured workflow templates and their task dependencies and service level agreement constraints, the performance profiles of each model for tasks in different workflows, and the current workflow queue to be executed and its execution status.

[0042] The candidate model filtering module is used to filter the list of available candidate models for each task based on an accuracy threshold, and when the list of available candidate models is empty, it reverts the available models for that task to all available models.

[0043] The unified scheduling module is used to uniformly schedule all workflows, such as Figure 3 As shown, this module retrieves ready tasks one by one according to their ready time in an event-driven manner. For each ready task, it selects the model expected to complete the task earliest from its candidate model list and executes it. It updates the latest busy time of the model and the ready time of subsequent tasks in the workflow until all tasks are scheduled. Finally, it outputs the basic task execution plan and its completion time.

[0044] The budget-inclusive upgrade module, constrained by the completion time budget, attempts to replace each task in the basic task execution plan with a more accurate model, ranked from highest to lowest based on the ratio of accuracy gain to time increment. It verifies whether the completion time constraint is met and overall accuracy is improved by replaying all replaced tasks. Replacements that meet the conditions are accepted, and the upgraded task execution plan is output. For example... Figure 4 As shown, this module continuously constructs candidate replacements, sorts cost-effectiveness, performs replay verification and acceptance judgment through iteration, until no acceptable replacement is found in a certain round of traversal.

[0045] The execution module is used to distribute the upgraded task execution plan to the heterogeneous model service cluster for execution.

[0046] In some embodiments, the on-budget upgrade module includes: a candidate replacement construction unit, used to traverse each assigned task and its current model in the basic task execution plan, construct the model whose accuracy is strictly improved after replacement as a candidate replacement, and pre-screen it with the requirement that the serial chain length of the workflow in which the task is located after replacement does not exceed the completion time budget; a sorting unit, used to calculate the cost-effectiveness of each candidate replacement, the cost-effectiveness being the ratio of accuracy gain to time increment, and sort the candidate replacements from high to low according to the ratio; a replay verification unit, used to execute the candidate replacements one by one according to the sort, and after each replacement, call the event-driven method of the unified scheduling module to re-simulate the execution process of all tasks to obtain new completion time and overall accuracy; and a judgment and iteration unit, used to accept the replacement and update the current execution plan when the new completion time does not exceed the completion time budget and the new overall accuracy is higher than the overall accuracy before replacement, otherwise cancel the replacement and continue to try the next candidate replacement until there is no acceptable replacement in a certain round of traversal, and output the upgraded task execution plan.

[0047] In some embodiments, the data acquisition module further includes: a configuration unit for providing a workflow template, the workflow template including a task set, dependencies between tasks, and workflow-level service level protocol constraints; a performance profile library unit for providing accuracy and latency statistics for each model-task combination, the latency statistics being obtained by repeatedly sampling each model-task combination and calculating multiple quantile values ​​of its latency; and a queue management unit for maintaining the current workflow queue to be executed and the execution status of each workflow.

[0048] This invention also provides a multi-task flow scheduling system based on SLO constraints, comprising the following modules: a first module, used to read all configured workflow templates from a configuration file and read the performance results of different models for tasks in different workflows from performance test results; a second module, used to read the current workflow queue, filter the list of selectable candidate models for each task in each workflow according to an accuracy threshold, and revert the selectable models for the task to all available models when the list of candidate models is empty; and a third module, used to uniformly schedule all workflows: retrieving them one by one according to their readiness time in an event-driven manner. For each ready task, the model expected to complete the task earliest is selected from its candidate model list for execution. The latest busy time of the model and the ready time of subsequent tasks in the workflow are updated until all tasks are scheduled, resulting in a basic task execution plan and its completion time. The fourth module is used for on-budget accuracy upgrades: with the completion time after replacement not exceeding the budgeted completion time, tasks in the basic task execution plan are replaced one by one with models with higher accuracy in descending order of the ratio of accuracy gain to time increment. After each replacement, the feasibility is verified by replay and then accepted, resulting in an upgraded task execution plan.

[0049] This invention uses model performance profiles as its data foundation, employing accuracy threshold filtering and an empty set backoff mechanism to ensure scheduling robustness and accuracy minimums. It utilizes an event-driven heap for unified scheduling of all workflows, selecting the most accurate model for each task based on the earliest completion criterion. This global perspective coordinates model queuing and competition, yielding a feasible basic solution that balances speed and accuracy. Furthermore, using completion time as an explicit budget, tasks are replaced with more accurate models based on the ratio of accuracy gain to time increment, and this replacement is validated and accepted. Within the budget, overall accuracy is systematically improved, achieving a continuous and controllable trade-off between time cost and accuracy. Finally, budget scanning generates a trade-off curve composed of non-dominated solutions, providing the most accurate execution plan under any deadline constraint. This comprehensive mechanism effectively overcomes the shortcomings of local optima, model binding, and discrete threshold adjustment, significantly improving the overall efficiency and scheduling flexibility of heterogeneous model clusters while ensuring service quality constraints.

[0050] The following further describes specific embodiments of the present invention, algorithm examples, and experimental verification.

[0051] A multi-task flow scheduling method based on SLO constraints, see [link / reference]. Figure 2The main process includes: reading all configured workflow templates from the configuration file and reading the performance results of different models for tasks in different workflows from the performance test results; reading the current workflow queue and, for each task in each workflow, filtering the list of available candidate models based on an accuracy threshold, with the list of candidate models being empty and reverting to all available models for the current task; uniformly scheduling all workflows, retrieving ready tasks one by one according to their ready time in an event-driven manner, and selecting the model expected to complete the task earliest from the candidate model list for each ready task to obtain the basic task execution plan and its completion time; subject to a constraint of not exceeding the completion time budget, replacing each task in the basic task execution plan with a more accurate model in descending order of the ratio of accuracy gain to time increment, and accepting the replacement after feasibility verification to obtain the upgraded task execution plan. Correspondingly, see [link to relevant documentation]. Figure 1 Furthermore, a multi-task flow scheduling system based on SLO constraints is provided. This system includes a data acquisition module, a candidate model screening module, a unified scheduling module, an in-budget upgrade module, and an execution module. These modules are used to implement the data acquisition, candidate screening, unified scheduling, in-budget upgrade, and scheme execution functions mentioned above. The modules work together to complete the entire scheduling process from workflow and performance data input to the issuance of the upgraded execution scheme. This overcomes the technical shortcomings of existing scheduling schemes, which only select locally optimal models for individual tasks, have fixed model bindings lacking a global perspective, resulting in low utilization of heterogeneous models, peak queuing causing latency spikes and SLO defaults, and lack a scheduling mechanism that can continuously balance time cost and accuracy.

[0052] Specifically, the multi-task flow scheduling method based on SLO constraints includes the following steps: (1) Read all configured workflow templates from the configuration file and read the performance results of different models for tasks in different workflows from the performance test results; (2) Read the current workflow queue, and for each task in each workflow, filter the list of available candidate models according to the accuracy threshold; when the candidate model list of a task is empty, roll back the candidate model list of the task to all available models of the current task. (3) Unified scheduling of all workflows: Retrieve ready tasks one by one according to their ready time in an event-driven manner, and select each ready task from its candidate model list. Select the model that is expected to complete the task earliest, that is, select the model with the earliest expected completion time that satisfies the following formula. : in, Representation Model The busiest time at the latest Indicates the task's readiness time. This indicates that the task is in the model The execution time is calculated; the latest busy time of the model is updated and the readiness time of subsequent tasks in the workflow is advanced until all tasks are scheduled, resulting in the basic task execution plan and its completion time. ; (4) Improved accuracy within budget: Set a budget for completion time The aforementioned completion time budget The maximum allowable completion time, i.e., the upper limit of the time from the start of scheduling to the completion of the entire workflow, is ensured that the replacement completion time does not exceed the budget. To constrain this process, tasks in the basic task execution plan are replaced one by one with models that have higher accuracy: each time, the replacement with the largest ratio of accuracy gain to time increment is selected from all candidate replacements and executed. After replacement, replay verification shows that the completion time does not exceed the budget. If the overall accuracy improves, the replacement is accepted; otherwise, it is rejected and a suboptimal replacement is attempted. This process is repeated until no acceptable replacement is found, resulting in an upgraded task execution plan that is then issued for execution.

[0053] Preferably, step (1) includes the following sub-steps: (1-1) Read the workflow template from the configuration file. The workflow template contains a set of tasks, dependencies between tasks, and workflow-level SLO constraints (including deadline and accuracy lower limit). (1-2) Read the accuracy and latency statistics of each model-task combination from the performance test results. The latency statistics are obtained by repeatedly sampling each model-task combination and calculating the quantiles (p50, p90, p99). All performance results constitute the model performance profile library. During scheduling, the corresponding quantile values ​​are selected according to the degree of conservatism to participate in the time estimation, where p90 and p99 are used to construct the worst-case budget.

[0054] Preferably, in step (2), the candidate model list for the i-th task is as follows: in, Indicates the first One task, Representation model, Representation Model In the Accuracy on each task This is the accuracy threshold; when it meets... When the model set is empty, the candidate model list is rolled back to all available models for the current task.

[0055] Preferably, in step (3), the criterion for selecting a model for each ready task is the earliest completion criterion: in, Representation Model The busiest time at the latest Represents workflow The Middle The ready time of each task. Indicates task In the model The execution time is calculated on the [database name]; once a task is ready, it is assigned to the model expected to complete the task earliest. and update .

[0056] Preferably, in step (3), an event-driven heap is used to implement unified scheduling: the first task of all workflows is pushed into the min-heap according to its ready time; the ready task with the earliest ready time is popped in a loop, and a model is selected and assigned according to the earliest completion criterion; after the task assignment is completed, the ready time of the next task in the workflow is updated to the expected completion time of the task and pushed into the min-heap; scheduling ends when the min-heap is empty, and the completion time of all workflows is... .

[0057] Preferably, in step (4), the candidate set for each replacement is constructed as follows and sorted from high to low according to the ratio of accuracy gain to time increment: (6-1) For each assigned task in the basic task execution plan and its current model Traverse all available models Construct candidate replacements ,in The accuracy of the task must be substantially improved after the replacement, that is... ; (6-2) The cost-effectiveness of each candidate replacement is defined as the ratio of accuracy gain to time increment: (6-3) according to Try candidate replacements one by one from high to low: After performing the replacement, replay all tasks in an event-driven manner to obtain the new completion time. With overall accuracy ,when and If the overall accuracy is higher than before the replacement, accept the replacement; otherwise, cancel the replacement and try the next candidate. (6-4) Repeat the above process until no acceptable replacement is found in a certain round of scanning, at which point the upgrade ends and the upgraded task execution plan is output.

[0058] Preferably, when constructing candidate replacements, the workflow serial chain length is used as a necessary condition for pre-screening: only if the serial chain length of the workflow to which the task belongs after replacement (the sum of the execution times of all tasks in the workflow on the assigned model) does not exceed the budget. Only when the replacement is deemed unfeasible will it be included in the candidate set, in order to avoid performing replay verification on obviously infeasible replacements and reduce verification overhead.

[0059] Preferably, it also includes adjusting the completion time budget. Generate a set of time-accuracy tradeoff solutions: for budget Take multiple candidate values ​​within its range, and perform step (4) for each candidate value to obtain the corresponding completion time. With overall accuracy This forms a time-accuracy tradeoff curve consisting of a series of non-dominated solutions. Based on the workflow-level SLO constraints, a task execution plan that satisfies the constraints is selected from the tradeoff curve and then issued for execution.

[0060] Preferably, the accuracy By referee model to model For the task The generated results are scored as follows: for language tasks, a continuous score of 0 to 1 is applied based on the semantic fit between the answer and the reference answer; for code tasks, a continuous score of 0 to 1 is applied based on the correctness of the code's functionality. The score result is the accuracy rate. Because the score uses continuous values ​​of 0 to 1 rather than discrete levels, the accuracy differences between different models on the same task can be finely characterized, giving the accuracy gain calculation for the within-budget upgrade continuous resolution capability.

[0061] Preferably, the referee model is deployed independently of the scheduling system and is not part of the candidate model set being evaluated. For code-related tasks, the accuracy is primarily scored based on the pass rate of automated unit tests; the referee model only scores the correctness of code that cannot be automatically executed. For language-related tasks, the referee model scores continuously from 0 to 1. Before calibration, the referee model is verified using a calibration subset annotated by human experts to ensure that the rank correlation coefficient between its score and the human annotation is not lower than a preset threshold; otherwise, the scoring criteria are iteratively modified. During calibration, the median score is taken for repeated scoring of the same generated result, and the presentation order is randomized, using length-independent scoring prompts to suppress systematic bias. Since the scheduling decision only depends on the relative difference in accuracy between models on the same task, the common biases of the referee model on the same task are partially offset in the cost-effectiveness ranking, further ensuring the reliability of the upgrade decision.

[0062] Figure 2The implementation flow of the scheduling method of the present invention is shown, including steps (1) to (4), wherein step (2) includes candidate model screening and empty set rollback, step (3) includes event-driven earliest completion greedy scheduling, and step (4) includes in-budget accuracy upgrade.

[0063] like Figure 2 As shown, this invention provides a multi-task flow scheduling method based on SLO constraints, comprising the following steps: (1) Read all configured workflow templates from the configuration file and read the performance results of different models for tasks in different workflows from the performance test results; Specifically, this step includes the following sub-steps: (1-1) Read the workflow template from the configuration file. The workflow template is a directed acyclic graph, where nodes are tasks and edges are dependencies between tasks; the template also contains workflow-level SLO constraints, including deadlines and accuracy lower bounds.

[0064] (1-2) Read the accuracy and latency statistics for each model-task combination from the performance test results. The performance test results are generated during the calibration phase: each model-task combination is repeatedly executed (including isolation tests and concurrent tests), and the p50, p90, and p99 quantiles of latency and accuracy are statistically analyzed to form a model performance profile library. During scheduling, the corresponding quantile values ​​are selected according to the degree of conservatism to participate in time estimation: p50 is used as the normal budget, and p90 and p99 are used to construct the worst-case budget.

[0065] The advantage of the above sub-steps (1-1) to (1-2) is that the scheduling decision is driven by the measured performance data, so that the scheduling result matches the real system behavior and can be transferred to any heterogeneous model cluster.

[0066] (2) Read the current workflow queue, and for each task in each workflow, filter the list of available candidate models based on the accuracy threshold; Specifically, this step includes the following sub-steps: (2-1) Read the current workflow queue to be executed, which contains the tasks submitted and execution status of each workflow; (2-2) For each task in each workflow, based on the accuracy threshold A list of candidate models is selected from all models. The candidate model list for each task is as follows: in, Indicates the first One task, Representation model, Representation Model In the Accuracy on each task This is the accuracy threshold. When it meets... When the model set is empty, the candidate model list is rolled back to all available models for the current task, i.e. This is the set of all available models for the current task, ensuring that scheduling is not interrupted due to strict filtering.

[0067] The advantages of the above sub-steps (2-1) to (2-2) are: on the one hand, filtering out models that do not meet the requirements by using an accuracy threshold ensures the lower limit of the accuracy of the scheduling results from the source; on the other hand, setting an empty set backoff mechanism ensures that any task can be performed under any circumstances. There are candidate models to choose from for all possible values, and the scheduling is robust.

[0068] (3) All workflows are scheduled in a unified manner to obtain the basic task execution plan and its completion time. .

[0069] Specifically, this step uses an event-driven heap to uniformly schedule ready tasks across all workflows. Each model service is strictly serial, single-slot (only one request can be processed at a time, and requests are queued in arrival order), and the current latest busy time for each model is recorded as follows: Tasks within each workflow are executed sequentially according to their dependencies, the first... The next task can only be executed after the previous task is completed. One task.

[0070] Figure 3 The detailed process of the event-driven earliest completion greedy scheduling in step (3) is shown, including initializing the heap, cyclically popping the earliest ready task, selecting the model according to the earliest completion criterion, updating the busy time of the model and the ready time of the subsequent task, until the heap is empty.

[0071] like Figure 3 As shown, this step includes the following sub-steps: (3-1) Initialization: For each model ,make For all workflows, set the readiness time of the first task to 0 and push it onto a min-heap. (3-2) Repeat the process until the min-heap is empty: (3-2-1) Pop the earliest ready task from the min-heap Skip if the task has already been assigned. (3-2-2) Obtain a candidate model list for this task. For each of these models Calculate the estimated completion time ; (3-2-3) Choice , will the task Assigned to model ; (3-2-4) Update If workflow There is a next task; set its ready time to... And push them into a min-heap; (3-3) Scheduling ends when the min-heap is empty. The completion time for the entire workflow is... Output the basic task execution plan.

[0072] Its pseudocode is as follows: Input: Candidate model list {C_i}, performance profiles acc(m,t) and t(q,m), and dependency chains of each workflow task. Output: Basic task - model allocation scheme (assign) and estimated completion time (W) 1. Initialize busy[m] = 0 (all models); cursor[w] = 0 (number of scheduled tasks in each workflow) 2. Push (0, w) into the min-heap (the first task of the entire workflow, ready time is 0). 3. While the min-heap is not empty: 4 arrive, w = Pop the top of the heap # The workflow with the earliest ready time 5. If cursor[w] has completed all tasks in this workflow: continue 6 idx = cursor[w]; cursor[w]+= 1; q = wfs[w][idx] 7 For each m ∈ C(q): fin[m] = max(busy[m], arrive) + t(q, m) 8 m* = argmin fin[m] # Earliest completion criterion 9 assign[w, idx] = m*; busy[m*]= fin[m*] 10. If w has a (idx+1)th task: push (fin[m*], w) into a min-heap. 11 end 12 W = max(busy[m]); returns the assign and W. The advantages of the above step (3) are: taking "expected to complete earliest" as the unified criterion, the most idle and fastest model is selected from the candidate models, and the load of each model is naturally balanced from the global perspective of multiple workflows, thus explicitly avoiding peak queuing; at the same time, the workflow dependency serial constraints are characterized by the advance of ready time, taking into account both the workflow chain length and model queuing time factors, so that the scheduling result matches the actual execution behavior under strict serial caliber, and the algorithm has low complexity and can be solved in real time.

[0073] (4) Improved accuracy within budget: Set a budget for completion time The aforementioned completion time budget The maximum allowed completion time is the upper limit from the start of scheduling to the completion of the entire workflow; the completion time after replacement shall not exceed [a certain value]. Under the constraints, the tasks in the basic task execution scheme are replaced one by one with more accurate models to obtain the upgraded task execution scheme.

[0074] Figure 4 The detailed process of upgrading the accuracy within budget in step (4) is shown, including inputting the basic scheme and budget, constructing candidate replacements (including pre-screening by serial chain length), calculating the cost-effectiveness of each candidate and sorting them, trying replacements in sequence, verifying by replay, accepting if the budget is met and the accuracy is improved, otherwise withdrawing, until there are no candidates acceptable.

[0075] like Figure 4 As shown, this step includes the following sub-steps: (4-1) Input is the basic task execution plan Its completion time With overall accuracy and completion time budget ( Typical or (Add allowable relaxation amount). (4-2) Constructing a candidate replacement set: For Each assigned task and its current model Traverse all available models If the accuracy rate is significantly improved after replacement ( And after the replacement, the serial chain length of the workflow containing this task does not exceed [a certain value]. If the result is not specified, it will be included in the candidate set. The serial chain length pre-screening utilizes the necessary condition that "the completion time is not less than the serial chain length of any workflow" to eliminate a large number of obviously infeasible candidates without performing replay. (4-3) Calculate the cost-effectiveness of each candidate replacement: That is, the accuracy gain gained per unit time increment, which will be used to select candidates by... Sort from highest to lowest; (4-4) Try candidate replacements one by one in order: After the replacement is performed, replay all tasks in the event-driven manner of step (3) to obtain the new completion time. With overall accuracy .when and Accept the replacement and update. , and Otherwise, cancel the replacement and try the next candidate. (4-5) Repeat steps (4-2) to (4-4). The upgrade ends when no acceptable replacement is found in a certain round of scanning, and the upgraded task execution plan is output.

[0076] Its pseudocode is as follows: Input: Basic solution assignment (completion time W, overall accuracy A), budget B, performance profile Output: Upgraded task-model allocation scheme 1 while true: 2 cands = [] 3 for (w, i, m_cur) in assign: 4 for m in All available models: 5 Δacc = acc(m, q[w][i]) - acc(m_cur, q[w][i]) 6 if Δacc <= 0: continue # Only consider the more accurate model 7 if chain length(w, after replacement)>B: continue # Pre-screening of necessary conditions for serial chain length 8 Δt = t(q[w][i], m) - t(q[w][i], m_cur) 9 cands.append( (-Δacc / Δt, w, i, m_cur, m) ) 10. If cands is empty: break 11. Sort cands in descending order by ROI 12 applied = false 13 for (_, w, i, m_cur, m) in cands[:K]: # Evaluate the top K high ROI candidates one by one 14 assign[w, i] = m 15 W', A' = Replay(assign) # Event-driven replay verification 16 if W'<= B and A'>A: W, A = W', A'; applied = true; break 17 assign[w, i] = m_cur # Cancel 18 if not applied: break 19 return assign The advantage of step (4) above is that: Firstly, the selection key is "the ratio of accuracy gain to time increment", which directly optimizes the accuracy gained per unit time. Every second of time increment is used for the most cost-effective replacement (i.e., the replacement with the largest ratio of accuracy gain to time increment). In contrast, the existing accuracy threshold method can only indirectly control the quality by filtering scalars in a "one-size-fits-all" manner. Secondly, the replay verification mechanism ensures that each accepted replacement is within budget and the overall accuracy is strictly improved, and the upgrade process is monotonous and does not deteriorate: even if the budget is equal to the completion time of the basic solution (zero relaxation), it can still automatically discover and utilize the gap that "the replacement model does not increase the completion time but is more accurate", and achieve an accuracy improvement with zero time cost. Third, the budget is a continuous knob, and any value will produce an effective solution, avoiding the problems of discrete levels, plateaus and high threshold congestion failures in threshold scanning. Fourth, serial chain length pre-screening and The sorting prior significantly reduces the number of candidates that need to be replayed for verification, enabling the entire upgrade process to be completed in polynomial time and achieving real-time performance.

[0077] This step also includes adjusting the completion time budget. Generate a set of time-accuracy tradeoff schemes (such as...) Figure 5 As shown): for the budget Take multiple candidate values ​​within its range (e.g., from the basic completion time) For each candidate value (up to the maximum allowed deadline), perform step (4) to obtain the corresponding completion time. With overall accuracy ; including the non-dominant points (i.e., those where no other budget exists); Simultaneously satisfy and (And at least one of them must be strictly true) connects to form a time-accuracy trade-off curve. During scheduling, based on the workflow-level SLO constraints (deadline and lower bound of accuracy), the budget that meets the constraints and its corresponding task execution plan are selected from the trade-off curve and issued for execution; given a deadline, the plan with the highest accuracy within the constraint can be quickly located through a binary search of the budget. Figure 5 A schematic diagram of the time-accuracy tradeoff curve for adjusting the completion time budget B is shown. The horizontal axis represents completion time, and the vertical axis represents overall accuracy. The diagram marks the base solution point (B=W, zero budget), several upgrade points, and the threshold scan curve as a comparison. In the diagram, red represents the upgrade frontier within the budget of this invention (budget B continuously widens from W), and gray represents the pure threshold θ scan curve. The upgrade frontier has superior accuracy across the board at the same or shorter completion time, and produces effective solutions at each budget level (monotonically non-degrading). The pure threshold scan fails in the high θ range due to backoff contention (below 63.6s / 95% when θ≥0.98, strictly dominated by the upgrade frontier).

[0078] In a specific embodiment, the complete process of joint adjustment and failure handling is as follows: Assume the SLO constraint of the workflow queue is a deadline D = 47.6 seconds and an accuracy lower limit of 99.0%. First, take the accuracy threshold θ = 0.85 and execute steps (2) and (3) to obtain the basic solution with a completion time W = 31.3 seconds; then, budget scan within the range B ∈ [31.3, 47.6] seconds to obtain the trade-off curve, where B = 47.6 seconds corresponds to 99.1% accuracy and 47.6 seconds completion, which simultaneously satisfies the deadline and the accuracy lower limit, and the output is selected. If the user tightens the deadline to D = 35 seconds, B = 35 seconds on the trade-off curve corresponds to approximately 98.5% accuracy, which cannot meet the 99.0% lower limit: at this time, the system determines that the two constraints are mutually exclusive, returns two coexisting alternative solutions (35 seconds / 98.5% and 47.6 seconds / 99.1%) to the scheduling platform and explains the breach dimension, which is then decided by the upper layer according to business priority. If the deadline is further tightened to D=28 seconds, which is lower than the fastest completion time under any threshold (28.8 seconds for pure speed greed), then a downgrade process is triggered: the fastest feasible solution marked as SLO default is output, and new workflow submissions are rejected or queued according to the policy.

[0079] In one specific embodiment, the joint adjustment process of θ and B is as follows: Steps (2) and (3) and budget scanning are performed on θ∈{0.70,0.75,0.80,0.85,0.90} respectively to obtain five trade-off curves; under a given deadline D and accuracy lower limit A, the parameter pair that satisfies the constraints and minimizes θ is selected. In actual tests, the combination of θ=0.85 and B∈[W,D] gives feasible solutions for all deadline levels, while θ=0.98 causes congestion in the slow model due to the narrowing of the candidate set, and the completion time deteriorates to more than 63 seconds, verifying that the joint strategy of using the minimum feasible θ in conjunction with budget upgrade is better than the strategy of raising the threshold alone.

[0080] In one specific embodiment, the deployment and verification process of the referee model is as follows: The referee model adopts an independent general-purpose large model service with capabilities higher than all candidate models (parameter scale from 0.5B to 4B); the score of code-type tasks is directly given by the pass rate of automatic unit tests, and the score of language-type tasks is output by the referee model according to the scoring criteria, with a continuous score of 0 to 1. The referee model first verifies consistency on a manually labeled subset that is distributed in the same way as the calibration set, and only after the rank correlation coefficient reaches the preset threshold is it put into full calibration; when scoring, each question is repeated 3 times and the median is taken, and the order is randomized, and the scoring prompts explicitly require that the length of the answer and the expression style be ignored. The obtained accuracy is written into the model performance profile library, which serves as a unified data source for the threshold screening in step (2) and the cost-effectiveness calculation in step (4).

[0081] In one specific embodiment, the method of the present invention is applied to an inference cluster comprising 10 workflows and sharing 10 heterogeneous models (parameter sizes ranging from 0.5B to 4B). Each workflow contains 100 tasks (50 language tasks and 50 code tasks), with task dependencies forming a sequential chain. During the calibration phase, each model-task combination is repeatedly sampled to obtain accuracy and latency profiles; the accuracy of both language and code tasks is obtained by a referee model scoring them continuously from 0 to 1.

[0082] Take accuracy threshold By executing steps (2) and (3), the basic task execution plan is obtained, and its completion time is determined. Seconds, overall accuracy rate 98.3%. Then, the completion time budget was calculated. Taking multiple candidate values ​​and performing step (4), some of the resulting trade-offs are shown in Table 1 below: Table 1 As can be seen from the table: zero budget ( Even when the time limit is reached, three replacements can still be completed with zero time cost, and the overall accuracy rate is improved from 98.29% to 98.31% for free. When the budget is relaxed to 41.3 seconds, 109 tasks are upgraded to a more accurate model, with an overall accuracy rate of 98.94% and a completion time of 40.9 seconds, which is within the budget.

[0083] As a comparison, the accuracy threshold Candidate values ​​are selected within the range of 0.70 to 1.00, and steps (2) and (3) are executed, but step (4) is not executed, to obtain the pure threshold scan trade-off point: 31.3 seconds / 98.3% 38.3 seconds / 98.9% 39.2 seconds / 98.5% Time: 63.4 seconds / 94.4%. Pure threshold scanning exhibits two types of failures: firstly, The same solution was produced within the range of 0.70 to 0.85 (plateau segment), and threshold adjustment had no effect; secondly, When the threshold is above 0.98, many tasks fail to reach the threshold and are rolled back, leading to a concentration of competition for the fastest model. This deteriorates the completion time to over 63 seconds and the accuracy drops below 95%. In contrast, the budget upgrade method of this invention produces monotonically non-degrading effective solutions for all budget values. Points on its tradeoff curve strictly dominate most points on the pure threshold scan curve: for example, the upgrade solution with a budget of 41.3 seconds (40.9 seconds / 98.9%) is significantly better than the pure threshold solution. The previous solution (39.2 seconds / 98.5% or higher accuracy levels were all unavailable) and the upgraded solution (47.6 seconds / 99.1%) with a budget of 47.6 seconds are significantly better than the pure threshold solution, with a completion time 16 seconds less and an accuracy rate 3.5 percentage points higher. The plan (63.4 seconds / 94.4%).

[0084] In multiple rounds of field testing under the same model server environment, the method of this invention was compared with several baseline solutions. All solutions used the same question bank (10 workflows, 100 tasks per workflow), the same model performance profile, and the same event-driven replay approach. The results are shown in the table below: Table 2 Actual measurements show that: First, compared with baseline 1 (fixed binding comprehensive most accurate model), this invention achieves higher accuracy (99.1% vs. 98.5%) while reducing completion time by approximately 97%, eliminating the computing power bottleneck caused by serial execution of a single model; Secondly, compared with baseline 2 (fixed binding of the fastest model), baseline 3 (random allocation), and baseline 6 (minimum load priority), this invention improves the accuracy by approximately 20, 9, and 9 percentage points respectively, under the premise of shorter or equivalent completion time. This shows that pursuing model speed alone or pursuing load balancing alone cannot guarantee accuracy. Third, compared to baseline 4 (pure speed-greedy algorithm), the completion time of this invention (zero-budget upgrade) only increases by about 2.5 seconds (31.3 seconds vs. 28.8 seconds), while the accuracy improves from 84.2% to 98.3%. That is, an accuracy improvement of about 14 percentage points is achieved at the cost of about 8.7% of time, and this cost is entirely within the budget. It is controllable within the adjustment range; Fourth, compared with baseline 5 (the most accurate greedy algorithm for a single problem), this invention (budget upgrade, The completion time was reduced by approximately 61% (47.6 seconds to 123.7 seconds), with an accuracy reduction of only 0.3 percentage points while still meeting SLO requirements; Fifth, compared with baseline 7 (optimal setting for pure threshold scanning), the present invention has higher accuracy at similar completion times and can provide any solution with a cutoff time between the two thresholds. For example, with a cutoff time of 35 seconds, pure threshold scanning has no corresponding setting and can only revert to 31.3 seconds / 98.3%, while the present invention can obtain a solution of about 98.5% with a budget of 35 seconds, indicating that the continuous controllability of the budget knob is significantly better than that of the threshold knob.

[0085] The meanings of the symbols used in this instruction manual are uniformly agreed upon as follows: Indicates task In the model Execution time; Representation Model For the task The accuracy rate; Indicates the accuracy threshold; Indicates task A list of candidate models; Represents workflow The Middle The ready time of each task; Representation Model The busiest time of the night; Indicates the completion time of the entire workflow; Indicate the estimated completion time; Represents workflow The Middle A model in which tasks are assigned; This represents the ratio of the accuracy gain of the candidate replacement to the time increment.

[0086] In summary, this invention provides a multi-task flow scheduling method and system based on SLO constraints. It uses pre-calibrated model performance profiles as data foundation, accuracy thresholds as filtering knobs, and an event-driven heap to uniformly schedule ready tasks across all workflows. For each ready task, the earliest expected completion model is selected from the candidate model list to obtain a basic execution plan that balances speed and accuracy minimums. Then, using completion time as an explicit budget, tasks are replaced one by one with more accurate models in descending order of accuracy gain to time increment, and accepted after replay verification. Within the budget, the accuracy margin between fast and slow models is systematically extracted, thus providing a continuously adjustable trade-off between time cost and accuracy. This method, through unified scheduling from a global perspective and budget-driven accuracy upgrades, effectively overcomes the shortcomings of existing scheduling methods such as single-task local optima, fixed model binding, and discrete threshold adjustment, significantly improving the overall efficiency and scheduling flexibility of heterogeneous model clusters.

[0087] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following significant technical advantages: (1) This invention employs steps (1) and (2), manages workflow templates uniformly through configuration files, constructs a model performance profile library based on performance test results, characterizes the distribution characteristics of model latency using quantile statistics, and bases the analysis on accuracy thresholds. A candidate model list is selected for each task and an empty set fallback mechanism is set to provide data support for global scheduling. This ensures both the lower limit of the accuracy of the scheduling results and avoids scheduling interruption due to the lack of available models caused by strict filtering. (2) This invention adopts step (3) to uniformly schedule the ready tasks of all workflows using an event-driven heap. For each ready task, the model expected to complete the task earliest is selected from its candidate model list. The queuing competition of shared model services is coordinated from the global perspective of multiple workflows. At the same time, the serial constraints of the task chain are explicitly characterized by the ready time of the workflow. A globally feasible basic execution scheme is obtained with lower computational complexity, avoiding delay defaults caused by local optima and peak queuing. (3) This invention adopts step (4), with completion time as the explicit budget and the ratio of accuracy gain to time increment as the selection key, and replaces the tasks in the basic execution plan one by one with a more accurate model and accepts it after replay verification, directly optimizing the goal of "accuracy gained per unit time": it can achieve free accuracy improvement with zero time cost when the budget is equal to the completion time of the basic plan, and can continuously and controllably balance time cost and accuracy through the budget knob, overcoming the defects of discrete adjustment of accuracy threshold and congestion failure due to narrow candidates under high threshold; (4) This invention uses a budget scanning to generate a time-accuracy trade-off curve, which enables the scheduler to provide the execution plan with the highest accuracy under any deadline constraint: the budget value is continuous and each budget level produces an effective plan, overcoming the shortcomings of existing methods that can only choose between "fixed optimal accuracy" and "fixed minimum delay" or threshold scanning produces plateau segments and failure points in some intervals.

[0088] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0089] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0090] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0091] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc or CD-ROM; magnetic surface memory can be disk storage or magnetic tape storage. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0092] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0093] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0094] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0095] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0097] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0098] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0099] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0100] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A multi-task flow scheduling method based on SLO constraints, characterized in that, Includes the following steps: (1) Read the configured workflow template from the configuration file and read the performance results of different models for tasks in different workflows from the performance test results; (2) Read the current workflow queue, filter the list of available candidate models for each task in each workflow according to the accuracy threshold, and when the list of available candidate models is empty, roll back the available models of the task to all available models. (3) Unified scheduling of all workflows: In an event-driven manner, ready tasks are retrieved one by one according to their ready time. For each ready task, the model expected to complete the task earliest is selected from its candidate model list for execution. The latest busy time of the model and the ready time of subsequent tasks in the workflow are updated until all tasks are scheduled and the basic task execution plan and its completion time are obtained. (4) Upgrade the accuracy within budget: Set a completion time budget, and take the completion time after replacement as a constraint that does not exceed the completion time budget. In order of the ratio of accuracy gain to time increment from high to low, replace the tasks in the basic task execution plan one by one with the model with higher accuracy. After each replacement, verify the feasibility. Accept the replacement that meets the constraint and improves the overall accuracy until there is no acceptable replacement. The upgraded task execution plan is obtained and issued for execution.

2. The method according to claim 1, characterized in that, Step (1) includes: Read the workflow template from the configuration file. The workflow template contains a set of tasks, dependencies between tasks, and workflow-level service level agreement constraints. The accuracy and latency statistics of each model-task combination are read from the performance test results. The latency statistics are obtained by repeatedly sampling each model-task combination and statistically analyzing its latency in multiple quantiles. All performance results constitute a model performance profile library.

3. The method according to claim 1, characterized in that, In step (2), the candidate model list is filtered as follows: each available model is judged to determine whether its accuracy on the task reaches the accuracy threshold, and the model that reaches the threshold is included in the candidate model list of the task; when no model has an accuracy that reaches the accuracy threshold, the candidate model list of the task is reverted to all available models.

4. The method according to claim 1, characterized in that, In step (3), the criteria for selecting a model for each ready task are as follows: obtain the latest busy time of each model in the candidate model list of the task and the ready time of the task; for each candidate model, take the larger value between its latest busy time and the task ready time as the starting time of the model to start processing the task, and then add the starting time to the execution time of the task on the model to obtain the expected completion time of the task on the model. The model with the shortest estimated completion time is selected as the execution model for the task; after task allocation, the latest busy time of the execution model is updated to the estimated completion time.

5. The method according to claim 4, characterized in that, In step (3), an event-driven heap is used to implement unified scheduling: Push the first task of each workflow into a min-heap according to its ready time; The following process is executed cyclically: pop the earliest ready task from the min-heap, select a model for the task according to the criteria and assign it; after the task is assigned, update the ready time of the next task in the workflow to the estimated completion time of the task, and push the next task into the min-heap. Scheduling ends when the min-heap is empty. The completion time of all workflows is taken as the maximum value among the latest busy times of all models, and the basic task execution plan is output.

6. The method according to claim 1, characterized in that, In step (4), the on-budget accuracy upgrade includes: For each assigned task and its current model in the basic task execution scheme, traverse all available models and construct a candidate replacement set. The candidate replacement satisfies the condition that the accuracy of the task after replacement is strictly higher than the accuracy of the current model on the task. Calculate the cost-effectiveness of each candidate replacement, defined as the ratio of the accuracy gain after replacement to the time increment, and arrange all candidate replacements in descending order of the ratio; Try candidate replacements one by one in the order of arrangement: After the replacement is executed, re-simulate the execution process of all tasks in the event-driven manner in step (3) to obtain the new completion time and the new overall accuracy; determine whether the new completion time does not exceed the completion time budget and whether the new overall accuracy is higher than the overall accuracy before the replacement. When both conditions are met, accept the replacement and update the current execution plan and the corresponding completion time and overall accuracy. Otherwise, cancel the replacement and continue to try the next candidate replacement in the order of arrangement. When all candidate replacements have been tried and no replacement has been accepted in a certain round of traversal, the upgrade process ends and the upgraded task execution plan is output.

7. The method according to claim 6, characterized in that, In step (4), when constructing the candidate replacement set, for each replacement to be evaluated, the serial chain length of the workflow in which the task is located after the replacement is first calculated. The serial chain length is the sum of the execution time of all tasks in the workflow on the allocated model. The replacement is included in the candidate replacement set only if the serial chain length does not exceed the completion time budget; otherwise, it is directly excluded.

8. The method according to claim 1, characterized in that, The process also includes, after step (4), selecting multiple different candidate budget values ​​within the range of the completion time budget, executing step (4) independently for each candidate budget value to obtain the actual completion time and overall accuracy corresponding to each candidate budget value; selecting non-dominated solutions from the solutions corresponding to all candidate budget values, wherein the non-dominated solution is a solution for which there is no other candidate solution whose actual completion time is not greater than that solution and whose overall accuracy is not lower than that solution, and at least one of these conditions is strictly met; sorting and connecting the non-dominated solutions according to their actual completion time to form a time-accuracy trade-off curve; and selecting the corresponding task execution solution that satisfies both constraints from the trade-off curve and issuing it for execution, based on the deadline and accuracy lower limit in the workflow-level service level agreement constraints.

9. The method according to claim 1, characterized in that, The performance results read in step (1) include accuracy, which is obtained by the judge model scoring the model's generated results for the task: for language tasks, continuous scoring is performed based on the semantic fit between the answer and the reference answer; for code tasks, continuous scoring is performed based on the correctness of the code's functional implementation; and the scoring results are used as the accuracy value of the model on the task.

10. A multi-task flow scheduling system based on SLO constraints, characterized in that, include: The data acquisition module is used to acquire the configured workflow templates and their task dependencies and service level agreement constraints, the performance profiles of each model for tasks in different workflows, and the current workflow queue to be executed and its execution status. The candidate model filtering module is used to filter the list of available candidate models for each task based on an accuracy threshold, and to revert the available models for the task to all available models when the list of candidate models is empty. The unified scheduling module is used to retrieve ready tasks one by one according to their ready time in an event-driven manner. For each ready task, it selects the model that is expected to complete the task earliest from its candidate model list and executes it. It updates the latest busy time of the model and the ready time of subsequent tasks in the workflow until all tasks are scheduled. It then outputs the basic task execution plan and its completion time. The budget-inclusive upgrade module is used to replace each task in the basic task execution plan with a higher-accuracy model in descending order of the ratio of accuracy gain to time increment, within the constraint of not exceeding the completion time budget. It verifies whether the completion time constraint is met and the overall accuracy is improved by replaying all the replaced tasks. Replacements that meet the conditions are accepted and the upgraded task execution plan is output. The execution module is used to distribute the upgraded task execution plan to the heterogeneous model service cluster for execution.

Images