Large language model application workload scheduling method, system and equipment

Abstract

The invention provides a large language model application workload scheduling method, system and device, and relates to the technical field of model load scheduling, and the method comprises the steps: modeling a composite large language model application into a directed acyclic graph comprising a conventional stage, an LLM stage and a dynamic stage; modeling execution correlation among stages in the directed acyclic graph through a Bayesian network, dynamically predicting duration distribution of uncompleted stages, and calibrating a duration estimated value of an LLM stage in combination with a real-time batch processing size of an LLM executor; the uncertainty reduction amount of each ready stage is quantitatively scheduled based on information entropy; an epsilon-greedy strategy is adopted, and a JCT priority queue and an uncertainty reduction priority queue are combined to allocate scheduling resources; and assigning the task to a corresponding executor for execution, and repeating the above process until all operations are completed. The technical problem that the scheduling technology in the prior art is difficult to effectively deal with the execution time uncertainty and the structure uncertainty of the composite LLM application is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of model load scheduling, in particular to a large language model application workload scheduling method, system and device. BACKGROUND

[0002] In the prior art, the scheduling schemes for multi-stage tasks (usually modeled as a directed acyclic graph DAG) are mainly divided into three categories: one is a job-independent scheme, such as first-come-first-served (FCFS), fair scheduling (FairScheduling), etc., which does not depend on the specific characteristics of the job information, but only depends on the task arrival order or resource allocation fairness for scheduling; the second is a job duration-dependent scheme, such as shortest job first (SJF), which determines the priority by estimating the total execution time of the job, and preferentially schedules tasks with shorter estimated duration; the third is a job topology structure-dependent scheme, such as Decima, Argus, etc., which analyzes the structural characteristics (such as node depth, number of child nodes, etc.) of the DAG to make scheduling decisions. The common premise of these existing schemes is that relatively accurate and static job execution time information and DAG structure information can be obtained, so that scheduling decisions are made based on fixed characteristics or rules.

[0003] However, with the rise of composite large language model (LLM) applications (such as code generation, intelligent task planning, multi-turn dialogue systems, etc.), existing scheduling schemes face many challenges: on the one hand, the execution time of composite LLM applications has inherent high uncertainty, especially the LLM inference stage, whose duration is affected by dynamic factors such as input text complexity, model load, batch processing strategy, etc., with significant fluctuations, making it difficult for duration estimation-dependent schemes (such as SJF) to accurately predict, leading to incorrect scheduling decisions and reducing overall scheduling performance and increasing average job completion time (JCT); on the other hand, the execution flow (i.e. DAG structure) of a large number of composite LLM applications is not fixed in advance, but is dynamically generated by the runtime output of the predecessor LLM stage (such as adding new sub-tasks or adjusting dependencies in real time based on inference results), and this structural uncertainty makes it impossible for static topology information-dependent schemes (such as Decima) to adapt, or the scheduling performance is reduced due to incomplete information.

[0004] Therefore, existing scheduling techniques cannot effectively cope with the execution time uncertainty and structural uncertainty of composite LLM applications, and there is an urgent need for a scheduling method that can specifically address these problems to improve the scheduling efficiency of composite LLM application workloads. SUMMARY

[0005] The present application provides a large language model application workload scheduling method, system and device to address the technical problem that existing scheduling techniques cannot effectively cope with the execution time uncertainty and structural uncertainty of composite LLM applications.

[0006] The technical solution of the present application to solve the above technical problems is as follows: In a first aspect, the present application provides a scheduling method for large language model application workloads, comprising: When a job request of a composite large language model application arrives, the composite large language model application is modeled as a directed acyclic graph containing a regular stage, an LLM stage, and a dynamic stage; The execution correlation between stages in the directed acyclic graph is modeled by a Bayesian network, the duration distribution of an unfinished stage is dynamically predicted, and the duration estimate of the LLM stage is calibrated in combination with the real-time batch size of the LLM executor; The uncertainty reduction amount of each ready stage is quantified based on information entropy; An ε-greedy strategy is used to allocate and schedule resources in combination with a JCT priority queue and an uncertainty reduction priority queue; Tasks are assigned to corresponding executors for execution, and the above process is repeated until all jobs are completed.

[0007] In a second aspect, the present application provides a scheduling system for large language model application workloads, comprising: A DAG modeling module for modeling a composite large language model application as a directed acyclic graph containing a regular stage, an LLM stage, and a dynamic stage when a job request of the composite large language model application arrives; A duration prediction module for modeling the execution correlation between stages in the directed acyclic graph by a Bayesian network, dynamically predicting the duration distribution of an unfinished stage, and calibrating the duration estimate of the LLM stage in combination with the real-time batch size of the LLM executor; An uncertainty calculation module for quantifying the uncertainty reduction amount of each ready stage based on information entropy; A resource scheduling module for using an ε-greedy strategy to allocate and schedule resources in combination with a JCT priority queue and an uncertainty reduction priority queue; An output execution module for assigning tasks to corresponding executors for execution, and repeating the above process until all jobs are completed.

[0008] In a third aspect, the present application provides an electronic device, comprising: A memory for storing a computer software program; A processor for reading and executing the computer software program, thereby implementing a scheduling method for large language model application workloads provided by the present application.

[0009] The present application has the following advantages: Compared with the prior art, firstly, when the job request of the composite large language model application arrives, the composite large language model application is modeled as a directed acyclic graph including a regular stage, an LLM stage and a dynamic stage, effectively solving the problem of structural uncertainty, providing a unified structured model for subsequent analysis, and supporting subsequent uncertainty quantification and resource allocation decision. Secondly, the execution correlation between stages in the directed acyclic graph is modeled by a Bayesian network, the time length distribution of the unfinished stage is dynamically predicted, and the time length estimate of the LLM stage is calibrated by combining the real-time batch size of the LLM executor. The actual data of the completed stage is used as evidence to real-time correct the time length estimate of the unfinished stage, so that the prediction changes from the static initial value to the dynamic updated value, effectively solving the problem of time uncertainty. Thirdly, the uncertainty reduction amount of each ready stage is quantified based on information entropy, the inherent uncertainty of each stage is quantified by information entropy, and the correlation strength between stages is measured by mutual information. Finally, the uncertainty reduction amount is calculated by combining the time length distribution range, which can be used to evaluate the value of executing a stage to reduce uncertainty and provide a quantitative basis for the subsequent epsilon-greedy strategy. Further, the epsilon-greedy strategy is adopted, and the JCT priority queue and the uncertainty reduction priority queue are combined to allocate and schedule resources, so that in a complex dynamic environment, the job can be quickly promoted, and the long-term inaccuracy can be avoided by reducing uncertainty, improving the overall scheduling efficiency of the composite LLM application. Finally, the task is assigned to the corresponding executor for execution, and the above process is repeated until all jobs are completed, effectively balancing the scheduling goals of quickly promoting job progress and reducing uncertainty, and improving the accuracy of resource allocation.

[0010] Through the above technical solution, the application solves the structural uncertainty of the composite LLM application through dynamic stage modeling, solves the execution time uncertainty of the LLM inference stage through Bayesian network dynamic prediction and batch processing calibration, and overcomes the scheduling inaccuracy problem caused by the dependence of the existing scheme on static information. Based on the information entropy quantification of the uncertainty reduction amount, the epsilon-greedy strategy is combined to balance the short-term quick job completion and the long-term uncertainty reduction, so that the scheduling decision is more in line with the dynamic characteristics of the composite LLM application, reducing the waiting and blocking caused by estimation deviation or unreasonable resource allocation, and greatly shortening the average JCT. In this way, a complete solution is provided for the composite LLM application, which takes into account dynamic adaptability and scheduling efficiency, effectively improving resource utilization and job processing efficiency in complex scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 A flowchart of a large language model application workload scheduling method provided by the application; Figure 2 A program block diagram of a large language model application workload scheduling method provided by the application; Figure 3 A schematic diagram of the structure of a scheduling system for large language model application workloads provided by the present invention; Figure 4 This is a schematic structural diagram of the electronic device provided by the present invention.

[0012] In the accompanying drawings, the components represented by the reference numerals are as follows: DAG modeling module 11, duration prediction module 12, uncertainty calculation module 13, resource scheduling module 14, output execution module 15, electronic device 200, memory 210, processor 220, computer program 211. DETAILED DESCRIPTION

[0013] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0014] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the specified features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0015] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or illustration". Any embodiment of the present invention described as "for example" is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any person skilled in the art to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed herein.

[0016] Example 1, as Figure 1 As shown, an embodiment of the present invention provides a method for scheduling a large language model application workload, including: S10: When a job request of a composite large language model application arrives, model the composite large language model application as a directed acyclic graph containing a regular stage, an LLM stage, and a dynamic stage.

[0017] The execution time of a composite LLM application has inherent high uncertainty, especially the LLM inference stage, which is affected by dynamic factors such as input text complexity, model load, and batch processing strategy, and has significant fluctuations, making it difficult for solutions that rely on time length estimation (such as SJF) to accurately predict and make incorrect scheduling decisions, thereby reducing overall scheduling performance and increasing the average job completion time (JCT). The execution flow of a large number of composite LLM applications (i.e., the DAG structure) is not fixed in advance, but is dynamically generated by the runtime output of the predecessor LLM stage (such as adding subtasks or adjusting dependencies in real time based on inference results). This structural uncertainty makes it impossible for solutions that rely on static topology information (such as Decima) to adapt, or the scheduling performance is reduced due to incomplete information.

[0018] To solve the above problems, when a job request of a composite large language model application arrives, the composite large language model application is modeled as a directed acyclic graph containing a regular stage, an LLM stage, and a dynamic stage.

[0019] Specifically, step S10 in the method includes: The regular stage is used to execute non-LLM tasks, and the execution time fluctuation range does not exceed a preset threshold. The LLM stage is used to execute LLM inference tasks, and the execution time is affected by input characteristics, hardware load, and batch processing strategy. The dynamic stage is a placeholder node, and the sub-stages and dependencies between sub-stages contained therein are dynamically generated at runtime by the output results of its predecessor LLM stage.

[0020] In an embodiment of the present application, as shown in Figure 2 First, determine the three stages of a composite large language model (LLM) application in a directed acyclic graph (DAG): a regular stage, an LLM stage, and a dynamic stage.

[0021] Among them, the conventional stage is a node that executes deterministic non-LLM tasks, such as data cleaning (such as format conversion, field verification, etc.), third-party API calls (such as calling map services, payment interfaces, etc.), fixed logic calculations (such as statistical data volume, file format conversion, etc.), etc. These tasks do not involve LLM inference, rely on pre-set program logic or external service interfaces, and the execution time is determined by the task input size and fixed logic. The fluctuation range is strictly controlled within the pre-set threshold, for example, the time-consuming of a certain API call is stable at 100ms±5ms, and the fluctuation does not exceed 5%. This stability makes it consistent with the characteristics of traditional computing tasks, and can be used as data preprocessing before LLM inference and result storage after inference in the directed acyclic graph (DAG), avoiding scheduling logic confusion caused by full-process uncertainty.

[0022] Among them, the LLM stage is a node that executes LLM inference tasks in the directed acyclic graph (DAG), such as text generation (such as generating reports, code, etc. according to requirements), semantic understanding (such as parsing user query intent, etc.), logical inference (such as planning task steps, etc.), etc. It relies on LLM instances deployed on special executors (such as servers equipped with GPUs). Further, since the length (such as generating 100 words or 1000 words) and complexity (such as simple instructions or multi-round logical inference) of the input text directly affect the LLM inference time; the GPU / CPU utilization and memory occupancy of the LLM executor dynamically change, and the single-task delay will significantly increase under high load; the number of tasks processed simultaneously by the LLM executor (batch size) affects resource allocation, and the larger the batch size, the higher the single-task waiting and processing delay. Therefore, the LLM stage has the characteristics of time uncertainty, providing a targeted analysis object for subsequent Bayesian network dynamic prediction of time length and batch processing calibration, avoiding scheduling errors caused by inaccurate time estimation.

[0023] The dynamic stage is a placeholder for structural uncertainty, and adapts to dynamically generated processes. The dynamic stage is a node in the directed acyclic graph (DAG) that carries sub-tasks and dependencies generated at runtime. Its core feature is structural uncertainty. When the composite LLM application is initially submitted, the dynamic stage only exists as an empty node in the DAG, without specific sub-tasks or dependencies. Its internal structure (sub-stages, dependencies between sub-stages) is determined by the output of the predecessor LLM stage. For example, in a code generation and debugging application, if the output of the predecessor LLM stage is "there is a syntax error", the dynamic stage will generate sub-stages and dependencies for "fixing syntax errors to verifying the repair results". If the output of the predecessor LLM stage is "logic is correct", the dynamic stage is empty. Through the placeholder + runtime filling method, the DAG can adapt to the scenario of non-predefined processes in the composite LLM application in real time, solve the problem of traditional static DAG that cannot describe dynamic structure, and provide complete and real-time task flow information for the scheduler.

[0024] Further, when the job request of the composite large language model application arrives, the composite large language model application is modeled as a directed acyclic graph containing a regular stage, an LLM stage, and a dynamic stage. For example, if the user complaint automatic handling system of an e-commerce platform receives a user request: "The earphone I bought last week has no sound, and the refund has not arrived. Help me solve it.", the system needs to complete the process of "request analysis, responsibility determination, solution generation, and result notification". Its DAG modeling is as follows: 1. Determine the regular stage, such as format standardization (e.g. remove colloquial words, extract key information), user information verification (query whether the user order number exists), these tasks depend on fixed rules (e.g. text cleaning templates, database query interfaces), the execution time is stable and the process is fixed, and in the DAG, it is used as a starting node. 2. Determine the LLM stage, such as complaint intent analysis (call LLM executor to identify user core demands), generate answers (call LLM executor to analyze responsible parties), these tasks have large time fluctuations, if the user complaint content is short (e.g. earphone is broken), LLM analysis takes about 1 second, if the content is lengthy (contains multiple rounds of historical dialogue), the time consumption may be up to 5 seconds; and affected by hardware load, if the LLM executor is processing multiple complaint analysis tasks at the same time, the batch processing delay increases, and the time consumption may be extended to 8 seconds, in the DAG, the output of the regular stage (standardized complaint content) determines the structure of the subsequent dynamic stage. 3. Determine the dynamic stage, the initial state is a placeholder node in the DAG, without specific sub-tasks, its structure is determined by the output of the LLM stage, with structural uncertainty.

[0025] Exemplarily, if the LLM stage determines that the responsibility is on the merchant (earphone quality problem), sub-stages are generated: [sub-stage 1: call merchant interface to negotiate exchange] to [sub-stage 2: generate exchange process description]; if the LLM stage determines that the responsibility is on the platform (refund process problem): sub-stages are generated: [sub-stage 3: trigger financial system to urgently refund] to [sub-stage 4: verify the status of the refund to account] to [sub-stage 5: generate a refund progress notification]; if the LLM stage determines that further verification is needed (insufficient information), sub-stages are generated: [sub-stage 6: call LLM to generate follow-up questions] to [sub-stage 7: wait for user response and re-analyze]. In this way, a directed acyclic graph is obtained: [regular stage: input information standardization, information verification] to [LLM stage: complaint analysis, responsibility determination] to [dynamic stage (placeholder, sub-process determined by LLM stage output result)] to [regular stage: result notification].

[0026] In summary, compared with the prior art, when a job request of a composite large language model is applied, the composite large language model is modeled as a directed acyclic graph including a regular stage, an LLM stage, and a dynamic stage. In this way, the problem of structural uncertainty is effectively solved, a unified structured model is provided for subsequent analysis, and uncertainty quantification and resource allocation decisions are supported.

[0027] S20: modeling the execution correlation between stages in the directed acyclic graph through a Bayesian network, dynamically predicting the time length distribution of the unfinished stage, and combining the real-time batch size of the LLM executor to calibrate the time length estimation value of the LLM stage.

[0028] There is a natural correlation between the stages of the composite LLM application, such as the execution time length of LLM stage A affecting the time length of subsequent LLM stage B. Therefore, based on this correlation, the actual data of the completed stage can be used as evidence to real-time correct the time length estimation of the unfinished stage, so that the prediction changes from a static initial value to a dynamic updated value, greatly improving the accuracy.

[0029] To solve the above problems, the execution correlation between stages in the directed acyclic graph is modeled through a Bayesian network, the time length distribution of the unfinished stage is dynamically predicted, and the time length estimation value of the LLM stage is calibrated in combination with the real-time batch size of the LLM executor.

[0030] Specifically, step S20 in the method includes: discretize the time length distribution of each stage into a preset number of intervals, and take each stage as a variable in the Bayesian network; training a Bayesian network based on historical execution data to capture execution dependencies between stages and derive conditional probability distributions of the duration of each stage and its dependencies, wherein if there is a direct path between two stages in the Bayesian network, it is determined that the two stages have execution dependencies, and the duration distribution of a child stage is modeled by the probability distribution of a parent stage; obtaining the actual duration of a completed stage and mapping it to its corresponding discretized interval, inputting the interval as evidence into the Bayesian network, and outputting the posterior conditional probability distribution of an uncompleted stage that has execution dependencies with the stage; updating the duration estimation of the uncompleted stage according to the posterior conditional probability distribution.

[0031] In the embodiments of the present application, the duration distribution of each stage is first discretized into a preset number of intervals, and each stage is taken as a variable in the Bayesian network, which facilitates the calculation of probability. The Bayesian network is a probability-based graphical model that intuitively represents the dependency relationship between variables through a directed acyclic graph (DAG), and quantifies these relationships using conditional probability distributions, and is particularly good at handling discrete variables. For example, if the LLM stage includes stages A and B, the dependency relationship between them is complaint analysis to responsibility determination, the duration distribution of stage A is discretized into 3 intervals: [1-2 seconds], [2-4 seconds], [4-7 seconds], and the duration distribution of stage B is discretized into the same intervals, which are taken as variables in the Bayesian network.

[0032] Secondly, the Bayesian network is trained based on historical execution data to capture execution dependencies between stages and derive conditional probability distributions of the duration of each stage and its dependencies, wherein if there is a direct path between two stages in the Bayesian network, it is determined that the two stages have execution dependencies, and the duration distribution of a child stage is modeled by the probability distribution of a parent stage. For example, during the training process, joint distribution data of the duration intervals of multiple stages with execution dependencies is extracted from historical execution data: after discretizing the historical duration of each stage into preset intervals, the joint occurrence frequency of the duration intervals of different stages is counted to construct a training sample set, for example, it is counted that the proportion of samples whose duration of stage A falls into [1-2 seconds] and whose duration of stage B falls into [1-2 seconds] is 80%, and the proportion of samples whose duration of stage A falls into [2-4 seconds] and whose duration of stage B falls into [2-4 seconds] is 70%, and the joint distribution data is taken as training samples, based on which the Bayesian network is trained to learn and derive the conditional probability distribution (CPD) between stages, i.e. given the duration interval of a parent stage, the probability of a child stage falling into each duration interval is output.

[0033] Next, the actual duration of the completed stage is obtained and mapped to its corresponding discretized interval. This interval is input as evidence into the Bayesian network, and the posterior conditional probability distribution of the unfinished stage that has execution relevance to this stage is output. For example, if the actual execution duration of stage A in the LLM stage is 4 seconds, which falls within the preset [2-4 seconds] interval, then the duration of stage A ∈ [2-4 seconds] is input as evidence into the Bayesian network. Based on the pre-trained conditional probability distribution, the Bayesian network outputs the posterior conditional probability distribution of the unfinished stage (such as stage B) that has execution relevance to stage A. For example, the probability that the duration of stage B falls within [2-4 seconds] is 60%, the probability that it falls within [4-6 seconds] is 30%, and the probability that it falls within [6-8 seconds] is 10%.

[0034] Finally, the duration estimate of the unfinished stage is updated according to the posterior conditional probability distribution to achieve dynamic adjustment, thereby reducing the impact of uncertainty on scheduling decisions. For example, if the initial duration estimate of the unfinished stage B is set to 5 seconds based on historical average data, the duration estimate of the unfinished stage is updated according to the posterior conditional probability distribution. For example, if the posterior conditional probability distribution shows that the probability of the duration of stage B falling into [2-4 seconds] is 60%, the probability of falling into [4-6 seconds] is 30%, and the probability of falling into [6-8 seconds] is 10%, then the duration estimate of stage B is updated from the initial 5 seconds to a high probability of falling into the 2-4 second range. In this way, the duration uncertainty of stage B is reduced, and the scheduler can adjust resource allocation accordingly.

[0035] Furthermore, the “calibrating the estimated duration of the LLM phase in combination with the real-time batch size of the LLM executor” includes: Get the LLM phase duration recorded under the reference batch size and the corresponding average decoding delay; Get the target batch size of the current LLM executor and the corresponding average decoding delay; Use the following calibration formula to correct the estimated duration under the target batch size: ; in, is the target batch size The estimated duration of the next is the reference batch size The duration of the recording, is the target batch size The average decoding delay recorded below, is the reference batch size The average decoding delay recorded below.

[0036] In the embodiments of the present application, when processing tasks, the LLM executor (such as a server equipped with a GPU) will package multiple tasks into batches for simultaneous processing (i.e., batch processing), and the number of tasks processed at the same time is referred to as the batch size, which directly affects the execution delay of a single task: the larger the batch size, the more intense the competition for resources such as GPU computing power and memory, and the longer the waiting time and processing time of a single task (i.e., the higher the average decoding delay); conversely, the smaller the batch size, the lower the single task delay. Since the batch size is dynamically changing, such as when the executor load fluctuates, the batch size may be adjusted from 2 to 5, and if only a fixed historical time length is used to estimate the execution time of the LLM stage, there will be a deviation due to not considering the real-time batch processing state. Therefore, it is necessary to calibrate the estimated time length based on the ratio of the real-time batch size to the delay characteristic of the historical reference batch, specifically: First, the recorded LLM stage time length under the reference batch size is obtained, and the corresponding average decoding delay, wherein the reference batch size refers to the complete batch size commonly used in historical execution, such as the most commonly used batch size of 2 or 4 in a production environment, which is used as a reference for calibration. The recorded LLM stage time length under the reference batch size refers to the actual execution time of a certain LLM stage under this batch size, for example, the actual execution time of this stage is 4 seconds when the reference batch size is 2. The average decoding delay corresponding to the reference batch size refers to the average delay of a single task from input to output under this batch size, for example, the average decoding delay of each task is 2 seconds / task when the reference batch size is 2. The average decoding delay can reflect the inherent delay characteristics of the batch size.

[0037] Second, the target batch size of the current LLM executor and the corresponding average decoding delay are obtained. Exemplarily, the target batch size refers to the batch size currently processed by the current LLM executor, such as the current batch size being adjusted to 5 due to increased load, and the average decoding delay corresponding to the target batch size is obtained by real-time monitoring of the average delay of a single task under this batch size, for example, the average decoding delay of each task is 5 seconds / task when the target batch size is 5, which is higher than the average decoding delay corresponding to the reference batch size due to more intense resource competition.

[0038] Finally, the estimated time length under the target batch size is corrected using the following calibration formula: ; Wherein, is the estimated time length under the target batch size, is the recorded time length under the reference batch size, is the average decoding delay corresponding to the target batch size, is the average decoding delay corresponding to the reference batch size, is the average decoding delay corresponding to the target batch size, the average decoding delay of the records under the target batch size, is the average decoding delay of the records under the reference batch size the average decoding delay of the records under the target batch size, i.e. the ratio of the average decoding delay of the records under the target batch size to the average decoding delay of the records under the reference batch size, scales the duration of the records under the reference batch size, if the delay of the target batch is higher ( > ), the ratio is greater than 1, and the calibrated is greater than , which conforms to the actual situation that the larger the batch is, the longer the single task takes, if the delay of the target batch is lower ( < ), the ratio is less than 1, and the calibrated is less than .

[0039] Exemplarily, if the reference batch size = 2 (2 tasks are processed at the same time), the duration of the records under the reference batch size in the historical records is = 4 seconds, and the average decoding delay is = 2 seconds / task, at this time, due to the increase of the executor load, the current target batch size = 5 (5 tasks are processed at the same time), the average decoding delay under the batch is = 5 seconds / task, and the calibration formula is = 10 seconds, the estimated duration of the LLM stage is calibrated from the reference value of 4 seconds to 10 seconds when the batch size is 5, which is more consistent with the actual execution delay that the single task takes longer due to the increase of the batch size and the resource competition. In this way, through calibration, the duration estimation of the LLM stage is no longer dependent on the fixed historical reference value, but can dynamically adapt to the real-time batch state of the executor, reducing the estimation error caused by the batch fluctuation, which provides more accurate time basis for the scheduler, so that it is more accurate in resource allocation (such as reserving GPU duration) and task sorting (such as preferentially scheduling tasks with shorter calibrated duration), thereby reducing resource idling or task waiting, and improving overall scheduling efficiency.

[0040] In summary, compared with the prior art, the present application models the execution correlation between each stage in the directed acyclic graph through the Bayesian network, dynamically predicts the duration distribution of the unfinished stage, and calibrates the duration estimation value of the LLM stage in combination with the real-time batch size of the LLM executor. In this way, the actual data of the completed stage is taken as evidence to real-time correct the duration estimation of the unfinished stage, so that the prediction changes from a static initial value to a dynamically updated value, effectively solving the problem of time uncertainty.

[0041] S30: Quantify the uncertainty reduction amount of each ready stage based on information entropy.

[0042] When the execution of the ready stage has an execution correlation with the related unscheduled stage, the execution result of the ready stage affects the uncertainty of the related unscheduled stage, which is specifically manifested as a decrease in the correlation stage information entropy. For such a ready stage, the scheduling priority should be quantitatively evaluated based on the uncertainty reduction amount that the ready stage can bring. The greater the uncertainty reduction amount, the more significant the decrease in the overall uncertainty after the ready stage is scheduled, and the higher priority the ready stage should be given in the scheduling sequence.

[0043] To solve the above problems, the information entropy of each ready stage is quantitatively evaluated based on the uncertainty reduction amount that the ready stage can bring.

[0044] Specifically, step S30 in the method includes: The random variables corresponding to the regular stage, the LLM stage and the dynamic stage are defined respectively, and the information entropy of each stage is calculated based on Shannon entropy. A set of unscheduled stages related to the current ready stage is determined. The mutual information between the current ready stage X and the related unscheduled stage Y is calculated using the following mutual information formula: ; Where p(x, y) is the joint probability of X and Y, and p(x) and p(y) are the marginal probabilities of X and Y, respectively. The time length distribution range of the related unscheduled stage is obtained, and the uncertainty reduction amount of scheduling the current ready stage is calculated based on the mutual information and the time length distribution range.

[0045] In the embodiments of the present application, first, the random variables corresponding to the regular stage, the LLM stage and the dynamic stage are defined respectively, and the information entropy of each stage is calculated based on Shannon entropy. The random variables include, for example, whether to execute, time length interval, and whether the sub-stage exists, etc. Shannon entropy can quantify the uncertainty of each stage itself. The higher the entropy value, the greater the uncertainty.

[0046] Secondly, a set of unscheduled stages related to the current ready stage is determined. For example, the unscheduled stages having a dependency relationship with the current ready stage are identified, because scheduling the current ready stage can affect the uncertainty of these related unscheduled stages.

[0047] Thirdly, the mutual information between the current ready stage X and the related unscheduled stage Y is calculated using the following mutual information formula: ; Where p(x, y) is the joint probability of X and Y, and p(x) and p(y) are the marginal probabilities of X and Y, respectively. Mutual information is used to measure the correlation strength between the current ready stage X and the related unscheduled stage Y. The higher the mutual information, the more the execution result of the current ready stage can reduce the uncertainty of the related unscheduled stage.

[0048] Exemplarily, if X and Y are independent, p(x, y) = p(x)p(y), mutual information = 0, i.e. the result of X does not affect Y; if the probability of the current ready stage X occurring in [1-3 seconds] is 0.5, the probability of the current ready stage X occurring in [3-6 seconds] is 0.5, the probability of the related unscheduled stage Y occurring in [1-3 seconds] is 0.5, the probability of the related unscheduled stage Y occurring in [3-6 seconds] is 0.5, the probability of the current ready stage X occurring in [1-3 seconds] and the related unscheduled stage Y occurring in [1-3 seconds] is 0.4, the probability of the current ready stage X occurring in [1-3 seconds] and the related unscheduled stage Y occurring in [3-6 seconds] is 0.1, the probability of the current ready stage X occurring in [3-6 seconds] and the related unscheduled stage Y occurring in [1-3 seconds] is 0.1, and the probability of the current ready stage X occurring in [3-6 seconds] and the related unscheduled stage Y occurring in [3-6 seconds] is 0.4, the mutual information at this time indicates that the current ready stage X and the related unscheduled stage Y have strong correlation.

[0049] Finally, the time length distribution range of the related unscheduled stage is obtained, and the uncertainty reduction amount of scheduling the current ready stage is calculated based on the mutual information and the time length distribution range, wherein the time length distribution range is the difference between the maximum value and the minimum value of the time length, reflecting the fluctuation range of the time length of the related unscheduled stage, and the larger the time length distribution range, the higher the uncertainty magnitude. Exemplarily, by combining the mutual information and the time length distribution range, the reduction degree of the overall uncertainty after scheduling the current ready stage is quantified comprehensively, and the higher the uncertainty reduction amount, the greater the value of scheduling the stage.

[0050] Specifically, the "defining corresponding random variables for the regular stage, the LLM stage and the dynamic stage, and calculating the information entropy of each stage based on Shannon entropy" comprises: defining corresponding random variables for the regular stage, the LLM stage and the dynamic stage, and determining the probability distribution of the values of the random variables, and calculating the information entropy of the regular stage, the LLM stage and the dynamic stage based on the following Shannon entropy formula: ; wherein, is the information entropy of each stage, is the probability of each possible value of the random variable.

[0051] Further, the calculation method of the information entropy for the regular stage, the LLM stage and the dynamic stage is specifically as follows: defining whether the regular stage is executed as a binary random variable X, wherein X ∈ {0, 1}, 1 represents execution, and 0 represents non-execution; obtaining the execution probability p and the non-execution probability 1-p of the regular stage; The information entropy of the LLM stage is calculated by using the following entropy calculation formula : ; The time length distribution of the LLM stage is discretized into k intervals, and non-execution is regarded as a time length of 0 s, and a discrete random variable containing k+1 values is defined; The probability of non-execution of the LLM stage is obtained , and the probability of the time length falling into the i-th interval is (i=1, 2, …, k); The information entropy of the LLM stage is calculated by using the following entropy calculation formula : ; The candidate sub-stage set of the dynamic stage is determined and the candidate dependent edge set E; Whether the candidate sub-stage exists or not is defined as a binary random variable , wherein ∈{0, 1}, 1 represents that the sub-stage exists, and 0 represents that the sub-stage does not exist; Whether the candidate dependent edge exists or not is defined as a binary random variable , wherein ∈{0, 1}, 1 represents that the dependent edge exists, and 0 represents that the dependent edge does not exist; The entropy of each candidate sub-stage is calculated and the entropy of each candidate dependent edge is calculated , and the information entropy of the dynamic stage is calculated by using the following formula : .

[0052] In the embodiments of the present application, the core of the Shannon entropy is to quantify the uncertainty of the random variable, the more dispersed the possible values of the variable are and the more average the probability is, the higher the entropy value is, and the greater the uncertainty is. The information entropy of each stage is calculated for the regular stage, the LLM stage and the dynamic stage respectively. Specifically, First, for the regular stage, the random variable of the regular stage is whether to execute or not, so whether to execute or not is defined as a binary random variable X∈{0, 1}, 1 represents execution, and 0 represents non-execution. Exemplarily, the execution probability p is obtained through historical data statistics, such as 0.9, then the non-execution probability 1-p=1-0.9=0.1, the information entropy =0.47, which indicates that the uncertainty is low.

[0053] Secondly, for the LLM stage, the random variables of the LLM stage are whether to execute and the execution duration. Therefore, the duration distribution of the LLM stage is discretized into k intervals, and non-execution is regarded as a duration of 0s, and a discrete random variable containing k+1 values ​​is defined. For example, the duration distribution of the LLM stage is discretized into k intervals, such as k=3, plus non-execution (duration 0 seconds), for a total of k+1 values, corresponding to non-execution and three duration distribution intervals, respectively. The probability of non-execution is statistically calculated through historical data. , and the probability that the duration falls into the i-th interval (i=1, 2, ..., k), for example, the probability of non-execution of a certain LLM stage =0.1, the probability of falling into [1-3 seconds] =0.6, the probability of falling into [3-6 seconds] =0.2, the probability of falling into [6-10 seconds] =0.1, then information entropy This is because the probability of the duration interval is more dispersed and the uncertainty is higher than that in the normal stage. The information entropy can comprehensively reflect the comprehensive uncertainty of the execution duration and whether it is executed.

[0054] Finally, for the dynamic stage, the random variables of the dynamic stage are structurally uncertain, that is, whether there are sub-stages and the dependencies between sub-stages, so the uncertainty of candidate sub-stages and candidate dependency edges is considered separately. For example, the set of candidate sub-stages of the dynamic stage is determined and the candidate dependency edge set E, where the candidate sub-stage set There are 2 candidate sub-stages in ={c1,c2}, there is one candidate dependency edge in the candidate dependency edge set E, E={e1}, if the probability of c1 existing is 0.5, the probability of c2 existing is 0.8, and the probability of e1 existing is 0.6, then the entropy =1, entropy =0.72, entropy =0.97, then the information entropy in the dynamic stage ,This is because the information entropy is usually higher than the conventional and LLM stages due to the high structural uncertainty.

[0055] Specifically, the step of “obtaining a duration distribution range of the relevant unscheduled phase, and calculating an uncertainty reduction amount for scheduling the current ready phase based on the mutual information and the duration distribution range” includes: Obtain the evidence set E of the completed phase duration and the duration distribution range of the related unscheduled phase Y ; The uncertainty reduction is calculated using the following formula: ; in, denotes mutual information, is the unscheduled stage related to X, M is the total number of unscheduled stages related to X, m is an index variable traversing the unscheduled stages related to X, and m ranges from 1 to M.

[0056] In the embodiments of the present application, first, the evidence set E of the completed stage duration is obtained, the duration distribution range of the related unscheduled stage Y is obtained, wherein the evidence set E is the actual duration of the completed stage mapped to the corresponding discretization interval, and the duration distribution range of the related unscheduled stage Y is the difference between the maximum and minimum values of the possible duration value interval of Y under the evidence set E. For example, if the completed stage includes a regular stage A (actual duration of 2 seconds, falling into the interval [1-3 seconds]) and an LLM stage B (actual duration of 5 seconds, falling into the interval [4-6 seconds]), then the evidence set E = {A ∈ [1-3 seconds], B ∈ [4-6 seconds]}, and then the difference between the maximum and minimum values of all possible value intervals of the posterior duration distribution of Y based on the evidence set E is calculated, for example, if the posterior duration distribution of the related unscheduled stage Y under the evidence E can fall into [2-4 seconds], [4-6 seconds], and [6-8 seconds], then the duration distribution range = 8 seconds - 2 seconds = 6 seconds, and the duration distribution range reflects the uncertainty degree of Y.

[0057] Secondly, the uncertainty reduction amount is calculated by the following formula: ; wherein, denotes mutual information, is the unscheduled stage related to X. Specifically, is the total mutual information between the current ready stage X and all related unscheduled stages under the evidence set E, is the sum of the duration distribution ranges of all related unscheduled stages, reflecting the total magnitude of the uncertainty of these stages, denotes the reduction amount of the overall uncertainty after scheduling the current ready stage X, the higher the value is, the greater the value of scheduling X is. For example, if the current ready stage X has two related unscheduled stages Y1 and Y2, the total mutual information = 0.5 under the evidence set E, the duration distribution range of Y1 is 4 seconds, the duration distribution range of Y2 is 6 seconds, then the sum of the duration distribution ranges of all related unscheduled stages = 4 + 6 = 10 seconds, and the uncertainty reduction amount 0.5 * 10 = 5, and if another ready stage The total mutual information is also 0.5, but the sum of the time length distribution range of all relevant stages of the relevant unscheduled stage is 15 seconds, so the uncertainty reduction amount 0.5*15=7.5, which indicates that the scheduling The uncertainty reduction effect of the unscheduled stage is more significant, and should be selected by the uncertainty reduction priority queue.

[0058] In summary, compared with the prior art, the present application quantifies the uncertainty reduction amount of each ready stage based on information entropy. In this way, by quantifying the inherent uncertainty of each stage by information entropy, by measuring the correlation strength between stages by mutual information, and finally by calculating the uncertainty reduction amount combined with the time length distribution range, the value of executing a certain stage to reduce uncertainty can be evaluated, which provides a quantitative basis for the subsequent ε-greedy strategy.

[0059] S40: Adopting the ε-greedy strategy, combining the JCT priority queue and the uncertainty reduction priority queue to allocate scheduling resources.

[0060] The foregoing steps obtain the execution time, uncertainty reduction amount and information entropy of each ready task, and the ε-greedy strategy can be used for scheduling resource allocation based on the foregoing.

[0061] To solve the above problems, the present application adopts the ε-greedy strategy, combines the JCT priority queue and the uncertainty reduction priority queue to allocate scheduling resources.

[0062] Specifically, step S40 in the method comprises: All ready tasks are sorted according to the shortest remaining time first principle as a JCT priority queue; All ready tasks are sorted in descending order of uncertainty reduction amount as an uncertainty reduction priority queue; Each time the scheduling is performed, a task is selected from the uncertainty reduction priority queue with a preset probability ε, and a task is selected from the JCT priority queue with a probability 1-ε, wherein the preset probability ε is determined according to the information entropy; When executing the task from the uncertainty reduction priority queue, part of the subtasks can be selectively executed.

[0063] In the embodiments of the present application, as Figure 2As shown, first, all ready tasks are sorted in the order of the shortest remaining time as the JCT priority queue, where the core goal of the JCT (Job Completion Time) priority queue is to shorten the overall job completion time. For example, all ready tasks (such as the regular stage, the LLM stage, and the sub-stage of the dynamic stage) are sorted in the order of the shortest remaining time based on the dynamic prediction of the remaining time by the Bayesian network (the shorter the remaining time of a task, the higher the priority of the task in the queue), for example, if the predicted remaining time of a ready task A is 2 seconds and the predicted remaining time of a ready task B is 5 seconds, the ready task A is arranged in front of the ready task B in the JCT queue, and the ready task A can be completed faster when being selected first, thereby reducing the overall job blocking.

[0064] Secondly, all ready tasks are sorted in the order of the descending uncertainty reduction amount as the uncertainty reduction priority queue, where the core goal of the uncertainty reduction priority queue is to reduce the overall uncertainty by executing the ready tasks to provide more accurate information for subsequent scheduling. For example, the ready tasks are sorted in the order of the descending uncertainty reduction amount (the greater the uncertainty reduction amount, the higher the priority of the task in the queue), for example, if the uncertainty reduction amount of a ready task C is 10 and the uncertainty reduction amount of a ready task D is 3, the ready task C is arranged in front of the ready task D, and the ready task C can reduce the system uncertainty faster when being selected first.

[0065] Thirdly, at each scheduling, a task is selected from the uncertainty reduction priority queue with a preset probability ε and a task is selected from the JCT priority queue with a probability of 1-ε, where the preset probability ε is determined according to the information entropy. Specifically, the ε-greedy strategy is a classic exploration-exploitation balance algorithm, at each scheduling, a task is selected from the uncertainty reduction priority queue with a preset probability ε (exploration behavior), and a task is selected from the JCT priority queue with a probability of 1-ε (exploitation behavior). When a task is selected from the JCT priority queue, the shortest remaining time is directly utilized to quickly advance the job completion and avoid resource idling. When a task is selected from the uncertainty reduction priority queue, the overall uncertainty is reduced by executing the task to provide a more accurate basis for subsequent scheduling. For example, the value of ε is not fixed but is dynamically adjusted according to the overall information entropy (the sum of the information entropy of all incomplete stages): when the overall information entropy is high, it indicates that the overall uncertainty is strong, such as the structure of multiple dynamic stages being unknown and the time length of the LLM stage fluctuating greatly, and ε can be increased, such as from 0.2 to 0.5, to preferentially select a task that can reduce the uncertainty and avoid long-term scheduling errors due to insufficient information; when the overall information entropy is low, it indicates that the overall uncertainty is weak, such as most stage time lengths being clear and the structure of the dynamic stage being determined, and ε can be appropriately reduced, such as from 0.5 to 0.1, to preferentially select a task that can be completed quickly to shorten the overall job time.

[0066] Finally, when executing the task from the uncertainty reduction priority queue, part of the sub-tasks can be selectively executed. Exemplarily, for the task from the uncertainty reduction priority queue, part of the sub-tasks can be selectively executed instead of all, and the sub-tasks can be sorted according to the contribution of the uncertainty reduction amount, and the sub-tasks with a cumulative contribution reaching a preset threshold (such as 80%) are preferentially executed. In this way, under the premise of significantly reducing uncertainty, unnecessary calculation resource consumption is reduced, and key evidence is quickly obtained for updating the probability distribution of the related stage.

[0067] In summary, compared with the prior art, the present application adopts the ε-greedy strategy, and combines the JCT priority queue and the uncertainty reduction priority queue to allocate and schedule resources. In this way, in a complex dynamic environment, the work can be quickly promoted, and long-term inaccuracy can be avoided by reducing uncertainty, thereby improving the overall scheduling efficiency of the composite LLM application.

[0068] S50: dispatch the task to the corresponding executor for execution, and repeat the above process until all jobs are completed.

[0069] In the embodiments of the present application, the task selected by the ε-greedy strategy is allocated to the executor matching its type for running, such as a general CPU executor for a regular stage, a GPU dedicated executor for an LLM stage, and the like, and the actual execution result of the task, such as the time length, the dynamic stage substructure, and the like, is obtained. Subsequently, the result is used as new evidence to update the posterior distribution of the Bayesian network, and the information entropy and the uncertainty reduction amount of each stage are recalculated, and the JCT priority queue and the uncertainty reduction priority queue and the probability ε are adjusted. The task selection and dispatch are performed again, and the cycle is repeated until all stages in the directed acyclic graph of the composite LLM application are executed and the job is completed. In this way, the scheduling objectives of quickly promoting the job progress and reducing the uncertainty are effectively balanced, the overall scheduling efficiency of the composite LLM application is improved, the job completion time (JCT) is shortened, the adaptability to the dynamic execution environment is enhanced, and the accuracy of resource allocation is improved.

[0070] In summary, the embodiments of the present application have at least the following technical effects: Compared with the prior art, the present application first models the composite large language model application as a directed acyclic graph including a regular stage, an LLM stage, and a dynamic stage. In this way, the problem of structural uncertainty is effectively solved, a unified structured model is provided for subsequent analysis, and the subsequent uncertainty quantification and resource allocation decision are supported.

[0071] Secondly, the application models the execution correlation between stages in the directed acyclic graph through a Bayesian network, dynamically predicts the time length distribution of an unfinished stage, and calibrates the time length estimate of the LLM stage in combination with the real-time batch size of the LLM executor.

[0072] Thirdly, the application quantifies the uncertainty reduction amount of each ready stage based on information entropy. In this way, by quantifying the inherent uncertainty of each stage through information entropy, measuring the correlation strength between stages through mutual information, and finally calculating the uncertainty reduction amount in combination with the time length distribution range, the value of executing a stage in reducing uncertainty can be evaluated, thereby providing a quantitative basis for the subsequent ε-greedy strategy.

[0073] Further, the application adopts an ε-greedy strategy, and allocates and schedules resources in combination with the JCT priority queue and the uncertainty reduction priority queue. In this way, in a complex dynamic environment, the job can be quickly promoted, and long-term inaccuracy can be avoided by reducing uncertainty, thereby improving the overall scheduling efficiency of the composite LLM application.

[0074] Finally, the application assigns tasks to corresponding executors for execution, and repeats the above process until all jobs are completed. In this way, the scheduling goals of quickly promoting job progress and reducing uncertainty are effectively balanced, and the accuracy of resource allocation is improved.

[0075] Through the above technical solutions, the application fully considers the influence of chronic inflammation of patients with renal anemia on monitoring results, dynamically optimizes and adjusts the preset iron metabolism monitoring period by adjusting the monitoring influence coefficient, and obtains an optimized monitoring period. In this way, the accuracy of iron metabolism monitoring for renal anemia is improved.

[0076] As shown in Embodiment Two, Figure 3 Based on the same inventive concept as the method for scheduling a large language model application workload provided in Embodiment One, the embodiments of the application further provide a system for scheduling a large language model application workload, comprising: A DAG modeling module 11 is configured to model a composite large language model application as a directed acyclic graph comprising a regular stage, an LLM stage, and a dynamic stage when a job request of the composite large language model application arrives; A time length prediction module 12 is configured to model the execution correlation between stages in the directed acyclic graph through a Bayesian network, dynamically predict the time length distribution of an unfinished stage, and calibrate the time length estimate of the LLM stage in combination with the real-time batch size of the LLM executor; An uncertainty calculation module 13 is used to quantify the uncertainty reduction of each ready stage of scheduling based on information entropy; The resource scheduling module 14 is configured to allocate scheduling resources by adopting an ε-greedy strategy and combining a JCT priority queue with an uncertainty reduction priority queue; The output execution module 15 is used to dispatch the task to the corresponding executor for execution, and repeat the above process until all jobs are completed.

[0077] The DAG modeling module 11 is specifically used for: The conventional stage is used to execute non-LLM tasks, and its execution time fluctuation range does not exceed the preset threshold. The LLM stage is used to execute LLM reasoning tasks, and its execution time is affected by input characteristics, hardware load and batch processing strategy. The dynamic stage serves as a placeholder node, and the sub-stages it contains and the dependencies between sub-stages are dynamically generated by the output results of its predecessor LLM stage at runtime.

[0078] The duration prediction module 12 is specifically configured to: Discretize the duration distribution of each stage into a preset number of intervals, and use each stage as a variable in the Bayesian network; A Bayesian network is trained based on historical execution data to capture the execution correlation between stages and derive the conditional probability distribution of each stage's duration and its correlation. If there is a direct path between two stages in the Bayesian network, the two stages are considered to have execution correlation, and the duration distribution of the child stage is modeled using the probability distribution of the parent stage. Obtain the actual duration of the completed phase and map it to its corresponding discretized interval. Input the interval as evidence into the Bayesian network and output the posterior conditional probability distribution of the unfinished phase that has execution correlation with the phase. The duration estimate of the unfinished stage is updated according to the posterior conditional probability distribution.

[0079] Specifically, the “calibrating the estimated duration of the LLM phase in combination with the real-time batch size of the LLM executor” includes: Get the LLM phase duration recorded under the reference batch size and the corresponding average decoding delay; Get the target batch size of the current LLM executor and the corresponding average decoding delay; Use the following calibration formula to correct the estimated duration under the target batch size: ; in, is the target batch size The estimated duration of the next is the reference batch size The duration of the recording, is the target batch size average decoding delay of the lower record, is the reference batch size average decoding delay of the lower record.

[0080] The uncertainty calculation module 13 is specifically configured to: define corresponding random variables for the regular stage, the LLM stage and the dynamic stage, and calculate the information entropy of each stage based on Shannon entropy; determine a set of unscheduled stages related to the current ready stage; The mutual information formula is used to calculate the mutual information between the current ready stage X and the related unscheduled stage Y: ; Where p(x,y) is the joint probability of X and Y, and p(x) and p(y) are the respective marginal probabilities; Obtain the time length distribution range of the related unscheduled stage, and calculate the uncertainty reduction amount of scheduling the current ready stage based on the mutual information and the time length distribution range.

[0081] Specifically, the "defining corresponding random variables for the regular stage, the LLM stage and the dynamic stage, and calculating the information entropy of each stage based on Shannon entropy" includes: Define corresponding random variables for the regular stage, the LLM stage and the dynamic stage, and determine the probability distribution of the random variable value, and calculate the information entropy of the regular stage, the LLM stage and the dynamic stage based on the following Shannon entropy formula: ; Where, is the information entropy of each stage, is the probability of each possible value of the random variable.

[0082] Further, for the regular stage, the LLM stage and the dynamic stage, the calculation method of the information entropy is specifically as follows: Define whether to execute the regular stage as a binary random variable X, where X∈{0,1}, 1 represents execution, and 0 represents non-execution; Obtain the execution probability p and the non-execution probability 1-p of the regular stage; The information entropy of the regular stage is calculated using the following entropy calculation formula : ; Discretize the time length distribution of the LLM stage into k intervals, and consider non-execution as time length 0s, and define a discrete random variable containing k+1 values; Obtain the probability of non-execution of the LLM stage and the probability that the length falls into the ith interval (i = 1, 2, …, k); The information entropy of the LLM stage is calculated by using the following entropy calculation formula : ; The candidate sub-stage set of the dynamic stage and the candidate dependent edge set E are determined ; Whether the candidate sub-stage exists is defined as a binary random variable , wherein ∈ {0, 1}, 1 represents that the sub-stage exists, and 0 represents that the sub-stage does not exist; Whether the candidate dependent edge exists is defined as a binary random variable , wherein ∈ {0, 1}, 1 represents that the dependent edge exists, and 0 represents that the dependent edge does not exist; The entropy of each candidate sub-stage and the entropy of each candidate dependent edge are calculated ; The information entropy of the dynamic stage is calculated by using the following formula : .

[0083] Further, the “obtaining the length distribution range of the related unscheduled stage, and calculating the uncertainty reduction amount of scheduling the current ready stage based on the mutual information and the length distribution range” comprises: obtaining the evidence set E of the completed stage length and the length distribution range of the related unscheduled stage Y ; The uncertainty reduction amount is calculated by using the following formula: ; wherein represents the mutual information, is an unscheduled stage related to X, M is the total number of unscheduled stages related to X, and m is an index variable for traversing the unscheduled stages related to X, and the value range of m is 1 to M.

[0084] The resource scheduling module 14 is specifically configured to: all ready tasks are sorted according to the shortest remaining time first principle as a JCT priority queue; all ready tasks are sorted in descending order of uncertainty reduction amount as an uncertainty reduction priority queue; each time of scheduling, a task is selected from the uncertainty reduction priority queue with a preset probability ε, and a task is selected from the JCT priority queue with a probability of 1 / ε; When executing a task from the uncertainty reduction priority queue, some subtasks may be selectively executed.

[0085] The output execution module 15 is specifically configured to: Assign tasks to the corresponding executors for execution, and repeat the above process until all jobs are completed.

[0086] In summary, the embodiments of the present application have at least the following technical effects: Compared with the existing technology, this application first uses the DAG modeling module to model the composite large language model application as a directed acyclic graph including a regular stage, an LLM stage, and a dynamic stage when a job request for the composite large language model application arrives. This effectively solves the problem of structural uncertainty and provides a unified structured model for subsequent analysis, supporting subsequent uncertainty quantification and resource allocation decisions. Secondly, through the duration prediction module, the execution correlation between the stages in the directed acyclic graph is modeled through a Bayesian network, and the duration distribution of the unfinished stage is dynamically predicted. The duration estimate of the LLM stage is calibrated in combination with the real-time batch size of the LLM executor. The actual data of the completed stage is used as evidence to correct the duration estimate of the unfinished stage in real time, so that the prediction changes from a static initial value to a dynamically updated value, effectively solving the problem of time uncertainty. Next, the uncertainty calculation module quantifies the uncertainty reduction of each ready stage based on information entropy. The inherent uncertainty of each stage is quantified using information entropy, and the strength of the correlation between stages is measured using mutual information. Finally, the uncertainty reduction is calculated based on the duration distribution range. This allows the value of executing a stage in reducing uncertainty to be assessed, providing a quantitative basis for the subsequent ε-greedy strategy. Furthermore, the resource scheduling module employs the ε-greedy strategy, combining JCT priority queuing with uncertainty reduction priority queuing to allocate scheduling resources. This allows for rapid job progress in complex dynamic environments while also reducing uncertainty and avoiding long-term misalignment, improving the overall scheduling efficiency of composite LLM applications. Finally, the output execution module dispatches tasks to the corresponding executors for execution. This process is repeated until all jobs are completed, effectively balancing the scheduling goals of rapidly advancing jobs and reducing uncertainty, while improving the accuracy of resource allocation. This provides a complete solution for composite LLM applications that balances dynamic adaptability and scheduling efficiency, effectively improving resource utilization and job processing performance in complex scenarios.

[0087] Example 3, as Figure 4 As shown, an embodiment of the present invention provides an electronic device 200, including a memory 210, a processor 220, and a computer program 211 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 211, a scheduling method for a large language model application workload in embodiment one is implemented.

[0088] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, since various changes and modifications within the spirit of the application will become apparent to those skilled in the art from this detailed description.

[0089] A person skilled in the art would understand that embodiments of the present application can be provided as methods, systems, or computer program products. Thus, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0090] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 means for performing each of the one or more functions specified in the flowchart illustrations and / or block diagrams.

[0091] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 means for performing each of the one or more functions specified in the flowchart illustrations and / or block diagrams.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 means for performing each of the one or more functions specified in the flowchart illustrations and / or block diagrams.

[0093] Although preferred embodiments of the application have been described, a person of ordinary skill in the art can make additional changes and modifications to these embodiments once armed with the present disclosure.

[0094] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover modifications and variations of this application provided they come within the scope of the application and their equivalent technology.

Claims

1. A method for scheduling large language model application workloads, characterized in that: The method comprises: When a job request for a composite large language model application arrives, the composite large language model application is modeled as a directed acyclic graph including a regular phase, an LLM phase, and a dynamic phase; The execution correlation between the stages in the directed acyclic graph is modeled using a Bayesian network, the duration distribution of unfinished stages is dynamically predicted, and the duration estimate of the LLM stage is calibrated based on the real-time batch size of the LLM executor. The uncertainty reduction of each ready stage of scheduling is quantified based on information entropy; Adopting the ε-greedy strategy, combining JCT priority queue and uncertainty reduction priority queue to allocate scheduling resources; Assign tasks to the corresponding executors for execution, and repeat the above process until all jobs are completed.

2. The method according to claim 1, characterized in that The conventional stage is used to execute non-LLM tasks, and its execution time fluctuation range does not exceed the preset threshold. The LLM stage is used to execute LLM reasoning tasks, and its execution time is affected by input characteristics, hardware load and batch processing strategy. The dynamic stage serves as a placeholder node, and the sub-stages it contains and the dependencies between sub-stages are dynamically generated by the output results of its predecessor LLM stage at runtime.

3. The method according to claim 1, characterized in that The execution correlation between the stages in the directed acyclic graph is modeled using a Bayesian network, and the duration distribution of unfinished stages is dynamically predicted, including: Discretize the duration distribution of each stage into a preset number of intervals, and use each stage as a variable in the Bayesian network; A Bayesian network is trained based on historical execution data to capture the execution correlation between stages and derive the conditional probability distribution of each stage's duration and its correlation. If there is a direct path between two stages in the Bayesian network, the two stages are considered to have execution correlation, and the duration distribution of the child stage is modeled using the probability distribution of the parent stage. Obtain the actual duration of the completed phase and map it to its corresponding discretized interval. Input the interval as evidence into the Bayesian network and output the posterior conditional probability distribution of the unfinished phase that has execution correlation with the phase. The duration estimate of the unfinished stage is updated according to the posterior conditional probability distribution.

4. The method according to claim 1, wherein Calibrate the LLM phase duration estimate based on the LLM executor's real-time batch size, including: Get the LLM phase duration recorded under the reference batch size and the corresponding average decoding delay; Get the target batch size of the current LLM executor and the corresponding average decoding delay; Use the following calibration formula to correct the estimated duration under the target batch size: ； in, is the target batch size The estimated duration of the next is the reference batch size The duration of the recording, is the target batch size The average decoding delay recorded below, is the reference batch size The average decoding delay recorded below.

5. The method according to claim 1, wherein The uncertainty reduction of each ready stage of scheduling is quantified based on information entropy, including: Corresponding random variables are defined for the conventional stage, LLM stage and dynamic stage, and the information entropy of each stage is calculated based on Shannon entropy; Determine the set of unscheduled stages associated with the current ready stage; The mutual information between the current ready stage X and the related unscheduled stage Y is calculated using the following mutual information formula: ； Where p(x,y) is the joint probability of X and Y, and p(x) and p(y) are their respective marginal probabilities; A duration distribution range of the relevant unscheduled phase is obtained, and based on the mutual information and the duration distribution range, an uncertainty reduction amount of scheduling the current ready phase is calculated.

6. The method according to claim 5, characterized in that Corresponding random variables are defined for the conventional stage, LLM stage, and dynamic stage, and the information entropy of each stage is calculated based on Shannon entropy, including: For the conventional stage, LLM stage, and dynamic stage, corresponding random variables are defined respectively, and the probability distribution of the values ​​of the random variables is determined. The information entropy of the conventional stage, LLM stage, and dynamic stage is calculated based on the following Shannon entropy formula: ； in, is the information entropy of each stage, is the probability of each possible value of the random variable.

7. The method according to claim 5, characterized in that Obtaining a duration distribution range of the relevant unscheduled phases, and calculating an uncertainty reduction amount for scheduling the current ready phase based on the mutual information and the duration distribution range, including: Obtain the evidence set E of the completed phase duration and the duration distribution range of the related unscheduled phase Y ; The uncertainty reduction is calculated using the following formula: ； in, represents mutual information, is the unscheduled phase associated with X, M is the total number of unscheduled phases associated with X, and m is the index variable for traversing the unscheduled phases associated with X. The value of m ranges from 1 to M.

8. The method according to claim 1, characterized in that Adopt the ε-greedy strategy, combining the JCT priority queue and the uncertainty reduction priority queue to allocate scheduling resources, including: All ready tasks are sorted by the shortest remaining time priority principle as the JCT priority queue; Sort all ready tasks in descending order of uncertainty reduction as an uncertainty reduction priority queue; Each time a task is scheduled, a task is selected from the uncertainty reduction priority queue with a preset probability ε, and a task is selected from the JCT priority queue with a probability 1ε; When executing a task from the uncertainty reduction priority queue, some subtasks may be selectively executed.

9. A scheduling system for large language model application workloads, characterized in that: Used to perform the method according to any one of claims 1 to 8, comprising: A DAG modeling module is configured to, when a job request for a composite large language model application arrives, model the composite large language model application as a directed acyclic graph comprising a regular phase, an LLM phase, and a dynamic phase; A duration prediction module is used to model the execution correlation between the stages in the directed acyclic graph through a Bayesian network, dynamically predict the duration distribution of unfinished stages, and calibrate the duration estimate of the LLM stage based on the real-time batch size of the LLM executor; Uncertainty calculation module, used to quantify the uncertainty reduction of each ready stage of scheduling based on information entropy; The resource scheduling module is used to allocate scheduling resources using the ε-greedy strategy, combining the JCT priority queue and the uncertainty reduction priority queue; The output execution module is used to dispatch tasks to the corresponding executors for execution, and the above process is repeated until all jobs are completed.

10. An electronic device, characterized in that: include: Memory for storing computer software programs; A processor is used to read and execute the computer software program, thereby implementing the scheduling method for a large language model application workload as described in any one of claims 1-8.

Images