Scheduling method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a scheduling method and device, electronic equipment, a storage medium and a program product, and is applied to a scheduling system.The method comprises the steps that a target request is obtained, and the target request comprises input information to be input into a preset model; according to the input information, predicting an index parameter corresponding to each instance scheduling scheme in a plurality of preset instance scheduling schemes; the index parameter corresponding to the instance scheduling scheme is used for representing the performance of scheduling the instance according to the instance scheduling scheme to execute the target task; according to the index parameter corresponding to each instance scheduling scheme, determining a target scheduling scheme from a plurality of preset instance scheduling schemes; and according to the target scheduling scheme, the target task is completed through the scheduling instance. According to the embodiment of the invention, the system can adaptively adopt the instance scheduling scheme adaptive to the target request in the reasoning process, and the limitation that the scheduling system only supports one instance scheduling scheme is overcome, so that efficient reasoning of the target request is realized, and the performance of the whole system is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to a scheduling method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the continuous development and popularization of deep learning technology, efficient large-scale model inference is essential for supporting various industries to make good use of large-scale models.

[0003] In related technologies, model inference scheduling systems typically support only one instance scheduling scheme during the inference process. This results in large models only being able to use one instance scheduling scheme during inference, that is, executing tasks by scheduling instances according to one instance scheduling scheme. Consequently, it is difficult to adaptively adopt a scheduling scheme that matches the target request during the inference task. Summary of the Invention

[0004] The purpose of this application is to provide a scheduling method, apparatus, electronic device, storage medium, and program product. These enable the adaptive adoption of an instance scheduling scheme adapted to the target request during the execution of a target task, thereby improving processing efficiency.

[0005] To achieve the above objectives, the embodiments of this application provide the following solutions:

[0006] Firstly, this application provides a scheduling method applied to a scheduling system. The method includes: obtaining a target request, the target request being used to request the execution of a target task using a preset model, the target request including input information to be input to the preset model; predicting, based on the input information, the indicator parameters corresponding to each of a plurality of preset instance scheduling schemes; the instance scheduling scheme representing the number of tasks and task stages that an instance of the preset model can execute in a target time period; the indicator parameters corresponding to the instance scheduling scheme being used to characterize the performance of scheduling instances to execute the target task according to the instance scheduling scheme; determining a target scheduling scheme from the plurality of preset instance scheduling schemes based on the indicator parameters corresponding to each instance scheduling scheme; and completing the target task by scheduling instances according to the target scheduling scheme.

[0007] This scheduling system can be applied to scenarios involving large-scale model inference. It supports multiple preset instance scheduling schemes, such as a first, second, and third instance scheduling scheme, meaning it integrates various schemes to suit different target requests (e.g., input information of varying lengths). This eliminates the need for multiple independent systems or tools to handle different target requests separately, demonstrating good versatility, adaptability, and flexibility. Based on this, by using the performance parameters corresponding to each of the preset instance scheduling schemes, the system can determine the performance of each scheme when executing a target request. Then, using the performance parameters of each scheme, it determines the target inference schedule and completes the target task by scheduling instances. In this way, the system can adaptively adopt an instance scheduling scheme that matches the target request during inference, overcoming the limitation of related technologies where scheduling systems only support one instance scheduling scheme. This achieves efficient inference for target requests and improves the overall system performance.

[0008] In one possible implementation, the metric parameters include a combined latency value and / or a combined throughput value; the combined latency value represents the total latency of the target task's task phases, and the combined throughput value represents the total throughput of the target task's task phases. Thus, because the metric parameters include a combined latency value and / or a combined throughput value, they can more comprehensively reflect the performance of scheduling instances to execute the target task according to the instance scheduling scheme.

[0009] In one possible implementation, determining the target scheduling scheme from multiple preset instance scheduling schemes based on the index parameters corresponding to each instance scheduling scheme includes: determining the instance scheduling scheme with the smallest overall latency or the largest overall throughput among the multiple preset instance scheduling schemes as the target scheduling scheme.

[0010] The target scheduling scheme is determined by selecting the instance scheduling scheme with the lowest overall latency among multiple preset instance scheduling schemes. This reduces the total execution time latency when scheduling instances to execute the target task based on the instance scheduling scheme with the lowest overall latency, resulting in good performance. This embodiment can be applied to scenarios where users need faster task result feedback. Simultaneously, the target scheduling scheme is determined by selecting the instance scheduling scheme with the highest overall throughput among multiple preset instance scheduling schemes. This allows the system to process more tasks or data per unit time when scheduling instances to execute the target task based on the instance scheduling scheme with the highest overall throughput, resulting in good performance. This embodiment can be applied to scenarios requiring large-scale data processing and high-concurrency requests.

[0011] In one possible implementation, based on the input information, the indicator parameters corresponding to each of the multiple preset instance scheduling schemes are predicted, including: obtaining relevant information of historical tasks; predicting the target information length based on the input information and the relevant information of historical tasks, where the target information length is the length of the output information corresponding to the target task; and predicting the indicator parameters corresponding to each instance scheduling scheme based on the length of the input information, the target information length, and the target performance parameters. The target performance parameters are used to characterize the performance of the instance in executing the target task.

[0012] By leveraging relevant information from historical tasks, we can gain a relatively accurate understanding of their characteristics and requirements, better predict the length of target information, and thus more accurately predict the indicator parameters corresponding to different instance scheduling schemes. This enables the scheduling system to select a more suitable scheduling scheme for executing the target task, thereby achieving better task execution results and improving the efficiency and performance of large model inference.

[0013] In one possible implementation, before predicting the metric parameters corresponding to each instance scheduling scheme based on the length of the input information, the length of the target information, and the target performance parameters, the method further includes estimating the target performance parameters based on relevant information from historical tasks. In this way, by using relevant information from historical tasks, the first and second performance parameters are accurately estimated, thereby enabling the accurate calculation of the metric parameters corresponding to each instance scheduling scheme subsequently.

[0014] In one possible implementation, the indicator parameters include a comprehensive latency value, which represents the total latency of each task stage of the target task; the target performance parameters include performance parameters corresponding to each task stage of the target task; based on the length of the input information, the length of the target information, and the target performance parameters, the indicator parameters corresponding to each instance scheduling scheme are predicted as follows: for each instance scheduling scheme, based on the length of the input information, the length of the target information, and the performance parameters corresponding to each task stage of the target task, the stage latency value of each task stage of the target task is predicted; based on the stage latency value of each task stage of the target task, the comprehensive latency value corresponding to the instance scheduling scheme is obtained.

[0015] By combining the length of the input information, the length of the target information, and the performance parameters corresponding to each stage of the target task to predict the metric parameters of each instance scheduling scheme, we can more comprehensively consider the initial scale of the target task (such as its length), the corresponding output scale (such as the length of the target information), and the performance of the instance at different stages of task processing. This makes the prediction results more accurate and better able to predict the performance of different instance scheduling schemes in actual target task processing.

[0016] In one possible implementation, the task phase includes a prefill phase and a decode phase; the target performance parameters include a first performance parameter and a second performance parameter, the first performance parameter being used to characterize the average latency of the instance when executing the prefill phase of the target task, and the second performance parameter being used to characterize the average latency of the instance when executing the decode phase of the target task; the phase latency value of the prefill phase of the target task is directly proportional to the first performance parameter; the phase latency value of the decode phase of the target task is directly proportional to the second performance parameter.

[0017] By establishing the direct proportionality between the prefill stage latency and a first performance parameter, and the direct proportionality between the decode stage latency and a second performance parameter, the direct influence of the first and second performance parameters on the respective task stage latency values ​​is demonstrated. For example, a larger first performance parameter results in a larger prefill stage latency. Conversely, a larger second performance parameter results in a larger decode stage latency.

[0018] In one possible implementation, the instance scheduling scheme includes a first instance scheduling scheme; the first instance scheduling scheme indicates that an instance executes either the prefill phase or the decode phase of the same task within a target time period; under the first instance scheduling scheme, the phase delay value of the prefill phase of the target task is a first P delay value, and the phase delay value of the decode phase is a first D delay value; the first P delay value conforms to the following relationship with the length of the input information and the first performance parameter: In the formula, Let P be the first time delay value, α be the first adjustment coefficient, and N be the second time delay value. p T represents the length of the input information. p The first performance parameter is defined as follows; the first D-delay value, the target information length, and the second performance parameter conform to the following relationship: In the formula, T is the first D delay value. wait To wait for scheduling delay, T represents the length of the target information. d This is the second performance parameter.

[0019] For the first instance scheduling scheme, by calculating the first P-latency value of the prefill stage and the first D-latency value of the decode stage of the target task respectively, the time delay of each stage of the target task execution can be accurately quantified. This allows the scheduling system to accurately obtain the latency of different stages of the target task execution, providing specific data for subsequent scheduling decisions and performance optimization.

[0020] In one possible implementation, the instance scheduling scheme includes a second instance scheduling scheme; the second instance scheduling scheme indicates that an instance executes the prefill and decode phases of different tasks within a target time period; under the second instance scheduling scheme, the stage delay value of the prefill phase of the target task is a second P delay value, and the stage delay value of the decode phase is a second D delay value; the second P delay value conforms to the following relationship with the length of the input information and the first performance parameter: In the formula, The second P delay value, β is the second adjustment coefficient, and T p The primary performance parameter is... The length of the prefill block. The second D-delay value is determined based on the length of the input information and the preset block size; the second D-delay value conforms to the following relationship with the target information length and the second performance parameter: In the formula, This is the second D delay value. T represents the length of the target information. d This is the second performance parameter.

[0021] The above calculation method fully considers the parallel processing of different task stages (i.e., the prefill stage and the decode stage) and the reasonable calculation of latency, improves the accuracy and rationality of the calculation of the second P latency value and the second D latency value, avoids errors caused by repeated calculations, and makes the whole calculation process more accurate, which meets the actual task execution and latency value calculation requirements of different stages.

[0022] In one possible implementation, the instance scheduling scheme includes a third instance scheduling scheme; the third instance scheduling scheme indicates that multiple instances execute the prefill and decode phases of different tasks respectively within a target time period; under the third instance scheduling scheme, the phase delay value of the prefill phase of the target task is the third P delay value, and the phase delay value of the decode phase is the third D delay value; the third P delay value conforms to the following relationship with the length of the input information and the first performance parameter: In the formula, The third P-delay value, γ is the third adjustment coefficient, and N p The length of the input information, T is the KV multiplexing length of the input information. p The first performance parameter is defined as follows; the third D-delay value, the target information length, and the second performance parameter conform to the following relationship: In the formula, For the third D delay value, T wait To wait for scheduling delay, For KV transmission delay, It was determined based on relevant information from historical missions. For the length of the target information, The length of the prefill block. T is determined based on the length of the input information and the preset block size. d This is the second performance parameter.

[0023] The above calculation method fully considers the parallel processing of different task stages (i.e., the prefill stage and the decode stage) in different instances and the reasonable calculation of latency, improves the accuracy and rationality of the calculation of the third P latency value and the third D latency value, makes the whole calculation process more accurate, and thus determines the target scheduling scheme more accurately.

[0024] In one possible implementation, completing the target task through scheduling instances according to the target scheduling scheme includes: predicting the target hyperparameters corresponding to the target scheduling scheme; configuring instances of preset models according to the target scheduling scheme and the target hyperparameters to obtain target instances; using the target instances to execute the target task based on the target hyperparameters and according to the target scheduling scheme; and executing the target task by scheduling the target instances.

[0025] By using the target scheduling scheme and target hyperparameter configuration of the preset model instance, the target instance is obtained. This allows the preset model instance to dynamically adjust the scheduling scheme and model scheduling parameters for executing the target task according to the target task, thereby achieving a high degree of adaptability between the target instance and the target task and improving the efficiency and accuracy of the target task execution.

[0026] In one possible implementation, predicting the target hyperparameters corresponding to the target scheduling scheme includes: obtaining the current system load and relevant information of historical tasks of the scheduling system; predicting the target information length based on the input information and relevant information of historical tasks, wherein the target information length is the length of the output information corresponding to the target task; and predicting the target hyperparameters corresponding to the target scheduling scheme based on the length of the input information, the target information length, and the current system load.

[0027] By combining the length of the input information, the length of the target information, and the current system load, the target hyperparameters corresponding to the target scheduling scheme are predicted, achieving more refined prediction of the scheduling scheme and obtaining more accurate target hyperparameters. More accurate prediction of target hyperparameters enables the maintenance of high resource utilization efficiency and performance levels during the actual execution of target tasks, improving system flexibility and reliability. Furthermore, predicting corresponding target hyperparameters for different target requests allows for dynamic adjustment of the target hyperparameters, making them more suitable for different target requests.

[0028] In one possible implementation, the relevant information about historical tasks includes the lengths of historical input and output information corresponding to the historical tasks, the temporal distribution of the lengths of historical input and output information, historical performance parameters, and the distribution information of historical key-value caches (KV caches). The historical performance parameters characterize the performance of instances executing historical tasks. In this way, the relevant information about historical tasks covers multiple aspects of data, providing rich information and offering multi-dimensional references for predicting the performance parameters corresponding to each instance scheduling scheme, thus making the predictions more accurate.

[0029] Secondly, this application provides a scheduling device applied to a scheduling system. The device includes: an acquisition module for acquiring a target request, the target request requesting the execution of a target task using a preset model, the target request including input information to be input to the preset model; a prediction module for predicting, based on the input information, the indicator parameters corresponding to each of a plurality of preset instance scheduling schemes; the instance scheduling scheme represents the number of tasks and task stages that an instance of the preset model can execute in a target time period; the indicator parameters corresponding to the instance scheduling scheme characterize the performance of scheduling instances to execute the target task according to the instance scheduling scheme; a determination module for determining a target scheduling scheme from a plurality of preset instance scheduling schemes based on the indicator parameters corresponding to each instance scheduling scheme; and a scheduling module for completing the target task by scheduling instances according to the target scheduling scheme.

[0030] Thirdly, this application provides an electronic device, including: a processor and a memory for storing processor-executable instructions; when the processor is configured to execute instructions, the electronic device implements the scheduling method described above.

[0031] Fourthly, this application provides a storage medium storing computer program instructions thereon; when the computer program instructions are executed by an electronic device, the electronic device implements the scheduling method described above.

[0032] Fifthly, this application provides a program product including a computer-readable storage medium storing a computer program, which, when at least one processor executes the computer program, causes at least one processor to execute the scheduling method described above. Attached Figure Description

[0033] Figure 1 A flowchart illustrating the application scenarios provided in this application embodiment;

[0034] Figure 2 This is a schematic diagram corresponding to a first instance scheduling scheme provided in an embodiment of this application;

[0035] Figure 3 This is a schematic diagram corresponding to a second instance scheduling scheme provided in an embodiment of this application;

[0036] Figure 4 This is a schematic diagram corresponding to a third instance scheduling scheme provided in an embodiment of this application;

[0037] Figure 5 A schematic diagram of the structure of a scheduling system provided in an embodiment of this application;

[0038] Figure 6 This is a schematic diagram of another scheduling system provided in an embodiment of this application;

[0039] Figure 7 This is a schematic diagram of another scheduling system provided in an embodiment of this application;

[0040] Figure 8 A flowchart illustrating a scheduling method provided in an embodiment of this application;

[0041] Figure 9 This is a flowchart illustrating how a target task is completed through a scheduling instance based on a target scheduling scheme, as provided in an embodiment of this application.

[0042] Figure 10 This is a schematic diagram illustrating another process for completing a target task through a scheduling instance based on a target scheduling scheme, as provided in an embodiment of this application.

[0043] Figure 11 This is a schematic diagram illustrating another process for completing a target task through a scheduling instance based on a target scheduling scheme, as provided in an embodiment of this application.

[0044] Figure 12 This is a schematic diagram of the structure of a scheduling device provided in an embodiment of this application;

[0045] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0046] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0047] In the following description of the embodiments of this application, unless otherwise stated, "a plurality of" means two or more.

[0048] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0049] The following provides an exemplary description of the application scenarios of the embodiments of this application.

[0050] With the continuous development and popularization of deep learning technology, efficient large-model inference is crucial for supporting the effective use of large models across various industries. The scheduling scheme provided in this application can be applied to scenarios involving large-model inference. The task phase in the large-model inference process consists of two stages: the prefill stage and the decode stage. The prefill stage involves the input information (requesting the use of the large model to perform the task) entering the large model for the first stage of inference and outputting a token (the token output by the prefill stage, hereinafter referred to as prefill output information). The prefill stage is computationally intensive. The decode stage involves inputting the token generated in the previous stage (e.g., prefill output information) into the large model for inference to generate a new token. This step is repeated multiple times to generate the final output information (hereinafter referred to as the output information corresponding to the input information). The decode stage is memory-intensive. Figure 1 As shown, the input information of the request is input to the large model, such as "What's the weather like today?". The large model (specifically, an instance of the scheduling large model) processes the request in the prefill phase and outputs prefill information. Then, the large model processes the prefill output information in the decode phase to generate a new token, and repeats this step multiple times to obtain the output information corresponding to the input information, such as "Sunny turning cloudy".

[0051] Based on the two phases of large-model inference, large-model inference has multiple instance scheduling schemes. An instance scheduling scheme represents the number of tasks and task phases that instances of a predefined model can execute within a target time period. Multiple instance scheduling schemes can include several of the following: a first instance scheduling scheme, a second instance scheduling scheme, and a third instance scheduling scheme. Specifically, a first instance scheduling scheme represents an instance executing the prefill or decode phase of the same task within the target time period (e.g., a PD selection scheduling scheme). A second instance scheduling scheme represents an instance executing the prefill and decode phases of different tasks within the target time period (e.g., a PD fusion scheduling scheme). A third instance scheduling scheme represents multiple instances executing the prefill and decode phases of different tasks separately within the target time period (e.g., a PD separation scheduling scheme).

[0052] Understandably, the first or second instance scheduling scheme executes the prefill and decode phases of a task by scheduling one instance. The third instance scheduling scheme executes the prefill and decode phases of a task by scheduling multiple (e.g., two) instances. Specifically, when an instance executes the prefill phase of a task, input information enters the prefill phase, and the instance processes the input information to obtain the prefill output information. When an instance executes the decode phase of a task, the information entering the decode phase is the decode input information. For example, the decode input information can be the prefill output information (e.g., the token generated in the previous phase mentioned above), or it can be the decode loop input information. The decode loop input information refers to the information that enters each round when the instance performs multiple rounds of reasoning on the information entering the decode phase (e.g., the information that enters the decode phase each time the above step is repeated multiple times).

[0053] For example, the first instance scheduling scheme involves scheduling an instance to execute both the prefill and decode phases of a task, but at any given time, the instance either executes the prefill phase of the task (e.g., task A) or the decode phase of the task. For example, as... Figure 2As shown, this instance processes the input information (e.g., the input information corresponding to task A) in the prefill phase at time t1, and processes the prefill output information (e.g., the prefill output information corresponding to task A) in the decode phase at time t2. Time t2 is after time t1. The input information corresponding to the task is the input information of the request corresponding to that task; the prefill output information corresponding to the task represents the information output from the prefill phase when the instance executes that task.

[0054] For example, the second instance scheduling scheme involves scheduling one instance to execute the prefill and decode phases of a task, and within the same time period, this instance can simultaneously execute the prefill and decode phases of different tasks (e.g., task A, task B, and task C). For example, as... Figure 3 As shown, this instance simultaneously executes the prefill phase of task B and the decode phase of task A at time t1. Specifically, while processing the input information (e.g., the input information corresponding to task B) in the prefill phase, the instance also processes the prefill output information (e.g., the prefill output information corresponding to task A) in the decode phase. When executing tasks according to the second instance scheduling scheme, the instance's execution of the task's prefill phase involves dividing the input information into multiple blocks, and the instance processes each block of the input information. In this way, if the instance receives input information for a request that needs to be processed first, the instance can truncate the currently processed input information (e.g., the truncated input information at time t1) and process the input information for the request that needs to be processed first. Similarly, this instance can also simultaneously execute the prefill and decode phases of different tasks at time t2. For example, the input information truncated at time t1 can continue to be processed in the prefill phase at time t2, and at the same time, the prefill output information (e.g., the prefill output information corresponding to task C) can also be processed in the decode phase at time t2.

[0055] For example, the third instance scheduling scheme completes the prefill and decode phases by scheduling at least two instances (e.g., instance A and instance B), with instance A completing the prefill phase and instance B completing the decode phase. For example, as... Figure 4As shown, at time t1, instance A processes the input information (e.g., the input information corresponding to task A) in the prefill phase, and the generated prefill output information (e.g., the prefill output information corresponding to task A) is stored in the key-value cache (KV cache) and transmitted to instance B. At time t2, instance B can simultaneously process the prefill output information (e.g., the prefill output information corresponding to task A) and the decode loop input information (e.g., the decode loop input information corresponding to task B) in the decode phase. The decode loop input information corresponding to the task represents the decode loop input information when the instance enters the decode phase during the execution of that task.

[0056] Based on the above-mentioned instance scheduling schemes, each scheme is suitable for different inference scenarios. Specifically: the first instance scheduling scheme (e.g., the PD selection scheduling scheme) is suitable for inference scenarios where the request is a short input sequence (e.g., the length of the requested input information is <1K). The second instance scheduling scheme (e.g., the PD fusion scheduling scheme) is suitable for inference scenarios where the request is a medium-length input sequence (e.g., the length of the requested input information is >1K). The third instance scheduling scheme (e.g., the PD separation scheduling scheme) is suitable for inference scenarios where the request is a medium-length input (e.g., the length of the requested input information is between 1K and 5K), and the corresponding output information is a long output sequence (e.g., the length of the output information is >1K).

[0057] Currently, model inference scheduling systems typically support only one instance scheduling scheme during the inference process. For example, tools used for text generation inference (TGI), a single language model (vLLM), Mooncake, Splitwise, or DistServ can only use one instance scheduling scheme during inference, making it difficult to adaptively adopt a scheduling scheme that matches the target request during the inference task.

[0058] In view of this, embodiments of this application provide a scheduling method applied to a scheduling system. This scheduling method selects a target scheduling scheme adapted to the target request from a variety of preset instance scheduling schemes, and completes the target task corresponding to the target request by scheduling instances according to the target scheduling scheme. Thus, by selecting the most suitable scheduling scheme for the target request from a variety of preset instance scheduling schemes supported by the scheduling system, the system's flexibility, adaptability, and versatility are improved. This allows the scheduling system to adaptively adopt an instance scheduling scheme adapted to the target request during the execution of the target task, enabling different tasks to use the optimal scheduling scheme, ensuring full utilization of instances, reducing waiting time, and improving processing efficiency.

[0059] In some embodiments, the scheduling method provided in this application is applied to a scheduling system. First, a target request is obtained, which requests the execution of a target task using a preset model. The target request includes input information to be input to the preset model. Second, based on the input information, indicator parameters corresponding to each of a variety of preset instance scheduling schemes are predicted. An instance scheduling scheme represents the number of tasks and task stages that an instance of the preset model can execute within a target time period. The indicator parameters corresponding to the instance scheduling scheme characterize the performance of scheduling instances to execute the target task according to the instance scheduling scheme. Then, based on the indicator parameters corresponding to each instance scheduling scheme, a target scheduling scheme is determined from the variety of preset instance scheduling schemes. Finally, the target task is completed by scheduling instances according to the target scheduling scheme.

[0060] In this embodiment, the scheduling system can be applied to scenarios involving large-scale model inference. This system supports multiple preset instance scheduling schemes, such as PD selection scheduling, PD fusion scheduling, and PD separation scheduling. In other words, the system integrates various instance scheduling schemes, making it suitable for different target requests (e.g., input information of varying lengths). It eliminates the need for multiple independent systems or tools to process different target requests separately, demonstrating good versatility, adaptability, and flexibility. Based on this, by using the performance parameters corresponding to each of the preset instance scheduling schemes, the performance of each scheme in executing the target request can be determined. Then, the target inference schedule is determined using the performance parameters of each scheme, and the target inference schedule is completed by scheduling instances according to the target inference schedule. This allows the system to adaptively adopt an instance scheduling scheme adapted to the target request during inference, overcoming the limitation of related technologies where scheduling systems only support one instance scheduling scheme. This achieves efficient inference of the target request and improves the overall system performance.

[0061] The system architecture of the embodiments of this application will be described exemplarily below.

[0062] like Figure 5 and Figure 6 As shown in the figure, this application provides a scheduling system, which includes a global coordinator.

[0063] The global coordinator is used to obtain target requests, which are used to request the execution of target tasks using a preset model. The target request includes input information to be input to the preset model.

[0064] The global coordinator includes a global simulator. The global simulator predicts the metric parameters for each of several preset instance scheduling schemes based on input information. An instance scheduling scheme represents the number of tasks and task stages that instances in a preset model can execute within a target time period. The metric parameters corresponding to the instance scheduling scheme characterize the performance of scheduling instances to execute the target task according to the scheme.

[0065] For example, the global simulator is a trained prediction model that can predict the metric parameters corresponding to each of a variety of preset instance scheduling schemes based on the input information.

[0066] In some embodiments, the global coordinator further includes a global scheduler. The global scheduler is used to determine a target scheduling scheme from a variety of preset instance scheduling schemes based on the metric parameters corresponding to each instance scheduling scheme. The global scheduler is also used to complete the target task by scheduling instances according to the target scheduling scheme.

[0067] For example, the global scheduler and the global simulator can run asynchronously. For instance, the global scheduler and the global simulator can each perform their own tasks within the same time period.

[0068] In one implementation, the global scheduler completes the target task by scheduling instances according to the target scheduling scheme. This includes: the global scheduler configures instances of a preset model based on the target scheduling scheme and target hyperparameters to obtain target instances. The target instances are then used to execute the target task according to the target scheduling scheme based on the target hyperparameters. The global scheduler completes the target task by scheduling these target instances.

[0069] For example, the target hyperparameters are related to the input information, the target information length, and the current system load. The target information length is used to represent the length of the output information corresponding to the target task.

[0070] Optionally, the global scheduler configures instances of a preset model based on the target scheduling scheme and target hyperparameters to obtain target instances. This includes: the global scheduler determining the number of instances corresponding to the target scheduling scheme and selecting instances from the instance pool corresponding to that number of instances. The global scheduler then configures instances corresponding to the number of instances corresponding to the target scheduling scheme based on the target scheduling scheme and target hyperparameters to obtain target instances.

[0071] For example, there can be multiple instances of the preset model. If the target scheduling scheme is a third instance scheduling scheme, then the number of target instances is at least two (e.g., instance A and instance B). Instance A can be scheduled to execute the prefill or decode phase of the target task according to the third instance scheduling scheme, and instance B can also be scheduled to execute the prefill or decode phase of the target task according to the third instance scheduling scheme. If the target scheduling scheme is a first instance scheduling scheme or a second instance scheduling scheme, then the number of target instances is one (e.g., instance A or instance B). Instance A or instance B can be scheduled to execute the prefill or decode phase of the target task according to the first instance scheduling scheme or the second instance scheduling scheme. When there are multiple instances of the preset model, this scheduling system can be applied to cluster instance scheduling scenarios for large model inference, and multiple instances can be deployed on one or more electronic devices.

[0072] For example, if there are multiple instances of the preset model, these multiple instances form an instance pool. The global scheduler is also used to manage the instance pool.

[0073] Optionally, the global scheduler configures instances of a preset model based on the target scheduling scheme and target hyperparameters to obtain the target instance. This includes: the global scheduler selecting an instance corresponding to the target scheduling scheme from the pre-configured instances in the instance pool; and the global scheduler configuring target hyperparameters for the instance corresponding to the target scheduling scheme to obtain the target instance. Here, the pre-configured instance represents an instance capable of executing the target task according to the instance scheduling scheme. The instance corresponding to the target scheduling scheme represents an instance capable of executing the target task according to the target scheduling scheme.

[0074] In another implementation, the global scheduler configures an instance of the preset model based on the target scheduling scheme and target hyperparameters. After obtaining the target instance, the global scheduler is also used to put the input information into the target instance.

[0075] In other embodiments, the global simulator is also used to predict the target hyperparameters corresponding to the target scheduling scheme.

[0076] In other embodiments, the scheduling system further includes a local scheduler A. Local scheduler A is used to determine a target scheduling scheme from a variety of preset instance scheduling schemes (e.g., a first instance scheduling scheme and a second instance scheduling scheme) based on the indicator parameters corresponding to each instance scheduling scheme. Local scheduler A is also used to execute the target task by scheduling target instances according to the target scheduling scheme.

[0077] For example, when local scheduler A executes a target task by scheduling a target instance, it is specifically responsible for batch scheduling (batch scheduling means processing multiple input information as a batch) and KV cache management in the prefill and / or decode stages.

[0078] In other embodiments, the scheduling system also includes an inference engine responsible for inference and KV cache sending during the prefill phase; and / or responsible for inference and KV cache receiving during the decode phase.

[0079] In some embodiments, the global coordinator also includes a global collector. The global collector is used to obtain relevant information about historical tasks. This information includes the lengths of historical input and output information corresponding to the historical task, the temporal distribution of the lengths of the historical input and output information, historical performance parameters, and historical KVcache distribution information. Historical performance parameters are used to characterize the performance of instances executing historical tasks.

[0080] In other embodiments, the global collector is also used to update information related to historical tasks.

[0081] In other embodiments, the local scheduler A includes a local collector A. The local collector A is used to obtain relevant information about the current task. This relevant information includes the current performance parameters and network transmission efficiency corresponding to the target task. The local collector A is also used to feed back the relevant information about the current task to the global collector, so that the global collector updates the relevant information about historical tasks.

[0082] In other embodiments, the scheduling system also includes a memory pool. The memory pool includes transmission tools and a link management unit, wherein the link management unit is responsible for managing the transmission links of the KV cache.

[0083] In other embodiments, the global coordinator also includes an application programming interface (API). The API is responsible for providing requests (such as target requests) to the scheduling system and providing responses to target requests.

[0084] In other embodiments, the global coordinator also includes a resource management module. The resource management module is used to manage the resources of the scheduling system.

[0085] In other embodiments, the global coordinator also includes a request-response module. The request-response module is used to process the request-response stream.

[0086] In other embodiments, the scheduling system further includes an instance deployment unit. The instance deployment unit is used to deploy instances of a preset model.

[0087] This application provides a scheduling system applicable to large-scale model inference scenarios. Specifically, it can be applied to cluster instance scheduling scenarios for large-scale model inference. This scheduling system is an adaptive scheduling framework for various requests, supporting multiple preset instance scheduling schemes, such as PD selection scheduling schemes, PD fusion scheduling schemes, and PD separation scheduling schemes. In other words, the system integrates multiple instance scheduling schemes, making it suitable for various target requests without requiring multiple independent systems or tools to handle different target requests separately. It exhibits good versatility, adaptability, and flexibility. Furthermore, this scheduling system can adaptively adjust scheduling parameters (i.e., target hyperparameters) to match the target request, improving the system's ability to handle diverse tasks.

[0088] like Figure 7 As shown in the embodiment of this application, another scheduling system is provided, which includes a local scheduler B.

[0089] Local scheduler B is used to acquire target requests, which request the execution of a target task using a preset model. The target request includes input information to be fed into the preset model. Local scheduler B includes a local simulator. The local simulator is used to predict the metric parameters corresponding to each of several preset instance scheduling schemes (e.g., PD selection scheduling scheme and PD fusion scheduling scheme) based on the input information. Local scheduler B is also used to determine the target scheduling scheme from among the several preset instance scheduling schemes based on the metric parameters corresponding to each instance scheduling scheme. And according to the target scheduling scheme, it completes the target task by scheduling instances.

[0090] For example, the local scheduler B and the local emulator can run asynchronously. For instance, the local scheduler B and the local emulator can each perform their own tasks within the same time period.

[0091] For example, the local simulator is a trained prediction model that can predict the metric parameters corresponding to each of a variety of preset instance scheduling schemes based on the input information.

[0092] In one implementation, the local scheduler B completes the target task by scheduling an instance according to the target scheduling scheme, including: the local scheduler B completes the target task by scheduling the target instance.

[0093] For example, the preset model has one instance. This scheduling system can be applied to scenarios involving the scheduling of a single instance of a large model. In this way, the number of target instances is also one, and the scheduling system can still support multiple preset instance scheduling schemes, thereby enabling the scheduling system to adaptively adjust to the scheduling scheme that matches the target request and improve system processing efficiency.

[0094] In other embodiments, the local simulator is also used to predict the target hyperparameters corresponding to the target scheduling scheme.

[0095] For example, when the local scheduler B executes the target task by scheduling the target instance, it is specifically responsible for batch scheduling and KV cache management in the prefill and decode phases.

[0096] In other embodiments, the local simulator B also includes a local collector B. The local collector B is used to obtain relevant information about historical tasks.

[0097] In other embodiments, the local collector B is also used to obtain relevant information about the current task and update relevant information about historical tasks based on the relevant information about the current task.

[0098] In other embodiments, the scheduling system may also include one or more of the inference engine, memory pool, API, request-response module, resource management module, and instance deployment unit described above, etc.

[0099] In this embodiment, the scheduling system is an adaptive scheduling framework for multiple requests. It is mainly applied to the scenario of single instance scheduling of large models and can support multiple preset instance scheduling schemes, such as PD selection scheduling scheme and PD fusion scheduling scheme. In other words, the system integrates multiple instance scheduling schemes, which can be applied to various different target requests. It has good adaptability and flexibility, and improves the system's ability to handle diverse tasks.

[0100] The aforementioned scheduling system can be deployed on one or more electronic devices. These electronic devices can be mobile phones, tablets, handheld computers, personal computers (PCs), cellular phones, personal digital assistants (PDAs), wearable devices (such as smartwatches and smart bracelets), smart home devices (such as televisions), in-vehicle systems (such as vehicle-mounted computers), smart screens, laptops, desktop computers, or all-in-one computers. The electronic device can also be a server.

[0101] The server can be a physical server or a virtualization platform. There can be one physical server or two or more physical servers that share different responsibilities, and the physical servers work together to realize the various functions of the server.

[0102] For example, the server can be a blade server, a high-density server, a rack server, or a tower server, etc.

[0103] The aforementioned scheduling system can be integrated into a large-scale model inference scheduler. The large-scale model inference scheduler can be a software product, a hardware / software integrated product, or a cloud service product.

[0104] It should be noted that the system architecture and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0105] For ease of understanding, the scheduling method provided in this application is described below with reference to the accompanying drawings. This scheduling method is applicable to... Figure 5 , Figure 6 ,or Figure 7 The scheduling system shown.

[0106] like Figure 8 As shown in the embodiments of this application, a scheduling method is provided, which can be executed by a scheduling system. The scheduling method includes:

[0107] S801, the scheduling system obtains the target request.

[0108] The target request is used to request the execution of a target task using a preset model. The target request includes the input information to be input to the preset model.

[0109] For example, there can be one or more target requests. Multiple target requests can correspond to multiple target tasks or a single target task. For instance, a specific target task could be: generating an accurate answer to the question "How is the weather today?". A specific target request could be: Please answer "How is the weather today?".

[0110] For example, the input information can be text or other input data. For instance, the specific meaning of the input information could be "How is the weather today?"

[0111] For example, the input information can be input by a user into a preset model or by an electronic device into a preset model.

[0112] For example, the default model is a large model.

[0113] In this embodiment of the application, by obtaining the target request, the specific content of the target task can be clearly understood, the target task to be executed and the corresponding input information can be determined, so that the preset model can accurately schedule and process the target request to execute the target task.

[0114] S802, the scheduling system predicts the index parameters corresponding to each of the various preset instance scheduling schemes based on the input information.

[0115] The instance scheduling scheme represents the number of tasks and task stages that instances of the preset model can execute within a target time period. The corresponding metric parameters of the instance scheduling scheme are used to characterize the performance of scheduling instances to execute target tasks according to the instance scheduling scheme.

[0116] For example, the multiple preset instance scheduling schemes include several of the first instance scheduling scheme, the second instance scheduling scheme, and the third instance scheduling scheme. In this way, the scheduling system integrates multiple different instance scheduling schemes, which can cope with diverse task requirements and inference scenarios, thereby achieving a more flexible and efficient scheduling mechanism to better adapt to target requests, thus improving the adaptability, versatility, and compatibility of the scheduling system.

[0117] In some embodiments, the metric parameters include a total latency value and / or a total throughput value. The total latency value represents the total latency of the task phase of the target task. The total throughput represents the total throughput of the task phase of the target task.

[0118] For example, the overall latency value can be the sum of the latency value of the prefill phase of the target task and the latency value of the decode phase of the target task.

[0119] For example, the overall throughput can be the sum of the stage throughput of the prefill stage of the target task and the stage throughput of the decode stage of the target task.

[0120] In the embodiments of this application, the indicator parameters include the overall latency value and / or the overall throughput, which can more comprehensively reflect the performance of scheduling instances to execute target tasks according to the instance scheduling scheme.

[0121] In some embodiments, the scheduling system predicts the corresponding indicator parameters for each of a variety of preset instance scheduling schemes based on input information, including: obtaining relevant information about historical tasks; predicting the target information length based on the input information and the relevant information about historical tasks; and predicting the indicator parameters for each instance scheduling scheme based on the length of the input information, the target information length, and the target performance parameters.

[0122] The target information length refers to the length of the output information corresponding to the target task. The target performance parameter is used to characterize the performance of the instance executing the target task.

[0123] In one implementation, since the basic unit of information (e.g., input information, output information corresponding to the target task) in large model inference scenarios can be represented as a token, the length of information can be specifically represented as the number of tokens in the information.

[0124] For example, the length of the input information is equal to the number of tokens in the input information. For instance, if the input information means "How's the weather today?", the number of tokens in this input information might be 10, thus the length of the input information would be 10.

[0125] For example, the length of the target information is the number of tokens in the output information corresponding to the target task.

[0126] Understandably, since a token is generated for each output in the decode phase, and the number of tokens is equivalent to the length of the output information corresponding to the target task, the length of the target information is equivalent to the number of output rounds when executing the decode phase of the target task.

[0127] In this embodiment of the application, by using the relevant information of historical tasks, the characteristics and requirements of historical tasks can be understood relatively accurately, the length of target information can be better predicted, and the index parameters corresponding to different instance scheduling schemes can be predicted more accurately, so that the scheduling system can select a more suitable scheduling scheme to execute the target task, thereby achieving better task execution results and improving the efficiency and performance of large model inference.

[0128] Optionally, the relevant information for historical tasks includes: the lengths of the historical input and output information corresponding to the historical task, the time distribution of the lengths of the historical input and output information, historical performance parameters, and several of the historical KVcache distribution information. Among them, the historical performance parameters are used to characterize the performance of the instance executing historical tasks.

[0129] For example, the length of the historical input information is equal to the number of tokens in the historical input information. The length of the historical output information is equal to the number of tokens in the historical output information.

[0130] For example, the temporal distribution of the lengths of historical input and output information is used to represent the lengths of historical input and output information across different time periods. For instance, in the first time period, the length of the historical input information is 10 (i.e., the number of tokens in the historical input information is 10), and the length of the historical output information is 6 (i.e., the number of tokens in the historical output information is 6). In the second time period, the length of the historical input information is 8, and the length of the historical output information is 12. Thus, by analyzing the temporal distribution of the lengths of historical input and output information, the changes in the lengths of historical input and output information can be determined.

[0131] For example, historical KV cache distribution information is used to represent information about the electronic devices on which historical KV caches were distributed. It is understood that in a scheduling system, there can be multiple instances of a preset model, and these instances can be distributed across one or more electronic devices. Therefore, the KV caches generated by multiple instances may also be distributed across different electronic devices. Specifically, historical KV cache distribution information can be obtained by acquiring the electronic devices on which the KV caches generated by each instance in a historical task were distributed.

[0132] Optionally, the relevant information for historical tasks may also include one or more of the following: memory usage, network transmission efficiency, network bandwidth usage, KV cache storage, and load information for each instance.

[0133] For example, memory usage is used to indicate the amount of memory that has been used by the scheduling system, either partially or entirely.

[0134] For example, network transmission efficiency includes the number of KV cache blocks.

[0135] For example, the storage capacity of the KV cache is used to represent the total amount of data stored in the KV cache.

[0136] In this embodiment of the application, the relevant information of historical tasks covers the above-mentioned multiple aspects of data, which is rich in information and provides a multi-dimensional reference for predicting the indicator parameters corresponding to each instance scheduling scheme, making the prediction more accurate.

[0137] In one implementation, a global simulator or a local simulator is used to predict the target information length based on the input information and relevant information from historical tasks. Specifically, the input information and relevant information from historical tasks are input into the global simulator or local simulator, and the global simulator or local simulator performs modeling and analysis on the input information and relevant information from historical tasks to obtain the target information length output by the global simulator or local simulator.

[0138] For example, the target performance parameters include performance parameters corresponding to each task phase of the target task.

[0139] Specifically, the target performance parameters include a first performance parameter and a second performance parameter. The first performance parameter characterizes the average latency of the instance during the prefill phase of the target task, and the second performance parameter characterizes the average latency of the instance during the decode phase of the target task.

[0140] More specifically, the first performance parameter is the average latency of the instance processing each token of input information during the prefill phase of the target task. The second performance parameter is the average latency of the instance processing each token of information input during the decode phase (e.g., prefill output information) during the decode phase of the target task.

[0141] In one implementation, before predicting the index parameters corresponding to each instance scheduling scheme based on the length of the input information, the length of the target information, and the target performance parameters, the method further includes: estimating the target performance parameters based on relevant information of historical tasks.

[0142] For example, the first performance parameter is estimated based on the length of the historical input information. For instance, the first performance parameter and the length of the historical input information have the following relationship: T p =A×n p +B×n p 2 +C. Where T is the formula. p n is the first performance parameter. p The length of the historical input information is A, B, and C, which are constants of the instance when executing the historical task.

[0143] For example, the second performance parameter is estimated based on the length of the historical input information and the number of KV cache blocks. For instance, the second performance parameter has the following relationship with the length of the historical input information and the number of KV cache blocks: T d =A×n p +B×e+C. Where, T d The second performance parameter is 'e', ​​where 'e' is the number of KV cache blocks.

[0144] In this embodiment of the application, the first performance parameter and the second performance parameter are accurately estimated by using relevant information of historical tasks, thereby enabling the subsequent accurate calculation of the index parameters corresponding to each instance scheduling scheme.

[0145] In some embodiments, predicting the metric parameters corresponding to each instance scheduling scheme based on the length of the input information, the length of the target information, and the target performance parameters includes: for each instance scheduling scheme, predicting the stage latency value of each task stage of the target task based on the length of the input information, the length of the target information, and the performance parameters corresponding to each task stage of the target task. Based on the stage latency value of each task stage of the target task, the overall latency value corresponding to the instance scheduling scheme is obtained.

[0146] In this embodiment, by combining the length of the input information, the length of the target information, and the performance parameters corresponding to each task stage of the target task to predict the index parameters of each instance scheduling scheme, it is possible to more comprehensively consider the initial scale of the target task (such as length), the corresponding output scale (such as the length of the target information), and the performance of the instance at different stages of task processing. This makes the prediction results more accurate and better able to predict the performance of different instance scheduling schemes in actual target task processing.

[0147] For example, the task phases include a prefill phase and a code decoding phase.

[0148] In some embodiments, the stage latency of the prefill stage of the target task is directly proportional to a first performance parameter. The stage latency of the decode stage of the target task is directly proportional to a second performance parameter.

[0149] In this embodiment, by determining the proportional relationship between the stage latency value of the prefill stage of the target task and a first performance parameter, and the proportional relationship between the stage latency value of the decode stage of the target task and a second performance parameter, the direct influence trend of the first and second performance parameters on the stage latency values ​​of their respective tasks is shown. For example, the larger the first performance parameter, the larger the stage latency value of the prefill stage of the target task. Similarly, the larger the second performance parameter, the larger the stage latency value of the decode stage of the target task.

[0150] In one implementation, under the first instance scheduling scheme, the stage delay value of the prefill stage of the target task is the first P delay value, and the stage delay value of the decode stage is the first D delay value.

[0151] The first P-delay value, the length of the input information, and the first performance parameter conform to the following relationship: In the formula, Let P be the first time delay value, α be the first adjustment coefficient, and N be the second time delay value. p T represents the length of the input information. p This is the primary performance parameter.

[0152] In this formula, under the first instance scheduling scheme, when an instance executes the prefill phase of the target task, it processes each token of the input information. Since the first performance parameter is the average latency of the instance processing each token in the input information during the prefill phase of the target task, the first P latency value can be accurately calculated by multiplying the length of the input information (i.e., the number of tokens in the input information) by the first performance parameter and adjusting it through the first adjustment coefficient.

[0153] The first D delay value, the target information length, and the second performance parameter conform to the following relationship: In the formula, T is the first D delay value. wait To wait for scheduling delay, T represents the length of the target information. d This is the second performance parameter.

[0154] In this formula, under the first instance scheduling scheme, T wait This can be represented as the scheduling latency that an instance waits for while executing the decode phase of the target task. For example, after executing the prefill phase of the target task, the instance does not immediately execute the decode phase. This might be because the instance prioritizes executing a higher-priority (e.g., more urgent) target task. Therefore, the instance needs to wait while executing the decode phase of that target task, which results in T. wait However, if an instance executes the decode phase of the target task directly after the prefill phase, then T... wait The value equals 0. Based on this, since the target information length is the number of tokens in the output information corresponding to the target task, and the second performance parameter is the average latency for each token of the information input during the decoding phase of the target task, the product of the target information length and the second performance parameter is calculated. This result is then added to the waiting scheduling latency to accurately calculate the first D latency value.

[0155] In this embodiment, for the first instance scheduling scheme, by calculating the first P-latency value of the prefill stage and the first D-latency value of the decode stage of the target task respectively, the time delay of each stage of executing the target task can be accurately quantified. This allows the scheduling system to accurately obtain the latency of different stages of executing the target task, providing specific data for subsequent scheduling decisions and performance optimization.

[0156] In one implementation, under the second instance scheduling scheme, the stage delay value of the prefill stage of the target task is the second P delay value, and the stage delay value of the decode stage is the second D delay value.

[0157] The second P-delay value, the length of the input information, and the first performance parameter conform to the following relationship: In the formula, The second P delay value, and β is the second adjustment coefficient. The length of the prefill block. The size is determined based on the length of the input information and the preset block size.

[0158] For example, the block prefill length, the length of the input information, and the preset block size conform to the following relationship: In the formula, S chunk The preset block size is used. For example, the block prefill length can refer to the length of multiple blocks of input information (i.e., the total number of blocks). The preset block size can refer to the preset length of each block (i.e., the number of tokens included in a block). For example, if a block includes 3 tokens, the preset block size is 3. If different blocks have different lengths, the preset block size can be the average length of each block. For example, if one block has a length of 2 and another has a length of 4, the preset block size can be 3. Because in the second instance scheduling scheme, when an instance executes the prefill phase of the target task, it specifically divides the input information into multiple blocks, and then processes each block of the input information in the prefill phase, the block prefill length in the input information can be accurately calculated using the length of the input information and the preset block size, thus providing accurate calculation parameters for the subsequent calculation of the second P delay value.

[0159] In the formula for calculating the second P delay value, since the prefill phase of the target task is executed under the second instance scheduling scheme, each block of input information is processed. Therefore, by calculating the product of the block prefill length determined by the length of the input information and the first performance parameter, and then adjusting the second adjustment coefficient, the second P delay value can be accurately calculated.

[0160] The second D delay value and the target information length and the second performance parameter conform to the following relationship: In the formula, This is the second D delay value.

[0161] In this formula, under the second instance scheduling scheme, an instance executes the prefill and decode phases of different tasks within a target time period. This allows the instance to simultaneously execute the prefill phases of other target tasks while executing the decode phase of the current target task. For example, during time period T1, the instance executes the prefill phase of target task B while simultaneously executing the decode phase of target task A. This means that the latency within the same target time period (e.g., time period T1) only needs to be calculated once, and the latency for this target time period has already been calculated in the second latency value P (because the second latency value P is the complete stage latency value for the entire prefill phase of the target task executed by the instance). Therefore, by calculating the difference between the target information length and the block prefill length, it is equivalent to removing the length of the information processed simultaneously with the prefill phase from the target information length (because the latency for processing this part of the information length has already been covered in the calculation of the second latency value P, avoiding duplicate calculation), thus obtaining the length of the target information processed separately when the instance executes the decode phase of the target task. Based on this, multiplying it by the second performance parameter, the second latency value D is accurately calculated.

[0162] In this embodiment of the application, the calculation method fully considers the parallel processing of different task stages (i.e., the prefill stage and the decode stage) and the reasonable calculation of latency, improves the accuracy and rationality of the calculation of the second P latency value and the second D latency value, avoids errors caused by repeated calculations, and makes the whole calculation process more accurate, which meets the actual task execution and latency value calculation requirements of different stages.

[0163] In one implementation, under the third instance scheduling scheme, the stage delay value of the prefill stage of the target task is the third P delay value, and the stage delay value of the decode stage is the third D delay value.

[0164] The third P-delay value conforms to the following relationship with the length of the input information and the first performance parameter: In the formula, The third P delay value, and γ is the third adjustment coefficient. The KV multiplexing length of the input information.

[0165] For example, the KV multiplexing length of input information can refer to the length of the portion of input information that is identical to the current input information, given that the delay has been calculated and the prefix is ​​the same as the current input information's prefix. For instance, if there exists input information before the current input information with the same prefix, then the length of that portion is the KV multiplexing length. For example, the prefix can refer to the header content of the information (e.g., a predefined sequence of bytes or characters at the beginning of the information). Information with the same prefix typically shares similarities in topic, semantic category, etc. For example, "how to improve learning efficiency" and "how to improve work performance" share the common prefix "how to improve". It is understandable that if there is no input information with the same prefix as the current input information in the calculated delay, then the KV multiplexing length of the input information is 0.

[0166] In this formula, under the third instance scheduling scheme, after completing the prefill phase of the target task, instance A stores the prefill output information in the KV cache and transmits the KV cache to instance B to execute the decode phase of the target task. Therefore, the information in the KV cache can be reused. Furthermore, since the KV multiplexing length of the input information is the length of the information whose delay has been calculated before the current input information, the processing result of processing the KV multiplexing length of the input information has been stored in the KV cache and can be used as part of the output information corresponding to the current input information. Therefore, by calculating the difference between the length of the input information and the KV multiplexing length of the input information, it is equivalent to removing the part of the input information length that does not need to be recalculated (the output information corresponding to this part of the information length can be directly applied since it is stored in the KV cache). Based on this, the product with the first performance parameter is calculated, and adjusted by the third adjustment coefficient, the third P delay value can be accurately calculated.

[0167] The third D delay value, the target information length, and the second performance parameter conform to the following relationship: In the formula, This is the third D delay value. For KV transmission delay, It was determined based on relevant information about the historical mission.

[0168] For example, KV transmission latency can refer to the latency between sending the KV cache and receiving the KV cache. KV transmission latency is related to network transmission efficiency, network bandwidth usage, and the storage capacity of the KV cache as follows: (W1×W2). In the formula, Tc is the storage amount of KV cache, W1 is the network bandwidth usage, and W2 is the network transmission efficiency.

[0169] In the formula for calculating the third D latency value, under the third instance scheduling scheme, after instance A completes the prefill phase of the current target task, it stores the prefill output information in the KV cache and transmits it to instance B, resulting in KV transmission latency during the transmission process. Furthermore, instance B may be executing the decode phase of other target tasks, causing the current target task's decode phase to not be executed directly, thus requiring waiting for instance B, which generates waiting scheduling latency. Moreover, during the same time period, while instance A is executing the prefill phase of target task A, instance B may be executing the decode phase of target task B. This means that the latency for the same target time period only needs to be calculated once, and this latency for the same target time period has already been calculated in the third P latency value (because the third P latency value is the calculated complete stage latency value of the entire prefill phase of the instance executing the target task). Therefore, by calculating the difference between the target information length and the block prefill length, it is equivalent to removing the length of the information processed simultaneously with the prefill stage from the target information length (this is because the delay of this information length has already been included in the calculation of the third P delay value, avoiding duplicate calculation), thus obtaining the length of the target information processed separately during the decode stage of instance B's target task. In summary, by calculating the difference between the target information length and the block prefill length, multiplying it by the second performance parameter, and adding it to the waiting scheduling delay and KV transmission delay, the third D delay value is accurately calculated.

[0170] In the embodiments of this application, the parallel processing of different task stages (i.e., the prefill stage and the decode stage) in different instances and the reasonable calculation of latency are fully considered, which improves the accuracy and rationality of the calculation of the third P latency value and the third D latency value, making the entire calculation process more accurate, thereby more accurately determining the target scheduling scheme.

[0171] In some embodiments, the overall throughput is determined based on the overall delay value, the length of the input information, and the length of the target information.

[0172] For example, the overall throughput = (length of input information + length of target information) % overall latency value. This allows for accurate prediction of the overall throughput, leading to a more accurate target scheduling scheme. It can be understood that the overall throughput corresponding to each instance scheduling scheme is equal to the ratio of the sum of the length of the input information and the length of the target information to the overall latency value corresponding to that instance scheduling scheme. Thus, after calculating the overall latency value, the overall throughput can be accurately determined based on that value.

[0173] S803, the scheduling system determines the target scheduling scheme from a variety of preset instance scheduling schemes based on the indicator parameters corresponding to each instance scheduling scheme.

[0174] In some embodiments, the scheduling system determines the target scheduling scheme from a variety of preset instance scheduling schemes based on the index parameters corresponding to each instance scheduling scheme, including: determining the instance scheduling scheme with the smallest comprehensive latency value among the various preset instance scheduling schemes as the target scheduling scheme.

[0175] In this embodiment, the instance scheduling scheme with the lowest overall latency value among various preset instance scheduling schemes is determined as the target scheduling scheme. Thus, when scheduling instances to execute the target task according to the instance scheduling scheme with the lowest overall latency value, the total task execution time latency can be reduced, resulting in good performance. This embodiment can be applied to scenarios where users need to obtain task results feedback more quickly.

[0176] In other embodiments, the scheduling system determines the target scheduling scheme from a variety of preset instance scheduling schemes based on the index parameters corresponding to each instance scheduling scheme, including: determining the instance scheduling scheme with the largest overall throughput among the various preset instance scheduling schemes as the target scheduling scheme.

[0177] In this embodiment, the instance scheduling scheme with the highest overall throughput among various preset instance scheduling schemes is determined as the target scheduling scheme. Thus, when scheduling instances to execute the target task according to the instance scheduling scheme with the highest overall throughput, the system can process more tasks or data per unit time, exhibiting good performance. This embodiment can be applied to scenarios requiring large-scale data processing and high-concurrency requests.

[0178] In other embodiments, the scheduling system determines the target scheduling scheme from a variety of preset instance scheduling schemes based on the index parameters corresponding to each instance scheduling scheme, including: sorting the various preset instance scheduling schemes in ascending order of comprehensive latency value or descending order of comprehensive throughput, and determining the instance scheduling scheme located at a preset position in the sort as the target scheduling scheme.

[0179] For example, or, the preset position could be the second position in the sorting.

[0180] In this application embodiment, a method for determining the target scheduling scheme is extended to achieve a more flexible determination method.

[0181] In other embodiments, the scheduling system determines the target scheduling scheme from a variety of preset instance scheduling schemes based on the index parameters corresponding to each instance scheduling scheme, including: determining the instance scheduling scheme that meets a first preset condition from the variety of preset instance scheduling schemes; and determining the instance scheduling scheme with the highest overall throughput among the instance scheduling schemes that meet the first preset condition as the target scheduling scheme.

[0182] For example, the first preset condition includes a comprehensive delay value that is less than or equal to a third preset delay value.

[0183] For example, the third preset delay value can be set to less than or equal to 0.3s. More specifically, the third preset delay value can be set to 0.1s, 0.2s, or 0.3s.

[0184] In this embodiment, the target scheduling scheme is determined by comprehensively considering both the overall latency and overall throughput, thus achieving a balance and coordination of system performance. Based on the target scheduling scheme selected under the above conditions, when the scheduling system performs scheduling according to the target scheme, it can avoid system failures or performance degradation caused by long waiting times or excessive resource consumption of certain tasks, thereby improving the reliability and stability of the scheduling system.

[0185] In other embodiments, the metrics parameters include the stage latency of the prefill phase of the target task, the stage latency of the decode phase of the target task, and the overall throughput.

[0186] In other embodiments, the scheduling system determines the target scheduling scheme from a variety of preset instance scheduling schemes based on the index parameters corresponding to each instance scheduling scheme, including: determining the instance scheduling scheme that meets a second preset condition from the variety of preset instance scheduling schemes; and determining the instance scheduling scheme with the highest overall throughput among the instance scheduling schemes that meet the preset condition as the target scheduling scheme.

[0187] The second preset condition includes that the stage delay value of the prefill stage of the target task is less than the first preset delay value, and the stage delay value of the decode stage of the target task is less than the second preset delay value.

[0188] Specifically, the first preset delay value can be set to less than or equal to 0.3s. More specifically, the first preset delay value can be set to 0.1s, 0.2s, or 0.3s.

[0189] Specifically, the second preset delay value can be set to less than or equal to 300ms. More specifically, the first preset delay value can be set to 100ms, 200ms, or 300ms.

[0190] In this embodiment, the target scheduling scheme is determined by comprehensively considering both stage latency and throughput, achieving a balance and coordination of system performance. On the one hand, by setting preset conditions including a first preset latency value and a second preset latency value, the user's real-time requirement for stage latency is met; on the other hand, by setting a maximum overall throughput, the system's work efficiency and processing capacity per unit time are improved. Based on the target scheduling scheme selected from the above conditions, when the scheduling system performs scheduling according to the target scheduling scheme, it can avoid system failures or performance degradation caused by long waiting times for certain tasks or excessive resource consumption, further improving the reliability and stability of the scheduling system.

[0191] S804, the scheduling system completes the target task through scheduling instances according to the target scheduling scheme.

[0192] In some embodiments, the scheduling system completes the target task by scheduling instances according to a target scheduling scheme, including: predicting the target hyperparameters corresponding to the target scheduling scheme; configuring instances of a preset model based on the target scheduling scheme and the target hyperparameters to obtain target instances; and executing the target task by scheduling the target instances.

[0193] In this embodiment, a target instance is obtained by configuring a preset model with a target scheduling scheme and target hyperparameters. This allows the preset model instance to dynamically adjust the scheduling scheme and scheduling parameters of the instance to execute the target task according to the target task, thereby improving the high adaptability between the target instance and the target task and enhancing the efficiency and accuracy of the target task execution.

[0194] The target instance is used to execute the target task according to the target scheduling scheme based on the target hyperparameters.

[0195] In one implementation, predicting the target hyperparameters corresponding to the target scheduling scheme includes: obtaining the current system load and relevant information of historical tasks in the scheduling system; predicting the target information length based on the input information and the relevant information of historical tasks; and predicting the target hyperparameters corresponding to the target scheduling scheme based on the length of the input information, the target information length, and the current system load.

[0196] In this embodiment, by analyzing relevant information from historical tasks, potential patterns and regularities can be uncovered; the current system load reveals the current resource usage and workload; and the predicted target information length allows for a preliminary estimate of the target task's scale and complexity before processing. Therefore, by combining the length of the input information, the target information length, and the current system load to predict the target hyperparameters corresponding to the target scheduling scheme, a more refined prediction of the scheduling scheme is achieved, resulting in more accurate target hyperparameters. Target hyperparameters include, but are not limited to, task allocation priority, resource allocation ratio, and the degree of concurrent processing. More accurate prediction of target hyperparameters maintains high resource utilization efficiency and performance levels during the actual execution of the target task, improving system flexibility and reliability. Furthermore, predicting corresponding target hyperparameters for different target requests allows for dynamic adjustment of the target hyperparameters, making them more suitable for different target requests.

[0197] For example, the scheduling system has a scheduling algorithm, and the hyperparameters of the scheduling algorithm have initial values, which are called initial hyperparameters.

[0198] For example, predicting the target hyperparameters corresponding to the target scheduling scheme based on the length of the input information, the length of the target information, and the current system load includes: inputting the length of the input information, the length of the target information, and the current system load into a global simulator or a local simulator; the global simulator or local simulator performs a grid search on the initial hyperparameters based on the length of the input information, the length of the target information, and the current system load; and determining the target hyperparameters corresponding to the target scheduling scheme from the initial hyperparameters. The hyperparameters of the scheduling algorithm are then adjusted to match the target hyperparameters.

[0199] In this embodiment of the application, the initial hyperparameters are optimized and dynamically updated by using grid search technology, thereby achieving efficient optimization and adjustment of the scheduling algorithm, so that the target hyperparameters of the scheduling algorithm can be better adapted to the target request.

[0200] In one implementation, configuring instances of a preset model based on the target scheduling scheme and target hyperparameters to obtain target instances includes: determining the number of instances corresponding to the target scheduling scheme, and selecting instances from the instance pool corresponding to the number of instances corresponding to the target scheduling scheme. Configuring instances corresponding to the number of instances corresponding to the target scheduling scheme based on the target scheduling scheme and target hyperparameters yields the target instances.

[0201] For example, if the target scheduling scheme is the third instance scheduling scheme, then the number of instances corresponding to the target scheduling scheme is two, and two instances are selected from the instance pool. Based on the target scheduling scheme and target hyperparameters, these two instances are configured to obtain the target instances.

[0202] In this embodiment, since the first or second instance scheduling scheme corresponds to one instance, while the third instance scheduling scheme corresponds to multiple instances, if the target scheduling scheme is the first or second instance scheduling scheme, one instance needs to be scheduled to execute the target task; if the target scheduling scheme is the third instance scheduling scheme, multiple instances need to be scheduled to execute the target task. Therefore, the number of instances is determined first, and the instances corresponding to the number of instances corresponding to the target scheduling scheme are selected. Based on this, the instances corresponding to the number of instances corresponding to the target scheduling scheme are configured to obtain the target instance. Furthermore, this embodiment can be applied to the scenario where the instance executes the target task for the first time according to the target scheduling scheme, improving the flexibility and adaptability of the scheduling system and better responding to various different task scenarios and needs.

[0203] In another implementation, based on the target scheduling scheme and target hyperparameters, configuring instances of a preset model to obtain the target instance includes: selecting an instance corresponding to the target scheduling scheme from pre-configured instances in the instance pool; configuring target hyperparameters for the instance corresponding to the target scheduling scheme to obtain the target instance. Here, the pre-configured instance represents an instance capable of executing the target task according to the instance scheduling scheme. The instance corresponding to the target scheduling scheme represents an instance capable of executing the target task according to the target scheduling scheme.

[0204] In this embodiment, the pre-configured instance can be some or all instances in the instance pool, already capable of executing the target task according to various preset instance scheduling schemes. Thus, by identifying the instance corresponding to the target scheduling scheme from the pre-configured instances in the instance pool, the instance to be invoked can be determined efficiently. Furthermore, configuring target hyperparameters for the instance corresponding to the target scheduling scheme facilitates dynamic adjustment of the instance's scheduling parameters (i.e., target hyperparameters), enabling the instance to more accurately adapt to the requirements of the target scheduling scheme and optimizing its performance when executing the target task.

[0205] In one embodiment, a target instance is provided with a target scheduling queue. The target scheduling queue stores information corresponding to target tasks executed according to a target scheduling scheme. This information includes input information for entering the prefill phase when executing the target task, and decoding input information for entering the decoding phase when executing the target task. The target scheduling queue includes multiple types of a first scheduling queue, a second scheduling queue, and a third scheduling queue, each corresponding to multiple types of the first instance scheduling scheme, the second instance scheduling scheme, and the third instance scheduling scheme, respectively.

[0206] For example, the first scheduling queue is used to store information corresponding to the target tasks executed according to the first instance scheduling scheme. The second scheduling queue is used to store information corresponding to the target tasks executed according to the second instance scheduling scheme. The third scheduling queue is used to store information corresponding to the target tasks executed according to the third instance scheduling scheme.

[0207] In one embodiment, this is primarily applied to cluster instance scheduling scenarios for large model inference. A global scheduler determines a target scheduling scheme from multiple preset instance scheduling schemes based on the metric parameters corresponding to each instance scheduling scheme (here, these multiple preset instance scheduling schemes may include a first instance scheduling scheme, a second instance scheduling scheme, and a third instance scheduling scheme). Then, according to the target scheduling scheme, instances are scheduled to complete the target task. Specifically, during the process of scheduling instances to complete the target task, the local scheduler A is responsible for batch scheduling and KV cache management in the prefill and / or decode phases.

[0208] For example, such as Figure 9 and Figure 10 As shown, based on the target request, the global scheduler receives input information and determines the target scheduling scheme. Specifically, determining the target scheduling scheme includes comparing the overall latency values ​​corresponding to multiple preset instance scheduling schemes. If the overall latency value corresponding to the third instance scheduling scheme is the smallest, then the target scheduling scheme is determined to be the third instance scheduling scheme; if the overall latency value corresponding to the third instance scheduling scheme is the largest, then the target scheduling scheme is determined to be either the first instance scheduling scheme or the second instance scheduling scheme.

[0209] like Figure 9As shown, if the target scheduling scheme is a first instance scheduling scheme or a second instance scheduling scheme, the following is an exemplary explanation of how to complete the target task by scheduling instances according to the target scheduling scheme: Since the number of target instances corresponding to the first instance scheduling scheme or the second instance scheduling scheme is one, this target instance is one instance (such as instance A or instance B). Based on this, firstly, the global scheduler puts the input information into instance A (or instance B). Then, the local scheduler A puts the input information into the target scheduling queue. Specifically, putting the input information into the target scheduling queue includes: comparing the comprehensive latency values ​​corresponding to the first instance scheduling scheme and the second instance scheduling scheme. If the comprehensive latency value corresponding to the first instance scheduling scheme is the smallest, the input information is put into the first scheduling queue; if the comprehensive latency value corresponding to the second instance scheduling scheme is the smallest, the input information is put into the second scheduling queue. Secondly, the local scheduler A performs batch scheduling on the input information stored in the target scheduling queue. During batch scheduling, the local scheduler A allocates a KV cache for the input information requested by the target. Then, after the batch scheduling is completed, the local scheduler A sends the batch-scheduled input information to the inference engine. Finally, the inference engine performs inference on the batch-scheduled input information (including prefilling the target request and decoding the prefill output information). After inference is complete, the inference engine outputs the corresponding output information (which can be output in token form).

[0210] like Figure 10As shown, if the target scheduling scheme is a third instance scheduling scheme, the following is an exemplary description of how the target task is completed through scheduling instances according to the target scheduling scheme: The third scheduling queue includes a P scheduling queue and a D scheduling queue. The P scheduling queue is used to store the input information entering the prefill stage of the target task executed according to the third instance scheduling scheme, and the D scheduling queue is used to store the decoding input information entering the decoding stage of the target task executed according to the third instance scheduling scheme. Given that the number of target instances corresponding to the third instance scheduling scheme is at least two (e.g., instance A and instance B), where instance A is used to execute the prefill stage of the target task and has a P scheduling queue, and instance B is used to execute the decoding stage of the target task and has a D scheduling queue. Based on this, firstly, the global scheduler puts the input information into instance A and instance B. Secondly, the local scheduler A puts the input information into the P scheduling queue of instance A. The local scheduler A performs batch scheduling on the input information stored in the P scheduling queue (hereinafter referred to as the first batch scheduling); during the first batch scheduling process, a KV cache is allocated for each input information. After the first batch scheduling is completed, local scheduler A sends the input information after the first batch scheduling to the inference engine. Then, the inference engine performs prefill inference on the input information after the first batch scheduling. After the inference engine completes the prefill inference, it outputs prefill output information (which can be returned to the user as a streaming response). The inference engine stores the prefill output information in the KV cache and transmits it to instance B. After the KV cache transmission is complete, local scheduler A puts the prefill output information into the D scheduling queue of instance B. Then, local scheduler A performs batch scheduling on the prefill output information stored in the D scheduling queue (hereinafter referred to as the second batch scheduling). After the second batch scheduling is completed, local scheduler A sends the prefill output information after the second batch scheduling to the inference engine. The inference engine performs the decode phase processing on the prefill output information after the second batch scheduling. Finally, after inference is completed, the inference engine outputs the output information corresponding to the input information (this output information can be output in token form).

[0211] Furthermore, regarding the input information placed into instance B, the local scheduler A determines whether the input information of instance B needs to undergo a decoding phase. Specifically, the local scheduler A determines whether the input information of instance B has undergone a prefill phase. If the input information of instance B has undergone a prefill phase, then the input information of instance B is placed into the D scheduling queue in instance B, and subsequent batch scheduling and inference processes are performed. If the input information of instance B has not undergone a prefill phase, then the input information of instance B is not processed in the decoding phase.

[0212] For example, the global coordinator returns the output information to the user as a streaming response.

[0213] In this embodiment, the system can be applied to cluster instance scheduling scenarios for large-model inference. Since task types and requirements are diverse in these scenarios, the collaborative scheduling method involving a global scheduler and local schedulers, along with the support of multiple instances for various preset instance scheduling schemes, effectively adapts to different task requirements. By evaluating and selecting the index parameters of various preset instance scheduling schemes, the most suitable target instance and the most appropriate scheduling scheme for processing input information can be flexibly chosen. This is especially true when a wider variety of preset instance scheduling schemes are available, enabling the target task to execute according to the optimal decision, thereby enhancing the versatility and adaptability of the scheduling system to different target tasks and input information requirements.

[0214] In another embodiment, it is mainly applied to the scenario of single instance scheduling for large model inference. The local scheduler B determines the target scheduling scheme from a variety of preset instance scheduling schemes based on the index parameters corresponding to each instance scheduling scheme (here, the variety of preset instance scheduling schemes may include a first instance scheduling scheme and a second instance scheduling scheme), and completes the target task by scheduling instances according to the target scheduling scheme.

[0215] like Figure 11As shown, an exemplary description is provided for completing a target task through scheduling instances according to a target scheduling scheme: Given that the number of target instances corresponding to either the first instance scheduling scheme or the second instance scheduling scheme is one, this target instance is a single instance (e.g., instance A). Based on this, firstly, after the input information is placed into instance A, the local scheduler B determines the target scheduling scheme (specifically, it can be determined from the first and second instance scheduling schemes based on the corresponding indicator parameters). More specifically, determining the target scheduling scheme includes comparing the comprehensive latency values ​​corresponding to the first and second instance scheduling schemes. If the comprehensive latency value corresponding to the first instance scheduling scheme is the smallest, the input information is placed into the first scheduling queue; if the comprehensive latency value corresponding to the second instance scheduling scheme is the smallest, the input information is placed into the second scheduling queue. Secondly, the local scheduler B performs batch scheduling on the input information stored in the target scheduling queue. During batch scheduling, the local scheduler B allocates a KV cache for each input information. After batch scheduling is completed, the local scheduler B sends the batch-scheduled input information to the inference engine. Finally, the inference engine performs inference on the batch-scheduled input information (including prefilling the target request and decoding the prefill output information). After inference is complete, the inference engine outputs the corresponding output information (which can be output in token form).

[0216] In this embodiment, in a single-instance scheduling scenario for large model inference, the local scheduler B directly determines the target scheduling scheme from preset instance scheduling schemes (such as the first instance scheduling scheme and the second instance scheduling scheme) based on indicator parameters. Since the target instance is only one, the decision-making process is reduced, lowering the system's computational complexity, while also improving the system's adaptability and flexibility. This allows users to choose between a single-instance scheduling scheme and a cluster instance scheduling scheme based on their actual needs, making the scheduling method more flexible and better meeting user requirements.

[0217] In some embodiments, after the scheduling system completes the target task through a scheduling instance according to the target scheduling scheme, it obtains relevant information about the current task and updates the relevant information about historical tasks based on this information. The relevant information about the current task includes the current performance parameters and network transmission efficiency corresponding to the target task. For example, the network transmission efficiency corresponding to the target task specifically includes the number of KV cache blocks.

[0218] For example, the relevant information for the current task may also include the load information for each instance of the current task. The load information for each instance of the current task includes the amount of input information entered during the prefill phase and the amount of information entered during the decode phase (i.e., the prefill output information).

[0219] In this embodiment, by acquiring relevant information about the current task, the actual parameters of the target task's execution can be accurately determined. Therefore, incorporating the current task's information into the historical task information enriches the scheduling system's historical data. As the number of tasks executed by the scheduling system increases, it accumulates execution data for various tasks under different scheduling schemes. This allows the scheduling system to make subsequent task scheduling decisions based on richer and more accurate historical experience, improving the accuracy and adaptability of the decision-making basis.

[0220] like Figure 12 As shown in the figure, this application embodiment provides a scheduling device 200, which is applied to a scheduling system. The scheduling device 200 includes: an acquisition module 21, a prediction module 22, a determination module 23, and a scheduling module 24. The acquisition module 21 is used to acquire a target request, which requests the execution of a target task using a preset model. The target request includes input information to be input to the preset model. The prediction module 22 is used to predict the indicator parameters corresponding to each of a variety of preset instance scheduling schemes based on the input information. An instance scheduling scheme represents the number of tasks and task stages that an instance of the preset model can execute within a target time period. The indicator parameters corresponding to the instance scheduling scheme characterize the performance of scheduling instances to execute the target task according to the instance scheduling scheme. The determination module 23 is used to determine a target scheduling scheme from a variety of preset instance scheduling schemes based on the indicator parameters corresponding to each instance scheduling scheme. The scheduling module 24 is used to complete the target task by scheduling instances according to the target scheduling scheme.

[0221] For example, the acquisition module 21 corresponds to the global coordinator or local scheduler B mentioned above.

[0222] For example, prediction module 22 corresponds to the global simulator or local simulator described above.

[0223] For example, the determination module 23 corresponds to the global scheduler, local scheduler A, or local scheduler B mentioned above.

[0224] For example, scheduling module 24 corresponds to the global scheduler, local scheduler A, or local scheduler B mentioned above.

[0225] like Figure 13As shown, this application embodiment provides an electronic device 500. The electronic device 500 includes a processor 510 and a memory 520 for storing processor-executable instructions. When the processor 510 is configured to execute instructions, the electronic device 500 implements the scheduling method described above.

[0226] Figure 13 The electronic device 500 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0227] Electronic device 500 is manifested in the form of a general-purpose computing device. The components of electronic device 500 may include, but are not limited to: one or more processors 510, memory 520, communication bus 540 connecting different system components (including memory 520 and processor 510), and communication interface 530.

[0228] The communication bus 540 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0229] Electronic devices 500 typically include a variety of computer system-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.

[0230] Memory 520 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Although Figure 13As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the communication bus 540 via one or more data media interfaces. The memory 520 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0231] A program / utility having a set (at least one) of program modules can be stored in memory 520. Such program modules include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of this application.

[0232] Electronic device 500 can also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), and with one or more devices that enable a user to interact with the electronic device, and / or with any device that enables the electronic device to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through communication interface 530. Furthermore, electronic device 500 can also communicate through a network adapter (… Figure 13 (Not shown) communicates with one or more networks (e.g., Local Area Network (LAN), Wide Area Network (WAN), and / or public networks, such as the Internet). The aforementioned network adapter can communicate with other modules of the electronic device via the communication bus 540. It should be understood that, although... Figure 13 As not shown, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, Redundant Arrays of Independent Drives (RAID) systems, tape drives, and data backup storage systems.

[0233] The processor 510 executes various functional applications and data processing by running programs stored in the memory 520, such as implementing the scheduling method provided in the embodiments of this application.

[0234] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 500. In other embodiments of this application, the electronic device 500 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0235] It is understood that the aforementioned electronic devices, etc., include hardware structures and / or software modules corresponding to perform each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, in conjunction with the exemplary units and algorithm steps described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0236] This application embodiment can divide the above-mentioned electronic device into functional modules according to the method example described above. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0237] This application also provides a storage medium, including a computer-readable storage medium storing a computer program, which, when executed by an electronic device, causes the electronic device to implement the scheduling method described above.

[0238] This application also provides a program product, which includes a computer program that, when at least one processor executes the computer program, causes the at least one processor to execute the scheduling method provided in this application.

[0239] The electronic devices, storage media, or computer program products provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0240] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0241] In the embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0242] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0243] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A scheduling method, characterized in that, Applied to a scheduling system, the method includes: Obtain a target request, the target request being used to request the execution of a target task using a preset model, the target request including input information to be input to the preset model; Based on the input information, predict the indicator parameters corresponding to each of the multiple preset instance scheduling schemes; the instance scheduling scheme represents the number of tasks and task stages that the instance of the preset model can execute in the target time period; the indicator parameters corresponding to the instance scheduling scheme are used to characterize the performance of scheduling the instance to execute the target task according to the instance scheduling scheme; Based on the indicator parameters corresponding to each instance scheduling scheme, the target scheduling scheme is determined from the multiple preset instance scheduling schemes; The target task is completed by scheduling the instance according to the target scheduling scheme.

2. The scheduling method according to claim 1, characterized in that, The metrics include a comprehensive latency value and / or a comprehensive throughput value; the comprehensive latency value represents the total latency value of the task phase of the target task, and the comprehensive throughput value represents the total throughput of the task phase of the target task.

3. The scheduling method according to claim 2, characterized in that, The step of determining the target scheduling scheme from the multiple preset instance scheduling schemes based on the indicator parameters corresponding to each instance scheduling scheme includes: The instance scheduling scheme with the smallest overall latency or the largest overall throughput among the various preset instance scheduling schemes is determined as the target scheduling scheme.

4. The scheduling method according to any one of claims 1-3, characterized in that, The step of predicting the index parameters corresponding to each of the multiple preset instance scheduling schemes based on the input information includes: Obtain relevant information about historical tasks; Based on the input information and the relevant information of the historical task, the target information length is predicted, where the target information length is the length of the output information corresponding to the target task. Based on the length of the input information, the length of the target information, and the target performance parameters, predict the indicator parameters corresponding to each instance scheduling scheme; the target performance parameters are used to characterize the performance of the instance in executing the target task.

5. The scheduling method according to claim 4, characterized in that, Before predicting the metric parameters corresponding to each instance scheduling scheme based on the length of the input information, the length of the target information, and the target performance parameters, the method further includes: The target performance parameters are estimated based on the relevant information from the historical missions.

6. The scheduling method according to claim 4 or 5, characterized in that, The indicator parameters include a comprehensive latency value, which represents the total latency value of the task phase of the target task; the target performance parameters include performance parameters corresponding to each task phase of the target task. The step of predicting the metric parameters corresponding to each instance scheduling scheme based on the length of the input information, the length of the target information, and the target performance parameters includes: For each instance scheduling scheme, the stage delay value of each task stage of the target task is predicted based on the length of the input information, the length of the target information, and the performance parameters corresponding to each task stage of the target task. Based on the stage delay value of each task stage of the target task, the comprehensive delay value corresponding to the instance scheduling scheme is obtained.

7. The scheduling method according to claim 6, characterized in that, The task phase includes a prefill phase and a decode phase; the target performance parameters include a first performance parameter and a second performance parameter, wherein the first performance parameter is used to characterize the average latency of the instance when executing the prefill phase of the target task, and the second performance parameter is used to characterize the average latency of the instance when executing the decode phase of the target task; The stage delay value of the prefill stage of the target task is directly proportional to the first performance parameter; the stage delay value of the decode stage of the target task is directly proportional to the second performance parameter.

8. The scheduling method according to claim 7, characterized in that, The instance scheduling scheme includes a first instance scheduling scheme; the first instance scheduling scheme indicates that an instance performs the prefill phase or decode phase of the same task in the target time period. Under the first instance scheduling scheme, the stage delay value of the prefill stage of the target task is the first P delay value, and the stage delay value of the decode stage is the first D delay value. The first P-delay value, the length of the input information, and the first performance parameter conform to the following relationship: In the formula, Let P be the first time delay value, α be the first adjustment coefficient, and N be the N-th time delay value. p T is the length of the input information. p The first performance parameter; The first D-delay value, the target information length, and the second performance parameter conform to the following relationship: In the formula, Let T be the first D delay value. wait To wait for scheduling delay, T is the length of the target information. d This refers to the second performance parameter.

9. The scheduling method according to claim 7 or 8, characterized in that, The instance scheduling scheme includes a second instance scheduling scheme; the second instance scheduling scheme indicates that an instance performs different tasks in the prefill and decode phases during the target time period. Under the second instance scheduling scheme, the stage delay value of the prefill stage of the target task is the second P delay value, and the stage delay value of the decode stage is the second D delay value. The second P-delay value, the length of the input information, and the first performance parameter conform to the following relationship: In the formula, The second P delay value, β is the second adjustment coefficient, and T p The first performance parameter, The length of the prefill block. Determined based on the length of the input information and the preset block size; The second D delay value, the target information length, and the second performance parameter conform to the following relationship: In the formula, This is the second D delay value. T is the length of the target information. d This refers to the second performance parameter.

10. The scheduling method according to any one of claims 7-9, characterized in that, The instance scheduling scheme includes a third instance scheduling scheme; the third instance scheduling scheme means that multiple instances execute different tasks in the prefill and decode phases during the target time period. Under the third instance scheduling scheme, the stage delay value of the prefill stage of the target task is the third P delay value, and the stage delay value of the decode stage is the third D delay value. The third P-delay value, the length of the input information, and the first performance parameter conform to the following relationship: In the formula, The third P delay value is given, γ is the third adjustment coefficient, and N is the third adjustment coefficient. p The length of the input information. T is the KV multiplexing length of the input information. p The first performance parameter; The third D delay value, the target information length, and the second performance parameter conform to the following relationship: In the formula, For the third D delay value, T wait To wait for scheduling delay, For KV transmission delay, It was determined based on relevant information about the historical mission. The length of the target information. The length of the prefill block. T is determined based on the length of the input information and the preset block size. d This refers to the second performance parameter.

11. The scheduling method according to any one of claims 1-10, characterized in that, The step of completing the target task by scheduling the instance according to the target scheduling scheme includes: Predict the target hyperparameters corresponding to the target scheduling scheme; Based on the target scheduling scheme and the target hyperparameters, an instance of the preset model is configured to obtain a target instance; the target instance is used to execute the target task according to the target scheduling scheme based on the target hyperparameters. The target task is executed by scheduling the target instance.

12. The scheduling method according to claim 11, characterized in that, The predicted target hyperparameters corresponding to the target scheduling scheme include: Obtain relevant information about the current system load and historical tasks of the scheduling system; Based on the input information and the relevant information of the historical task, the target information length is predicted, where the target information length is the length of the output information corresponding to the target task. Based on the length of the input information, the length of the target information, and the current system load, predict the target hyperparameters corresponding to the target scheduling scheme.

13. The scheduling method according to any one of claims 4, 5, 10, and 12, characterized in that, The relevant information of the historical task includes the length of the historical input information and historical output information corresponding to the historical task, the time distribution of the length of the historical input information and historical output information, historical performance parameters, and historical key-value cache (KV cache) distribution information; the historical performance parameters are used to characterize the performance of the instance in executing the historical task.

14. A scheduling device, characterized in that, The device, used in a scheduling system, includes: The acquisition module is used to acquire a target request, which is used to request the execution of a target task using a preset model. The target request includes input information to be input to the preset model. The prediction module is used to predict the indicator parameters corresponding to each of a variety of preset instance scheduling schemes based on the input information; the instance scheduling scheme represents the number of tasks and task stages that the instance of the preset model can execute in a target time period; the indicator parameters corresponding to the instance scheduling scheme are used to characterize the performance of scheduling the instance to execute the target task according to the instance scheduling scheme. The determination module is used to determine the target scheduling scheme from the multiple preset instance scheduling schemes based on the indicator parameters corresponding to each instance scheduling scheme. The scheduling module is used to complete the target task by scheduling the instance according to the target scheduling scheme.

15. An electronic device, characterized in that, include: A processor and a memory for storing processor-executable instructions; When the processor is configured to execute the instructions, the electronic device implements the scheduling method as described in any one of claims 1-13.

16. A storage medium storing computer program instructions thereon; characterized in that, When the computer program instructions are executed by the electronic device, the electronic device implements the scheduling method as described in any one of claims 1-13.

17. A program product, characterized in that, The system includes a computer-readable storage medium storing a computer program, which, when executed by at least one processor, causes the at least one processor to perform the scheduling method according to any one of claims 1-13.

Images