Large model reasoning scheduling method and electronic equipment

Abstract

The invention provides a large model reasoning scheduling method and electronic equipment, and relates to the technical field of computers. The method comprises the steps of obtaining service performance parameters corresponding to inference service of a large model within target time; the service performance parameters comprise the input data length, the output data length, the throughput and the service standard of the inference service and hardware state parameters supporting the inference service; determining a service performance index corresponding to the reasoning service based on the service performance parameter; the service performance indexes comprise the target throughput of the inference service, the load of the inference service, the type of the inference service and the state of hardware supporting the inference service; adjusting the current resource configuration of the large model based on a scheduling strategy corresponding to a scheduling condition satisfied by the service performance index to obtain a target resource configuration; wherein each scheduling condition has a corresponding weight, and the weight is obtained based on an association relationship between the scheduling target corresponding to the reasoning service and the scheduling condition.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a large-scale model inference scheduling method and electronic device. Background Technology

[0002] In Large Language Model (LLM) inference services, a fixed resource configuration is typically used, such as a fixed prefill / decode replica ratio, a fixed tensor parallelism, and a fixed data parallelism. Prefill replicas are primarily used to process the LLM input data and construct initial context information, while decode replicas are used to progressively generate the LLM output based on the output of the prefill replicas. However, this fixed resource configuration approach cannot meet the complex and ever-changing real-world inference needs, making it difficult to achieve optimal system performance and efficient resource utilization. Summary of the Invention

[0003] This disclosure provides a large model inference scheduling method and electronic device to at least solve the above-mentioned technical problems existing in the prior art.

[0004] According to a first aspect of this disclosure, a large-model inference scheduling method is provided, comprising: obtaining service performance parameters corresponding to the inference service of a large model within a target time; the service performance parameters including the input data length, output data length, throughput, service standard, and hardware status parameters supporting the inference service; determining service performance indicators corresponding to the inference service based on the service performance parameters; the service performance indicators including the target throughput of the inference service, the load of the inference service, the type of the inference service, and the hardware status supporting the inference service; adjusting the current resource configuration of the large model based on the scheduling policy corresponding to the scheduling conditions satisfied by the service performance indicators to obtain a target resource configuration; wherein each scheduling condition has a corresponding weight, and the weight is obtained based on the correlation between the scheduling target corresponding to the inference service and the scheduling condition.

[0005] In one possible implementation, the scheduling conditions include at least one of the following: the target throughput of the inference service is less than a first threshold; the change in the load of the inference service is greater than a second threshold; the type of the inference service changes; and the hardware state satisfies a third threshold.

[0006] In one possible implementation, adjusting the current resource configuration of the large model based on the scheduling policy corresponding to the scheduling conditions satisfied by the service performance indicators to obtain the target resource configuration includes: in response to the existence of multiple scheduling conditions with weights greater than a fourth threshold, determining the execution order of the scheduling policy corresponding to the scheduling conditions with weights greater than the fourth threshold based on the weights of the scheduling conditions; and executing the scheduling policy corresponding to the scheduling conditions with weights greater than the fourth threshold based on the execution order to adjust the current resource configuration of the large model to obtain the target resource configuration.

[0007] In one embodiment, a large model inference scheduling method further includes: in response to the existence of a scheduling condition with a weight greater than a fifth threshold, adjusting the current resource configuration of the large model based on the scheduling strategy corresponding to the scheduling condition with the largest weight to obtain a target resource configuration; wherein the fifth threshold is greater than the fourth threshold.

[0008] In one possible implementation, the weights corresponding to the scheduling conditions are determined as follows: based on the input information of the large model, a scheduling target corresponding to the inference service and a first weight of the scheduling target are determined; based on the association between the scheduling target and the scheduling conditions, a second weight of each scheduling condition for the scheduling target is determined; and based on the first weight and the second weight, the weight of the scheduling condition is determined.

[0009] In one possible implementation, the scheduling strategy for the scheduling conditions corresponding to the load of the inference service includes: in response to an increase in the load of the inference service, reducing the data parallelism in the current resource configuration of the large model by a sixth threshold and increasing the tensor parallelism in the current resource configuration of the large model by a sixth threshold; in response to a decrease in the load of the inference service, increasing the data parallelism in the current resource configuration of the large model by a sixth threshold and decreasing the tensor parallelism in the current resource configuration of the large model by a sixth threshold.

[0010] In one implementation, the scheduling strategy for the scheduling conditions corresponding to the type of the inference service includes: in response to the inference service becoming input-intensive, increasing the number of first execution replicas in the current resource configuration of the large model by a seventh threshold; the first execution replicas are used to process the input of the inference service; in response to the inference service becoming output-intensive, increasing the number of second execution replicas in the current resource configuration of the large model by a seventh threshold; the second execution replicas are used to generate the output of the inference service.

[0011] In one implementation, the scheduling strategy for the scheduling conditions corresponding to the target throughput includes: increasing the number of the first and second execution replicas in the current resource configuration of the large model by an eighth threshold.

[0012] In one implementation, the scheduling strategy for the scheduling conditions corresponding to the hardware state includes reducing the number of both the first and second execution replicas in the current resource configuration of the large model by a ninth threshold.

[0013] According to a second aspect of this disclosure, an electronic device is provided, comprising: a scheduling unit; and a plurality of processors configured to execute an inference service for a large model; the scheduling unit being configured to: acquire service performance parameters corresponding to the inference service of the large model within a target time period; the service performance parameters including the input data length, output data length, throughput, service standard, and hardware status parameters supporting the inference service; determine a service performance index corresponding to the inference service based on the service performance parameters; the service performance index including the target throughput of the inference service, the load of the inference service, the type of the inference service, and the hardware status supporting the inference service; and adjust the current resource configuration of the large model based on a scheduling strategy corresponding to a scheduling condition satisfied by the service performance index to obtain a target resource configuration; wherein each scheduling condition has a corresponding weight, the weight being obtained based on the correlation between the scheduling target corresponding to the inference service and the scheduling condition.

[0014] According to a third aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this disclosure.

[0015] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0016] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0017] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0018] Figure 1 This illustration shows a flowchart of a large model inference scheduling method according to an embodiment of the present disclosure. Figure 1 ;

[0019] Figure 2 This illustration shows a flowchart of a large model inference scheduling method according to an embodiment of the present disclosure. Figure 2 ;

[0020] Figure 3 This illustration shows a flowchart of a large model inference scheduling method according to an embodiment of the present disclosure. Figure 3 ;

[0021] Figure 4 This illustration shows a flowchart of a large model inference scheduling method according to an embodiment of the present disclosure. Figure 4 ;

[0022] Figure 5 A schematic diagram of the composition structure of an electronic device according to an embodiment of the present disclosure is shown;

[0023] Figure 6 A schematic diagram of a scenario for an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0024] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0025] Figure 1 This illustration shows a flowchart of a large model inference scheduling method according to an embodiment of the present disclosure. Figure 1 ,like Figure 1 As shown, a large-model inference scheduling method includes:

[0026] Step S101: Obtain the service performance parameters corresponding to the inference service of the large model within the target time period.

[0027] In this embodiment, it is first necessary to obtain the service performance parameters of the large model inference service within a certain target time. These service performance parameters are those that quantify the performance of the inference service, including the input data length, output data length, throughput, service standards, and hardware status parameters supporting the inference service. Specifically, the input data length is the length of the input data received by the inference service; for example, in a natural language processing task, the input data length can be the number of words or characters in a text paragraph. The output data length is the length of the output data generated by the inference service; for example, in a text generation task, the output data length can be the number of words or characters in the generated text. Throughput refers to the amount of data the inference service can process per unit of time, directly reflecting the performance of the inference service. Service standards are the performance requirements or service quality indicators that the inference service needs to meet, such as response time and accuracy. Service standards can be represented by Service Level Objectives (SLOs). Hardware status parameters include various status information of the hardware supporting the inference service, such as CPU (Central Processing Unit) utilization, GPU (Graphics Processing Unit) utilization, and memory usage.

[0028] Step S102: Based on the service performance parameters, determine the service performance indicators corresponding to the inference service.

[0029] In this embodiment, it is also necessary to determine the service performance indicators corresponding to the inference service based on the service performance parameters. The service performance indicators can intuitively reflect the performance status and operating status of the inference service. Based on the service performance indicators, it can be determined how to adjust the current resource configuration of the large model. The service performance indicators include the target throughput of the inference service, the load of the inference service, the type of the inference service, and the hardware status supporting the inference service.

[0030] In one implementation, the target throughput is the maximum request rate that the inference service can stably handle while meeting service standards. The target throughput is obtained based on the service standards and throughput acquired in step S101. For example, the maximum throughput that can meet the service standards and can be achieved under existing resource conditions is taken as the target throughput. The load of the inference service is the workload currently undertaken by the inference service. It can be determined by the number of requests currently being processed and the number of requests waiting in the queue, or by the length of the input data and the length of the output data acquired in step S101. For example, the load of the inference service can be determined based on the changing trends of the input data length and the output data length, combined with the changes in throughput. The type of the inference service is the task processed by the inference service. The type, such as input-intensive task or output-intensive task, can be determined by analyzing the length of the input and output data in the service performance parameters. For example, when the length of the input data is much longer than the length of the output data, it can be identified as an input-intensive task; when the length of the output data is much longer than the length of the input data, it can be identified as an output-intensive task. The hardware status refers to the current state of the hardware that supports the operation of the inference service, which reflects the availability and performance of the hardware that supports the operation of the inference service. The hardware status can be determined based on the hardware status parameters obtained in step S101. For example, if the GPU utilization rate is consistently higher than a certain threshold for a certain period of time, and the memory usage is also consistently close to the upper limit, it can be determined that the GPU and memory are in an overloaded state.

[0031] Step S103: Based on the scheduling strategy corresponding to the scheduling conditions met by the service performance indicators, adjust the current resource configuration of the large model to obtain the target resource configuration.

[0032] In this embodiment, the scheduling conditions are pre-set based on service performance metrics. When the service performance metrics meet the scheduling conditions, the corresponding scheduling strategy is triggered. Different scheduling conditions correspond to different scheduling strategies, which can be used to adjust the current resource configuration of the large model to obtain the target resource configuration. For example, when the target throughput is lower than a threshold, the scheduling condition corresponding to the target throughput is triggered. The scheduling strategy corresponding to this condition might be to increase PD replicas to improve throughput. When the load on the inference service increases, the scheduling condition corresponding to the load on the inference service is triggered. The scheduling strategy corresponding to this condition might be to increase parallelism to adapt to load changes. When the type of the inference service changes, the scheduling condition corresponding to the type of inference service is triggered. The scheduling strategy corresponding to this condition might be to adjust the proportion of PD replicas to adapt to the resource requirements of different types of tasks. In one example, the scheduling conditions can be represented in the form of triggers. When the service performance metrics meet the scheduling conditions, the corresponding triggers are triggered.

[0033] In one implementation, each scheduling condition has a corresponding weight, which is derived from the correlation between the scheduling objective and the scheduling condition corresponding to the inference service. The scheduling objective refers to the performance and resource management goals that the inference service needs to achieve during operation, such as optimal performance, optimal cost, highest throughput, and meeting a specific performance metric. Different application scenarios and user needs may have different scheduling objectives. For example, in application scenarios with high real-time requirements, optimal performance may be the primary scheduling objective; while in cost-sensitive application scenarios, optimal cost may be the primary scheduling objective. The correlation refers to the relationship between the scheduling objective and the scheduling condition. Different scheduling objectives have different sensitivities to scheduling conditions, therefore, the correlation is needed to determine the weight of each scheduling condition. For example, for the optimal performance scheduling objective, the scheduling condition corresponding to the target throughput may be the most important scheduling condition, thus the scheduling condition corresponding to the target throughput will have a relatively high weight; while for the optimal cost scheduling objective, the scheduling condition corresponding to the hardware state may be the most important scheduling condition, thus the scheduling condition corresponding to the hardware state will have a relatively high weight.

[0034] In one implementation, when adjusting the current resource configuration of a large model based on the scheduling strategy corresponding to the scheduling conditions met by the service performance indicators, if there are multiple scheduling conditions met by the service performance indicators, the scheduling strategies corresponding to the scheduling conditions can be sorted or selected based on the weight of each scheduling condition, thereby obtaining a target resource configuration that better meets the current scheduling objectives.

[0035] In this embodiment, service performance parameters corresponding to the inference service of the large model within a target time period are obtained. Based on these parameters, service performance indicators are determined. Then, based on the scheduling policy corresponding to the scheduling conditions satisfied by the service performance indicators, the current resource configuration of the large model is adjusted to obtain the target resource configuration. Therefore, during the inference service process of the large model, the current resource configuration can be dynamically adjusted in real time based on the service performance parameters of the inference service, so that the obtained target resource configuration can meet the inference requirements of different inference services, improving the performance and resource utilization efficiency of the inference service.

[0036] In another embodiment, the scheduling conditions include at least one of the following:

[0037] The target throughput of the inference service is less than the first threshold; the change in the load of the inference service is greater than the second threshold; the type of the inference service changes; and the hardware status meets the third threshold.

[0038] In this embodiment, the scheduling condition corresponding to the target throughput can be that the target throughput of the inference service is less than a first threshold. For example, in an online inference service scenario, if the target throughput is 800 requests / second and the first threshold is 1000 requests / second, the scheduling condition corresponding to the target throughput is considered to be triggered because the target throughput is less than the first threshold.

[0039] In this embodiment, the scheduling condition corresponding to the load of the inference service can be that the change in the load of the inference service is greater than a second threshold. For example, if the load of the inference service increases from 100 requests / second to 500 requests / second in a short period of time, and the change is greater than the set second threshold, it proves that the load of the inference service has changed significantly, and then it can be considered that the scheduling condition corresponding to the load of the inference service has been triggered.

[0040] In this embodiment, the scheduling condition corresponding to the type of inference service can be a change in the type of inference service. For example, in a natural language processing task, if the inference service changes from a text classification task (input-intensive task) to a text generation task (output-intensive task), then the scheduling condition corresponding to the type of inference service is considered to have been triggered.

[0041] In this embodiment, the scheduling condition corresponding to the hardware state can be that the hardware state meets a third threshold. For example, if the GPU utilization exceeds 90%, the GPU hardware state is considered abnormal, which triggers the scheduling condition corresponding to the hardware state; if the GPU utilization is less than 10%, the GPU hardware state is also considered abnormal, which triggers the scheduling condition corresponding to the hardware state.

[0042] Figure 2 This illustration shows a flowchart of a large model inference scheduling method according to an embodiment of the present disclosure. Figure 2 ,like Figure 2 As shown, a large-model inference scheduling method includes:

[0043] Step S201: Obtain the service performance parameters corresponding to the inference service of the large model within the target time period.

[0044] Step S202: Based on the service performance parameters, determine the service performance indicators corresponding to the inference service.

[0045] The specific implementation of steps S201-S202 is similar to that of steps S101-S102, and will not be repeated here.

[0046] Step S203: In response to the existence of multiple scheduling conditions with weights greater than the fourth threshold, the execution order of the scheduling policy corresponding to the scheduling conditions with weights greater than the fourth threshold is determined based on the weights of the scheduling conditions.

[0047] In this embodiment, if the service performance index satisfies multiple scheduling conditions, and among these multiple scheduling conditions, there are multiple scheduling conditions with weights greater than the fourth threshold, then the execution order of the scheduling policies corresponding to the scheduling conditions with weights greater than the fourth threshold is determined based on the weights of the scheduling conditions. That is, the scheduling policy corresponding to the scheduling condition with the higher weight is executed first. Here, the fourth threshold can be less than 0.5. For example, suppose the service performance index satisfies scheduling conditions A, B, and C, with weights of 0.3, 0.4, and 0.1 respectively, and the fourth threshold is 0.25. Obviously, scheduling conditions A and B are scheduling conditions with weights greater than the fourth threshold. Therefore, the execution order of the scheduling policies corresponding to scheduling conditions A and B is the scheduling policy corresponding to scheduling condition B, followed by the scheduling policy corresponding to scheduling condition A.

[0048] Step S204: Based on the execution order, the scheduling policy corresponding to the scheduling condition with an execution weight greater than the fourth threshold is used to adjust the current resource configuration of the large model to obtain the target resource configuration.

[0049] In this embodiment, scheduling policies corresponding to scheduling conditions with weights greater than the fourth threshold can be executed sequentially according to a determined execution order to obtain the target resource configuration. For example, if the execution order of the scheduling policies corresponding to scheduling conditions A and B obtained in step S203 is the scheduling policy corresponding to scheduling condition B, then the scheduling policy corresponding to scheduling condition A, and scheduling condition A is for the target throughput of the inference service being less than the first threshold, while scheduling condition B is for the load change value of the inference service being greater than the second threshold, then the scheduling policy corresponding to scheduling condition B can be executed first, such as increasing the parallelism, and then the scheduling policy corresponding to scheduling condition A can be executed, such as increasing PD replicas. In one example, after executing each scheduling policy, it can be checked whether the resource configuration meets the requirements of the target resource configuration, such as whether the current throughput meets the throughput requirements of the target resource configuration. If it does, the execution of subsequent scheduling policies is stopped; if it does not, the next scheduling policy is executed, thereby reducing the power consumption required for scheduling.

[0050] In one implementation, if the service performance index meets only one scheduling condition, the current resource configuration of the large model is adjusted directly based on the scheduling policy corresponding to that scheduling condition to obtain the target resource configuration.

[0051] In another possible implementation, if multiple scheduling conditions are met for the service performance metrics, but none of them have a weight greater than the fourth threshold, then the current resource configuration of the large model is not adjusted based on the scheduling strategy that meets the met scheduling conditions. The adjustment can wait until the next detection cycle to prevent instability. For example, suppose the service performance metrics meet scheduling conditions A, B, C, and D, with weights of 0.2, 0.1, 0.22, and 0.15 respectively, and the fourth threshold is 0.25. Since the weights of scheduling conditions A through D are all less than the fourth threshold, it proves that scheduling conditions A through D have a relatively small impact on the performance and resource management of the inference service. Therefore, the current resource configuration of the large model is not adjusted based on the scheduling strategy that meets scheduling conditions A through D.

[0052] In this disclosure, in response to the existence of multiple scheduling conditions with weights greater than a fourth threshold, the execution order of the scheduling policies corresponding to the scheduling conditions with weights greater than the fourth threshold is determined based on the weights of the scheduling conditions. Then, based on the execution order, the scheduling policies corresponding to the scheduling conditions with weights greater than the fourth threshold are executed to adjust the current resource configuration of the large model and obtain the target resource configuration. This ensures that when multiple triggered scheduling conditions with weights greater than the fourth threshold exist, the scheduling policies corresponding to these scheduling conditions are executed in a reasonable order, thereby achieving efficient scheduling of the large model inference service resources.

[0053] In another embodiment, a large model inference scheduling method further includes:

[0054] In response to the existence of a scheduling condition with a weight greater than the fifth threshold, the current resource configuration of the large model is adjusted based on the scheduling strategy corresponding to the scheduling condition with the largest weight to obtain the target resource configuration; the fifth threshold is greater than the fourth threshold.

[0055] In this embodiment, if multiple scheduling conditions are satisfied by the service performance indicators, and among these conditions, there exists a scheduling condition with a weight greater than the fifth threshold, then the current resource configuration of the large model is adjusted based on the scheduling strategy corresponding to the scheduling condition with the highest weight to obtain the target resource configuration. Here, the fifth threshold is greater than the fourth threshold, and the fifth threshold can be greater than 0.5. In one example, the sum of the weights of all scheduling conditions is 1. If the fifth threshold is greater than 0.5, and there exists a scheduling condition with a weight greater than the fifth threshold, it proves that the weights of the other satisfied scheduling conditions will not be greater than 0.5. That is, the weights of the other satisfied scheduling conditions may be much smaller than the scheduling condition with a weight greater than the fifth threshold. In this case, the scheduling condition with a weight greater than the fifth threshold has a significant impact on the performance and resource management of the inference service, while the other satisfied scheduling conditions can be ignored. Therefore, the current resource configuration of the large model can be directly adjusted based on the scheduling strategy corresponding to the scheduling condition with the highest weight, i.e., the scheduling condition with a weight greater than the fifth threshold.

[0056] In this disclosure, if multiple scheduling conditions are satisfied by the service performance indicators, and among these conditions, there is a scheduling condition with a weight greater than a fifth threshold, then the current resource configuration of the large model is adjusted based on the scheduling strategy corresponding to the scheduling condition with the highest weight to obtain the target resource configuration. This ensures that when the weight of a certain scheduling condition is significantly greater than the weights of other scheduling conditions, the current resource configuration of the large model is adjusted only based on the scheduling strategy corresponding to the scheduling condition with the highest weight. This ensures the efficiency and accuracy of scheduling decisions, avoids resource waste, performance degradation, and system instability caused by adjusting the current resource configuration of the large model based on scheduling conditions with lower weights, and improves the performance and scheduling efficiency of the inference service.

[0057] In another embodiment, in a large model inference scheduling method, the weights corresponding to the scheduling conditions are determined based on the following:

[0058] Based on the input information of the large model, the scheduling target and the first weight of the scheduling target corresponding to the inference service are determined; based on the correlation between the scheduling target and the scheduling conditions, the second weight of each scheduling condition on the scheduling target is determined; based on the first weight and the second weight, the weight of the scheduling condition is determined.

[0059] In this embodiment, the scheduling objectives for the inference service and the first weight of each scheduling objective need to be determined based on the input information of the large model. The input information of the large model may include user questions and business requirements. For example, if the input information indicates that the current inference service is an online service with high real-time requirements, then optimal performance may be the primary scheduling objective, and optimal cost may be the secondary scheduling objective. The first weight of different scheduling objectives can be manually set by the user or automatically determined based on the input information. For example, the user might set the first weight for optimal performance to 0.7 and the first weight for optimal cost to 0.3. In one example, the scheduling objective x of the inference service, based on the input information of the large model, can be expressed as follows: Formula 1:

[0060] x=∑λ i ×obj(i) Formula 1

[0061] Where, λ i Let ∑λ be the first weight of the i-th scheduling target, and obj(i) be the i-th scheduling target. i =1.

[0062] In this embodiment, it is also necessary to determine the second weight of each scheduling condition for the scheduling target based on the correlation between each scheduling target and the scheduling condition. For example, for the performance-optimal scheduling target, the target throughput has a significant impact on it; therefore, the second weight of the scheduling condition corresponding to the target throughput can be set to 0.8. For the cost-optimal scheduling target, the hardware state has a significant impact on it; therefore, the second weight of the scheduling condition corresponding to the hardware state can be set to 0.9. In one example, based on the correlation between each scheduling target and the scheduling condition, each scheduling target obj(i) can be represented by the following formula 2:

[0063] obj(i)=∑w j ×condition(j) Formula 2

[0064] Among them, w j Let w be the second weight of the j-th scheduling condition on the scheduling target obj(i), and condition(j) be the j-th scheduling condition. j ∈[0,1],∑w j =1.

[0065] In this embodiment, it is also necessary to determine the weight of each scheduling condition based on the first weight and the second weight. In one example, the weight of each scheduling condition can be determined based on the weighted average method. For example, if the first weight of scheduling target A is 0.7 and the second weight of scheduling condition X on scheduling target A is 0.8, then the weight of scheduling condition X can be calculated as 0.7 × 0.8 = 0.56.

[0066] In another example, for each inference service, its scheduling target X, taking into account scheduling conditions, can be represented based on the following Formula 3:

[0067] X=∑λ i ×obj(i)=∑ i λ i ×(∑ j w j ×condition(j)) Formula 3

[0068] After normalizing Equation 3, the final scheduling target X can be represented by the following Equation 4:

[0069] X=∑ i w′ i ×condition(i) Formula 4

[0070] Among them, w′ i Let w' be the weight of the i-th scheduling condition, condition(i) be the i-th scheduling condition, and w' be the weight of the i-th scheduling condition. i ∈[0,1],∑ i w′ i =1.

[0071] In this disclosure, based on the input information of the large model, the scheduling target and the first weight of the scheduling target corresponding to the inference service are determined, and based on the correlation between the scheduling target and the scheduling conditions, the second weight of each scheduling condition on the scheduling target is determined. Then, based on the first weight and the second weight, the weight of the scheduling condition is determined, which can obtain a more reasonable weight, thereby better balancing the relationship between the scheduling target and different scheduling conditions, and improving the performance and resource utilization efficiency of the large model after inference scheduling.

[0072] In another embodiment, the scheduling strategy for the scheduling conditions corresponding to the load of the inference service includes:

[0073] In response to an increase in the load on the inference service, the data parallelism in the current resource configuration of the large model is reduced by the sixth threshold, and the tensor parallelism in the current resource configuration of the large model is increased by the sixth threshold.

[0074] In response to the reduced load on the inference service, the data parallelism in the current resource configuration of the large model is increased by the sixth threshold, and the tensor parallelism in the current resource configuration of the large model is decreased by the sixth threshold.

[0075] In this embodiment, if the scheduling condition corresponding to the load of the inference service is triggered, the scheduling policy corresponding to the scheduling condition needs to be executed. The corresponding scheduling policies are divided into policies when the load of the inference service increases and policies when the load of the inference service decreases.

[0076] Specifically, if the load on the inference service increases, the sixth threshold is lowered for the data parallelism (DP) in the current resource configuration of the large model, while the sixth threshold is increased for the tensor parallelism (TP). Data parallelism refers to the degree of parallelism in processing multiple data replicas simultaneously; reducing data parallelism can decrease communication overhead between model replicas and improve the computational efficiency of a single replica. Tensor parallelism refers to the degree to which tensors are partitioned and computed in parallel within the model; increasing tensor parallelism can fully utilize hardware resources and improve the model's computational power. For example, assuming the current data parallelism is 4 and tensor parallelism is 2, when the load increases, the data parallelism can be reduced to 3 while the tensor parallelism is increased to 3 to better adapt to changes in the inference service load.

[0077] Specifically, if the load on the inference service decreases, the data parallelism in the current resource configuration of the large model is increased by a sixth threshold, while the tensor parallelism is decreased by a sixth threshold. Increasing data parallelism improves the model's processing power and better utilizes hardware resources; decreasing tensor parallelism avoids resource waste and performance degradation caused by excessive parallelism under light loads. For example, assuming the current data parallelism is 3 and tensor parallelism is 3, when the load decreases, the data parallelism can be increased to 4 while the tensor parallelism is decreased to 2 to adapt to the load change.

[0078] In another embodiment, the scheduling strategy for the scheduling conditions corresponding to the type of inference service includes:

[0079] In response to the inference service becoming input-intensive, a seventh threshold is added to the number of first execution replicas in the current resource allocation of the large model; the first execution replicas are used to process the input of the inference service.

[0080] In response to the inference service type becoming output-intensive, a seventh threshold is added to the number of second execution replicas in the current resource configuration of the large model; the second execution replicas are used to generate the output of the inference service.

[0081] In this embodiment, if the scheduling condition corresponding to the type of the inference service is triggered, the scheduling policy corresponding to the scheduling condition needs to be executed. The corresponding scheduling policies are divided into a policy that changes the type of the inference service to input-intensive and a policy that changes the type of the inference service to output-intensive.

[0082] If the inference service becomes input-intensive, the number of first execution copies needs to be increased. The first execution copies are primarily used to process the input of the inference service, such as data preprocessing and feature extraction. In one example, the first execution copy could be a prefill copy. Prefill copies are mainly used to process the input data of large models and build initial context information. Increasing the number of first execution copies can improve the efficiency of input processing, thus better adapting to the needs of input-intensive tasks. For example, assuming the current number of first execution copies is 2, when the inference service becomes input-intensive, the number of first execution copies can be increased to 4 to improve input processing capacity.

[0083] If the inference service becomes output-intensive, the number of second execution copies needs to be increased. These second execution copies are primarily used to generate the output of the inference service, such as text generation or image generation. In one example, the second execution copy could be a Decode copy. The Decode copy is mainly used to gradually generate the output of the large model based on the output of the Prefill copy. Increasing the number of second execution copies can improve the efficiency of output generation, thus better adapting to the needs of output-intensive tasks. For example, assuming the current number of second execution copies is 2, when the inference service becomes output-intensive, the number of second execution copies can be increased to 4 to improve output generation capabilities.

[0084] In another embodiment, the scheduling strategy for the scheduling conditions corresponding to the target throughput includes:

[0085] The number of first and second execution replicas in the current resource configuration of the large model is increased by an eighth threshold.

[0086] In this embodiment, if the scheduling condition corresponding to the target throughput is triggered, the scheduling policy corresponding to that condition needs to be executed. The corresponding scheduling policy is to increase the data of both the first and second execution replicas in the current resource configuration of the large model by an eighth threshold. The first and second execution replicas are used to process the input information and generate output information for the inference service, respectively. Increasing the number of replicas can improve their processing capacity, thereby helping to increase the overall throughput of the inference service and alleviating hardware pressure. For example, assuming the current number of the first and second execution replicas is 1 and 2 respectively, when the scheduling condition corresponding to the target throughput is triggered, the number of the first execution replica can be increased to 2, and the number of the second execution replica can be increased to 3 to improve throughput.

[0087] In another embodiment, the scheduling strategy for the scheduling conditions corresponding to the hardware state includes:

[0088] The number of first and second execution replicas in the current resource configuration of the large model is reduced by the ninth threshold.

[0089] In this embodiment, if the scheduling condition corresponding to the hardware state is triggered, the scheduling policy corresponding to that condition needs to be executed. The corresponding scheduling policy is to reduce the number of both the first and second execution replicas in the current resource configuration of the large model by the ninth threshold. Reducing the number of replicas can optimize resource utilization efficiency and avoid resource waste. For example, assuming that the current number of the first and second execution replicas are 4 and 4 respectively, when the scheduling condition corresponding to the hardware state is triggered, the number of the first execution replica can be reduced to 3, and the number of the second execution replica can be reduced to 3 to release hardware resources, avoid hardware resource overload, and ensure the stable operation of the inference service. It should be emphasized that the scheduling condition corresponding to the hardware state being triggered here refers to the hardware state being less than the third threshold, such as GPU utilization being less than 10%, that is, the GPU load is low; if the hardware state is greater than the third threshold, such as GPU utilization exceeding 90%, the corresponding scheduling policy is the same as the scheduling policy corresponding to the target throughput, which will not be elaborated here.

[0090] For ease of understanding of this disclosure, the following is based on Figure 3 and Figure 4 The overall process of a large-scale model inference scheduling method disclosed herein is explained as follows:

[0091] Figure 3 This illustration shows a flowchart of a large model inference scheduling method according to an embodiment of the present disclosure. Figure 3 ,like Figure 3 As shown, users input data onto a large model, and a monitoring system monitors the service performance parameters of the large model during inference services. A large model inference scheduling method includes:

[0092] Step S301: Obtain input information. That is, obtain the user's input information on the large model.

[0093] Step S302 involves determining the weights of the scheduling conditions based on the scheduling objective. That is, the scheduling objective can be determined based on the input information, and the weights of the scheduling conditions can be determined based on the scheduling objective. The specific implementation details of step S302 have been discussed above and will not be repeated here.

[0094] In one possible implementation, after obtaining the input information, a default attribute f(x,y,z) can be assigned to the inference service based on the model parameters, where:

[0095] x represents the maximum sequence length supported by the large model (max sequence num), y represents the number of attention heads in the large model, and z is the size of the large model. A larger f(x,y,z) indicates that the model tends to process long input text, setting its default inference service attributes to input-intensive and high-load tasks; conversely, a smaller f(x,y,z) sets the task to output-intensive and low-load tasks.

[0096] Then, based on the inference service attribute f(x,y,z), multiple resource configurations that can be obtained under the current resources can be determined. These multiple resource configurations are shown in Table 1.

[0097] Table 1

[0098]

[0099]

[0100] Then, based on performance parameters and scheduling objectives derived from input information, an initial resource configuration can be determined from multiple resource configurations. For example, if the scheduling objectives are: optimal performance with a first weight of 0.7 and optimal cost with a first weight of 0.3, then the top n resource configurations in terms of performance data can be determined first, and then the resource configuration with the lowest cost can be selected from the top n resource configurations in terms of performance data as the initial resource configuration.

[0101] After determining the initial resource configuration, inference services can be performed based on the initial resource configuration. During the inference service of the large model, the monitoring system monitors the service performance parameters.

[0102] Step S303: Obtain service performance parameters.

[0103] Step S304: Obtain service performance metrics.

[0104] Step S305: Trigger scheduling conditions. That is, there are scheduling conditions that meet the service performance indicators.

[0105] If the number of triggered scheduling conditions is no more than 1, that is, only one scheduling condition is triggered, then step S306 can be executed directly to execute the scheduling policy corresponding to the scheduling condition.

[0106] If the number of triggered scheduling conditions is greater than 1, then the weight of each scheduling condition needs to be determined:

[0107] If there is a scheduling condition with a weight greater than the fifth threshold, then step S308 is executed, and the scheduling policy corresponding to the scheduling condition with the largest weight is executed.

[0108] If multiple scheduling conditions have weights greater than the fourth threshold, then step S309 is executed, and the corresponding scheduling strategies are executed in descending order of weight. That is, the scheduling strategies corresponding to the scheduling conditions are executed based on their weights. The fourth threshold is less than the fifth threshold.

[0109] If all triggered scheduling conditions are less than the fourth threshold, then step S310 is executed, which only counts and waits for processing in the next cycle. That is, the current resource configuration of the large model is not adjusted this time.

[0110] Figure 4 This illustration shows a flowchart of a large model inference scheduling method according to an embodiment of the present disclosure. Figure 4 ,like Figure 4 As shown, after the scheduling conditions are triggered in step S401, if one of the scheduling conditions corresponding to the target throughput and the scheduling conditions corresponding to the type of inference service, the load of inference service, and the hardware status is triggered simultaneously, that is, the target throughput decreases, accompanied by one of the following: a change in the type of inference service, a change in the load of inference service, and excessive hardware load, then the corresponding scheduling strategy can be executed.

[0111] If the target throughput decreases, accompanied by a change in the type of inference service, and the inference service becomes an input-intensive task, then step S402 is executed, increasing the number of the first execution replicas by the seventh threshold. If the inference service becomes an output-intensive task, then step S403 is executed, increasing the number of the second execution replicas by the seventh threshold. If adjusting the number of the first or second execution replicas restores the target throughput to its normal value, then the current scheduling ends; otherwise, monitoring of service performance parameters continues.

[0112] If the target throughput decreases, accompanied by a change in the load of the inference service, and the load of the inference service increases, then step S404 is executed: the data parallelism is decreased by the sixth threshold, and the tensor parallelism is increased by the sixth threshold. If the load of the inference service decreases, then step S405 is executed: the data parallelism is increased by the sixth threshold, and the tensor parallelism is decreased by the sixth threshold. If the target throughput recovers after adjusting the data parallelism and tensor parallelism, that is, the target throughput returns to its normal value, then the current scheduling ends; otherwise, the monitoring of service performance parameters continues.

[0113] If the target throughput decreases while the hardware load is too high, step S406 is executed, increasing the number of both the first and second execution replicas by the eighth threshold. If the target throughput recovers after adjusting the number of the first and second execution replicas, i.e., the target throughput returns to normal, the current scheduling ends; otherwise, monitoring of service performance parameters continues.

[0114] If a scheduling condition of low hardware load is triggered alone, step S407 is executed, reducing the number of both the first and second execution replicas by the ninth threshold, while retaining the released resources. In other words, the resources released after reducing the number of the first and second execution replicas are temporarily retained so that they can be quickly reallocated when needed.

[0115] If the throughput does not decrease after both the number of the first and second execution replicas are reduced to the ninth threshold, then the current resources are still excessive, and step S407 can be further executed. If the throughput decreases, then step S408 is executed to roll back the operation, redeploying the first and second execution replicas on the reserved resources. In other words, if the reduction of the first and second execution replicas causes performance issues, then a rollback operation is performed, reallocating the resources reserved in step S407 to the first and second execution replicas. The adjustment process ends when system resources and target throughput reach a balance.

[0116] Figure 5 A schematic diagram of the composition structure of an electronic device according to an embodiment of the present disclosure is shown, such as... Figure 5 As shown, an electronic device includes:

[0117] Scheduling unit 10;

[0118] Multiple processors are used to execute inference services for large models; multiple processors are such as processor 1, processor 2, ..., processor n;

[0119] Scheduling unit 10 is used to execute:

[0120] Obtain the service performance parameters corresponding to the inference service of the large model within the target time period; the service performance parameters include the input data length, output data length, throughput, service standard, and hardware status parameters supporting the inference service;

[0121] Based on the service performance parameters, determine the service performance indicators corresponding to the inference service; the service performance indicators include the target throughput of the inference service, the load of the inference service, the type of the inference service, and the hardware status that supports the inference service.

[0122] Based on the scheduling strategy corresponding to the scheduling conditions met by the service performance indicators, the current resource configuration of the large model is adjusted to obtain the target resource configuration;

[0123] Each scheduling condition has a corresponding weight, which is obtained based on the relationship between the scheduling target and the scheduling condition corresponding to the inference service.

[0124] In one possible implementation, the scheduling conditions include at least one of the following: the target throughput of the inference service is less than a first threshold; the change in the load of the inference service is greater than a second threshold; the type of the inference service changes; or the hardware state meets a third threshold.

[0125] In one embodiment, the scheduling unit 10 is further configured to: in response to the existence of multiple scheduling conditions with weights greater than the fourth threshold, determine the execution order of the scheduling strategies corresponding to the scheduling conditions with weights greater than the fourth threshold based on the weights of the scheduling conditions; and, based on the execution order, execute the scheduling strategies corresponding to the scheduling conditions with weights greater than the fourth threshold to adjust the current resource configuration of the large model and obtain the target resource configuration.

[0126] In one embodiment, the scheduling unit 10 is further configured to: in response to the existence of a scheduling condition with a weight greater than the fifth threshold, adjust the current resource configuration of the large model based on the scheduling strategy corresponding to the scheduling condition with the largest weight, and obtain the target resource configuration; the fifth threshold is greater than the fourth threshold.

[0127] In one embodiment, the scheduling unit 10 is further configured to perform: determining the scheduling target and a first weight of the scheduling target corresponding to the inference service based on the input information of the large model; determining a second weight of each scheduling condition on the scheduling target based on the correlation between the scheduling target and the scheduling conditions; and determining the weight of the scheduling conditions based on the first weight and the second weight.

[0128] In one embodiment, the scheduling unit 10 is further configured to: in response to an increase in the load of the inference service, decrease the data parallelism in the current resource configuration of the large model by a sixth threshold and increase the tensor parallelism in the current resource configuration of the large model by a sixth threshold; in response to a decrease in the load of the inference service, increase the data parallelism in the current resource configuration of the large model by a sixth threshold and decrease the tensor parallelism in the current resource configuration of the large model by a sixth threshold.

[0129] In one embodiment, the scheduling unit 10 is further configured to: in response to the inference service becoming input-intensive, increase the number of first execution replicas in the current resource configuration of the large model by a seventh threshold; the first execution replicas are used to process the inputs of the inference service; in response to the inference service becoming output-intensive, increase the number of second execution replicas in the current resource configuration of the large model by a seventh threshold; the second execution replicas are used to generate the outputs of the inference service.

[0130] In one implementation, the scheduling unit 10 is also configured to: increase the number of both the first and second execution replicas in the current resource configuration of the large model by an eighth threshold.

[0131] In one implementation, the scheduling unit 10 is also configured to: reduce the number of both the first and second execution replicas in the current resource configuration of the large model by a ninth threshold.

[0132] Figure 6 A schematic diagram of a scenario for an electronic device according to an embodiment of the present disclosure is shown, such as... Figure 6 As shown, user-submitted jobs (i.e., input information) Job1, Job2, ..., Jobn first enter the job queue, awaiting processing. The job preprocessing component retrieves jobs from the queue and performs preprocessing, such as preliminary analysis and classification of job attributes. The performance analysis component performs more in-depth performance analysis of the jobs, including parallel configuration analysis, PD replica (i.e., first and second execution replicas) orchestration analysis, and heterogeneous GPU cost analysis. The runtime management component manages the runtime environment, including model loading (VLLM), compilation (sglang), etc., and sends the Service Levels (SLOs), input data lengths (Input tokens), and output data lengths (Output tokens) that need to be monitored to the monitoring component. The monitoring component monitors the values ​​of the parameters sent by the runtime management component in real time. The evaluator evaluates the job's performance and resource requirements based on the data provided by the performance analysis and runtime management components. The control layer includes a lightweight rescheduling component, used to perform dynamic adjustments based on the data obtained from the monitoring component and the evaluation results, such as adjusting the number of PD replicas, parallelism, and hardware resource types. The scheduler resides in the infrastructure layer and is responsible for distributing jobs to different Pods within the Kubernetes cluster for execution. The lightweight rescheduling component in the control layer can serve as... Figure 5 The scheduling unit 10 in the device cluster allows each device (or Pod) to contain one or more processors. Figure 5 The processors (1-n) are used to perform inference services for large models.

[0133] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0134] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0135] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A large-scale model inference scheduling method, comprising: Obtain the service performance parameters corresponding to the inference service of the large model within the target time period; The service performance parameters include the input data length, output data length, throughput, service standard, and hardware status parameters supporting the inference service. Based on the service performance parameters, the service performance indicators corresponding to the inference service are determined; the service performance indicators include the target throughput of the inference service, the load of the inference service, the type of the inference service, and the hardware status supporting the inference service. Based on the scheduling strategy corresponding to the scheduling conditions satisfied by the service performance indicators, the current resource configuration of the large model is adjusted to obtain the target resource configuration; Each of the scheduling conditions has a corresponding weight, which is obtained based on the correlation between the scheduling target corresponding to the inference service and the scheduling condition.

2. The method according to claim 1, wherein the scheduling conditions include at least one of the following: The target throughput of the inference service is less than a first threshold; The change in the load of the inference service is greater than the second threshold. The type of inference service changes; The hardware state meets the third threshold.

3. The method according to claim 1, wherein adjusting the current resource configuration of the large model based on the scheduling strategy corresponding to the scheduling conditions satisfied by the service performance indicators to obtain the target resource configuration includes: In response to the existence of multiple scheduling conditions with weights greater than the fourth threshold, the execution order of the scheduling policy corresponding to the scheduling conditions with weights greater than the fourth threshold is determined based on the weights of the scheduling conditions. Based on the execution order, the scheduling policy corresponding to the scheduling condition with an execution weight greater than the fourth threshold is used to adjust the current resource configuration of the large model to obtain the target resource configuration.

4. The method according to claim 3, further comprising: In response to the existence of a scheduling condition with a weight greater than the fifth threshold, the current resource configuration of the large model is adjusted based on the scheduling strategy corresponding to the scheduling condition with the largest weight to obtain the target resource configuration; the fifth threshold is greater than the fourth threshold.

5. The method according to claim 1, wherein the weights corresponding to the scheduling conditions are determined based on the following method: Based on the input information of the large model, the scheduling target corresponding to the inference service and the first weight of the scheduling target are determined; Based on the correlation between the scheduling target and the scheduling conditions, a second weight for each scheduling condition to the scheduling target is determined; The weights of the scheduling conditions are determined based on the first weight and the second weight.

6. The method according to claim 1, wherein the scheduling strategy for the scheduling conditions corresponding to the load of the inference service includes: In response to an increase in the load on the inference service, the data parallelism in the current resource configuration of the large model is reduced by a sixth threshold, and the tensor parallelism in the current resource configuration of the large model is increased by a sixth threshold. In response to a decrease in the load on the inference service, the data parallelism in the current resource configuration of the large model is increased by a sixth threshold, and the tensor parallelism in the current resource configuration of the large model is decreased by a sixth threshold.

7. The method according to claim 1, wherein the scheduling strategy for the scheduling conditions corresponding to the type of the inference service includes: In response to the inference service becoming input-intensive, the number of first execution replicas in the current resource configuration of the large model is increased by a seventh threshold. The first execution copy is used to process the input of the inference service; In response to the inference service becoming output-intensive, the number of second execution replicas in the current resource configuration of the large model is increased by a seventh threshold. The second execution copy is used to generate the output of the inference service.

8. The method according to claim 1, wherein the scheduling strategy for the scheduling conditions corresponding to the target throughput includes: The number of the first and second execution replicas in the current resource configuration of the large model is increased by an eighth threshold.

9. The method according to claim 1, wherein the scheduling strategy for the scheduling conditions corresponding to the hardware state includes: The number of the first and second execution replicas in the current resource configuration of the large model is reduced by the ninth threshold.

10. An electronic device, comprising: Scheduling unit; Multiple processors, the multiple processors being used to execute inference services for large models; The scheduling unit is used to execute: Obtain the service performance parameters corresponding to the inference service of the large model within the target time period; the service performance parameters include the input data length, output data length, throughput, service standard, and hardware status parameters supporting the inference service; Based on the service performance parameters, the service performance indicators corresponding to the inference service are determined; the service performance indicators include the target throughput of the inference service, the load of the inference service, the type of the inference service, and the hardware status supporting the inference service. Based on the scheduling strategy corresponding to the scheduling conditions satisfied by the service performance indicators, the current resource configuration of the large model is adjusted to obtain the target resource configuration; Each of the scheduling conditions has a corresponding weight, which is obtained based on the correlation between the scheduling target corresponding to the inference service and the scheduling condition.

Images