High-throughput large model reasoning method based on accuracy scaling

Abstract

The invention discloses a high-throughput large model reasoning method based on accuracy scaling, and belongs to the field of artificial intelligence of computer science. The method comprises the following steps: according to historical user request quantity data, predicting a user request quantity of a next time period through an EWMA prediction algorithm; inputting the predicted request quantity into a strategy generation algorithm based on a complete knapsack problem, allocating acceleration cards by using a resource allocation mechanism, and converting a multi-task problem into a single-task sub-problem; modeling the single task sub-problem into a complete knapsack problem; after all the complete backpack problems are solved one by one, a scheduling strategy result is output to a large model reasoning system; according to a model scheduling strategy, smooth transition of model replacement is realized by adopting a batch progressive adjustment mode and a mode of alternately adjusting cluster resources in proportion. According to the method, the inference model in the large model inference system is adaptively adjusted through the accuracy scaling technology, the throughput of the large model inference system is improved at zero cost, and the SLO conflict rate is reduced.

Description

Technical Field

[0001] This invention relates to the field of computer science and artificial intelligence, and in particular to a high-throughput large model inference method based on accuracy scaling. Background Technology

[0002] Large-scale model inference systems (LLMs) have become central to the field of artificial intelligence, demonstrating unprecedented capabilities in natural language processing, code generation, and multimodal interaction. However, with the explosive growth in the adoption of large models, large-scale model inference systems often struggle to simultaneously guarantee Service Level Response (SLO) and support dynamically changing peak user request volumes. Existing cloud service providers use dynamic hardware scaling to address these issues. However, with the exponential growth in the parameter size of large models, the demand for accelerator cards for individual model replica inference increases continuously, leading to enormous deployment costs for cloud service providers. Furthermore, due to the dynamic nature of user request volumes, cloud service providers' cluster resources need to be continuously scaled up and down at the hardware level based on user request volumes, resulting in wasted cluster resources. This approach is particularly unsuitable for large-scale model inference systems with limited cluster resources. Therefore, designing a high-throughput model scheduling optimization method for large-scale model inference systems to reduce deployment costs for cloud service providers has become crucial. Summary of the Invention

[0003] To address the issue that large model inference systems cannot simultaneously guarantee SLO and maintain high throughput under dynamic load conditions, the present invention aims to provide a high-throughput large model inference method based on accuracy scaling.

[0004] The objective of this invention is achieved through the following technical solution: a high-throughput large-model inference method based on accuracy scaling, comprising:

[0005] Based on historical user request volume data, the EWMA prediction algorithm is used to predict the user request volume for the next period in advance.

[0006] Based on the predicted user request volume for the next time period, a model scheduling strategy for the next time period is generated; the accelerator card is allocated through a resource allocation mechanism based on user request volume, transforming the multi-task problem into a single-task sub-problem.

[0007] The single-task sub-problem is modeled as a complete knapsack problem. After solving the complete knapsack problem one by one based on the dynamic programming algorithm, the model scheduling strategy result is output to the large model inference system.

[0008] According to the model scheduling strategy, a phased and gradual adjustment approach is adopted, and a smooth transition for model replacement is achieved by adjusting cluster resources proportionally.

[0009] Furthermore, the method of allocating accelerator cards through a resource allocation mechanism based on user request volume, transforming the multi-task problem into a single-task sub-problem, includes:

[0010] Calculate and allocate the minimum number of accelerator cards required to meet the user request volume for different tasks. Since the model with higher throughput corresponds to a smaller parameter scale and fewer accelerator cards are used, the minimum number of accelerator cards used for each task only needs to consider the case of a single model and does not need to consider the combination of multiple models.

[0011] The remaining accelerator cards are allocated based on the number of user requests, and the final number of accelerator cards allocated to each type of task is calculated.

[0012] Furthermore, the step of modeling the single-task sub-problem as a complete knapsack problem, solving the complete knapsack problem one by one based on a dynamic programming algorithm, and then outputting the model scheduling strategy result to the large model inference system includes:

[0013] Determine the prediction domain for user request volume using the amplification factor;

[0014] Dynamic programming is performed based on the maximum value of the prediction domain. The dynamic programming includes three loops: the outermost loop iterates through candidate models, the middle loop iterates through the total number of user requests from 0 to the maximum value, and the innermost loop iterates through the number of accelerator cards from 0 to the total number. That is, sub-problems are processed first, and then the original problem is solved based on the results of the sub-problems. Inside the loop, the maximum value obtained by using or not using the current candidate model is compared to determine whether to use the current candidate model. If the maximum value is obtained by using the current candidate model, the candidate model is recorded. After the dynamic programming iteration is completed, the optimal scheduling scheme with all throughputs in the prediction domain is found based on the average performance.

[0015] The number of replicas of the candidate model is calculated by back-calculating the candidate model information recorded during dynamic programming execution, and the final scheduling strategy is generated.

[0016] Furthermore, the method of proportionally rotating and adjusting cluster resources includes: prioritizing the loading of models with parameter sizes smaller than preset values.

[0017] The present invention also provides a high-throughput large model inference system based on accuracy scaling, comprising one or more processors for implementing the high-throughput large model inference method based on accuracy scaling.

[0018] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the high-throughput large model inference method based on accuracy scaling.

[0019] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the high-throughput large model inference method based on accuracy scaling.

[0020] The beneficial effects of this invention are as follows: This invention utilizes accuracy scaling technology to design a high-throughput model scheduling optimization method for large-model inference systems. It achieves a good trade-off between model performance and throughput through a policy generation algorithm based on the complete knapsack problem. It achieves a smooth transition during model replacement by adjusting cluster resources in a proportional rotation manner. The overall effect is to increase the throughput of large-model inference systems at zero cost and reduce the SLO conflict rate. The computational model scheduling strategy of this invention requires only seconds and can handle large-model inference system scenarios. Attached Figure Description

[0021] Figure 1 This is a design framework diagram of the present invention;

[0022] Figure 2 The diagram shows the test results of this invention and other solutions. Detailed Implementation

[0023] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0024] The terms used in this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention. The singular forms "a," "the," and "the" used in this invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0025] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0026] like Figure 1 As shown, this embodiment of the invention provides a high-throughput large model inference method based on accuracy scaling. Table 1 shows the modeling variables and symbols that appear in this embodiment of the invention.

[0027] Table 1: Variables and symbols used in modeling

[0028] Variables and symbols describe N_tasks Number of task types α User Request Volume Increase Factor <![CDATA[R i ]]> User request volume for the i-th type of task N_gpus Total number of accelerator cards in the computing cluster <![CDATA[GPU i ]]> Number of accelerator cards used to process the i-th type of task in the current time period M Set of all candidate models <![CDATA[M i ]]> The set of candidate models that can handle the i-th task <![CDATA[M i,j ]]> The j-th candidate model in the candidate model set for the i-th task gpu_cost Number of accelerator cards required for candidate model inference t Throughput of candidate models p Model performance of candidate models v The composite attribute of the candidate model: the product of throughput t and model performance p. <![CDATA[cnt i,j ]]> The number of replicas of the j-th candidate model in the candidate model set for the i-th task in the scheduling strategy.

[0029] The method specifically includes the following steps:

[0030] Step 1: Predict the user request volume for the next time period in advance using an EWMA-based user request volume prediction algorithm, calculated using the following recursive formula:

[0031] R (k+1)′ =β·R k +(1-β)·R k ′.

[0032] Among them, R k+1’ R represents the predicted user request volume for the (k+1)th time period. k’ R represents the predicted user request volume for the k-th time period. k This represents the actual number of user requests in the k-th time period for any task; β is a smoothing factor (0 < β ≤ 1), used to control the decay rate of historical data.

[0033] Step 2: Input the predicted user request volume into the policy generation algorithm based on the complete knapsack problem. Allocate accelerator cards through a resource allocation mechanism based on user request volume, transforming the multi-task problem into a single-task sub-problem, as shown in Algorithm 1:

[0034]

[0035]

[0036] The input to Algorithm 1 is the total number of accelerator cards N_gpus in the large model inference system and the number of user requests R for all tasks. The output of the algorithm is the scheduling policy cnt, and it can be divided into the following sub-steps:

[0037] 2.1. Calculate the minimum number of accelerator cards required to meet the user request volume for each task (lines 02-08). Since the model with higher throughput has a smaller parameter scale and uses fewer accelerator cards, the minimum number of accelerator cards used for each task only needs to consider the case of a single model, and does not need to consider the combination of multiple models.

[0038] 2.2. Allocate the remaining acceleration cards according to the number of user requests, and calculate the final number of acceleration cards allocated for each type of task (lines 11-12), transforming the multi-task large model inference system problem into a single-task large model inference system problem.

[0039] Step 3: The policy generation algorithm based on the complete knapsack problem models the single-task subproblems as complete knapsack problems. This achieves the goal of increasing the throughput of the large model inference system while minimizing the degradation of the average performance of the large model through accuracy scaling techniques. After solving all single-task subproblems one by one, the scheduling policy results are output back to the large model inference system, as shown in Algorithm 2:

[0040]

[0041]

[0042] The input to Algorithm 2 is the number of accelerator cards (n_gpus) in the large model inference system and the total number of user requests (R0) for task 0 (using task 0 as an example for a single task). The output of the algorithm is the optimal scheduling policy (cnt). 0，j It can be divided into the following sub-steps:

[0043] 3.1. Use the amplification factor α (here α = 1.25) to determine the prediction domain of user request volume [R0, R0*α] (line 01).

[0044] 3.2. A three-loop dynamic programming approach is used based on the maximum value R' in the prediction domain. The outermost loop iterates through candidate models, the middle loop iterates through the total number of user requests from 0 to R', and the innermost loop iterates through the number of accelerator cards from 0 to n_gpus. In other words, sub-problems are processed first, and then the original problem is solved based on the results of the sub-problems. Inside each loop, the maximum value obtained by using or not using the current model is compared (lines 07-14) to determine whether to use the current candidate model. If the maximum value is obtained by using the candidate model, it is recorded. After the dynamic programming iterations are completed, the optimal scheduling scheme with all throughputs within the prediction domain is found based on the average performance (lines 16-18).

[0045] 3.3. Based on dp[max_v][N] and the candidate model information recorded in the record, the number of replicas of the candidate model is inferred, thereby generating the final scheduling strategy (lines 20-24). Since all scheduling schemes that keep the total throughput within the user request prediction domain are desirable, and this algorithm selects the scheduling scheme with the highest average performance, the final scheduling scheme achieves the optimal value of average performance while ensuring throughput. This achieves the goal of increasing the throughput of the large model inference system while minimizing the decrease in the average performance of the large model.

[0046] Step 4: Based on the output model scheduling strategy, a phased and gradual adjustment approach is adopted. This involves adjusting cluster resources proportionally to achieve a smooth transition during model replacement. Since the new scheduling strategy is generated before the start of the next time period, a phased and gradual adjustment strategy is used, meaning model updates are performed in multiple rounds, with each round updating only a certain percentage of the model (e.g., five rounds, updating 20% ​​of the model at a time). Furthermore, models with smaller parameter sizes in the new scheduling strategy should be loaded as early as possible. This can potentially increase the throughput of the large model inference system after a partial update, reducing the SLO conflict rate. Conversely, models with larger parameter sizes in the new scheduling strategy should be loaded after the throughput of the large model inference system has increased.

[0047] Example 1

[0048] To verify the feasibility of this method, the following specific experiments were conducted in this embodiment of the invention:

[0049] Experimental setup:

[0050] (1) Operating system: Ubuntu 18.04.3LTS;

[0051] (2) CPU: Model 8-core Intel(R) Xeon(R) Gold 6126 CPU@2.60GHz, equipped with 32GB DRAM;

[0052] (3) GPU: 64 Tesla V100 GPUs with 32GB of video memory;

[0053] (4) Storage devices: 512GB SK hynix SC311 SATA SSD; Western Digital WDCWD40EZRZ-75G HDD; Intel Optane NVM 256GB;

[0054] Model configuration:

[0055] Task types: text generation, multimodal understanding and generation, code generation and completion, mathematical and symbolic reasoning, instruction following and task planning, dialogue systems and multi-turn interaction, information retrieval and question answering.

[0056] Model:

[0057]

[0058]

[0059] Dataset: Using Twitter's open-source user request dataset.

[0060] This invention is compared with other conventional solutions, such as... Figure 2As shown. The blue curve represents the actual user request volume changing over time, while the other curves represent the changes over time under different baselines and Dp (in this invention) under different metrics. The test results show that:

[0061] In terms of throughput, Dp, INFaaS, and Proteus all demonstrated the ability to quickly adjust to the gradual fluctuations in actual user request volume. Only Clipper's throughput remained consistently low, failing to flexibly respond to changes in actual user request volume. Regarding effective accuracy, Dp consistently maintained a very high effective accuracy. INFaaS experienced a decrease in effective accuracy when actual user request volume increased. Proteus's effective accuracy remained unstable, while Clipper's effective accuracy remained very low. In terms of maximum accuracy decline, Dp exhibited the lowest decline, while INFaaS, Proteus, and Clipper's declines were 4, 6, and 5 times that of Dp, respectively. Regarding SLO violation rate, Dp had the lowest SLO violation rate, while INFaaS and Proteus's SLO violation rates were 2.5 and 1.5 times that of Dp, respectively. Clipper's SLO violation rate was very high.

[0062] Considering the above indicators, Dp (this invention) is optimal in all aspects. It can not only use accuracy scaling technology to flexibly change the throughput of the large model inference system according to the actual changes in user request volume, and minimize the SLO conflict rate, achieving a 1.5 times improvement compared to the best solution of the baselines, but also minimize the reduction in the average performance of the large model inference system and minimize the drop in maximum accuracy, achieving a 4 times improvement compared to the best solution of the baselines.

[0063] This invention also provides a high-throughput large model inference system based on accuracy scaling, including one or more processors, for implementing the high-throughput large model inference method based on accuracy scaling described in any embodiment.

[0064] It should be noted that the system embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, and will not be repeated here.

[0065] This invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the high-throughput large model inference method based on accuracy scaling as described in any embodiment.

[0066] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the high-throughput large model inference method based on accuracy scaling described in any embodiment.

[0067] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0068] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0069] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A high-throughput large model inference method based on accuracy scaling, characterized in that, Includes the following steps: Based on historical user request volume data, the EWMA prediction algorithm is used to predict the user request volume for the next period in advance. Based on the predicted user request volume for the next time period, a model scheduling strategy is generated for the next time period. By allocating accelerator cards based on a resource allocation mechanism that takes into account the volume of user requests, the multi-task problem is transformed into a single-task sub-problem. The single-task sub-problem is modeled as a complete knapsack problem. After solving the complete knapsack problem one by one based on the dynamic programming algorithm, the model scheduling strategy result is output to the large model inference system. According to the model scheduling strategy, a phased and gradual adjustment approach is adopted, and a smooth transition for model replacement is achieved by adjusting cluster resources proportionally.

2. The high-throughput large model inference method based on accuracy scaling according to claim 1, characterized in that, The method of allocating accelerator cards based on user request volume, transforming the multi-task problem into a single-task sub-problem, includes: Calculate and allocate the minimum number of accelerator cards required to meet the user request volume for each different task. The remaining accelerator cards are allocated based on the number of user requests, and the final number of accelerator cards allocated to each type of task is calculated.

3. The high-throughput large model inference method based on accuracy scaling according to claim 1, characterized in that, The process of modeling a single-task sub-problem as a complete knapsack problem, solving the complete knapsack problem one by one using a dynamic programming algorithm, and then outputting the model scheduling strategy results to the large model inference system includes: Determine the prediction domain for user request volume using the amplification factor; Dynamic programming is performed based on the maximum value of the prediction domain. The dynamic programming includes three loops: the outermost loop iterates through candidate models, the middle loop iterates through the total number of user requests from 0 to the maximum value, and the innermost loop iterates through the number of accelerator cards from 0 to the total number. Within each loop, the maximum value obtained by using or not using the current candidate model is compared to determine whether to use it. If the maximum value is obtained by using the current candidate model, then the candidate model is recorded. After the dynamic programming iteration is completed, the optimal scheduling scheme with all throughputs within the prediction domain is found based on the average performance. The number of replicas of the candidate model is calculated by back-calculating the candidate model information recorded during dynamic programming execution, and the final scheduling strategy is generated.

4. The high-throughput large model inference method based on accuracy scaling according to claim 1, characterized in that, The method of proportionally rotating and adjusting cluster resources includes: prioritizing loading models with parameter sizes smaller than preset values.

5. A high-throughput large-model inference system based on accuracy scaling, characterized in that, It includes one or more processors for implementing the high-throughput large model inference method based on accuracy scaling as described in any one of claims 1-4.

6. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the high-throughput large model inference method based on accuracy scaling as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the high-throughput large model inference method based on accuracy scaling as described in any one of claims 1-4.

Images