Quantization-based multi-task large language model service system

Abstract

The invention discloses a multi-task large language model service system based on quantification. The system comprises a dynamic multi-task quantification module and a multi-task request scheduling module, wherein the dynamic multi-task quantification module is used for performing initial multi-task joint quantification and deployment on a deployed model, and when a subsequent new task needs to be introduced, the dynamic multi-task quantification module is also responsible for performing re-quantification on the model; the request pool is used for caching requests sent by various clients and giving the requests to the subsequent multi-task request scheduling module; the multi-task request scheduling module schedules and arranges requests of multiple tasks; whenever a multi-task request scheduling module gives a group of inputs similar to an existing service system, the inputs are sent to a reasoning process to be processed, predicted values corresponding to the outputs are obtained and decoded, and lexical elements obtained through decoding are output in a streaming mode and returned to the scheduling module to wait for next round of scheduling. The method has the advantages of higher availability, better expandability and better user experience.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and relates to a multi-task large language model service system based on quantization. Background Art

[0002] With the remarkable effects of large language models (LLMs) demonstrated in numerous fields, the application demands for downstream tasks such as text summarization, machine translation, and dialogue systems are increasing day by day. Due to the explosive growth of model scale and the limitations of device resources, "efficient fine-tuning - quantization deployment" has become the most common application paradigm for large language models in downstream tasks.

[0003] On the one hand, efficient fine-tuning techniques represented by LoRA (Low Rank Adaptation) only train additional small-scale adapters for the base model to adapt to specific tasks, significantly reducing the cost of model fine-tuning. Its principle is to replace the change of the original weight matrix of the model (ΔW ∈ R mxn ) with the product of two small low-rank matrices (A ∈ R rxn , B ∈ R mxr , r << m, n), that is, the result of multiplying these two matrices can approximately simulate the change of the original matrix, thereby performing fine-tuning training on the model.

[0004] On the other hand, model low-bit quantization techniques represented by GPTQ (Generative Pretrained Transformer Quantization) and AWQ (Activation-aware Weight Quantization) can significantly reduce the video memory requirements for large model deployment and reduce the memory access overhead during the inference process while maintaining the model effect. Its principle is to reduce the bit width of the model's weights and activation values to reduce the model size and improve the calculation efficiency. There are many quantization techniques, among which post-training quantization (PTQ) is the most common. The following is a typical PTQ process:

[0005]

[0006] Where X R represents the real-valued parameter before quantization, and X INTRepresents the parameters after quantization to int, Qmin and Qmax represent the minimum and maximum values ​​of the quantization range, and α represents the scaling ratio. Different PTQ methods calculate α in different ways, or use different approximation methods. In order to improve the accuracy of the model after quantization, it is usually necessary to correct the numerical distribution based on the target data set to better obtain parameters such as α. Many existing works have shown that the quantization accuracy with data set correction is often significantly higher than quantization without data set correction. Therefore, this patent conducts research on quantization with data set correction.

[0007] Although many mainstream LLM service systems (such as vLLM, TensorRT-LLM) have integrated support for quantitative deployment of fine-tuned models, these mainstream systems focus on the scenario of a single downstream task. With the increasing demand for various downstream tasks, it is necessary to serve multiple different tasks by simply switching the LoRA adapter of different tasks while deploying only one base model. Therefore, how to efficiently and quantitatively deploy a service system that supports multiple tasks has gradually become a crucial issue.

[0008] There are two existing multi-task LLM service systems, Punica and S-LoRA. They make multi-task service possible by solving the problem that single-task service systems cannot process LoRA adapters of different tasks in one batch at the same time.

[0009] The Punica system serves multiple LoRA models by optimizing GPU resource utilization. The core of its innovation is the introduction of the innovative CUDA core, Segmented Gather Matrix-Vector Multiplication (SGMV). SGMV enables batch GPU operations for parallel execution of different LoRA models, greatly improving the efficiency of GPU in memory and computing. The Punica system ensures that user requests are processed within milliseconds by optimizing the decoding stage. In addition, Punica optimizes resource allocation and utilization by periodically moving active requests to release allocated GPU resources.

[0010] S-LoRA adds several major innovative components based on Punica's kernel. It proposes a batch processing strategy to decompose the calculations between the basic model and the LoRA adapter. In addition, S-LoRA uses a unified memory pool to store KV cache and adapter weights in a paged manner, reducing fragmentation and balancing the dynamic changes in the size of the KV cache and adapter weights.

[0011] Although multi-task service systems such as Punica and S-LoRA theoretically realize the possibility of multi-task service, their actual availability is very poor.

[0012] First, given that quantization deployment has become the de facto standard for the deployment of inference service systems in the industry, and existing multi-task service systems do not support joint quantization of multiple tasks, it is impossible to apply it on a large scale: Currently, two simple ideas are tried to directly transplant the quantization method in the single-task service system to the multi-task service system. The first is the quantization algorithm such as Float point Quantization (FP) that does not require data set correction. Its scaling ratio α is obtained only through the weight itself. Such a quantization method has a poor effect, especially in the case of low-bit quantization. It is almost unusable. At the same time, it is also unstable and sensitive to the task type. The effect of some tasks is greatly reduced. Therefore, it cannot be used in multi-task service systems; the second is the single-task quantization algorithm that requires data set correction, such as GPTQ and AWQ. They rely on sampling the data set of the corresponding task to fit the activation and correct the quantization error, so as to better obtain the scaling ratio α that adapts to the task, and often obtain more stable and accurate quantization results than the first quantization method. However, in a multi-task service system, if this method is used directly to quantize multiple tasks, the data of different tasks will affect each other, thus fitting the wrong results, resulting in a decrease in the quantization effect of all tasks. If you want to maintain accuracy and quantize separately, you need to quantize a base model for each task, which greatly wastes video memory and computing resources. Therefore, previous naive attempts have failed. At present, there is an urgent need for a joint quantization algorithm suitable for multi-task service scenarios, so that different tasks can share a quantized base model with an accuracy comparable to that of single-task quantization.

[0013] Secondly, in actual service scenarios, the types of tasks of system services cannot remain unchanged, and there is a need to dynamically add and delete service tasks. However, systems such as Punica and S-LoRA have not considered this issue and only support a static number of tasks, and cannot dynamically add or delete different tasks. In addition, in quantitative deployment scenarios, existing work cannot support the introduction of quantification and deployment of new tasks without affecting old tasks. Therefore, when a new task needs to be added, the existing system needs to terminate and restart the service, which seriously affects the stability of online services and the user experience is extremely poor.

[0014] Finally, service requests for different tasks will inevitably have different loads (such as request length, processing time, etc.), and different LoRA adapters need to be loaded for processing. Existing systems do not take these issues into consideration when performing service scheduling, so they often need to load a large number of adapters in a single schedule and frequently switch the adapters that need to be loaded between adjacent schedules, resulting in performance limitations. Summary of the invention

[0015] In view of the problems existing in the prior art, the object of the present invention is to provide a multi-task large language model service system based on quantization.

[0016] The present invention proposes a novel multi-LoRA task quantization service system MQLserve (Multi-taskQuantization-based LoRA serve system), which aims to provide flexible and efficient multi-LoRA deployment services in resource-constrained environments by adopting innovative joint quantization, dynamic task addition and scheduling technologies, so as to make the multi-task service system truly available.

[0017] First of all, this patent proposes an innovative dynamic multi-task quantization algorithm Multi LoRA GenerativePretrained Transformer Quantization (MLGPTQ), which is based on the single-task quantization algorithm GPTQ. It can use the data of multiple tasks to jointly quantize the base model, so that the quantized base model can be shared among multiple tasks while maintaining high precision, which solves the shortcomings of directly transplanting naive quantization methods. At the same time, this algorithm can losslessly quantize new tasks added dynamically with almost no impact on online service performance.

[0018] Secondly, to address the problems of uneven load and resource waste in multi-tasking scenarios, this patent proposes a new scheduling algorithm based on output length prediction and clustering, which effectively reduces the video memory occupancy and video memory switching overhead in multi-tasking scenarios, and greatly improves system performance.

[0019] Based on this, this patent developed an end-to-end multi-LoRA task quantization service system MQLserve, which adopts advanced scheduling algorithms and dynamic task addition strategies, and integrates MLGPTQ quantization, which significantly improves its operating efficiency in resource-constrained environments and greatly enhances the availability of multi-task systems.

[0020] The technical solution of the present invention is:

[0021] A multi-task large language model service system based on quantization, characterized by comprising a multi-task joint quantization module and a dynamic task adding module;

[0022] The processing flow of the multi-task joint quantization module includes:

[0023] 11) Initialization phase

[0024] For each task i, the activation value X of task i is i and its corresponding low-rank adaptation matrix A i , B iInput the base model, which calculates the activation value X of the forward propagation according to the activation values ​​of all current tasks and their corresponding low-rank adaptation matrix; where the low-rank adaptation matrix B i , A i A weight change ΔW of a linear layer weight matrix W for simulating the base model;

[0025] Initialize an error matrix E for storing quantization errors; and initialize a matrix Q for storing pseudo quantization values;

[0026] Calculate the inverse Hessian matrix H by Cholesky decomposition -1 , that is, H -1 ←Cholesky((2XX T +λI) -1 ) T ; where λ is a regularization parameter used to prevent the matrix from being close to singular, and I represents the identity matrix;

[0027] 12) Weight Correction Phase

[0028] Each time, C columns are selected from the linear layer weight matrix W of the base model as a batch in batch processing. The starting column in each batch is the jth column in the linear layer weight matrix W. For each column in each batch, let it correspond to the kth column in the linear layer weight matrix W, denoted as (W ：，k ); where j = [0, C, 2C, 3C, ...], k∈[j, j+C-1]; first, (W ：，k ) is pseudo-quantized to obtain the corresponding pseudo-quantized value Q ：，k And store it in the kth column of the matrix Q; then calculate the error E corresponding to the kth column of the linear layer weight matrix W ：，k-j ←(W ：，k -Q ：，k ) / (H -1 ) kk And store it into the error matrix E, (H -1 ) kk is the inverse Hessian matrix H -1 The value of the kth row and kth column in the linear layer weight matrix W is then updated to the weight of each column in the batch with the initial column being the jth column in the linear layer weight matrix W. ：，k：(j+C) ←W ：，k：(j+C) -E ：，k-j ·(H -1 ) ：，k：(j+C) ; Among them, W ：，k：(j+C) represents the weights of the kth to jth + Cth columns of the linear layer weight matrix, E ：，k-j represents the kjth column of the error matrix, (H -1 ) ：，k：(j+C) represents the k~j+Cth column of the Hessian inverse matrix;

[0029] By W ：，(j+C)： ←W ：，(j+C)： -E·(H -1 ) j：(j+C)，(j+C) : Correct the initial column to the delayed weight W from the j+Cth column to the last column in the linear layer weight matrix W ；，(j+C) ; Among them, (H -1 ) j：(j+C)，(j+C)： Indicates (H -1 ) from the jth to j+Cth rows to the j+Cth column to the last column;

[0030] 13) Model quantization update phase

[0031] Quantize the linear layer weight matrix W corrected in step 12) to obtain the weight W of the quantized model int ;

[0032] The processing flow of the dynamic task adding module includes:

[0033] 21) Initialization phase

[0034] The linear layer weight matrix W before correction is stored in the memory of the CPU of the server where the multi-task service system is located;

[0035] Initialize message queue H queue , add the Hessian matrix of the task previously processed by the quantization model in the CPU to the information queue H queue ;

[0036] Set the activation value X of the new task t t and its corresponding low-rank adaptation matrix A t , B t Cache in the CPU;

[0037] 22) Asynchronous aggregation phase

[0038] Start an asynchronous thread for asynchronously aggregating new task information; load the linear layer weight W of the current layer of the base model before correction into the GPU of the server where the multi-task service system is located, and then for each new task t, the asynchronous thread according to the activation value X t , low-rank adaptation matrix A t , B t Calculate the activation value X of the new task t in the next layer of the base model t ←(W+B t A t )X t , update the activation value X of the new task t t ;

[0039] Calculate the Hessian matrix of the new task t And store it in the information queue H queue Then, the information of the new task t is aggregated using the set aggregation operator H aggrcgalcd ←Aggregate(H qucue );

[0040] 23) Layer-by-layer requantization stage:

[0041] Using the aggregation result H aggregated Re-quantize, that is, use the aggregation result H of the current layer aggregated Instead of the inverse Hessian matrix H -1 Executing the multi-task joint quantization module;

[0042] 24) Asynchronous LoRA correction phase

[0043] Calculate the error obtained by subtracting the quantized weight result containing the new task t after layer-by-layer re-quantization from the linear layer weight matrix W of the base model before correction, and perform low-rank decomposition on the error to obtain A error and B error And add them to the low-rank adaptation matrix A of the new task t respectively t and B t Update the low-rank adaptation matrix A of the new task t t and B t ;

[0044] 25) Seamless Integration

[0045] The updated low-rank adaptation matrix A of the new task t t and B t Seamlessly add it to the GPU for online task processing; update the low-rank adaptation matrix A of the new task t t and B t Integrated into the LoRA pool.

[0046] Furthermore, it also includes a multi-task request scheduling module, whose processing flow includes: using a set model to predict the output length of each task request; then based on the predicted output length, adopting the shortest remaining time priority strategy to determine the scheduling batch of each task request; wherein, a clustering coefficient β is set during the scheduling process, and the number of task types corresponding to the task requests in the same scheduling batch is less than β.

[0047] Furthermore, the specific processing flow of the multi-task request scheduling module is as follows:

[0048] 31) System initialization and cold start:

[0049] The multi-task service system creates and maintains four queues:

[0050] prefill_reqs queue: used to store new requests that have not yet entered the prefill stage;

[0051] decoding_reqs queue: used to store requests in the decoding stage;

[0052] hungry_prefill_reqs queue: used to store new requests that have exceeded the set time or scheduling times but have not been serviced by reasoning;

[0053] hungry_decoding_reqs queue: used to store requests that have been unprocessed for a set period of time and are in the decoding stage;

[0054] First, check whether the scheduling batch running_batch being processed is empty. If it is empty, it means that the system is in the cold start state; otherwise, it is in the scheduling stage;

[0055] (1) Cold start phase

[0056] Get multiple requests from the prefill_reqs queue to generate a new request batch new_batch; if the generated new_batch is not empty, perform a prefill operation on the batch new_batch and set it as running_batch;

[0057] If new_batch is empty, it remains idle and waits for new requests;

[0058] (2) Normal scheduling stage

[0059] According to the preset hyperparameter decode_count for storing the number of consecutive decodings, the hyperparameter max_cont_decode for storing the maximum number of consecutive decodings, the hyperparameter decode_count_one_batch for storing the number of consecutive decodings of the same request batch, and the hyperparameter max_cont_decode_one_batch for storing the maximum number of consecutive decodings of the same request batch, the scheduling is determined according to the following conditions:

[0060] If decode_count ≥ max_cont_decode, schedule a new request batch new_batch from the prefill_reqs queue and the hungry_prefill_reqs queue, set it as the scheduled batch running_batch being processed, and perform prefill operations on it;

[0061] If decode_count_one_batch ≥ max_cont_decode_one_batch, a new request batch new_batch is generated from the decoding_reqs queue and the hungry_decoding_reqs queue according to the task clustering strategy with the shortest remaining time first, and it is set as the scheduling batch running_batch being processed;

[0062] If it is other cases, continue to process the current scheduled batch running_batch and perform the decoding operation;

[0063] (3) Scheduling ends

[0064] When all task batches are processed, the scheduling stops; otherwise, the normal scheduling phase is repeated until all requests are served by inference.

[0065] Further, through The activation value X of the forward propagation is calculated, where W is the linear layer weight matrix of the base model.

[0066] Furthermore, the regularization parameter λ is set to 0.001.

[0067] Further, through Quantize the modified linear layer weight matrix W to obtain the weight W of the quantized model INT ; Among them, Q min and Q max They represent the minimum and maximum ranges after quantization, and α represents the scaling ratio.

[0068] The advantages of the present invention are as follows:

[0069] The MQLserve system proposed in the present invention is a brand-new multi-task service system that can be efficiently deployed under resource-constrained conditions. Compared with previous multi-task service systems, MQLserve has higher availability, better scalability and better user experience.

[0070] 1) The quantization algorithm MLGPTQ proposed in this system can make multiple tasks share the same quantized base model, which is impossible for previous methods. For n tasks, it can reduce the video memory resources by n times. At the same time, it maintains a high quantization accuracy, and its average accuracy loss is 3.17 times lower than that of other feasible methods.

[0071] 2) Compared with the previous SOTA system S-LoRA, the system MQLserve proposed in this patent has an average throughput ratio 30% higher, an average delay 27% lower than S-LORA, and an average task completion time 28% lower than S-LoRA.

[0072] 3) The system MLGPTQ proposed in this patent has better scalability. When the number of service tasks is increased from 2 to 100, the throughput of MQLserve only decreases by <10%, and the previous SOTA system is 36% higher.

[0073] 4) The dynamic task adding technology proposed in this patent has been proven to ensure that no matter how many tasks are added, the online service always maintains a high level of throughput and will not be interrupted, which is something that previous systems could not do. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 This is the architecture diagram of the MQLserve system design of the present invention.

[0075] Figure 2 Updated flow chart for asynchronous LoRA.

[0076] Figure 3 Flowchart for multi-task scheduling. DETAILED DESCRIPTION

[0077] The present invention is further described in detail below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0078] The core of this patented system MQLserve consists of two modules with three key technical points, namely the dynamic multi-task quantification module (responsible for multi-task joint quantification and dynamic task addition), and the multi-task request scheduling module (responsible for multi-task load scheduling). The overall system architecture is shown in the figure below: Figure 1 As shown, the dynamic multi-task quantization module is used to perform initial multi-task joint quantization and deployment on the deployed model. When new tasks need to be introduced later, the module is also responsible for re-quantizing the model; the request pool is used to cache requests from various clients and hand them over to the subsequent multi-task request scheduling module; the multi-task request scheduling module schedules and arranges multi-task requests based on the scheduling algorithm proposed in this patent; whenever the multi-task request scheduling module gives a set of inputs, similar to the existing service system, MQLserve sends the input to the inference pipeline for processing, obtains the corresponding output prediction value (Logits) and decodes it, and the decoded tokens are streamed and returned to the scheduling module, waiting for the next round of scheduling. Below we will introduce the specific implementation of the three key points we proposed.

[0079] 1. Multi-task quantization module

[0080] As mentioned above, since mainstream quantization methods require the use of task-related target datasets for correction and need to load different LoRA adapters for calculation, there are differences in the base models obtained by quantizing different tasks. In order to share quantization models in multi-task service scenarios, there are two naive approaches, namely, not using dataset correction or mixing the datasets of all tasks for unified quantization. However, these two approaches cannot accurately capture the individual numerical distributions for each task, and therefore will result in serious model accuracy loss.

[0081] To this end, this patent proposes a quantization method in a multi-task scenario - MLGPTQ (Multi-LoRA GPTQ), which aims to solve the shortcomings of existing quantization methods that cannot accurately capture the characteristics of each task, and can handle multiple different tasks while maintaining the accuracy of the quantized model. The advantage is that it can jointly quantize multiple tasks on the basis of retaining only one quantized base model, maintaining almost no loss of accuracy, and efficiently serving multiple tasks.

[0082] The core idea of ​​the MLGPTQ algorithm is to select the corresponding LoRA adapter (low-rank adaptation matrix) according to the task type of the input data, and use the corresponding quantization aggregation method to extract important information of the corresponding task, thereby increasing accuracy while reducing the storage requirements and computational overhead of the model. The detailed process is as follows:

[0083] (1) Initialization phase

[0084] For each task i, enter the activation value X for each task i i , and two low-rank adaptation matrices LoRA adapter (A i and B i , using B i A i To simulate the weight change of the base model ΔW), and merge the results of all tasks through the linear layer and LoRA to get the activation value X, that is, through Get the activation value X of the forward propagation, where W represents the linear layer weight matrix of the base model, which is shared by all tasks and is also our quantization target; X i represents the activation value of task i, i = 1 ~ n, n is the total number of tasks, and the activation value of each task depends on the position of the layer to be quantized in the model. For the first layer, the activation value is the input of the corresponding data, and for the i-th layer, the activation value is the output of the i-1th layer of the model, so the activation value X i The core of this method is to simulate the real activation values ​​of different tasks in the forward process, and the activation value X correctly captures the data distribution of all tasks.

[0085] Initialize the matrix E to store the quantization error; initialize the matrix Q to store the intermediate results of pseudo quantization.

[0086] Calculate the inverse Hessian matrix H by Cholesky decomposition -1 , that is, H -1 ←Cholesky((2XX T +λI) -1 ) T , used to maintain the numerical stability of the inverse matrix during the inverse matrix calculation process, where λ is the regularization parameter to prevent the matrix from being close to singularity and is set to 0.001, and I represents the identity matrix. When calculating the inverse matrix, if the original matrix is ​​close to singularity (i.e., irreversible or nearly irreversible), adding a small regularization parameter λ can improve numerical stability.

[0087] (2) Weight Correction Process

[0088] Using batch processing, we process a batch of size C at a time, that is, we process C columns in the W matrix at a time, traverse all batches, and loop j, k according to the following steps, where the outer loop j = [0, C, 2C, 3C, ...] and the inner loop k∈[j, j+C-1]:

[0089] First, it is pseudo-quantized (Q ：，k ←quant(W ：，k ), that is, the k-th column of W is pseudo-quantized and stored in the k-th column of the matrix Q; k∈[j, j+C-1];

[0090] Then use E ：，k-j ←(W ：，k -Q ：，k ) / (H -1 ) kk Update the error to guide weight correction; (H 1 ) kk is the inverse Hessian matrix H -1 The value of row k and column k in ;

[0091] Use W ：,k：(j+C) ←W ：，k：(j+C) -E ：，k-j ·(H -1 ) ：，k：(j+C) To update the weights, it is convenient for the final quantization, where W ：，k：(j+C) represents the weights of the kth to j+Cth columns of the linear layer weight matrix W, E ：，k-j represents the kjth column of the error matrix, (H -1 ) ：，k：(j+C)represents the kth to j+Cth columns of the Hessian inverse matrix;

[0092] Finally, after each outer loop is completed (after each batch of internal weight correction is completed), an additional delayed weight correction is required to correct the impact of this batch on all subsequent batches. The formula is: W ：，(j+C)： ←W ：，(j+C)： -E·(H -1 ) j：(j+C)，(j+C) :, where w ：，(j+C)： represents the j+Cth column to the last column of W, (H -1 ) j：(j+C)，(j+C)： Indicates (H -1 ) from the jth to j+Cth rows to the j+Cth column to the last column.

[0093] (3) Model quantization update

[0094] The present invention believes that the weights corrected by this method are suitable for quantization, so the linear layer weight matrix W of the corrected base model is used Quantize and get the weight W of the quantized model INT . Where Q min and Q max They represent the minimum and maximum ranges after quantization, which are -127 and 128 respectively if 8-bit quantization is used; α represents the scaling ratio, which is obtained by using the group quantization method (known technology); the Clip function clips the obtained value so that the result is within the Q range. min and Q max between.

[0095] 2. Dynamic task addition

[0096] In a multi-tasking scenario, there is a need to add dynamic tasks (i.e., add a LoRA adapter), but the existing multi-tasking service systems do not support this function. This technology aims to solve the model solidification problem faced when adding dynamic tasks, so that the newly added tasks can be adapted to the existing model in real time without affecting the stability and performance of the online service. Through the asynchronous LoRA update mechanism, it is ensured that the model dynamically adds the LoRA parameters of the new task to the LoRA pool of the online service without occupying a large amount of video memory and affecting GPU resources, thereby realizing flexible dynamic task addition.

[0097] The detailed process is as follows Figure 2 As shown, the specific steps are:

[0098] (1) Initialization phase:

[0099] The linear layer weight matrix W in the system has been pre-weighted and quantized to become WINT , stored in the GPU for online service. At the same time, we retain a copy of the original linear layer weight matrix W before weight correction and quantization, and fix it in the CPU memory of the server where the multi-task service system is located for subsequent possible dynamic task addition and quantization.

[0100] The quantized model also stores the Hessian matrix H of the old task in the CPU. cached , initialize the message queue H queue =[], using H queue .append(H cached ) Add the old task Hessian matrix to the information queue to provide support for future weight updates.

[0101] The input activation value X of the new task t and LoRA adapter A t , B t Temporarily stored in the CPU to facilitate subsequent capture of new task information.

[0102] (2) Asynchronous aggregation:

[0103] Start an asynchronous thread to perform asynchronous aggregation of new task information. Load the original linear layer weight W of the current layer of the base model fixed in the CPU to the GPU of the server where the multi-task service system is located, and then for each new task t, execute:

[0104] Set this task's X t ,LoRA adapter A t and B t Passed into the thread for calculation, that is, X t ←(W+B t A t )X t .

[0105] Calculate the Hessian matrix of task t: Used to store the data distribution characteristic information of the task.

[0106] Temporarily store the information in the information queue H queue Middle: H qucuc ·append(H t ).

[0107] Then, operators such as average aggregation and maximum aggregation are used to aggregate the new task information: H aggrcgatcd←Aggregate(H qucuc ).

[0108] (3) Layer-by-layer quantification:

[0109] The aggregation results are used for weight quantification, that is, H aggregated Replace H in MLGPTQ -1 The multi-task quantization described above is performed layer by layer and asynchronously. That is, we open another asynchronous CPU thread to load and quantize new tasks layer by layer to minimize the preemption of online service resources. At the same time, the asynchronous background CPU process is used for heavy quantization, which will not affect the GPU process of the online service. The quantization can be completed with the minimum resources without interrupting the multi-task service in progress online.

[0110] (4) Asynchronous LoRA correction:

[0111] The error obtained by subtracting the quantized weights containing new task information from the non-quantized base model weights after layer-by-layer re-quantization is corrected to LoRA instead of the base model to reduce the amount of transmission. The core of this asynchronous update process is to perform low-rank decomposition of the error to obtain A error and B error , respectively added to the low-rank matrix A of the LoRA adapter of the new task t t and B t It can be seamlessly added to the LoRA adapter instead of the base model, without interrupting the base model in service, and without waiting for the computing task to be completed. Since the entire process is mainly performed on the CPU, GPU resources are almost unaffected, and the system's online task processing performance will not be reduced.

[0112] (5) Seamless integration:

[0113] After the new LoRA adapter is modified and quantized, it is seamlessly added to the GPU for online task processing.

[0114] The LoRA adapter of the new task t is integrated into the LoRA pool and can share the same base model with the original task to achieve dynamic adaptation of multiple tasks.

[0115] 3. Multi-task request scheduling module

[0116] Although there have been many studies on the scheduling algorithm of LLM service requests, the special properties of multi-task scenarios have not been considered: (1) The total length distribution of input and output of different tasks is very different, while the same task shows a clustering effect. (2) Due to the randomness and non-fixed nature of task arrival, there will be multiple tasks in a single schedule, and the types of tasks between two adjacent batches of scheduling are very different.

[0117] This paper proposes a scheduling strategy based on output length prediction and task clustering, which solves the above problems well.

[0118] First, the BERT distillation model is used to predict the output length of the request. Based on this prediction, this paper adopts the theoretically optimal scheduling strategy - the shortest remaining time first strategy (SRTF) algorithm for scheduling. Secondly, in the scheduling process, in order to prevent the excessive number of tasks in the same scheduling batch from causing the LoRA adapter to occupy video memory and cause waste, this system sets the clustering coefficient β to make the number of task types in a scheduling batch as small as possible. At the same time, the system will give priority to scheduling the task type in the previous batch to avoid a large difference in the types of two adjacent tasks. Finally, in order to avoid starvation, the system sets a starvation queue and will give priority to scheduling starvation requests. The specific process is as follows:

[0119] (4) System initialization and cold start:

[0120] The system maintains four queues:

[0121] prefill_reqs: stores new requests that have not yet entered the prefill phase.

[0122] decoding_reqs: stores requests in the decoding phase.

[0123] hungry_prefill_reqs: stores new requests that have not been served for a long time and need to be processed quickly, such as new requests that have not been scheduled in 100 times.

[0124] hungry_decoding_reqs: Stores requests that have been unprocessed for a long time and are in the decoding stage.

[0125] The system first checks whether running_batch, that is, the scheduling batch being processed, is empty. If it is empty, it means that the system is in a cold start state.

[0126] (5) Cold start phase:

[0127] If running_batch is empty, call the generate_new_batch function to get multiple requests from prefill_reqs to generate a new request batch new_batch.

[0128] If the generated new_batch is not empty, a prefill operation is performed on the batch and it is set as running_batch.

[0129] If new_batch is empty, the system remains idle, waiting for new requests.

[0130] (6) Normal scheduling stage:

[0131] The present invention determines what kind of scheduling to perform according to the following conditions based on the pre-set system hyperparameters decode_count (number of consecutive decodes), max_cont_decode (maximum number of consecutive decodes), decode_count_one_batch (number of consecutive decodes of the same batch) and max_cont_decode_one_batch (maximum number of decodes of the same batch):

[0132] If decode_count ≥ max_cont_decode, the system calls generate_new_batch to schedule a new task batch from prefill_reqs and hungry_prefill_reqs. The specific process is to first sort the requests according to their input length, and then limit the number of task types in the same batch according to the set threshold β. Finally, new_batch is obtained, set as running_batch, and Prefill operation is performed.

[0133] If decode_count_one_batch ≥ max_cont_decode_one_batch, the system calls the schedule_new_batch function, schedules from decoding_reqs and hungry_decoding_reqs according to the shortest remaining time first (SRTF), and uses β to ensure that β tasks of the same type in the previous running_batch are scheduled first, and set as running_batch.

[0134] In other cases, we continue to process the current running_batch and perform the decode operation.

[0135] (7) Scheduling ends:

[0136] When all task batches are processed, the system stops scheduling. Otherwise, the normal scheduling phase is repeated until all requests are served.

[0137] This algorithm maximizes the efficiency of task processing through reasonable threshold settings and scheduling strategies, while reducing the overhead caused by resource switching. It is particularly suitable for complex scenarios with multiple tasks and multiple types of requests. Figure 3 shown.

[0138] The technical solution proposed by the present invention can support multi-task joint quantization, dynamic task addition and integrate a scheduling system designed specifically for multi-task scenarios. Generally, existing multi-task service systems do not support quantization or dynamic task addition, and they do not have a scheduling algorithm designed specifically for this scenario, which is very inefficient.

[0139] The functional performance of the product side can include serving as a multi-task scheduling system to provide flexible and efficient multi-LoRA deployment services in resource-constrained environments; sharing high-precision quantitative base models between multiple tasks, dynamically adding new tasks losslessly, and with almost no impact on online service performance; effectively reducing video memory usage and video memory switching overhead in multi-tasking scenarios, greatly improving system performance, and making the multi-task service system truly practical.

[0140] There are no mature products on the market, only relevant open source software, including punica from punica-ai, S-LoRA from UC Berkeley and Stanford University, etc. However, these open source software can perform multi-task services in theory, but in actual deployment, they are very inefficient and waste resources, and cannot be used in actual application scenarios.

[0141] The functional comparison between the system of the present invention and the existing system is shown in Table 1. It is easy to see that the existing technology has great problems in industrial deployment and practical application. We reduce the difficulty of its actual deployment and increase the user experience by solving its problems in quantification and scheduling.

[0142] Table 1. Comparison of supported functions of different service systems

[0143]

[0144]

[0145] Although the specific embodiments of the present invention are disclosed for the purpose of illustration, the purpose is to help understand the content of the present invention and implement it accordingly, those skilled in the art will understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the best embodiment, and the scope of the present invention is subject to the scope defined in the claims.

Claims

1. A quantization-based multi-task large language model service system, characterized in that: Including multi-task joint quantization module and dynamic task adding module; The processing flow of the multi-task joint quantization module includes: 11) Initialization phase For each task i, the activation value X of task i is i and its corresponding low-rank adaptation matrix A i , B i Input the base model, which calculates the activation value X of the forward propagation according to the activation values ​​of all current tasks and their corresponding low-rank adaptation matrix; where the low-rank adaptation matrix B i , A i A weight change ΔW of a linear layer weight matrix W for simulating the base model; Initialize an error matrix E for storing quantization errors; and initialize a matrix Q for storing pseudo quantization values; Calculate the inverse Hessian matrix H by Cholesky decomposition -1 , that is, H -1 ←CHolesky((2XX T +λI) -1 ) T ; where λ is a regularization parameter used to prevent the matrix from being close to singular, and I represents the identity matrix; 12) Weight Correction Phase Each time, C columns are selected from the linear layer weight matrix W of the base model as a batch in batch processing. The starting column in each batch is the jth column in the linear layer weight matrix W. For each column in each batch, let it correspond to the kth column in the linear layer weight matrix W, denoted as (W ：，k ); where j = [0, C, 2C, 3C, ...], k∈ [j, j+C-1]; first, (W ：，k ) performs pseudo-quantization to obtain the corresponding pseudo-quantization value Q:, k and stores it in the kth column of the matrix Q; then calculates the error E corresponding to the kth column of the linear layer weight matrix W ：k-j ←(W ：k- Q ：k ) / (H -1 ) kk And store it into the error matrix E, (H -1 ) kk is the inverse Hessian matrix H -1 The value of the kth row and kth column in the linear layer weight matrix W is then updated to the weight of each column in the batch with the initial column being the jth column in the linear layer weight matrix W. ：，k：(j+C) ←W ：，k(j+C) -E ：k-j ·(H -1 ) ：k：(j+C) ; Among them, W :,k:(j+C) represents the weights of the kth to jth + Cth columns of the linear layer weight matrix, E :,k-j represents the kjth column of the error matrix, (H -1 ) :,k:(j+C) represents the k~j+Cth column of the Hessian inverse matrix; By W ：，(j+C)： ←W ：，(j+C)： -E·(H -1 ) j：(j+C)，(j+C)： Correct the initial column to be the delayed weight W from the j+Cth column to the last column in the linear layer weight matrix W :,(j+C) ; Among them, (H -1 ) j:(j+C),(j+C): Indicates (H -1 ) from the jth to j+Cth rows to the j+Cth column to the last column; 13) Model quantization update phase Quantize the linear layer weight matrix W corrected in step 12) to obtain the weight W of the quantized model int ; The processing flow of the dynamic task adding module includes: 21) Initialization phase The linear layer weight matrix W before correction is stored in the memory of the CPU of the server where the multi-task service system is located; Initialize message queue H queue , add the Hessian matrix of the task previously processed by the quantization model in the CPU to the information queue H queue ; Set the activation value X of the new task t t and its corresponding low-rank adaptation matrix A t , B t Cache in the CPU; 22) Asynchronous aggregation phase Start an asynchronous thread for asynchronously aggregating new task information; load the linear layer weight W of the current layer of the base model before correction into the GPU of the server where the multi-task service system is located, and then for each new task t, the asynchronous thread according to the activation value X t , low-rank adaptation matrix A t , B t Calculate the activation value X of the new task t in the next layer of the base model t ←(W+B t A t )X t , update the activation value X of the new task t t ; Calculate the Hessian matrix of the new task t And store it in the information queue H queue Then, the information of the new task t is aggregated using the set aggregation operator H aggregatcd ←Aggregate(H qucuc ); 23) Layer-by-layer requantization stage: Using the aggregation result H aggregated Re-quantize, that is, use the aggregation result H of the current layer aggregated Instead of the inverse Hessian matrix H -1 Executing the multi-task joint quantization module; 24) Asynchronous LoRA correction phase Calculate the error obtained by subtracting the quantized weight result containing the new task t after layer-by-layer re-quantization from the linear layer weight matrix W of the base model before correction, and perform low-rank decomposition on the error to obtain A error and B error And add them to the low-rank adaptation matrix A of the new task t respectively t and B t Update the low-rank adaptation matrix A of the new task t t and B t ; 25) Seamless Integration The updated low-rank adaptation matrix A of the new task t t and B t Seamlessly add it to the GPU for online task processing; update the low-rank adaptation matrix A of the new task t t and B t Integrated into the LoRA pool.

2. The multi-task large language model service system according to claim 1, characterized in that: It also includes a multi-task request scheduling module, whose processing flow includes: using a set model to predict the output length of each task request; then based on the predicted output length, adopting the shortest remaining time priority strategy to determine the scheduling batch of each task request; wherein, a clustering coefficient β is set during the scheduling process, and the number of task types corresponding to the task requests in the same scheduling batch is less than β.

3. The multi-task large language model service system according to claim 2, characterized in that: The specific processing flow of the multi-task request scheduling module is as follows: 31) System initialization and cold start: The multi-task service system creates and maintains four queues: prefill_reqs queue: used to store new requests that have not yet entered the prefill stage; decoding_reqs queue: used to store requests in the decoding stage; hungry_prefill_reqs queue: used to store new requests that have not been processed for a set period of time or scheduling times; hungry_decoding_reqs queue: used to store requests that have not been processed for a set period of time and are in the decoding stage; First, check whether the scheduling batch running_batch being processed is empty. If it is empty, it means that the system is in the cold start state; otherwise, it is in the scheduling stage; (1) Cold start phase Get multiple requests from the prefill_reqs queue to generate a new request batch new_batch; if the generated new_batch is not empty, perform a prefill operation on the batch new_batch and set it as running_batch; if new_batch is empty, remain idle and wait for new requests; (2) Normal scheduling stage According to the preset hyperparameter decode_count for storing the number of consecutive decodings, the hyperparameter max_cont_decode for storing the maximum number of consecutive decodings, the hyperparameter decode_count_one_batch for storing the number of consecutive decodings of the same request batch, and the hyperparameter max_cont_decode_one_batch for storing the maximum number of decodings of the same request batch, the scheduling is determined according to the following conditions: If decode_count ≥ max_cont_decode, schedule a new request batch new_batch from the prefill_reqs queue and the hungry_prefill_reqs queue, set it as the scheduled batch running_batch being processed, and perform prefill operations on it; If decode_count_one_batch ≥ max_cont_decode_one_batch, a new request batch new_batch is generated from the decoding_reqs queue and the hungry_decoding_reqs queue according to the task clustering strategy with the shortest remaining time first, and it is set as the scheduling batch running_batch being processed; If it is other cases, continue to process the current scheduled batch running_batch and perform the decoding operation; (3) Scheduling ends When all task batches are processed, the scheduling stops; otherwise, the normal scheduling phase is repeated until all requests are served by inference.

4. The multi-task large language model service system according to claim 1, 2 or 3, characterized in that: pass The activation value X of the forward propagation is calculated, where W is the linear layer weight matrix of the base model.

5. The multi-task large language model service system according to claim 1, 2 or 3, characterized in that: The regularization parameter λ is set to 0.

001.

6. The multi-task large language model service system according to claim 1, 2 or 3, characterized in that: pass Quantize the modified linear layer weight matrix W to obtain the weight W of the quantized model INT ; Among them, Q min and Q max They represent the minimum and maximum ranges after quantization, and α represents the scaling ratio.

Images