An iterative quantization-aware training method for large language model end-side deployment

Abstract

The application provides an iterative quantization-aware training method for large language model end-side deployment, which can be applied to the technical field of large language models. The method comprises the following steps: dynamically allocating a weight quantization ratio by a proportional scheduler according to a training stage, avoiding excessive compression of key parameters, and reducing performance loss caused by low-precision calculation of end-side devices; selectively retaining original parameters by a Boolean mask matrix, reducing quantization errors of key weights in the self-attention layer, and maintaining the accuracy of the generative large language model in end-side decoding; the sparse parameter structure generated by the mask matrix can trigger the sparse calculation optimization of the hardware accelerator, thereby improving the inference throughput; in addition, the multi-stage quantization parameter updating strategy provided by the application allows adjusting the quantization granularity for different hardware, thereby avoiding the cross-platform compatibility problem caused by traditional one-time quantization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model, in particular to an iterative quantization-aware training method for large language model end-side deployment. BACKGROUND

[0002] With the development of generative large language model, the model parameters grow explosively, and the parameters of the generative large language model are saved in FP16 floating point (Half-precision floating-point), that is, at least 2 bytes are occupied for one parameter. In order to be able to deploy the generative large language model completely, the server needs to have high hardware performance, such as high bandwidth, large amount of high-performance memory, large amount of high-performance graphics card, etc., which undoubtedly increases the deployment cost and application cost of the generative large language model. In addition, due to the need for high hardware performance of the generative large language model, the pre-trained generative large language model cannot be directly deployed on the end-side with much lower hardware performance than the server, so users usually interact with the generative large language model deployed on the server through the network, which reduces the convenience of user use, especially in the case of network service blockage. At the same time, the server deployment model has the risk of user privacy leakage, therefore, the generative large language model deployed on the server usually needs parameter quantization technology to reduce the storage space occupied by the model parameters, and then deploy the parameter quantized generative large language model on the end-side. However, the existing parameter quantization technology still needs to use a large amount of hardware resources in the parameter quantization process of the generative large language model, and the existing parameter quantization technology mainly focuses on post-training quantization technology, which has the advantages of small training amount and fast quantization speed, but also has the problems of poor quantization effect, etc. SUMMARY

[0003] In view of the above problems, the present application provides an iterative quantization-aware training method for large language model end-side deployment, which is used to at least solve one of the above technical problems.

[0004] According to the first aspect of the present application, an iterative quantization-aware training method for large language model end-side deployment is provided, comprising:

[0005] randomly selecting text data samples used in the current training stage from a text training data set, performing initial quantization of all original parameters in the target large language model deployed on the server in the current training stage to obtain an initial quantization parameter matrix;

[0006] determining the weight quantization ratio of the current training stage by using a proportional scheduler, and generating a Boolean mask matrix of the current training stage by using a weight mask function;

[0007] The initial quantization parameter matrix is replaced with original parameters of the target large language model by using a weight quantization ratio and a Boolean mask matrix to obtain a quantization parameter matrix.

[0008] The target large language model with the quantization parameter matrix is used to perform self-attention mechanism-based encoding and cross-attention mechanism-based decoding on text data samples of a current training stage to obtain a text data processing result, and a loss value of the current training stage is obtained by using the text data processing result.

[0009] The target large language model is updated in the current training stage by using the quantization parameter matrix, the loss value of the current training stage, and a preset learning rate.

[0010] The operation in the current training stage is repeated for each training stage until a preset training condition is met, and a large language model after parameter quantization is obtained, and the large language model after parameter quantization is deployed to a client, wherein the large language model after parameter quantization is used to process text data of the client.

[0011] According to the embodiments of the present application, the above-mentioned iterative quantization-aware training method for large language model end-side deployment further comprises:

[0012] The original parameters of the pre-trained large language model are limited in value range by a parameter truncation operation to obtain a target large language model.

[0013] The original parameters in the pre-trained large language model are initially scored by a parameter scoring operation to obtain an initial score matrix, and a weight quantization ratio used in a first round of training stage is initialized.

[0014] According to the embodiments of the present application, the original parameters of the target large language model have the same floating-point precision as the original parameters of the pre-trained large language model.

[0015] The initial score matrix has the same shape as the parameter matrix of the pre-trained large language model and is used to screen quantized model parameters in the parameter quantization process.

[0016] According to the embodiments of the present application, the determination of the weight quantization ratio of the current training stage by the ratio scheduler comprises:

[0017] The weight quantization ratio of the current training stage is determined by the ratio scheduler in a linearly increasing manner based on the total number of stages used for training.

[0018] According to the embodiments of the present application, the Boolean mask matrix has the same shape as the initial quantization parameter matrix.

[0019] The part of elements in the Boolean mask matrix are set to a preset value according to a weight quantization ratio of the current training stage.

[0020] According to the embodiment of the present application, the Boolean mask matrix of the current training stage is generated by using the weight mask function, which includes:

[0021] The score matrix of the current training stage is created or updated by using the preset calibration data, the quantization parameter matrix of the previous training stage, and the original parameters of the target large language model.

[0022] The Boolean mask matrix of the current training stage is obtained by processing the score matrix of the current training stage and the weight quantization ratio by using the preset selection function.

[0023] According to the embodiment of the present application, the score matrix of the current training stage is created or updated by using the preset calibration data, the quantization parameter matrix of the previous training stage, and the original parameters of the target large language model, which includes:

[0024] The local quantization loss of the current training stage is calculated by using the preset calibration data, the quantization parameter matrix of the previous training stage, and the original parameters of the target large language model.

[0025] The second-order partial derivative of the original weight of the local quantization loss of the current training stage is calculated to obtain the Hessian matrix of the current training stage.

[0026] The score matrix of the current training stage is obtained by processing the Hessian matrix of the current training stage, the quantization parameter matrix of the previous training stage, and the original parameters of the target large language model.

[0027] According to the embodiment of the present application, the parameter update of the target large language model in the current training stage by using the quantization parameter matrix, the loss value of the current training stage, and the preset learning rate includes:

[0028] The loss value of the current training stage is calculated by using the quantization parameter matrix, the original parameters of the target large language model, and the text data sample used in the current training stage.

[0029] The gradient value of the current training stage is obtained by deriving the loss value of the current training stage.

[0030] The parameter update of the target large language model in the current training stage is performed by using the preset learning rate and the gradient value through the gradient descent operation.

[0031] According to the embodiment of the present application, the target large language model includes a multi-language machine translation model, and the text training data set includes a corpus with a source language and a target language alignment relationship.

[0032] The client includes a mobile communication device and a body-integrated intelligent robot.

[0033] According to the embodiment of the present application, the above-mentioned target large language model includes an intelligent question and answer model, and the above-mentioned text training data set includes a structured knowledge base with question-text answer labels and a multi-modal data set with image-question-text answer triple labels.

[0034] The client includes an intelligent home terminal, an intelligent medical terminal, a customer service terminal, and an intelligent education terminal.

[0035] The iterative quantization-aware training method for large language model end-side deployment provided by the present application dynamically allocates weight quantization ratios by a proportional scheduler according to training stages, retains more original model performance during quantization, avoids the serious damage to the basic performance of the model caused by traditional static quantization, and reduces the performance loss caused by low-precision calculation of end-side devices; the original parameters are selectively retained by a Boolean mask matrix, the quantization error of key weights in the self-attention layer is reduced, and the accuracy of the generative large language model in end-side decoding is maintained; the sparse parameter structure generated by the mask matrix can trigger the sparse calculation optimization of the hardware accelerator, and improve the inference throughput; in addition, the quantization parameter updating strategy provided by the present application allows adjusting the quantization granularity for different hardware, and avoids the cross-platform compatibility problem caused by traditional one-time quantization. BRIEF DESCRIPTION OF DRAWINGS

[0036] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application taken in conjunction with the accompanying drawings, in which:

[0037] Figure 1 is an application scenario diagram of the iterative quantization-aware training method for large language model end-side deployment according to the embodiment of the present application;

[0038] Figure 2 is a flowchart of the iterative quantization-aware training method for large language model end-side deployment according to the embodiment of the present application;

[0039] Figure 3 is a structural block diagram of the iterative quantization-aware training device for large language model end-side deployment according to the embodiment of the present application;

[0040] Figure 4 is a block diagram of an electronic device suitable for implementing the iterative quantization-aware training method for large language model end-side deployment according to the embodiment of the present application. DETAILED DESCRIPTION

[0041] Embodiments of the present application will be described below with reference to the accompanying drawings. It should be understood, however, that the description that follows is merely exemplary and is not intended to limit the scope of the application. In the following detailed description of embodiments of the present application, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to one skilled in the art that one or more embodiments of the present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present application.

[0042] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "includes" and tautological equivalents thereof, means that the named feature, step, operation, and / or component is present, but not excluding the presence or addition of one or more other features, steps, operations, or components.

[0043] All terms used herein including technical and scientific terms have the same meanings as commonly understood by one of ordinary skill in the art unless otherwise defined herein. It should be noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification, and not be interpreted in an idealized or overly formal way.

[0044] In the case where expressions such as "at least one of A, B, and C, etc." are used, it generally should be interpreted that the meaning is the same as "at least one of the group consisting of A, B, and C" unless otherwise defined herein (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C together, etc.).

[0045] With the development of generative large language models, model parameters have exploded, such as the DeepSeek-R1 model with 671 billion parameters, which requires at least 1248G memory for full deployment. Using a server equipped with 8 NVIDIA A100 80G graphics cards for deployment requires two such servers. These model parameters are saved in FP16 floating point numbers, and one parameter occupies 16 bits (2 bytes). Quantization is to reduce the bit width of the parameters to low precision (8 bits, 4 bits, etc.), which can reduce the resource occupation of the model. However, reducing precision inevitably damages the performance of the model, and research on parameter quantization is to alleviate these problems.

[0046] The current popular quantization method mainly focuses on post-training quantization technology, which has the advantages of small training amount and fast quantization speed, but also has obvious disadvantages: poor quantization effect. Existing researches can obtain better effect than post-training quantization through quantization-aware training under the same setting, but in the scene of low-bit quantization, directly performing quantization-aware training is not the optimal choice, because it will make the model lose most of the basic ability and cannot fully utilize the basic ability of the model. In addition, the existing technical solutions still need to use a large amount of hardware resources in the parameter quantization process.

[0047] To at least solve one of the problems in the prior art, the present application provides a low-cost and efficient parameter quantization method for end-side deployment of a generative large language model.

[0048] Figure 1 is an application scenario diagram of the iterative quantization-aware training method for end-side deployment of a large language model according to an embodiment of the present application.

[0049] As Figure 1 shown, the application scenario 100 according to this embodiment can include a large language model and an end-side device inference acceleration, etc. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0050] The user can use the first terminal device 101, the second terminal device 102, the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0051] The first terminal device 101, the second terminal device 102, the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smartphones, tablet computers, laptop computers and desktop computers, etc.

[0052] The server 105 can be a server providing various services, such as a background management server supporting the website browsed by the user using the first terminal device 101, the second terminal device 102, the third terminal device 103 (only as an example). The background management server can analyze and process the received user request data, etc., and feed back the processing result (such as a web page, information, or data, etc. obtained or generated according to the user request) to the terminal device.

[0053] It should be noted that the iterative quantization-aware training method for edge deployment of large language models provided in this embodiment of the invention can generally be executed by server 105. Correspondingly, the iterative quantization-aware training device for edge deployment of large language models provided in this embodiment of the invention can generally be located in server 105. The iterative quantization-aware training method for edge deployment of large language models provided in this embodiment of the invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the iterative quantization-aware training device for edge deployment of large language models provided in this embodiment of the invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0054] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0055] The following will be based on Figure 1 The described scene, through Figure 2 The iterative quantization-aware training method for edge deployment of large language models, according to the disclosed embodiments, is described in detail.

[0056] Figure 2 This is a flowchart of an iterative quantization-aware training method for edge deployment of a large language model according to an embodiment of the present invention.

[0057] like Figure 2 As shown, the iterative quantization-aware training method for deploying large language models on the edge in this embodiment includes operations S210 to S260.

[0058] In operation S210, text data samples used in the current training phase are randomly selected from the text training dataset, and all original parameters in the target large language model deployed on the server are initially quantized for the current training phase to obtain the initial quantization parameter matrix.

[0059] According to an embodiment of the present invention, the target large language model includes a multilingual machine translation model, and the text training dataset includes a corpus with alignment relationships between the source language and the target language; wherein, the client includes a mobile communication device and an embodied intelligent robot.

[0060] According to an embodiment of the present application, the large language model comprises an intelligent question and answer model, the text training data set comprises a structured knowledge base with question-text answer labels and a multi-modal data set with image-question-text answer triple labels; and the client comprises a smart home terminal, a smart medical terminal, a customer service terminal and a smart education terminal.

[0061] In operation S220, a weight quantization ratio of the current training stage is determined by using a proportional scheduler, and a Boolean mask matrix of the current training stage is generated by using a weight mask function.

[0062] In operation S230, part of the parameters in the initial quantized parameter matrix are replaced with original parameters of the large language model by using the weight quantization ratio and the Boolean mask matrix, to obtain a quantized parameter matrix.

[0063] In operation S240, the large language model with the quantized parameter matrix is used to encode the text data sample of the current training stage based on a self-attention mechanism and decode based on a cross-attention mechanism, to obtain a text data processing result, and the loss value of the current training stage is obtained by using the text data processing result.

[0064] In operation S250, the large language model is updated by using the quantized parameter matrix, the loss value of the current training stage and a preset learning rate.

[0065] In operation S260, the operation of the current training stage is repeated for each training stage until a preset training condition is met, to obtain a large language model after parameter quantization, and the large language model after parameter quantization is deployed to the client, wherein the large language model after parameter quantization is used to process text data of the client.

[0066] The iterative quantization-aware training method for large language model end-side deployment provided by the present application dynamically allocates a weight quantization ratio by using a proportional scheduler according to a training stage, avoids over-compression of key parameters by traditional static quantization, reduces performance loss caused by low-precision calculation of end-side devices, selectively retains original parameters by using a Boolean mask matrix, reduces quantization error of key weights in a self-attention layer, maintains accuracy of a generative large language model in end-side decoding, triggers sparse calculation optimization of a hardware accelerator by using a sparse parameter structure generated by the mask matrix, improves inference throughput, and the quantization parameter update strategy provided by the present application allows adjusting quantization granularity for different hardware, and avoids cross-platform compatibility problems caused by traditional one-time quantization.

[0067] According to an embodiment of the present application, the above-mentioned iterative quantization-aware training method for large language model end-side deployment further comprises: limiting the value range of the original parameters of the pre-trained large language model through a parameter truncation operation to obtain a target large language model; and performing initial scoring on the original parameters in the pre-trained large language model through a parameter scoring operation to obtain an initial score matrix, and initializing the weight quantization ratio used in the first round of training.

[0068] The above-mentioned pre-trained large language model, for example, LLaMA model, deepseek model, GPT model, Claude model, ChatGLM-6B model, etc.

[0069] The above-mentioned embodiment limits the parameters in the hardware-friendly interval through the truncation operation, avoids quantization overflow caused by extreme values, and improves the calculation stability of the mobile terminal chip.

[0070] According to an embodiment of the present application, the original parameters of the target large language model have the same floating-point precision as the original parameters of the pre-trained large language model; and the initial score matrix has the same shape as the parameter matrix of the pre-trained large language model and is used to screen the quantized model parameters in the parameter quantization process.

[0071] According to an embodiment of the present application, the above-mentioned determination of the weight quantization ratio of the current training stage by the ratio scheduler comprises: determining the weight quantization ratio of the current training stage by the ratio scheduler in a linearly increasing manner based on the total number of stages used for training.

[0072] The ratio scheduler will be described in detail through a specific implementation manner.

[0073] The ratio scheduler is responsible for determining the proportion of weights to be quantized at each step The present application fully considers the hardware cost problem in the parameter quantization process, and therefore directly selects a simple scheduling method: linear increase. From 0 to 1 linearly within half of the total number of training steps, as shown in formula (1):

[0074] (1)

[0075] Where t is the training step, T is the total number of training steps.

[0076] According to an embodiment of the present application, the Boolean mask matrix has the same shape as the initial quantization parameter matrix; and part of the elements in the Boolean mask matrix are set to a preset value according to the weight quantization ratio of the current training stage.

[0077] According to an embodiment of the present application, the above-mentioned generating the Boolean mask matrix of the current training stage by using the weight mask function comprises: creating or updating the score matrix of the current training stage by using the preset calibration data, the quantized parameter matrix of the previous training stage and the original parameter of the target large language model; processing the score matrix of the current training stage and the weight quantization ratio by using the preset selection function to obtain the Boolean mask matrix of the current training stage.

[0078] According to an embodiment of the present application, the above-mentioned creating or updating the score matrix of the current training stage by using the preset calibration data, the quantized parameter matrix of the previous training stage and the original parameter of the target large language model comprises: calculating the local quantization loss of the current training stage by using the preset calibration data, the quantized parameter matrix of the previous training stage and the original parameter of the target large language model; calculating the second-order partial derivative of the original weight of the local quantization loss of the current training stage to obtain the Hessian matrix of the current training stage; processing the Hessian matrix of the current training stage, the quantized parameter matrix of the previous training stage and the original parameter of the target large language model to obtain the score matrix of the current training stage.

[0079] The above-mentioned embodiments can accurately identify the parameters that have the greatest impact on the model through the second-order derivative analysis of the local quantization loss, and avoid the blindness of traditional uniform quantization; and the dynamic evaluation of the importance of the parameters is realized by combining the historical quantized parameters and the current Hessian matrix, thereby improving the stability of the end-side deployment.

[0080] The weight mask function will be further described in detail through a specific implementation manner.

[0081] The overall calculation manner of the weight mask function is shown in formula (2):

[0082] (2)

[0083] wherein w is the unquantized FP16 weight, a score matrix of the same shape is generated, and a Boolean matrix is generated by the function, which has the same shape as , wherein the elements of are 1, and the elements of are 0.

[0084] The calculation method of is shown in formula (3) and (4):

[0085] (3)

[0086] (4)

[0087] in, These are unquantized FP16 weights. These are the quantified weights. It is an input from calibration data. Is L relative to The Hessian matrix, This is the loss when calculating the Hessian matrix. It is the first of the unquantized FP16 weights Line number Column elements, It is the th in the fractional matrix Line number Column elements, It is the first of the quantized weights Line number Column elements, It is the first inverse matrix of Hessian. Line number The elements of the column.

[0088] The Sel function chosen here is the TOP function, which sorts the data in descending order. The front of the middle The elements are set to 1, and the rest are 0. The weight mask function described above is denoted as max_to_min.

[0089] All modules here are pluggable. This invention also includes two weight mask functions: min_to_max: the Sel function is replaced with the minimum value, i.e., the value after ascending sorting. The front of the middle The elements are set to 1, and the rest are 0; random: The elements in the sample are uniformly distributed between 0 and 1, and the Sel function is the top function.

[0090] According to an embodiment of the present invention, the above-mentioned parameter update of the target large language model in the current training stage using the quantization parameter matrix, the loss value of the current training stage, and the preset learning rate includes: calculating the loss value of the current training stage using the quantization parameter matrix, the original parameters of the target large language model, and the text data samples used in the current training stage; differentiating the loss value of the current training stage to obtain the gradient value of the current training stage; and updating the parameters of the target large language model in the current training stage using the preset learning rate and the gradient value through gradient descent operation.

[0091] The gradient calculation in the above embodiments considers both quantized parameters and original parameters, avoiding the gradient mismatch problem of traditional PTQ (post-training quantization) and improving the stability of edge model fine-tuning; the learning rate dynamically controls the parameter update magnitude to prevent low-precision devices from causing numerical overflow due to gradient explosion.

[0092] The above iterative parameter quantization-aware training method provided by the present application is further described in detail through specific embodiments.

[0093] The iterative parameter quantization-aware training framework provided by the present application is a gradient descent-based quantization-aware training framework, and the core innovation includes two parts of a proportion scheduler and a weight mask function. First, given a model saved in FP16 format, a batch of training data, and start the training loop. In each training step, first use the proportion scheduler to determine the weight quantization proportion in the range of 0~1 , only weights will be quantized in this step. Then use the weight mask function to generate a Boolean mask matrix with the same shape as the weight, where elements are 1 and the remaining elements are 0. The weights with 1 in the mask matrix will be quantized, and the weights with 0 in the mask matrix will not be quantized. Then forward propagation, calculate the loss according to the loss function. That is, only weights are introduced into the quantization error in the forward propagation process. Then the FP16 weight is back propagated to update the original FP16 weight.

[0094] The advantages of the above method provided by the present application are further verified through specific experiments.

[0095] The present application mainly carries out experiments on the LLaMA (Large Language Model Meta AI) series model to verify the superiority of the method, compares the GPTQ, AWQ two post-training methods, and the LLM-QAT, BitDistiller two quantization-aware training methods. Table 1 shows the results of IQAT on the LLaMA-3.1-8B model, and the training data set is a subset of the FineWeb data set, wherein Ours indicates that the method provided by the present application is adopted, and from Table 1, it can be seen that, regardless of the loss function corresponding to the LLM-QAT quantization-aware training method or the loss function corresponding to the BitDistiller quantization-aware training method, the method provided by the present application has achieved good technical effects on the PPL (Perplexity) and AVG indicators compared with the GPTQ, AWQ two post-training methods, wherein PPL is evaluated on the WikiText-2 data set, and AVG is the average of the results of the model on HellaSwag, Winogrande, PIQA, ARC-c, GSM8k, GSM8k-COT, +random in Table 1 indicates that the weights are randomly selected according to the scores, +max_to_min in Table 1 indicates that the weights are selected from large to small according to the scores, +min_to_max in Table 1 indicates that the weights are selected from small to large according to the scores, and avg in Table 1 indicates the average of +random, +max_to_min and +min_to_max. Table 2 is the result on the LLaMA3.2-3B model, wherein Ours indicates that the method provided by the present application is adopted, and avg, +random, +max_to_min, +min_to_max and AVG in Table 2 have the same meaning as avg, +random, +max_to_min, +min_to_max and AVG in Table 1, and from Table 2, it can be seen that, regardless of the loss function corresponding to the LLM-QAT quantization-aware training method or the loss function corresponding to the BitDistiller quantization-aware training method, the method provided by the present application has achieved good technical effects on the PPL (Perplexity) and AVG indicators compared with the GPTQ, AWQ two post-training methods. Table 3 is the result on the dialogue model, wherein MMLU is the data set of various subject test questions, MT-Bench is the data set used to evaluate the ability of large models to reply to dialogues in real scenarios, and Ours indicates that the method provided by the present application is adopted, wherein 8B represents LLaMA-3.1-8B-instruct model, 3B represents LLaMA-3.2-3B model, and the average percentage of improvement represents the average improvement of the method (Ours) provided by the present application on the PPL, MMLU and MT-Bench three evaluation indicators compared with the method corresponding to the upper row.

[0096] Table 1: IQAT results on LLaMA-3.1-8B model

[0097]

[0098] Table 2: IQAT results on LLaMA-3.2-3B model

[0099]

[0100] Table 3: Results on LLaMA-3.1-8B-instruct model and LLaMA-3.2-3B-instruct model

[0101]

[0102] Based on the above iterative quantization-aware training method for large language model end-side deployment, the application further provides an iterative quantization-aware training device for large language model end-side deployment. The following will be described in detail in combination with Figure 3 the device.

[0103] Figure 3 is a structural block diagram of the iterative quantization-aware training device for large language model end-side deployment according to an embodiment of the application.

[0104] As Figure 3 shown, the iterative quantization-aware training device for large language model end-side deployment 300 of the embodiment includes a parameter initial quantization module 310, a quantization ratio and mask matrix acquisition module 320, a quantized parameter matrix acquisition module 330, a model training module 340, a parameter update module 350, and an iterative training and deployment module 360.

[0105] The parameter initial quantization module 310 is configured to randomly select text data samples used in the current training stage from the text training data set, perform initial quantization of all original parameters in the target large language model deployed on the server in the current training stage, and obtain an initial quantized parameter matrix. In an embodiment, the parameter initial quantization module 310 can be configured to perform the operation S210 described above, and thus will not be described again.

[0106] The quantization ratio and mask matrix acquisition module 320 is configured to determine the weight quantization ratio of the current training stage by using a ratio scheduler, and generate a Boolean mask matrix of the current training stage by using a weight mask function. In an embodiment, the quantization ratio and mask matrix acquisition module 320 can be configured to perform the operation S220 described above, and thus will not be described again.

[0107] The quantization parameter matrix obtaining module 330 is configured to replace part of parameters in the initial quantization parameter matrix with original parameters of the target large language model by using the weight quantization scale and the Boolean mask matrix to obtain a quantization parameter matrix. In an embodiment, the quantization parameter matrix obtaining module 330 can be configured to perform the operation S230 described above, and details are not repeated here.

[0108] The model training module 340 is configured to perform self-attention mechanism-based encoding and cross-attention mechanism-based decoding on the text data sample of the current training stage by using the target large language model with the quantization parameter matrix to obtain a text data processing result, and obtain a loss value of the current training stage by using the text data processing result. In an embodiment, the model training module 340 can be configured to perform the operation S240 described above, and details are not repeated here.

[0109] The parameter updating module 350 is configured to perform parameter updating of the target large language model in the current training stage by using the quantization parameter matrix, the loss value of the current training stage, and a preset learning rate. In an embodiment, the parameter updating module 350 can be configured to perform the operation S250 described above, and details are not repeated here.

[0110] The iterative training and deployment module 360 is configured to repeat the operation of the current training stage for each training stage until a preset training condition is met to obtain a large language model after parameter quantization, and deploy the target large language model after parameter quantization to the client, wherein the large language model after parameter quantization is used to process text data of the client. In an embodiment, the iterative training and deployment module 360 can be configured to perform the operation S260 described above, and details are not repeated here.

[0111] According to embodiments of the present application, any multiple of the parameter initial quantization module 310, the quantization scale and mask matrix obtaining module 320, the quantization parameter matrix obtaining module 330, the model training module 340, the parameter updating module 350 and the iterative training and deployment module 360 can be combined in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of other modules and implemented in one module. According to embodiments of the present application, at least one of the parameter initial quantization module 310, the quantization scale and mask matrix obtaining module 320, the quantization parameter matrix obtaining module 330, the model training module 340, the parameter updating module 350 and the iterative training and deployment module 360 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or in any one of the three implementation ways of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the parameter initial quantization module 310, the quantization scale and mask matrix obtaining module 320, the quantization parameter matrix obtaining module 330, the model training module 340, the parameter updating module 350 and the iterative training and deployment module 360 can be at least partially implemented as a computer program module which, when executed, can perform the corresponding functions.

[0112] Figure 4 is a block diagram of an electronic device suitable for implementing the iterative quantization-aware training method for large language model end-side deployment according to embodiments of the present application.

[0113] As shown in Figure 4 , the electronic device 400 according to embodiments of the present application includes a processor 401 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset, and / or a special-purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), etc. The processor 401 can also include an on-board memory for cache use. The processor 401 can include a single processing unit or multiple processing units for performing different actions of the method processes according to embodiments of the present application.

[0114] In the RAM 403, various programs and data required for the operation of the electronic device 400 are stored. The processor 401, the ROM 402, and the RAM 403 are connected to each other via the bus 404. The processor 401 performs various operations of the method flow according to the embodiments of the present application by executing the programs in the ROM 402 and / or the RAM 403. It should be noted that the programs can also be stored in one or more memories other than the ROM 402 and the RAM 403. The processor 401 can also perform various operations of the method flow according to the embodiments of the present application by executing the programs stored in the one or more memories.

[0115] According to the embodiments of the present application, the electronic device 400 can further include an input / output (I / O) interface 405, which is also connected to the bus 404. The electronic device 400 can further include one or more of the following components connected to the input / output (I / O) interface 405: an input part 406 including a keyboard, a mouse, etc.; an output part 407 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 408 including a hard disk, etc.; and a communication part 409 including a network interface card such as a LAN card, a modem, etc. The communication part 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as necessary. A removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 410 as necessary, so that a computer program read out therefrom is installed in the storage part 408 as necessary.

[0116] The present application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, when the one or more programs are executed, the method according to the embodiments of the present application is implemented.

[0117] According to embodiments of the present application, the computer readable storage medium can be a non-transitory computer readable storage medium, such as, for example, without limitation, a portable computer diskette, a hard disk, random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present application, the computer readable storage medium can include the ROM 402 and / or the RAM 403 described above, and / or one or more other memories not expressly described above.

[0118] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functional processes, and operational processes, according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0119] Those skilled in the art will understand that features recited in various embodiments of the present application can be combined and / or integrated in various ways, even if such combinations or integrations are not expressly noted in the present application. In particular, features recited in various embodiments of the present application can be combined and / or integrated in ways that are not expressly noted in the present application, without departing from the spirit and teachings of the present application. All such combinations and / or integrations are within the scope of the present application.

[0120] The embodiments of the present application described above are merely intended to illustrate the present application. These embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present application. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Various alternatives and modifications can be made to the embodiments of the present application by those skilled in the art without departing from the scope of the present application, and such alternatives and modifications are intended to fall within the scope of the present application.

Claims

1. An iterative quantization-aware training method for large language model end-side deployment, characterized in that, The method comprises: randomly selecting text data samples used in the current training stage from a text training data set, performing initial quantization of all original parameters in the target large language model deployed on the server in the current training stage to obtain an initial quantization parameter matrix; determining the weight quantization ratio of the current training stage using a proportion scheduler, and generating a Boolean mask matrix of the current training stage using a weight mask function; replacing part of the parameters in the initial quantization parameter matrix with the original parameters of the target large language model using the weight quantization ratio and the Boolean mask matrix to obtain a quantization parameter matrix; performing encoding based on a self-attention mechanism and decoding based on a cross-attention mechanism on the text data samples of the current training stage using the target large language model with the quantization parameter matrix to obtain a text data processing result, and obtaining a loss value of the current training stage using the text data processing result; performing parameter updating of the target large language model in the current training stage using the quantization parameter matrix, the loss value of the current training stage, and a preset learning rate; repeating the operation of the current training stage for each training stage until a preset training condition is met to obtain a large language model after parameter quantization, and deploying the target large language model after parameter quantization to a client, wherein the large language model after parameter quantization is used to process text data of the client. Determining the weight quantization ratio of the current training stage using a proportion scheduler comprises: determining the weight quantization ratio of the current training stage in a linearly increasing manner using the proportion scheduler based on the total number of stages used for training.

2. The method of claim 1, wherein, Further comprising: limiting the value range of the original parameters of the pre-trained large language model through a parameter truncation operation to obtain the target large language model; initially scoring the original parameters in the pre-trained large language model through a parameter scoring operation to obtain an initial score matrix, and initializing the weight quantization ratio used in the first round of training.

3. The method of claim 2, wherein, The original parameters of the target large language model have the same floating point precision as the original parameters of the pre-trained large language model. The initial score matrix has the same shape as the parameter matrix of the pre-trained large language model and is used to screen quantized model parameters during parameter quantization.

4. The method of claim 1, wherein, The Boolean mask matrix has the same shape as the initial quantization parameter matrix. According to the weight quantization ratio of the current training stage, part of the elements in the Boolean mask matrix are set to a preset value.

5. The method of claim 4, wherein, Generating the Boolean mask matrix of the current training stage using a weight mask function comprises: creating or updating the score matrix of the current training stage using preset calibration data, the quantization parameter matrix of the previous training stage, and the original parameters of the target large language model; processing the score matrix and the weight quantization ratio of the current training stage using a preset selection function to obtain the Boolean mask matrix of the current training stage.

6. The method of claim 5, wherein, The creating or updating of the score matrix of the current training stage comprises: calculating a local quantization loss of the current training stage by using the preset calibration data, the quantization parameter matrix of the previous training stage and the original parameters of the target large language model; calculating a second-order partial derivative of the original weight of the local quantization loss of the current training stage to obtain a Hessian matrix of the current training stage; processing the Hessian matrix of the current training stage, the quantization parameter matrix of the previous training stage and the original parameters of the target large language model to obtain the score matrix of the current training stage.

7. The method of claim 1, wherein, The parameter updating of the target large language model in the current training stage comprises: calculating a loss value of the current training stage by using the quantization parameter matrix, the original parameters of the target large language model and the text data sample used in the current training stage; deriving the loss value of the current training stage to obtain a gradient value of the current training stage; performing parameter updating of the target large language model in the current training stage by gradient descent operation by using the preset learning rate and the gradient value.

8. The method according to any one of claims 1 to 7, characterized in that, The target large language model comprises a multi-language machine translation model, and the text training data set comprises a corpus with a source language and a target language alignment relationship. The client comprises a mobile communication device and an embodied intelligent robot.

9. The method according to any one of claims 1 to 7, characterized in that, The target large language model comprises an intelligent question and answer model, and the text training data set comprises a structured knowledge base with question-text answer labels and a multi-modal data set with image-question-text answer triple labels. The client comprises an intelligent home terminal, an intelligent medical terminal, a customer service terminal and an intelligent education terminal.

Images