Model quantification method combined with efficient parameter transfer learning

Abstract

The invention discloses a model quantification method combined with efficient parameter transfer learning, which comprises the following steps: constructing an M1 model based on a pre-trained ViT model, and adding a low-rank adaptive matrix in an attention layer of the M1 model to obtain an M2 model added with a LoRA module; a plurality of sub-modules in the M2 model are converted into quantization modules, and a quantized M3 model is obtained; setting a weight quantization bit number and an activation quantization bit number, and calibrating the model M3 combined with the preset transfer learning mode to obtain each calibrated model; respectively training each calibrated model according to training logic by utilizing a preset training set; and evaluating the trained QATL-M3 model, the trained PTLQ-M3 model and the trained GQTL-M3 model by using a preset verification set to obtain an evaluation result. The invention relates to the field of deep learning, and solves the technical problems that an existing model quantification method is high in model calculation cost and prone to overfitting under the condition of low-bit quantification.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and relates to model quantization technology, specifically a model quantization method combined with efficient parameter transfer learning. Background Art

[0002] Pre-training and fine-tuning of large-scale datasets are the cornerstones of continuous breakthroughs in the field of vision, and they have demonstrated excellent performance in tasks such as classification, detection, and segmentation. As the versatility of pre-trained models increases, fine-tuning and feature alignment for downstream tasks become particularly critical. Full parameter fine-tuning, that is, initializing the model with pre-trained weights and then performing comprehensive parameter adjustments, is a common strategy to adapt to specific downstream tasks. However, this approach requires storing independent parameter sets for each task and deployment environment, which significantly increases storage costs. At the same time, when downstream task data is scarce, full parameter fine-tuning may lead to model overfitting.

[0003] To address the challenge of fine-tuning large models, parameter efficient transfer learning (PETL) provides a solution that focuses on tuning a very small portion of parameters, typically less than 1% of the total trainable parameters, thereby significantly reducing storage requirements and simplifying the optimization process. This approach helps alleviate the overfitting problem when applying a wide range of pre-trained models to a specific dataset, allowing the PETL strategy to achieve or even exceed the performance of full parameter fine-tuning. However, existing PETL methods mainly focus on reducing the training cost of downstream tasks, while ignoring the storage and computational costs of the model, which is particularly important when deploying large visual models on embedded platforms.

[0004] Model quantization is an effective way to reduce memory usage by converting floating-point data to low-precision fixed-point data without changing the model architecture. Some studies have tried to combine LoRA (Low-Rank Adaptation, a model parameter adjustment technique) with model quantization on large language models, but due to the differences in data representation, variability, and semantics between visual tasks and language tasks, visual tasks face greater domain adaptability challenges. In addition, downstream tasks usually provide less image data, making the combination of transfer learning and quantization more difficult. Combining quantization-aware training (QAT) with transfer learning may introduce global errors, which is only constrained by task losses, exacerbating the overfitting problem of PETL on a small amount of downstream task data; while combining post-training quantization (PTQ) with transfer learning, quantization is performed after PETL fine-tuning. At this time, the focus is on the quantization loss based on block reconstruction rather than the downstream task loss, which may cause the upper limit of quantization accuracy to be lower than the full-precision result of PETL, especially in the case of low-bit quantization, which is prone to underfitting problems. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a model quantization method combined with efficient parameter transfer learning, which is used to solve the technical problems that the existing model quantization method has high model calculation cost and is prone to overfitting under low-bit quantization conditions.

[0006] To achieve the above object, the present invention provides a model quantization method combined with efficient parameter transfer learning, comprising:

[0007] S1, builds the M1 model based on the pre-trained ViT model, and adds a low-rank adaptive matrix to the attention layer of the M1 model to obtain the M2 model after adding the LoRA module;

[0008] S2, converting several submodules in the M2 model into quantized modules to obtain the quantized M3 model; wherein the several submodules include several attention layers, several linear layers and several convolutional layers;

[0009] S3, setting the number of weight quantization bits and the number of activation quantization bits, calibrating the model M3 combined with the preset transfer learning method, and obtaining the calibrated QATL-M3 model, PTLQ-M3 model and the calibrated GQTL-M3 model; wherein the preset transfer learning methods include: quantization-aware transfer learning QATL, post-transfer learning quantization PTLQ and progressive quantization joint transfer learning GQTL;

[0010] S4, using the preset training set to perform single-stage training on the calibrated QATL-M3 model, double-stage training on the PTLQ-M3 model, and single-stage training on the calibrated GQTL-M3 model, to obtain the trained QATL-M3 model, PTLQ-M3 model, and GQTL-M3 model;

[0011] S5, use the preset validation set to evaluate the trained QATL-M3 model, PTLQ-M3 model and GQTL-M3 model, and obtain the evaluation results of the M3 model under different preset transfer learning methods.

[0012] Based on the above technical process, the present invention can effectively improve quantization efficiency, reduce the amount of calculation and solve the overfitting problem. Among them, in S1, by adding a low-rank adaptive matrix to the attention layer of the ViT model to construct the M2 model, the number of trainable parameters is reduced, which is conducive to subsequent quantization processing; S2 quantizes the key submodules, directly reduces memory usage and improves quantization efficiency, and low-precision data operations also reduce the amount of calculation; S3-S4 uses a variety of preset transfer learning methods for calibration and training, QATL makes the model adapt to quantization early, PTLQ uses the full-precision training advantage before quantization, GQTL trains block by block to reduce the amount of calculation and combines multiple losses to avoid overfitting, and designs the optimal quantization scheme for different quantization paradigms to improve the overall quantization efficiency. At the same time, the characteristics of the LoRA module and various transfer learning methods effectively alleviate the overfitting problem, which helps to improve the generalization ability and robustness of the model.

[0013] Furthermore, adding a low-rank adaptive matrix in the attention layer of the M1 model includes:

[0014] S11, obtain the initial weight matrix W0 of the attention layer of the M1 model; R represents a real number, d in represents the input dimension of W0, d out represents the output dimension of W0;

[0015] S12, adding low-rank adaptive matrices A and B to W0 to obtain the M2 model after adding the LoRA module; r represents the rank of the low-rank adaptive matrix.

[0016] By adding low-rank adaptive matrices A and B, the adjustment of the original weight matrix can be converted into an operation on the low-rank matrix. In this way, in the subsequent quantization process, only a small number of low-rank matrix parameters need to be processed, which greatly reduces the number of parameters that need to be stored and reduces storage costs. At the same time, in the calculation process, since the amount of matrix multiplication calculation is related to the matrix dimension, the computational complexity of the low-rank matrix is ​​much lower than the operation of the original high-dimensional weight matrix, which significantly reduces the amount of calculation and improves the quantization efficiency.

[0017] Furthermore, the conversion of several submodules in the M2 model into quantization modules includes:

[0018] S21, weight quantization and activation quantization are performed on the attention layer in the M2 model, and attention calculation is performed on the weight quantization results and the activation quantization results to obtain an attention layer quantization module;

[0019] S22, using a quantizer to quantize the linear layer to obtain a linear layer quantization module;

[0020] S23, using a quantizer to quantize the convolution layer to obtain a convolution layer quantization module.

[0021] Attention layers, linear layers, and convolutional layers are usually the main contributors to computation and storage in deep learning models. Since quantization can convert the original floating-point data into low-precision fixed-point data, reducing the number of bytes of data storage, quantizing these submodules can directly reduce the model's memory requirements.

[0022] Furthermore, the weight quantization of the attention layer in the M2 model includes:

[0023] S21-11, add the LoRA module to Q and V in the attention layer, and initialize the low-rank matrix B' in the LoRA module to obtain the initialized low-rank matrix B';

[0024] S21-12, merge the initialized low-rank matrix B' into Q and V, and use a weight quantizer to quantize the weights of Q, K, and V respectively to obtain the quantized weights of Q, K, and V;

[0025] S21-13, concatenate the quantized Q, K, and V along dimension 0 to obtain QKV.

[0026] It should be noted that Q, K, and V represent the query vector, key vector, and value vector in the attention layer of the ViT model, respectively.

[0027] Adding LoRA modules to Q and V in the attention layer and merging and quantizing after initializing the low-rank matrix B' is a step-by-step optimization strategy. First, the addition of the LoRA module provides a more flexible way to adjust the weights. By adjusting the low-rank matrix, it can adapt to different task requirements without changing the original weight structure too much. Initializing the low-rank matrix B' provides a reasonable initial value for subsequent weight adjustments, which helps the model converge faster. It is then merged into Q and V for weight quantization. The weights can be quantized more accurately while considering the influence of the low-rank matrix, so that the quantized weights can better reflect the needs of the model, improve quantization accuracy, and reduce the impact of quantization errors on model performance.

[0028] Furthermore, the activation quantization process includes:

[0029] S21-21, obtain input feature x∈(B,N,C); where B represents the batch size, that is, the number of samples processed at one time, N represents the sequence length, that is, the number of image blocks, and C represents the feature dimension;

[0030] S21-22, reshape the QKV into (3, B, num_heads, N, C / / num_heads), and use the permute operation to separate the reshaped QKV to obtain Q, K, and V; where num_heads represents the number of attention heads;

[0031] S21-23, using the activated quantizer to quantize Q, K, and V respectively, to obtain activated quantized Q', K', and V'.

[0032] In the attention layer, Q, K, and V are intermediate results obtained based on input features and weight quantization. This calculation method can make full use of the input feature information and the quantized weight information, so that the generated QKV can more accurately reflect the computing requirements of the model in a quantized environment, providing a more suitable basis for subsequent attention calculations, helping to improve the accuracy of quantization and reduce information loss caused by quantization.

[0033] Furthermore, performing attention calculation on the weight quantization result and the activation quantization result includes:

[0034] S21-31, perform dot product operation on the transpose of Q' and K' and multiply by the scaling factor to obtain the attention score matrix;

[0035] S21-32, after normalizing the attention score matrix using the softmax function, the drop function is applied to discard it to obtain the processed attention score;

[0036] S21-33, quantize the processed attention score using a quantizer, and multiply it by V' to obtain a quantized and weighted attention score;

[0037] S21-34, linearly transform the quantized and weighted attention scores and apply the drop function to obtain the output result of the attention calculation.

[0038] The quantization operation reduces the amount of calculation and speeds up the calculation process; multiplication by V' implements a weighted operation based on the attention score, allowing the model to focus on important information; the linear transformation further adjusts the output features, and the application of the drop function helps prevent overfitting. By randomly discarding some information, the generalization ability of the model is enhanced, so that the model can achieve a better balance between performance and efficiency in the attention calculation process after quantization, and adapt to different task requirements.

[0039] Furthermore, the calibration of the model M3 combined with the preset transfer learning method includes:

[0040] For the M3 model combined with QATL, the activation quantization is calibrated first, and then the weight quantization is calibrated;

[0041] For the M3 model combined with PTLQ, calibration is performed after the LoRA module training is completed;

[0042] For the M3 model combined with GQTL, only the calibration of weight quantization was performed.

[0043] In model quantization, the calibration methods of model M3 are different under different transfer learning methods. For QATL, activation quantization calibration is performed first because its activation value changes dynamically. Calibration can determine the quantization parameters according to its distribution, reduce information loss and calculation error, stabilize training, and then optimize weight quantization based on this to balance weight accuracy and storage calculation overhead. PTLQ is calibrated after the training of the LoRA module. It can take advantage of the full-precision training to learn features, avoid quantization interference, reduce the impact of global errors on LoRA parameter adjustment, and quantize on the basis of retaining valid features. GQTL only performs weight quantization calibration because the block-by-block training focuses on weight quantization, which has an impact on each block. The activation impact is indirectly considered through the joint task loss, and the adjustment of block-by-block training parameters is simplified to achieve performance optimization.

[0044] Furthermore, the single-stage training of the calibrated QATL-M3 model includes:

[0045] S41-1, obtaining an ordered list of several modules in the calibrated QATL-M3 model and quantized scaling parameters;

[0046] S41-2, the initial weight matrix W0 of the M1 model in the QATL-M3 model after freezing calibration;

[0047] S41-3, use the optimizer and scheduler to dynamically adjust the learning rate, and according to the formula Calculate the loss of the first task, and update the low-rank adaptive matrices A and B as well as the quantization scaling parameters and learning rate through back propagation; where arg min represents the value of the input parameter when the formula reaches the minimum value, s w Represents the quantization scale of the weight, s a represents the activation quantization scale, L T1 represents the first task loss, FQNet represents the calibrated QATL-M3 model, i.e., the fully quantized network, represents the weight matrix after quantization, represents the activated feature after quantization, and y represents the true label;

[0048] S41-4, repeat S41-1 to S41-3 to perform a training cycle until a first preset number of training times is reached.

[0049] By obtaining the module list and learnable parameters, building training dependencies, and clarifying the range of learnable parameters, unnecessary calculations can be avoided; then freezing the initial weight matrix is ​​to retain the pre-training knowledge and prevent overfitting, while focusing on the adjustment of low-rank matrices and quantization parameters, so that the model can adapt to downstream tasks and optimize quantization based on pre-training knowledge; then calculating the task loss and updating the parameters according to the formula can accurately reflect the prediction error under the quantization state, and continuously optimize the model, which helps to improve the training effect; finally, repeating the training cycle allows the model to gradually optimize the performance and stop at the appropriate training stage to prevent underfitting or overfitting and improve generalization ability.

[0050] Furthermore, the two-stage training of the PTLQ-M3 model includes:

[0051] S42-1, obtaining an ordered list of several modules in the calibrated PTLQ-M3 model, and obtaining parameters of the LoRA module to be optimized from the ordered list to obtain LoRA parameters;

[0052] S42-2, use the optimizer and scheduler to dynamically adjust the learning rate, and according to the formula Calculate the loss of the second task, update the LoRA parameters and learning rate through back propagation, and complete the first stage of training, where L T2 represents the second task loss, FPnet represents the calibrated QATL-M3 model, i.e., the full-precision network, W represents the full-precision matrix weight without quantization, and x represents the full-precision activation feature without quantization;

[0053] S42-3, using a quantizer to perform weight quantization calibration and activation quantization calibration on the PTLQ-M3 model that has completed the first stage of training, to obtain a calibrated PTLQ-M3 model;

[0054] S42-4, train the quantization scale of the calibrated PTLQ-M3 model block by block, through the formula Calculate the reconstruction loss LT3 between the quantized output and the floating-point output, and perform back-propagation to update the quantization parameters to complete the second stage of training; Represents the unquantized floating-point output of the lth module. represents the quantized output of the lth module after quantization processing, W x represents the weight activation multiplication without quantization, represents the multiplication of quantized weight activations, and F represents the Frobenius norm;

[0055] S42-5, repeat S42-1 to S42-4 to perform a training cycle until a second preset number of training times is reached.

[0056] In the first stage of PTLQ, full-precision training aims to leverage the powerful feature extraction capabilities of the pre-trained model to adapt to downstream tasks by adjusting the LoRA module parameters; the task loss function is then used to measure how well the model fits the downstream tasks at full precision, and the optimizer is used to adjust the LoRA parameters by minimizing the loss; in the second stage of PTLQ, the focus is on optimizing the performance of the quantized model to ensure that the output of the quantized model is as close as possible to the output of the full-precision model; the reconstruction loss measures the difference in output between the quantized model and the full-precision model. By minimizing the reconstruction loss to adjust the quantization parameters, the quantization accuracy can be improved.

[0057] By separating the LoRA module training and quantization calibration, the global error that may occur during the quantization process interferes with the LoRA module parameter adjustment. During the LoRA module training process, the model focuses on adapting to downstream tasks by adjusting a small number of parameters without being affected by the reduction in numerical accuracy and error accumulation caused by quantization.

[0058] Furthermore, the calibrated GQTL-M3 model is trained block by block in a single stage, including:

[0059] S43-1, obtaining an ordered list of several modules in the calibrated GQTL-M3 model, and sequentially obtaining the lth block of modules from the ordered list, and freezing the first l-1 blocks of modules;

[0060] S43-2, obtain the learnable parameters of the lth block module, use the optimizer and scheduler to dynamically adjust the learning rate, according to the alternating quantization migration loss function formula Calculate the joint task loss L T4 , and update the learnable parameters and learning rate through back propagation; where FQNet' represents the quantized network of the first l blocks of the calibrated GQTL-M3 model; represents the quantized weight W and feature x of the first l-1 blocks, {W i>l ,x i>l} represents the unquantized weight W and feature x after the lth block module, θ q With θ t represents a learnable parameter, and and Denote the quantization scale of the weight of the lth block module and the quantization scale of the activation, respectively, and B i>l and A i>l Indicates the LoRA parameters after the lth module;

[0061] S43-3, repeat S43-1 to S41-2, and perform a block-by-block training cycle until a number of modules are traversed and a third preset number of training times is reached.

[0062] GQTL adopts a block-by-block training method based on the idea of ​​modular model optimization. In a deep learning model, each module has relatively independent functions. Block-by-block training can more finely control the model training process and optimize according to the characteristics of each module. At the same time, freezing the previously processed modules helps to simplify the training process, reduce computational complexity, and avoid unnecessary adjustments to the trained parts.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] In terms of reducing computational costs, the present invention reduces the number of trainable parameters and computational complexity by introducing a low-rank adaptive matrix into the model, and also reduces the amount of computation in the quantization process; then, the key submodules are quantized, the data is converted into low-precision fixed-point data, and the high efficiency of hardware operations on low-precision data is used to accelerate the reasoning process, so that the quantization of each submodule can be coordinated to optimize the calculation; at the same time, the training methods under different transfer learning methods are also optimized: the QATL-M3 model freezes the initial weight matrix, focuses on the low-rank matrix and quantization scaling parameter adjustment to avoid repeated calculations; the PTLQ-M3 model is optimized in stages, the first stage is full-precision training of the LoRA module to avoid quantization interference, and the second stage optimizes the quantization parameters block by block; the GQTL-M3 model balances tasks and quantization through block-by-block training, only processing one module at a time and freezing the previous module, reducing the number of computational parameters and reducing computational complexity; through transfer learning optimization of different quantization paradigms, the model can still run efficiently at extremely low bit quantization, which is particularly suitable for resource-constrained environments such as embedded devices;

[0065] In alleviating the overfitting problem, the present invention limits the scope of parameter adjustment through the application of low-rank matrices, avoids the model's over-adaptation to the training data, and thus can effectively control the amplitude of parameter adjustment under various transfer learning methods, thereby enhancing the generalization ability of the model; QATL stabilizes the training process by balancing model learning and quantization effects at an early stage, activating quantization calibration first and then weight quantization calibration; PTLQ takes advantage of full-precision training before quantization to avoid quantization interference with LoRA module parameter adjustment; GQTL further alleviates the risk of overfitting and underfitting of the model through block-by-block training combined with an alternating quantization transfer loss function, thereby enhancing the robustness of the model under different conditions, ensuring that the model can maintain good performance under extremely low-bit quantization, and improving the stability and practicality of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 A flow chart of a model quantization method combined with efficient parameter transfer learning provided by the present invention;

[0068] Figure 2 and Figure 3 Evaluation and comparison results of the trained GQTL-M3 model provided by the present invention on the VTAB-1K dataset. DETAILED DESCRIPTION

[0069] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] See also Figure 1 The first aspect of the present invention provides a model quantization method combined with efficient parameter transfer learning, including:

[0071] S1, builds the M1 model based on the pre-trained ViT model, and adds a low-rank adaptive matrix to the attention layer of the M1 model to obtain the M2 model after adding the LoRA module;

[0072] S2, converting several submodules in the M2 model into quantized modules to obtain the quantized M3 model; wherein the several submodules include several attention layers, several linear layers and several convolutional layers;

[0073] S3, setting the number of weight quantization bits and the number of activation quantization bits, calibrating the model M3 combined with the preset transfer learning method, and obtaining the calibrated QATL-M3 model, PTLQ-M3 model and GQTL-M3 model; wherein the preset transfer learning methods include: quantization-aware transfer learning QATL, post-transfer learning quantization PTLQ and progressive quantization joint transfer learning GQTL;

[0074] S4, using the preset training set to perform single-stage training on the calibrated QATL-M3 model, double-stage training on the PTLQ-M3 model, and single-stage training on the calibrated GQTL-M3 model, to obtain the trained QATL-M3 model, PTLQ-M3 model, and GQTL-M3 model;

[0075] S5, use the preset validation set to evaluate the trained QATL-M3 model, PTLQ-M3 model and GQTL-M3 model, and obtain the evaluation results of the M3 model under different preset transfer learning methods.

[0076] The present invention aims to solve the problems of high computational cost and easy overfitting in the existing model quantization method under low-bit quantization. The overall method includes building and optimizing the model based on the pre-trained ViT model, quantizing some submodules of the model, calibrating and training with different transfer learning methods combined with quantization, and finally evaluating the performance of the model under different preset transfer learning methods.

[0077] Specifically, in step S1, the M1 model is first built based on the pre-trained ViT model, and a low-rank adaptive matrix is ​​added to the attention layer of the M1 model to obtain the M2 model after adding the LoRA module:

[0078] S11, obtain the initial weight matrix W0 of the attention layer of the M1 model; R represents a real number, d in represents the input dimension of W0, d out represents the output dimension of W0;

[0079] S12, add low-rank adaptive matrix A in W0 and B The M2 model after adding the LoRA module is obtained; wherein r represents the rank of the low-rank adaptive matrix.

[0080] LoRA is a commonly used model fine-tuning method. It freezes the model parameters and only trains two low-rank matrices A and B. The matrix is ​​used to approximate the change ΔW of the entire model weight parameter during the fine-tuning process. Since the gradient of the entire model does not need to be maintained during back propagation, the fine-tuning cost of the model is reduced.

[0081] In this embodiment, by introducing a low-rank matrix without changing the original model architecture, the model has greater flexibility in adapting to downstream tasks. While maintaining most of the pre-trained knowledge of the model, it can quickly adapt to new tasks by adjusting the low-rank matrix, avoiding the risk of overfitting caused by full parameter adjustment, and can better balance model performance and quantization accuracy during quantization, because it can more effectively explore suitable quantization parameters in the low-rank space.

[0082] Then in step S2, several attention layers, several linear layers, and several convolutional layers in the M2 model are converted into quantization modules to obtain the quantized M3 model:

[0083] First, the weight quantization and activation quantization of the attention layer:

[0084] S21-1, for weight quantization, first add the LoRA module to Q and V in the attention layer, initialize the low-rank matrix B', then merge them into Q and V, and use the weight quantizer WeightQuantizer to quantize the weights of Q, K, and V respectively; then concatenate the quantized Q, K, and V along dimension 0 to obtain QKV;

[0085] It should be noted that Q, K, and V represent the query vector, key vector, and value vector in the attention layer of the ViT model, respectively; and the LoRA module consists of the following three parts:

[0086] Low rank adaptive matrix A Low rank adaptive matrix B These two matrices simulate the update of the model parameters through the matrix multiplication AB, whose product has the same dimensions as the full fine-tuning update, but significantly reduces the number of parameters that must be trained;

[0087] And the scaling factor α: used to scale the low-rank update matrix AB, the default value is 1, and the importance of the pre-trained model and the new specific task adaptation can be balanced by changing the value of α.

[0088] S21-2, for activation quantization, first obtain the input feature x∈(B,N,C); where B represents the batch size, that is, the number of samples processed at a time, N represents the sequence length, that is, the number of image blocks, and C represents the feature dimension; then reshape the QKV obtained by weight quantization to (3, B, num_heads, N, C / / num_heads), and then use the permute operation to separate the reshaped QKV to obtain Q, K, and V; finally, use the activation quantizer ActivationQuantizer to quantize Q, K, and V respectively to obtain Q', K', and V' after activation quantization; where num_heads represents the number of attention heads;

[0089] The activation value changes dynamically during model calculation, and its quantization needs to consider the combined effects of input features and model weights. Reshaping QKV and separating it using the permute operation before quantizing it is based on the different computational roles and statistical characteristics of Q, K, and V in the attention mechanism. The reshaping and separation operations can reorganize QKV according to the logical structure of attention calculation, so that each vector can be independently quantized according to its own characteristics, thereby improving the performance of the entire attention calculation process after quantization;

[0090] After weight quantization and activation quantization, attention calculation is performed on the weight quantization results and activation quantization results to obtain the linear layer quantization module:

[0091] S21-3, perform dot product operation on the transpose of Q' and K' and multiply it by the scaling factor to obtain the attention score matrix; where the scaling factor is d k is the dimension of K; then the softmax function is used to normalize the attention score matrix, and then the drop function is applied to discard it to obtain the processed attention score;

[0092] Then, the processed attention score is quantized by the quantizer and multiplied by V' to obtain the quantized and weighted attention score; finally, the quantized and weighted attention score is linearly transformed and the drop function is applied to obtain the output result of the attention calculation, thereby obtaining the calculation process of the attention layer quantization module;

[0093] In deep learning models, the attention mechanism is an important information processing method, and its calculation process needs to comprehensively consider weights and activation information. The transpose of Q' and K' is dot-producted and multiplied by the scaling factor, and then normalized using the softmax function to ensure the stability of the attention score calculation. During the quantization process, due to the reduction in data accuracy, the numerical range may change, and it is easy for the value to be too large or too small, affecting the calculation results. The scaling factor can adjust the numerical range of the dot product result to make it more suitable for the calculation of the softmax function, and the softmax normalization converts the attention score into a reasonable probability distribution, ensuring the stability and accuracy of the attention calculation in a quantized environment, so that the model can correctly allocate attention weights;

[0094] S22, for the quantization of the linear layer: the parameter setting of the linear layer forward propagation does not require additional parameters, only the input and weights are needed, and then the linear layer is quantized using a quantizer to convert the floating-point data in the linear layer into low-precision fixed-point data, thereby obtaining a linear layer quantization module;

[0095] S23, for the quantization of the convolution layer: the parameter setting of the forward propagation of the convolution layer requires multiple parameters to define the convolution operation, including stride, padding, expansion and number of groups. The convolution layer is then quantized using a quantizer to quantize the weights and activation values ​​in the convolution layer to obtain a convolution layer quantization module.

[0096] Different sub-modules in the model have different characteristics in computing and storage. Targeted optimization of key sub-modules can effectively improve the efficiency of the entire system.

[0097] In this embodiment, the quantized submodule can take advantage of low-precision data operations during the calculation process. Low-precision data operations are faster and can accelerate the reasoning process of the model. For example, at the hardware level, many processors support integer operations more efficiently. The quantized linear layer and convolutional layer can use integer operations instead of floating-point operations, reducing the calculation time and improving the calculation efficiency of the overall model, so that the model can process input data faster after quantization, meeting the application scenarios with high real-time requirements.

[0098] In step S3, by setting the weight quantization bit number and the activation quantization bit number, the model M3 combined with different transfer learning methods is calibrated to obtain the calibrated QATL-M3 model, PTLQ-M3 model and GQTL-M3 model;

[0099] Specifically, first, a preset number of training samples (such as 256) are loaded to form a preset training set. Then, during the training process, the context is set so that no gradient calculation is required. Then, different quantization paradigms are combined with transfer learning to form a new quantization method, and calibration is performed according to the following logic:

[0100] Calibration of Quantization Aware Transfer Learning (QATL), first calibrate activation quantization, then calibrate weight quantization;

[0101] Post-transfer learning quantization (PTLQ) calibration will be performed after the LoRA training in the training phase;

[0102] The calibration of Progressive Quantization Joint Transfer Learning (GQTL) only performs calibration of weight quantization.

[0103] QATL combines quantization-aware training with transfer learning, and considers the impact of quantization on the model during the training process, trying to adapt the model to the numerical changes brought about by quantization during the training phase. It uses the pre-trained model knowledge in transfer learning to introduce quantization operations during the fine-tuning process, with the aim of achieving efficient quantization while maintaining model performance. Therefore, in QATL, the calibration process needs to first focus on activation quantization so that its quantization parameters can adapt to changes in input data; then, weight quantization calibration uses the calibrated activation quantization information to ensure that weight quantization does not affect the expressiveness of the model while reducing storage and computational overhead. This calibration strategy helps the QATL-M3 model gradually adapt to quantization changes during training, while using pre-training knowledge to balance accuracy and efficiency;

[0104] The PTLQ method is to pre-train and fine-tune the full-precision model to adapt to downstream tasks, and then quantize it. Therefore, in the calibration stage, the model has learned effective feature representations. Quantization at this time can maintain model performance while reducing storage and computing costs. In addition, separating LoRA module training from quantization calibration can reduce the interference of quantization on model parameter adjustment, ensuring that the model can still maintain the performance of full-precision training after quantization;

[0105] GQTL gradually optimizes model performance by quantizing block by block. In this method, weight quantization is the focus of calibration because weights play a core role in the model and are relatively stable. Therefore, a block-by-block training approach is adopted, allowing fine quantization parameter adjustment of the weights of each module to maintain computational accuracy and reduce storage requirements. At the same time, the impact of activation quantization is indirectly considered through the calculation of the joint task loss without the need for separate calibration. This design simplifies the calibration process, allowing GQTL to gradually optimize model performance in block-by-block training and improve overall performance under low-bit quantization.

[0106] According to the different quantization paradigms mentioned in S3, combined with transfer learning, a new quantization method is formed. They carry out corresponding training processes respectively:

[0107] S41, a single-stage training method is used for the calibrated QATL-M3 model:

[0108] The training process of QATL is a single-stage training based on task loss. First, an ordered list of modules in the model and learnable parameters are obtained. Then, the Adam optimizer and CosineAnnealingLR scheduler are defined to dynamically adjust the learning rate. Finally, a training loop is performed to calculate the task loss using the cross entropy loss function, compare the model output with the true label and perform backpropagation to calculate the gradient, and update the model parameters and learning rate.

[0109] In QATL, the weight W0 of the pre-trained model is frozen, and only B, A and quantization scaling factors of LoRA are trained. The optimized loss and parameters can be expressed as: Among them, arg min represents the value of the input parameter when the formula reaches the minimum value, s w Represents the quantization scale of the weight, s a represents the activation quantization scale, L T1 represents the first task loss calculated using the cross entropy loss function, FQNet represents the calibrated QATL-M3 model, i.e., the fully quantized network, represents the weight matrix after quantization, represents the activated feature after quantization processing, and y represents the real label; and the above steps are trained in a loop until a first preset number of training times is reached; wherein the first preset number of training times is determined according to the experimental equipment and experimental requirements.

[0110] S42, a two-stage training method is used for the PTLQ-M3 model:

[0111] PTQ uses a small amount of calibration data to achieve quantization under a pre-trained model. Since downstream tasks require model migration and adaptation, the training process of PTLQ is a two-stage training process;

[0112] The first stage is to perform transfer learning for LoRA training. At this time, the model is a floating-point model and does not participate in quantization. First, get the ordered list of model modules, then get the learnable parameters for LoRA training, then set the Adam optimizer and the learning rate scheduler of cosine decay (CosineAnnealingLR), and finally perform a training loop to update the LoRA parameters through back propagation;

[0113] The goal of transfer learning at this stage is to obtain a full-precision model for downstream tasks. This stage can be expressed as the following optimization problem: Among them, L T2 represents the second task loss calculated using the cross entropy loss function, FPNet represents the calibrated QATL-M3 model, i.e., the full-precision network, W represents the full-precision matrix weights that have not been quantized, and x represents the full-precision activation features that have not been quantized; at the same time, in this stage, only A and B and the classification head of the model are learnable, and other parameters are frozen;

[0114] In the second stage, the model is trained for post-migration quantization. The model is first calibrated for weight quantization and then for activation quantization. The model accuracy is evaluated and the accuracy on the test set before quantization is recorded. Then, the quantization scale is gradually trained for each block of the model. The mean square error loss function is used for optimization. Finally, the quantization parameters are updated by calculating the reconstruction loss between the quantized output and the floating-point output.

[0115] During the parameter calibration process, the quantization scale s w ,s a It is adjusted in blocks. Specifically, when adjusting the Lth block, the quantized output of the previous block is used as input, and then the loss between the quantized output and the full-precision output of the Lth block is calculated. This process can be described as follows: in, Represents the unquantized floating-point output of the lth module. represents the quantized output of the lth module after quantization processing, W x represents the weight activation multiplication without quantization, represents the multiplication of quantized weight activations, and F represents the Frobenius norm; finally, the above steps are repeated to perform a training cycle until a second preset number of training times is reached; wherein the second preset number of training times is determined according to the experimental equipment and experimental requirements.

[0116] It should be noted that the Frobenius norm is a method for measuring the size of a matrix, defined as the square root of the sum of the squares of all elements in the matrix. Using the Frobenius norm means calculating the mean square error between the quantized activation and the theoretical activation, taking into account the differences in all elements.

[0117] S43, the calibrated GQTL-M3 model is trained in a block-by-block single-stage manner:

[0118] The training process of GQTL is a single-stage training process based on the reconstruction task loss, which is carried out block by block. First, all quantization blocks before the lth block must be frozen and their parameters are excluded from the optimization range to prevent quantization overfitting; then the operation on the current quantization block is: first initialize the activation quantization parameters, calibrate the activation and weight quantization of the current block without calculating the gradient, then select the corresponding learnable parameters and define the optimizer, and finally merge the calculated task loss and quantization loss in the training cycle to optimize the learnable parameters in the update model; compare the floating-point output and quantization output of the current module, and use the local quantization loss to optimize the quantization scaling parameters of the current quantization block, while continuing to obtain the downstream global task loss through the remaining unquantized blocks, and optimize the LoRA parameters, LayerNorm parameters and quantization scaling parameters of the current quantization block and subsequent blocks to compensate for the underfitting problem caused by the accumulated quantization error;

[0119] In GQTL, an alternating quantization transfer loss is designed to ensure the transfer and quantization performance of the model. As time goes by, the alternating quantization transfer loss function alternates the parameter optimization direction between the transfer task and quantization. The optimization parameters of the task will gradually decrease, and the quantization process will gradually saturate.

[0120] Specifically, the loss function is constructed as follows: Among them, FQNet' represents the quantized network of the first l blocks of modules in the calibrated GQTL-M3 model; represents the quantized weight W and feature x of the first l-1 blocks, {W i>l ,x i>l} represents the unquantized weight W and feature x after the lth block module, θ q With θ t represents a learnable parameter, and and Denote the quantization scale of the weight of the lth block module and the quantization scale of the activation, respectively, and B i>l and A i>l Indicates the LoRA parameters after the lth module;

[0121] Repeat the above steps to perform a block-by-block training cycle until a number of modules are traversed and a third preset number of training times is reached; similarly, the third preset number of training times is determined according to the experimental equipment and experimental requirements.

[0122] Finally, in S5, the trained QATL-M3 model, PTLQ-M3 model, and GQTL-M3 model are evaluated using the preset validation set to obtain the evaluation results of the M3 model under different preset transfer learning methods.

[0123] The second aspect of the present invention provides an evaluation result of the trained GQTL-M3 model using a preset validation set:

[0124] In this example, the performance of the GQTL-M3 model trained by the method of the present invention was evaluated on the VTAB-1K dataset and compared with different quantization methods: LoRA, LSQ, BRECQ and RepQ-ViT. The evaluation results are shown in Figure 2. Figure 2 and Figure 3 As shown;

[0125] Among them, the VTAB-1K dataset is a benchmark dataset designed to evaluate general visual representations. It contains 19 image classification tasks and is divided into three areas:

[0126] Natural image datasets: such as Caltech101, CIFAR-100, 102Category Flower Dataset, Oxford IIIT Pet Dataset, SUN397, SVHN, etc.

[0127] Domain-specific datasets: for example, Diabetic Retinopathy, Eurosat, PatchCamelyon, Resisc45, etc.

[0128] Structured task datasets: such as CLEVR Distance Prediction, CLEVR Counting, DSPrites Orientation Prediction, DSPrites Location Prediction, Small NORBAzimuth Prediction, Small NORB Elevation Prediction, Kitti Distance Prediction, etc.; each category represents a specific dataset.

[0129] The average accuracy in the figure refers to the experimental results of the natural data set to find an average accuracy x1, the experimental results of the specific field data set to find an average accuracy x2, and the experimental results of the structured data set to find an experimental result x3; then the three accuracy values ​​are summed and averaged to get the average accuracy; and the bold numbers in the figure represent the maximum values ​​of the model evaluation results obtained by different quantization methods under the same data set and the same number of quantization bits;

[0130] As can be seen from the figure, the trained GQTL-M3 model is competitive at the same bit level, and the overall results exceed other quantization methods. In the ViT-B / 16 model, compared with LSQ, BRECQ and RepQ-ViT, the GQTL-M3 model achieved an average accuracy improvement of 10.7%, 16% and 6.9% under 4-bit quantization, reflecting the effectiveness and advantages of the method of the present invention in the model quantization process.

[0131] This embodiment shows that the GQTL-M3 model can not only effectively reduce the model calculation cost under low-bit quantization conditions, but also maintain a high model performance through reasonable quantization strategies and training methods, reducing the accuracy loss caused by quantization. Especially when processing different types of data set tasks, the GQTL-M3 model can better adapt to various data features and task requirements, whether it is the classification of common objects in natural images, or professional image analysis in specific fields, and information prediction in structured tasks, it has demonstrated strong generalization and robustness. This is due to the model quantization method combined with efficient parameter transfer learning in the present invention. Through block-by-block quantization, optimization of quantization parameters and reasonable training methods, the model can better balance the relationship between computational cost and model performance during the quantization process, providing strong support for the deployment of the model in practical applications, especially in resource-constrained environments, such as embedded devices, etc., to achieve efficient model reasoning and accurate task execution.

[0132] Therefore, through comprehensive evaluation on the VTAB-1K dataset, the trained GQTL-M3 model stands out in comparison with other quantization methods, proving that the model quantization method proposed in the present invention has significant advantages in improving model quantization performance, reducing computational costs, and enhancing model generalization capabilities, and has important practical application value.

[0133] Part of the data in the above formula is calculated by removing the dimension and taking its numerical value. The formula is a formula closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0134] Working principle of the present invention:

[0135] First, the M1 model is initialized using the weights of the pre-trained model to obtain a good initial feature representation, which provides a basis for subsequent training. The LoRA module is added to approximate the weight changes through the low-rank matrices A and B to reduce the fine-tuning calculation cost. At the same time, the model is flexibly adjusted without changing the original weights, so that it can better adapt to different tasks, and the M2 model after adding the LoRA module is obtained;

[0136] Then, the sub-modules such as the attention layer, linear layer, and convolutional layer are converted into quantization modules, and the floating-point data is converted into low-precision fixed-point data to reduce the model memory usage and speed up the calculation, thus obtaining the M3 model.

[0137] Next, appropriate quantization parameters are determined for transfer learning methods of different quantization paradigms (QATL, PTLQ, GQTL). For QATL, the weights are activated first and then calibrated to balance the impact of quantization during training; PTLQ is calibrated after LoRA training to avoid quantization interfering with transfer learning; GQTL only calibrates weight quantization to simplify the adjustment of training parameters block by block; and the calibrated models are obtained;

[0138] Secondly, the preset training sets were used to perform single-stage training on the calibrated QATL-M3 model, double-stage training on the PTLQ-M3 model, and single-stage training on the calibrated GQTL-M3 model, to obtain the trained QATL-M3 model, PTLQ-M3 model, and GQTL-M3 model, to improve the accuracy and generalization ability of each model after quantification;

[0139] Finally, the trained QATL-M3 model, PTLQ-M3 model and GQTL-M3 model were evaluated using the preset validation set to obtain the evaluation results of the M3 model under different preset transfer learning methods.

[0140] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A model quantization method combined with efficient parameter transfer learning, characterized in that: include: S1, builds the M1 model based on the pre-trained ViT model, and adds a low-rank adaptive matrix to the attention layer of the M1 model to obtain the M2 model after adding the LoRA module; S2, converting several submodules in the M2 model into quantized modules to obtain the quantized M3 model; wherein the several submodules include several attention layers, several linear layers and several convolutional layers; S3, setting the number of weight quantization bits and the number of activation quantization bits, calibrating the model M3 combined with the preset transfer learning method, and obtaining the calibrated QATL-M3 model, PTLQ-M3 model and the calibrated GQTL-M3 model; wherein the preset transfer learning methods include: quantization-aware transfer learning QATL, post-transfer learning quantization PTLQ and progressive quantization joint transfer learning GQTL; S4, using the preset training set to perform single-stage training on the calibrated QATL-M3 model, double-stage training on the PTLQ-M3 model, and single-stage training on the calibrated GQTL-M3 model, to obtain the trained QATL-M3 model, PTLQ-M3 model, and GQTL-M3 model; S5, use the preset validation set to evaluate the trained QATL-M3 model, PTLQ-M3 model and GQTL-M3 model, and obtain the evaluation results of the M3 model under different preset transfer learning methods.

2. The model quantization method combined with efficient parameter transfer learning according to claim 1, characterized in that: The low-rank adaptive matrix is ​​added to the attention layer of the M1 model, including: S11, obtain the initial weight matrix W0 of the attention layer of the M1 model; R represents a real number, d in represents the input dimension of W0, d out represents the output dimension of W0; S12, adding low-rank adaptive matrices A and B to W0 to obtain the M2 model after adding the LoRA module; r represents the rank of the low-rank adaptive matrix.

3. The model quantization method combined with efficient parameter transfer learning according to claim 1, characterized in that: The method of converting several submodules in the M2 model into quantization modules includes: S21, weight quantization and activation quantization are performed on the attention layer in the M2 model, and attention calculation is performed on the weight quantization results and the activation quantization results to obtain an attention layer quantization module; S22, using a quantizer to quantize the linear layer to obtain a linear layer quantization module; S23, using a quantizer to quantize the convolution layer to obtain a convolution layer quantization module.

4. The model quantization method combined with efficient parameter transfer learning according to claim 3, characterized in that: The weight quantization of the attention layer in the M2 model includes: S21-11, add the LoRA module to Q and V in the attention layer, and initialize the low-rank matrix B' in the LoRA module to obtain the initialized low-rank matrix B'; S21-12, merge the initialized low-rank matrix B' into Q and V, and use a weight quantizer to quantize the weights of Q, K, and V respectively to obtain the quantized weights of Q, K, and V; S21-13, concatenate the quantized Q, K, and V along dimension 0 to obtain QKV.

5. The model quantization method combined with efficient parameter transfer learning according to claim 3, characterized in that: The activation quantization process includes: S21-21, obtain input feature x∈(B,N,C); where B represents the batch size, that is, the number of samples processed at one time, N represents the sequence length, that is, the number of image blocks, and C represents the feature dimension; S21-22, reshape the QKV into (3, B, num_heads, N, C / / num_heads), and use the permute operation to separate the reshaped QKV to obtain Q, K, and V; where num_heads represents the number of attention heads; S21-23, using the activated quantizer to quantize Q, K, and V respectively, to obtain activated quantized Q', K', and V'.

6. The model quantization method combined with efficient parameter transfer learning according to claim 3, characterized in that: The attention calculation of the weight quantization result and the activation quantization result includes: S21-31, perform dot product operation on the transpose of Q' and K' and multiply by the scaling factor to obtain the attention score matrix; S21-32, after normalizing the attention score matrix using the softmax function, the drop function is applied to discard it to obtain the processed attention score; S21-33, quantize the processed attention score using a quantizer, and multiply it by V' to obtain a quantized and weighted attention score; S21-34, linearly transform the quantized and weighted attention scores and apply the drop function to obtain the output result of the attention calculation.

7. The model quantization method combined with efficient parameter transfer learning according to claim 1, characterized in that: The calibrating the model M3 combined with the preset transfer learning method includes: For the M3 model combined with QATL, the activation quantization is calibrated first, and then the weight quantization is calibrated; For the M3 model combined with PTLQ, calibration is performed after the LoRA module training is completed; For the M3 model combined with GQTL, only the calibration of weight quantization was performed.

8. The model quantization method combined with efficient parameter transfer learning according to claim 1, characterized in that: The single-stage training of the calibrated QATL-M3 model includes: S41-1, obtaining an ordered list of several modules in the calibrated QATL-M3 model and quantized scaling parameters; S41-2, the initial weight matrix W0 of the M1 model in the QATL-M3 model after freezing calibration; S41-3, use the optimizer and scheduler to dynamically adjust the learning rate, and according to the formula Calculate the loss of the first task, and update the low-rank adaptive matrices A and B as well as the quantization scaling parameters and learning rate through back propagation; where arg min represents the value of the input parameter when the formula reaches the minimum value, s w and a represents the quantization scaling parameter, and s w Represents the quantization scale of the weight, s a represents the activation quantization scale, L T1 represents the first task loss, FQNet represents the calibrated QATL-M3 model, i.e., the fully quantized network, represents the weight matrix after quantization, represents the activated feature after quantization, and y represents the true label; S41-4, repeat S41-1 to S41-3 to perform a training cycle until a first preset number of training times is reached.

9. The model quantization method combined with efficient parameter transfer learning according to claim 1, characterized in that: The PTLQ-M3 model is trained in two stages, including: S42-1, obtaining an ordered list of several modules in the calibrated PTLQ-M3 model, and obtaining parameters of the LoRA module to be optimized from the ordered list to obtain LoRA parameters; S42-2, use the optimizer and scheduler to dynamically adjust the learning rate, and according to the formula Calculate the loss of the second task, update the LoRA parameters and learning rate through back propagation, and complete the first stage of training, where L T2 represents the second task loss, FPNet represents the calibrated QATL-M3 model, i.e., the full-precision network, W represents the full-precision matrix weight without quantization, and x represents the full-precision activation feature without quantization; S42-3, using a quantizer to perform weight quantization calibration and activation quantization calibration on the PTLQ-M3 model that has completed the first stage of training, to obtain a calibrated PTLQ-M3 model; S42-4, train the quantization scale of the calibrated PTLQ-M3 model block by block, through the formula Calculate the reconstruction loss LT3 between the quantized output and the floating-point output, and perform back-propagation to update the quantization parameters to complete the second stage of training; Represents the unquantized floating-point output of the lth module. represents the quantized output of the lth module after quantization processing, W x represents the weight activation multiplication without quantization, represents the multiplication of quantized weight activations, and F represents the Frobenius norm; S42-5, repeat S42-1 to S42-4 to perform a training cycle until a second preset number of training times is reached.

10. The model quantization method combined with efficient parameter transfer learning according to claim 1, characterized in that: The calibrated GQTL-M3 model is trained block by block in a single stage, including: S43-1, obtaining an ordered list of several modules in the calibrated GQTL-M3 model, and sequentially obtaining the lth block of modules from the ordered list, and freezing the first l-1 blocks of modules; S43-2, obtain the learnable parameters of the lth block module, use the optimizer and scheduler to dynamically adjust the learning rate, and use the alternating quantization migration loss function formula Calculate the joint task loss L T4 , and update the learnable parameters and learning rate through back propagation; where FQNet' represents the quantized network of the first l blocks of the calibrated GQTL-M3 model; represents the quantized weight W and feature x of the first l-1 blocks, {W i>l ,x i>l } represents the unquantized weight W and feature x after the lth block module, θ q With θ t represents a learnable parameter, and and Denote the quantization scale of the weight of the lth block module and the quantization scale of the activation, respectively, and B i>l and A i>l Indicates the LoRA parameters after the lth module; S43-3, repeat S43-1 to S41-2, and perform a block-by-block training cycle until a number of modules are traversed and a third preset number of training times is reached.

Images