A mixed precision quantization perception training method based on neural network structure search
By searching neural network architectures and constructing supernet models, mixed-precision configurations are directly searched on the target dataset, solving the problems of low efficiency and high cost of existing mixed-precision quantization methods, and achieving more efficient mixed-precision model training and improved generalization performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TIANJIN UNIV
- Filing Date
- 2023-04-11
- Publication Date
- 2026-05-12
AI Technical Summary
Existing mixed-precision quantization methods suffer from low search efficiency, high computational cost, and fixed-precision quantization ignores the position and structural characteristics of network layers, resulting in suboptimal performance.
A mixed-precision quantization-aware training method based on neural network architecture search is adopted. By constructing a supernet model, the mixed-precision configuration is directly searched on the target dataset by taking advantage of parameter sharing and computational equivalence of convolution operators. The generalization ability is improved by minimizing the target loss value and the sharpness of the quantization loss.
Without increasing computational costs, it improves the generalization performance of mixed-precision models, reduces memory consumption and computational complexity during the search process, and enhances the training efficiency and accuracy of low-bit quantization models.
Smart Images
Figure CN116721305B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, and mainly relates to the model quantization of deep neural networks, specifically to a hybrid precision quantization perception training method based on neural network structure search. Background Technology
[0002] Model quantization is an important research direction for the industrialization of deep learning. Most existing quantization methods use fixed-precision quantization (also known as uniform-precision quantization), where the weights and activations of all layers in the network are typically quantized using the same bit width. Network models quantized with fixed precision are favored because they are well supported on conventional hardware such as CPUs and FPGAs. However, fixed-precision quantization ignores the location, structure, number of parameters, and FLOPs of network layers within the network model. Under the same network parameter count and computational complexity, fixed-precision quantization may lead to suboptimal performance. Therefore, mixed-precision quantization has emerged, aiming to assign different quantization bit widths to the weights and activations of different layers, thus addressing the aforementioned limitations to some extent. Compared to fixed-precision quantization, mixed-precision quantization is more flexible and can further save memory and computational costs without sacrificing network performance. Furthermore, hardware supporting mixed-precision inference (such as A12 and Turing GPUs) has accelerated the practical application of mixed-precision models.
[0003] Existing mixed-precision quantization techniques can be categorized into rule-based and learning-based methods. Rule-based methods utilize specific metrics to determine the optimal quantization bit width for each layer. For example, the HAWQ method uses the Hessian matrix as a metric to determine the layer-level quantization bit width of the network. However, these rule-based metrics typically rely on heuristics provided by domain experts, thus limiting their scalability in practice. Inspired by Neural Architecture Search (NAS) techniques, researchers have proposed learning-based methods to automatically search for the optimal bit width of network layers. These algorithms are built upon Deep Reinforcement Learning (DRL) or basis-differentiable NAS methods. Although existing mixed-precision quantization methods have achieved some success, they still suffer from drawbacks such as low search efficiency and high computational cost. Summary of the Invention
[0004] To overcome the shortcomings of the existing technologies, the present invention aims to propose a mixed-precision quantization-aware training method based on neural network architecture search. This method searches for the optimal mixed-precision configuration of the model under constraints, employs parameter sharing to reduce memory requirements during the search process, utilizes the computational equivalence of convolution operators, and replaces expensive parallel convolution modules with composite convolution modules. This ensures that the computational complexity of the supernet remains unchanged, decoupling the size of the search space from the computational cost of the supernet. This allows for direct searching of the mixed-precision configuration of large networks without relying on proxy tasks. After obtaining the mixed-precision configuration of the target model, the generalization ability of the quantization model is improved by simultaneously minimizing the target loss value and the sharpness of the quantization loss. This alleviates the training difficulties and significant performance degradation caused by low-bit quantization, further enhancing the generalization ability of low-bit quantization models or mixed-precision models containing low-bit quantization layers.
[0005] To achieve the above objectives, the technical means employed in this invention are as follows:
[0006] A mixed-precision quantization-aware training method based on neural network architecture search, the specific steps of which are as follows:
[0007] (1) Process the original image dataset and divide it into a training dataset Dtrain and a validation dataset Dval;
[0008] (2) Obtain the gradient of the supernet with respect to the training set: Sample a batch of data samples from the training dataset Dtrain in step (1), input them into the supernet for forward inference, and obtain the current target loss value; obtain the gradient of the weights, weight bit importance parameters and activation bit importance parameters in the supernet with respect to the target loss value according to backpropagation.
[0009] (3) Update the supernet weights: Update the weights of the current supernet model using gradient descent;
[0010] (4) Obtain the gradient of the supernet with respect to the validation set: Sample a batch of data samples from the validation dataset Dval in step (1), input them into the supernet for forward inference, and obtain the current target loss value; obtain the gradient of the bit importance parameter in the supernet with respect to the target loss value according to backpropagation;
[0011] (5) Update the bit importance parameters of the supernet: Update the weight bit importance parameters and activation bit importance parameters of the supernet using the gradient descent method;
[0012] (6) Save the current mixed precision configuration: Take the quantization bit corresponding to the item with the highest probability from the weight bit importance parameters and activation bit importance parameters of the supernet as the mixed precision configuration of the current target network, and save the mixed precision configuration in the form of a file.
[0013] (7) Repeat steps (2) to (5) until the set number of iterations is completed or the complexity of the current mixed precision configuration is lower than the expected complexity;
[0014] (8) By reading the contents of the mixed precision configuration file saved in step (6), the mixed precision configuration set of the target network under different constraints is obtained;
[0015] (9) Starting from the flatness of the minimum region of the model loss function, quantitative perception training is carried out on the mixed precision network under different constraints.
[0016] The expected complexity mentioned in step (7) is the complexity set by the user based on the requirements of the model computation cost in the actual application scenario.
[0017] The different constraints mentioned in steps (8) and (9) include different model sizes and different computational costs, i.e., computational complexity.
[0018] The specific method described in step (9) is as follows:
[0019] (9-1) The mixed precision configuration obtained in step (6) is read from the configuration file by the program and mapped onto the bit distribution of the target neural network model. The maximum perturbation coefficient and training configuration are set.
[0020] (9-2) Perform end-to-end quantization-aware training on the learnable parameters of the quantization model, namely the weights and the quantization step size. For the weights, an optimization method based on loss sharpness is used for updating, and for the quantization step size, a standard gradient descent method is used for updating.
[0021] In step (9), the optimization method based on loss sharpness is used, and the magnitude of the perturbation needs to decrease as the number of training iterations increases.
[0022] Compared with the prior art, the advantages of the present invention are:
[0023] This invention leverages parameter sharing and the computational equivalence of convolution operators to search for the optimal mixed-precision configuration of the model under constraints at a low computational cost. During the mixed-precision search, to alleviate overfitting of bit importance parameters in single-layer and two-layer optimization, this invention utilizes regularization, treating the loss term of the supernet model on the validation set as a regularization term. By adding this additional constraint term, the training loss and validation set loss regarding bit importance parameters are minimized to obtain a better mixed-precision configuration. Furthermore, considering that small changes in weights due to quantization noise or gradient updates can cause significant fluctuations in the loss value within the local optimum range, making low-precision models more likely to fall into poor local minima during optimization and leading to a significant decrease in model accuracy, this invention improves the generalization performance of low-precision models by simultaneously minimizing the target loss value and the sharpness of the quantization loss, providing a general quantization-aware training mode for low-bit quantization networks.
[0024] 1. This invention eliminates the need for proxy tasks to reduce the extremely high computational requirements of directly searching for the optimal mixed-precision configuration of the target network on a large dataset. By constructing the supernet in step (2), and utilizing the computational equivalence of parameter sharing and convolution operators, the mixed-precision search is performed directly on the target dataset during the training of the supernet in steps (2) to (5). The mixed-precision configurations that satisfy different constraints (model size and computational cost of the model) are saved in step (6), thereby reducing the memory consumption and computational complexity during the search process.
[0025] 2. This invention also considers the training difficulties and significant performance degradation caused by low-bit quantization. Compared with existing methods that optimize the model from the perspective of training, this invention proposes a quantization-aware training strategy QSAMDecay based on minimizing the target loss sharpness, starting from the flatness of the minimum region of the model loss function. The specific content is as described in step (9). While minimizing the task loss, it also uses an optimization method based on sharpness loss to make the target loss function plane of the quantization model flatter, thereby improving the generalization performance of the low-bit quantization model. Attached Figure Description
[0026] Figure 1 This invention compares the computational graph design in the mixed-precision search process with that of traditional computational graphs. Figure 1 (a) Traditional mixed-precision search algorithms require maintaining multiple copies of full-precision weight parameters; Figure 1 (b) is the method proposed in this invention. Detailed Implementation
[0027] The present invention will now be described in detail with reference to the accompanying drawings.
[0028] according to Figure 1 The present invention compares the computational graph design in the mixed-precision search process with that of traditional computational graphs, wherein... Figure 1 (a) shows that the traditional mixed-precision search algorithm needs to maintain multiple copies of the full-precision weight parameters; Figure 1 (b) illustrates the method proposed in this invention. This method requires only storing one weight tensor, which is then aggregated after passing through multiple quantization branches, ultimately requiring only one convolution operation. It is important to emphasize that each branch possesses an independent learnable quantization stride, ensuring that each branch has dynamic and flexible quantization capabilities during supernet training, further improving the convergence speed and search efficiency of supernet training.
[0029] Example 1
[0030] Mixed-precision search was performed on the ResNet-20 network on the CIFAR-10 dataset. The performance of the EMPS method was compared with classic fixed-precision quantization methods such as Dorefa, PACT, LQ-Ne, and LSQ. Furthermore, to demonstrate the effectiveness of the hybrid two-layer optimization strategy designed in this invention in the supernet search process, the performance of the EMPS method was compared with the existing mixed-precision method HAWQ. [5] BP-NAS [6] and SSPS [7] Comparison. In a mixed-precision search of the target network ResNet-20:
[0031] Building the supernet: First, a supernet with the same macrostructure as the ResNet20 network was constructed. Then, the supernet model was initialized using the pre-trained full-precision model parameters for search.
[0032] Dataset Splitting: To train the supernet, 50% of the training samples in the CIFAR-10 dataset will be split.
[0033] The remaining 50% of the training samples are used as the validation set, and the batch size B is set to 128.
[0034] Set the search bit width set: In each search cell in the supernet, the candidate bit width set for weights and activation values is set to {2,3,4,5};
[0035] Update the supernet parameters: For the supernet weights W, use the SGD optimizer with an initial learning rate of 0.2, momentum of 0.9, and weight decay factor of 5e-4. For the bit importance parameters of the supernet weights and activation values, use the Adam optimizer with an initial learning rate of 5e-3 and weight decay factor of 1e-3. Throughout the training process of the supernet, cosine decay is used to control the change in the learning rate.
[0036] Obtaining Mixed Precision Configurations (MPCs): After training the supernet using the training pipeline, a set of mixed precision configurations (MPCs) for the target network under different computational constraints can be obtained.
[0037] Retraining of mixed-precision models: Quantization-aware training is performed on quantization models with mixed-precision configurations of different complexities, and the models are retrained for 200 epochs.
[0038]
[0039] Table 1 shows the quantization results of the ResNet-20 network on the CIFAR-10 dataset. Where "MP" represents mixed precision quantization, "WComp" represents the compression ratio of the model weights, and "B-Comp" represents the model inference performance.
[0040] The compression ratio of the number of bit operations (BOPs) in the process; "Ave-Bits" represents the average number of bit operations.
[0041] The results in Table 1 show that the proposed EMPS hybrid precision quantization method outperforms all comparable methods compared to fixed-precision quantization methods. At the same model weight compression ratio (W-Comp), our EMPS method achieves a Top-1 accuracy of 93.29%, representing improvements of 2.19%, 1.69%, and 0.99% compared to PACT, LQ-Net, and LSQ methods, respectively. At nearly the same BOPs bit operation compression ratio (B-Comp), our EMPS method achieves a Top-1 accuracy of 93.00%, representing improvements of 1.90%, 1.40%, and 0.70% compared to PACT, LQ-Net, and LSQ methods, respectively.
[0042] Furthermore, compared to existing BP-NAS and SSPS mixed-precision methods, the mixed-precision quantization model obtained by the method proposed in this invention shows significant improvements in accuracy across the model parameter compression ratio (W-Comp), bit operation compression ratio (B-Comp), and average bit operations (Ave-Bits). It can be seen that the mixed-precision configuration of the target model searched by the EMPS method can achieve better performance under different computational cost constraints.
[0043] Example 2
[0044] Mixed-precision search was performed on the MobileNetV2 network on the CIFAR-100 dataset. The process of performing mixed-precision search on the CIFAR-100 dataset for the MobileNetV2 model is similar to that of the ResNet20 model on the CIFAR-10 dataset. 50% of the training samples in the CIFAR-100 dataset were used as the training set, and the remaining 50% of the training samples were used as the validation set. The batch size was set to 128. The difference is that after obtaining different mixed-precision configurations, the quantized model was retrained for 300 epochs using all training samples, and no weight decay was applied to the learnable parameters of the batch normalization layer.
[0045]
[0046] Table 2 shows the quantization results of the MobileNetV2 network on the CIFAR-100 dataset. Here, "MP" represents mixed precision quantization, "W-Comp" indicates the compression ratio of the model weights, "B-Comp" indicates the compression ratio of the number of bit operations (BOPs) during model inference, "Ave-Bits" represents the average number of bit operations, and "-" indicates that the result was not reported.
[0047] Table 2 compares the accuracy of different quantization methods for the MobileNetV2 model under 4-bit and 3-bit constraints. Compared to fixed-precision quantization, EMPS shows a significant performance advantage under the same model compression ratio and the same bit operation compression ratio. For example, with approximately 4-bit quantization for weights and activations, EMPS not only achieves a 0.6% higher Top-1 accuracy than the current best-performing fixed-precision quantization method LSQ, but also a 1.78× higher model compression ratio. Furthermore, we compare the quantization results with existing mixed-precision methods such as HAQ and SAMQ; under similar average bit operation (Ave-Bits) metrics, EMPS demonstrates a significant accuracy advantage.
[0048] Example 3
[0049] To verify the effectiveness of the proposed mixed-precision algorithm on large datasets and network models of different depths, we conducted experiments on the ImageNet ILSVRC-2012 dataset to validate ResNet-18, ResNet-50, and MobileNetV2 network models. During hypernet training, 75% of the training samples were selected as the training set Dtrain, and the remaining 25% as the validation set Dval. Detailed information on the search training hyperparameters and retraining hyperparameters for different network depths is shown in Table 3. In the experiments, training images were randomly cropped and their resolution adjusted to 224×224. Data augmentation methods included random horizontal flipping and color augmentation (brightness = 0.2, contrast = 0.2, saturation = 0.2, hue = 0.1). Validation images were center-cropped and their resolution adjusted to 224×224. Furthermore, a label smoothing strategy (label_mooth = 0.1) was used to add regularization during training.
[0050]
[0051] Table 3 Training hyperparameter settings. * indicates that the weight decay coefficient of the BN layer parameter is 0.
[0052] The proposed method is compared with currently popular quantization methods (such as PACT, LSQ, BP-NAS, SAMQ, and SSPS). Similar to the experimental procedure described above, the experimental results were obtained by retraining the mixed-precision model searched by the EMPS method using the QSAMDecay training method. Table 4 shows the quantization results of different deep networks on the ImageNet dataset. First, compared with fixed-precision quantization methods, under the same model compression ratio and higher BOPs bit operation compression ratio, the proposed method achieves better Top-1 accuracy. For example, when the target network is ResNet-18, ResNet-50, and MobileNetV2, our method outperforms the current best-performing fixed-precision quantization method LSQ by 0.19%, 0.34%, and 0.32% in Top-1 accuracy, respectively.
[0053] Secondly, compared to existing differentiable mixed-precision search methods such as DNAS, BP-NAS, and SSPS, our method only stores one copy of the weight tensor shared by each branch during the search process and utilizes the computational equivalence of convolution operators to perform equivalent transformations in the forward computation, avoiding large memory requirements and training computation costs. Finally, compared to reinforcement learning-based mixed-precision search methods such as HAQ, AutoQ, and SAMQ, our method does not require long training cycles or large amounts of computational resources (e.g., thousands of GPUs) in terms of training cost, making it more suitable for directly searching for mixed-precision configurations of deep networks on large datasets. Regarding quantization model accuracy, our algorithm outperforms HAQ by 1.8% on ResNet-50 and by 1.06% and 0.26% on MobileNetV2, respectively.
[0054]
[0055]
[0056]
[0057] Table 4 compares the performance of quantized network models on the ImageNet dataset. "-" indicates that the results were not reported.
[0058] Example 4
[0059] To verify the effectiveness of the proposed method on object detection tasks, we used the ResNet-50 mixed-precision network selected in Experiment 3 as the backbone on the COCO detection dataset to train mixed-precision models for both the two-stage Faster-RCNN detector and the single-stage RetinaNet detector. For the Faster-RCNN detector, in addition to the backbone, the weights and activations in the RPN region proposal module and the ROIHead module were also quantized to 4 bits. During quantization-aware training, we fine-tuned the Faster-RCNN model for 12 epochs on four GPUs, with a batch size of 8. Furthermore, we chose SGD as the base optimizer for QSAMDeay and then used the QSAMDeay optimization method for quantization-aware training of the detection network. The initial learning rate was set to 0.02, the momentum to 0.9, the weight decay coefficient to 10⁻⁴, and the maximum perturbation coefficient ρmax to 0.05. During training, we used the MultiStepLR (milestones = [5, 8, 10]) learning rate decay strategy. For the RetinaNet network framework, except for the last layer in the detection head module which uses 8-bit quantization, the feature pyramid and other parts in the detection head module all use 4-bit quantization.
[0060]
[0061] Table 5 compares the performance of Faster-RCNN and RetinaNet detection models on the COCO dataset.
[0062] Table 5 compares the proposed hybrid precision quantization perceptual training method EMPS with FQN, Auxi, BP-NAS, and SSPS algorithms. FQN and Auxi are fixed precision quantization methods, while BP-NAS and SSPS are hybrid precision quantization algorithms. Experimental results show that after obtaining the hybrid precision configuration of the ResNet-50 network using the EMPS method on the ImageNet classification task, directly applying this configuration to the backbone of the Faster-RCNN detection model and then fine-tuning the training results in detector performance that is 4.5%, 1.8%, and 0.2% higher than FQN, BP-NAS, and SSPS, respectively. On the RetinaNet detection model, the proposed model quantization method also achieved the best detection results compared to other quantization methods.
[0063] The experimental results above demonstrate that the method proposed in this invention has good transferability in target detection tasks.
[0064] In summary, this invention discloses an efficient mixed-precision search method (EMPS). Benefiting from an efficient search computation graph design, this method allows for direct mixed-precision search of the target network on the target dataset without any proxy tasks. Furthermore, a two-layer hybrid optimization strategy is used to update the bit importance parameters in the supernet, mitigating overfitting during training by embedding more training information. Considering the training difficulties and significant performance degradation of network models during low-bit quantization, this invention proposes a quantization-aware training method (QSAMDecay) that minimizes the sharpness of the target loss. This method analyzes the model parameter perturbation and improves the generalization ability of the quantization model by simultaneously minimizing the sharpness of both the target loss and quantization loss. Through image classification tasks on CIFAR-10, CIFAR-100, and ImageNet datasets, and object detection tasks on the COCO dataset, it can be seen that the proposed mixed-precision search method achieves better performance than existing comparative methods under the same computational constraints. This further accelerates the practical application of mixed-precision models.
Claims
1. A hybrid precision quantization-based perceptual training method based on neural network architecture search, characterized in that: The specific steps are as follows: (1) Process the original image dataset and divide it into a training dataset Dtrain and a validation dataset Dval; (2) Construct a supernet. Each quantizable layer in the supernet uses a composite convolution module. Each composite convolution module contains multiple quantization branches. Each branch has an independent learnable quantization stride. All branches share the same full-precision weight tensor. During forward propagation, the computational equivalence of the convolution operator is used to aggregate the outputs of multiple quantization branches and perform a convolution operation only once. A batch of data samples is sampled from the training dataset Dtrain, input into the supernet for forward inference, and the current target loss value is obtained; The gradients of the weights, weight bit importance parameters, and activation bit importance parameters in the supernet with respect to the target loss value are obtained through backpropagation. (3) Update the supernet weights: Update the weights of the current supernet model using gradient descent; (4) Obtain the gradient of the supernet with respect to the validation set: Sample a batch of data samples from the validation dataset Dval, input them into the supernet for forward inference, and obtain the current target loss value; obtain the gradient of the bit importance parameter in the supernet with respect to the target loss value according to backpropagation; (5) Update the bit importance parameters of the supernet: Update the weight bit importance parameters and activation bit importance parameters of the supernet using the gradient descent method; (6) Save the current mixed precision configuration: Take the quantization bit corresponding to the item with the highest probability from the weight bit importance parameters and activation bit importance parameters of the supernet as the mixed precision configuration of the current target network, and save the mixed precision configuration in the form of a file. (7) Repeat steps (2) to (5) until the set number of iterations is completed or the complexity of the current mixed precision configuration is lower than the expected complexity; (8) By reading the contents of the mixed precision configuration file saved in step (6), the mixed precision configuration set of the target network under different constraints is obtained; (9) Starting from the flatness of the minimum region of the model loss function, quantitative perception training is carried out on the mixed precision network under different constraints.
2. The hybrid precision quantization perception training method based on neural network structure search according to claim 1, characterized in that: The expected complexity mentioned in step (7) is the complexity set by the user based on the requirements of the model computation cost in the actual application scenario.
3. The hybrid precision quantization perception training method based on neural network structure search according to claim 1, characterized in that: The different constraints mentioned in steps (8) and (9) include different model sizes and different computational costs, i.e., computational complexity.
4. The hybrid precision quantization perception training method based on neural network structure search according to claim 1, characterized in that: The specific method for step (9) is as follows: (9-1) The mixed precision configuration obtained in step (6) is read from the configuration file by the program and mapped onto the bit distribution of the target neural network model. The maximum perturbation coefficient and training configuration are set. (9-2) Perform end-to-end quantization-aware training on the learnable parameters of the quantization model, namely the weights and the quantization step size. For the weights, an optimization method based on loss sharpness is used for updating, and for the quantization step size, a standard gradient descent method is used for updating.
5. The hybrid precision quantization perception training method based on neural network structure search according to claim 4, characterized in that: In step (9-2), the optimization method based on loss sharpness is used, and the magnitude of the perturbation needs to decrease as the number of training iterations increases.