Model quantification method based on long-distance dependency relationship and zero sample generation

Abstract

The invention discloses a model quantification method based on a long-distance dependency relationship and zero sample generation. The method comprises the following steps: step 1, constructing a synthetic data generation network; 2, inputting the standard Gaussian noise into the synthetic data generation network for training until the loss of the synthetic data generation network is converged, and obtaining synthetic data; 3, performing post-training quantization on the target image classification model by taking the synthetic data as calibration data to obtain a quantization model; thirdly, constructing a distillation network by combining the quantitative model with a full-precision model; and then, inputting the synthetic data into the distillation network for training until the loss of the distillation network is converged so as to finely adjust the quantitative model therein, and obtaining the fine-adjusted quantitative model. According to the method, generalization of the quantification model to various data is improved, common image classification models can be compressed and quantified, and excellent quantification performance is shown on widely-used image data sets.

Description

Technical Field

[0001] The present invention relates to the field of model quantization methods, and in particular to a model quantization method based on long-distance dependency and zero-sample generation. Background Art

[0002] The purpose of the model quantization task is to map the weights and activation values ​​of the deep learning model from 32-bit floating point numbers to low-bit integers to reduce the model size, and then deploy the model on edge devices with limited computing resources. Post-training quantization (PTQ) can perform fast inference with a small amount of data and does not require a lot of calculations. However, in reality, access to some original data may be severely restricted, so zero-sample quantization (ZSQ) was proposed. It can quantize deep models without any original data. The synthetic data generated by the pre-trained full-precision model has a similar distribution to the original data, allowing the quantized model to retain higher accuracy. Summary of the invention

[0003] The present invention provides a model quantization method based on long-distance dependency and zero-sample generation to enhance the ability of existing zero-shot quantization methods to perceive the target as a whole, avoid the homogenization of the same type of data in synthetic calibration data, and affect the generalization of the quantization model to real-world data.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] A model quantization method based on long-distance dependency and zero-shot generation, comprising the following steps:

[0006] Step 1: construct a synthetic data generation network, wherein the synthetic data generation network includes a long-distance information fusion generator and an edge information enhancement confrontation module; wherein the long-distance information fusion generator establishes a long-distance dependency relationship for pixels at different distances in an image to obtain global features of the image, thereby generating synthetic data; the edge information enhancement confrontation module adds a new angle deviation to the cosine similarity calculation between the intra-class feature vector and the cluster center to expand the distance between the vector and the center and reduce the similarity between the two;

[0007] Step 2: configure the hyperparameters of the synthetic data generation network training, and then input standard Gaussian noise into the synthetic data generation network for training until the loss of the synthetic data generation network converges, thereby obtaining synthetic data that matches the distribution of real data;

[0008] Step 3: Use the synthetic data as calibration data to perform post-training quantization on the target image classification model to obtain a quantized model;

[0009] Next, the obtained quantized model is combined with the full-precision model to construct a distillation network. The distillation network uses a decoupled knowledge distillation network to take KL divergence as the loss function for the target class and the background class for joint fine-tuning training, and introduces hyperparameters to alleviate the sample imbalance problem, thereby transferring knowledge from the full-precision model to the quantized model.

[0010] Then, the synthetic data is input into the distillation network for training until the loss of the distillation network converges, so as to fine-tune the quantization model therein and obtain a fine-tuned quantization model.

[0011] Furthermore, during training in step 2, the standard Gaussian noise is input into the long-distance information fusion generator in the synthetic data generation network to obtain synthetic data, and then the loss of the synthetic data to the batch normalization statistics and the edge information enhancement adversarial module is calculated as the loss of the synthetic data generation network.

[0012] Furthermore, in step 3, a zero-shot quantization method is used to compress the target image classification model without using the original data, and then the target image classification model is post-trained and quantized using the synthetic data as calibration data to obtain a quantized model.

[0013] Furthermore, in step 3, the full-precision model is a pre-trained full-precision model, and the full-precision model uses the BNS loss L BNS to constrain the activation distribution.

[0014] Furthermore, in step 3, when the distillation network uses a decoupled knowledge distillation network to take KL divergence as the loss function for joint fine-tuning training for the target class and the background class respectively, the knowledge distillation is decomposed into target class knowledge distillation and non-target class knowledge distillation, and TCKD is set to represent the similarity between the binary classification probabilities of the teacher and the student of the target class, and NCKD is set to represent the similarity between the probabilities of the teacher and the student of the non-target class, and the NCKD loss is converted into a Focal Loss-like form and combined with the TCKD loss as the loss function during the joint fine-tuning training of the distillation network.

[0015] Furthermore, in step 3, when the synthetic data is input into the distillation network for training, the quantization model is fine-tuned using the cross entropy loss, and the output of the full-precision model and the quantization model output are trained using the loss of the de-distillation network to align the accuracy, thereby constructing the total loss for quantization model training.

[0016] The method of the present invention proposes a model quantization method based on long-distance dependency and zero-sample generation. In the method, a synthetic data generation network and a distillation network based on attention mechanism and reinforcement learning are designed and constructed for zero-shot quantization of the target model. The synthetic data generation network consists of a long-distance information fusion generator and a data edge information enhanced adversarial module. In order to enhance the model's ability to perceive the target as a whole and obtain global features and local context information, the long-distance information fusion generator uses a long-distance information fusion method to generate richer synthetic calibration data. Compared with other zero-shot quantization methods, the target model is more sensitive to the global features of the image after quantization using this batch of synthetic calibration data.

[0017] In order to avoid the homogenization of the same type of data in the synthetic calibration data and affect the generalization of the quantization model to real-world data, the present invention adopts an edge information enhancement adversarial module to amplify the intra-class angle between the feature vector and the cluster center, thereby expanding the intra-class feature vector distance, effectively retaining and enhancing the intra-class heterogeneity of the synthetic data, and improving the generalization of the quantization model to various types of data.

[0018] The present invention also adopts a decoupled knowledge distillation network to transfer local information to the student network to fine-tune the model so as to align the accuracy of the quantization model and the floating-point model.

[0019] The present invention can compress and quantize common image classification models and show excellent quantization performance on widely used image datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The figure is a schematic diagram of a model quantization method according to an embodiment of the present invention.

[0021] Figure 2 This is a comparison chart of the visualization effects of quantizing the ResNet18 floating-point model to 4 bits wide using the method of the embodiment of the present invention and different ZSQ methods. DETAILED DESCRIPTION

[0022] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0023] like Figure 1 As shown, this embodiment discloses a model quantization method based on long-distance dependency and zero-sample generation, comprising the following steps:

[0024] Step 1: Construct a synthetic data generation network, wherein the synthetic data generation network includes a long-distance information fusion generator and an edge information enhancement adversarial module.

[0025] Among them, the long-distance information fusion generator establishes long-distance dependencies for pixels at different distances in the image to obtain the global features of the image, thereby generating synthetic data. The generated synthetic data can enhance the ability of the pre-trained model to perceive the target as a whole, effectively alleviate problems such as target occlusion and blur, and improve the overall performance of the image classification quantization model.

[0026] The edge information enhancement adversarial module adds a new angle deviation in the calculation of the cosine similarity between the intra-class feature vector and the cluster center to increase the distance between the vector and the center and reduce the similarity between the two, increasing the difficulty of model convergence. This edge information enhancement adversarial module effectively retains and enhances the intra-class heterogeneity of the generated synthetic data, enhancing the generalization of the quantitative model to real-world data.

[0027] In this embodiment, the synthetic data generation network is integrated with the attention mechanism into the long-range dependency, that is, the long-distance information fusion generator is integrated, so that the synthetic data generation network can synthesize diversified data containing global features, which helps to effectively utilize the comprehensive knowledge of the pre-trained model in the data generation process. In addition, the long-distance information fusion generator enhances the final classification results in the network fine-tuning stage.

[0028] The long-distance information fusion generator consists of three parts: dilated convolution, spatial long-distance convolution, and point-by-point convolution. Under this design, long-distance dependencies can be well processed with less computation. After obtaining the long-distance relationship, this embodiment can estimate the importance of a point and generate an attention map. Long-distance attention can be defined as shown in the following formula:

[0029] LRA=C Conv (SLR Conv (SL Conv (V)))

[0030] Where V represents the input feature vector, SL Conv Represents dilated convolution, SLR Conv represents the spatial long-distance convolution with dilation d, C Conv represents point-wise convolution, and LRA represents long-range attention map. Then, the output feature can be expressed as shown below:

[0031]

[0032] in, is the output feature, Represents element-wise multiplication.

[0033] Assuming that the input and output features are H×W×C, where H is the height of the input feature map, W is the width of the input feature map, C is the number of input feature channels, and P(K,d) is the input parameter, the number of P(K,d) is calculated as shown below:

[0034]

[0035] Where d represents the dilation rate and K represents the kernel size. Compared with the self-attention method, the long-range attention mechanism of this embodiment covers both global dependencies and local context information. This novel feature is crucial for generating diverse samples with complex textures that are richer and closer to real data. In addition, as a fixed-shape kernel, LRA can more easily train the generator with less synthetic data.

[0036] In this embodiment, the edge information enhancement adversarial module is integrated into the synthetic data generation network so that the generated synthetic data has higher diversity, and the pre-trained full-precision model can obtain more feature information during post-training quantization and fine-tuning. In contrast, if the model is quantized or fine-tuned using mostly homogeneous synthetic data within each class, the quantized model has low accuracy for real-world data and weak generalization ability. In order to more accurately classify all categories and enable the synthetic data generation network to enhance intra-class discrimination while forming category-related features, the edge information enhancement adversarial module of this embodiment adds a new angle deviation in the calculation of the cosine similarity between the feature vector and the class center, and the edge information enhancement adversarial module increases the intra-class distance by expanding the angle.

[0037] The edge information enhanced adversarial module increases the difficulty of convergence of the loss function, further increasing the difficulty of model training. However, the goal of the loss function is to reduce the difficulty of convergence, that is, the training of the adversarial module is opposite to the original loss function, forming an adversarial process. It should be noted that this adversarial process makes training more difficult, but it can benefit performance.

[0038] Specifically, in this embodiment, represents the feature vector produced by the pre-trained full-precision model express Assume that The label is c, M c is a collection of synthetic data belonging to class c. In this embodiment, the class center can be defined as M c The average feature vector of all synthetic images in the formula can be expressed as follows:

[0039]

[0040] Since the feature vectors are distributed around the center of each category in the present embodiment, and An additional angle deviation m is added between to enhance the intra-class data heterogeneity, as shown in the following formula:

[0041] L AMA (θ) = max(λ l -cos(θ+m),0)+max(cos(θ+m)-λ u ,0)

[0042] Where θ represents the deflection angle, represents the return cosine distance, m represents the margin added in this embodiment, and λ l and λ u They are The lower and upper bounds of the cosine distance to its class center, L AMA (θ) represents the loss function of the edge information enhanced adversarial module. Specifically, the angle deviation m is forced into the calculation of the cosine similarity through the above formula.

[0043] Step 2: Configure the hyperparameters for training the synthetic data generation network, and then input standard Gaussian noise into the synthetic data generation network for training until the loss of the synthetic data generation network converges, and obtain synthetic data that matches the distribution of real data. The details are as follows:

[0044] The standard Gaussian distribution I is used as input data. The data generation process converts I into synthetic data through a long-distance information fusion generator. To match the distribution of real data well. Specifically, the synthetic data generation network contains three types of convolution blocks: hole convolution, spatial long-distance convolution, and point-by-point convolution. A long-distance attention mechanism is included after the second convolution block. In this stage, the BN layer, the long-distance information fuser, and a feedforward neural network are stacked in sequence to extract global features and local context features, and all layers maintain the same output size.

[0045] In this embodiment, a long-distance information fusion device is first used to derive Next, calculate The loss of the batch normalization statistic BNS and the edge information enhanced adversarial module AMA is used as the loss of the synthetic data generation network as shown in the following formula:

[0046] L G =L BNS +L AMA

[0047] Among them, L G Generate the total loss function for network training for synthetic data, L BNS The activation distribution loss of the BN layer of the network and the pre-trained full-precision model for synthetic data generation, L AMA The loss function of the edge information enhanced adversarial module AMA.

[0048] Step 3: First, the synthetic data is used as calibration data to perform post-training quantization on the target image classification model to obtain a quantized model.

[0049] In this embodiment, in order to better deploy the target image classification model on the edge device, an efficient zero-shot quantization method is used to compress the target image classification model to 1 / 4 of the original full-precision model without using the original data and retain almost lossless classification accuracy. The specific principle is to quantize the weights and activation values ​​of the 32-bit floating-point value x in the full-precision model to 8-bit integer x. q , the process can be expressed as follows:

[0050]

[0051] Where: S is the uniformly mapped scaling factor. Z is the integer offset. x is a 32-bit floating point value, and N is the quantized fixed point value x. q The bit width is usually set to 8. α and β represent the upper and lower bounds of the clipping range, respectively, and round(·) is a rounding operation that maps the value to the nearest N-bit integer. It is a 32-bit integer after inverse quantization. Due to the poor representation capability of low-bit values, the quantization process introduces noise and requires fine-tuning to restore the degraded performance.

[0052] Next, a distillation network is constructed by combining the obtained quantization model with the full-precision model. The distillation network uses a decoupled knowledge distillation network to take KL divergence as the loss function for the target class and the background class respectively for joint fine-tuning training, and introduces hyperparameters to alleviate the sample imbalance problem, thereby transferring knowledge from the full-precision model to the quantization model.

[0053] The distillation network constructed in this embodiment adopts a decoupled knowledge distillation network to take KL divergence as the loss function for the target class and the background class respectively, and introduces hyperparameters to alleviate the sample imbalance problem, and transfers knowledge from the full-precision model to the quantized model, effectively aligning the classification accuracy of the quantized model and the pre-trained model.

[0054] The decoupled knowledge distillation network uses KL divergence as the loss function for joint fine-tuning training for the target class and the background class respectively. In order to better handle the adversarial process in the edge information enhancement adversarial module, the quantization model needs to be further fine-tuned. Therefore, this embodiment introduces decoupled knowledge distillation. The traditional knowledge distillation method only uses KL divergence as the loss function, while the decoupled knowledge distillation uses the binary probability between the non-target classes to reconstruct the classic knowledge distillation. Decoupled knowledge distillation first decomposes the knowledge distillation into target class knowledge distillation and non-target class knowledge distillation. Let TCKD represent the similarity between the binary classification probabilities of the teacher and the student of the target class, and NCKD represent the similarity between the probabilities of the teacher and the student of the non-target class (background class). The loss function of the knowledge distillation network is as follows:

[0055] L KD =L TCKD +(1-p t )L NCKD

[0056] Where: p t is the probability of the target class. KD represents the loss function of the knowledge distillation network, L TCKD represents the similarity between the binary classification probabilities of the teacher and the student of the target class, L NCKD Represents the similarity between the probabilities of the teacher and the student for the non-target class (background class).

[0057] Initially, the distillation network used hyperparameters to set the coefficients of TCKD and NCKD losses. However, since both losses are based on KL divergence, it is difficult for the model to fully solve the problem of class imbalance. In order to enhance the learning ability of difficult samples, this embodiment converts the NCKD loss into a Focal Loss-like form instead of a hyperparameter. The loss function of the improved distillation network in this embodiment is defined as follows:

[0058] Ldkd=L TCKD +γ(1-pt) 2 L NCKD

[0059] Where: γ is the weight coefficient of NCKD, which is set to 8.

[0060] Different from the decoupled distillation network, two hyperparameter weight coefficients are set to balance L TCKD and L NCKD The importance of this embodiment is to use γ(1-p t ) 2To solve the problem of inter-class imbalance, instead of using hyperparameters. Through more efficient training methods, the improved decoupled distillation network can better transfer the knowledge learned by the edge information enhanced adversarial module to the quantized model. This further enables the quantized model to learn knowledge about intra-class heterogeneity.

[0061] In order to imitate the real data distribution, this embodiment makes full use of the pre-trained full-precision model to generate synthetic data. The pre-trained full-precision model uses the BNS loss L BNS To constrain the activation distribution, as shown below:

[0062]

[0063] in, and are the mean and variance stored for the kth BN layer of the pre-trained full-precision model. and They represent the running mean and variance of the k-th layer BN respectively. BNS Represents the activation distribution loss of the BN layer of the synthetic data generation network and the pre-trained full-precision model.

[0064] Then, the synthetic data is input into the distillation network for training until the loss of the distillation network converges, so as to fine-tune the quantization model therein and obtain a fine-tuned quantization model.

[0065] Specifically, when this embodiment inputs synthetic data into the distillation network for training, the quantization model is fine-tuned using the cross entropy loss, and the output of the full-precision model and the quantization model output are trained using the loss of the de-distillation network to align the accuracy, thereby constructing the total loss for quantization model training.

[0066] Assume that the input generated synthetic data The cross entropy loss CE(·) is used to fine-tune the quantization model. The formula can be expressed as follows:

[0067]

[0068] Where: y is the true label of the real data. CE is the cross entropy loss, CE(·) is the cross entropy function, is the output of the quantization model.

[0069] This embodiment uses the improved decoupled distillation network loss to train the output of the pre-trained full-precision model and the output of the quantized model to align the accuracy. The formula can be expressed as shown below:

[0070]

[0071] Among them, LDKD represents the distillation loss between the full-precision model and the quantized model, To quantify the model output, Output for full-precision models.

[0072] L CE , L DKD The joint training forms the total loss of the quantized model training. The total training loss function can be expressed as follows:

[0073] L F =L CE +λL DKD

[0074] Where: λ is a hyperparameter used to balance the weight of the two types of losses. F Represents the total loss of quantized model training

[0075] After obtaining the fine-tuned quantization model in this embodiment, various data sets are input into the fine-tuned quantization model to obtain corresponding image classification results. The method of this embodiment is further described below in conjunction with experiments.

[0076] The experiments were verified on ImageNet and CIFAR-10 / 100, and implemented using Pytorch. For ImageNet, ResNet18, MobileNetV1, and MobileNetV2 were quantized in the experiment; for CIFAR-10 / 100, ResNet20 was quantized in this experiment. Adam was applied in the data generation process, and the momentum and initial learning rate were set to 0.9 and 0.5. The learning rate was decayed by 0.1 every 1000 updates to the synthetic data, and the batch size was set to 256. The images generated by the synthetic data generation network were generated using stochastic gradient descent (SGD), 150 fine-tuning epochs were provided, and the weight decay was set to 10-4. For CIFAR-10 / 100 and ImageNet, the batch sizes used for fine-tuning were 256 and 16, respectively. In addition, the learning rates for CIFAR-10 / 100 and ImageNet were 10e-4 and 10e-6, respectively. And they decayed by 0.1 every 100 epochs.

[0077] This experiment first analyzes the performance of several ZSQ methods on the ImageNet dataset, including GZNQ, ClusterQ, Qimera, IntraQ and the ZSQ method proposed in this embodiment. MobileNetV1, MobileNetV2 and ResNet18 were selected as target models. This experiment shows the classification results of 5-bit and 4-bit quantization models. The comparison index is the classification accuracy. The objective evaluation results are shown in Table 1 below, which shows the effectiveness of the model quantization method based on long-distance dependencies and zero-sample generation in this embodiment. Table 1 is as follows:

[0078] Table 1: Classification comparison results of the quantized ResNet of the present invention and various algorithms on the CIFAR-10 dataset

[0079]

[0080] The experimental results on the data set show that the model quantization method based on long-distance dependency and zero-sample generation in this embodiment can be effectively used for the quantization task of the image classification model and can achieve relatively good results. Figure 2 , which is a visualization of the quantization of the ResNet18 floating-point model to 4 bits wide using the model quantization method based on long-distance dependencies and zero-sample generation disclosed in this embodiment and different ZSQ methods.

[0081] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. The embodiments described in the present invention are merely descriptions of the preferred embodiments of the present invention, and do not limit the concept and scope of the present invention. The various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction, and such combinations should also be regarded as the contents disclosed in the present disclosure as long as they do not violate the concept of the present invention. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0082] The present invention is not limited to the specific details of the above-mentioned embodiments. Within the technical concept of the present invention and without departing from the design concept of the present invention, various modifications and improvements made to the technical solution of the present invention by technical personnel in this field should fall within the protection scope of the present invention. The technical contents for which protection is sought in the present invention have been fully recorded in the claims.

Claims

1. A model quantization method based on long-distance dependency and zero-sample generation, characterized in that: The following steps are involved: Step 1: construct a synthetic data generation network, wherein the synthetic data generation network includes a long-distance information fusion generator and an edge information enhancement confrontation module; wherein the long-distance information fusion generator establishes a long-distance dependency relationship for pixels at different distances in an image to obtain global features of the image, thereby generating synthetic data; the edge information enhancement confrontation module adds a new angle deviation to the cosine similarity calculation between the intra-class feature vector and the cluster center to expand the distance between the vector and the center and reduce the similarity between the two; Step 2: configure the hyperparameters of the synthetic data generation network training, and then input standard Gaussian noise into the synthetic data generation network for training until the loss of the synthetic data generation network converges, thereby obtaining synthetic data that matches the distribution of real data; Step 3: Use the synthetic data as calibration data to perform post-training quantization on the target image classification model to obtain a quantized model; Next, the obtained quantized model is combined with the full-precision model to construct a distillation network. The distillation network uses a decoupled knowledge distillation network to take KL divergence as the loss function for the target class and the background class for joint fine-tuning training, and introduces hyperparameters to alleviate the sample imbalance problem, thereby transferring knowledge from the full-precision model to the quantized model. Then, the synthetic data is input into the distillation network for training until the loss of the distillation network converges, so as to fine-tune the quantization model therein and obtain a fine-tuned quantization model.

2. A model quantization method based on long-distance dependency and zero-sample generation according to claim 1, characterized in that: During training in step 2, the standard Gaussian noise is input into the long-distance information fusion generator in the synthetic data generation network to obtain synthetic data, and then the loss of the synthetic data to the batch normalization statistics and the edge information enhancement adversarial module is calculated as the loss of the synthetic data generation network.

3. The model quantization method based on long-distance dependency and zero-sample generation according to claim 1, characterized in that: In step 3, a zero-shot quantization method is used to compress the target image classification model without using the original data, and then the target image classification model is post-trained and quantized using the synthetic data as calibration data to obtain a quantized model.

4. The model quantization method based on long-distance dependency and zero-sample generation according to claim 1, characterized in that: In step 3, the full-precision model is a pre-trained full-precision model, and the full-precision model uses BNS loss to constrain the activation distribution.

5. The model quantization method based on long-distance dependency and zero-sample generation according to claim 1, characterized in that: In step 3, when the distillation network uses a decoupled knowledge distillation network to take KL divergence as the loss function for joint fine-tuning training for the target class and the background class respectively, the knowledge distillation is decomposed into target class knowledge distillation and non-target class knowledge distillation, TCKD is set to represent the similarity between the binary classification probabilities of the teacher and the student of the target class, and NCKD is set to represent the similarity between the probabilities of the teacher and the student of the non-target class, the NCKD loss is converted into a Focal Loss-like form and combined with the TCKD loss as the loss function during the joint fine-tuning training of the distillation network.

6. The model quantization method based on long-distance dependency and zero-sample generation according to claim 1, characterized in that: In step 3, when the synthetic data is input into the distillation network for training, the quantization model is fine-tuned using the cross entropy loss, and the output of the full-precision model and the quantization model output are trained using the loss of the de-distillation network to align the accuracy, thereby constructing the total loss of the quantization model training.

Images