Generator-based deep learning model diversity data-free quantification method

Abstract

The invention provides a deep learning model diversity data-free quantification method based on a generator. The method comprises the steps of obtaining basic feature representation of a category label of target generation data; the generator maps generation features of different levels through a normalized stream, and each layer of attention mechanism adds one mapped noise data to learn the relationship between different categories of labels to generate synthetic data; inputting the synthetic data into a full-precision model and a quantitative model at the same time, and extracting different levels of recognition features from the full-precision model; the multi-layer feature mixer obtains identification features of the same type of synthetic data at different levels and carries out linear fusion, and a fusion result is used for initializing basic feature representation; and aligning each level of the quantitative model and each level of the full-precision model, and calibrating the quantitative model by using the distribution difference output by the full-precision model and the quantitative model. According to the method, the diversity quantification requirement of the data-free quantification model is met, and the effective improvement of the model quantification effect is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and relates to a method for quantifying the diversity of deep learning models based on a generator without data. Background Art

[0002] Model quantization is an effective method for compressing deep neural networks and accelerating the inference time on edge devices. Quantization can be roughly divided into two categories: quantization-aware training (QAT) quantizes the weights while training a full-precision model using the original data. Post-training quantization (PTQ), on the other hand, is performed after the full-precision model is trained. Usually, PTQ requires the original training data to calibrate the quantized model to obtain better performance; otherwise, the model performance will drop significantly. However, due to privacy issues, the original data is usually not available in practical scenarios. For example, medical images or face data are all private data. Therefore, it is necessary to develop a data-free quantization method.

[0003] The generator-based framework can be used for data-free quantization methods, which can quantize the model without the original training data. It uses a generator to synthesize training data and then uses these fake data to calibrate the quantized model. The inspiration for this framework comes from the generative adversarial network (GAN). This framework consists of three components: a generator, a discriminator (i.e., the full-precision model), and a quantization model. The generator learns randomly initialized label embeddings to represent categories and then samples from random variables to generate synthetic data. Subsequently, these synthetic data are used to calibrate the quantization model. The role of the discriminator is to force the generator to synthesize correct data according to the category labels.

[0004] Although the generator-based data-free quantization framework has achieved good results in data-free scenarios, it still has the following problems:

[0005] First of all, the generator method based on GAN usually encounters the mode collapse problem, that is, the generator performs poorly in synthesizing diverse data. When the generator collapses at some data points, it can only generate a small number of highly similar images for each category. Due to the lack of diverse data, it is difficult for the quantization model to be accurately calibrated, resulting in a performance drop compared to when using the original data for calibration.

[0006] Secondly, under the mode collapse problem, the activation feature values of the data generated by the deep learning model show a spike and long-tail distribution, which is very unfavorable for quantization in the activation layer, especially in the case where existing data-free quantization methods all use a fixed truncation range. This will result in the weak performance ability of the weights when using a fixed number of quantization bits (bit), affecting the effect of the model.

[0007] Therefore, how to provide a data-free quantization method for the diversity of deep learning models based on a generator that can effectively improve the model quantization effect on a zero-shot basis is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0008] In view of this, the present invention proposes a data-free quantization method for the diversity of deep learning models based on a generator, which meets the diversity quantization requirements of data-free quantization models and ensures an effective improvement in the model quantization effect.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] The present invention discloses a data-free quantization method for the diversity of deep learning models based on a generator. The model architecture includes: a generator, a full-precision model, a quantization model, and a multi-layer feature mixer. The generator includes an attention module based on normalizing flow. The full-precision model uses a deep learning model. The quantization model is used to perform a quantization process on the full-precision model. The method includes the following steps:

[0011] Obtain the basic feature representation of the category label of the target generated data, and use the basic feature representation and noise data as the input of the generator.

[0012] Each layer of the feature extraction network in the generator embeds the attention module based on normalizing flow, maps the generated features at different levels through normalizing flow, and adds a mapped noise data to calculate the attention weight for each layer of the attention mechanism, learning the relationship between different category labels to generate synthetic data.

[0013] The generated synthetic data is simultaneously input into the full-precision model and the quantization model, and recognition features at different levels are extracted from the full-precision model. The multi-layer feature mixer obtains the recognition features of the same class of synthetic data at different levels and performs linear fusion. The fusion result is used to initialize the basic feature representation, serving as the diversity feature representation of the category label to synthesize the training data of the full-precision model.

[0014] During the discriminative classification process of the full-precision model, align the quantization model with each layer of the full-precision model, and use the distribution difference between the outputs of the full-precision model and the quantization model to calibrate the quantization model.

[0015] Preferably, the target generated data is a target image.

[0016] Preferably, the noise data is collected from Gaussian noise.

[0017] Preferably, the step of extracting recognition features at different levels from the full-precision model includes:

[0018] A classification layer is inserted into the full-precision models at different depths for identifying feature extraction, and the output dimension of each classification layer is the same as the number of categories.

[0019] Preferably, before the step of extracting different levels of identification features from the full-precision model, there is also an optimization step for the classification layer:

[0020] The outputs of different classification layers are aligned with the output of the full-precision model to complete the optimization of the classification layer. The identification features extracted by the optimized classification layer are used to initialize the basic feature representation.

[0021] Preferably, the operation steps of the alignment include:

[0022] Calculate the KL divergence between the outputs of each classification layer and the output of the full-precision model, and use the KL divergence as a loss function to finely tune the corresponding classification layer structure parameters.

[0023] Preferably, the generator adopts a regularization loss function based on similarity, including:

[0024] By calculating the cosine similarity between the current activation features of the synthetic data generated by the generator passing through the last activation layer of the full-precision model and the historical activation feature center of the generated data of the same class during the iteration process, it is used to constrain the distribution of the synthetic data generated by the generator in the feature space.

[0025] Preferably, the loss function of the generator is:

[0026]

[0027] SimLoss1 = ∑ d Cosine(P a (synthetic_data d _, f yd ));

[0028] In the formula, synthetic_data d represents the time-series data generated by the generator, d represents a sample in a set of sampling samples, yd represents the label of synthetic_data d , P a (synthetic_data d ) represents the activation features from the last activation layer of the full-precision model, f yd represents the moving average of the activation features of all the generated data belonging to the label yd in the historical iteration, and Cosine represents the cosine similarity metric;

[0029] Denotes the synthetic data generated by the generator, and P is the full-precision model. Denotes the discrimination result of using the full-precision model to discriminate the synthetic data. Denotes the cross-entropy loss.

[0030] Preferably, the quantization model adopts a regularization loss function based on similarity, including:

[0031] Introduce a learnable truncation variable to limit the activation value of the quantization model within the dynamic range, align the activation distributions between the quantization model and the full-precision model layer by layer, and learn the learnable truncation variable by minimizing the difference between the activation distributions of the two.

[0032] Preferably, the loss function of the quantization model is:

[0033]

[0034] In the formula, a represents the output value of the activation layer of the l-th layer in the quantization model. Denotes the KL divergence. Denotes the synthetic data generated by the generator, and P is the full-precision model. Denotes the discrimination result of using the full-precision model to discriminate the synthetic data, Q is the quantization model. Denotes the discrimination result of using the quantization model to discriminate the synthetic data.

[0035] Through the above technical solutions, compared with the prior art, the beneficial effects of the present invention include:

[0036] The quantization method of the present invention enables the full-precision model to ensure the classification accuracy rate. At the same time, under different quantization bit (bit) positions, the accuracy of the quantization model of the present invention significantly exceeds the existing data-free quantization method based on the generator.

[0037] The present invention learns the relationship between various categories through a multi-layer feature mixer, and at the same time proposes a normalized flow attention mechanism to focus on features at different levels, enabling the information of the full-precision model to enhance the diversity between and within classes. The dataset generated without using the original training dataset is closer to the original dataset, and the mode collapse problem is fundamentally alleviated.

[0038] The regularization loss proposed by the present invention is used to generate diverse data with more complex feature patterns. By aligning the activation distributions of the full-precision model and the quantization model layer by layer through the KL divergence, and combining the optimization of the adaptive truncation range, it effectively retains the high-level semantic information and reduces the information loss caused by low-precision representation.

[0039] In privacy-sensitive scenarios such as medical imaging and face recognition, the present invention can complete high-precision model quantization without using any original training data, meeting privacy compliance requirements.

[0040] The low-bit (bit) quantization of the present invention combined with a lightweight generator framework helps to achieve end-to-end efficient deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0042] Figure 1 FIG. is a system architecture diagram of a data-free quantization method for diversity of deep learning models based on a generator provided by an embodiment of the present invention;

[0043] Figure 2 FIG. is a flowchart of a data-free quantization method for diversity of deep learning models based on a generator provided by an embodiment of the present invention;

[0044] Figure 3 FIG. is a schematic diagram of the comparison of synthetic data generated by a generator in different methods provided by an embodiment of the present invention;

[0045] Figure 4 FIG. is a schematic diagram of the comparison of the distribution effects of synthetic data sets and original data sets generated by different methods based on a small amount of original training data provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0047] An embodiment of the present invention discloses a data-free quantization method for diversity of deep learning models based on a generator, as Figure 1 shown. The model architecture on which it is based includes: a generator, a full-precision model, a quantization model, and a multi-layer feature mixer, where:

[0048] The generator includes an attention module based on normalizing flow.

[0049] The full-precision model can be a deep learning model stored in 16-bit or 32-bit, such as model architectures like ViT and ResNet.

[0050] The quantization model is used to perform the quantization process on the full-precision model. The quantization process refers to converting parameters such as weights and activation values in the full-precision model network from high precision to low precision. This process aims to reduce the model size and computational requirements, while also facilitating the improvement of the inference efficiency during the inference stage of the full-precision model. The quantization model can use a deep learning model stored with less than 16-bit.

[0051] As Figure 2 shown, it specifically includes the following steps:

[0052] Obtain the basic feature representation of the class label of the target generated data, and use the basic feature representation and noise data as the input of the generator; among them, the basic feature representation means that before the pre-generated data, the class label will be converted into an initial embedding vector through a simple embedding layer. For example, the class label is embedded through Embedding.

[0053] Each layer of the feature extraction network in the generator is embedded with a normalizing flow-based attention module. Different levels of generated features are mapped through the normalizing flow, and a mapped noise data is added to each layer of the attention mechanism to calculate the attention weights, learning the relationships between different class labels to generate synthetic data; it should be noted that the existing self-attention process requires stacking multiple attention mechanism blocks as the layers of the full-precision model to specifically learn the detailed features at different levels, but this will lead to excessive parameters. Due to the technical conditions of the present invention lacking original training data, optimizing these parameters becomes very challenging. Therefore, it is proposed to use flow mapping to learn different non-Gaussian distributions;

[0054] The generated synthetic data is simultaneously input into the full-precision model and the quantization model, and recognition features at different levels are extracted from the full-precision model; the multi-layer feature mixer obtains the recognition features of the same class of synthetic data at different levels and performs linear fusion, and the fusion result is used to initialize the basic feature representation, as the diversity feature representation of the class label to synthesize the training data of the full-precision model.

[0055] During the discriminative classification process of the full-precision model, align the quantization model with each layer of the full-precision model, and use the distribution difference between the outputs of the full-precision model and the quantization model to calibrate the quantization model.

[0056] In one embodiment, the target generated data is a target image.

[0057] In one embodiment, the noise data is collected from Gaussian noise, and the Gaussian noise satisfies the normal distribution.

[0058] In one embodiment, the steps of extracting recognition features at different levels from the full-precision model include:

[0059] First, the generator is used to pre-generate data for several rounds. A classification layer is inserted into full-precision models with different depths to extract recognition features based on this data, and the output dimension of each classification layer is the same as the number of categories. The recognition features at different levels of the full-precision model are used to guide the generator to generate synthetic data.

[0060] In one embodiment, before the steps of extracting recognition features at different levels from the full-precision model, there is also an optimization step for the classification layer:

[0061] The outputs of different classification layers are aligned with the output of the full-precision model to complete the optimization of the classification layer. The recognition features extracted by the optimized classification layer are used to initialize the basic feature representation.

[0062] In this embodiment, the alignment operation steps include:

[0063] The KL divergence is calculated between the outputs of each classification layer and the output of the full-precision model, and the KL divergence is used as a loss function to fine-tune the corresponding classification layer structure parameters. Specifically, during execution, the KL divergence is calculated between the logits of the outputs of the classification layers added to the model at different depths and the logits of the last layer, and this is used as a loss function to fine-tune the classification layer.

[0064] In one embodiment, the generator first learns the relationship between embeddings of different categories based on the attention mechanism of normalizing flow, and uses a linear network to map features at different levels. A mapped noise is added to the attention mechanism of each layer for training. For example, for the resnet network architecture, noise is sampled and incorporated into each block. Then, an upsampling convolutional network is used to generate synthetic data, such as synthetic images.

[0065] In one embodiment, the loss function of the overall model architecture starts from the activation values of the full-precision model to generate diverse data for facilitating the fine-tuning of the model. This loss can be divided into two parts, one part is the constraint on the generator and the other part is the constraint on the quantization model.

[0066] In this embodiment, the generator adopts a similarity-based regularization loss function, including:

[0067] During the optimization of the generator, record the feature centers of all the generated data, and use a similarity loss to constrain the data generated by the generator to ensure data diversity. That is, by calculating the cosine similarity between the current activation features of the synthetic data generated by the generator after the last activation layer of the full-precision model and the historical activation feature centers of the generated data of the same class during the iteration process, it is used to constrain the distribution of the synthetic data generated by the generator in the feature space.

[0068] During specific execution, the loss function of the generator is:

[0069]

[0070] SimLoss1 = ∑ d Cosine(P a (synthetic_data d ), f yd );

[0071] In the formula, synthetic_data d represents the time-series data generated by the generator. For the sampling process of the generated data and the label y, multiple samples are sampled at a time, that is, one batch. d represents a sample in this batch, and yd represents the label of synthetic_data d . P a (synthetic_data d ) represents the activation features from the last activation layer of the full-precision model, and f yd represents the moving average of the activation features of all the generated data belonging to the label yd in the historical iteration. Cosine represents the cosine similarity metric; represents the synthetic data generated by the generator, and P is the full-precision model. represents the discrimination result of using the full-precision model to discriminate and output the synthetic data, represents the cross-entropy loss. This loss function is added to the training process of the generator to guide the generator to generate data.

[0072] In this embodiment, the quantization model adopts a regularization loss function based on similarity, including:

[0073] Use a learnable truncation variable to replace the fixed truncation variable, and learn this variable by aligning with the difference between the full-precision model. So that the quantization model can better adapt to the information loss caused by the long-tail distribution. That is, introduce a learnable truncation variable to limit the activation values of the quantization model within the dynamic range, align the activation distributions between the quantization model and the full-precision model layer by layer, and learn the learnable truncation variable by minimizing the difference between the activation distributions of the two.

[0074] It should be noted that the truncation range refers to the maximum and minimum values of model quantization. The activation value refers to the value output by each neuron (or node) after applying the activation function during the forward propagation of the network. These values reflect the result of the information processed by this layer when the input data is passed through the network layer.

[0075] During specific execution, the loss function of the quantization model is:

[0076]

[0077] In the formula, a represents the output value of the activation layer of the l-th layer in the quantization model, represents the KL divergence, represents the synthetic data generated by the generator, and P is the full-precision model. represents the discrimination result of using the full-precision model to discriminate the synthetic data, Q is the quantization model, represents the discrimination result of using the quantization model to discriminate the synthetic data.

[0078] It should be noted that in this embodiment, distillation adjustment is used for the activation layers in the full-precision model and the quantization model to learn to adapt to diverse synthetic data, so as to determine the appropriate pruning range of the activation layer. For the l-th activation layer, where 1 < l < J, and J represents the total number of selected layers in the full-precision model and the quantization model, initialize a learnable pruning range [0, beta]. Then, pass each l-th activation value in the quantization model through the pruning range [0, beta] to approximate the activation value of the corresponding activation layer in the full-precision model.

[0079] It can be understood that because of the long-tailed distribution of the activation values and their very low information entropy, such a constraint can enable it to adapt to a suitable pruning range in data with large distribution differences.

[0080] As shown in Table 1, through comparative experimental data, the significant advantages of the method of the present invention in low-bit (bit) quantization tasks are intuitively demonstrated. Among them, the method of the present invention performs excellently on different models (from lightweight MobileNetV2 to complex ResNet-50) and datasets (from Cifar-100 to ImageNet), reflecting its strong adaptability. Especially in extreme quantization configurations (such as 3w3a) and large-scale tasks (such as ImageNet), the performance improvement is more prominent.

[0081] Table 1 Comparison table of model quantization results

[0082]

[0083] Such as Figure 3As shown, the data generated by the present invention also exceeds the data diversity generated by existing methods in terms of diversity. Among them, Figure (a) is the synthetic data generated by the method of the present invention, and Figure (b) is the synthetic data generated by other methods.

[0084] As Figure 4 shown, the dataset generated by the method of the present invention without using the original training dataset is closer to the original dataset in terms of distribution. The yellow data points in the figure are the synthetic dataset of the method of the present invention, the green is the original dataset, and the red and blue are the synthetic datasets of other methods.

[0085] The above has introduced in detail the method for quantifying the diversity of the deep learning model based on the generator provided by the present invention. In this embodiment, specific examples are used to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

[0086] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined in this embodiment can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this embodiment, but will conform to the widest scope consistent with the principles and novel features disclosed in this embodiment.

Claims

1. A data-free quantization method for the diversity of deep learning models based on a generator, characterized in that The model architecture based on includes: a generator, a full-precision model, a quantization model, and a multi-layer feature mixer. The generator includes an attention module based on normalizing flow. The full-precision model uses a deep learning model. The quantization model is used to perform a quantization process on the full-precision model. The steps are as follows: Obtain the basic feature representation of the class label of the target generated data, and use the basic feature representation and noise data as the input of the generator. Each layer of the feature extraction network in the generator embeds the attention module based on normalizing flow. Different levels of generated features are mapped through normalizing flow, and a mapped noise data is added to each layer of the attention mechanism to calculate the attention weight, learning the relationship between different class labels to generate synthetic data. The generated synthetic data is simultaneously input into the full-precision model and the quantization model. Recognition features at different levels are extracted from the full-precision model. The multi-layer feature mixer obtains the recognition features of the same class of synthetic data at different levels and performs linear fusion. The fusion result is used to initialize the basic feature representation, serving as the diversity feature representation of the class label to synthesize the training data of the full-precision model. During the discriminative classification process of the full-precision model, align the quantization model with each level of the full-precision model, and use the distribution difference between the outputs of the full-precision model and the quantization model to calibrate the quantization model.

2. The method for data-free quantization of the diversity of a deep learning model based on a generator according to claim 1, wherein, The target generated data is a target image.

3. A data-free quantization method for the diversity of a deep learning model based on a generator according to claim 1, characterized in that The noise data is collected from Gaussian noise.

4. A data-free quantization method for the diversity of a deep learning model based on a generator according to claim 1, characterized in that The step of extracting recognition features at different levels from the full-precision model includes: Insert a classification layer in the full-precision model at different depths to extract recognition features, and the output dimension of each classification layer is the same as the number of classes.

5. A data-free quantization method for the diversity of a deep learning model based on a generator according to claim 4, characterized in that Before the step of extracting recognition features at different levels from the full-precision model, there is also an optimization step for the classification layer: Align the outputs of different classification layers with the output of the full-precision model to complete the optimization of the classification layer. The recognition features extracted by the optimized classification layer are used to initialize the basic feature representation.

6. A data-free quantization method for deep learning model diversity based on a generator according to claim 5, characterized in that The operation steps of the alignment include: Calculate the KL divergence between the outputs of each classification layer and the output of the full-precision model, and use the KL divergence as a loss function to fine-tune the corresponding classification layer structure parameters.

7. A data-free quantization method for the diversity of deep learning models based on a generator according to claim 1, characterized in that The generator uses a regularization loss function based on similarity, including: By calculating the cosine similarity between the current activation feature of the synthetic data generated by the generator passing through the last activation layer of the full-precision model and the historical activation feature center of the same class of generated data during the iteration process, it is used to constrain the distribution of the synthetic data generated by the generator in the feature space.

8. A data-free quantization method for the diversity of a deep learning model based on a generator according to claim 7, characterized in that The loss function of the generator is: SimLoss1 = ∑ d Cosine(P a (synthetic_data d ), f yd ); where synthetic_data d represents the time series data generated by the generator, d represents a sample in a set of sampling samples, and yd represents the d label of synthetic_data a (synthetic_data d ) represents the activation features from the last activation layer of the full-precision model, and f yd represents the moving average of the activation features of all the generated data belonging to the label yd in the historical iterations; Cosine represents the cosine similarity metric; Denotes the synthetic data generated by the generator, and P is the full-precision model. Denotes the discrimination result of using the full-precision model to discriminate and output the synthetic data, Denotes the cross-entropy loss.

9. A data-free quantization method for the diversity of deep learning models based on a generator according to claim 1, characterized in that The quantization model uses a regularization loss function based on similarity, including: Introduce a learnable truncation variable to limit the activation value of the quantization model within a dynamic range, align the activation distributions between the quantization model and the full-precision model layer by layer, and learn the learnable truncation variable by minimizing the difference between the two activation distributions.

10. A method for diversity-based data-free quantization of a deep learning model based on a generator, characterized in that, The loss function of the quantization model is: where a represents the output value of the activation layer in the l-th layer of the quantization model, represents the KL divergence, represents the synthetic data generated by the generator, and P is the full-precision model. represents the discrimination result of using the full-precision model to discriminate the synthetic data, and Q is the quantization model, represents the discrimination result of using the quantization model to discriminate the synthetic data.

Images