A neural network quantization method based on multi-model joint learning

Abstract

The application provides a neural network quantization method based on multi-model joint learning, relates to the technical field of neural network quantization, and utilizes a full-precision model to optimize noise samples with Gaussian distribution to obtain pseudo samples, and utilizes the pseudo samples and the full-precision model to quantize to-be-quantized models with different bit widths, and simultaneously through joint learning of multiple to-be-quantized models, multiple high-precision quantization models with different bit widths are obtained, so that the problems of serious precision reduction of a quantization model in a scene without original training data and serious time consumption in a model quantization process are solved.

Description

Technical Field

[0001] This application relates to the field of neural network quantization technology, and in particular to a neural network quantization method based on multi-model joint learning. Background Technology

[0002] In recent years, neural networks based on deep learning technology have made significant progress in fields such as computer vision, natural language processing, and speech recognition. Numerous neural networks have been deployed in small devices and edge devices, such as mobile phones and cameras, that rely on batteries or solar power. However, neural networks have a large number of parameters and are computationally complex, requiring substantial computing resources and bandwidth, thus limiting their application on resource-constrained platforms. In particular, large neural networks have numerous layers and nodes, making it crucial to consider how to reduce their memory and computational requirements. In this context, compressing neural network models with a large number of parameters and high computational complexity plays a vital role in the development of deep learning technology.

[0003] To address the challenge of lightweighting neural network models, current mainstream research methods primarily focus on model compression to reduce storage and computational costs, including pruning, quantization, and knowledge distillation. Quantization, in particular, is a widely applicable neural network compression technique. Currently, the most prevalent quantization schemes are PTQ (Post-Training Quantization) and QAT (Quantization Aware Training), but both require real training data to calibrate the quantized model. However, in many scenarios, due to privacy, copyright, or confidentiality concerns, the original data used to train the teacher model is not always available, such as medical privacy data and sensitive user information.

[0004] Therefore, there is an urgent need for a neural network quantization method based on multi-model joint learning. Summary of the Invention

[0005] In view of the above problems, embodiments of this application provide a neural network quantization method based on multi-model joint learning, so as to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect, this application provides a neural network quantization method based on multi-model joint learning, the method comprising:

[0007] Obtain a pre-trained full-precision model and multiple models to be quantized with different bit widths;

[0008] Based on the full-precision model, the multiple quantization models with different bit widths are quantized to obtain multiple high-precision quantization models with different bit widths that match the full-precision model. The high-precision quantization models are obtained through the following steps:

[0009] S1: For the full-precision model, noise samples are initialized using a Gaussian distribution to generate multiple pseudo-samples relative to the full-precision model, and a pseudo-label is sampled for each pseudo-sample;

[0010] S2: Input the pseudo-sample into the full-precision model, calculate the first target loss function, and optimize the pixel value of the pseudo-sample based on the first target loss function;

[0011] S3: Determine the second objective loss function based on the divergence between the full-precision model and each of the models to be quantized, as well as the divergence between each pair of the models to be quantized;

[0012] S4: Based on the difference between the pseudo-label of the optimized pseudo-sample and the output result after inputting the optimized pseudo-sample into each of the models to be quantized, calculate the third objective loss function corresponding to each of the models to be quantized.

[0013] S5: Summing the second objective loss function and the third objective loss function to obtain the fourth objective loss function, and updating each of the models to be quantized based on the fourth objective loss function to obtain multiple high-precision quantization models.

[0014] Optionally, the first target loss function is calculated through the following steps:

[0015] The first loss function is determined using the feature parameter information in the batch normalization layer of the full-precision model.

[0016] The second loss function is determined based on the intra-class heterogeneity among samples of the same category in the training samples during the actual model training process.

[0017] The third loss function is determined based on the inter-class homogeneity of samples of different categories in the training samples during the actual model training process.

[0018] The fourth loss function is determined based on the difference between the output of the full-precision model and the pseudo-labels corresponding to the pseudo-samples;

[0019] The first target loss function is obtained by summing the first loss function, the second loss function, the third loss function, and the fourth loss function;

[0020] The fourth loss function is obtained by the following formula:

[0021]

[0022] Where CE represents the cross-entropy loss function, Y represents the output of the full-precision model, and Y represents the pseudo-label.

[0023] Optionally, before inputting the plurality of pseudo-samples into the full-precision model, the method further includes:

[0024] The original feature image corresponding to each pseudo-sample is randomly cropped according to a first preset probability and a preset scaling rate to obtain a pseudo-sample that possesses some features of the original feature image; or,

[0025] The pixel values ​​in any feature channel of the original feature image corresponding to each pseudo-sample are enhanced or weakened according to a second preset probability to enrich the brightness and chromaticity information of the original feature image.

[0026] Optionally, determining the first loss function using the feature parameter information in the batch normalization layer of the full-precision model includes:

[0027] The multiple pseudo-samples are sequentially input into all batch normalization layers of the full-precision model, and the mean and variance of the multiple pseudo-samples in each batch normalization layer are calculated.

[0028] The first loss function is determined based on the difference between the mean of the multiple pseudo-samples calculated in each batch normalization layer and the mean of each batch normalization layer, and the difference between the variance of the multiple pseudo-samples calculated in each batch normalization layer and the variance of each batch normalization layer. The first loss function is shown in the following formula:

[0029]

[0030] Wherein, μ′ l and σ′ l These represent the mean and variance of the pseudo-samples calculated in the l-th batch normalization layer, respectively. and These represent the mean and variance stored in the l-th batch normalization layer, respectively. This represents the square of the L2 norm of the eigenvector.

[0031] Optionally, the second loss function is obtained by the following formula:

[0032]

[0033] in, This serves as the lower bound for the intra-class heterogeneity distance constraint. This is the upper bound of the intra-class heterogeneity distance constraint; Dis() is a function that measures the similarity between two feature vectors. This represents the feature vector of a pseudo-sample after passing through the last feature extraction layer of the full-precision model. The class center represents the average value of all samples within a class.

[0034] Optionally, the third loss function is obtained by the following formula:

[0035]

[0036] in, This serves as the lower bound of the inter-class homogeneity distance constraint. This is the upper bound of the inter-class homogeneity distance constraint; Dis() is a function that measures the similarity between two feature vectors. This represents the feature vector of a pseudo-sample after passing through the last feature extraction layer of the full-precision model. This represents the average class center of all different classes of samples. The average class center is the center point obtained by averaging the class centers of multiple classes.

[0037] Optionally, determining the second objective loss function based on the divergence between the full-precision model and each of the models to be quantized, as well as the pairwise divergence between each of the models to be quantized, includes:

[0038] The first divergence is obtained by summing the divergences between the full-precision model and each of the models to be quantized.

[0039] The second divergence is obtained by summing the divergences between each pair of the models to be quantized.

[0040] The sum of the first divergence and the second divergence is determined as the second objective loss function, wherein the second objective loss function is shown in the following formula:

[0041]

[0042] Where λ i and λ i,j For adaptive loss coefficients; This represents the KL divergence calculation of the full-precision model as the teacher model relative to the i-th model to be quantized as the student model; This indicates the full-precision model's performance on the optimized pseudo-samples. The output; The i-th model to be quantized uses the optimized pseudo-samples The output; This represents the KL divergence calculation between the i-th model to be quantized as the teacher model and the j-th model to be quantized as the student model in a pairwise quantization model. This represents the j-th model to be quantized for the optimized pseudo-samples. The output.

[0043] Optionally, the adaptive loss coefficient is obtained by the following method:

[0044] Obtain the output result after inputting the optimized pseudo-sample into the full-precision model, and the output result after inputting the optimized pseudo-sample into the model to be quantized;

[0045] The adaptive coefficients are determined based on the output results of inputting the optimized pseudo-samples into the full-precision model and the output results of inputting the optimized pseudo-samples into the model to be quantized. The adaptive coefficients are calculated using the following formula:

[0046]

[0047]

[0048] Where Exp(·) represents an exponential function with the natural constant e as its base.

[0049] Optionally, the step of calculating the third objective loss function corresponding to each of the quantized models based on the difference between the pseudo-labels of the optimized pseudo-samples and the output results after inputting the optimized pseudo-samples into each of the quantized models includes:

[0050] Obtain the pseudo-labels corresponding to each of the optimized pseudo-samples, and input each of the optimized pseudo-samples into the output results of each of the models to be quantized.

[0051] Calculate the cross-entropy loss function between the output of each model to be quantized and the pseudo-label of each optimized pseudo-sample, and sum the obtained cross-entropy loss functions to obtain the third objective loss function, which is obtained by the following formula:

[0052]

[0053] Among them, Let be the cross-entropy loss function between the i-th model to be quantized and the pseudo-label.

[0054] Optionally, the plurality of quantization models with different bit widths include a plurality of quantization models with a first bit width and a plurality of quantization models with a second bit width.

[0055] Wherein, the full-precision model is defined as a teacher model to guide the parameter updates of the plurality of first-bit-width quantized models defined as student models and the plurality of second-bit-width quantized models defined as student models; and,

[0056] The first bit-width model to be quantized is defined as the teacher model to guide the parameter updates of the plurality of second bit-width models to be quantized, wherein the first bit-width is greater than the second bit-width.

[0057] The beneficial effects of this application are:

[0058] This application provides a neural network quantization method based on multi-model joint learning. The method includes: acquiring a pre-trained full-precision model and multiple models to be quantized with different bit widths; quantizing the multiple models to be quantized with different bit widths based on the full-precision model to obtain multiple high-precision quantization models with different bit widths that match the full-precision model. The high-precision quantization models are obtained through the following steps: S1: For the full-precision model, initializing noise samples using a Gaussian distribution to generate multiple pseudo-samples relative to the full-precision model, and sampling a pseudo-label for each pseudo-sample; S2: Inputting the pseudo-samples into the full-precision model to calculate a first target loss function. S3: Based on the divergence between the full-precision model and each of the models to be quantized, and the divergence between each pair of models to be quantized, a second objective loss function is determined; S4: Based on the difference between the pseudo-label of the optimized pseudo-sample and the output result after inputting the optimized pseudo-sample into each of the models to be quantized, a third objective loss function is calculated for each model to be quantized; S5: The second objective loss function and the third objective loss function are summed to obtain a fourth objective loss function, and each model to be quantized is updated based on the fourth objective loss function to obtain multiple high-precision quantization models. This application obtains pseudo-samples by applying a Gaussian distribution to the full-precision model, and uses the pseudo-samples and the full-precision model to quantize models to be quantized with different bit widths. At the same time, through joint learning of multiple models to be quantized, multiple high-precision quantization models with different bit widths are obtained, thereby solving the problems of severe accuracy degradation of quantization models in scenarios without original training data and severe time consumption in the model quantization process. Attached Figure Description

[0059] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a flowchart illustrating the steps of a neural network quantization method based on multi-model joint learning provided in an embodiment of this application.

[0061] Figure 2 This is a flowchart illustrating the calculation of the first loss function based on the feature parameters of a batch normalization layer using a full-precision model, as provided in an embodiment of this application.

[0062] Figure 3 This is a schematic diagram of a process for constructing a second loss function based on intra-class heterogeneity of real samples, provided in an embodiment of this application.

[0063] Figure 4 This is a schematic diagram of a process for constructing a third loss function based on the inter-class homogeneity of real samples, provided in an embodiment of this application.

[0064] Figure 5 This is a schematic diagram illustrating the calculation process of a second objective loss function provided in an embodiment of this application;

[0065] Figure 6 This is a flowchart of a neural network quantization method based on multi-model joint learning provided in an embodiment of this application;

[0066] Figure 7 This is a schematic diagram of a neural network quantization system based on multi-model joint learning provided in an embodiment of this application. Detailed Implementation

[0067] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0068] In a first aspect, this application provides a neural network quantization method based on multi-model joint learning, the method as follows: Figure 1 As shown, it includes:

[0069] Step S101: Obtain the pre-trained full-precision model and multiple models to be quantized with different bit widths;

[0070] Specifically, in this step, a high-precision neural network model needs to be pre-trained and defined as a full-precision model. This model is trained on a large-scale dataset. Then, multiple models with different bit widths need to be obtained for quantization.

[0071] Step S102: Based on the full-precision model, quantize the multiple models to be quantized with different bit widths to obtain multiple high-precision quantization models with different bit widths that match the full-precision model.

[0072] Specifically, multiple models to be quantized with different bit widths are quantized using a full-precision model, thereby obtaining multiple high-precision quantization models with different bit widths that match the full-precision model. These high-precision quantization models are obtained through the following steps:

[0073] S1: For the full-precision model, noise samples are initialized using a Gaussian distribution to generate multiple pseudo-samples relative to the full-precision model, and a pseudo-label is sampled for each pseudo-sample;

[0074] Specifically, in step S1, noise samples are first initialized for the full-precision model using a Gaussian distribution, resulting in multiple pseudo-samples relative to the full-precision model. A pseudo-label is then sampled for each pseudo-sample. The size of the initialized Gaussian noise samples (pseudo-samples) is consistent with the input of the corresponding full-precision model. For example, for models pre-trained on the CIFAR-10 and CIFAR-100 datasets, the pseudo-sample size is 32×32, while for models pre-trained on the ImageNet1K dataset, the pseudo-sample size is 224×224. In practical applications, a batch of Gaussian noise samples is initialized each time, and the batch of samples itself is treated as a learnable object and fed into the optimizer. The optimizer optimizes the pixel values ​​of the pseudo-samples. Simultaneously, samples are taken from the uniform distribution U(0,9) / U(0,999) to serve as the label and classification value for each pseudo-sample in that batch, respectively.

[0075] S2: Input the pseudo-sample into the full-precision model, calculate the first target loss function, and optimize the pixel value of the pseudo-sample based on the first target loss function;

[0076] In this step, the generated pseudo-samples are sequentially input into the full-precision model. The first target loss function is calculated through the full-precision model, and the pixel values ​​of each pseudo-sample are updated by the reverse of the first target loss function.

[0077] Preferably, the first target loss function is calculated through the following steps:

[0078] The first loss function is determined using the feature parameter information in the batch normalization layer of the full-precision model.

[0079] The second loss function is determined based on the intra-class heterogeneity among samples of the same category in the training samples during the actual model training process.

[0080] The third loss function is determined based on the inter-class homogeneity of samples of different categories in the training samples during the actual model training process.

[0081] The fourth loss function is determined based on the difference between the output of the full-precision model and the pseudo-labels corresponding to the pseudo-samples;

[0082] The first target loss function is obtained by summing the first loss function, the second loss function, the third loss function, and the fourth loss function;

[0083] Specifically, in this embodiment, determining the first loss function using the feature parameter information in the batch normalization layer of the full-precision model includes:

[0084] The multiple pseudo-samples are sequentially input into all batch normalization layers of the full-precision model, and the mean and variance of the multiple pseudo-samples in each batch normalization layer are calculated.

[0085] The first loss function is determined based on the difference between the mean calculated by the multiple pseudo-samples in each batch normalization layer and the mean of each batch normalization layer, and the difference between the variance calculated by the multiple pseudo-samples in each batch normalization layer and the variance of each batch normalization layer.

[0086] In this embodiment, multiple pseudo-samples are sequentially input into the batch normalization layer of the pre-trained full-precision model, and the mean and variance of each pseudo-sample in each batch normalization layer are calculated. Furthermore, based on the differences between the mean stored in each batch normalization layer of the pre-trained full-precision model and the mean of each pseudo-sample in each batch normalization layer, as well as the differences between the variance stored in each batch normalization layer of the pre-trained full-precision model and the variance of each pseudo-sample in each batch normalization layer, a first loss function is determined. The first loss function is shown in the following formula (1):

[0087]

[0088] Wherein, μ′ l and σ′ l These represent the mean and variance of the pseudo-samples calculated in the l-th batch normalization layer, respectively. and These represent the mean and variance stored in the l-th batch normalization layer, respectively. This represents the square of the L2 norm of the eigenvector.

[0089] In a preferred embodiment, as follows is provided Figure 2 The diagram shows the flowchart for calculating the first loss function based on the feature parameters of the batch normalization layer in a full-precision model.

[0090] Specifically, the pseudo-samples are first input into the full-precision model. Through the forward propagation unit, the variance and mean of each pseudo-sample are calculated after being input into each batch normalization layer of the full-precision model. Through the matching module of the batch normalization layer, the difference between the variance obtained by inputting each pseudo-sample into each batch normalization layer of the full-precision model and the variance stored in each batch normalization layer is compared. The difference between the mean obtained by inputting each pseudo-sample into each batch normalization layer of the full-precision model and the mean stored in each batch normalization layer is also compared. Finally, these differences are integrated through the batch normalization layer matching loss summation unit to obtain the first loss function.

[0091] Preferably, considering that in real training data, different samples of the same class often have different features, their feature distributions often differ during feature extraction, i.e., intra-class heterogeneity among samples. This application constructs a second loss function to guide feature learning, enabling pseudo-samples to form heterogeneous intra-class features, i.e., intra-class heterogeneous features. Specifically, the second loss function is obtained through the following formula:

[0092]

[0093] in, This serves as the lower bound for the intra-class heterogeneity distance constraint. This is the upper bound of the intra-class heterogeneity distance constraint; Dis() is a function that measures the similarity between two feature vectors. This represents the feature vector of a pseudo-sample after passing through the last feature extraction layer of the full-precision model. The class center represents the average value of all samples within a class.

[0094] In practical applications, the function that measures the similarity between two feature vectors can be the cosine similarity distance.

[0095] In a preferred embodiment, a method such as... is provided. Figure 3 The diagram shows the process of constructing a second loss function based on intra-class heterogeneity of real samples.

[0096] Specifically, the pseudo-samples obtained in the aforementioned steps are first input into the feature extraction unit of the full-precision model for feature extraction. Then, the cosine similarity calculation unit calculates the cosine similarity between each pair of feature vectors. At the same time, the class center of the same type of samples is introduced. The cosine similarity distance is further constrained by the distance constraint unit, and finally the second loss function is determined by the intra-class heterogeneity calculation module.

[0097] In a preferred embodiment, the class center of similar samples It is calculated using the following formula (3):

[0098]

[0099] in, This indicates the sample currently being optimized; Tol c Indicates the sum of other optimized values. The set of all samples of the same category.

[0100] Preferably, considering that in the feature distribution of real training data, there is a certain overlap in the distribution of different categories of samples in the feature space, that is, the homogeneity of samples of different categories. Therefore, when synthesizing samples, this embodiment introduces a third loss function to recover some of the inter-class homogeneity features of the samples. Specifically, the third loss function is obtained by the following formula (4):

[0101]

[0102] in, This serves as the lower bound of the inter-class homogeneity distance constraint. This is the upper bound of the inter-class homogeneity distance constraint; Dis() is a function that measures the similarity between two feature vectors. This represents the feature vector of a pseudo-sample after passing through the last feature extraction layer of the full-precision model. This represents the average class center of all different classes of samples. The average class center is the center point obtained by averaging the class centers of multiple classes.

[0103] In practical applications, the function that measures the similarity between two feature vectors can be the cosine similarity distance.

[0104] In a preferred embodiment, a method such as... is provided. Figure 4 The diagram shows the process of constructing a third loss function based on the inter-class homogeneity of real samples.

[0105] Specifically, the pseudo-samples obtained in the aforementioned steps are first input into the feature extraction unit of the full-precision model for feature extraction. Then, the cosine similarity calculation unit calculates the cosine similarity between each pair of feature vectors. At the same time, the class centers of different classes of samples are introduced. The cosine similarity distance is further constrained by the distance constraint unit, and finally, the third loss function is determined by the inter-class homogeneity calculation module.

[0106] In a preferred embodiment, the average class center of samples from different classes is... It is calculated using the following formula (5):

[0107]

[0108] Where |OC| represents the number of all categories previously optimized, excluding the current category; Tol i This represents a set of samples that does not belong to the current category.

[0109] Preferably, a fourth loss function is determined based on the difference between the output of the full-precision model and the pseudo-labels corresponding to the pseudo-samples; wherein the fourth loss function is obtained by the following formula (6):

[0110]

[0111] Where CE represents the cross-entropy loss function, Y represents the output of the full-precision model, and Y represents the pseudo-label.

[0112] Preferably, after determining the first loss function, the second loss function, the third loss function, and the fourth loss function, these loss functions are summed to obtain the first target loss function, wherein the first target loss function is calculated by the following formula (7):

[0113]

[0114] Among them, L D-G Represents the first objective loss function; Represents the first loss function; This represents the second loss function; Represents the third loss function; This represents the fourth loss function.

[0115] In a preferred embodiment, before inputting the plurality of pseudo-samples into the full-precision model, the method further includes:

[0116] The original feature image corresponding to each pseudo-sample is randomly cropped according to a first preset probability and a preset scaling rate to obtain a pseudo-sample that possesses some features of the original feature image; or,

[0117] The pixel values ​​in any feature channel of the original feature image corresponding to each pseudo-sample are enhanced or weakened according to a second preset probability to enrich the brightness and chromaticity information of the original feature image.

[0118] Specifically, in this embodiment, data augmentation is performed on the initial noise samples, i.e., pseudo-samples, before inputting them into the pre-trained full-precision model. This causes the pseudo-samples to be transformed with different probabilities, including local information augmentation and channel augmentation.

[0119] In practical applications, data generation focuses solely on reconstructing the original data distribution using a pre-trained, full-precision model throughout the process. Given prior labels with initial loss, the entire image is optimized during generation. Therefore, synthetic images often result in the target object covering the entire image, and synthetic images with the same prior label become highly similar, leading to significant feature similarity among similar objects. However, in real-world data distributions, the features learned by the model for pseudo-samples are distributed across different locations within the image. Furthermore, similar samples may exhibit different proportions. Therefore, the synthesized pseudo-samples must closely approximate the real-world situation—the learned information must appear at different locations within the sample, while also considering the chromaticity and brightness variations of the synthesized samples.

[0120] Based on the above analysis, this embodiment proposes data augmentation before initializing noisy samples (i.e., pseudo-samples) into the pre-trained full-precision model. This allows the samples to be transformed with different probabilities. In this embodiment, random sample pruning and channel chroma adjustment operations are employed to enrich the information diversity of the generated samples, specifically including:

[0121] (1) Local Information Enhancement: For each generated pseudo-sample image, this application selects to randomly crop a small patch of the image with probability p1, instead of directly feeding the entire image into the pre-trained full-precision model for optimization. For each random crop, the scaling factor is sampled from a uniform distribution U(η,1), where η is a hyperparameter controlling the minimum scaling factor after cropping.

[0122] (2) Channel enhancement: In order to enrich the brightness and chromaticity information of the original training data, this embodiment uses probability p2 to perform channel enhancement. A channel is randomly selected, and the pixel value corresponding to the pseudo sample in the channel is enhanced or weakened to optimize the chromaticity difference of the synthesized sample.

[0123] In this application, the above content focuses on how to obtain pseudo samples by applying a Gaussian distribution to the full-precision model, and optimize the pseudo samples by using a first objective loss function, so that the pseudo samples can be closer to the original training data used to train the full-precision model, thus providing data preparation for obtaining the quantized model in the future.

[0124] S3: Determine the second objective loss function based on the divergence between the full-precision model and each of the models to be quantized, as well as the divergence between each pair of the models to be quantized;

[0125] In a preferred embodiment, the plurality of quantization models with different bit widths include a plurality of quantization models with a first bit width and a plurality of quantization models with a second bit width.

[0126] Wherein, the full-precision model is defined as a teacher model to guide the parameter updates of the plurality of first-bit-width quantized models defined as student models and the plurality of second-bit-width quantized models defined as student models; and,

[0127] The first bit-width model to be quantized is defined as the teacher model to guide the parameter updates of the plurality of second bit-width models to be quantized, wherein the first bit-width is greater than the second bit-width.

[0128] Specifically, in this embodiment, a multi-model joint learning paradigm is proposed, in which models to be quantized with different bit widths are selected to participate in the update simultaneously. The pre-trained full-precision model serves as the true teacher model. The low bit width models to be quantized with different bit widths simultaneously receive guidance from high bit width models to be quantized with different knowledge levels. The low bit width models to be quantized with different bit widths are divided into multiple first bit width models and multiple second bit width models, where the first bit width is greater than the second bit width.

[0129] In this embodiment, the quantized model with a second bit width simultaneously receives guidance from teacher models of different knowledge levels, effectively avoiding the problem of low quantization accuracy caused by excessive differences in knowledge levels. Unlike the multi-teacher paradigm, the quantized model with a first bit width simultaneously acts as both a teacher and a student model. That is, the first bit width quantized model serves both as a student model receiving guidance from the full-precision teacher model for parameter optimization, and as a teacher model guiding other low-bit-width quantized models for parameter optimization. Parameter updates are performed concurrently during training, and after a single training iteration, high-precision models with different quantization bit widths can be obtained, greatly improving the efficiency of the model quantization process. It is important to note that the first bit width quantized model does not refer to a single bit width, but specifically refers to multiple bit width quantized models with a bit width greater than a preset bit width. Similarly, the second bit width quantized model specifically refers to multiple bit width quantized models with a bit width less than or equal to a preset bit width.

[0130] In a preferred embodiment, determining the second objective loss function based on the divergence between the full-precision model and each of the models to be quantized, and the pairwise divergence between each of the models to be quantized, includes:

[0131] The first divergence is obtained by summing the divergences between the full-precision model and each of the models to be quantized.

[0132] The second divergence is obtained by summing the divergences between each pair of the models to be quantized.

[0133] The sum of the first divergence and the second divergence is determined as the second objective loss function, wherein the second objective loss function is as shown in the following formula (8):

[0134]

[0135] Where λ i and λ i,j For adaptive loss coefficients; This represents the KL divergence calculation of the full-precision model as the teacher model relative to the i-th model to be quantized as the student model; This indicates the full-precision model's performance on the optimized pseudo-samples. The output; This represents the spurious sample after optimization of the i-th model to be quantized. The output; This represents the KL divergence calculation between the i-th model to be quantized as the teacher model and the j-th model to be quantized as the student model in a pairwise quantization model. This represents the j-th model to be quantized for the optimized pseudo-samples. The output.

[0136] In a preferred embodiment, when the number of models to be quantized is large, the number of loss functions will also be relatively large. Assuming there are n models to be quantized besides the teacher model (full-precision model), the number of KL divergence loss functions used by the teacher model in the calculation is n; the number of KL divergences calculated among the remaining n models is as follows: The total number of KL divergences involved in the calculation is:

[0137] In multi-task training, where multiple models to be quantized exist, there are multiple different loss functions. In actual parameter updates, loss functions with larger values ​​often correspond to larger task gradients and faster update speeds. Conversely, loss functions with smaller values ​​have smaller gradients and slower update speeds. Therefore, if multiple loss functions converge at the same rate (with the same coefficients), the overall training process may favor a specific task, resulting in only one or a few specific models to be quantized being updated during training, while the convergence performance of other models is poor. To avoid this situation, this embodiment proposes a method such as... Figure 5 The diagram shown illustrates the calculation process of the second objective loss function, which is used to adaptively and dynamically adjust various loss functions during training. This involves inputting optimized pseudo-samples into the full-precision model and various different models to be quantized, obtaining the loss function corresponding to each model, and then using the adaptive loss controller unit to count all loss functions and calculate the corresponding coefficient for each loss, which is then multiplied by the initial loss.

[0138] Preferably, the calculation process of the adaptive coefficient is shown in formulas (9) and (10):

[0139]

[0140]

[0141] Where Exp(·) represents an exponential function with the natural constant e as its base.

[0142] S4: Based on the difference between the pseudo-label of the optimized pseudo-sample and the output result after inputting the optimized pseudo-sample into each of the models to be quantized, calculate the third objective loss function corresponding to each of the models to be quantized.

[0143] In a preferred embodiment, the step of calculating the third objective loss function corresponding to each of the quantized models based on the difference between the pseudo-labels of the optimized pseudo-samples and the output results after inputting the optimized pseudo-samples into each of the quantized models includes:

[0144] Obtain the pseudo-labels corresponding to each of the optimized pseudo-samples, and input each of the optimized pseudo-samples into the output results of each of the models to be quantized.

[0145] Calculate the cross-entropy loss function between the output of each model to be quantized and the pseudo-label of each optimized pseudo-sample, and sum the obtained cross-entropy loss functions to obtain the third objective loss function, wherein the third objective loss function is obtained by the following formula (11):

[0146]

[0147] Among them, Let be the cross-entropy loss function between the i-th model to be quantized and the pseudo-label.

[0148] Specifically, in this embodiment, the pseudo-labels corresponding to each optimized pseudo-sample are obtained respectively, and the optimized pseudo-labels are input into each quantization model to be quantized to obtain the output results of each quantization model. The cross-entropy loss function between the output results of each quantization model and the pseudo-labels corresponding to each optimized pseudo-sample is calculated respectively, and the multiple cross-entropy loss functions are summed to obtain the third target loss function.

[0149] S5: Summing the second objective loss function and the third objective loss function to obtain the fourth objective loss function, and updating each of the models to be quantized based on the fourth objective loss function to obtain multiple high-precision quantization models.

[0150] Preferably, the second and third objective loss functions obtained above are summed to obtain the fourth objective loss function, which is the final quantization loss function. The fourth objective loss function is then used to optimize each model to be quantized through backpropagation.

[0151] Preferably, the fourth objective loss function is shown in the following formula (12):

[0152] L Q =L CE +β·L KL (12)

[0153] Among them, L Q L represents the fourth objective loss function; CE The third objective loss function is represented; β is a hyperparameter; L KL This is the second objective loss function.

[0154] This application provides a neural network quantization method based on multi-model joint learning. The method includes: acquiring a pre-trained full-precision model and multiple models to be quantized with different bit widths; quantizing the multiple models to be quantized with different bit widths based on the full-precision model to obtain multiple high-precision quantization models with different bit widths that match the full-precision model. The high-precision quantization models are obtained through the following steps: S1: For the full-precision model, initializing noise samples using a Gaussian distribution to generate multiple pseudo-samples relative to the full-precision model, and sampling a pseudo-label for each pseudo-sample; S2: Inputting the multiple pseudo-samples into the full-precision model to calculate a first target loss function. S3: Based on the divergence between the full-precision model and each of the models to be quantized, and the divergence between each pair of models to be quantized, a second target loss function is determined; S4: Based on the difference between the pseudo-label of the optimized pseudo-sample and the output result after inputting the optimized pseudo-sample into each of the models to be quantized, a third target loss function corresponding to each model to be quantized is calculated; S5: The second target loss function and the third target loss function are summed to obtain a fourth target loss function, and each model to be quantized is updated based on the fourth target loss function to obtain multiple high-precision quantization models. This application obtains pseudo-samples by applying a Gaussian distribution to the full-precision model, and uses pseudo-samples and the full-precision model to quantize models to be quantized with different bit widths. At the same time, through joint learning of multiple models to be quantized, multiple high-precision quantization models with different bit widths are obtained, thereby solving the problems of severe decrease in quantization model accuracy and severe time consumption in the model quantization process in scenarios without original training data.

[0155] In summary, the embodiments of this application divide the quantization process of the entire neural network model into two stages. In the first stage, noise samples are used as input, and the characteristics of intra-class heterogeneity and inter-class homogeneity are considered to optimize the noise samples and generate pseudo samples needed for the quantization stage. In the second stage, multiple models are jointly trained to train models to be quantized with different bit widths, thereby improving training accuracy and shortening training time.

[0156] In a preferred embodiment, a method such as... is provided. Figure 6 The flowchart shown here illustrates a neural network quantization method based on multi-model joint learning, which specifically includes:

[0157] First, noisy samples of the full-precision model are initialized using a Gaussian distribution to obtain pseudo-samples, and a corresponding pseudo-label is sampled for each pseudo-sample. Next, local feature enhancement is performed on each pseudo-sample. Then, the enhanced pseudo-samples are input into the full-precision model, and the first, second, third, and fourth loss functions are calculated respectively. Finally, the first, second, third, and fourth loss functions are summed to obtain the first target loss function, and the pixel values ​​of each pseudo-sample are optimized using the first target loss function.

[0158] Further, determine whether the number of batches of samples obtained is greater than or equal to the preset number. If yes, proceed to the next step; otherwise, return to the step of initializing the noise samples of the full-precision model with a Gaussian distribution.

[0159] Furthermore, multiple models to be quantized with different bit widths are selected, and the second objective loss function and the third objective loss function corresponding to each model to be quantized are calculated respectively.

[0160] Furthermore, the second objective loss function and the third objective loss function are summed to obtain the fourth objective loss function;

[0161] Furthermore, each model to be quantized is optimized using a fourth objective loss function.

[0162] In a preferred embodiment, this application provides a method such as Figure 7 The illustrated neural network quantization system based on multi-model joint learning includes:

[0163] The acquisition module 201 is used to acquire the pre-trained full-precision model and multiple models to be quantized with different bit widths;

[0164] Quantization module 202 is used to quantize the plurality of models to be quantized with different bit widths based on the full-precision model, to obtain a plurality of high-precision quantization models with different bit widths that match the full-precision model, wherein the high-precision quantization model is obtained through the following steps:

[0165] The Gaussian noise initialization module 203 is used to initialize noise samples with a Gaussian distribution for the full-precision model, generate multiple pseudo samples relative to the full-precision model, and sample a pseudo label for each pseudo sample.

[0166] The first target loss function summation module 204 is used to input the multiple pseudo samples into the full-precision model, calculate the first target loss function, and update the pixel value of each pseudo sample based on the first target loss function.

[0167] The second objective loss function calculation module 205 is used to determine the second objective loss function based on the divergence between the full-precision model and each of the models to be quantized, as well as the divergence between each pair of the models to be quantized.

[0168] The third objective loss function calculation module 206 is used to calculate the third objective loss function corresponding to each of the quantized models based on the difference between the pseudo label of the optimized pseudo sample and the output result after inputting the optimized pseudo sample into each of the quantized models.

[0169] The fourth objective loss function summing module 207 is used to sum the second objective loss function and the third objective loss function to obtain the fourth objective loss function, and update each of the models to be quantized based on the fourth objective loss function to obtain multiple high-precision quantization models.

[0170] In a preferred embodiment, the first objective loss function summing module 204 includes:

[0171] The batch normalization layer matching module is used to determine the first loss function using the feature parameter information in the batch normalization layer of the full-precision model;

[0172] The intra-class heterogeneity calculation module is used to determine the second loss function based on the intra-class heterogeneity between samples of the same class in the training samples during the actual model training process.

[0173] The inter-class homogeneity calculation module is used to determine the third loss function based on the inter-class homogeneity between samples of different classes in the training samples during the actual model training process.

[0174] The cross-entropy calculation module is used to determine the fourth loss function based on the difference between the output of the full-precision model and the pseudo-labels corresponding to the pseudo-samples.

[0175] Each embodiment in this specification focuses on the differences from other embodiments. For the same or similar parts between the embodiments, please refer to each other.

[0176] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0177] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0178] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0180] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0181] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0182] The above provides a detailed description of a neural network quantization method based on multi-model joint learning. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A neural network quantization method based on multi-model joint learning, characterized in that, The method includes: Obtain a pre-trained full-precision model and multiple models to be quantized with different bit widths; Based on the full-precision model, the multiple quantization models with different bit widths are quantized to obtain multiple high-precision quantization models with different bit widths that match the full-precision model. The high-precision quantization models are obtained through the following steps: S1: For the full-precision model, noise samples are initialized using a Gaussian distribution to generate multiple pseudo-samples relative to the full-precision model, and a pseudo-label is sampled for each pseudo-sample; S2: Input the pseudo-sample into the full-precision model, calculate the first target loss function, and optimize the pixel value of the pseudo-sample based on the first target loss function; S3: Determine the second objective loss function based on the divergence between the full-precision model and each of the models to be quantized, as well as the divergence between each pair of the models to be quantized; S4: Based on the difference between the pseudo-label of the optimized pseudo-sample and the output result after inputting the optimized pseudo-sample into each of the models to be quantized, calculate the third objective loss function corresponding to each of the models to be quantized. S5: Summing the second objective loss function and the third objective loss function to obtain the fourth objective loss function, and updating each of the models to be quantized based on the fourth objective loss function to obtain multiple high-precision quantization models; The process includes, before inputting the plurality of pseudo-samples into the full-precision model, the following steps: The original feature image corresponding to each pseudo-sample is randomly cropped according to a first preset probability and a preset scaling rate to obtain a pseudo-sample that possesses some features of the original feature image; or, The pixel values ​​in any feature channel of the original feature image corresponding to each pseudo-sample are enhanced or weakened according to a second preset probability to enrich the brightness and chromaticity information of the original feature image. The plurality of quantization models with different bit widths include a plurality of quantization models with a first bit width and a plurality of quantization models with a second bit width. Wherein, the full-precision model is defined as a teacher model to guide the parameter updates of the plurality of first-bit-width quantized models defined as student models and the plurality of second-bit-width quantized models defined as student models; and, The first bit-width model to be quantized is defined as the teacher model to guide the parameter updates of the plurality of second bit-width models to be quantized, wherein the first bit-width is greater than the second bit-width.

2. The neural network quantization method based on multi-model joint learning according to claim 1, characterized in that, The first target loss function is calculated through the following steps: The first loss function is determined using the feature parameter information in the batch normalization layer of the full-precision model. The second loss function is determined based on the intra-class heterogeneity among samples of the same category in the training samples during the actual model training process. The third loss function is determined based on the inter-class homogeneity of samples of different categories in the training samples during the actual model training process. The fourth loss function is determined based on the difference between the output of the full-precision model and the pseudo-labels corresponding to the pseudo-samples; The first target loss function is obtained by summing the first loss function, the second loss function, the third loss function, and the fourth loss function; The fourth loss function is obtained by the following formula: Among them, Represents the cross-entropy loss function. This represents the output of the full-precision model. This indicates a pseudo-tag.

3. The neural network quantization method based on multi-model joint learning according to claim 2, characterized in that, The step of determining the first loss function using the feature parameter information in the batch normalization layer of the full-precision model includes: The multiple pseudo-samples are sequentially input into all batch normalization layers of the full-precision model, and the mean and variance of the multiple pseudo-samples in each batch normalization layer are calculated. The first loss function is determined based on the difference between the mean of the multiple pseudo-samples calculated in each batch normalization layer and the mean of each batch normalization layer, and the difference between the variance of the multiple pseudo-samples calculated in each batch normalization layer and the variance of each batch normalization layer. The first loss function is shown in the following formula: Among them, and These represent the mean and variance of the pseudo-samples calculated in the l-th batch normalization layer, respectively. and These represent the mean and variance stored in the l-th batch normalization layer, respectively. Representing the eigenvector The square of the norm.

4. The neural network quantization method based on multi-model joint learning according to claim 2, characterized in that, The second loss function is obtained by the following formula: in ， This serves as the lower bound for the intra-class heterogeneity distance constraint. This is the upper bound of the intra-class heterogeneity distance constraint; Dis() It is a function that measures the similarity between two feature vectors; This represents the feature vector of a pseudo-sample after passing through the last feature extraction layer of the full-precision model. The class center represents the average value of all samples within a class.

5. The neural network quantization method based on multi-model joint learning according to claim 2, characterized in that, The third loss function is obtained by the following formula: in, This serves as the lower bound of the inter-class homogeneity distance constraint. This serves as the upper bound for the inter-class homogeneity distance constraint. Dis() It is a function that measures the similarity between two feature vectors; This represents the feature vector of a pseudo-sample after passing through the last feature extraction layer of the full-precision model. This represents the average class center of all different classes of samples. The average class center is the center point obtained by averaging the class centers of multiple classes.

6. The neural network quantization method based on multi-model joint learning according to claim 1, characterized in that, The step of determining the second objective loss function based on the divergence between the full-precision model and each of the models to be quantized, as well as the pairwise divergence between each of the models to be quantized, includes: The first divergence is obtained by summing the divergences between the full-precision model and each of the models to be quantized. The second divergence is obtained by summing the divergences between each pair of the models to be quantized. The sum of the first divergence and the second divergence is determined as the second objective loss function, wherein the second objective loss function is shown in the following formula: in and For adaptive loss coefficients; This indicates that the full-precision model as the teacher model is relative to the first... i KL divergence calculation for the model to be quantized, which is used as a student model; This indicates the full-precision model's performance on the optimized pseudo-samples. The output; Indicates the first i The model to be quantized uses pseudo-samples after optimization. The output; This represents the first teacher model in the pairwise unquantified model. i The model to be quantized is relative to the first student model. j KL divergence calculation for each model to be quantized; Indicates the first j The model to be quantized uses pseudo-samples after optimization. The output of .

7. The neural network quantization method based on multi-model joint learning according to claim 6, characterized in that, The adaptive loss coefficient is obtained through the following method: Obtain the output result after inputting the optimized pseudo-sample into the full-precision model, and the output result after inputting the optimized pseudo-sample into the model to be quantized; Based on the output results of inputting the optimized pseudo-samples into the full-precision model and the output results of inputting the optimized pseudo-samples into the model to be quantized, the adaptive coefficients are determined, wherein the adaptive coefficients are calculated using the following formula: in Represented by natural constant e An exponential function with base 0.

8. The neural network quantization method based on multi-model joint learning according to claim 1, characterized in that, The step of calculating the third objective loss function for each of the quantized models based on the difference between the pseudo-labels of the optimized pseudo-samples and the output results after inputting the optimized pseudo-samples into each of the quantized models includes: Obtain the pseudo-labels corresponding to each of the optimized pseudo-samples, and input each of the optimized pseudo-samples into the output results of each of the models to be quantized. Calculate the cross-entropy loss function between the output of each model to be quantized and the pseudo-label of each optimized pseudo-sample, and sum the obtained cross-entropy loss functions to obtain the third objective loss function, which is obtained by the following formula: Among them, Let be the cross-entropy loss function between the i-th model to be quantized and the pseudo-label.

Images