Image classification method based on full-quantization VGG style convolutional neural network

Abstract

The invention relates to an image classification method based on a full-quantization VGG style convolutional neural network, and belongs to the technical field of artificial intelligence and image processing. Firstly, a full-precision image classification network is pre-trained, and a good initialization weight with high precision and friendly quantization is provided for low-bit quantization. And then performing low-bit quantization perception training to obtain a low-bit quantization VGG style convolutional neural network weight with excellent performance. And the low-precision-high-precision digital conversion mapping, the batch regularization layer and the activation layer in the quantized network are fused and merged into lookup table parameters, and the remaining convolutional layer network parameters are converted into real low-precision integer network parameters from floating-point number representation low-pseudo low-precision integers for deployment. And carrying out image classification by using the converted image classification network. According to the method, the performance of the low-bit quantized VGG style convolutional neural network is greatly improved, the image classification accuracy and reasoning speed are improved, and the method has high practical application value.

Description

Technical Field

[0001] The present invention relates to an image classification method based on a fully quantized VGG style convolutional neural network, and in particular to an image classification method based on a lookup table to implement fully quantized reasoning of a VGG style convolutional neural network, and belongs to the technical field of artificial intelligence and image processing. Background Art

[0002] Image classification is an important image understanding task, which aims to determine the specific category of an input image. In recent years, deep neural networks (DNNs) have made significant progress in computer vision, natural language processing, and speech recognition, and have also achieved remarkable results in image classification tasks. Thanks to the increasingly effective network structure and the increasing amount of computation and parameters, the performance of image classification algorithms based on deep neural networks has been continuously improved. However, during the use phase, the growing demand for computing power of these algorithms has forced them to be deployed on high-performance computing devices (such as GPU and FPGA servers) with very high energy consumption. Many practical application scenarios can only provide very limited computing power and energy, such as application scenarios such as autonomous driving, always-on devices, and mobile devices. Therefore, it is very important to design an image classification algorithm with a simple structure and lower energy consumption, while maintaining the performance of the most advanced algorithm, for the practical application of image classification technology.

[0003] Existing research on image classification algorithms and efficient deep neural network inference techniques mainly includes network search, network pruning and network quantization. Among them, network quantization is the most direct and effective way. Its purpose is to reduce the bit width of DNN weights and activation values, and use lower precision integers or even binary exclusive OR operations (XNOR) for DNN inference, thereby significantly reducing the hardware resource overhead during network model inference, saving memory usage and reducing energy consumption in the inference phase. Uniform quantization is a commonly used quantization method that uniformly maps full-precision values ​​to integer representations with the same step size. DoReFa-Net uses quantized gradients for network quantization. LSQ uses back-propagated gradients to learn quantization steps. Some existing works have also considered applying non-uniform mapping quantization. LQ-Nets obtain non-uniform quantization levels by learning a set of floating-point values. Miyashita uses power of two (PoT) values ​​to represent weights and activations. Li further proposed additive power of two (APoT) to better match the data distribution. Recently, Wang proposed learning lookup tables as quantizers. These efforts enable DNNs to work at very low bits with even better performance. However, these network quantization efforts usually focus on quantizing convolutional and fully connected layers, because they occupy the vast majority of the computational workload during DNN inference, while their computational operations are still calculated at full precision, such as batch normalization, residual, first and last layers, etc. This problem requires the use of high-precision computing hardware when deploying image classification networks, and introduces data precision conversion issues, which to some extent offsets the image classification efficiency advantages brought by quantization.

[0004] At present, the second shortcoming of image classification algorithm research is that it rarely considers the joint design with network architecture and network quantization. Instead, they usually apply quantization algorithms to existing classic networks for testing, such as VGG and ResNet. This leads to two problems. First, many state-of-the-art networks are designed to strive for the best performance in the full precision domain. Although these networks have very efficient network architectures and very advanced performance, and the amount of computation and the amount of parameters are balanced to a certain extent during the design process, their more complex network structure and data flow also lead to additional hardware resource overhead. Related studies have shown that many efficient network architectures may experience performance degradation after quantization, such as EfficientNet and MobileNets. This quantization collapse problem limits the performance of image classification algorithms in scenarios with high energy efficiency requirements.

[0005] Recently, reparameterization techniques have provided a new design perspective for improving the efficiency of image classification algorithms. Reparameterization introduces structural priors through reparameterization to help simple network modules learn optimal parameters. It simplifies the network structure during inference and reduces computational overhead with the same performance. Simple reparameterization strategies usually use BN folding and convolution fusion strategies. RepVGG enhances the performance of VGG-type networks through structural priors and retains only convolution and activation functions for inference. This greatly simplifies the model structure, improves the friendliness to computing devices, and takes reparameterization to a new level. MobileOne extends the reparameterized network to mobile devices, achieving advanced performance with millisecond latency. Recently, reparameterization techniques for network structures such as CNN, MLP, and Transformer have been widely explored. These strategies have been widely used in low-level vision tasks and various high-level vision tasks.

[0006] Although the reparameterization technique brings simplified network structure and advanced performance, it has the same quantization collapse problem as many efficient image classification algorithms. Some works have tried to quantize RepVGG to INT8, but it is still difficult to perform lower bit quantization. In addition, the reparameterized RepVGG no longer has batch normalization, which makes it difficult to apply the quantization-aware training (QAT) method; it is very difficult to add BN to the reparameterized network without compromising performance. The quantization of the reparameterized network is still an urgent problem to be solved. Summary of the invention

[0007] The purpose of the present invention is to creatively propose an image classification method based on a fully quantized VGG style convolutional neural network in view of the shortcomings and defects of the existing technology. The present invention improves the strategy of using reparameterization to enhance the network quantization performance, greatly improves the performance of the VGG style image classification network after low-bit full quantization, and improves the accuracy and reasoning speed of image classification.

[0008] This method mainly includes a pre-training full-precision image classification network stage, a low-bit quantization perception training stage, a classification model conversion stage and a classification model deployment stage.

[0009] In the pre-training full-precision image classification network stage, the improved quantization-friendly reparameterized network is pre-trained to obtain a full-precision image classification network with excellent performance and quantization-friendly. This stage provides good network initial values ​​for subsequent low-bit quantization perception training, helping to improve the network performance after low-bit quantization. Since the existing reparameterized networks have large variances of weights and activation values, although they have very advanced performance, they are not conducive to network quantization and lead to degradation of performance after quantization. A good quantization initialization weight should have two characteristics, one is to have a smaller variance of weights and activation values, and the other is to have a smaller quantization error. In order to solve this problem, the present invention selects an improved quantization-friendly reparameterized network for full-precision pre-training. First, in addition to the original three reparameterized branches, an additional batch normalization layer is added to reduce the heteroscedasticity problem caused by branch addition. Second, the batch normalization layer in the residual branch and the 1×1 branch is removed to avoid extreme values ​​and variance amplification problems caused by the ReLU activation function. This improvement significantly improves the performance of the quantized model.

[0010] In the low-bit quantization-aware training stage, the learning quantization step size (LSQ) algorithm is used to achieve low-bit quantization. Since the re-parameterized network does not retain batch normalization in the inference stage, there are problems of network gradient explosion and gradient vanishing when performing quantization-aware training (QAT) with learned quantization step size, which makes it difficult for the network to converge effectively and leads to the degradation of image classification performance. In order to address this problem, the present invention proposes to add an additional batch normalization layer in the re-parameterized main branch, and the batch normalization layer can be retained in the quantization-aware training stage, thereby effectively stabilizing the gradient during the training process. This strategy greatly improves the performance of the network and enables the network to run at even a 2-bit bit width.

[0011] During the classification model conversion phase, the trained low-bit quantized network is converted into network weight data that can be actually deployed. In order to further accelerate reasoning and completely avoid full-precision calculations during the reasoning phase, we propose to use a lookup table to replace the remaining full-precision operations in the quantized network. These operations include the low-precision-high-precision number conversion mapping generated by the quantization strategy, batch normalization layers, and activation layers. Thanks to rigorous mathematical derivation and code implementation, the converted network model can replace the original computing operations with more efficient operations without any performance loss.

[0012] During the classification algorithm deployment phase, the converted low-bit quantized network is imported into the computing hardware (ASIC) for actual inference calculations. The computing hardware is implemented using FPGA and only runs the implemented perception algorithm with low-precision integers.

[0013] In order to achieve the above object, the present invention adopts the following technical solutions.

[0014] An image classification method based on a fully quantized VGG style convolutional neural network comprises the following steps:

[0015] Step 1: Pre-train a full-precision image classification network to provide good initialization weights for the quantization stage; by constructing a quantization-friendly reparameterized image classification network model and optimizing the reparameterized network structure, quantization-friendly network weights are obtained while maintaining accuracy.

[0016] The reparameterized network introduces structural priors. During training, the original reparameterized module includes a 3×3 convolution branch difference, a 1×1 convolution branch, and a residual branch with batch normalization. Each branch has a batch normalization layer. The operation results of the three branches are finally added together and activated using the ReLU function. Additional batch normalization is added after the three branches, and batch normalization in the residual branch and the 1×1 branch is removed.

[0017] Step 2: The obtained quantization-friendly reparameterized image classification network model is trained using a low-bit quantization-aware training method to obtain a low-precision VGG style image classification network;

[0018] During training, keep the batch normalization of the main branch, and use the lookup table to replace the batch normalization in subsequent deployment;

[0019] Step 3: Calculate the lookup table parameters based on the image classification network parameters;

[0020] Convert the resulting low-precision VGG-style image classification network to a fully quantized image classification network using lookup table inference;

[0021] Step 4: Deploy the converted image classification network to the computing system and use the image classification network to perform image classification. In this process, the network parameters are fixed and no longer updated or converted.

[0022] Beneficial Effects

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] This paper proposes for the first time the use of a lookup table for low-bit reasoning of deep neural networks, and constructs the first low-bit quantized VGG-style convolutional neural network image classification algorithm. This method uses a reparameterization strategy and a stabilized quantization-aware training strategy to greatly improve the performance of low-bit quantized VGG-style convolutional neural networks. At the same time, this paper provides a new idea for the quantization and deployment of neural networks, greatly improving the accuracy and reasoning speed of image classification, and has high practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the process of the present invention;

[0026] Figure 2 It is the overall framework diagram of the present invention;

[0027] Figure 3 This is a schematic diagram of the fully quantized efficient reasoning principle of the present invention;

[0028] Figure 4 is the training curve of the present invention;

[0029] Figure 5 It is a visualization diagram of weights and activation values ​​of the present invention. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0031] like Figure 1 As shown, an image classification method based on a fully quantized VGG style convolutional neural network includes the following steps:

[0032] Step 1: Pre-train a full-precision image classification network. This includes the following steps:

[0033] First, a quantization-friendly reparameterized image classification network model is constructed.

[0034] Pre-trained full-precision image classification networks are used to provide good initialization weights for the quantization stage. A good initialization weight should have two characteristics: one is good accuracy, and the other is quantization-friendly. Reparameterization technology can retain the VGG-style minimalist network structure and maintain unchanged performance during inference, however, this technology has an inherent quantization collapse problem. The current reparameterization technology does not retain batch normalization during quantization, which makes the network have gradient vanishing and gradient exploding problems during quantization-aware training.

[0035] In order to solve the above problems, the present invention obtains quantization-friendly network weights while maintaining accuracy by optimizing the re-parameterized network structure.

[0036] Specifically, the reparameterized network introduces structural priors. During training, each reparameterized module includes multiple branches and multiple convolution and batch normalization operations. These operations are all linear transformations and can be integrated into a single convolution module during inference, thereby reducing the amount of network calculations and speeding up the calculations.

[0037] The original reparameterized module includes a 3×3 convolution branch difference, a 1×1 convolution branch, and a residual branch with batch normalization. Each branch has a batch normalization layer. The results of the three branches are finally added together and activated using the ReLU function. Through batch normalization folding and convolution kernel fusion, the three branches can be converted into a single convolution with almost no performance loss in full precision. However, this structure causes the network weights and activation values ​​to have large variance and large weight heteroscedasticity, and makes them inherently quantized. Collapse problem.

[0038] To solve this problem, the present invention improves the reparameterized structural prior. The addition of the three branches leads to the heteroscedasticity problem, which can be solved by adding additional batch regularization after the three branches. At the same time, adding batch regularization in the residual and 1×1 branches will lead to extreme value and variance amplification problems caused by the previous ReLU activation function. Therefore, the batch regularization in them should be removed. By taking the above two improvements, quantization-friendly initialized full-precision weight parameters can be obtained, which significantly improves the performance of the final quantized network.

[0039] For example, the full-precision model is trained on the ImageNet-1K image classification dataset for 120 cycles, and each iteration uses a small batch of data for gradient descent optimization, and the batch size is set to 256. The SGD optimizer is used, the initial learning rate is set to 0.1, and the learning rate is gradually reduced to 0 according to the cosine curve. After completing the training of the full-precision network, the three branches of the network are folded into a single 3×3 convolution to obtain the deployment state, so as to initialize the quantized network later.

[0040] The conversion process includes convolution and batch normalization fusion strategy and multi-convolution kernel fusion strategy. First consider the batch normalization operation, which is expressed as:

[0041]

[0042] Among them, BN stands for batch normalization, X is the input of BN, μ and σ are the mean and variance of X, respectively, and γ and β are learnable affine transformation parameters. During training, μ and σ are updated using the sliding average strategy (EMA), and γ and β are trained directly through gradients.

[0043] Since batch normalization and convolution are both linear transformations, batch normalization can be integrated into the convolution module, and the final parameters are expressed as:

[0044]

[0045] Among them, W and W′ are the convolution kernels before and after fusion, respectively, and b and b′ are the biases before and after fusion, respectively.

[0046] When fusing multiple convolution kernels of different sizes, the final convolution kernel is the largest one among all the convolution kernels to be fused, and the remaining convolution kernels are aligned to the largest convolution kernel through padding. Finally, the multi-convolution kernel fusion strategy can be expressed as:

[0047]

[0048] in, represents the fused convolution kernel, W′0 represents the unit matrix corresponding to the residual, W′1 represents the fused 1×1 convolution kernel, and W′3 represents the 3×3 convolution kernel before fusion.

[0049] Step 2: Low-bit quantization-aware training: The obtained quantization-friendly reparameterized image classification network model is trained using a low-bit quantization-aware training method to obtain a low-precision VGG-style image classification network using low-precision integer calculations.

[0050] Specifically, to solve the training instability problem caused by not retaining batch normalization during inference of the re-parameterized network, the batch normalization of the main branch is retained in the quantization-aware training phase, and a lookup table is used to replace the batch normalization in subsequent deployment.

[0051] The learning quantization step size method (LSQ) is used to train the network for quantization perception. LSQ is a uniform quantization method that constructs a mapping relationship between floating point numbers and quantized low-bit integers through a linear relationship. For the quantization compensation S, the floating point number F before quantization in the network and the integer I after quantization. The relationship between them can be expressed as:

[0052]

[0053] Among them, Q n =-2 n-1 and Q p =2 n-1 -1, respectively, represents the upper and lower bounds of the value represented by the n-bit integer. Clip(x) is a truncation operation, which truncates x to a floating point number between -1 and 1. Indicates the rounding operation. For an integer I, it can be reversely mapped to the floating point domain and is called a pseudo-quantized floating point number. It is expressed as:

[0054]

[0055] In the process of quantization-aware training, through the above mapping relationship, it is possible to use quantized pseudo-quantized floating-point numbers to simulate integer uniform velocity with floating-point operations. However, since quantization rounding is a non-differentiable operation, there are inherent difficulties in gradient return and network training. A commonly used method is direct gradient return (STE), that is, directly returning the gradient to the variable through the rounding operation.

[0056] Use the full-precision network parameters obtained in step 1 as initial values ​​and keep the batch normalization layer in quantization-aware training.

[0057] In this embodiment, the network is trained for a total of 90 cycles, and a cosine curve is used for learning rate decay, and the initial learning rate is set to 0.01.

[0058] Step 3: Calculate the lookup table parameters based on the image classification network parameters.

[0059] The obtained low-precision VGG-style image classification network is converted into a fully quantized image classification network using lookup table inference, further reducing the computational overhead and hardware resource consumption of the image classification network.

[0060] The VGG-style convolutional neural network has a very simple structure and avoids jump connections during reasoning. In the quantized network, the operations between two low-precision convolutional layers are: converting the activation output of the previous convolutional module from low precision to full precision, BN, ReLU, and converting the activation from full precision to low precision integers, and sending them to the next convolutional module for calculation. Figure 2 As shown in the figure, all these transformations start from low-precision integers and eventually output low-precision integers, especially BN and ReLU need to run at full precision. Therefore, all these non-convolution operations are transformed into lookup table operations to achieve fully quantized neural network inference.

[0061] For the output Y of the previous convolution layer as a low-precision integer, and the Z to be fed into the next convolution layer as a low-precision integer, all transformations between them can be expressed as:

[0062]

[0063] Where LUT is the lookup table operation and Θ is the parameter of the lookup table. z Represents the quantization multiplier of the next convolutional layer activation value, S y Represents the quantization multiplier of the current convolution layer activation value, S w Represents the quantization multiplier of the current convolutional layer weight; M and N represent the quantization multiplier and bias after fusion, respectively. Clip represents.

[0064] like Figure 3As shown in the figure, the lookup table transforms the original four calculations into one simple and fast lookup table operation, and avoids full-precision calculations. Although full quantization inference can be achieved through quantized BN, this method will make training unstable, resulting in accuracy degradation, and the other three calculation operations still need to be retained. The hardware implementation of LUT is very efficient. Especially in specially optimized computing devices, it has significantly lower hardware overhead.

[0065] Step 4: Use the image classification network to perform image classification reasoning and use. In this process, the network parameters are fixed and no longer updated or converted.

[0066] Specifically, the converted image classification network is deployed to a computing system, which can be a server supporting low-precision computing or a dedicated FPGA computing device.

[0067] After the input image data is preprocessed by image cropping and resizing, it is sent to the computing system for image classification calculation and reasoning, and the deep features or category information of the image are output. The accuracy of the network classification can be calculated by comparing the image category information with the true label to evaluate the accuracy of the image classification network.

[0068] The image data to be processed is sent to the computing system for calculation and reasoning to obtain the extracted image depth features or image category information. According to the real label of the image data, the accuracy of the network classification is calculated to determine the final accuracy performance.

[0069] The deep features extracted by the image classification network can also be further applied to other applications such as image re-identification, image search, and abnormal image detection, providing reliable technical support for various applications and having high practical application value.

Claims

1. An image classification method based on a fully quantized VGG-style convolutional neural network, characterized in that: Includes pre-trained full-precision image classification network, low-bit quantization-aware training, classification model conversion, and classification model deployment; Step 1: Pre-train a full-precision image classification network to provide good initialization weights for the quantization stage; by building a quantization-friendly reparameterized image classification network model and optimizing the reparameterized network structure, quantization-friendly network weights are obtained while maintaining accuracy; The reparameterized network introduces structural priors. During training, the original reparameterized module includes a 3×3 convolution branch difference, a 1×1 convolution branch, and a residual branch with batch normalization. Each branch has a batch normalization layer. The operation results of the three branches are finally added together and activated using the ReLU function. Additional batch normalization is added after the three branches, and batch normalization in the residual branch and the 1×1 branch is removed. Step 2: The obtained quantization-friendly reparameterized image classification network model is trained using a low-bit quantization-aware training method to obtain a low-precision VGG style image classification network; During training, keep the batch normalization of the main branch, and use the lookup table to replace the batch normalization in subsequent deployment; Step 3: Calculate the lookup table parameters based on the image classification network parameters; Convert the resulting low-precision VGG-style image classification network to a fully quantized image classification network using lookup table inference; Step 4: Deploy the converted image classification network to the computing system and use the image classification network to perform image classification. In this process, the network parameters are fixed and no longer updated or converted.

2. The image classification method based on a fully quantized VGG style convolutional neural network as claimed in claim 1, characterized in that: In step 1, the batch normalization operation is expressed as: Among them, BN stands for batch normalization, X is the input of BN, μ and σ are the mean and variance of X respectively, γ and β are learnable affine transformation parameters; in training, μ and σ are updated using the sliding average strategy, and γ and β are trained directly through gradients; Batch normalization is integrated into the convolution module, and the final parameters are expressed as: Among them, W and W′ are the convolution kernels before and after fusion, respectively, and b and b′ are the biases before and after fusion, respectively.

3. The image classification method based on a fully quantized VGG style convolutional neural network according to claim 1, characterized in that: In step 1, when multiple convolution kernels of different sizes are fused, the final convolution kernel is the largest one among all the convolution kernels to be fused, and the remaining convolution kernels are aligned to the largest convolution kernel by padding. Finally, the multi-convolution kernel fusion strategy is expressed as: in, represents the fused convolution kernel, W′0 represents the unit matrix corresponding to the residual, W′1 represents the fused 1×1 convolution kernel, and W′3 represents the 3×3 convolution kernel before fusion.

4. The image classification method based on a fully quantized VGG style convolutional neural network according to claim 1, characterized in that: In step 2, the network is trained for quantization awareness by learning the quantization step size. For the quantization compensation S, the floating point number F before quantization and the integer I after quantization in the network are expressed as: Among them, Q n =-2 n-1 and Q p =2 n-1 -1, respectively, represents the upper and lower bounds of the value represented by the n-bit integer; Clip(x) is a truncation operation, which truncates x to a floating point number between -1 and 1; Indicates rounding operation; For integer I, it can be reversely mapped to the floating point domain and is called pseudo-quantized floating point number It is expressed as: In the process of quantization-aware training, the above mapping relationship is used to use pseudo-quantized floating-point numbers after quantization to simulate integer uniform speeds with floating-point operations; at the same time, the gradient is directly returned to the variable through rounding operations.

5. The image classification method based on a fully quantized VGG style convolutional neural network as claimed in claim 1, characterized in that: In step 3, all non-convolution operations are transformed into lookup table operations to achieve fully quantized neural network reasoning; The output Y of the previous convolution layer is a low-precision integer, and the Z to be fed into the next convolution layer is also a low-precision integer. All transformations between them are expressed as: Where LUT is the lookup table operation, Θ is the parameter of the lookup table; S z Represents the quantization multiplier of the next convolutional layer activation value, S y Represents the quantization multiplier of the current convolution layer activation value, S w represents the quantization multiplier of the current convolutional layer weight; M and N represent the quantization multiplier and bias after fusion respectively; ReLU is the activation function; clip represents; BN represents batch normalization, X is the input of BN, μ and σ are the mean and variance of X respectively, γ and β are learnable affine transformation parameters; Q n =-2 n-1 and Q p =2 n-1 -1, indicating the upper and lower bounds of the value represented by the n-bit integer.

6. The image classification method based on a fully quantized VGG style convolutional neural network according to claim 1, characterized in that: In step 4, the converted image classification network is deployed to the computing system; The image data to be processed is sent to the computing system for calculation and reasoning to obtain the extracted image depth features or image category information; based on the true label of the image data, the accuracy of the network classification is calculated to determine the final accuracy performance.

7. In the image classification method based on the fully quantized VGG style convolutional neural network as described in claim 6, the computing system is a server that supports low-precision computing.

8. In the image classification method based on a fully quantized VGG style convolutional neural network as described in claim 6, the computing system is an FPGA computing device.

Images