Convolutional neural network-oriented collaborative sparse quantization method

Abstract

The invention discloses a collaborative sparse quantization method for a convolutional neural network, and the method comprises the three steps: carrying out the theoretical analysis of a collaborative sparse quantization error, and carrying out the main analysis of the collaborative sparse quantization error; the step 2 is a collaborative sparse method based on channel correlation, and the part is dominant; the step 3 is a block logarithm domain mapping average quantization method, and the step 3 is mainly used.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of neural network lightweight, and particularly relates to a convolutional neural network-oriented collaborative sparse quantization method. BACKGROUND

[0002] The convolutional neural network has achieved great breakthroughs in multiple fields including target detection, semantic segmentation, target tracking and the like in remote sensing tasks. However, the superlative effect of the neural network is often accompanied by huge computing power and hardware resource overhead. For the on-board and airborne edge devices, considering the strict volume, power consumption, mass limitation and limited computing power, it is still quite challenging to deploy the convolutional neural network.

[0003] In recent years, a large number of related researches have been made from the aspects of sparse pruning and quantization. For model sparse pruning, although the common unstructured sparse pruning technology can make the model have a high sparsity, the irregular parameters and input features lead to the fact that this way is not hardware-friendly. Therefore, the common edge device deployment usually uses the structured pruning way to prune the model in the channel dimension and the like by using the statistical way, so as to reduce the parameter quantity and operation quantity of the model. For model quantization, the common quantization way usually uses the symmetric quantization way to quantize the parameters and input features, and adopts the maximum value of the input and feature to scale. However, this quantization way often causes the model accuracy to degenerate due to the outliers in the input and parameters. Therefore, the common quantization way can process the outliers by setting a threshold according to the data distribution, using an activation function with an upper bound and the like. In addition, the combination of pruning and quantization is also an effective means for model lightweight. However, researches in recent years show that pruning and quantization are usually not orthogonal, so the combination of the two usually causes the model accuracy to decrease. The specific application method of sparse pruning and quantization still has the research significance and practical value of exploration. SUMMARY

[0004] To address the issue of additional errors introduced by existing sparse pruning and quantization methods when applied to convolutional neural networks (CNNs), this invention proposes a structured collaborative sparse quantization method. This method compresses CNNs from three perspectives: error theory analysis, a structured sparsity construction strategy, and a logarithmic block quantization method design. From the error theory analysis perspective, this method achieves both sparsity and quantization simultaneously through training, relying on a loss function and optimizer to achieve optimal model performance during backpropagation. The structured sparsity construction strategy differs from traditional methods by selecting discardable data from all pixels of the convolutional kernel parameters through training rather than statistical analysis. Leveraging the correlation between parameters across different dimensions, independent sparse kernel parameters are averaged at a certain granularity along the channel dimension and mapped to a mask using a Heaviside function to ensure the structured characteristics of the pruned kernel parameters. From the perspective of the logarithmic block quantization method design, to address outlier issues, this method statistically calculates the scaling weights for each output channel, averages them in the logarithmic domain at a certain granularity, and then maps them back to the original domain as scaling values, effectively mitigating the impact of outliers on quantization.

[0005] The method of this invention is achieved through the following technical solution: A cooperative sparse quantization method for convolutional neural networks includes: Step 1: Error Theory Analysis: Based on mathematical derivation, the errors of sequential sparse quantization and cooperative sparse quantization are analyzed and compared.

[0006] The convolution operation process can be represented by the following formula: ; in and , representing the input and output features respectively. and These represent the number of channels, width, and height of the input and output features, respectively.

[0007] The error obtained by applying sparsity and quantization methods to this process is shown below: ; Where W and M represent the original weight matrix and the original sparse matrix that are iteratively updated during training, respectively. , , These represent the errors introduced by quantization, sparsity, and their combinations, respectively.

[0008] Based on the above formula, the proposed sparsity and quantization strategies jointly influence the final error during training. To mitigate this error, the proposed structured collaborative sparse quantization method employs a backpropagation mechanism, automatically optimizing the error through the co-operation of the optimizer and the loss function. Backpropagation during training can be expressed as: ; ; ; in, This represents the loss function, which directly updates the sparse kernel. and weight kernel Clearly, the above process effectively reduces... , and These three types of errors thus optimize Furthermore, it is worth noting that since sparsity and quantization are performed during the same training process, they can influence each other and optimize synergistically, achieving better performance compared to methods that apply sparsity and quantization independently.

[0009] Regarding the impact of sparsity on quantization, the sparse kernel M directly filters out certain elements in the weight kernel W, ensuring that only when... hour, Only then will it be updated. This process is similar to dropout, which effectively prevents overfitting and thus enhances the robustness of the model.

[0010] Regarding the impact of quantization on sparsity, during backpropagation, each element in the weight kernel... They can all directly participate in the corresponding sparse kernels The update mechanism effectively alters the structure of the sparse kernel, allowing the sparsity rate to adaptively change according to the model's accuracy during training.

[0011] The second step is to construct a structured sparsity strategy: the parameters should be mapped based on the correlation of the network parameter input channels to ensure that the model parameters have fine-grained structured sparsity characteristics.

[0012] To ensure This paper proposes a structured sparsity construction strategy that combines structured sparsity with trainability. It utilizes the inter-channel correlation of convolutional kernel parameters to construct sparse kernels. This is a feasible method. To ensure... To address the structured sparsity, this paper proposes constraining the sparse kernel from the channel dimension during forward inference propagation. This process can be represented as: ; in, Indicates sparsity granularity. It is an iterator that traverses all granularities. and These represent the sparse matrices before and after pruning, respectively.

[0013] As shown in the formula above, the sparse matrix is ​​averaged along the input channel dimension with adjustable granularity. Therefore, during forward inference, the sparse matrix values ​​within the same granularity are identical, exhibiting a structured characteristic. Furthermore, it is assumed that the sparse matrix... Following a normal distribution, and because the proposed structuring strategy is linear, the processed sparse matrix... It will still follow a normal distribution.

[0014] After implementing structured features, further processing of the sparse matrix is ​​needed to facilitate pruning and training. Therefore, this paper introduces the Heaviside function to map floating-point numbers to 0 and 1. The mapping process in forward propagation is as follows: ; in, It's the Heaviside function. It is the sparse kernel that ultimately exhibits structured sparsity.

[0015] As shown in the formula above, this mapping strategy effectively ensures Furthermore, the sparsity of the sparse matrix is ​​addressed by introducing a direct-through estimator (STE) method during backpropagation. This means that the sparse matrix is ​​obtained in floating-point form through training and reaches a local optimum based on the loss function and optimizer, thus effectively solving the problem that common pruning methods rely on complex search state space exploration.

[0016] Step 3: Design of logarithmic domain block quantization method: Based on the convex property of the logarithmic function, the influence of parameter outliers on the accuracy of the quantized model is eliminated by the logarithmic domain mapping average of the quantization weights.

[0017] Common Quantization-Aware Training (QAT) methods often face the problem of outliers in the parameters causing performance degradation. To address this issue, a logarithmic block quantization method is proposed. This method leverages the inter-channel parameter influence to divide several channels in each neural network layer into a granular unit, and calculates the final scaling value based on the scaling value of the channels within each granular unit. This process can be represented as: ; ; in This represents all scaling values ​​for a given layer. This represents the scaling value for the i-th granularity. Indicates the granularity of quantization. This represents the scaling value of the j-th channel within the granularity.

[0018] This quantization method maps the scaling values ​​of different channels within the same granularity to the logarithmic domain, averages them, and then maps them back to the original domain. Utilizing the gradually slowing growth of the logarithmic function, this method effectively mitigates the impact of a few outliers in the neural network parameters on the quantization scaling values. Furthermore, to facilitate co-training with sparse kernels, the STE method is also introduced during the backpropagation process of the weight kernels.

[0019] Beneficial effects: The cooperative sparse quantization method for convolutional neural networks proposed in this invention has the following beneficial effects: 1. It can simultaneously perform sparsity and quantization during training, reducing the training time and resource overhead of lightweight model compression; 2. Compared with traditional methods, it can achieve a compression ratio of more than ten times for convolutional neural network models with less reduction in detection accuracy, and more efficiently complete the lightweight compression of neural networks. Attached Figure Description

[0020] Figure 1 Flowchart of the algorithm of this invention; Figure 2 A schematic diagram of the structured sparsity construction strategy; Figure 3 A schematic diagram of the logarithmic field block quantization method; Figure 4 A schematic diagram of the algorithm's backpropagation; Figure 5 Actual test results. Detailed Implementation

[0021] The embodiments of the method of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] A collaborative sparse quantization method for convolutional neural networks, with the following workflow: Figure 1 As shown, it includes: Step 1: Error Theory Analysis The convolution operation process can be represented by the following formula: ; in and , representing the input and output features respectively. and These represent the number of channels, width, and height of the input and output features, respectively.

[0023] The error obtained by applying sparsity and quantization methods to this process is shown below: ; Where W and M represent the original weight matrix and the original sparse matrix that are iteratively updated during training, respectively. , , These represent the errors introduced by quantization, sparsity, and their combinations, respectively.

[0024] Based on the above formula, the proposed sparsity and quantization strategies jointly influence the final error during training. To mitigate this error, the proposed structured collaborative sparse quantization method employs a backpropagation mechanism, automatically optimizing the error through the co-operation of the optimizer and the loss function. Backpropagation during training can be expressed as: ; ; ; in, This represents the loss function, which directly updates the sparse kernel. and weight kernel Clearly, the above process effectively reduces... , and These three types of errors thus optimize Furthermore, it is worth noting that since sparsity and quantization are performed during the same training process, they can influence each other and optimize synergistically, achieving better performance compared to methods that apply sparsity and quantization independently.

[0025] Regarding the impact of sparsity on quantization, the sparse kernel M directly filters out certain elements in the weight kernel W, ensuring that only when... hour, Only then will it be updated. This process is similar to dropout, which effectively prevents overfitting and thus enhances the robustness of the model.

[0026] Regarding the impact of quantization on sparsity, during backpropagation, each element in the weight kernel... They can all directly participate in the corresponding sparse kernels The update mechanism effectively alters the structure of the sparse kernel, allowing the sparsity rate to adaptively change according to the model's accuracy during training.

[0027] Step 2: Structured Sparsity Construction Strategy After error analysis, to ensure This paper proposes a structured sparsity construction strategy that combines structured sparsity with trainability. It utilizes the inter-channel correlation of convolutional kernel parameters to construct sparse kernels. This is a feasible method. To ensure... To address the structured sparsity, this paper proposes constraining the sparse kernel from the channel dimension during forward inference propagation. The process is as follows: Figure 2 As shown, it can be formalized as follows: ; in, Indicates sparsity granularity. It is an iterator that traverses all granularities. and These represent the sparse matrices before and after pruning, respectively.

[0028] As shown in the formula above, the sparse matrix is ​​averaged along the input channel dimension with adjustable granularity. Therefore, during forward inference, the sparse matrix values ​​within the same granularity are identical, exhibiting a structured characteristic. Furthermore, it is assumed that the sparse matrix... Following a normal distribution, and because the proposed structuring strategy is linear, the processed sparse matrix... It will still follow a normal distribution.

[0029] After implementing structured features, further processing of the sparse matrix is ​​needed to facilitate pruning and training. Therefore, this paper introduces the Heaviside function to map floating-point numbers to 0 and 1. The mapping process in forward propagation is as follows: ; in, It's the Heaviside function. It is the sparse kernel that ultimately exhibits structured sparsity.

[0030] As shown in the formula above, this mapping strategy effectively ensures The sparsity of the data. Furthermore, a straight-through estimator (STE) method is introduced during backpropagation, such as... Figure 4 As shown, this means that the sparse matrix will be obtained in floating-point form through training and will reach a local optimum based on the loss function and optimizer, thus effectively solving the problem that common pruning methods rely on complex search state space exploration.

[0031] Step 3: Logarithmic Block Quantization Method Common Quantization-Aware Training (QAT) methods often face the problem of outliers in the parameters causing performance degradation. To address this issue, a logarithmic block quantization method is proposed. This method leverages the inter-channel parameter influence to divide several channels in each neural network layer into a granular unit, and calculates the final scaling value based on the scaling value of the channels within each granular unit. The process is as follows: Figure 3 As shown, it can be formalized as follows: ; ; in This represents all scaling values ​​for a given layer. This represents the scaling value for the i-th granularity. Indicates the granularity of quantization. This represents the scaling value of the j-th channel within the granularity.

[0032] like Figure 3 The quantization method shown maps the scaling values ​​of different channels within the same granularity to the logarithmic domain, averages them, and then maps them back to the original domain. Utilizing the gradually slowing growth of the logarithmic function, this method effectively mitigates the impact of a few outliers in the neural network parameters on the quantization scaling values. Furthermore, to facilitate co-training with sparse kernels, the STE method is also introduced during the backpropagation of the weight kernels. Figure 4 As shown.

[0033] Example To verify the proposed collaborative sparse quantization method for convolutional neural networks, data validation was performed.

[0034] In the object detection task, experiments were conducted using the NWPUVHR10 and DOTA-v1 datasets. The NWPUVHR10 dataset was used in an 8:2 ratio for experiments, while the official training and validation sets of the DOTA-v1 dataset were also used. In the object detection task, all images were resized to 640×640 pixels.

[0035] Regarding model selection, YOLOv5s was used for the detection task. Considering that pruning and quantizing specific structures of Convolutional Neural Networks (CNNs) models can significantly impact model performance, the application scope of collaborative sparse quantization methods is limited. For the detection task, the proposed collaborative sparse quantization method prunes and W4A8 quantizes the backbone and head networks, but does not prune or quantize the first and last layers on the NWPUVHR10 and DOTA-v1 datasets. It is worth noting that since many previous studies only implemented model pruning, this paper uses the proposed quantization method to quantize the pruned model for comparison. When the granularity is not divisible by the number of input channels of a layer, the granularity is set to be consistent with the number of channels.

[0036] During model training, for the object detection task, all hyperparameter settings, except for the batch size being set to 32, were consistent with the official YOLOv5 implementation. The training process consisted of a total of 150 training epochs. In both the sparsity-only and co-sparse-quantization cases, the model was trained directly for 150 epochs starting from floating-point parameters to obtain the final result. To allow the sparse matrix in the CTSQ method to transition better near zero, the initialization formula for the sparse matrix is ​​as follows: ; in, Indicates a Gaussian distribution. This indicates the number of parameters in a sparse matrix.

[0037] The performance metrics and comparisons of the YOLOv5s network on the NWPUVHR10 and DOTA-v1 datasets are shown in Tables 1 and 2. The method proposed in this invention achieves a compression ratio of 13.81 times on the DOTA-v1 dataset. Some detection results are shown below. Figure 5 As shown.

[0038]

[0039] Table 1. Experimental results of the cooperative sparsity quantization method for convolutional neural networks based on YOLOv5s on the NWPUVHR10 dataset.

[0040] Table 2. Experimental results of the cooperative sparsity quantization method for convolutional neural networks based on YOLOv5s on the DOTA-v1 dataset. The results in Tables 1 and 2, and the comparison with previous methods, show that the collaborative sparse quantization method for convolutional neural networks proposed in this invention can effectively compress the model size while maintaining the accuracy of the detection model.

[0041] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cooperative sparse quantization method for convolutional neural networks, characterized in that, include: Step 1: Error Theory Analysis: Based on mathematical derivation, the errors of sequential sparse quantization and cooperative sparse quantization are analyzed and compared. The second step, the structured sparsity construction strategy, is to map the parameters based on the correlation of the network parameter input channels to ensure that the model parameters have fine-grained structured sparsity characteristics. Step 3: Design of logarithmic domain block quantization method: Based on the convex property of the logarithmic function, the influence of parameter outliers on the accuracy of the quantized model is eliminated by the logarithmic domain mapping average of the quantization weights.

2. The collaborative sparse quantization method for convolutional neural networks as described in claim 1, characterized in that, In the first step, the convolution operation uses sparsity and quantization methods, and the resulting error is shown below: ; Where W and M represent the original weight matrix and the original sparse matrix that are iteratively updated during training, respectively. , , These represent the errors introduced by quantization, sparsity, and their combinations, respectively.

3. The collaborative sparse quantization method for convolutional neural networks as described in claim 1, characterized in that, In the first step, the backpropagation during training can be described as follows: ; ; ; in, This represents the loss function, which directly updates the sparse kernel. and weight kernel The above process effectively reduces , and These three types of errors thus optimize ; Regarding the impact of sparsity on quantization, the sparse kernel M directly filters out certain elements in the weight kernel W, ensuring that only when... hour, It will only be updated then; Regarding the impact of quantization on sparsity, during backpropagation, each element in the weight kernel... They can all directly participate in the corresponding sparse kernels Update.

4. The collaborative sparse quantization method for convolutional neural networks as described in claim 1, characterized in that, In the second step, the sparse kernel is constrained from the channel dimension during the forward inference propagation process. This process is formulated as follows: ； in, Indicates sparsity granularity. It is an iterator that traverses all granularities. and These represent the sparse matrices before and after pruning, respectively.

5. The collaborative sparse quantization method for convolutional neural networks as described in claim 1, characterized in that, In the third step, the logarithmic field block quantization method is formulated as follows: ； ； in This represents all scaling values ​​for a given layer. This represents the scaling value for the i-th granularity. Indicates the granularity of quantization. This represents the scaling value of the j-th channel within the granularity.

Images