A cnn fast pruning method for target detection based on filter weight clustering

Abstract

The application provides a CNN fast pruning method for target detection based on filter weight clustering, which comprises a target detection model initialization step; an importance score exponential moving average processing step to obtain a sliding average value of the importance score of each filter; a threshold calculation step: calculating the importance score of the convolution layer, and then taking the convolution layer with the lowest importance score as the target convolution layer for this pruning; calculating the variance of the importance score of each filter in the target convolution layer to obtain the importance score threshold of the target convolution layer; a pruning step: clustering the filter parameter vector of the target convolution layer; in each cluster of the target convolution layer, pruning the filter with the lowest sliding average value of the importance score and the sliding average value of the importance score lower than the importance score threshold. The application improves the efficiency of the convolutional neural network filter pruning for target detection under the condition of ensuring that the overall pruning effect is not reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the CNN technology for object detection, in particular to the pruning technology of CNN for object detection. BACKGROUND

[0002] In recent years, deep learning related technologies are widely used in various computer vision tasks. Convolutional neural networks have many advantages such as being friendly to image form input, being able to be converted into matrix multiplication operation to facilitate optimization on various different architecture processors, etc. Therefore, in many basic tasks such as object detection and tracking, OCR, behavior recognition, etc., many commonly used methods, especially the neural network models running on edge devices, use convolutional neural networks as feature extraction backbone network, various task heads, etc. In the information age, smart phones, smart wearable devices and unmanned vehicles, etc. terminals need deep learning related technologies to perform environment perception, autonomous decision making and other tasks. Whether it is to perform computation on a cloud server through network communication or to perform neural network model inference on the side, it needs huge computing overhead. Neural network model compression technology can reduce the computational amount, memory occupation and demand for memory bandwidth of the model while keeping the accuracy of the original model basically unchanged. On the cloud server, it means that the same hardware resources can support higher QPS, and on the end-side device, it means lower power consumption and shorter latency.

[0003] As a method for compressing convolutional neural networks for object detection, the core problem of filter pruning is the importance evaluation method of each filter. Existing pruning algorithms mostly solve this problem based on loss function and gradient, similarity, channel attention, etc.

[0004] Although a large number of studies have focused on this core problem, the time complexity of the pruning method itself is often ignored. At present, many pruning methods adopt the form of three stages of training, pruning and fine-tuning for model compression. There are also some pruning methods that choose to perform training and pruning at the same time in order to save overall computing overhead, but pruning the model that has not been trained obviously lacks cognitive information about the target task, although strategies such as filter recovery can be proposed to make up for it, but it will inevitably affect the performance of the entire object detection network. SUMMARY

[0005] The technical problem to be solved by the present application is that the information globality and pruning efficiency are difficult to be considered at the same time in the pruning method, and a fast pruning method for convolutional neural networks for object detection based on exponential moving average and filter parameter clustering is provided.

[0006] The application adopts the technical scheme of a CNN fast pruning method for target detection based on filter weight clustering to solve the above technical problems, and comprises the following steps:

[0007] A target detection model initialization step: a target detection model is trained using a target data set to obtain a target detection model that has not undergone pruning;

[0008] An exponential moving average processing step of importance scores: a fixed pruning interval is set, and the importance scores of each filter in each convolution layer of the target detection model are calculated in each iteration; then, the exponential moving average method is used to accumulate the moving average values of the importance scores of each filter from the start of pruning to the present, and the moving average value of the importance scores obtained in the last iteration that meets the set pruning interval is output to the importance score threshold value calculation step of pruning the convolution layer;

[0009] An importance score threshold value calculation step of pruning the convolution layer: the moving average values of all filters in each convolution layer are added and averaged to obtain the importance score of the convolution layer, and the convolution layer with the lowest importance score is taken as the target convolution layer for pruning this time; the variance of the importance scores of each filter in the target convolution layer is calculated; and the result obtained by subtracting the variance weighted by the hyperparameter from the importance score of the convolution layer is taken as the importance score threshold value of the target convolution layer;

[0010] A pruning step: the filter parameter vector of the target convolution layer is clustered; in each cluster of the target convolution layer, the filter with the lowest moving average value of the importance score and the moving average value of the importance score lower than the importance score threshold value is pruned; and then, it is judged whether the overall sparsity degree of the neural network of the current target detection model reaches the expectation, and if yes, the current target detection model is used for target detection, and if not, the exponential moving average processing step of the importance scores is returned.

[0011] The application has the beneficial effects that the global importance score information is accumulated using the exponential moving average method, the overall pruning efficiency is improved using the clustering method, and the efficiency of pruning the filters of the convolutional neural network for target detection is improved under the condition that the overall pruning effect is not reduced. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 The figure is a flowchart of the embodiment. DETAILED DESCRIPTION

[0013] The applicant has found that in the existing global filter pruning method, the basis for pruning each time is usually the average importance score of the first 10 mini-batches. This processing method can ensure that pruning can be performed at a high frequency throughout the training process, but it does not fully utilize the information of the entire data set, because the least important filter can be different for different batches of samples.

[0014] The embodiment is implemented on a server equipped with an NVIDIA Titan X Pascal GPU, and is improved based on the paper Group Fisher Pruning for Practical Network Compression published in the ICML conference in 2021. The experimental object of the pruning method is the FCOS target detection model. Since the grouping regularization layer in the detection head of the model does not support pruning, the layer needs to be removed first, which has a slight impact on the original detection performance. The data set uses the target detection data set of the laboratory.

[0015] First, the exponential moving average importance score processing method in the application is described. Specifically, assuming that the original importance score of the filter f calculated at the tthiteration is score(f, t), the importance score Score ori (f, t) obtained by processing score(f, t) when pruning according to the existing method is:

[0016]

[0017] The exponential moving average processing method for score(f, t) proposed in the application is:

[0018] Score EMA (f, t) = a x score(f, t) + (1-a) x Score EMA (f, t-1)

[0019] Wherein, a is a hyperparameter for adjusting the smoothing degree, Score EMA (f, t-1) is the importance score obtained by exponential moving average processing in the last iteration.

[0020] Using the exponential moving average method to process the importance score can obtain more global information. On this basis, at least one filter can be selected for pruning based on the clustering method, combined with the global importance score and the parameter similarity in a single layer, to improve the efficiency of the overall pruning.

[0021] Specifically, for a convolutional layer, its parameters are a 4-dimensional array: C in x Kh ×K w ×C out +C out , where C in and C out are the number of input channels and output channels of the convolutional layer, respectively, K h ×K w is the size of the convolution kernel. Since in the pruning method, the unit of the filter is divided according to the output channel, a convolutional layer is regarded as having C out filters, for the convenience of representation, let D = C in ×K h ×K w + 1 represent the parameter amount of each filter.

[0022] The core steps of the pruning method of the embodiment are shown in Figure 1 , and the specific steps are as follows:

[0023] Step 1: Train the FCOS target detection model on the target data set to obtain the model weight parameters and performance indicators before pruning.

[0024] Step 2: Start an iteration, t = t + 1, based on the importance evaluation method used in Group Fisher Pruning, calculate the importance score of each filter in each convolutional layer according to the gradient information of back propagation; where the initial value of t is 0.

[0025] Step 3: Use the exponential moving average method to process the importance score score(f, t) obtained in this calculation to obtain the sliding average value Score EMA (f, t) from the beginning of training to the present.

[0026] Step 4: Determine whether the current iteration number t reaches 10 times, if yes, go to Step 5, otherwise repeat Steps 2 and 3.

[0027] Step 5: Calculate the average value of the importance scores of all filters in each convolutional layer L in the FCOS target detection model as the importance score of the convolutional layer L. where represents the filter corresponding to the i-th output channel in the convolutional layer L, and N is the total number of all filters in the convolutional layer L.

[0028] Step 6: Find the convolutional layer with the lowest importance score Score(L, t) of all convolutional layers, as the target layer of this pruning, denoted as L target .

[0029] Step 7: Calculate the variance of the importance scores of all filters in the selected target convolutional layer L target .

[0030]

[0031] Then according to Var(L) target ,t) to obtain the importance score threshold for this pruned convolutional layer:

[0032] Thr(L target ,t)=Score(L target ,t)-β×Var(L target ,t);

[0033] β is a hyperparameter that adjusts the strictness of the screening.

[0034] Step 8: Apply the selected target convolutional layer L target The weight parameters and offset parameters are divided according to the output channel to obtain C. out The dimension is D=C in ×K h ×K w +1 parameter vector; use K-means clustering to cluster this set of filter parameter vectors to obtain K clusters.

[0035] Step 9: Apply the target convolutional layer L target For each cluster, select the filter with the lowest importance score. If the importance score of this filter is lower than the dynamic threshold Thr(L) calculated in step 7, then... target If t), then the filter is pruned.

[0036] Step 10: Determine whether the sparsity of the FCOS object detection model has reached the expected level or the number of iterations has reached the upper limit. If so, proceed to step 11; otherwise, return to step 2.

[0037] Step 11: Fine-tune the pruned FCOS object detection model and use the fine-tuned FCOS object detection model for final object detection.

Claims

1. A CNN fast pruning method for target detection based on filter weight clustering, characterized in that, The method comprises the following steps: a target detection model initialization step: training a target detection model using a target data set to obtain an unpruned target detection model; an importance score exponential moving average processing step: setting a fixed pruning interval, and calculating the importance scores of the filters in each convolution layer of the target detection model in each iteration; using an exponential moving average method to accumulate the moving average values of the importance scores of the filters from the start of pruning to the present, and outputting the moving average value of the importance score obtained in the last iteration that meets the set pruning interval to the importance score threshold value calculation step for pruning the convolution layer; an importance score threshold value calculation step for pruning the convolution layer: adding the moving average values of all filters in each convolution layer and averaging the result to obtain the importance score of the convolution layer, and taking the convolution layer with the lowest importance score as the target convolution layer for pruning this time; calculating the variance of the importance scores of the filters in the target convolution layer; and taking the result obtained by subtracting the variance weighted by a hyperparameter from the importance score of the convolution layer as the importance score threshold value of the target convolution layer; a pruning step: clustering the filter parameter vectors of the target convolution layer; in each cluster of the target convolution layer, pruning the filters with the lowest moving average value of the importance score and the moving average value of the importance score lower than the importance score threshold value; then determining whether the overall sparsity of the neural network of the current target detection model reaches the expectation, and if yes, returning to the importance score exponential moving average processing step, and if not, using the current target detection model for target detection.

2. The method of claim 1, wherein, The calculation of the moving average value of the importance score of the filter is: Score EMA (f, t) = a x score(f, t) + (1 - a) x Score EMA (f, t - 1) wherein score(f, t) is the importance score computed by the filter f at the tth iteration; Score EMA (f, t-1) is the importance score obtained at the t-1th iteration subjected to exponential smoothing, and a is a hyperparameter used to adjust the smoothing degree.

3. The method of claim 1, wherein, The output interval of the exponential moving average processing is the set pruning interval.

4. The method of claim 2, wherein, The calculation of the importance score of the convolution layer is: wherein Score(L, t) is the importance score of the convolutional layer L at the tth iteration, N is the total number of all filters in the convolutional layer L, represents the filter corresponding to the i th output channel in the convolutional layer L.

5. The method of claim 4, wherein, Target convolutional layer L target The variance of the importance scores of its individual filters is computed in the middle as: Among them, Var(L target ,t) is the target convolutional layer L target The variance of the importance score at the t-th iteration, Sco re (L target ,t) is the target convolutional layer L target The importance score at the t-th iteration. For the target convolutional layer L target The moving average of the importance score of the filter corresponding to the i-th output channel in the t-th iteration.

6. The method of claim 5, wherein, The calculation of the importance score threshold value of the target convolution layer is: Thr(L target , t) = Score(L target , t) - β x Var(L target , t) where Thr(L target is the importance score threshold of the target convolutional layer L target The importance score threshold β at the t-th iteration is a hyper-parameter that adjusts the strictness of the screening.

Images