A yolofastest-xl network convolutional layer pruning method based on convolutional layer feature space similarity

Abstract

The application belongs to the computer field and discloses a YOLO-Fastest-xl network convolutional layer pruning method based on convolutional layer feature space similarity, wherein a more shallow neural network can be obtained by pruning the convolutional layers with high similarity, the inference calculation amount of the network is reduced, and the detection speed on a resource-limited embedded platform is improved. The residual module of the YOLO-Fastest-xl network is regularly pruned to ensure the consistency of the parameter dimensions under the consistency principle of the grouped convolution channels, the edge feature similarity between the convolutional layers is approximated to the similarity between the layers, the edge features of the feature maps of the second and third convolutional layers of the residual module are extracted through a canny operator, then the spatial edge similarity between the adjacent layer feature map sets is calculated, one layer of the adjacent layers with high similarity is pruned, and after the convolutional layers are pruned, not only the parameter amount and the calculation amount are reduced, but also the I / O time of the interlayer data is reduced, and the detection speed is accelerated.

Description

Technical Field

[0001] This invention belongs to the field of computer science, and specifically relates to a method for pruning convolutional layers in YOLO-Fastest-xl networks based on the similarity of convolutional layer feature spaces. Background Technology

[0002] The computational and storage resources required for deep convolutional neural networks (CNNs) severely limit their deployment on resource-constrained embedded platforms. To improve algorithm efficiency, pruning, as one of the most widely used methods for model compression, reduces the network's width or depth by removing redundant convolutional kernels, thereby reducing the number of parameters and computational load, and accelerating forward inference. While convolutional kernel pruning can result in a narrower network structure and accelerate inference, data I / O between convolutional layers still consumes a significant amount of time if the network is deep. Addressing the difficulty in measuring the similarity between adjacent convolutional layers in the YOLO-Fastest-xl network, this invention designs a layer pruning method based on feature space similarity. Since the differences between feature maps largely depend on the differences in spatial edge features, the minimum difference in edge features between adjacent layer feature map sets is calculated. A smaller difference indicates higher similarity between adjacent layers and more similar feature extraction capabilities. During layer pruning, only one of the two can be retained. Removing redundant convolutional layers accelerates network inference and facilitates network deployment. Summary of the Invention

[0003] This invention proposes a convolutional layer pruning method for the YOLO-Fastest-xl network based on the similarity of convolutional layer feature spaces. This method addresses the difficulty in measuring convolutional layer similarity and allows for the removal of redundant convolutional layers based on inter-layer similarity. Starting with a coarse-grained pruning approach, this method reduces the number of network parameters and computational cost by removing one of the adjacent convolutional layers in the residual module, thereby accelerating the network's forward inference speed and making it easier to deploy the network model on embedded platforms.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] A method for pruning convolutional layers in the YOLO-Fastest-xl network based on the similarity of convolutional layer feature space is described below:

[0006] Step 1: Sparsely train the network until convergence. Sort the convolutional kernels of each convolutional layer according to the L1 norm, gradient and activation value in the residual modules of the network by voting. Then, determine the number of convolutional kernels to be retained for pruning according to the constraint relationship between the number of channels and the number of groups of grouped convolutions. Prune the redundant convolutional kernels of the first and second layers from low to high importance.

[0007] Step 2: Retrain the network until convergence, take multiple images from the test set as input, save the feature map of each convolutional kernel in each layer, and use the Canny operator to obtain the edge feature information of each feature map;

[0008] Step 3: Calculate the distance between each feature map in each layer and the feature map in the next layer in a loop. Take the minimum distance as the similarity between convolutional kernels. Add up the similarity of all convolutional kernels in each layer to get the similarity between two adjacent convolutional layers.

[0009] Step 4: Based on the similarity calculation results, sort the similarity of all convolutional layers, then select a ratio to determine the adjacent convolutional layers to be pruned according to the similarity from high to low.

[0010] Step 5: Modify the `groups` parameter of the grouped convolutions in the residual module where the convolutional layers to be pruned need to be made consistent with the number of output channels. Retrain the network, then prune one of the adjacent convolutional layers to be pruned, and then fine-tune the model until it converges again.

[0011] The calculation method for determining the number of convolution kernels to be retained during pruning in step one is as follows:

[0012]

[0013] n1'=n2'=κ×n3

[0014] In the formula, n2 and n3 are the number of convolution kernels in the second and third layers, and n1′ and n2′ are the number of convolution kernels in the second and third layers after removing redundant convolution kernels.

[0015] In step three, the distance between each feature map and the next layer feature map is calculated as follows:

[0016]

[0017] N = max(sum(edge) 1 ),sum(edge 2 ))

[0018]

[0019] In the formula, edge 1 and edge 2 The edge feature information of the two feature maps is given by w and h, which are the width and height of the feature maps, respectively, and dist is the distance between the two feature maps.

[0020] The advantages of this invention compared to the prior art are:

[0021] (1) The present invention can obtain a network model with fewer parameters and less computation, accelerate the detection speed, and enable the network to be deployed on embedded platforms with limited resources.

[0022] (2) This invention starts from the granularity of the convolutional layer and simplifies the network model in depth compared with convolutional kernel pruning, which can more effectively accelerate the detection speed.

[0023] (3) The present invention can fully release the performance of each convolutional layer, reduce the number of network parameters and computation, reduce the time consumption of network data I / O, reduce detection time, and is conducive to deployment on embedded devices with limited resources. Attached Figure Description

[0024] Figure 1 This is a flowchart of the pruning process of the present invention.

[0025] Figure 2 This is the feature map and edge information difference map of the present invention.

[0026] Figure 3 This is a transition diagram for the residual module pruning of the present invention. Detailed Implementation

[0027] To better understand the pruning scheme of the present invention, a further detailed description is provided below in conjunction with the accompanying drawings.

[0028] This invention discloses a method for pruning convolutional layers in the YOLO-Fastest-xl network based on the similarity of the convolutional layer feature space. The specific steps are as follows:

[0029] Step 1: Sparsely train the network until convergence. Sort the convolutional kernels of each convolutional layer according to the L1 norm, gradient and activation value in the residual modules of the network by voting. Then, determine the number of convolutional kernels to be retained for pruning according to the constraint relationship between the number of channels and the number of groups of grouped convolutions. Prune the redundant convolutional kernels of the first and second layers from low to high importance.

[0030] Based on the divisor relationship between the number of channels and the groups parameter, the number of convolutional kernels to be retained during pruning can be determined as follows:

[0031]

[0032] n1'=n2'=κ×n3

[0033] In the formula, n2 and n3 are the number of convolution kernels in the second and third layers, and n1′ and n2′ are the number of convolution kernels in the second and third layers after removing redundant convolution kernels.

[0034] Step 2: Retrain the network until convergence, using 100 images from the test set as input, save the feature map of each convolutional kernel in each layer, and use the Canny operator to obtain the edge feature information of each feature map;

[0035] The faster Canny operator is selected to extract edge features from the feature map. For the edge feature edge of the feature map feature_map, edge = Canny(feature_map).

[0036] Step 3: Calculate the distance between each feature map in each layer and the feature map in the next layer in a loop. Take the minimum distance as the similarity between convolutional kernels. Add up the similarity of all convolutional kernels in each layer to get the similarity between two adjacent convolutional layers.

[0037] The calculation method for any two feature maps is as follows:

[0038]

[0039] N = max(sum(edge) 1 ),sum(edge 2 ))

[0040]

[0041] In the formula, edge 1 and edge 2 The edge feature information of the two feature maps is given by w and h, which are the width and height of the feature maps, respectively, and dist is the distance between the two feature maps.

[0042] The similarity distance is defined as the ratio of the number of differing pixels between edge features to the number of pixels with the largest difference. A smaller ratio indicates greater similarity. The similarity between adjacent convolutional layers is described by the sum of the minimum similarity distances between their feature maps. Assuming the similarity distance is calculated using the `cal_dist()` function:

[0043] feature_map i-1 ={feature_map0,feature_map1,...,feature_map n}

[0044] feature_map i ={feature_map0,feature_map1,...,feature_map m}

[0045] dist k =min(cal_dist(feature_map) kfeature_map i-1 ))

[0046]

[0047] feature_map i and feature_map i-1 Let feature_map represent the feature map sets of the i-th and (i-1)-th convolutional layers, respectively. n The feature map represents the feature map generated by the nth convolutional kernel in the (i-1)th layer. m The feature map represents the feature map generated by the m-th convolutional kernel in the i-th layer. k Dist represents the feature map generated by the k-th convolutional kernel in the i-th layer. k Dist represents the similarity between the feature map generated by the k-th convolutional kernel in the i-th layer and the feature map of the (i-1)-th convolutional layer. i,i-1 This indicates the similarity between two convolutional layers.

[0048] Deep learning convolutional kernels extract features locally, which is consistent in principle with filters in traditional image processing. To better distinguish objects, filters often focus on edge features when processing images, extracting edge information by calculating the vertical and horizontal gradients of the image, resulting in more refined information. The role of convolutional layers in deep learning is also to extract image features; the calculation method is the same as that of filters when convolving two-dimensional feature maps. Therefore, the convolution operation of convolutional layers should focus on the gradient of grayscale changes in the image, i.e., edge features. If the edge information difference between the feature maps of two layers is small, they can be substituted for each other for subsequent convolutional layers. The difference between the feature map and the edge information is as follows: Figure 2 As shown.

[0049] Step 4: Based on the similarity calculation results, sort the similarity of all convolutional layers, then select a ratio to determine the adjacent convolutional layers to be pruned according to the similarity from high to low.

[0050] The forward propagation formula between the three layers of the network is expressed as follows:

[0051]

[0052]

[0053]

[0054] When the spatial similarity of the output feature maps of layer l+1 and layer l+2 is very high:

[0055] x l+1 ≈x l+2

[0056] The output of layer l+1 extracts similar feature information after being convolved by layer l+3, and can replace the output of layer l+2. The convolution kernel and other parameters can be updated through training. The subscript new indicates new parameters that are different from the original parameters. The output will not have a significant impact on the feature extraction of subsequent layers.

[0057]

[0058]

[0059] Furthermore, the high spatial similarity between the output feature maps of layer l+1 and layer l+2 indicates that the convolutional kernel of one of the two layers is redundant. The feature map generated by the input of layer l+1 after two convolutions has extremely high edge feature similarity, which is equivalent to a linear transformation process and can be replaced by a single convolution. Therefore, the two convolutional layers can also be replaced by layer l+2 (i.e., the result of layer l+2 is obtained through the parameters of layer l+2 and the input of layer l+1).

[0060]

[0061] In the formula, x l x l+1 x l+2 and x l+3 These are the calculation results for layers l, l+1, l+2, and l+3 during forward propagation, respectively, γ l γ l+1 and γ l+2 These are the scaling factor coefficients for layers l+1, l+2, and l+3, respectively, W l W l+1 and W l+2 These are the convolutional kernel weight matrices for layers l+1, l+2, and l+3, respectively, β l β l+1 and β l+2 These are the offsets for layers l+1, l+2, and l+3, respectively. and These are the new scaling factor coefficients for layers l+1, l+2, and l+3, respectively. and These are the new offsets for layers l+1, l+2, and l+3, respectively. These are the batch normalization processes for the (l+1), (l+2), and (l+3)th layers, respectively. It is a new batch normalization process at level l+2, and its subscript... These are the mean, variance, and smallest non-zero values ​​of the new BN at layer l+2. and These are the new convolutional kernel weight matrices for the (l+1), (l+2), and (l+3)th layers, respectively.

[0062] Step 5: Modify the `groups` parameter of the grouped convolutions in the residual module where the convolutional layers to be pruned need to be matched with the number of output channels. Retrain the network, then prune one of the adjacent convolutional layers to be pruned, and fine-tune the model until it converges again.

[0063] When pruning the third convolutional layer in the residual structure, because the number of channels in the residual output is small, it is necessary to modify the grouped convolution of the second convolutional layer. The principle is as follows: Figure 3 As shown.

[0064] Overall, the convolutional layer pruning method for YOLO-Fastest-xl networks based on convolutional layer feature space similarity described in this invention can prune one of the adjacent layers with high similarity in the network residual module, thereby reducing the number of parameters and computational load, reducing I / O time for inter-layer data, accelerating detection speed, and facilitating the deployment of detection networks on resource-constrained embedded devices. The techniques and principles not described in detail in this invention are well-known to those skilled in the art.

Claims

1. A method for pruning convolutional layers in a YOLO-Fastest-xl network based on the similarity of the feature space of convolutional layers, characterized in that, The specific steps are as follows: Step 1: Sparsely train the network until convergence. Sort the convolutional kernels of each convolutional layer according to the L1 norm, gradient and activation value in the residual modules of the network by voting. Then, determine the number of convolutional kernels to be retained for pruning according to the constraint relationship between the number of channels and the number of groups of grouped convolutions. Prune the redundant convolutional kernels of the first and second layers from low to high importance. Step 2: Retrain the network until convergence, take multiple images from the test set as input, save the feature map of each convolutional kernel in each layer, and use the Canny operator to obtain the edge feature information of each feature map; Step 3: Calculate the distance between each feature map in each layer and the feature map in the next layer in a loop. Take the minimum distance as the similarity between convolutional kernels. Add up the similarity of all convolutional kernels in each layer to get the similarity between two adjacent convolutional layers. Step 4: Based on the similarity calculation results, sort the similarity of all convolutional layers, then select a ratio to determine the adjacent convolutional layers to be pruned according to the similarity from high to low. Step 5: Modify the `groups` parameter of the grouped convolutions in the residual module where the convolutional layers to be pruned need to be made consistent with the number of output channels. Retrain the network, then prune one of the adjacent convolutional layers to be pruned, and then fine-tune the model until it converges again.

2. The YOLO-Fastest-xl network convolutional layer pruning method based on convolutional layer feature space similarity as described in claim 1, characterized in that, The calculation method for determining the number of convolution kernels to be retained during pruning in step one is as follows: n1'=n2'=κ×n3 In the formula, n2 and n3 are the number of convolution kernels in the second and third layers, and n1' and n2' are the number of convolution kernels in the first and second layers after removing redundant convolution kernels.

3. The YOLO-Fastest-xl network convolutional layer pruning method based on convolutional layer feature space similarity as described in claim 1, characterized in that, In step three, the distance between each feature map and the next layer feature map is calculated as follows: N=max(sum(edge 1 ),sum(edge 2 )) In the formula, edge 1 and edge 2 The edge feature information of the two feature maps is given by w and h, which are the width and height of the feature maps, respectively, and dist is the distance between the two feature maps.

Images