A network pruning method

Abstract

The present invention provides a network pruning method, including: obtaining the convolution kernel and output feature map of each layer; calculating the convolution kernel of each layer in the convolution layer and generating a two-norm; calculating the output feature map of each layer in the convolution layer and generating a batch average rank; multiplying the two-norm and the batch average rank, and generating a product value corresponding to the layer; sorting the calculated product values, selecting at least one product value with a small value according to a preset pruning rate, and then setting the channel convolution kernel parameter in the convolution kernel corresponding to the selected product value to 0, and updating the convolution layer; training the updated convolution layer through a convolutional neural network, and obtaining the trained convolution layer; repeating N times; removing the channel corresponding to the channel convolution kernel parameter of 0 in the convolution layer and loading it into the network model to form a network pruning model. Compared with the related art, the network pruning method of the present invention can reduce the size of the network model and has high network detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a network pruning method. Background Art

[0002] In recent years, deep learning-based convolutional neural networks represented by AlexNet have received widespread attention and have been deeply developed. With the increase of training data and the enhancement of the computing power of hardware devices, network models are getting larger and larger, and the number of parameters and storage space occupied are also increasing. In order to solve the problem that large models cannot be deployed on edge devices and wearable devices with limited resources, network model compression without affecting the performance of the network model has become an important solution technology.

[0003] The network model compression in related technologies generally adopts a pruning method based on the convolutional kernel norm.

[0004] However, the pruning method based on the convolution kernel norm in the related art has the following problems: (1) The convolution kernel norm should meet two ideal conditions: 1) The standard deviation of the filter norm is also high; 2) The minimum norm of the filter should be close to 0. However, most filters do not meet the above conditions, especially the target detection network filter. (2) The pruning method based on the norm criterion ignores the output of the network. The output of the network is the result of the network learning and should be considered accordingly. Therefore, how to make the pruning method can greatly reduce the size of the network model and ensure the detection accuracy of the network is a technical problem to be solved.

[0005] Therefore, it is necessary to provide a new module and method to solve the above problems. Summary of the invention

[0006] In view of the above deficiencies in the prior art, the present invention proposes a network pruning method that can reduce the size of a network model and has high network detection accuracy.

[0007] In order to solve the above technical problems, an embodiment of the present invention provides a network pruning method, which is applied to a convolutional neural network. The method comprises the following steps:

[0008] Step S1, obtaining a convolutional layer in a target network model, and deriving a convolution kernel and an output feature map of each layer in the convolutional layer according to the convolutional layer;

[0009] Step S2, calculating the convolution kernel of each layer in the convolution layer, and generating a binary norm corresponding to the convolution kernel of the layer;

[0010] Step S3, calculating the output feature map of each layer in the convolutional layer, and generating a batch average rank corresponding to the convolution kernel of the layer;

[0011] Step S4, multiplying the calculated binary norm corresponding to each layer in the convolutional layer and the batch average rank, and generating a product value corresponding to the layer;

[0012] Step S5, sorting the calculated product values ​​according to their sizes, selecting at least one product value with a small value according to a preset pruning rate, setting the channel convolution kernel parameter in the convolution kernel corresponding to the selected product value to 0, and updating the convolution layer;

[0013] Step S6, training the updated convolutional layer through the convolutional neural network, and obtaining a trained convolutional layer;

[0014] Step S7, repeating steps S1 to S6 N times, N≥2, N is a positive integer;

[0015] Step S8: remove the channels corresponding to the channel convolution kernel parameters of 0 in the convolution layer and load them into the network model to form a network pruning model.

[0016] Preferably, in step S2, the weight of each convolutional layer is weight; the binary norm is L2, and satisfies the following formula:

[0017] Weight = weight.view(c out , c in ×k×k) (1);

[0018] L2=torch.norm(Weight, p=2, dim=1) (2);

[0019] Among them, view() is the dimension transformation operation function, c out is the number of channels that output the convolution kernel, c in is the number of channels of the input convolution kernel, k is the width and height of the convolution kernel, torch.norm() is the norm operation function for the Weight tensor, p=2 is the parameter for the second norm operation; dim=1 is the parameter for the norm operation on the second dimension.

[0020] Preferably, in step S3, the batch average rank is rank, and satisfies the following formula:

[0021] a=Torch.matrix_rank(batchsize×p out ×h×w) (3);

[0022] b=a.view(batchsize,-1) (4);

[0023] rank=b.sum(0) / batchsi(5);

[0024] Among them, a is a variable, Torch.matrix_rank() is the rank function for obtaining the feature map, batchsize×p out ×h×w is the tensor of the output feature map, batchsize is the number of images input to the network in one iteration of training, and p out is the tensor of the output feature map, h is the height of the output feature map, w is the width of the output feature map, and b.sum(0) is the operation function that sums the first dimension of b.

[0025] Preferably, in step S4,

[0026] Sorted(Result=L2*rank) (6);

[0027] weight(c min ×c in ×k×k)=0 (7);

[0028] Among them, Sorted() is a function for sorting by size, Result = L2 * rank is the product value, c min The number of output channels with the smaller product value.

[0029] Preferably, in step S5, the pruning rate is 0.4; and in step S7, N is 250.

[0030] Preferably, step S8 comprises the following steps:

[0031] Step S81, removing the channels corresponding to the channel convolution kernel parameters of 0 in the convolution layer and obtaining a network pruned convolution layer;

[0032] Step S82, fine-tuning the network pruning convolutional layer according to a preset network model fine-tuning algorithm;

[0033] Step S83: Load the fine-tuned network pruning convolutional layer into the network model to form the network pruning model.

[0034] Preferably, step S82 includes the following steps:

[0035] Step S821, loading the preset parameters to be retained into the network pruned convolutional layer, training the network through the convolutional neural network, and obtaining the trained network pruned convolutional layer;

[0036] Step S822: training the trained network pruned convolutional layer through the convolutional neural network, repeating M times, M≥2, M is a positive integer.

[0037] Preferably, M is 100.

[0038] Compared with the related art, the network pruning method of the present invention calculates the convolution kernel of each layer in the convolution layer and generates a binary norm; calculates the output feature map of each layer in the convolution layer and generates a batch average rank; multiplies the binary norm and the batch average rank, and generates a product value corresponding to the layer; sorts the calculated product values, selects at least one product value with a small value according to the preset pruning rate, and then sets the channel convolution kernel parameter in the convolution kernel corresponding to the selected product value to 0, and updates the convolution layer; trains the updated convolution layer through a convolutional neural network, and obtains the trained convolution layer; repeats N times; removes the channel corresponding to the channel convolution kernel parameter of 0 in the convolution layer and loads it into the network model to form a network pruning model. According to the operations of the above steps S1 to S8, processing is performed based on the two criteria of binary norm and batch average rank, and the size of the network pruning model is reduced. Compared with the related art that adopts a single criterion based on the convolution kernel norm, the network pruning method of the present invention reduces the size of the network model and has high network detection accuracy.

[0039] Figures: The present invention is described in detail below in conjunction with the accompanying drawings. The above and other aspects of the present invention will become clearer and easier to understand through the detailed description made in conjunction with the following drawings. In the accompanying drawings:

[0040] Figure 1 It is a flowchart of the network pruning method of the present invention;

[0041] Figure 2 This is a flowchart of step S8 in the network pruning method of the present invention. DETAILED DESCRIPTION

[0042] The specific implementation of the present invention will be described in detail below with reference to the accompanying drawings.

[0043] The specific implementation modes / embodiments recorded herein are specific implementation modes of the present invention, which are used to illustrate the concept of the present invention, are explanatory and exemplary, and should not be interpreted as limiting the implementation modes of the present invention and the scope of the present invention. In addition to the embodiments recorded herein, those skilled in the art can also adopt other obvious technical solutions based on the contents disclosed in the claims and the specification of this application, and these technical solutions include any obvious replacement and modification of the embodiments recorded herein, which are within the protection scope of the present invention.

[0044] The present invention provides a network pruning method, which is applied to a convolutional neural network.

[0045] Please refer to Figure 1 As shown, Figure 1 The figure is a flowchart of the network pruning method of the present invention.

[0046] The network pruning method comprises the following steps:

[0047] Step S1: obtain a convolutional layer in a target network model, and derive a convolution kernel and an output feature map of each layer in the convolutional layer according to the convolutional layer.

[0048] In this embodiment, the target network model is a yolov5 network. The data set from which the convolution layer comes is a PCB data set containing 6 types of defects, with a total of 3,000 pictures, and a training set, a validation set, and a test set are obtained in a ratio of 8:1:1.

[0049] Step S2: Calculate the convolution kernel of each layer in the convolution layer and generate a binary norm corresponding to the convolution kernel of the layer.

[0050] In the step S2, the weight of each convolutional layer is weight; the second norm is L2, and satisfies the following formula:

[0051] Weight = weight.view(c out , c in ×k×k) (1);

[0052] L2=torch.norm(Weight, p=2, dim=1) (2);

[0053] Among them, view() is the dimension transformation operation function, c out is the number of channels that output the convolution kernel, c in is the number of channels of the input convolution kernel, k is the width and height of the convolution kernel, torch.norm() is the function of performing norm operation on the Weight tensor, p=2 is the parameter for performing the second norm operation, and dim=1 is the parameter for performing the norm operation on the second dimension. Performing the norm operation on the second dimension means that for a two-dimensional tensor, the norm operation is performed along the row.

[0054] Step S3: Calculate the output feature map of each layer in the convolutional layer and generate a batch average rank corresponding to the convolution kernel of the layer.

[0055] In step S3, the batch average rank is rank, and satisfies the following formula:

[0056] a=Torch.matrix_rank(batchsize×p out ×h×w) (3);

[0057] b=a.view(batchsize,-1) (4);

[0058] rank=b.sum(0) / batchsize(5);

[0059] Among them, a is a variable, Torch.matrix_rank() is the rank function for obtaining the feature map, batchsize×p out ×h×w is the tensor of the output feature map, batchsize is the number of images input to the network in one iteration of training, and p out is the tensor of the output feature map, h is the height of the output feature map, w is the width of the output feature map, and b.sum(0) is the operation function that sums the first dimension of b.

[0060] Step S4: multiply the calculated binary norm corresponding to each layer in the convolutional layer by the batch average rank, and generate a product value corresponding to the layer.

[0061] In the step S4,

[0062] Sorted(Result=L2*rank) (6);

[0063] weight(c min ×c in ×k×k)=0 (7);

[0064] Among them, Sorted() is a function for sorting by size, Result = L2 * rank is the product value, c min The number of output channels with the smaller product value.

[0065] Step S5, sort the calculated product values ​​according to size, select at least one product value with a small value according to a preset pruning rate, set the channel convolution kernel parameter in the convolution kernel corresponding to the selected product value to 0, and update the convolution layer.

[0066] In this embodiment, the pruning rate is 0.4. The pruning rate refers to the proportion of the number of pruning in each convolutional layer.

[0067] Step S6: training the updated convolutional layer through the convolutional neural network to obtain a trained convolutional layer.

[0068] In this embodiment, the learning rate during the convolutional neural network training is 0.0025, and the optimizer is SGD. Step S7, repeating steps S1 to S6 N times, wherein N≥2, N is a positive integer.

[0069] In this embodiment, in step S7, N is 250. That is, the number of training rounds of the convolutional neural network is 250.

[0070] Step S8: remove the channels corresponding to the channel convolution kernel parameters of 0 in the convolution layer and load them into the network model to form a network pruning model.

[0071] Please refer to Figure 2 As shown, Figure 2 This is a flowchart of step S8 in the network pruning method of the present invention.

[0072] Specifically, step S8 includes the following steps:

[0073] Step S81, removing the channels corresponding to the channel convolution kernel parameters of 0 in the convolution layer and obtaining a network pruned convolution layer.

[0074] Step S82: fine-tune the network pruned convolutional layer according to a preset network model fine-tuning algorithm.

[0075] The step S82 comprises the following steps:

[0076] Step S821: After loading the preset parameters to be retained into the network pruned convolutional layer, the convolutional neural network is used for training, and a trained network pruned convolutional layer is obtained.

[0077] Step S822: training the trained network pruned convolutional layer through the convolutional neural network, and repeating M times, where M≥2, and M is a positive integer.

[0078] In this embodiment, M is 100. That is, the number of rounds of fine-tuning the network pruning convolutional layer is 100.

[0079] In this embodiment, the preset network model fine-tuning algorithm is to load the preset parameters to be retained into the network pruning convolution layer, train it through the convolutional neural network, and obtain the trained network pruning convolution layer, and then train the trained network pruning convolution layer through the convolutional neural network M times. Among them, the preset parameters to be retained are only loaded into the network pruning convolution layer once; the preset parameters to be retained are selected according to actual design requirements, for example, the parameters are small learning rates.

[0080] It should be pointed out that the fine-tuning is a commonly used term in the field of convolutional neural network training of the present invention, and does not mean the fine-tuning commonly used in daily life. Here, it will not be described in detail.

[0081] Step S83: Load the fine-tuned network pruning convolutional layer into the network model to form the network pruning model.

[0082] The network pruning model after the steps S1 to S8 are implemented is smaller than the original target network model. The network pruning model is a small model.

[0083] In this embodiment, through the implementation of the steps S1 to S8, the detection accuracy of the network pruning model after test pruning is only 0.1% lower than the accuracy of the original target network model, and the detection accuracy reaches 99% of the original target network model, and the number of parameters of the network pruning model is reduced by nearly 51%. In summary, the network pruning method of the present invention can reduce the size of the network model and has high network detection accuracy.

[0084] Compared with the related art, the network pruning method of the present invention calculates the convolution kernel of each layer in the convolution layer and generates a binary norm; calculates the output feature map of each layer in the convolution layer and generates a batch average rank; multiplies the binary norm and the batch average rank, and generates a product value corresponding to the layer; sorts the calculated product values, selects at least one product value with a small value according to the preset pruning rate, and then sets the channel convolution kernel parameter in the convolution kernel corresponding to the selected product value to 0, and updates the convolution layer; trains the updated convolution layer through a convolutional neural network, and obtains the trained convolution layer; repeats N times; removes the channel corresponding to the channel convolution kernel parameter of 0 in the convolution layer and loads it into the network model to form a network pruning model. According to the operations of the above steps S1 to S8, processing is performed based on the two criteria of binary norm and batch average rank, and the size of the network pruning model is reduced. Compared with the related art that adopts a single criterion based on the convolution kernel norm, the network pruning method of the present invention reduces the size of the network model and has high network detection accuracy.

Claims

1. A network pruning method, applied to a convolutional neural network, characterized in that: The method comprises the following steps: Step S1, obtaining a convolutional layer in a target network model, and deriving a convolution kernel and an output feature map of each layer in the convolutional layer according to the convolutional layer; Step S2, calculating the convolution kernel of each layer in the convolution layer, and generating a binary norm corresponding to the convolution kernel of the layer; Step S3, calculating the output feature map of each layer in the convolutional layer, and generating a batch average rank corresponding to the convolution kernel of the layer; Step S4, multiplying the calculated binary norm corresponding to each layer in the convolutional layer and the batch average rank, and generating a product value corresponding to the layer; Step S5, sorting the calculated product values ​​according to their sizes, selecting at least one product value with a small value according to a preset pruning rate, setting the channel convolution kernel parameter in the convolution kernel corresponding to the selected product value to 0, and updating the convolution layer; Step S6, training the updated convolutional layer through the convolutional neural network, and obtaining a trained convolutional layer; Step S7, repeating steps S1 to S6 N times, N≥2, N is a positive integer; Step S8, removing the channels corresponding to the channel convolution kernel parameters of 0 in the convolution layer and loading them into the network model to form a network pruning model; The convolutional layer comes from a data set containing 6 types of defects in a total of 3,000 images, and a training set, a validation set, and a test set are obtained in a ratio of 8:1:

1.

2. The network pruning method according to claim 1, characterized in that: In the step S2, the weight of each convolutional layer is weight; the second norm is L2, and satisfies the following formula: Weight=weight.view( , )(1); L2=torch.norm(Weight, p=2,dim=1)(2); Among them, view() is the dimension transformation operation function, is the number of channels that output the convolution kernel, is the number of channels of the input convolution kernel, is the width and height of the convolution kernel, torch.norm() is the norm operation function for the Weight tensor, p=2 is the parameter for the second norm operation, and dim=1 is the parameter for the second dimension.

3. The network pruning method according to claim 2, characterized in that: In step S3, the batch average rank is rank, and satisfies the following formula: a=Torch.matrix_rank( )(3); b=a.view( ,-1)(4); rank=b.sum(0) / (5); Among them, a is a variable, Torch.matrix_rank() is the rank function for obtaining the feature map, is the tensor of the output feature map, is the number of images fed into the network in one iteration of training, is the tensor of the output feature map, is the height of the output feature map, is the width of the output feature map, and b.sum(0) is the operation function that sums the first dimension of b.

4. The network pruning method according to claim 3, characterized in that: In the step S4, Sorted(Result=L2 *rank)(6); weight( )=0(7); Among them, Sorted() is a function for sorting by size, and Result=L2*rank is the product value. The number of output channels with the smaller product value.

5. The network pruning method according to claim 1, characterized in that: In the step S5, the pruning rate is 0.4; in the step S7, N is 250.

6. The network pruning method according to claim 1, characterized in that: The step S8 comprises the following steps: Step S81, removing the channels corresponding to the channel convolution kernel parameters of 0 in the convolution layer and obtaining a network pruned convolution layer; Step S82, fine-tuning the network pruning convolutional layer according to a preset network model fine-tuning algorithm; Step S83: Load the fine-tuned network pruning convolutional layer into the network model to form the network pruning model.

7. The network pruning method according to claim 6, characterized in that: The step S82 includes the following steps: Step S821, loading the preset parameters to be retained into the network pruned convolutional layer, training the network through the convolutional neural network, and obtaining the trained network pruned convolutional layer; Step S822: training the trained network pruned convolutional layer through the convolutional neural network, repeating M times, M≥2, M is a positive integer.

8. The network pruning method according to claim 7, characterized in that: M is 100.

Images