An improved filter soft pruning method for convolutional neural network models

Abstract

The present invention discloses an improved filter soft pruning method for a convolutional neural network model, comprising the steps of: S1, parameterizing the neural network model; S2, training and updating the parameters of the neural network model, and applying a sparse penalty to the scaling factor γ of the BN layer; S3, setting some convolutional layer filter parameters to zero according to the sparsely distributed scaling factor γ; S4, performing pruning based on the parameter zeroing condition to obtain a lightweight neural network model. The present invention optimizes the process of filter soft pruning in the neural network model, removes the pre-pruning stage in the soft pruning process, combines a parameter optimizer that integrates historical gradients, and uses a pruning standard that does not depend on the filter parameters (i.e., the scaling factor γ of the BN layer), thereby solving the problem that soft pruning cannot take effect when facing a ReLU-like activation function, overcoming the defects of existing filter soft pruning methods, thereby giving full play to the potential of the soft pruning method and obtaining better neural network model performance.

Description

Technical Field

[0001] The present invention relates to the technical field of lightweight neural network models, and in particular to an improved filter soft pruning method for a convolutional neural network model. Background Art

[0002] Convolutional Neural Networks (CNNs) have achieved remarkable success in various computer vision tasks, including image classification, object detection, and instance segmentation. With the development of deep learning technology, the depth of CNN models has gradually increased. For example, the emergence of residual structures has increased the depth of CNN models to 152 layers, significantly improving their performance. However, while increasing network depth improves performance, it also increases network complexity and computational and storage costs, making it difficult to deploy on resource-constrained edge devices such as smartphones and car computers. Therefore, it is necessary to study lightweight technologies for deep CNN models.

[0003] Existing deep model lightweighting technologies can be roughly divided into parameter quantization, knowledge distillation, model pruning, etc. However, not all methods can be used for lightweighting of deep models, because some methods require specially designed software, hardware or are only targeted at certain specific models. Filter pruning, as a model pruning method, is one of the most commonly used and very efficient model lightweighting methods. It only compresses the model regularly and can achieve a high compression rate and low accuracy loss. A standard filter pruning process is as follows: 1) Train a deep model to convergence; 2) According to a specific standard, identify unimportant filters in the model and prune them; 3) Retrain the pruned model to restore model accuracy.

[0004] The development of filter pruning can be divided into two types according to the process: hard pruning (Hard Filter Pruning) and soft pruning (Soft Filter Pruning). Hard pruning strictly follows the three-step process mentioned above, which means that once a filter is removed during the pruning stage, it will not participate in the retraining process. In contrast, soft pruning will reset the weights of filters that are judged to need pruning during the training process instead of removing them, thus retaining the possibility that the weights of the zeroed filters will be restored to non-zero in the next training round. In this way, the parameter optimization space of the model during training will become larger, providing the model with the opportunity to achieve better performance. After the training is completed, unimportant filters are removed to obtain a compressed model. Soft pruning has advantages over hard pruning, and the comparison between the two can be seen in Figure 1. However, existing soft pruning methods fail when faced with ReLU-like activation functions, that is, the zeroing operation combined with the ReLU-like activation function will cause the zeroed filter to lose the ability to be restored, making it impossible to expand the parameter optimization space. This is a defect of the soft pruning method. In addition, the pruning criteria based on norms (such as L2-norm) commonly used in soft pruning methods are heavily dependent on the parameter values ​​of the filter. After the zeroed filter is restored, its L2-norm is still small due to the small amplitude of the parameter update, and it will still be zeroed in the next round of pruning, making it impossible to achieve true recovery.

[0005] In order to remedy the above-mentioned defects of the soft pruning method, the present invention provides an improved soft pruning method for filters of a convolutional neural network model. When using a parameter optimizer that integrates historical gradients, by removing the pre-pruning stage in the soft pruning method process, the ability to theoretically restore the zeroed filters is provided. In addition, by abandoning the L2-norm-based pruning standard commonly used in the soft pruning method and selecting a pruning standard that does not depend on the filter parameters, the potential of the soft pruning method is truly brought into play, that is, the parameter optimization space of the model during the training process is expanded to provide an opportunity to obtain better model performance.

[0006] Patent document CN112488304A discloses a heuristic filter pruning method and system in a convolutional neural network, comprising: obtaining the adjusted cosine similarity between two epochs for each filter in each convolutional layer, and soft-pruning the filters according to the adjusted cosine similarity; reconstructing the soft-pruned and updated convolutional neural network; repeating the soft-pruning and reconstruction process until a convolutional neural network with stable accuracy is obtained; obtaining the adjusted cosine similarity between two epochs for each filter in each convolutional layer of the convolutional neural network with stable accuracy, and hard-pruning the filters according to the adjusted cosine similarity; and fine-tuning the hard-pruned and updated convolutional neural network until the network accuracy of the convolutional neural network reaches a stable value. This method can solve the technical problem of serious accuracy loss in pruned convolutional neural networks caused by not considering the dynamic changes in distance and direction of filters during convolutional neural network training. However, this method also suffers from the problem that soft pruning is ineffective when using ReLU-like activation functions. Summary of the Invention

[0007] The purpose of the present invention is to provide an improved filter soft pruning method for convolutional neural network models to solve the problem that the existing soft pruning of neural network models is not effective when facing ReLU-like activation functions.

[0008] The purpose of the present invention can be achieved by the following technical solution: an improved filter soft pruning method for a convolutional neural network model, comprising the following steps:

[0009] S1. Select a neural network model and set parameters for the neural network model;

[0010] S2. Train the parameterized neural network model, update the neural network model setting parameters, and perform a sparse penalty on the scaling factor γ of the BN layer of the neural network model while updating the neural network model setting parameters;

[0011] S3, setting the filter parameters of some convolutional layers of the neural network model to zero according to the scaling factor γ after the sparsity penalty;

[0012] S4. Prune the neural network model according to the zeroing of the convolutional layer filter parameters to obtain a lightweight neural network model.

[0013] Furthermore: the parameter setting of the neural network model in S1 includes preprocessing of input data, initialization of model parameters and initialization of parameter optimizer.

[0014] Furthermore, the neural network model in S2 is trained to update the neural network model setting parameters, including the following steps:

[0015] S21, forward deducing the neural network model using input data to obtain output data and corresponding confidence levels;

[0016] S22. Based on the output data and the corresponding confidence level, the neural network model is gradient-backed to update the model parameters.

[0017] Furthermore, the step of performing a sparse penalty on the scaling factor γ of the BN layer of the neural network model while updating the neural network model setting parameters in S2 is:

[0018] S221. Calculate the gradient value of each filter parameter in the neural network model based on the input data and output data of the neural network model. The update formula of the filter parameter is:

[0019]

[0020] Where W is the model parameter, η is the learning rate, is the loss function;

[0021] S222. In the loss function The loss of adding the BN layer scaling factor γ is:

[0022]

[0023] Among them, (x, y) is the training data of the neural network model, x is the input data, y is the corresponding true value label, f represents the neural network model, λ is the sparse penalty amplitude value, γ is the scaling factor, Γ is the set of scaling factors γ of all BN layers in the neural network model, and g(·) is the sparse penalty term.

[0024] Furthermore, the step of setting some filter parameters of the neural network model to zero according to the scaling factor γ after the sparse penalty in S3 is:

[0025] S221, traverse the neural network model and count the number M of filters in each convolutional layer of the neural network model;

[0026] S222. Calculate the pruning threshold T subscript idx based on the preset pruning rate P:

[0027]

[0028] S223. Sort the filters in the convolutional layer according to the scaling factor γ of the BN layer corresponding to the convolutional layer;

[0029] S224: After the filters are sorted, a boundary filter is determined based on idx, and the value of the scaling factor γ corresponding to the boundary filter is the convolutional layer pruning threshold T;

[0030] S225. Set the filter parameters corresponding to the scaling factor γ in the convolutional layer that is less than the pruning threshold T to zero.

[0031] Furthermore, the step of pruning the neural network model according to the zeroing of the convolutional layer filter parameters in S4 is as follows:

[0032] S31. Prune the current convolution layer according to the zero setting of the convolution layer filter parameters. The number of convolution layer filters after pruning is M'. Then

[0033] M′=M-|PF|

[0034] Where PF is the number of parameter-zero filters in the current convolutional layer;

[0035] S32, use M' filters to reconstruct the convolution layer and BN layer;

[0036] S34, continue to prune the next convolutional layer;

[0037] S35. Repeat steps S31 to S34 until the neural network model is traversed.

[0038] Furthermore, the step of reconstructing the convolution layer and the BN layer using M' filters in S32 is:

[0039] S321, copy the M' filter parameters of the convolutional layer that have not been pruned and the corresponding BN layer parameters;

[0040] S322, filling the copied parameters into the reconstructed convolutional layer and BN layer;

[0041] S323. Set the number of filter channels in each filter in the reconstructed convolutional layer to the number of filters in the previous convolutional layer.

[0042] Beneficial effects of the present invention:

[0043] 1. The present invention optimizes the process of soft pruning of filters in neural network models. By removing the pre-pruning stage in the soft pruning method process, the problem that soft pruning is ineffective when facing ReLU-like activation functions is solved, and the defects of existing soft pruning methods for neural network model filters are overcome, thereby better unleashing the potential of the soft pruning method and facilitating better neural network model performance.

[0044] 2. This invention uses the BN layer scaling factor γ as the pruning criterion. By adding a loss for the BN layer scaling factor γ to the loss function, it becomes sparse during the training process, thereby having the ability to represent the importance of the filter. Compared with the norm-based pruning criterion used in the original filter soft pruning method, the BN layer scaling factor γ does not depend on the filter parameters and will not reduce the activation amplitude of the pruned filter due to the zeroing operation. This solves the problem that the existing soft pruning method cannot play its full role, which is conducive to obtaining better neural network model performance.

[0045] 3. The PointNet model in this invention has a baseline model accuracy of 90.7 / 86.5. When the pruning rate is set to 10%, 40%, and 50%, it can achieve better performance than the baseline model, which are 90.9 / 87.3, 90.8 / 86.9, and 90.2 / 87.0 respectively. At other pruning rates such as 60% and 70%, it can achieve better accuracy than the original soft pruning method. If the λ parameter is adjusted, such as setting λ = 10 -6 , even reaching an accuracy of 91.0 / 86.2 at a pruning rate of 20%. For the PointNet++ model, the baseline model accuracy is 93.0 / 90.0. When the pruning rate is set to 20%, 30%, 40%, and 50%, it can achieve better accuracy than the original soft pruning method. At 20%, it can reach the same accuracy as the baseline model (92.6 / 90.0). After optimizing the soft pruning process and pruning criteria, the soft pruning method can achieve a good balance between pruning rate and model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1Schematic diagram comparing the soft pruning process structure and the hard pruning process structure of the present invention;

[0047] Figure 2 Schematic diagram of the process of the improved filter soft pruning method for convolutional neural network model according to the present invention;

[0048] Figure 3a Schematic diagram of the structure of the prunable and non-pruneable network layers in the PointNet model of the present invention;

[0049] Figure 3b Schematic diagram of the structure of the prunable and non-pruneable network layers in the PointNet++ model of the present invention.

[0050] The accompanying drawings show the following descriptions in English and Chinese:

[0051] Layer: A combination structure in a deep neural network;

[0052] Pruning Pool: pruning pool;

[0053] Filter: Filter, the basic component of convolutional neural networks;

[0054] Filter Index: filter indicator number;

[0055] Original Layer: The layer that has not been pruned;

[0056] Pre-Pruning: Pre-pruning is the pruning operation before training starts in the filter soft pruning process;

[0057] After Pruning: After pruning;

[0058] (Re)Training Epoch: The epoch of model (re)training, Epoch 1 indicates the first epoch;

[0059] Pruned Layer: The pruned layer;

[0060] Soft Filter Pruning: filter soft pruning;

[0061] Hard Filter Pruning: Filter hard pruning

[0062] ISFP: The abbreviation of the English name of improved soft pruning, the full name is: Improved Soft Filter Pruning. DETAILED DESCRIPTION

[0063] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention, and are not to be construed as limiting the present invention.

[0064] like Figure 1 -3, the present invention discloses an improved filter soft pruning method for a convolutional neural network model, comprising the following steps:

[0065] S1. Select a neural network model and set parameters for the neural network model;

[0066] S2. Train the parameterized neural network model, update the neural network model setting parameters, and perform a sparse penalty on the scaling factor γ of the BN layer of the neural network model while updating the neural network model setting parameters;

[0067] S3, setting the filter parameters of some convolutional layers of the neural network model to zero according to the scaling factor γ after the sparsity penalty;

[0068] S4. Prune the neural network model according to the zeroing of the convolutional layer filter parameters to obtain a lightweight neural network model.

[0069] First, select a neural network model. The model to be trained and pruned can be selected from mainstream 2D ​​or 3D classification models, such as VGG, ResNet, PointNet, PointNet++, etc.

[0070] The preparation work before training the selected neural network model mainly includes parameter setting of the neural network model, including preprocessing of input data, initialization of model parameters and initialization of parameter optimizer.

[0071] Preprocessing of input data: If a 2D classification model is selected, the input data is an image. The input data preprocessing mainly includes unifying the image size and normalization processing, and converting the input data into tensor type data after processing; if a 3D classification model is selected, the input data is a point cloud. The input data preprocessing operation mainly includes point sampling, and converting the input point cloud data into tensor type data after processing.

[0072] The initialization of model parameters mainly includes the initialization of convolutional layer parameters and BN layer parameters. The initialization of parameters of other types of layers can use default values. The initialization of convolutional layer parameters depends on the strategy set by the selected model, such as (Heinitialization); for BN layer parameters, the scaling factor γ can be set to 1 and the bias term can be initialized to 0.

[0073] The parameter optimizer can be initialized according to the default parameters of the selected parameter optimizer. The selection of the parameter optimizer needs to consider the momentum, that is, the historical gradient of the parameters, such as when using the Adam optimizer.

[0074] To train the parameterized neural network model and update the neural network model parameters, the forward derivation and gradient backpropagation methods can be used:

[0075] S21, forward deducing the neural network model using input data to obtain output data and corresponding confidence levels;

[0076] S22. Based on the output data and the corresponding confidence level, the neural network model is gradient-backed to update the model parameters.

[0077] While updating the neural network model setting parameters, a sparse penalty is imposed on the scaling factor γ of the BN layer of the neural network model. The steps can be:

[0078] S221. Calculate the gradient value of each filter parameter in the neural network based on the input data and output data of the neural network model. The update formula of each filter parameter is:

[0079]

[0080] Where W is the model parameter, η is the learning rate, is the loss function;

[0081] S222. In the loss function The loss of adding the BN layer scaling factor γ is:

[0082]

[0083] Among them, (x, y) is the training data of the neural network model, x is the input data, y is the corresponding true value label, f represents the neural network model, λ is the sparse penalty amplitude value, γ is the scaling factor, Γ is the set of scaling factors γ of all BN layers in the neural network model, and g(·) is the sparse penalty term.

[0084] The steps of setting some of the neural network model filter parameters to zero according to the scaling factor γ after the sparsity penalty can be:

[0085] S221, traverse the neural network model and count the number M of filters in each convolutional layer of the neural network model;

[0086] S222. Calculate the pruning threshold T subscript idx based on the preset pruning rate P:

[0087]

[0088] S223. Sort the filters in the convolutional layer according to the scaling factor γ of the BN layer corresponding to the convolutional layer;

[0089] S224: After the filters are sorted, a boundary filter is determined based on idx, and the value of the scaling factor γ corresponding to the boundary filter is the convolutional layer pruning threshold T;

[0090] S225. Set the filter parameters in the convolutional layer whose scaling factor γ is less than the pruning threshold T to zero.

[0091] The steps for pruning the neural network model according to the zeroing of the convolutional layer filter parameters can be:

[0092] S31. Prune the current convolution layer according to the zero setting of the convolution layer filter parameters. The number of convolution layer filters after pruning is M'. Then

[0093] M'=M-|PF| (4)

[0094] Where PF is the number of parameter-zero filters in the current convolutional layer;

[0095] S32, use M' filters to reconstruct the convolution layer and BN layer;

[0096] S34, continue to prune the next convolutional layer;

[0097] S35. Repeat steps S31 to S34 until the neural network model is traversed.

[0098] The steps of reconstructing the convolution layer and BN layer using M' filters can be:

[0099] S321, copy the unpruned filter parameters of the convolutional layer to the pruned M' filters;

[0100] S322, reconstructing the convolutional layer and the BN layer according to the replication parameters of the M' filters;

[0101] S323. Set the number of filter channels in each filter in the reconstructed convolutional layer to the number of filters in the previous convolutional layer.

[0102] When constructing a new convolutional layer and BN layer, the number of filters contained in the convolutional layer is M', and the parameters of the unpruned filters are copied to the filters of the new convolutional layer. The BN layer can be processed in the same way, so that a compressed convolutional layer and BN layer can be obtained, and the compressed convolutional layer and BN layer have non-zero parameters corresponding to the original model.

[0103] The number of filter channels refers to the number of channels formed when a single filter in this convolutional layer receives the output data of all filters in the previous convolutional layer as input data. When constructing the next new convolutional layer, the influence of the previous convolutional layer is taken into account. For example, if the capacity of the newly constructed previous convolutional layer is reduced to M' compared to the original structure, then when constructing this layer, the number of channels of each filter needs to be reduced to M' accordingly. If the previous layer has not been reconstructed (that is, the previous layer is a structure that cannot be pruned), there is no need to reduce the number of filter channels and it can be kept the same.

[0104] Save the parameters of the newly constructed neural network model and the corresponding model structure to facilitate model construction during deployment and inference, and a compressed lightweight model is obtained.

[0105] Example description:

[0106] This embodiment uses the 3D classification model PointNet as the pruned model, and describes the specific implementation steps in detail. The flowchart is as follows: Figure 2 shown.

[0107] First, prepare the 3D classification model PointNet before model training, including preprocessing of input data, initialization of model parameters, initialization of parameter optimizer, etc.

[0108] For the PointNet model, the input data is point cloud, and the preprocessing operation is mainly point sampling. Finally, the point cloud is converted into tensor type data. For example, 1024 points can be sampled for each point cloud, and a batch contains 24 point clouds, that is, the tensor shape input to the model is 24*3*1024.

[0109] The initialization of the PointNet model parameters mainly includes the initialization of the convolutional layer parameters and the BN layer parameters. The parameters of other types of layers are initialized by default. The initialization strategy of the convolutional layer parameters is (He initialization); the fully connected layer can be regarded as a one-dimensional convolutional layer. The initialization strategy of the fully connected layer parameters is to initialize from the standard uniform distribution. For the BN layer parameters, the scaling factor γ is initialized to 1 and the bias term is initialized to 0.

[0110] Initialization of the PointNet model parameter optimizer. The default parameter optimizer is Adam (a commonly used parameter optimizer), its learning rate is initialized to 1e-03, the weight_deacy parameter is initialized to 1e-04, and other parameters are default.

[0111] Each round of training and pruning of the PointNet model can be divided into two stages: the training stage and the pruning stage. The training stage updates the parameters through forward deduction and gradient backpropagation, and uses the L1-norm to perform a sparse penalty on the scaling factor γ of the BN layer in the network so that it gradually becomes sparse during the training process; the pruning stage prunes according to the value of the scaling factor γ of the BN layer, that is, the parameters of the filter corresponding to the smaller scaling factor γ are set to zero.

[0112] The training phase of the PointNet model includes forward derivation, which inputs tensor data with a shape of 24*3*1024 into PointNet. The model processes the data and outputs the classification results and corresponding confidence levels; gradient backpropagation updates the model parameters.

[0113] The loss is calculated based on the true labels of the PointNet model output data and input data, and then the gradient value of each filter parameter of the PointNet model is calculated based on the loss, and the model parameters are updated according to the gradient value.

[0114] The overall parameter update process of the PointNet model is:

[0115]

[0116] Where W is the model parameter, η is the learning rate, is the loss function.

[0117] Specifically, for the jth filter of the i-th convolutional layer in the PointNet model, its parameters are The update process can be broken down into:

[0118]

[0119] Where * is a convolution operation or a matrix multiplication operation, which depends on the type of layer i. If it is a convolution layer, then * is a convolution operation, if it is a fully connected layer, then it is a matrix multiplication, σ(·) is the activation function, for The output, is the output of the jth filter of the i-th layer, which can be expressed as:

[0120]

[0121] where a i-1 is the output of all filters in the i-1th layer, is the bias term of the jth filter in the i-th layer, and its parameter update process is:

[0122]

[0123] The following describes the defects of existing soft pruning methods when facing ReLU-like activation functions:

[0124] Assume that the activation function of layer i is ReLU(z)=max(0,z), which satisfies: when z<0,ReLU'(z)=0, z>0,ReLU'(z)=1; and for z=0, the value of ReLU'(z) is not fixed, it can be 0 or 1 (that is, the sub-gradient of ReLU at 0 is not fixed); in fact, for deep learning frameworks such as PyTorch or TensorFlow, when z=0, ReLU'(0)=0; in formula (3), when the zeroing operation of soft pruning is When set to 0, is also 0, and in formula (2), 0 results in is 0, which leads to cannot be updated; and This has been set to zero before, which means that when the soft pruning method faces the ReLU-like activation function, the zeroed filters cannot be activated, resulting in the pruned filter set (in Figure 1 The pruning pool (called the Pruning Pool) cannot be updated, which makes it impossible to expand the parameter optimization space during training and the actual effect of the soft pruning method cannot be brought into play.

[0125] This application proposes a solution to the above defects. When using a parameter optimizer with historical gradients, even though the activation function such as ReLU makes the gradient of the current round 0, the parameters can still be updated by taking the historical gradients into account. Figure 1 The existence of pre-pruning makes the historical gradient of the pruned filter zero from the beginning, which is unacceptable. Therefore, the present invention proposes an improvement to the existing filter soft pruning method, that is, removing the pre-pruning, such as Figure 2 As shown by the red × in the middle, the soft pruning method can actually work under the influence of historical gradients.

[0126] The following is an explanation of the shortcomings of the L2-norm-based pruning criteria used in existing soft pruning methods:

[0127] The zeroing operation of soft pruning sets the parameters of the filter to zero. When using the norm-based pruning standard, this will affect the activation amplitude of the pruned filter. The reason is that the norm-based pruning standard is dominated by the filter parameter value, and the zeroing operation will make the filter parameter value 0. Even if it is subsequently activated, if the activation amplitude is small, it will still be set to 0 in the next round of pruning, which will cause activation failure.

[0128] To solve the above defects, the present invention proposes to use the scaling factor γ of the BN layer as the pruning criterion during soft pruning training. In order to increase the sparsity of the scaling factor γ, it is necessary to add a loss about the BN layer scaling factor γ to the loss function. The formula is:

[0129]

[0130] Among them, (x, y) is the training data, x is the input point cloud data, y is the real classification label corresponding to the point cloud, f represents the PointNet model, and λ controls the magnitude of the sparse penalty, which can be adjusted from 10 -3 ,10 -4 ,10 -5 ,10 -6 In this example, select λ = 10 -5 , γ is the scaling factor, Γ is the set of scaling factors γ of all BN layers in the model, and g(γ) is the sparse penalty term, specifically: g(γ) = |γ|.

[0131] The training phase of the PointNet model includes traversing the PointNet model to obtain each convolutional layer structure that can be pruned. Some layers in the PointNet model cannot be pruned, for example, the "9" shown in MLP(9, 256) and the "4096" shown in MLP(4096, 256) in Figure 3(a). For example, for the PointNet model, the last layer in the module for input transformation, the last layer in the module for feature transformation, and the first layer in the feature extraction module cannot be pruned.

[0132] Traverse each network layer in the network structure that can be pruned in the PointNet model. Each network layer includes a convolutional layer and a BN layer. For each network layer that can be pruned, calculate the pruning threshold T (Threshold).

[0133] The pruning threshold T (Threshold) can be calculated by first counting the number of filters M contained in this layer, that is, the capacity of this layer, and then calculating the pruning threshold T subscript idx based on the preset pruning rate P:

[0134]

[0135] Then, all BN layers in this layer of network structure are sorted according to the scaling factor γ of the BN layer, and then the corresponding scaling factor γ value is taken out according to idx. This value is the pruning threshold T.

[0136] Traverse the filters in this layer of network structure, obtain the scaling factor γ of the BN layer corresponding to each filter, and determine whether it is greater than the pruning threshold T. If it is greater, no processing is performed; otherwise, the parameters of the filter are set to zero.

[0137] After training, the distribution of the scaling factor γ of the BN layer in the model has been trained to be sparse enough. At this time, each layer of the model is pruned using the scaling factor γ as the pruning criterion to obtain a lightweight model.

[0138] According to the above pruning stage, the filters that need to be pruned in each layer are determined. Let the set of these filters be PF (Pruned Filters). Correspondingly, the set of all filters in this layer is F (Filters), satisfying M = |F|.

[0139] Construct the filter capacity M' of the new network layer:

[0140] M′=M-|PF|

[0141] Construct a new convolutional layer with M' filters and a corresponding BN layer. Then copy the parameters of the unpruned filters to the filters of the new convolutional layer. The BN layer is processed in the same way. In this way, a compressed network layer is obtained with the non-zero parameters of the corresponding network layer of the original model.

[0142] When constructing the next new network layer, it is necessary to consider the impact of the previous layer. For example, if the filter capacity of the newly constructed previous layer is reduced to M' compared to the original structure, then when constructing the network layer, the number of channels of each filter needs to be reduced to M' accordingly. If the previous layer has not been reconstructed (that is, the previous layer is a structure that cannot be pruned, such as the first layer of the feature extraction module of the PointNet model), there is no need to reduce the number of filter channels and it can be kept the same.

[0143] Save the parameters of the newly constructed neural network model and the corresponding model structure to facilitate model construction during deployment and inference. This will yield a compressed lightweight PointNet model.

[0144] For the PointNet model, the baseline model accuracy is 90.7 / 86.5. When the pruning rate is set to 10%, 40%, and 50%, it can achieve better performance than the baseline model, which are 90.9 / 87.3, 90.8 / 86.9, and 90.2 / 87.0 respectively. At other pruning rates such as 60% and 70%, it can achieve better accuracy than the original soft pruning method. If the λ parameter is adjusted, such as setting λ = 10 -6, with a pruning rate of 20%, it can even achieve an accuracy of 91.0 / 86.2. For the PointNet++ model, its baseline model accuracy is 93.0 / 90.0. When the pruning rate is set to 20%, 30%, 40%, and 50%, it can achieve better accuracy than the original soft pruning method. At 20%, it can achieve the same accuracy as the baseline model (92.6 / 90.0). The above data shows that after optimizing the soft pruning process and pruning standards, the soft pruning method can achieve a good balance between pruning rate and model performance.

[0145] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

[0146] It should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like, indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be understood as limiting the present invention.

[0147] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0148] In the present invention, unless otherwise specified or limited, the terms "installed," "connected," "connect," "fixed," etc. should be understood in a broad sense. For example, they can refer to fixed connection, detachable connection, or integration; mechanical connection, electrical connection, or communication; direct connection or indirect connection through an intermediate medium; internal communication between two elements or interaction between two elements, unless otherwise specified. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0149] In the present invention, unless otherwise expressly specified or limited, when a first feature is "above" or "below" a second feature, it may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediary. Furthermore, when a first feature is "above," "above," or "above" a second feature, it may mean that the first feature is directly above or diagonally above the second feature, or simply means that the first feature is at a higher level than the second feature. When a first feature is "below," "below," or "below" a second feature, it may mean that the first feature is directly below or diagonally below the second feature, or simply means that the first feature is at a lower level than the second feature.

Claims

1. An improved filter soft pruning method for convolutional neural network model, characterized in that: The following steps are involved: S1. Select a neural network model and set parameters for the neural network model; S2. Train the parameterized neural network model, update the neural network model setting parameters, and perform a sparse penalty on the scaling factor γ of the BN layer of the neural network model while updating the neural network model setting parameters; The preprocessing of input data includes: if the 2D classification model is selected, the input data is an image, and the input data preprocessing includes unifying the image size and normalizing the image, and converting the input data into tensor type data after processing; if the 3D classification model is selected, the input data is a point cloud, and the input data preprocessing operation is point sampling, and converting the input point cloud data into tensor type data after processing; The steps of performing sparse penalty on the scaling factor γ of the BN layer of the neural network model while updating the neural network model setting parameters in S2 are: S221. Calculate the gradient value of each filter parameter in the neural network model based on the input data and output data of the neural network model. The update formula of the filter parameter is: Where W is the model parameter, η is the learning rate, is the loss function; S222. In the loss function The loss of adding the BN layer scaling factor γ is: Where (x, y) is the training data of the neural network model, x is the input data, y is the corresponding true value label, f represents the neural network model, λ is the sparse penalty amplitude value, γ is the scaling factor, Γ is the set of scaling factors γ of all BN layers in the neural network model, and g(·) is the sparse penalty term; S3, setting the filter parameters of some convolutional layers of the neural network model to zero according to the scaling factor γ after the sparsity penalty; The step of setting some filter parameters of the neural network model to zero according to the scaling factor γ after the sparse penalty in S3 is: S221, traverse the neural network model and count the number M of filters in each convolutional layer of the neural network model; S222. Calculate the pruning threshold T subscript idx based on the preset pruning rate P: S223. Sort the filters in the convolutional layer according to the scaling factor γ of the BN layer corresponding to the convolutional layer; S224: After the filters are sorted, a boundary filter is determined based on idx, and the value of the scaling factor γ corresponding to the boundary filter is the convolutional layer pruning threshold T; S225, setting the filter parameters corresponding to the scaling factor γ in the convolutional layer that is less than the pruning threshold T to zero; S4. Prune the neural network model according to the zeroing of the convolutional layer filter parameters to obtain a lightweight neural network model.

2. The improved filter soft pruning method for a convolutional neural network model according to claim 1, characterized in that: The parameter setting of the neural network model in S1 includes preprocessing of input data, initialization of model parameters and initialization of parameter optimizer.

3. The improved filter soft pruning method for a convolutional neural network model according to claim 1, characterized in that: The neural network model in S2 is trained to update the neural network model setting parameters, including the following steps: S21, forward deducing the neural network model using input data to obtain output data and corresponding confidence levels; S22. Based on the output data and the corresponding confidence level, the neural network model is gradient-backed to update the model parameters.

4. The improved filter soft pruning method for a convolutional neural network model according to claim 3, characterized in that: The steps of pruning the neural network model according to the zeroing of the convolutional layer filter parameters in S4 are: S31. Prune the current convolution layer according to the zero setting of the convolution layer filter parameters. The number of convolution layer filters after pruning is M'. Then M′=M-|PF| Where PF is the number of parameter-zero filters in the current convolutional layer; S32, use M' filters to reconstruct the convolution layer and BN layer; S34, continue to prune the next convolutional layer; S35. Repeat steps S31 to S34 until the neural network model is traversed.

5. The improved filter soft pruning method for a convolutional neural network model according to claim 4, characterized in that: The steps of reconstructing the convolution layer and the BN layer using M' filters in S32 are: S321, copy the M' filter parameters of the convolutional layer that have not been pruned and the corresponding BN layer parameters; S322, filling the copied parameters into the reconstructed convolutional layer and BN layer; S323. Set the number of filter channels in each filter in the reconstructed convolutional layer to the number of filters in the previous convolutional layer.

Images