A neural network pruning method based on similarity regularization

Abstract

This invention discloses a neural network pruning method based on similarity regularization. The method comprises: 1) pre-training the neural network to be processed; 2) pruning the pre-trained neural network: first, setting the sparsity of the neural network to s%; then calculating the s-th quantile of the neural network parameters, and globally setting parameters whose absolute values ​​are less than this s-th quantile to zero, obtaining a mask m; updating the weight matrix w of the neural network based on the obtained mask m; 3) fine-tuning the pruned neural network. This invention reduces the dependence of the output on the input by increasing the cosine similarity between the network weight matrix and the zero matrix, thereby improving the robustness of prediction. This way, only a small portion of the input participates in the final calculation, while most of the input is discarded; therefore, not only is the prediction performance of the sparse network improved, but the neural network can also achieve higher sparsity.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and is an improved neural network pruning method that introduces similarity regularization. Background Technology

[0002] As a branch of artificial intelligence, neural networks have been widely used in computer vision, natural language processing, and other fields. However, a neural network often consists of tens of millions of parameters, which brings huge costs to model training and inference. Therefore, researchers have proposed many model compression methods. Among these methods, neural network pruning reduces the number of parameters by directly removing redundant weights. The model obtained by pruning a pre-trained dense network is called a sparse network, and the ratio of the removed weights to the original number of weights is denoted as sparsity. The higher the sparsity of the network, the smaller the number of parameters. Although its performance will be affected to some extent, the required computation and storage overhead is significantly reduced. Therefore, the industry often deploys extremely sparse networks on edge devices with weak computing power to improve response speed.

[0003] To improve the accuracy of sparse networks, Han et al. first proposed applying L2 regularization during the pre-training phase to simplify dense networks at NeurIPS 2015. Later, Liu et al. found that L1 regularization was even more effective at ICCV 2017. Generally, applying higher penalty weights during pre-training before network pruning can enhance the effect of regularization, leading to better performance of the resulting sparse network. However, excessively high penalty weights can interfere with the training process. Therefore, previous works have had to set very small penalty weights, resulting in limited performance improvements for sparse networks. Summary of the Invention

[0004] To address the problems existing in current technologies, the present invention aims to provide a neural network pruning method based on similarity regularization. This method introduces a strong regularization effect into neural network training, thereby simplifying the subsequent pruning process. Unlike previous norm-based regularization terms that make weights numerically close to zero, this invention makes weights statistically similar to zero. Specifically, by increasing the cosine similarity between the network weight matrix and the zero matrix, the dependence of the output on the input is reduced, thus improving the robustness of prediction. This way, only a small portion of the input participates in the final calculation, while most input is discarded. Correspondingly, the weights of the discarded inputs can also be safely pruned. Therefore, not only is the prediction performance of sparse networks improved, but neural networks can also achieve higher sparsity.

[0005] The technical solution of this invention is as follows:

[0006] A neural network pruning method based on similarity regularization includes the following steps:

[0007] 1) Pre-train the neural network to be processed:

[0008] 11) To calculate the weights w of the i-th layer of the neural network i Similarity to the 0 matrix is ​​first determined by its w. i Standardization is performed to eliminate dimensions, resulting in standardized weights. Calculate standardized weights The cosine similarity γ(w) with a 0.5 matrix (i.e., a regularized 0 matrix) i The neural network has n layers, where i = 1 to n, and n is a natural number greater than 1; the neural network is a neural network that performs one-way transmission of compressed features internally.

[0009] 12) Average the cosine similarities of all layers in the neural network to obtain the overall similarity Γ(w) between the neural network and the 0 matrix; then add the overall similarity Γ(w) to the loss function of the neural network to obtain the loss function. Where m is the total number of training samples in the training sample set x, and y j For the j-th training sample x j The label CE stands for cross-entropy function, λ is the penalty weight, and f w (x j ) is a neural network f w For the input training sample x j The predicted output results;

[0010] 13) Train the neural network using the training sample set x corresponding to the neural network, and use the obtained loss value to generate a gradient to optimize the parameters of the neural network;

[0011] 14) Repeat steps 11-13) multiple times to perform pre-training of the neural network;

[0012] 2) Pruning the pre-trained neural network: First, set the sparsity of the neural network to s%; then calculate the s-th quantile of the neural network parameters, and set the parameters whose absolute value is less than the s-th quantile to zero globally to obtain the mask m; update the weight matrix w of the neural network according to the obtained mask m.

[0013] 3) Fine-tune the pruned neural network:

[0014] 31) Update the loss function of the neural network to

[0015] 32) Train the neural network using the training sample set x, generate a gradient using the obtained loss value, and then optimize the parameters of the neural network by setting the obtained gradient to zero using a mask m;

[0016] 33) Repeat step 32) multiple times to fine-tune the neural network.

[0017] Furthermore, the neural network is a feedforward neural network or a convolutional neural network.

[0018] Furthermore, the weight matrix w of the neural network is updated using the formula w = w⊙m.

[0019] Furthermore, |·| represents the L1 norm, and ||·| represents the L2 norm. 0.5 It is a matrix of the same shape as w, with all internal elements being 0.5.

[0020] Furthermore, the neural network is a Faster RCNN network used for object detection tasks in edge devices.

[0021] Furthermore, the training samples are face images, and the target detection task is a face recognition task.

[0022] Furthermore, the training samples are vehicle images, and the target detection task is a license plate recognition task.

[0023] An image feature extraction method is characterized in that an image to be processed is input into a neural network obtained by the above method for feature extraction to obtain the features of the image to be processed; wherein the neural network is a CNN backbone network.

[0024] A face recognition method is characterized in that a face image to be recognized is input into a face recognition model to obtain a matching face; wherein the face recognition model adopts a neural network obtained by the above method.

[0025] A machine translation method is characterized in that a sentence to be translated is input into a neural network obtained by the above method for translation, and a translation result of the sentence to be translated is obtained; wherein the neural network is a large-scale pre-trained model.

[0026] This invention can be used not only for neural network models of target detection tasks deployed in edge computing scenarios such as face recognition and license plate recognition, but also to simplify large-scale pre-trained language models based on the Transformer architecture, thereby significantly reducing the deployment cost of the models. For example, face recognition and license plate recognition systems are typically deployed at the entrances of residential communities and office buildings, with all components such as cameras, display screens, and inference chips concentrated within the same edge device. Generally, real-time images captured by cameras are first processed by a backbone network for feature extraction, and then candidate boxes are generated via an RPN to achieve real-time target selection and classification. Due to cost constraints, these edge devices often have limited computing power, but still require high response speed and accuracy from the algorithm. Therefore, this invention can be applied to pruning the CNN backbone network in feature extraction, reducing the computing power and storage requirements of the device for model deployment by removing a large number of parameters, while improving response speed and quickly completing the feature extraction step for faces, allowing the feature map to be passed to subsequent steps to accelerate the recognition process of face or license plate information.

[0027] For large-scale pre-trained models, taking machine translation as an example, the user input sentence is segmented and dimensionality reduced before being fed into an encoder-decoder-based Transformer architecture, where each attention module has multiple layers of feedforward neural networks for feature alignment. These feedforward networks typically have a large number of redundant parameters, and the resulting matrix multiplication operations also impose a significant computational burden on model deployment and online inference. This invention can be applied to sparsify these feedforward neural networks, enabling machine translation service providers to complete online inference and translation result display faster under the same hardware conditions.

[0028] Example of application scenario:

[0029] This invention primarily modifies the loss function during neural network training. For an n-layer neural network, the optimization objective applied during training is as follows:

[0030]

[0031]

[0032] Where L represents the optimization objective, w represents the weights in the network, and x represents a training set with m training data points. In computer vision applications such as face recognition and license plate recognition, x... j This represents the 3D RGB values ​​of all the pixels that make up an image. In machine translation scenarios, x... j This represents the sequence of word vectors that make up a sentence. w (x j ) represents model w with respect to the j-th training data x in the training set x. jThe result of the reasoning. j For training sample x j The label CE stands for cross-entropy function. λ is the penalty weight for each layer w in the neural network. i In other words, k i This is the number of weights in the current layer, calculated by the cosine similarity between the weights of this layer and the zero matrix, i.e., γ(w i The similarity of all n layers is weighted and averaged before being added to the training objective, thus completing the present invention.

[0033] The advantages of this invention are as follows:

[0034] This invention proposes a similarity-based regularization term that can reduce interference with the pre-training process while introducing a strong regularization effect, thereby improving the accuracy of extremely sparse networks. Comparative analysis confirms that this invention significantly improves the accuracy of sparse networks, achieving better accuracy performance compared to other methods. Attached Figure Description

[0035] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0036] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0037] This invention can only be used in neural networks where compressed features are passed one-way internally, such as feedforward neural networks and convolutional neural networks. It is not suitable for scenarios where there are temporal relationships between input features, such as recurrent neural networks and graph networks.

[0038] Neural network pruning consists of three steps: pre-training, pruning, and fine-tuning. For a neural network with n convolutional or fully connected layers, after specifying the number of training rounds E, E rounds of pre-training are first performed. Then, the neural network is pruned to remove redundant parameters. Finally, E rounds of fine-tuning are performed to restore accuracy. This invention mainly changes the loss function during neural network pre-training, thereby simplifying the subsequent pruning process. It mainly includes the following steps:

[0039] 1. Pre-training

[0040] 1.1 Weights w of the i-th layer of the neural network i To eliminate dimensions, the following standardization is performed:

[0041]

[0042] in The elements are all matrices w i The matrix that has the minimum value and the same shape as it. It is matrix w i The maximum value in.

[0043] 1.2 For the i-th layer, calculate its cosine similarity to the 0 matrix. After regularizing both matrices, obtain... With 0.5 matrix w 0.5 We obtain the following formula:

[0044]

[0045] Where k i It is w i The number of elements in the matrix, |·| is the L1 norm, and ‖·‖ is the L2 norm.

[0046] 1.3 The average similarity of all layers in the neural network is taken to obtain the overall similarity of the network as follows:

[0047]

[0048] 1.4 Add the overall network similarity obtained in the previous step to the training objective of the current neural network:

[0049]

[0050] Where L represents the optimization objective, w represents the weights in the network, x represents a training set with m data points, and y represents the weights in the network. j For the j-th training sample x j The label CE stands for cross-entropy function. λ is the penalty weight, which needs to be specified manually.

[0051] 1.5 By calculating the gradient of L with respect to the weight w, gradient descent is performed on the neural network, as shown in the following formula for the p-th parameter of the j-th layer of the network at time t. Update:

[0052]

[0053] Here, α is the learning rate, which needs to be manually specified in advance, and is generally set to 0.1 or 0.001. Let x be the empirical loss of the neural network on the dataset x.

[0054] 1.6 Repeat steps 1.1-1.5 a total of E times to complete the pre-training.

[0055] 2. Pruning

[0056] 2.1 First, manually specify the network sparsity s%, where s is a natural number greater than zero and less than 100, that is, the proportion of zero in the network parameters relative to the global value.

[0057] 2.2 Calculate the s-th quantile of the network parameters, and globally set all parameters whose absolute values ​​are less than this quantile to zero to obtain the mask m.

[0058] 2.3 Update the network parameters w = w⊙m, that is, use the Hadamard multiplication to set the parameters to zero.

[0059] 3. Fine-tuning

[0060] 3.1 Calculate the training objective for the neural network:

[0061]

[0062] 3.2 The neural network is trained by generating gradients using the training objective as follows:

[0063]

[0064] However, the generated gradient is set to zero using the mask m obtained in step 2 to prevent the pruned parameters from being updated. 3.3 Repeat steps 3.1-3.3 a total of E times to complete the fine-tuning.

[0065] Scenario Examples

[0066] The following application scenarios use object detection tasks as an example, considering the use of the Faster R-CNN network in edge devices. As mentioned earlier, such methods first require using a convolutional neural network to extract features from the images captured by the camera. Therefore, this invention uses model compression for the backbone network involved in this step. Scenario 1 is a face recognition task, and scenario 2 is a license plate recognition task.

[0067] Scenario 1: For face recognition tasks, the distance between a face and the camera is often quite close in reality, allowing for the extraction of richer features. Therefore, an ImageNet image set with an input size of 224*224 is used as an example to verify the effectiveness of this invention. Specifically, the images are first input into a compressed backbone network, EfficientNet-B2, and the effectiveness of this invention is verified by its recognition accuracy on the validation set. Simultaneously, test results on another COCO image dataset, i.e., the comparison between the candidate boxes generated by the network and the actual answers, also reflect that it maintains high performance even with a significant reduction in the number of parameters.

[0068] Table 1. Test accuracy (%) of Sparse EfficientNet-B2 on ImageNet

[0069] l 90％ 95％ 99％ 0 71.03 66.95 25.83 0.001 72.03 66.81 27.05 0.01 70.86 66.99 28.19

[0070] Table 2. Faster-RCNN recognition results on COCO (mAP@.5%)

[0071] l 90％ 95％ 99％ 0 38.1 37.6 30.8 0.001 40.5 38.5 32.6 0.01 42.3 41.9 33.1

[0072] Scenario 2: For license plate recognition tasks, the distance between the camera and the vehicle is often far, and the conditions to be considered are more complex, such as lighting conditions and visibility. Therefore, the Tiny-ImageNet image dataset with an input of 64*64 pixels is used to verify the performance. By inputting the images into a compressed ResNet-18 neural network, the output classification prediction results are compared with the true labels, and the results are shown in Table 3.

[0073] Table 3. Test accuracy (%) of ResNet-18 on Tiny-ImageNet

[0074] l 95％ 99％ 99.9％ 0 60.03 52.14 30.44 0.1 60.54 53.02 30.43 0.5 61.02 55.19 36.90

[0075] Effect Analysis:

[0076] Table 4 compares the performance of this invention with several other regularization terms on AlexNet at different sparsities. Table 5 uses ResNet-20 as the network and CIFAR-10 as the dataset. In the comparison, confidence penalty is proposed by Pereyra et al. in the ICLR 2017 Workshop Track, while label smoothing is a regularization method proposed by Szegedy et al. in CVPR 2016. It can be seen that although these two methods help improve sparse networks to some extent, their effects are limited, and they cannot prevent a sharp decline in accuracy in extremely sparse networks. L1 is a 1-norm regularization method proposed by Liu et al. in ICCV 2017. Because it can generate sparse matrices, it is widely used in neural network pruning. L1 can achieve higher sparsity in neural networks under specific penalty weights, but the accuracy degradation is still severe. This invention significantly improves the accuracy of sparse networks, achieving better accuracy performance compared to other methods.

[0077] As can be seen from the table, the improvement of our method compared to the comparison method increases with the increase of sparsity. For AlexNet with 99.5% sparsity, the best performance that L1 can achieve is 67.53%, while the performance of our invention is 70.09%. On the other hand, in extremely sparse networks, the highest sparsity that confidence penalty and label smoothing can achieve is 99.5%, while our method can prune 99.8% of the weights, and the test accuracy is also significantly better than the existing L1 regularization term.

[0078] Table 4. Accuracy (%) of Sparse AlexNet on CIFAR-10.

[0079]

[0080]

[0081] Table 5. Accuracy (%) of Sparse ResNet-20 on CIFAR-10

[0082]

[0083] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A neural network pruning method based on similarity regularization, comprising the following steps: 1) Pre-train the neural network to be processed: 11) The weights w of the i-th layer of the neural network i Standardization is performed to eliminate dimensions, resulting in standardized weights. Calculate standardized weights The cosine similarity γ(w) with the 0.5 matrix, i.e., the regularized 0 matrix. i The neural network has n layers, where i = 1 to n, and n is a natural number greater than 1; the neural network is a neural network that compresses features and performs one-way transmission internally. 12) Average the cosine similarities of all layers in the neural network to obtain the overall similarity Γ(w) between the neural network and the 0 matrix; then add the overall similarity Γ(w) to the loss function of the neural network to obtain the loss function. Where m is the total number of training samples in the training sample set x, and y j For the j-th training sample x j The labels are: CE stands for cross-entropy function, λ is the penalty weight, and f... w (x j ) is a neural network f w For the input training sample x j The predicted output results; the training samples are face images or vehicle images; 13) Train the neural network using the training sample set x corresponding to the neural network, and use the obtained loss value to generate a gradient to optimize the parameters of the neural network; 14) Repeat steps 11-13) multiple times to perform pre-training of the neural network; 2) Pruning the pre-trained neural network: First, set the sparsity of the neural network to s%; then calculate the s-th quantile of the neural network parameters, and set the parameters whose absolute value is less than the s-th quantile to zero globally to obtain the mask m; update the weight matrix w of the neural network according to the obtained mask m. 3) Fine-tune the pruned neural network: 31) Update the loss function of the neural network to 32) Train the neural network using the training sample set x, generate a gradient using the obtained loss value, and then optimize the parameters of the neural network by setting the obtained gradient to zero using a mask m; 33) Repeat step 32) multiple times to fine-tune the neural network; 4) Target detection is performed using a fine-tuned neural network; wherein, when the training samples are face images, the target detection task is a face recognition task; when the training samples are vehicle images, the target detection task is a license plate recognition task.

2. The method according to claim 1, characterized in that, The neural network is either a feedforward neural network or a convolutional neural network.

3. The method according to claim 2, characterized in that, The weight matrix w of the neural network is updated using the formula w = w⊙m.

4. The method according to claim 1, 2, or 3, characterized in that, |·| represents the L1 norm, and ||·| represents the L2 norm. 0.5 It is a matrix whose internal elements are all 0.5 and are identical to the weight matrix w.

5. The method according to claim 1, characterized in that, The neural network is the Faster RCNN network used for object detection tasks in edge devices.

6. An image feature extraction method, characterized in that, A neural network obtained by the method described in claim 1 is input into an image to be processed to extract features, thereby obtaining the features of the image to be processed. The neural network mentioned therein is a CNN backbone network.

7. A face recognition method, characterized in that, A face image to be identified is input into a face recognition model to obtain a matching face; wherein the face recognition model adopts a neural network obtained by the method described in claim 1.

8. A machine translation method, characterized in that, A sentence to be translated is input into a neural network obtained by the method described in claim 1 for translation, and the translation result of the sentence to be translated is obtained; wherein the neural network is a large-scale pre-trained model.

Images