A method for sharing internal parameters of convolution kernels in deep learning networks

Abstract

The present invention discloses a method for sharing internal parameters of a deep learning network convolution kernel, including: (1) pre-training the network; (2) clustering the convolution input channels of each layer of the network in (1) using the Kmeans clustering function; (3) sharing a two-dimensional matrix parameter for the input channels belonging to the same category in (2); (4) retraining the network to improve the prediction results. According to tests, the present invention can reduce the parameters and calculation amount of the convolutional neural network by more than 40% without substantially affecting the accuracy, speeding up the reasoning speed of the neural network, and at a low sharing rate, can even improve the reasoning accuracy of the network test set, making its accuracy surpass the original neural network that has not been optimized by the method under the same training set and the same training batch.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method for sharing internal parameters of a deep learning network convolution kernel. Background Art

[0002] Convolutional Neural Network (CNN) is a feedforward neural network whose artificial neurons can respond to surrounding units within a certain coverage area, and performs well in large image processing. Due to the invariance of image translation, scaling and rotation, convolutional neural networks are widely used in the field of image recognition. Microsoft uses convolutional neural networks to make Arabic and Chinese handwriting recognition systems, and Google uses convolutional neural networks to recognize faces and license plates in street view images, etc.

[0003] Convolutional neural networks have developed rapidly, and their recognition accuracy has gradually increased, but this has come with a huge cost in terms of parameters and computation. For example, the winner of the ImageNet challenge increased the classification accuracy from 84.7% (AlexNet) in 2012 to 96.5% (ResNet-152) in 2015, but the computational effort increased from 1.4×10 10 FLOPs surged to 2.26×10 11 FLOPs, traditional CPUs are helpless for such a huge network. Only GPUs with high computing power can train the network relatively quickly. However, high-performance GPUs will inevitably bring huge power consumption, and its heat dissipation problem has become a challenge for embedded devices. Therefore, reducing the number of model parameters and the amount of calculation has become an urgent issue for neural network applications.

[0004] In order to reduce the parameters and computational complexity of the network, researchers have proposed many compression methods for neural network models, which can be divided into four categories: (1) parameter pruning and sharing; (2) low-rank factorization; (3) transfer / compact convolution filters; and (4) knowledge distillation. Parameter pruning and sharing were originally used to solve the overfitting problem, but are now more often used to reduce network complexity. Traditional parameter sharing results in high precision loss. Pruned networks are often difficult to train and suffer from precision loss. The invention proposed in this article mainly uses convolution and internal parameter sharing to further improve the efficiency of parameter sharing. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method for sharing internal parameters of a convolution kernel, which changes the structure of a neural network, reduces the amount of computation of the network, compresses the network parameters, and thus improves the reasoning speed.

[0006] To solve the above problems, the present invention provides a method for sharing internal parameters of a deep learning network convolution kernel, comprising the following steps:

[0007] A method for sharing internal parameters of a deep learning network convolution kernel comprises the following steps:

[0008] Step 1: Call the clustering method to cluster the two-dimensional matrix parameters corresponding to the input channel of each convolution kernel of the neural network model to be optimized, and divide the two-dimensional matrix parameters with the same characteristics into one category;

[0009] Step 2: Make the same type of two-dimensional matrix parameters share a new two-dimensional matrix parameter; by sharing, the parameters and calculation amount are reduced, and the total number of input channels minus the number of clustered categories is the reduced number of parameters;

[0010] Step 3: Design a new neural network model based on the new two-dimensional matrix parameters;

[0011] Step 4: The new neural network model uses the matrix point multiplication distribution law to change the original calculation method; the input channels that share a parameter are first added and then matrix multiplied. If there are n two-dimensional parameters in a clustering category, (n-1) addition operations and one multiplication operation are required, which reduces (n-1) multiplication operations compared to the original convolution method;

[0012] Step 5: Train the new neural network model, calculate the gradient of each neuron, and update the shared weight parameters; repeatedly adjust the shared parameters until the predetermined accuracy is achieved.

[0013] Furthermore, before step 1, the neural network model to be optimized is pre-trained so that the accuracy of the network to be optimized in the test set reaches a high level. Before clustering, the network is trained to a level with a high accuracy, and cluster sharing is performed on this basis, which has little impact on the accuracy.

[0014] Furthermore, by calling Kmeans clustering or other clustering methods, each input channel corresponding to the convolution kernel is clustered. Kmeans is an unsupervised clustering algorithm. The clustering effect is excellent and the convergence speed is fast. For a given sample set, the samples are divided into K clusters according to the distance between the samples, so that the points in the cluster are connected as closely as possible, and the distance between clusters is as large as possible. Parameters with the same characteristics through Kmeans clustering are clustered into one category, and the input channels of one category share one parameter.

[0015] Further, in step 2, the two-dimensional matrix parameters before sharing are expressed as where w i represents the weight of the i-th layer, L represents the number of layers of the convolutional neural network, and N i Represents the number of input channels of the i-th layer; according to clustering, for the weight parameter w i Any convolution kernel w nGet K clusters: C n =C1,C2,…C k , each cluster C j , 1≦j≦k, the parameters contained in are the parameters that need to be optimized and shared; the sharing method is:

[0016]

[0017] W i =f(C n ,w i )for 1≦n≦N i+1 (2)

[0018] Where W n is an optimized convolution kernel, W i is the optimized weight parameter of the i-th layer, and w is the parameter within the j-th cluster.

[0019] Furthermore, in step 4, when training a new neural network model, all input feature maps that share the same parameter are first added and then convolved using the distributive law of matrix dot multiplication. The formula is:

[0020]

[0021] Where W a×b (a <N i+1 ,b <N i ) is the two-dimensional matrix parameter of the a-th channel of the i-th layer corresponding to the b-th input feature map, N i is the number of input channels of the i-th layer is the input, is the output.

[0022] Furthermore, the back propagation algorithm is used to fine-tune both the pre-training of the neural network model to be optimized and the training of the new neural network model.

[0023] The beneficial effects of the present invention are as follows: the present invention proposes a method for sharing internal parameters of a deep learning network convolution kernel, by clustering and sharing convolution kernel input channels with the same characteristics, so that two or more input channels with the same characteristics share one parameter, and improves the convolution kernel operation method, and the input images sharing one parameter can be added first and then multiplied by the weight parameter, thereby greatly reducing the number of parameters and the amount of calculation while ensuring the accuracy of the calculation. The experiment uses VGG16 and Resnet50 to test cifar10, cifar100, and Imagenet data sets.

[0024] The present invention can be used in various mobile embedded devices, and is particularly suitable for autonomous driving technology. For normal autonomous driving input of images of size 1920×1024, GPU is used for inference, and only 11-20 images can be inferred per second, which is not allowed for occasions requiring fast inference. The present invention proposes a method for sharing internal parameters of a deep learning network convolution kernel, which can reduce the amount of calculation by more than 50%. After optimization, the number of inference images per second can reach more than 90, which basically meets the needs of autonomous driving technology. In addition, the use of the distribution law in step 4 of the present invention can greatly reduce multiplication operations. We know that for ASIC chips, the area and power consumption of the multiplier are much greater than those of the addition tree. Therefore, the advantage of the present invention in reducing the number of multiplications is also of great application value for designing AI chips.

[0025] Table 1 lists the number of images inferred per second on different networks before and after the internal parameter sharing method of the convolution kernel of the learning network with different GPU applications.

[0026] BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of internal sharing of convolution kernel.

[0028] Figure 2 Schematic diagram of internal sharing of convolution kernels used in VGG13.

[0029] Figure 3 This is a network structure diagram that uses internal sharing of convolution kernels.

[0030] Figure 4 Schematic diagram of the internal sharing operation of the convolution kernel.

[0031] Figure 5 The result curve graph shared by VGG16.

[0032] Figure 6 The result curve graph shared by Resnet50.

[0033] Figure 7 It is the intermediate feature map of Rresnet50 when it is not shared.

[0034] Figure 8 It is the intermediate feature map of Resnet50 when the sharing rate is 0.7. DETAILED DESCRIPTION

[0035] A method for sharing internal parameters of a deep learning network convolution kernel has the following steps:

[0036] (1) Select a neural network model to be optimized (the present invention takes VGG13, VGG16, and Resnet50 as examples), and pre-train the neural network model to be optimized so that the accuracy of the network to be optimized in the test set reaches a high level.

[0037] (2) Cluster the parameters corresponding to each input channel of the convolution kernel by calling Kmeans clustering or other clustering methods. Kmeans is an unsupervised clustering algorithm. It has excellent clustering effect and fast convergence speed. For a given sample set, the samples are divided into K clusters according to the distance between the samples, so that the points in the clusters are connected as closely as possible, and the distance between clusters is as large as possible. Parameters with the same characteristics through Kmeans clustering are clustered into one category, and the input channels of one category share one parameter.

[0038] (3) Share the parameters of a class. After shared optimization, two or more input channels belonging to the same class share a two-dimensional matrix parameter. The shared parameter value is the arithmetic mean of the parameters corresponding to the model to be optimized. Based on the sharing results, a new neural network model is designed. In the new neural network, the distributive law of matrix dot multiplication is used to share all input feature maps that share the same parameter.

[0039] (4) Use the back propagation algorithm to train the new neural network model, find the gradient of each neuron, and update the shared weight parameters. Repeatedly adjust the shared parameters until a higher accuracy is achieved. The goal of achieving a higher accuracy is to control the accuracy drop to less than 1% compared with the original network.

[0040] The present invention proposes a method for sharing internal parameters of a deep learning network convolution kernel, wherein the convolution kernel internal parameters are shared as follows: Figure 1 As shown. Figure 1 It can be seen that the internal parameter sharing method of the convolutional neural network convolution kernel greatly reduces the connections between neurons, thereby greatly reducing the amount of calculation.

[0041] like Figure 2 As shown in the figure, the present invention does not change the network structure of VGG and Resnet, but changes the operation method of the convolution layer. For the output of the previous convolution layer, the input channel parameters of the convolution kernel are first clustered using a clustering function, and the input channels corresponding to the parameters belonging to the same category are first added and then dot-multiplied. Finally, the outputs of each convolution kernel are concat, and the obtained outputs are batch-normalized and input into the next layer through the relu function.

[0042] VGG13 using convolutional neural network convolution kernel internal parameter sharing, see Figure 3, the present invention uses pytorch to build a network training CIFAR-10, CIFAR100, ImageNet datasets. Taking the CIFAR-10 dataset as an example, the CIFAR-10 dataset is a three-channel color image with an image size of 32×32. Since the internal sharing of the first convolution layer and the 1*1 convolution kernel will greatly affect the accuracy, the first convolution layer kernel and the convolution layer with a convolution kernel size of 1*1 use classic convolution, and the rest use convolution kernel internal shared convolution (Internal shared convolution). The specific operation steps are as follows: first, the input image size is: 32*32*3, after a 3*3, step size of 1, padding of 1 standard convolution, the output channel is 64, then a convolution kernel size of 3*3, step size of 1, padding of 1, and the kernel size of 64 input and output channels are performed, and the kernel size is 3*3, the step size is 1, and the kernel size is 1. Shared convolution, after pooling and batch normalization, enter the second stage. The second stage includes 2 shared convolutions within the kernel, with output channels of 128, stride length, and padding of 1. After pooling and batch normalization, it enters the third stage. The third stage includes 3 convolutions with output channels of 128. The first two shared convolution kernels are 3*3, and the last standard convolution kernel is 1*1. After pooling and batch normalization, it enters the fourth stage. The fourth stage includes 2 3*3 internal shared convolution kernels and a 1*1 standard convolution. The output channels are 256, stride length is 1, and padding is 1. After pooling and batch normalization, it enters the fifth stage. The fifth stage includes 2 3*3 internal shared convolution kernels and a 1*1 standard convolution. The output channels are 512, stride length is 1, and padding is 1. Then it is output through 3 fully connected layers. Image classification is completed.

[0043] The operation method of internal sharing of convolution kernel is as follows Figure 4 As shown, the steps are as follows:

[0044] (1) Pre-train the network for 300 rounds;

[0045] (2) Performing convolution kernel internal sharing operations on network parameters;

[0046] (3) 3 rounds of fine-tuning;

[0047] (4) Test sharing results.

[0048] Wherein (2), (3), and (4) are repeated continuously. Based on the deep learning CNN network improvement method, the present invention proposes the sharing of the internal parameters of the convolution kernel, which greatly reduces the amount of calculation and the amount of parameters. Compared with the classical neural network, the parameters used are fewer and the amount of calculation is lower, indicating that the deep learning network convolution kernel internal parameter sharing method proposed by the present invention is effective. The results obtained by VGG16 are shown in Table 1, and the results obtained by Resnet50 are shown in Table 2.

[0049] Table 1 VGG16 calculation results

[0050]

[0051] Table 2 Resnet50 calculation results

[0052]

[0053]

[0054] In addition, the present invention also provides an intermediate feature map of CIFAR10 processed by Resnet50 (see attached figure), where Figure 7 is the intermediate feature map of Resnet50 without sharing optimization, Figure 8 This is the intermediate feature map when the sharing rate is 0.7. Comparing the two figures, it can be seen that the sharing optimization has little damage to feature extraction.

[0055] From the above data, we can find that the internal sharing of convolution kernels has obvious advantages over classic convolution. For VGG convolution operations, the number of parameters can be reduced by more than 80%, and the amount of parameter calculations can be reduced by nearly 80%. For Resnet, both the number of parameters and the amount of calculations can be reduced by more than 40%, and the accuracy loss is very small. Even when the sharing rate is low, the test set score of the model is even better than that of the unoptimized model.

[0056] The above are only preferred implementation modes of the present invention. It should be pointed out that the above implementation modes do not constitute limitations on the present invention. Various changes and modifications made by relevant personnel without departing from the technical idea of ​​the present invention all fall within the protection scope of the present invention.

Claims

1. An image classification method based on deep learning network convolution kernel internal parameter sharing, characterized in that: The steps include: First, the image is input and passed through a standard convolution of 3*3, stride 1, padding 1, and output channel 64. Then, a convolution kernel size of 3*3, stride 1, padding 1, and the input and output channels are both 64. The kernel size is shared in the kernel. After pooling and batch normalization, it enters the second stage; the second stage contains 2 shared convolutions in the kernel, and the output channels are 128, stride, and padding are 1. After pooling and batch normalization, it enters the third stage. The third stage contains 3 convolutions with output channels of 128. The first two convolutions have a shared kernel size of 3*3, stride 1, and padding of 1. The shared convolution kernel is 3*3, and the latter standard convolution kernel is 1*1. After pooling and batch normalization, it enters the fourth stage; the fourth stage includes 2 3*3 internal shared convolution kernels and a 1*1 standard convolution, the output channels are 256, the step size is 1, and the padding is 1. After pooling and batch normalization, it enters the fifth stage; the fifth stage includes 2 3*3 internal shared convolution kernels and a 1*1 standard convolution, the output channels are 512, the step size is 1, and the padding is 1; then it is output through 3 fully connected layers to complete the image classification; The operation method of internal sharing of the convolution kernel includes the following steps: Step 1: Call the clustering method to cluster the two-dimensional matrix parameters corresponding to the input channel of each convolution kernel of the neural network model to be optimized, and divide the two-dimensional matrix parameters with the same characteristics into one category; Step 2: Make similar two-dimensional matrix parameters share a new two-dimensional matrix parameter; Step 3: Design a new neural network model based on the new two-dimensional matrix parameters; Step 4: Use the matrix point multiplication distributive law in the new neural network model to change the original calculation method; Step 5: Train the new neural network model, find the gradient of each neuron, and update the shared weight parameters; repeatedly adjust the shared parameters until the predetermined accuracy is achieved; Before proceeding to step 1, pre-train the neural network model to be optimized; Step 2: The two-dimensional matrix parameters before sharing are expressed as where w i represents the weight of the i-th layer, L represents the number of layers of the convolutional neural network, and N i Represents the number of input channels of the i-th layer; according to clustering, for the weight parameter w i Any convolution kernel w n Get K clusters: C n =C1,C2,…C k , each cluster C j , 1≦j≦k, the parameters contained in are the parameters that need to be optimized and shared; the sharing method is: W i =f(C n ,w i )for 1≦n≦N i+1 (2) Where W n is an optimized convolution kernel, W i is the optimized weight parameter of the i-th layer, and w is the parameter within the j-th cluster; In step 4, when training a new neural network model, all input feature maps that share the same parameters are added and then convolved using the distributive law of matrix dot multiplication. The formula is: Where W a×n (a <N i+1 ,b <N i ) is the two-dimensional matrix parameter of the a-th channel of the i-th layer corresponding to the b-th input feature map, N i is the number of input channels of the i-th layer is the input, is the output; The back propagation algorithm is used for both pre-training the neural network model to be optimized and training the new neural network model.

Images