GCNN network parameter reduction method based on matrix decomposition

Abstract

The invention discloses a GCNN network parameter reduction method based on matrix decomposition, and the method comprises the following steps: 1, carrying out the sparse decomposition of an input image channel and a convolution kernel, carrying out the two times of sparse decomposition of an input feature map and the convolution kernel through a conversion matrix and a sparse base, and reducing the complexity and calculation amount of convolution operation; step 2, SVD decomposition of a full connection layer: carrying out SVD decomposition on a weight matrix of the full connection layer; according to the GCNN network parameter reduction method based on matrix decomposition, sparse decomposition is carried out on an input image channel and a convolution kernel, sparse decomposition is carried out on an input feature map and the convolution kernel twice, the complexity and the calculation amount of convolution operation are reduced through conversion of a matrix and a sparse base, sparse decomposition based on a channel base and a kernel base is mainly adopted, and the convolution efficiency is improved. According to the method, SVD decomposition is carried out on a weight matrix of a full connection layer by reducing network parameters, the weight matrix is split into two matrixes with fewer parameters, the parameter quantity and calculation time of the full connection layer are remarkably reduced through SVD decomposition, and the overall efficiency of the network is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of growable convolutional neural network related algorithms, and specifically to a GCNN network parameter reduction method based on matrix decomposition. Background Art

[0002] In recent years, complex visual recognition tasks require the construction of large-scale neural networks with huge parameters, resulting in low learning efficiency. For example, in the 2014 ILSVRC competition, VGG used a deep network with 16 convolutional layers, which took 1 month to train in parallel on 4 GPUs to obtain the results. Due to the high nonlinearity of neural networks, over-parameterization and random initial weights are usually used to avoid falling into local minima during training, and the convolution kernels of each layer are usually relatively independent. All of these lead to a large number of redundant parameters in the network, further increasing the time and space complexity of network training.

[0003] In order to solve the problem of learning efficiency, low-rank matrix decomposition can be used to achieve network parameter reduction and fast learning. For example, Denil et al. used low-rank matrix decomposition on the MLP network of the MNIST dataset to achieve a 95% parameter reduction effect. Jaderberg and Denton et al. adopted a tensor low-rank expansion strategy to speed up the training of convolutional neural networks, and achieved significant acceleration effects on the scene character classification dataset and ImageNet dataset, respectively, with an accuracy drop of less than 1%. However, these low-rank matrix decomposition methods are only applicable to larger convolution kernels, but not to smaller convolution kernels, such as the 3x3 convolution kernel widely used in CNN models. At the same time, some scholars have tried other strategies to improve the learning efficiency of the network. Vanhoucke et al. studied the impact of SIMD instructions, memory alignment, and fixed-point decomposition strategies on CPU-based general neural network training. Mathieu et al. used fast Fourier transform (FFT) to achieve a 2-fold acceleration effect on AlexNet, but this method is limited by the high consumption of FFT and is only applicable to large convolution kernels. Farabet et al. used FPGA training to obtain a large-scale convolutional neural network and completed the embedded real-time recognition task.

[0004] In response to the problem of test efficiency in complex tasks such as target detection, research in recent years has mainly focused on identifying regions of interest from a large number of candidate target regions to be classified. Girshick et al. used the selective search algorithm to extract candidate regions, which reduced the number of regions of interest and improved the accuracy of detection; however, there are still nearly 2,000 candidate regions in an image, making the test phase still relatively slow. To solve this problem, He et al. proposed the SPP-net structure, which replaces the last sampling layer before the fully connected layer in the traditional convolutional neural network with a spatial pyramid pooling layer. Subsequent operations only need to be performed on the corresponding regions of interest of the obtained convolutional feature map, thereby greatly improving the efficiency of the test. Compared with these algorithms, the parameters of the fully connected layer in the convolutional neural network account for a large proportion of the overall network parameters. Research has found that the time overhead of the fully connected layer in multiple interest areas in tasks such as target detection is very large. For real-time, dynamic, and continuous data, common problems in general CNN models such as overfitting, low learning and testing efficiency, and "data hunger" still exist in the GCNN model. The structure of GCNN determines that it contains a large number of parameters, the network complexity is very high, and the training and testing time is long, which does not meet the needs of real-time and dynamic data operations. The use of GPU can improve the computing efficiency of the network, but the more economical and practical solution is to optimize the parameter model of GCNN and simplify the network operation so that it can quickly complete training and testing on the CPU platform.

[0005] In practical applications, there are mainly the following problems:

[0006] (1) Over-parameterization leads to parameter redundancy: The highly nonlinear and over-parameterized design in neural networks leads to a large number of redundant parameters, which increases the time and space complexity of training and inference;

[0007] (2) Poor adaptability of small convolution kernels: Existing low-rank matrix decomposition methods are mainly applicable to larger convolution kernels, but they are not effective for the 3x3 small convolution kernels widely used in CNNs, which limits the scope of application of these methods;

[0008] (3) The time overhead of the fully connected layer is large: In complex tasks such as object detection, the parameters of the fully connected layer account for a large proportion of the overall network parameters, resulting in huge time overhead on multiple regions of interest, which seriously affects the efficiency of the testing phase. Summary of the invention

[0009] 1. Technical issues to be resolved

[0010] In view of the shortcomings of the prior art, the present invention provides a GCNN network parameter reduction method based on matrix decomposition, which solves the problems raised in the above background technology.

[0011] (II) Technical solution

[0012] A GCNN network parameter reduction method based on matrix decomposition, characterized by comprising the following steps:

[0013] first step:

[0014] Sparse decomposition of input image channels and convolution kernels. Through the conversion matrix and sparse basis, the input feature map and convolution kernel are sparsely decomposed twice to reduce the complexity and computational complexity of the convolution operation.

[0015] Step 2:

[0016] SVD decomposition of the fully connected layer, the weight matrix of the fully connected layer is decomposed by SVD, splitting it into two matrices with fewer parameters, significantly reducing the number of parameters and calculation time of the fully connected layer.

[0017] Preferably, the specific operation steps of the first step are: in a traditional convolutional neural network, setting the input feature map of a convolutional layer in the network to I, Where h is the height, w is the width, and m is the number of channels of the input feature map: the first layer is the number of RGB channels of the image, and after the second layer, m is the number of convolution kernels in the previous layer, and the convolution kernel is K. Where s is the size of the convolution kernel, n is the number of channels of the output feature map: the number of convolution kernels in this layer, so the output feature map of this convolution layer is O = K*I, that is,

[0018]

[0019] The specific goal is to replace the convolution operation with high computational complexity in the above formula with a fast sparse version consisting of multiple sparse matrix multiplications, and perform two sparse decompositions on the convolution operation, one based on the channel basis and the other based on the kernel basis, to gradually reduce the complexity of the convolution operation in the network. The specific method is as follows;

[0020] S1. Channel-based sparse decomposition: using a transformation matrix P∈R m×m The input feature map I and convolution kernel K are converted into and So that O≈K*I holds, the conversion relationship is as follows, where the conversion matrix P is the channel basis involved in the decomposition, and R is the corresponding sparse coefficient;

[0021]

[0022]

[0023] S2, kernel-based sparse decomposition: For each input channel i=1,…,m, Can be decomposed into a matrix and the tensor Q i ∈ The decomposition conversion relationship is as follows:

[0024]

[0025]

[0026] Where Q i To decompose the nuclei involved, q i To decompose each R(·,·,i,·) required basis Q i The number of i is a sparse kernel matrix, so

[0027]

[0028] S3. After the above two steps of sparse decomposition, the input feature map I is gradually combined with the channel basis P and the kernel basis Q i And the sparse kernel matrix S i After multiplication or convolution respectively, the original output feature map O is restored. Under the condition that the matrix parameters have a high sparsity, the operation of the convolution layer of the network will be greatly reduced, thereby improving the overall efficiency of the network. Then it is necessary to solve the matrices P and Q i and S i , i=1,…,m, so that q i Much smaller than s 2 , and the matrix S i Containing a large number of zero elements reduces the number of parameters and allows the new sparse convolution kernel R to produce output close to the original convolution kernel K. The matrices P, Q i and S i That is, the network parameters required to be solved in the new network obtained after matrix decomposition;

[0029] The solution of network parameters can be carried out in two steps. First, through initial decomposition, with the goal of minimizing the reconstruction error, a sparse initial parameter matrix is ​​obtained. The initial decomposition can be achieved by the following method: using a sparse dictionary learning algorithm to decompose K and R, where P, Q i is the basis; use PCA to decompose K and R, where P, Q i is the principal component; P and Q i Initialize to the unit matrix, keeping K and R unchanged;

[0030] Then, based on the initial parameter matrix, we add sparse constraints and continue to train the network, using the network's loss function as the error measurement criterion to fine-tune the network parameters, that is, optimize the following formula:

[0031]

[0032] where L net is the logical loss function of the network output layer. The second term in the above formula adds lasso constraints to each element of S i , and the third term takes each row of S i as a group variable to add group-lasso constraints. Through the above two different lasso constraints, we incorporate sparse constraints and low-rank constraints into the network structure to further reduce the redundancy of network parameters and improve the efficiency of the entire network. Through the above two-step solution and fine-tuning of network parameters, a highly sparse convolutional network can be obtained, thus greatly improving the training efficiency of GCNN.

[0033] Preferably, the specific implementation steps of the second step are as follows: Set the weight matrix of the fully connected layer to W ∈ R u×v , whose rank is t. Then, performing SVD factorization on it can obtain W ≈ U∑ t V T , where U = [u 1 , u 2 ,..., u t ∈ R u×t , u i is the i-th left singular vector of W, ∑ t ∈ R t×t is a diagonal matrix containing the first t singular values of W, V = [v 1 , v 2 ,..., v t ∈ R v×t , v i is the i-th right singular vector of W. Considering that W contains uv parameters, and through SVD decomposition, it is split into ∑ t V T and U, the total number of parameters is reduced to t(u + v) (t << min(u, v)), greatly reducing the parameters of the fully connected layer, thereby reducing the time consumption in the fully connected layer during the test phase. Based on this, we will replace the fully connected layer of the original GCNN with two linearly cascaded fully connected layers. The weight matrix of the first fully connected layer is ∑ t V T , and the second fully connected layer regards U as the weight matrix. On the basis of sparse decomposition of the channels and convolutional kernels of the input image, by compressing the parameters of the fully connected layer, the overall efficiency of the network is further improved.

[0034] (III) Beneficial Effects

[0035] The present invention provides a method for reducing GCNN network parameters based on matrix decomposition, having the following beneficial effects:

[0036] (1) The GCNN network parameter reduction method based on matrix decomposition performs sparse decomposition on the input image channels and convolution kernels, decomposes the input feature map and convolution kernels twice, and reduces the complexity and computational complexity of the convolution operation by converting matrices and sparse bases. It mainly uses sparse decomposition based on channel basis and kernel basis to reduce network parameters. It performs SVD decomposition on the weight matrix of the fully connected layer and splits it into two matrices with fewer parameters. The SVD decomposition significantly reduces the number of parameters and computational time of the fully connected layer, thereby improving the overall efficiency of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the principle of the present invention. DETAILED DESCRIPTION

[0038] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0039] The present invention provides a technical solution: the first step:

[0040] Sparse decomposition of input image channels and convolution kernels. Through the conversion matrix and sparse basis, the input feature map and convolution kernel are sparsely decomposed twice to reduce the complexity and computational complexity of the convolution operation.

[0041] The specific operation steps of the first step are: in a traditional convolutional neural network, the input feature map of a convolutional layer in the network is set to I, Where h is the height, w is the width, and m is the number of channels of the input feature map: the first layer is the number of RGB channels of the image, and after the second layer, m is the number of convolution kernels in the previous layer, and the convolution kernel is K. Where s is the size of the convolution kernel, n is the number of channels of the output feature map: the number of convolution kernels in this layer, so the output feature map of the convolution layer is O = K * I, that is,

[0042]

[0043] The specific goal is to replace the convolution operation with high computational complexity in the above formula with a fast sparse version consisting of multiple sparse matrix multiplications, and perform two sparse decompositions on the convolution operation, one based on the channel basis and the other based on the kernel basis, to gradually reduce the complexity of the convolution operation in the network. The specific method is as follows;

[0044] S1. Channel-based sparse decomposition: using a transformation matrix P∈R m×m The input feature map I and convolution kernel K are converted into and So that O≈K*I holds, the conversion relationship is as follows, where the conversion matrix P is the channel basis involved in the decomposition, and R is the corresponding sparse coefficient;

[0045]

[0046]

[0047] S2, kernel-based sparse decomposition: For each input channel i=1,…,m, Can be decomposed into a matrix and tensors The decomposition conversion relationship is as follows:

[0048]

[0049]

[0050] Where Q i To decompose the nuclei involved, q i To decompose each R(·,·,i,·) required basis Q i The number of i is a sparse kernel matrix, so

[0051]

[0052] S3. After the above two steps of sparse decomposition, the input feature map I is gradually combined with the channel basis P and the kernel basis Q i And the sparse kernel matrix S i After multiplication or convolution respectively, the original output feature map O is restored. Under the condition that the matrix parameters have a high sparsity, the operation of the convolution layer of the network will be greatly reduced, thereby improving the overall efficiency of the network. Then it is necessary to solve the matrices P and Q i and S i , i=1,…,m, so that q i Much smaller than s 2 , and the matrix S i Containing a large number of zero elements reduces the number of parameters and allows the new sparse convolution kernel R to produce output close to the original convolution kernel K. The matrices P, Q i and S i That is, the network parameters required to be solved in the new network obtained after matrix decomposition;

[0053] The solution of network parameters can be carried out in two steps. First, through initial decomposition, with the goal of minimizing the reconstruction error, a sparse initial parameter matrix is ​​obtained. The initial decomposition can be achieved by the following method: using a sparse dictionary learning algorithm to decompose K and R, where P, Q iis the basis; use PCA to decompose K and R, where P, Q i is the principal component; P and Q i Initialize to the unit matrix, keeping K and R unchanged;

[0054] Then, based on the initial parameter matrix, we add sparse constraints and continue to train the network, using the network's loss function as the error measurement criterion to fine-tune the network parameters, that is, optimize the following formula:

[0055]

[0056] Where L net is the logistic loss function of the network output layer. The second term in the above formula is in S i Add lasso constraints to each element of S, and the third term is to i Each row of is used as a group variable to add group-lasso constraints. Through the above two different lasso constraints, we integrate sparse constraints and low-rank constraints into the network structure to further reduce the redundancy of network parameters and improve the efficiency of the entire network. Through the above two steps of network parameter solution and fine-tuning, a highly sparse convolutional network can be obtained, which greatly improves the training efficiency of GCNN.

[0057] Step 2:

[0058] SVD decomposition of the fully connected layer: Perform SVD decomposition on the weight matrix of the fully connected layer and split it into two matrices with fewer parameters, which significantly reduces the number of parameters and calculation time of the fully connected layer;

[0059] The specific implementation steps of the second step are: set the weight matrix of the fully connected layer to W∈R u×v , whose rank is t, then SVD factorization can be obtained to obtain W≈U∑ t V T , where U=[u 1 ,u 2 ,...,u t ]∈R u×t ,u i is the i-th left singular vector of W, ∑ t ∈R t×t is a diagonal matrix containing the first t singular values ​​in W, V = [v 1 ,v 2 ,...,v t ]∈R v×t , v i is the i-th right singular vector of W. Considering that W contains uv parameters, it is decomposed into ∑ t V TAfter the two parts of U, the total number of parameters is reduced to t(u + v) (t << min(u, v)), which greatly reduces the parameters of the fully connected layer, thereby reducing the time consumption of the fully connected layer in the test phase. Based on this, we will replace the fully connected layer of the original GCNN with two linearly cascaded fully connected layers, where the weight matrix of the first fully connected layer is ∑ t V T , and for the second fully connected layer, U is regarded as the weight matrix. On the basis of the sparse decomposition of the channels and convolution kernels of the input image, by compressing the parameters of the fully connected layer, the overall efficiency of the network is further improved.

[0060] In summary, the method for reducing the parameters of the GCNN network based on matrix decomposition reduces the complexity and computational amount of convolution operations by performing two sparse decompositions on the input feature map and convolution kernel through the transformation matrix and sparse basis for the sparse decomposition of the channels and convolution kernels of the input image. Then, the weight matrix of the fully connected layer is decomposed by SVD and split into two matrices with fewer parameters, significantly reducing the number of parameters and computational time of the fully connected layer.

[0061] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device.

[0062] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principle and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A GCNN network parameter reduction method based on matrix decomposition, characterized by: The following steps are involved: first step: Sparse decomposition of input image channels and convolution kernels. Through the conversion matrix and sparse basis, the input feature map and convolution kernel are sparsely decomposed twice to reduce the complexity and computational complexity of the convolution operation. Step 2: SVD decomposition of the fully connected layer, perform SVD decomposition on the weight matrix of the fully connected layer, split it into two matrices with fewer parameters, reducing the number of parameters and calculation time of the fully connected layer.

2. The GCNN network parameter reduction method based on matrix decomposition according to claim 1, characterized in that: The specific operation steps of the first step are: in a traditional convolutional neural network, the input feature map of a convolutional layer in the network is set to Where h is the height, w is the width, and m is the number of channels of the input feature map: the first layer is the number of RGB channels of the image, and after the second layer, m is the number of convolution kernels in the previous layer. The convolution kernel is Where s is the size of the convolution kernel, n is the number of channels of the output feature map: the number of convolution kernels in this layer, so the output feature map of this convolution layer is O = K*I, that is, The specific goal is to replace the convolution operation with high computational complexity in the above formula with a fast sparse version consisting of multiple sparse matrix multiplications, and perform two sparse decompositions on the convolution operation, one based on the channel basis and the other based on the kernel basis, to gradually reduce the complexity of the convolution operation in the network. The specific method is as follows; S1. Channel-based sparse decomposition: using a transformation matrix P∈R m×m The input feature map I and convolution kernel K are converted into and So that O≈K*I holds, the conversion relationship is as follows, where the conversion matrix P is the channel basis involved in the decomposition, and R is the corresponding sparse coefficient; S2, kernel-based sparse decomposition: For each input channel i=1,…,m, Can be decomposed into a matrix and tensors The decomposition conversion relationship is as follows: Where Q i To decompose the nuclei involved, q i To decompose each R(·,·,i,·) required basis Q i The number of i is a sparse kernel matrix, so S3. After the above two steps of sparse decomposition, the input feature map I is gradually combined with the channel basis P and the kernel basis Q i And the sparse kernel matrix S i After multiplication or convolution, the original output feature map O is restored, and then the matrix P and Q need to be solved. i and S i , i=1,…,m, so that q i Much smaller than s 2 , and the matrix S i Containing a large number of zero elements reduces the number of parameters and allows the new sparse convolution kernel R to produce output close to the original convolution kernel K. The matrices P, Q i and S i That is, the network parameters required to be solved in the new network obtained after matrix decomposition; The solution of network parameters can be carried out in two steps. First, through initial decomposition, with the goal of minimizing the reconstruction error, a sparse initial parameter matrix is ​​obtained. The initial decomposition can be achieved by the following method: using a sparse dictionary learning algorithm to decompose K and R, where P, Q i is the basis; use PCA to decompose K and R, where P, Q i is the principal component; P and Q i Initialize to the unit matrix, keeping K and R unchanged; Then, based on the initial parameter matrix, we add sparse constraints and continue to train the network, using the network's loss function as the error measurement criterion to fine-tune the network parameters, that is, optimize the following formula: st‖P(.,j)‖2≤1,j=1,…,m ‖Q i (.,.,k)‖2≤1,i=1,...,m,k=1,...,q i Where L net is the logistic loss function of the network output layer. The second term in the above formula is in S i Add lasso constraints to each element of S, and the third term is to i Each row of is used as a group variable to add group-lasso constraints. Through the above two different lasso constraints, we integrate sparse constraints and low-rank constraints into the network structure to further reduce the redundancy of network parameters. Through the above two steps of solving and fine-tuning the network parameters, we can obtain a highly sparse convolutional network.

3. The GCNN network parameter reduction method based on matrix decomposition according to claim 1, characterized in that: The specific implementation steps of the second step are as follows: Set the weight matrix of the fully connected layer as \(W\in\mathbb{R}\) u×v , whose rank is \(t\). Then, performing SVD factorization on it can obtain \(W\approx U\sum\) t V T , where \(U = [u_1, u_2,\cdots, u\) t \(\in\mathbb{R}\) u×t , \(u\) i is the \(i\)-th left singular vector of \(W\), \(\sum\) t \(\in\mathbb{R}\) t×t is a diagonal matrix containing the first \(t\) singular values of \(W\), \(V = [v_1, v_2,\cdots, v\) t \(\in\mathbb{R}\) v×t , \(v\) i is the \(i\)-th right singular vector of \(W\). Considering that \(W\) contains \(uv\) parameters, and through SVD decomposition, it is split into \(\sum\) t V T and \(U\) two parts, the total number of parameters is reduced to \(t(u + v)\) (\(t\ll\min(u, v)\)). Based on this, we will replace the fully connected layer of the original GCNN with two linearly cascaded fully connected layers. The weight matrix of the first fully connected layer is \(\sum\) t V T , and the second fully connected layer regards \(U\) as the weight matrix. Based on the sparse decomposition of the channels of the input image and the convolutional kernel, the parameters of the fully connected layer are compressed.

Images