Neural network column sparse method based on weight saliency

Abstract

The invention discloses a neural network column sparse method based on weight saliency. When the neural network model carries out feedforward operation, the four-dimensional weight tensor can be expanded into a two-dimensional matrix in a tensor expansion mode, and therefore convolution operation is converted into matrix multiplication. Therefore, on the basis of a tensor expansion operation mode,the invention discloses a method for carrying out column sparsity on a weight matrix, namely deleting columns of the weight matrix in a neural network parameter layer to obtain a sparse model, and the dimension of the operation matrix is reduced at the moment, thereby reducing the model scale of the neural network and accelerating the operation of the neural network. Network re-training is performed on the remaining weight parameters to callback the accuracy, and re-training is stopped when the accuracy of the model is not increased any more to obtain a final model. According to the method, the deep learning model can be deployed on mobile and embedded equipment, a certain actual acceleration effect is obtained, and the application of an intelligent algorithm on a mobile terminal is promoted.

Description

technical field [0001] The invention relates to the fields of neural calculation, pattern recognition and computer vision, in particular to a method for pruning the weight of a neural network to enhance its generalization ability, reduce storage capacity and speed up operation. Background technique [0002] In recent years, Convolutional Neural Networks (CNNs) have increasingly outperformed traditional computer vision methods in classification, detection, and segmentation tasks. However, CNN also brings heavy computation and storage consumption, which hinders its deployment on mobile and embedded devices. Neural network parameter sparsity shows better performance in CNN acceleration by eliminating redundant parameters that do not contribute much to the output results. However, due to irregular sparseness caused by unstructured pruning, it is difficult to achieve acceleration on general hardware platforms. Even with sparse matrix kernels, the speedup is very limited. To so...

AI Technical Summary

Problems solved by technology

As shown in Figure 3(a) and Figure 3(b), the comparison of row and column sparsification shows that compared with the row sparsity, the column sparsity has a larger pruning space, which solves the limitation of the small pruning space in the structured row sparsity strategy sexual problems

The method of the present invention solves the problem of large storage capacity and large amount of calculation of the deep learning model represented by the convolutional neural network, and the problem that the precision drops more due to the small selection space existing in other structured row sparse work Improvements on the above, and propose an accelerated compression method for structured column sparseness

Images