Neural network structured sparse method based on incremental regularization

Abstract

The invention discloses a neural network structured sparse method based on incremental regularization. The method comprise the steps of when the branches of a neural network are pruned, according to the relative importance of each weight group, distributing different regularization increments to different weight groups step by step, then updating the regularization factors of the weight groups iteratively and continuously, and when the regularization factor of a certain weight group reaches a specified regularization upper limit, deleting the corresponding weight in the network permanently toincrease the structured sparsity of the network model; and when the sparsity of a certain layer reaches a preset sparsity rate, automatically stopping pruning of the layer until the pruning of all thelayers is completed; and finally, retraining the whole network to callback the accuracy, and when the accuracy of the model does not rise any more, stopping retraining to obtain a sparse model. According to the method, a large deep learning model can be deployed on the mobile and embedded equipment, a remarkable actual acceleration effect is obtained, and the application of a deep learning algorithm on a mobile terminal is promoted.

Description

technical field [0001] The invention relates to the fields of neural calculation, pattern recognition and computer vision, in particular to a method for pruning the weight of a neural network to enhance its generalization ability, reduce storage capacity and speed up operation. Background technique [0002] In recent years, deep convolutional neural networks (CNN) have achieved remarkable success in computer vision tasks by using large amounts of data to learn large-scale networks. However, CNNs usually incur heavy computation and storage consumption, hindering their deployment on mobile and embedded devices. To address this issue, much research work has focused on compressing the size of CNNs. As a more effective method, parameter pruning is used to compress and accelerate CNN models, and its purpose is to eliminate redundant model parameters within a tolerable performance loss range. However, the general parameter pruning method may generate unstructured random connectio...

AI Technical Summary

Problems solved by technology

[0004] In order to solve the problem of performance loss caused by using a large and constant regularization factor in the above-mentioned traditional regularization method to obtain a better model pruning effect, the present invention provides a neural network structure based on incremental regularization sparse method

At the same time, when pruning, this method first initializes all regularization factors to 0, and then gradually increases the regularization factors based on the relative importance of each weight group, which solves the traditional algorithm due to CNN The problem of weak expressiveness and difficult to bear large penalties at the beginning of pruning, thereby reducing the performance loss of the model

Images