Structured pruning method and device for lightweight neural network, medium and equipment

Abstract

The invention discloses a structured pruning method for a lightweight neural network. The method comprises the following steps: constructing a similarity matrix according to a three-dimensional tensorFilter in a to-be-pruned target layer in the neural network; clustering the three-dimensional tensor Filter corresponding to the similarity matrix based on spectral clustering to obtain a plurality of clustering clusters; determining the centroid of each clustering cluster and the distance between each three-dimensional tensor Filter in the same clustering cluster and the centroid; and deleting the three-dimensional tensor Filter corresponding to the centroid distance exceeding a set threshold to obtain a target neural network model. As a structured pruning method, the weight matrixes of theneural network subjected to structured pruning do not have an unstructured sparse phenomenon, existing software and hardware can be directly used for acceleration, and the invention can be very naturally compared with other lightweight neural network technologies. Knowledge distillation, weight quantification and the like are combined for use to further reduce the network redundancy.

Description

technical field [0001] The invention relates to the technical field of network compression, in particular to a structured pruning method, device, medium and equipment for a lightweight neural network. Background technique [0002] As the technical carrier of deep learning, the neural network model has greatly refreshed the performance accuracy of traditional signal processing methods in many fields. As deep learning tasks become more and more complex, the number of layers of the neural network model is gradually increasing, and the width is also increasing. Neural network models with increasingly complex topological structures may be able to run efficiently on servers with expensive Graphics Processing Units (GPUs); but because of their huge parameter complexity and computational complexity, it is difficult Stable operation on mobile terminal equipment. However, with the advent of the mobile intelligent society, mobile terminal applications such as mobile phones, wearable ...

AI Technical Summary

Problems solved by technology

Structured pruning methods (such as figure 1 Shown) is performed on the Kernel level (2-dimensional tensor) or Filter level (3-dimensional tensor) of the neural network model, because the pruning granularity is large, which may lead to wrong pruning, too much pruning or too little Pruning and other phenomena

[0004] The existing structured pruning methods basically use the importance criterion to discover the redundancy of the neural network, and the importance criterion of the structural parameters is actually difficult to obtain because of the internal unexplainable black box characteristics of the neural network. greatly affects the pruning performance

Images