Hierarchical sparse coding method of pruned deep neural network with extremely high compression ratio

Abstract

The invention provides a hierarchical sparse coding method of a pruning deep neural network with an extremely high compression ratio. The hierarchical sparse coding method comprises the steps of performing sparse processing on a hyper-parameterized DNN model by applying a pruning technology; providing a layered sparse coding method LSC, and the compression ratio of a trimmed DNN model being improved by designing a coding mechanism of metadata. A multi-process decoding mechanism is designed, so that matrix operation is supported without complete decoding, and the running memory is saved; the hierarchical sparse coding method maximizes the compression ratio by reducing the amount of metadata, and comprises the following steps: dividing a compression process into a block layer and a coding layer; in the block layer, dividing the sparse matrix into a plurality of small blocks, and then deleting zero blocks; in the coding layer, a novel marked relative indexing method SRI being provided tofurther code the non-zero value blocks; the multi-process decoding mechanism accelerating multiplication of a coding matrix in an inference phase; and finally, proving the effectiveness of the proposed LSC method relative to other sparse coding methods through experimental comparison.

Description

technical field [0001] The invention relates to a layered sparse coding method of a pruned deep neural network with a very high compression ratio, belonging to the field of machine learning. Background technique [0002] Deep Neural Networks (DNNs) have evolved into state-of-the-art in many fields, especially in computer vision, natural language processing, and audio processing. However, the massive growth of hidden layers and neurons consumes considerable hardware storage and memory bandwidth, which poses severe challenges for many resource-constrained scenarios in real-world applications. Especially the arrival of the post-Moore era has slowed down the hardware replacement cycle. Specifically, current DNNs have two major bottlenecks: 1. Conflicts with resource-constrained application platforms, such as self-driving tools, mobile phones, mobile robots, and augmented reality AR, which place a heavy burden on the energy consumption and computation of DNN models Quantity is ...

AI Technical Summary

Problems solved by technology

Some new methods have been proposed recently, such as bitmasks and relative indexing, which are able to encode sparse models, but they are procedural methods that are difficult to implement in parallel

In order to make full use of the resources of deep learning accelerators, a series of novel sparse coding methods have been proposed in recent years, including block compressed sparse column BCSC and nested bit mask NB, which are suitable for parallelization environment, but the compressed model still consumes Massive storage and memory bandwidth

[0006] A major challenge of the above two methods is that it is difficult to achieve high compression ratio and efficient calculation at the same time

Images