Systems and methods for block-sparse recurrent neural networks

Abstract

Described herein are systems and methods to prune deep neural network models in reducing the overall memory and compute requirements of these models. It is demonstrated that using block pruning and group lasso combined with pruning during training, block-sparse recurrent neural networks (RNNs) may be built as accurate as dense baseline models. Two different approaches are disclosed to induce blocksparsity in neural network models: pruning blocks of weights in a layer and using group lasso regularization to create blocks of weights with zeros. Using these techniques, it is demonstrated that block-sparse RNNs with high sparsity can be created with small loss in accuracy. Block-sparse RNNs eliminate overheads related to data storage and irregular memory accesses while increasing hardware efficiency compared to unstructured sparsity.

Description

technical field [0001] The present invention generally relates to systems and methods for computer learning that provide improved computer performance, features and usage. Background technique [0002] Recurrent Neural Networks (RNNs) are state-of-the-art models used in fields such as speech recognition, machine translation, language modeling, and more. Sparsity is a technique for reducing the computational and memory requirements of deep learning models. Sparse RNNs are easier to deploy on devices and high-end server processors. Although sparse operations require less computation and memory relative to their dense counterparts, the observed speedup using sparse operations is smaller than expected on different hardware platforms. Sparse formats do not utilize hardware resources efficiently due to storage overhead, irregular memory accesses, and inability to take advantage of array data paths in modern processors. [0003] Accordingly, systems and methods for neural networ...

AI Technical Summary

Problems solved by technology

Although sparse operations require less computation and memory relative to their dense counterparts, the observed speedup using sparse operations is smaller than expected on different hardware platforms

Sparse formats do not use hardware resources efficiently due to storage overhead, irregular memory accesses, and inability to take advantage of array data paths in modern processors

Images