System and method for accelerating neural networks using a unified sparse tensor core

Abstract

A system and method for accelerating a neural network model is provided, the method for accelerating a neural network model includes obtaining an original weight matrix corresponding to a trained neural network model, pruning the original weight matrix, retraining non-zero coefficients in the pruned weight matrix, compressing the retrained weight matrix, and performing a matrix multiplication operation based on inputting the compressed weight matrix and a set of input activations into the trained neural network model.

Description

[Technical field]

[0001]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority in the United States Patent and Trademark Office to U.S. Provisional Application No. 63 / 257,014, filed on October 18, 2021, U.S. Provisional Application No. 63 / 289,035, filed on December 13, 2021, and U.S. Application No. 17 / 956,036, filed on September 29, 2022, the entire disclosures of which are incorporated herein by reference.

[0002]

[0002] 1. Field The present disclosure relates to neural network model acceleration, and more particularly, to a unified sparse tensor core operation for neural network acceleration. [Background technology]

[0003] 2. Description of Related Art Deep Neural Networks (DNNs) have been used to solve a wide range of tasks for computer vision, natural language processing, etc. The enormous model capacity of deep network structures with a huge number of parameters leads to high prediction performance, but makes DNN models too expensive to be used in practice, especially for mobile and on-device applications with strong limitations on storage, computing power, and energy consumption. Therefore, reducing the cost of using DNN models has attracted attention in academia and industry.

[0004] Neural network compression is one method to reduce the size (i.e., storage required) and accelerate inference (e.g., classification) of large DNN models without compromising much performance (e.g., classification accuracy). Effective compression solutions usually require multidisciplinary knowledge from machine learning, computer architecture, hardware design, etc. Neural network compression may include various techniques such as weight pruning, weight quantization, low-rank factorization, and knowledge distillation. Among all efforts, weight pruning and weight quantization are the most common directions. In particular, weight pruning aims to remove unimportant weight coefficients and reduce redundancy in the network connections of the neural network being trained. Although they can achieve high compression ratios with little prediction loss, unstructured weight pruning methods often fail to improve inference operations (and in some cases even make the problem worse) due to the random memory accesses caused by unstructured sparsity in the pruned weight matrix.

[0004]

[0005] The NVIDIA Ampere GPU architecture introduced the concept of fine-grained structured sparsity to address the weaknesses of unstructured pruning. In the NVIDIA A100 GPU, the structure appears as a 2:4 pattern: out of every 4 coefficients, at least 2 must be zero. However, these zero coefficients are arranged in an unstructured manner. This approach reduces the data footprint and bandwidth of the weight tensor by 2x and doubles the throughput by skipping the calculation of zero values ​​using new hardware NVIDIA Sparse Tensor Cores. However, the maximum sparse rate and throughput improvement are limited to 2x due to the 2:4 sparsity pattern used in this architecture. Removing more weights usually causes a large degradation in prediction performance, especially for models like MobileNet that are already designed to be highly efficient. Summary of the Invention

[0005]

[0006] Since inference operations in deep learning systems are matrix multiplication intensive, high-performance general matrix-matrix multiplication (GEMM) is key to performing the inference operations. Provided are systems and methods for a unified pattern-based GEMM computation method for accelerating neural network models.

[0006]

[0007] According to a disclosed aspect, a method for accelerating a neural network model includes: obtaining an original weight matrix corresponding to a trained neural network model; pruning the original weight matrix; retraining non-zero coefficients in the pruned weight matrix; compressing the retrained weight matrix; and performing a matrix multiplication operation based on inputting the compressed weight matrix and a set of input activations into the trained neural network model.

[0007]

[0008] Pruning the weight matrix may include pruning the weight matrix to fit a 2:4 sparsity pattern, where in each group of four coefficients of the weight matrix, at least two coefficients of the weight matrix are non-zero.

[0008]

[0009] The step of retraining the weight matrix may include determining the smallest nonzero coefficient in each group of four coefficients of the pruned weight matrix; and retraining the absolute value of each nonzero coefficient in each group of four coefficients to be a power of two of the smallest nonzero coefficient in the group.

[0009]

[0010] The step of compressing the weight matrix includes: compressing the retrained weight matrix to be one-quarter the size of the original weight matrix; generating a non-zero flag array corresponding to the compressed weight matrix; generating a sign flag array corresponding to the compressed weight matrix; and The method may include generating a left-shift flag array corresponding to the compressed weight matrix, where the nonzero flag array is a 1-bit array used to track nonzero coefficients in the original weight matrix, the sign flag array is a 1-bit array used to track signs of the nonzero coefficients in the original weight matrix, and the left-shift flag array is a 2-bit array used to track power-of-two relationships in each group of four coefficients.

[0010]

[0011] The steps of performing a matrix multiplication operation based on inputting the compressed weight matrix and a set of input activations into a trained neural network model include: selecting input activations from the set of input activations based on the non-zero flag array, where only input activations that correspond to the non-zero flag array are selected; and The method may include performing a matrix multiplication operation on selected input activations, in which case multiplication operations corresponding to non-selected input activations are skipped.

[0011]

[0012] The steps for performing a matrix multiplication operation on selected input activations include: converting the multiple independent multiplication operations into one multiplication operation and multiple addition operations based on the sign flag array and the left shift flag array; and and performing one multiplication operation and multiple addition operations with the selected input activations based on the sign flag array and the left shift flag array. In this way, each of the "multiple independent multiplication operations" corresponding to the inference operations of the trained neural network model is converted into "one multiplication operation" and "multiple addition operations," and the "one multiplication operation" can be performed simultaneously.

[0012]

[0013] The method may further include obtaining an output of the neural network model based on the matrix multiplication operation, the output corresponding to an inference operation of the neural network model.

[0013]

[0014] According to an aspect of the present disclosure, a device for accelerating a neural network model includes a memory storing instructions; and at least one processor, the at least one processor performing the following steps: Obtaining the original weight matrix corresponding to the trained neural network model; A step of pruning the original weight matrix; retraining the non-zero coefficients in the pruned weight matrix; compressing the retrained weight matrix; and The method is configured to execute instructions by performing a matrix multiplication operation based on inputting the compressed weight matrix and the set of input activations into the trained neural network model.

[0014]

[0015] The processor may be further configured to execute instructions by: determining the smallest non-zero coefficient in each group of four coefficients of the pruned weight matrix; and retraining the absolute value of each non-zero coefficient in each group of four coefficients to be a power of two of the smallest non-zero coefficient in the group.

[0015]

[0016] The processor may be further configured to execute instructions by compressing the retrained weight matrix to be one-quarter the size of the original weight matrix; generating a non-zero flag array corresponding to the compressed weight matrix; generating a sign flag array corresponding to the compressed weight matrix; and generating a left-shift flag array corresponding to the compressed weight matrix.

[0016]

[0017] The processor may be further configured to execute instructions that include the steps of: selecting input activations from the set of input activations based on the non-zero flag array, where only input activations that correspond to the non-zero flag array are selected; and performing a matrix multiplication operation on the selected input activations, where multiplication operations that correspond to non-selected input activations are skipped.

[0017]

[0018] The processor may be further configured to execute the instruction by: converting the multiple independent multiplication operations into one multiplication operation and multiple addition operations based on the sign flag array and the left shift flag array; and performing the one multiplication operation and multiple addition operations with selected input activations based on the sign flag array and the left shift flag array.

[0018]

[0019] Each of the "multiple independent multiplication operations" corresponding to an inference operation of the trained neural network model is converted into a "single multiplication operation" and "multiple addition operations," and the "single multiplication operations" may be executed simultaneously.

[0019]

[0020] The processor may be further configured to execute instructions by obtaining an output of the neural network model based on the matrix multiplication operation, the output corresponding to an inference operation of the neural network model.

[0020] According to an aspect of the disclosure, a non-transitory computer readable storage medium is provided that stores computer readable program code or instructions, the program code or instructions being executable by a processor to perform operations to accelerate a neural network model, the operations including: Obtaining the original weight matrix corresponding to the trained neural network model; A step of pruning the original weight matrix; retraining the non-zero coefficients in the pruned weight matrix; compressing the retrained weight matrix; performing a matrix multiplication operation based on inputting the compressed weight matrix and the set of input activations into the trained neural network model; obtaining an output of the neural network model based on a matrix multiplication operation; , and the output corresponds to the inference operation of the neural network model.

[0021]

[0021] The operations may further include: determining the smallest non-zero coefficient in each group of four coefficients of the pruned weight matrix; and retraining the absolute value of each non-zero coefficient in each group of four coefficients to be a power of two of the smallest non-zero coefficient in the group.

[0022]

[0022] The operations may further include the steps of compressing the retrained weight matrix to be one-quarter the size of the original weight matrix; generating a non-zero flag array corresponding to the compressed weight matrix; generating a sign flag array corresponding to the compressed weight matrix; and generating a left-shift flag array corresponding to the compressed weight matrix.

[0023]

[0023] The operations may further include converting the multiple independent multiplication operations into one multiplication operation and multiple addition operations based on the sign flag array and the left shift flag array; and performing one multiplication operation and multiple addition operations using selected input activations based on the sign flag array and the left shift flag array.

[0024]

[0024] These and other aspects of the exemplary embodiments herein will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings. However, it should be understood that while the following description sets forth an exemplary embodiment and numerous specific details thereof, they are given by way of illustration and not limitation. Many changes and modifications may be made within the scope of the exemplary embodiments herein without departing from the spirit thereof, and the exemplary embodiments herein include all such modifications. [Brief description of the drawings]

[0025]

[0025] The above and other aspects, features and advantages of particular embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings. [Figure 1]

[0026] FIG. 1 is a diagram illustrating components of one or more devices according to various embodiments. [Diagram 2]

[0027] FIG. 2 is a diagram illustrating a unified sparse tensor core operation in accordance with various embodiments. [Diagram 3]

[0028] FIG. 3 is a diagram illustrating retraining of a weight matrix according to various embodiments. [Figure 4]

[0029] FIG. 4 is a diagram illustrating compression of a weight matrix according to various embodiments. [Diagram 5]

[0030] FIG. 5 is a diagram illustrating matrix multiplication in a unified sparse tensor core in accordance with various embodiments. [Figure 6]

[0031] FIG. 6 is a flow chart illustrating a method for accelerating a neural network model according to various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026]

[0032] Detailed Description The following detailed description of the exemplary embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings may identify the same or similar elements.

[0027]

[0033] The present disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may occur from practice of implementation. Moreover, one or more features or components of one embodiment may be incorporated or combined with another embodiment (or one or more features of another embodiment). Furthermore, in the flow charts and descriptions of operations provided below, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed simultaneously (at least partially simultaneously), and the order of one or more operations may be rearranged.

[0028]

[0034] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to any specific software code. It will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0029]

[0035] Although particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each separately recited dependent claim may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

[0030]

[0036] No element, act, or instruction used herein should be construed as critical or essential unless expressly described as such. Also, as used herein, the article terms "a" and "an" are intended to include one or more items and can be used interchangeably with "one or more." Where only one item is intended, the term "a" or similar language is used. Also, as used herein, the terms "has," "have," "include ... ","having," ","include," "including," and the like are intended to be open-ended terms. Additionally, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise. Additionally, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood as including only A, only B, or both A and B.

[0031]

[0037] As explained above, neural network compression using unstructured weight pruning techniques may achieve high compression rates with little prediction loss, but these techniques typically fail to improve inference behavior and may even increase prediction loss in some cases. A fine-grained structured sparsity technique may be denoted as a 2:4 pattern, where at least two out of every four coefficients must be zero. This technique reduces the data footprint and bandwidth of the weight tensor by half and potentially doubles the inference throughput by skipping the computation of zero-valued coefficients. However, the maximum sparsity rate and inference throughput improvement are limited to a factor of two due to the 2:4 sparsity pattern.

[0032]

[0038] Various embodiments according to the present disclosure provide systems and methods for unified sparse tensor core computing that combines fine-grained structured sparsity techniques with weight unification techniques to achieve higher inference throughput.

[0033]

[0039] 1 is a diagram illustrating components of one or more devices according to various embodiments. With reference to FIG. 1, device 100 may include a bus 110, one or more processors 120, a memory 130, a storage element 140, and a communication interface 150. It is understood that one or more components may be omitted and / or one or more additional components may be included.

[0034]

[0040] Bus 110 includes components that enable communication between components of device 100. Processor 120 may be implemented in hardware, firmware, or a combination of hardware and software. Processor 120 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a sparse tensor core, or another type of processing component. Processor 120 may include one or more processors. For example, processor 120 may include one or more CPUs, APUs, FPGAs, ASICs, sparse tensor cores, or another type of processing component. One or more processors of processor 120 may be programmable to perform functions.

[0035]

[0041] Memory 130 may include random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions used by processor 120.

[0036]

[0042] The storage element 140 stores information and / or software related to the operation and use of the device 100. For example, the storage element 140 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium along with a corresponding drive.

[0037]

[0043] Communications interface 150 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable device 100 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communications interface 150 can enable device 100 to receive information from another device and / or provide information to another device. For example, communications interface 150 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0038]

[0044] The device 100 is capable of performing one or more processes or functions described herein. The device 100 is capable of performing operations based on the processor 120 executing software instructions stored by a non-transitory computer-readable medium, such as the memory 130 and / or the storage element 140. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space distributed across multiple physical storage devices.

[0039]

[0045] The software instructions may be loaded into memory 130 and / or storage element 140 from another computer-readable medium or from another device via communications interface 150. When executed, the software instructions stored in memory 130 and / or storage element 140 may cause processor 120 to perform one or more processes described herein.

[0040]

[0046] Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to implement one or more of the processes described herein. Thus, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0041]

[0047] The number and arrangement of components shown in Figure 1 are provided as an example. In practice, device 100 may include additional, fewer, different, or differently arranged components relative to those shown in Figure 1. Additionally or alternatively, a set of components (e.g., one or more components) of device 100 may perform one or more functions that are described as being performed by another set of components of device 100.

[0042]

[0048] Any of the operations or processes described below (eg, FIGS. 2-6) may be implemented by or using any of the elements shown in FIG.

[0043]

[0049] FIG. 2 illustrates a unified sparse tensor core operation according to various embodiments. As shown in FIG. 2, the unified sparse tensor core operation includes obtaining a weight matrix 210. For example, the device 100 may obtain the weight matrix 210 from the memory 130, the storage element 140, or a location external to the device 100 via the communication interface 150. The weight matrix 210 may include trained weights (e.g., coefficient values) corresponding to a trained neural network. For example, the weight matrix 210 may include weights corresponding to a neural network trained to perform a classification task.

[0044]

[0050] Weight matrix 210 may be pruned to generate pruned weight matrix 220. Weight matrix 210 may be pruned using a fine-grained structured sparsity technique having a 2:4 sparsity pattern such that in each group of four coefficients in weight matrix 210, at least two coefficients are nonzero. For example, the first through fourth coefficients in weight matrix 210 may form a first group of coefficients, the fifth through eighth coefficients in weight matrix 210 may form a second group of coefficients, the ninth through twelfth coefficients in weight matrix 210 may form a third group of coefficients, and so on. The first group of coefficients may be pruned such that at least two coefficients in the first group are non-zero, the second group of coefficients may be pruned such that at least two coefficients in the second group are non-zero, and the third group of coefficients may be pruned such that at least two coefficients in the third group are non-zero.

[0045]

[0051] The coefficient values ​​in pruned weight matrix 220 may be retrained to generate a retrained weight matrix. For example, the absolute value of each nonzero coefficient in each group of four coefficients in pruned weight matrix 220 may be retrained to be a power of two of the smallest nonzero coefficient in the group.

[0046]

[0052] The retrained weight matrix can be compressed to generate a compressed weight matrix 230, a non-zero flag array 240, a sign flag array 250, and a left shift flag array 260. The compressed weight matrix 230 may be compressed to one-quarter the size of the original matrix (weight matrix 220). The non-zero flag array 240 may be a 1-bit array used to track the non-zero coefficients in the original matrix (weight matrix 220). The sign flag array 250 may be a 1-bit array used to track the signs of the non-zero coefficients in the original matrix (weight matrix 220). The left shift flag array 260 may be a 2-bit array used to track the power of two relationship in each group of four coefficients in the retrained pruned weight matrix.

[0047]

[0053] The sparse tensor core 280 can be controlled to perform a matrix multiplication operation as part of the inference operation. The sparse tensor core 280 can perform a matrix multiplication based on the packed weight matrix 230, the non-zero flag array 240, the sign flag array 250, the left shift flag array 260, and the input activation matrix 270 to obtain output activations. Corresponding values ​​in the packed weight matrix 230 (i.e., weight coefficients), the non-zero flag array 240 (i.e., non-zero flags), the sign flag array 250 (i.e., sign flags), the left shift flag array 260 (i.e., left shift flags), and the input activation matrix 270 (i.e., input activation coefficients) can be input to the sparse tensor core 280. 2, the weight coefficients in subset 231 of compressed weight matrix 230, the non-zero flag values ​​in subset 241 of non-zero flag array 240, the sign flag values ​​in subset 251 of sign flag array 250, the left-shift flag values ​​in subset 261 of left-shift flag array 260, and the input activation values ​​in subset 271 of input activation matrix 270 may be corresponding values. The values ​​in subsets 231, 241, 251, 261, and 271 may be provided as inputs to sparse tensor core 280.

[0048]

[0054] The sparse tensor core 280 may select input activation values ​​based on the non-zero flags and compute dot products with the selected activations. The sparse tensor core 280 may compute dot products between the selected activations and their corresponding coefficients in the compressed weight matrix 230 based on the sign flags and the left shift flags. The sparse tensor core 280 may provide the dot product results as output. The dot product results may be obtained as output activation coefficient values ​​and stored in the output activation matrix 290.

[0049]

[0055] FIG. 3 illustrates retraining of a weight matrix according to various embodiments. For example, the device 100 may obtain a pruned weight matrix 220 and retrain the coefficients in each group of four coefficients in the pruned weight matrix 220. As shown in FIG. 3, the group of four coefficients 310 may include a first coefficient 311 (represented by "a"), a second coefficient 312 (represented by "b"), a third coefficient 313 (represented by "c"), and a fourth coefficient 314 (represented by "d"). From among the group of four coefficients 310, a coefficient having a minimum non-zero value is determined. With reference to FIG. 3, the first coefficient 311 "a" may be determined as the minimum non-zero coefficient. The remaining non-zero coefficients (e.g., 312, 313, 314) in the group of four coefficients 310 may be retrained to be powers of two of the first coefficient 311. 3, the second coefficient 312 "b" can be retrained as a coefficient 322 that is a power of two ("-2a") of the first coefficient 311. Similarly, the third coefficient 313 "c" can be retrained as a coefficient 323 that is a power of two ("-8a") of the first coefficient 311, and the fourth coefficient 314 "d" can be retrained as a coefficient 324 that is a power of two ("4a") of the first coefficient 311. In this manner, each group of four coefficients in the pruned weight matrix 220 can be retrained to generate a retrained weight matrix. The retrained weight matrix can then be compressed.

[0050]

[0056] 4 illustrates compression of a weight matrix according to various embodiments. For example, device 100 may take pruned weight matrix 220 and retrain the coefficients in each group of four coefficients to generate a retrained weight matrix, and device 100 may compress the retrained weight matrix to generate compressed weight matrix 230, non-zero flag array 240, sign flag array 250, and left shift flag array 260. Non-zero flag array 240 may be a 1-bit array used to track the non-zero coefficients in the original matrix, as described above with respect to FIG. 2; sign flag array 250 may be a 1-bit array used to track the signs of the non-zero coefficients in the original matrix; left shift flag array 260 may be a 2-bit array used to track the power-of-two relationships in each group of four coefficients.

[0051]

[0057] 4, the group of four coefficients 310 can be compressed into a single coefficient 430 ("a") in the compressed weight matrix 230, thereby achieving one-quarter of the size of the original weight matrix. With further reference to FIGURE 4, a non-zero flag 440 is the value in the non-zero flag array 240 that corresponds to the group of four coefficients 310, a sign flag 450 is the value in the sign flag array 240 that corresponds to the group of four coefficients 310, and a left shift flag 460 is the value in the left shift flag array 260 that corresponds to the group of four coefficients 310.

[0052]

[0058] 5 illustrates a matrix multiplication operation in a unified sparse tensor core according to various embodiments. The unified sparse tensor core 580 can be controlled to perform matrix multiplication as part of an inference operation of a trained neural network. The unified sparse tensor core 580 can be provided with inputs including the compressed weight matrix 230 (i.e., weight coefficients), the non-zero flag array 240 (i.e., non-zero flags), the sign flag array 250 (i.e., sign flags), the left shift flag array 260 (i.e., left shift flags), and corresponding values ​​in the input activation matrix 270 (i.e., input activation coefficients).

[0053]

[0059] As shown in FIG. 5, the weight coefficients 530, the non-zero flags 540, the sign flags 550, the left shift flags 560, and the input activation coefficients 570 can be input to the unified sparse tensor core 580, and the output activation coefficients 590 can be obtained as the output of the unified sparse tensor core 580. The non-zero flags 540 can be used to select only non-zero values ​​in the input activation coefficients 570. In this way, the multiplication operations corresponding to the zero-value coefficients in the input activation coefficients 570 can be skipped to achieve twice the throughput of the conventional tensor core. Multiple multiplication operations between the selected input activation coefficients and the weight coefficients 530 can be converted into a single multiplication operation and multiple addition operations based on the sign flags 550 and the left shift flags 560. In this way, only half the multipliers in the unified sparse tensor core 580 are needed to compute each dot product result, allowing the unified sparse tensor core 580 to achieve four times the throughput of a conventional tensor core.

[0054]

[0060] In some embodiments, the absolute values ​​of all non-zero coefficients in each of the two groups of four coefficients in the pruned weight matrix 220 may be retrained to generate a unified pruned weight matrix. The absolute values ​​of all non-zero coefficients in each of the two groups of four coefficients may be retrained to be a power of two of the smallest non-zero coefficient in the two groups of four coefficients. By retraining the non-zero coefficients in each of the two groups of four coefficients, the unified sparse tensor core 580 may double the input and convert multiple independent multiplication operations corresponding to twice the input activation coefficients into one multiplication operation and multiple addition operations. In this way, only one-quarter of the multipliers in the unified sparse tensor core 580 are required to calculate the result of each dot product, and the unified sparse tensor core 580 may achieve eight times the throughput of a conventional tensor core.

[0055]

[0061] FIG. 6 is a flow chart illustrating a method 600 for accelerating a neural network model according to various embodiments. As shown in FIG. 6, at 601, the method 600 includes obtaining a weight matrix of a trained neural network model. For example, the device 100 may obtain a weight matrix 210 corresponding to the trained neural network model. The weight matrix 210 may be referred to as an original weight matrix. The weight matrix 210 may include a set of coefficient values ​​corresponding to the trained neural network model.

[0056]

[0062] At 602, method 600 includes pruning a weight matrix. For example, device 100 may prune the weight matrix to conform to a 2:4 sparsity pattern, where in each group of four coefficients in the weight matrix, at least two coefficients of the weight matrix are nonzero.

[0057]

[0063] At 603, method 600 includes retraining the weight matrix. For example, device 100 may retrain pruned weight matrix 220. Device 100 may retrain pruned weight matrix 220 by determining the smallest nonzero coefficient in each group of four coefficients of pruned weight matrix 220 and retraining the pruned weight matrix 220 such that the absolute value of each nonzero coefficient in each group of four coefficients is a power of two of the smallest nonzero coefficient in the group.

[0058]

[0064] At 604, the method 600 includes compressing the weight matrix. For example, the device 100 may compress the retrained weight matrix. The device 100 may compress the retrained weight matrix to a quarter of the size of the original weight matrix and generate a non-zero flag array corresponding to the compressed weight matrix, a sign flag array corresponding to the compressed weight matrix, and a left-shift flag array corresponding to the compressed weight matrix. The non-zero flag array may be a 1-bit array used to track non-zero coefficients in the original weight matrix, the sign flag array may be a 1-bit array used to track signs of non-zero coefficients in the original weight matrix, and the left-shift flag array may be a 2-bit array used to track power-of-two relationships in each group of four coefficients.

[0059]

[0065] At 605, the method 600 includes performing a matrix multiplication operation based on the compressed weight matrix and input activations of the neural network model. For example, the device 100 may perform the matrix multiplication operation based on inputting the compressed weight matrix and the set of input activations into the trained neural network model. The device 100 may select input activations from the set of input activations based on the non-zero flag array, such that only input activations corresponding to the non-zero flag array are selected. The device 100 may perform the matrix multiplication operation on the selected input activations, where multiplication operations corresponding to the non-selected input activations are skipped. In this manner, each of the "multiple independent multiplication operations" corresponding to the inference operations of the trained neural network model is converted into a "single multiplication operation" and a "multiple addition operation", and the converted "single multiplication operations" are executed simultaneously.

[0060]

[0066] At 606, the method 600 includes obtaining an inference of the neural network model. For example, the device 100 may obtain results of a matrix multiplication operation corresponding to each of a plurality of groups of four coefficients in the weight matrix. The device 100 may determine an output of the inference operation of the trained neural network model based on the plurality of matrix multiplication results.

[0061]

[0067] According to various exemplary embodiments, neural network models can be accelerated using unified pattern-based sparse tensor core operations for accelerating neural networks. The unified sparse tensor core operations combine fine-grained structured sparsity techniques with weight consolidation techniques to achieve higher inference throughput. Systems and methods are provided for performing high-performance generalized matrix-matrix multiplication (GEMM) by pruning, retraining, and compressing weight matrices, so that multiple independent multiplication operations can be converted into a single multiplication operation and multiple addition operations. In this way, inference operations of neural network models (which are matrix multiplication intensive) can be accelerated.

[0062]

[0068] The present disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of implementation.

[0063]

[0069] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of integration of technical details. Furthermore, one or more of the above components mentioned above may be implemented as instructions stored in a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or media) having computer-readable program instructions for causing a processor to perform operations.

[0064]

[0070] A computer readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as ridged structures or punch cards with instructions recorded in grooves, and any suitable combination of the above. Computer-readable storage medium, as used herein, should not be construed as being a transitory signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0065]

[0071] The computer readable program instructions described herein may be downloaded from a computer readable storage medium into each computing / processing device, or may be downloaded to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing / processing device.

[0066]

[0072] The computer readable program code / instructions for carrying out the operations may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform each aspect or operation.

[0067]

[0073] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to cause the machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for performing the functions / operations specified in one or more blocks of the flow charts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium, such that the instructions may direct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that a computer readable storage medium having instructions stored thereon includes a product containing instructions that implement aspects of the functions / operations specified in one or more blocks of the flow charts and / or block diagrams.

[0068]

[0074] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to result in a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / operations specified in one or more blocks of the flow charts and / or block diagrams.

[0069]

[0075] The flow and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flow chart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical functions. The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks relative to those depicted in the figures. In some alternative implementations, the functions noted in the blocks may occur in an order different from that noted in the figures. For example, two blocks shown in succession may in fact be executed concurrently or substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the example block diagrams and / or flow charts, as well as combinations of blocks in the example block diagrams and / or flow charts, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0070]

[0076] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to any specific software code. It will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

Claims

1. 1. A method for accelerating a neural network model, executed by at least one processor, the method comprising: Obtaining an original weight matrix corresponding to the trained neural network model; pruning the original weight matrix to generate a pruned weight matrix, where in each group of four coefficients in the pruned weight matrix, at least two coefficients are non-zero; retraining the non-zero coefficients in the pruned weight matrix; compressing the retrained weight matrix; and performing a matrix multiplication operation based on inputting the compressed weight matrix and a set of input activations into the trained neural network model; and the step of retraining the non-zero coefficients in the pruned weight matrix comprises: determining the smallest non-zero coefficient in each group of four coefficients of the pruned weight matrix; and retraining the absolute value of each non-zero coefficient in each group of four coefficients to be a power of two of the smallest non-zero coefficient in that group; A method comprising:

2. 2. The method of claim 1, wherein the step of pruning the original weight matrix comprises: pruning the weight matrix to fit a 2:4 sparsity pattern; A method comprising:

3. A method for accelerating a neural network model, executed by at least one processor, the method comprising: Obtaining an original weight matrix corresponding to the trained neural network model; pruning the original weight matrix to generate a pruned weight matrix, where in each group of four coefficients in the pruned weight matrix, at least two coefficients are non-zero; retraining the non-zero coefficients in the pruned weight matrix; compressing the retrained weight matrix; and performing a matrix multiplication operation based on inputting the compressed weight matrix and a set of input activations into the trained neural network model; and compressing the retrained weight matrix comprises: compressing the retrained weight matrix to be one-quarter the size of the original weight matrix; generating a non-zero flag array corresponding to the compressed weight matrix; generating a sign flag array corresponding to the compressed weight matrix; and generating a left-shift flag array corresponding to the compressed weight matrix; A method comprising:

4. 4. The method of claim 3, wherein the nonzero flag array is a 1-bit array used to track nonzero coefficients in the original weight matrix, the sign flag array is a 1-bit array used to track signs of nonzero coefficients in the original weight matrix, and the left shift flag array is a 2-bit array used to track power-of-two relationships in each group of four coefficients.

5. 4. The method of claim 3, wherein the step of performing a matrix multiplication operation based on inputting the compressed weight matrix and a set of input activations into the trained neural network model includes: selecting input activations from the set of input activations based on the non-zero flag array, where only input activations that correspond to the non-zero flag array are selected; and performing the matrix multiplication operation on the selected input activations, where multiplication operations corresponding to non-selected input activations are skipped; A method comprising:

6. 6. The method of claim 5, wherein performing the matrix multiplication operation on the selected input activations comprises: converting a plurality of independent multiplication operations into one multiplication operation and a plurality of addition operations based on the sign flag array and the left shift flag array; and performing the one multiplication operation and multiple addition operations with the selected input activations based on the sign flag array and the left shift flag array; A method comprising:

7. 7. The method of claim 6, wherein each of the "multiple independent multiplication operations" corresponding to the inference operations of the trained neural network model is converted into a "single multiplication operation" and "multiple addition operations," and the "single multiplication operations" are performed simultaneously.

8. 4. The method of claim 3 further comprising: obtaining an output of the neural network model based on the matrix multiplication operation, the output corresponding to an inference operation of the neural network model.

9. 1. A device for accelerating a neural network model, comprising: A memory storing program code; and at least one processor configured to execute said program code and to act as directed by said program code; said program code comprising: obtaining code configured to cause the at least one processor to perform the step of obtaining an original weight matrix corresponding to a trained neural network model; pruning code configured to cause the at least one processor to perform the step of pruning the original weight matrix; retraining code configured to cause the at least one processor to perform the step of retraining non-zero coefficients in the pruned weight matrix; compression code configured to cause the at least one processor to compress the retrained weight matrix; and matrix multiplication code configured to cause the at least one processor to perform a matrix multiplication operation based on inputting the compressed weight matrix and a set of input activations into the trained neural network model; and the program code further comprises: decision code configured to cause the at least one processor to perform the step of determining a smallest non-zero coefficient in each group of four coefficients of the pruned weight matrix; wherein the retraining code is configured to cause the at least one processor to perform the step of retraining an absolute value of each nonzero coefficient in each group of four coefficients to be a power of two of the smallest nonzero coefficient in the group.

10. A device for accelerating a neural network model, comprising: A memory storing program code; and at least one processor configured to execute said program code and to act as directed by said program code; said program code comprising: obtaining code configured to cause the at least one processor to perform the step of obtaining an original weight matrix corresponding to a trained neural network model; pruning code configured to cause the at least one processor to perform the step of pruning the original weight matrix; retraining code configured to cause the at least one processor to perform the step of retraining non-zero coefficients in the pruned weight matrix; compression code configured to cause the at least one processor to compress the retrained weight matrix; and matrix multiplication code configured to cause the at least one processor to perform a matrix multiplication operation based on inputting the compressed weight matrix and a set of input activations into the trained neural network model; the compression code configured to cause the at least one processor to perform the steps of compressing the retrained weight matrix to be one-quarter the size of the original weight matrix; The program code: generating a non-zero flag array corresponding to the compressed weight matrix; generating a sign flag array corresponding to the compressed weight matrix; and generating a left-shift flag array corresponding to the compressed weight matrix; a flag array generation code for causing the at least one processor to execute:

11. 11. The device of claim 10, wherein the program code further comprises: further comprising input selection code configured to cause the at least one processor to perform the steps of selecting input activations from the set of input activations based on the non-zero flag array, where only input activations corresponding to the non-zero flag array are selected; and The matrix multiplication code is configured to cause the at least one processor to perform the matrix multiplication operation on selected input activations, and multiplication operations corresponding to non-selected input activations are skipped.

12. 11. The device of claim 10, wherein the program code comprises: conversion code configured to cause the at least one processor to convert a plurality of independent multiplication operations into one multiplication operation and a plurality of addition operations based on the sign flag array and the left shift flag array; and wherein the matrix multiplication code is configured to cause the at least one processor to perform the one multiplication operation and multiple addition operations with selected input activations based on the sign flag array and the left shift flag array.

13. 13. The device of claim 12, wherein each of the "multiple independent multiplication operations" corresponding to an inference operation of the trained neural network model is converted into a "single multiplication operation" and "multiple addition operations," and the "single multiplication operations" are executed simultaneously.

14. 11. The device of claim 10, wherein the program code comprises: and output obtaining code configured to cause the at least one processor to perform a step of obtaining an output of the neural network model based on the matrix multiplication operation, the output corresponding to an inference operation of the neural network model.

15. A computer program product for causing a processor to carry out a method, the method comprising: Obtaining an original weight matrix corresponding to the trained neural network model; pruning the original weight matrix; retraining the non-zero coefficients in the pruned weight matrix; compressing the retrained weight matrix; and performing a matrix multiplication operation based on inputting the compressed weight matrix and a set of input activations into the trained neural network model; obtaining an output of the neural network model based on the matrix multiplication operation; the output corresponds to an inference operation of the neural network model, the method comprising: determining the smallest non-zero coefficient in each group of four coefficients of the pruned weight matrix; and retraining the absolute value of each non-zero coefficient in each group of four coefficients to be a power of two of the smallest non-zero coefficient in that group; A computer program comprising:

16. A computer program product for causing a processor to carry out a method, the method comprising: Obtaining an original weight matrix corresponding to the trained neural network model; pruning the original weight matrix; retraining the non-zero coefficients in the pruned weight matrix; compressing the retrained weight matrix; and performing a matrix multiplication operation based on inputting the compressed weight matrix and a set of input activations into the trained neural network model; obtaining an output of the neural network model based on the matrix multiplication operation; the output corresponds to an inference operation of the neural network model, the method comprising: compressing the retrained weight matrix to be one-quarter the size of the original weight matrix; generating a non-zero flag array corresponding to the compressed weight matrix; generating a sign flag array corresponding to the compressed weight matrix; and generating a left-shift flag array corresponding to the compressed weight matrix; A computer program comprising:

17. 17. The computer program of claim 16, wherein the method comprises: converting a plurality of independent multiplication operations into one multiplication operation and a plurality of addition operations based on the sign flag array and the left shift flag array; and performing the one multiplication operation and the multiple addition operations with selected input activations based on the sign flag array and the left shift flag array; A computer program comprising:

Images