Systems and methods for accelerating deep learning networks using sparsity

Abstract

A system and method for accelerating deep learning networks using sparsity includes communicating a bit vector to a scheduler that identifies which values ​​in an input tensor are non-zero, and for each lane of the input tensor, determining which values ​​need to be communicated for a multiply-accumulate (MAC) operation, including indicating performance of one of communicating the current value in the lane, communicating the next value in the same lane if such value is non-zero, communicating a value from a step ahead in time if such value is non-zero, and communicating a value from an adjacent lane if such value is non-zero, and outputting the values ​​of the MAC operation.

Description

[Technical Field]

[0001] The following relates generally to deep learning networks, and more particularly to systems and methods for accelerating deep learning networks using sparsity. [Background technology]

[0002] Previously, neural networks could be trained within hours on commodity servers, but training the best neural networks has become an exascale problem. State-of-the-art neural networks may require many graphics processors or specialized accelerators, such as tensor processing units (TPUs), Gaudi, DaVinci, or Cerebras CS1, to be trained within realistic time constraints. Tuning neural networks for best performance or accuracy during inference—for example, via hyperparameter search, or more generally, via network architecture search—further exacerbates the cost of training. Beyond the cost of acquiring or accessing such expensive computing resources, there are also operational costs and environmental impacts of training. Certain applications, such as refining existing models with user-specific information and inputs, may require training neural networks at the "edge." While the tradeoffs for edge devices differ from those for data centers or desktop machines, the need for faster execution time and improved energy efficiency under certain constraints remains. Summary of the Invention

[0003] In one aspect, a method for accelerating a deep learning network of processing elements using sparsity is provided, the method including receiving and storing an input tensor from each of a plurality of input streams; communicating a bit vector to a scheduler that identifies which values ​​in the input tensor are non-zero; determining, for each lane of the input tensor, which values ​​need to be communicated for a multiply-accumulate (MAC) operation, including indicating performance of one of communicating a current value in the lane, communicating a next value in the same lane if such value is non-zero, communicating a value from a step ahead in time if such value is non-zero, and communicating a value from an adjacent lane if such value is non-zero; communicating the indicated value for the MAC operation; and outputting the value of the MAC operation.

[0004] In a particular case of this method, the processing element is tiled with a plurality of other processing elements, and receiving and storing input tensors from each of a plurality of input streams includes processing elements along the same row of the input streams sharing inputs, and processing elements along the same column of the input streams sharing inputs.

[0005] Another version of this method is where each processing unit of the multiple processing units uses a separate scheduler.

[0006] In yet another version of this method, each processing unit along the same row uses a common scheduler.

[0007] In yet another version of this method, the lanes are processed so that they are arranged in a ring with the last lane of the lanes adjacent to the first lane of the lanes.

[0008] In yet another instance of this method, communicating a value from an adjacent lane includes communicating a value from an adjacent lane from one step ahead in time if such value is non-zero.

[0009] In yet another aspect of this method, receiving and storing input tensors from each of a plurality of input streams includes storing each of the input streams in a respective staging buffer.

[0010] In yet another version of this method, the staging buffer contains the current step of the input stream and one or more steps ahead in time.

[0011] In yet another version of this method, the scheduler uses a hierarchical priority scheme.

[0012] In yet another version of this method, the hierarchical priority scheme includes a tight schedule followed by multiple steps that are progressively further ahead in time, followed by multiple steps in adjacent lanes that are progressively further ahead in time.

[0013] In another aspect, a system for accelerating deep learning networks using sparsity is provided, the system including one or more processing units in communication with a data memory to execute processing elements (PEs), each processing element receiving a plurality of input streams, a staging buffer for receiving and storing an input tensor from each of the plurality of input streams, and a scheduler for receiving a bit vector from each staging buffer and identifying which values ​​in the input tensor are non-zero, the scheduler determining, for each lane of the input tensor, which values ​​need to be communicated to a MAC unit for a multiply-accumulate (MAC) operation, the determination including indicating performance of one of communicating a current value in the lane, communicating a next value in the same lane if such value is non-zero, communicating a value from a step ahead in time if such value is non-zero, and communicating a value from an adjacent lane if such value is non-zero; performing the MAC operation with the MAC unit, and outputting the values ​​of the MAC operation.

[0014] In this particular case of the system, the processing elements are collectively arranged in a tiled arrangement where processing elements along the same row of the input stream share inputs and processing elements along the same column of the input stream share inputs.

[0015] In another version of the system, each processing unit uses a separate scheduler.

[0016] In yet another version of the system, each processing unit along the same row uses a common scheduler.

[0017] In yet another version of this system, the lanes are arranged in a ring with the last lane of the lanes adjacent to the first lane of the lanes.

[0018] In yet another case of the system, communicating a value from an adjacent lane includes communicating a value from the adjacent lane from one step ahead in time if such value is non-zero.

[0019] In yet another version of this system, the staging buffer contains the current step of the input stream and one or more steps ahead in time.

[0020] In yet another version of the system, the scheduler uses a hierarchical priority scheme.

[0021] In yet another version of the system, the hierarchical priority scheme includes a tight schedule followed by multiple steps progressively further ahead in time, followed by multiple steps in adjacent lanes progressively further ahead in time.

[0022] In yet another version of this system, the scheduler receives two bit vectors from each of the staging buffers associated with the values ​​in each lane, bitwise ORs the two bit vectors to generate an output bit vector indicating there is at least one zero, and the scheduler uses those values ​​in the output bit vector to select a move for each lane such that as many non-zero values ​​as possible are processed in a single step.

[0023] These and other aspects are contemplated and described herein. It will be understood that the foregoing summary has presented exemplary aspects of embodiments to aid those skilled in the art in understanding the following detailed description.

[0024] A better understanding of the embodiments may be obtained with reference to the drawings. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a schematic diagram of a system for accelerating deep learning networks using sparsity, according to one embodiment. [Figure 2] FIG. 2 is a schematic diagram illustrating the system of FIG. 1 and an exemplary operating environment. [Figure 3] 1 is a flowchart of a method for accelerating deep learning networks using sparsity, according to one embodiment. [Figure 4] 10 is a chart showing an example of total work reduction potential for each of three operations per model from an exemplary experiment. [Figure 5] Here is an example of a forward pass of a deep learning network (DNN). [Figure 6] An example of determining the input gradient for the backward pass of a DNN is shown below. [Figure 7] An example of determining the weight gradients for the backward pass of a DNN is shown below. [Figure 8] 1 shows an example of a baseline processing element. [Figure 9A]Here is an example of processing 16 value pairs without removing invalid ones. [Figure 9B] We present an example of an approach that allows non-zero values ​​to move independently without restrictions in both time and space. [Figure 9C] An example diagram of the system of Figure 1 is shown, which exploits sparsity by moving a limited set of values ​​per lane. [Figure 9D] 2 is a diagram illustrating cycle 1 for an exemplary approach of the system of FIG. 1. [Figure 9E] 2 is a diagram illustrating cycle 2 for an exemplary approach of the system of FIG. 1. [Figure 10] 2 shows an exemplary diagram of a processing element of the system of FIG. 1; [Figure 11] 1 is a diagram showing an example of a connection for lane 1. [Figure 12] 1 shows an illustration of an exemplary implementation of a scheduler. [Figure 13] 2 shows a simplified core with a 2×2 tile configuration for the system of FIG. 1. [Figure 14] This chart shows the speedup achieved by incorporating the system in Figure 1 into the Tensorcore architecture of each model. [Figure 15] We show the runtime speedup achieved by the system in Figure 1 during training from the first epoch to convergence. [Figure 16] We show the energy efficiency of each model for core logic and whole-chip computation. [Figure 17] The energy consumed by the system in Figure 1 is shown relative to the baseline. [Figure 18] The number of rows per tile varies as the number of columns is fixed at 4 but varies from 1 up to 16.,Figure 1 shows the performance of the system. [Figure 19] We show the average performance improvement for the system of Figure 1 with a 3-deep staging buffer on top of a 16-wide processing element. [Figure 20]We show the speedup of the system in Figure 1 for random sparse tensors. DETAILED DESCRIPTION OF THE INVENTION

[0026] Embodiments will now be described with reference to the drawings. For simplicity and clarity of description, where considered appropriate, reference numerals may be repeated among the drawings to indicate corresponding or analogous elements. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be appreciated by those skilled in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the embodiments described herein. Additionally, this description should not be considered to limit the scope of the embodiments described herein.

[0027] Any module, unit, component, server, computer, terminal, or device illustrated herein that executes instructions may include or have access to a computer-readable medium such as a storage medium, computer storage medium, or data storage device (removable and / or non-removable), e.g., magnetic disk, optical disk, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and that can be accessed by an application, module, or both. Any such computer storage media may be part of the device or may be accessible or connectable to the device. Any application or module described herein may be implemented using computer-readable / executable instructions that can be stored or maintained by such computer-readable media.

[0028] Recently, significant effort has been expended on methods to reduce the execution time and energy cost of training. As an example, distributed training partitions the training workload across several computing nodes by leveraging model, data, and pipeline parallelism to reduce overall latency. Data blocking, reuse, and duplication of communication and computation within and between nodes can be used to orchestrate computing, memory hierarchy, and communication resources to improve performance and energy efficiency. Lossless and lossy compression can be used to reduce the footprint of the vast amounts of data processed during training. While single-precision floating-point data and operations have traditionally been used for training, more compact data types (e.g., half-precision floating-point FP16, bfloat16, dynamic floating-point, and flexpoint) reduce overall data volume and computational cost. A mixed-data type approach can further reduce costs by computing more using lower-cost representations and less using higher-cost representations.

[0029] Regardless of the above approaches, training remains a substantial exascale problem. The inventors have observed that many redundant computations naturally occur during training for various models. Therefore, the present embodiment advantageously provides an approach for improving training runtime and energy efficiency. A large portion of the energy consumption during training is due to the transfers and computations required to perform multiply-accumulate (MAC) operations. Because one of the operands in these MACs is often zero, these operations can be safely removed without affecting the values ​​generated during training and therefore the convergence and final accuracy. In many artificial neural networks, many zeros naturally occur in activations during the forward and backward passes, as well as in gradients during the backward pass. Sparsity, when present, represents an opportunity to improve performance and energy efficiency.

[0030] Additionally, zero values ​​and invalid operations occur during inference in both weights and activations. While some zero weights occur naturally, their occurrence can be significantly amplified by pruning. Zero activations also occur naturally during inference, occurring most frequently in models that use the rectified linear unit (ReLU) activation function. Therefore, the presence of zeros, known as sparsity, can be exploited. Some approaches target sparsity in weights, while others target sparsity in both activations and weights.

[0031] However, exploiting sparsity during training is substantially more technically challenging than during inference. First, the occurrence of zeros during inference does not imply that they will also appear during training. Typically, training begins with random initialization of a subset of weights and slowly adjusts them until the network converges. Eventually, some weights will become zero, but it is unclear how quickly this occurs and whether they will remain zero. Second, the location of zero weights during inference is known and does not change, so the sparsity pattern is static. As a result, for inference, computations can be scheduled in advance to fully exploit the sparsity in the weights. This is not the case during training, where weight values ​​are constantly changing, so the sparsity pattern is dynamic. This pattern typically changes with every sample and batch in the training dataset and also changes over time. Third, inference involves two input tensors (weights and activations), which are used in only one computation, typically a matrix-matrix or matrix-vector multiplication. Therefore, the two tensors can be laid out in memory in a way that provides specific access patterns that promote data parallelism and, therefore, energy-efficient fetching and execution. During training, there is typically a third tensor, the gradient, and each of the three tensors is used in two different computations. A particular challenge is that the way the tensors are used in each of these two computations may differ. For example, during the forward pass, a different set of weights may contribute to the output than during the backward pass. This makes it difficult to lay out values ​​in memory in a way that suits both computation needs; a layout that suits the forward pass may need to be "transposed" for the backward pass. Fourth, most inference accelerators that exploit sparsity operate on fixed-point values, while training typically requires floating-point values. The relative costs of the operations are different, so various tradeoffs may arise.

[0032] Sparsity, when present, represents an opportunity to improve performance and energy efficiency. To exploit this opportunity, the present embodiment improves performance and energy efficiency in the presence of sparsity while avoiding the otherwise significant performance and energy efficiency penalties. The present embodiment (which can be informally called "TensorDash") provides a runtime approach to eliminate invalid MACs using a combination of an inexpensive hardware scheduler located directly in front of the MAC units and a co-designed, sparse, low-cost data interconnect. The present embodiment can work with neural networks "out-of-the-box" and does not require modifications or any special annotations from the model developer. The present embodiment does not change any values ​​or functional units, and therefore does not affect convergence or accuracy.

[0033] In some cases, the present embodiment benefits from pruning and quantizing the network during training. Typically, the purpose of pruning is to convert weight values ​​to zero. Dynamic sparse reparameterization, sparse momentum, early pruning, and DropBack are training-time pruning methods that can achieve high sparsity levels with minimal or no impact on output accuracy. We investigated the interaction of the present embodiment with several of these methods. In general, the purpose of quantization is to reduce the data width used during inference. During training, quantization effectively clips otherwise low-magnitude values ​​to zero. Quantization methods include PACT (Parameterized Clipping Activation for Quantized Neural Networks) and LQ-Nets (LQ-Nets). The present embodiment can also benefit from selective backpropagation methods, which backpropagate losses for only some of the neurons. In some cases, selective backpropagation effectively converts many gradients to zero, which manifests as sparsity.

[0034] Advantageously, the embodiments described herein provide at least the following features and benefits: Exploits the sparsity that naturally arises during training, primarily manifesting in activations and gradients. The sparsity is dynamically exploited, and in most cases fully exploited in hardware using a low-overhead hardware scheduler to advance MAC operations in time (faster cycles) and space (separate MAC units), resulting in faster overall computation. The scheduler makes no assumptions about how the sparsity is distributed, allowing it to efficiently handle the dynamic sparsity patterns that arise during training. · It does not affect numerical fidelity, as it only removes MAC operations where at least one of the inputs is zero. Compatible with data-parallel processing elements that perform multiple MAC operations and accumulate them into a single output, and compatible with arbitrary data flows of such processing elements. The benefits can be amplified with training algorithms that incorporate quantization or pruning. · Can be configured to extract sparsity in one or both operands.

[0035] For clarity of illustration, this disclosure generally focuses on convolutional layers because the data flow in convolutional layers is more difficult than in other layers due to the use of activation windows. However, it is understood that the embodiments described herein can be applied to any suitable layer, such as pooling layers and fully connected layers. During training, the processing of layer i typically involves three main convolutions or operations: Forward Pass: A i +1=W i* A i (1) Backward Pass: G i -1=G i* W i (2) Backward Path: Gw i =G i* A i (3) In the formula, W i is the weight, and A i is the input activation, and A i+1 is the output activation, and G i-1 is the input activation gradient, and G i is the output activation gradient, and Gw i is the weight gradient. The first convolution is performed during the forward pass to compute the layer's output activations, and the next two convolutions are performed during the backpropagation pass to compute the input gradients and weight gradients, respectively. It has been observed that the activations, gradients, and / or weights of convolutional neural networks exhibit significant sparsity during training. The present embodiment advantageously exploits such sparsity to accelerate processing by eliminating corresponding multiply-accumulate (MAC) operations.

[0036] 1 and 2 , a system 100 for accelerating deep learning networks (informally referred to as “TensorDash”) using sparsity is shown, according to one embodiment. In this embodiment, system 100 executes on a computing device 26 and accesses content located on a server 32 via a network 24, such as the Internet. In further embodiments, system 100 may execute solely on device 26, solely on server 32, or may execute and / or be distributed on other computing devices, such as desktop computers, laptop computers, smartphones, tablet computers, servers, smartwatches, distributed or cloud computing device(s), etc. In some embodiments, components of system 100 are stored by and executed on a single computer system. In other embodiments, components of system 100 are distributed among two or more computer systems, which may be distributed locally or remotely.

[0037] FIG. 1 illustrates various physical and logical components of an embodiment of system 100. As shown, system 100 includes multiple physical and logical components, including a processing unit 102 (including one or more processors), random access memory (“RAM”) 104, input interfaces 106, output interfaces 108, network interfaces 110, non-volatile storage 112, and a local bus 114 that enables processing unit 102 to communicate with other components. Processing unit 102 can execute or direct the execution of various modules, as described in more detail below. RAM 104 provides relatively responsive volatile storage to processing unit 102. Input interface 106 allows an administrator or user to provide input via input devices such as a keyboard and mouse. Output interface 108 outputs information to output devices, such as a display and / or speakers. Network interface 110 enables communication with other systems, such as other computing devices and servers, located remotely from system 100, such as in a typical cloud-based access model. Non-volatile storage 112 stores the operating system and programs, including computer-executable instructions for implementing the operating system and modules, as well as any data used by these services. Additional stored data may be stored in database 116, as described below. During operation of system 100, the operating system, modules, and associated data may be retrieved from non-volatile storage 112 and placed in RAM 104 to facilitate execution.

[0038] In one embodiment, system 100 includes one or more processing elements (PEs) 122. In some cases, processing elements 122 may be tiled, as described herein. Each processing element 122 includes several modules, which include multiple input streams 124 (including pads for the input streams), multiple staging buffers 126, a scheduler 128, and a MAC unit 130. In some cases, MAC unit 130 may output to an output pad (C-pad). In some cases, some of the modules may be dedicated and at least partially executed on separate hardware, while in other cases, at least some of the functionality of some of the modules is executed on processing unit 102.

[0039] In some cases, weights may exhibit negligible sparsity during training unless the training incorporates pruning, but we have determined that activations and output gradients have sparsity that should be considered. Therefore, the first and second convolutions use A i and G i The sparsity of G can be exploited. i or A i The sparsity in G can be exploited, for example, whichever is higher. i and A i It is possible to simultaneously utilize the sparsity of both.

[0040] Figure 4 is a chart showing an example of the total work reduction potential for each of the three operations per model. One forward operation (A x W) and two backpropagation operations (A x G) and (W x G) each perform approximately the same number of MACs. The work reduction is shown as a speedup defined as:

number

[0041] Deep neural networks can often be trained using a variant of the gradient descent algorithm, where training examples are run through the network to find the prediction error (gradient) with respect to the corresponding label (forward pass), and then the gradient is backpropagated through the network layers to update the network parameters (backward pass).

[0042] During a forward pass, A * W is applied in sequence from the first layer to the last. At each layer, the weights are convolved with the input activations to generate output activations that are fed to the next layer. The output activations of the last layer are compared with the ground truth labels to generate gradients, which can then be backpropagated to update the overall weights. During backpropagation, the layers are called in reverse order, from last to first. Each layer convolves its output gradients with its weights to generate input gradients that are fed to the previous layer. Each layer also convolves its output gradients with its input activations to compute weight gradients. The weight gradients for each layer are accumulated across the training samples in a mini-batch and are used to update the weights once per mini-batch, or iteratively, as explained below:

number

[0043] Figures 5-7 show an example of training a CNN, illustrating the processing of one training sample. Figures 5-7 show the operation of only the convolutional layers. The fully connected layer can be treated as a special case of the convolutional layer where all input tensors are of the same size. In this example, the weights are updated batch by batch. The symbols used for activation, weight, activation gradient, and weight gradient are respectively:

number

[0044] Figure 5 shows an example of the forward pass. The convolutional layer performs a sliding-window three-dimensional (3D) convolution between the input activations and each of the weight filters, and generates one channel of output activations:

number

number

[0045] Figures 6 and 7 show an example of the backward pass. Figure 6 shows an example of determining the input gradients. In the convolutional layer, a sliding window 3D convolution is performed between reshaped versions of the filters using the activation gradients from the subsequent layer. The filters are reconstructed for each channel, rotated 180 degrees, and the activation gradients are dilated by stride st:

number

number

[0046] Figure 7 shows an example of weight gradient determination. For convolutional layers, weight gradients are accumulated over the entire batch of samples. For each sample, they are determined as a 2D convolution between the 2D channel of input activations and the channel of output gradients, dilated according to the stride:

number

number

[0047] An exemplary architecture for training artificial neural networks is an NVIDIA™ graphics processing unit (GPU) with Tensorcore extensions (TCs). In one example, the building block of a baseline accelerator according to this embodiment can include a Tensorcore-like unit. TCs are designed to maximize computational throughput under the data supply constraints of existing memory data paths, so their internal organization may vary from GPU generation to GPU generation. In a specific case, a TC can perform 4x4 floating-point matrix multiplications per cycle, i.e., 64 MACs per cycle. This can be implemented as a tile of 4x4 processing elements (PEs), each of which can perform 4 MACs per cycle and all contribute to the same output, as shown in Figure 8. For example, these could be four pairs (activation, weight) that all contribute to the same output activation, or four pairs (output gradient, weight) that all contribute to the same input gradient.

[0048] Such PEs are generally more energy efficient than single MAC units because they amortize the energy cost of updating the accumulator over several operations and the cost of the summation stage by fusing the MACs. Similar to TCs, PEs implement mixed-precision MAC operations, with multiplications performed in floating-point 16 (FP16) and accumulations performed in floating-point 32 (FP32). A processing element can have three local scratchpads (two for input and one for output). An accelerator may use a grid of these PEs, with each PE having a different scratchpad, or may organize several of them into a tiled grid that shares buffers to take advantage of time and space reuse. While the above case assumes mixed-precision floating-point values ​​(such as those in TCs), this embodiment is data-type agnostic and can work with any data type, e.g., bfloat16, fixed-point, or dedicated narrow floating-point.

[0049] In one example, consider the two input streams as A and B, and use C to denote the output. Figure 9A shows an example of how 16 value pairs can be processed if the system does not attempt to remove those invalid ones (where at least one of the two input values ​​is zero). The input values ​​are

number

number

[0050] Some approaches try to exploit sparsity during inference. Inference is performed on a dataset with weights known in advance. *Because only W convolutions are performed, this represents their sparsity pattern. With only one convolution and one pass, a single data flow is sufficient, allowing the system to arrange values ​​in memory in the order in which it wants to process them. The most challenging layers are generally convolutional layers, since they use a sliding window for their activations. This means that weights must be matched with activations that vary from window to window. Figure 9B shows an approach that allows nonzero values ​​from both sides to move independently without restrictions in both time and space. Here, nonzero values ​​in A are tightly packed one after the other in memory space, as are values ​​in B. Values ​​belonging to the same pair are no longer aligned in time or space. To avoid processing all invalid pairs, the system needs to identify pairs in which both values ​​are nonzero and funnel them into some multiplier. In general, it is preferable to keep as many multipliers busy as possible. This is a difficult task for two reasons: 1) performing arbitrary movement of values ​​in time and space is expensive in hardware, and 2) keeping the four multiplier lanes busy often requires retrieving values ​​from multiple rows from each buffer.

[0051] Some approaches exploit sparsity only on the weight side, for example, by packing nonzero weights densely in memory space so that PEs can access them one row at a time at runtime. Each weight is annotated with metadata, allowing the system to determine its dense (lane, time) location. A unit that maintains a pool of activation candidates finds each nonzero weight and pairs it with its activation. This unit, performing the crossbar function, is generally computationally expensive, so activations can reflect arbitrary weight movement in memory space. Other approaches exploit sparsity on both sides, allowing weights and activations to move freely in both time and space. In one example, an indexing module is tasked with matching nonzero weights and activations. Other approaches improve efficiency by imposing structural constraints on how the model is pruned. In practice, such approaches can remove invalid pairs only if n of the invalid pairs appear together within a single n-wide row. These architectural constraints must be imposed during pruning. Other approaches exploit sparsity on both sides, despite the cost of deploying separate buffer banks for each multiplier input (both A and B). Because they only support value movement in time, they cannot effectively handle work imbalance between lanes, where the "Struggler" lane becomes a bottleneck. In one example, nonzero weights and activations are tightly packed in memory, allowing runtime processing of only valid pairs where both operands are nonzero. To do this, values ​​are processed one channel at a time, ensuring that the product of any weight and any activation contributes to an output activation that assumes a stride of one. In this way, all data movement is avoided at the input. However, a crossbar is required to route these products to the target accumulator bank. The number of crossbars and banks is over-provisioned to avoid stalls due to bank contention, which would otherwise be significant.Other approaches use low-cost sparse interconnects and software schedulers on the front end to extract sparsity in the weights of pruned models without imposing restrictions on how the sparsity is structured. On the activation side, such approaches target in-value sparsity (bit-level sparsity) and use shift-and-add multiplier-based MAC units to do so. Other approaches target recurrent neural networks and extract sparsity from both weights and activations using efficient vector-based encodings that operate on fixed-point values.

[0052] None of the above approaches apply to training. The present embodiments advantageously address at least the following substantial problems in the art: During training, the sparsity pattern is always dynamic. In contrast, during inference, the weights are statically known and can therefore be pre-packed in memory after removing zero weights. During training, each tensor participates in two convolutions, or operations. Because the set of values ​​contributing to the output of each convolution is different, the order in which they are arranged must also be different. For example, the filter channels in the forward pass are different from the channels in the "reconstructed" filter in the backward pass. The "reconstructed" filter in the backward pass is formed by taking weights from the same channel across all filters, stacking them along the channel dimension, and then spatially rotating the filter by 180 degrees. Similarly, the gradients for the second convolution must be bundled together separately from the third convolution. These two convolutions are computed layer-by-layer during the backward pass, which avoids the need to spill gradients off-chip. Therefore, there is no single way to pack them into memory (effectively pre-scheduling them) that works for all cases in which they are used. Activations can be discarded after each layer during inference, but not during training where they are saved for use by the backward pass. Inference accelerators use narrow fixed-point arithmetic (e.g., 8b), while modern training is primarily done using floating-point (e.g., FP32, FP16, or Bfloat16). Floating-point values ​​are typically wide, which increases the cost of crossbars, and performing shift-and-add operations to extract bit-level sparsity is not straightforward in floating-point. Training starts with randomly initialized values ​​that continue to evolve throughout the training process.

[0053] To take advantage of dynamic sparsity patterns that may appear in any or all of the three tensors in some cases, the present embodiment provides an approach that can dynamically remove invalid value pairs at runtime from all three convolutions or operations involved in DNN training. To make this approach energy-efficient, we avoid the high hardware cost of a deep interconnect that allows arbitrary movement of operands, as well as the energy and latency costs of rearranging tensors in different ways to suit different operations.

[0054] System 100 advantageously eliminates invalid value pairs when processing input tensors (e.g., the exemplary input tensor of FIG. 9A ). In some cases, system 100 can be used as an extension of a Tensorcore-like processing unit that performs four MACs per cycle, although it will be appreciated that system 100 can be used to extend any data-parallel unit. For example, the inventors have performed exemplary experiments using system 100 for units up to 16 MACs wide. In one example of the present disclosure, system 100 is processing a 3D convolution of two input tensors A and B, and the processing elements can be understood to perform four MAC operations in parallel.

[0055] FIG. 10 shows an exemplary diagram of a sparse interconnect extending the processing elements 122, in this example, the processing elements of a Tensorcore. The sparse interconnect includes a staging buffer and a scheduler. In one example, there are A and B staging buffers, each with a predetermined parameter depth. In one example, assume this depth is 2 so that each staging buffer can hold up to two row values. Writes to these staging buffers are row-wise. There are four single-value-width read ports, each feeding directly into a multiplier input. As FIG. 9C shows, the connectivity per read port is sparse, and each port can read one of a limited set of values ​​(four values ​​in this example) in the staging buffer. In some cases, the set of values ​​each port can read is different but may overlap.

[0056] In one example, the scheduler can be a hardware scheduler that accepts a bit vector from each staging buffer and identifies which values ​​are non-zero. For a two-deep staging buffer, the bit vector would be 8b wide in this example. Each cycle, the scheduler selects up to four valid pairs from the staging buffer. It generates control signals for the read ports (2b per port in this example), and each port has a set of four possible values, so the corresponding value is read. The same control signals are shared between corresponding ports in the two staging buffers; that is, the same control signals are sent to port p in the horizontal and vertical staging buffers, so both operands move in tandem (4 x 2b control signals in total).

[0057] 9C shows how system 100 can exploit sparsity despite only allowing a limited set of value moves per lane. There are two types of moves: time-only or look-ahead, and space-time or look-aside. FIG. 9C shows one set of possible moves for the third multiplier lane, which are the original dense values ​​a2 0 or the next value a in the same lane 2 1 (look-ahead) or a value a that can be fetched from one of its two adjacent lanes one step ahead in time. 1 1 or a 3 1 (lookaside) or 1 (look-aside). The movements allowed by the other read ports are structurally identical compared to their lanes, and these ports are treated as if they were arranged in a ring with port 0 adjacent to port 3. Each port can access a different set of values, although these sets may overlap. Figures 9D and 9E show how system 100 reduces processing time to a minimum of two cycles using only a four-input multiplexer per multiplier input.

[0058] To improve performance, it is usually necessary to keep the staging buffer as full as possible. Therefore, the A and B buffers need to be banked accordingly to maintain higher read throughput. In this example, two banks are sufficient. Generally, having as many banks as lookaheads is sufficient, and the inventors have empirically determined that three lookaheads are sufficient for Tensorcore-like units. For wider units with more MACs, two banks of lookaheads may also be sufficient.

[0059] In our hardware scheduler, each PE accepts four pairs of (A, B) FP16 values ​​and performs four MACs per cycle. Possibly, a four-deep staging buffer is added to each input. As the example in Figure 11 shows, the staging buffer can hold four rows, each of which is four-valued, corresponding to the dense schedule of the current step in time (step +0) and the next three steps (+1, +2, and +3). For each lane, there is a multiplexer that implements the sparse connectivity pattern. Figure 11 shows the connections for lane 1. In addition to the original "dense" schedule values, there are three lookahead and four lookaside options per input. For example, the lane 1 multiplier can be fed a value in lane 1 from the current time slot or up to three time slots ahead. Alternatively, it can "grab" a value from an adjacent lane. For example, it can get a value from lane 2 from one time step ahead or a value from lane 3 from two steps ahead. Each lane has the same connectivity pattern shifted relative to its position (wrapping around the side edge), and the staging buffers are Z for the A and B staging buffers, respectively. A and Z B and indicates which of these values ​​are zero. The staging buffer has four write ports (one per row).

[0060] The scheduler takes two bit vectors Z from the A and B staging buffers. A and Z B It accepts and generates two sets of signals. The first set is four MS i (i=0...3) 3-bit signals (one per input lane). These are used as select signals for the multiplexers per lane. One MS per multiplier. iThe signal is present and is used by the multiplexers on both the A and B sides of that lane. The scheduler also generates a 2-bit AS signal that indicates the number of rows in the staging buffer that have been drained so they can be replenished from the banked scratchpad to keep the buffer full.

[0061] Z A and Z B When the 4x4b bit vectors of A, B are first bitwise ORed together, a 4x4b bit vector Z is produced. This indicates which (A, B) value pairs have at least one zero. These pairs are invalid and can be skipped. The scheduler then performs a total of four moves (MS i For each lane (signal), the scheduler selects one move per lane. The move selection that the scheduler makes for one time step can be called a schedule.

[0062] For each lane i, the scheduler can use a static priority scheme. In this example, it selects the first available option among eight options in the following order (the notations are (step, lane) as seen in Figure 11): (+0,i) (tight schedule), (+1,i) one step of look-ahead, (+2,i) two steps of look-ahead, (+3,i) three steps of look-ahead, and the look-aside options: (+1,i+1), (+1,i-1), (+2,i+2), and (+3,i+3). In this example, using an 8b-to-3b priority encoder is sufficient. However, if all lane selections were made independently, the same pair could be selected for multiple lanes, resulting in more than one multiplication and accumulation, which could lead to an invalid schedule.

[0063] To ensure that the scheduler always generates a valid schedule (one in which each value pair is selected once), a hierarchical scheme can be used in which scheduling occurs at four levels, as illustrated in the diagram in Figure 12. At each level, a subset of lanes (one lane in this example) make their decisions independently using the current value of the Z vector as input. The lanes assigned at each level are guaranteed by design to be unable to make overlapping choices. After multiple lanes at one level have made their choices, they "remove" these options (OR gates) from the Z vector before passing it on to the next level. Finally, any bits left enabled in Z can cause an AS signal to be generated. While the above is described in steps, it is understood that the scheduler is combinatorial and can operate in a single cycle.

[0064] In the above example, a single processing element (PE) is described and can exploit sparsity in both operands. Tensor cores can be implemented as tiles (e.g., 4x4) of such PEs. PEs can only exploit temporal reuse, but spatial data reuse is also possible by having PEs along the same row share the same B input and PEs along the same column share the same A input. For example, in the forward pass and for convolutional layers, each row can be processing a different filter, while the column can be processing a different window. In this arrangement, each PE processes a unique combination of B and A inputs. If a scheduler and staging buffer are used per PE, it is still possible to skip zeros on both the A and B sides.

[0065] In this example, there is sufficient sparsity in one of the operands in each of the three main operations to extract significant benefits from the sparsity extracted from only the B-side. Figure 13 shows a simplified core with a 2x2 tile configuration. Each row of PEs uses a common scheduler and shares the same staging buffer and multiplexer block on the B-side. On the A-side, there is a single staging buffer per column and a dedicated multiplexer block per PE. The A-side multiplexer blocks per row receive the same MS from the row scheduler. i Each scheduler now only needs to see the Z vector from its corresponding B-side staging buffer. We evaluated both TensorDash and Tensorcore (4x4 tiles of PEs) configurations as the main building blocks.

[0066] During training, each tensor can be used in more than one computation. For example, a weight can be convolved with an activation in the forward pass, but convolved with the output gradient in the backward pass. Each operation has a different set of weights that contribute to the output value. This is true for weights, activations, and gradients as well. This impacts the memory hierarchy, and data may need to be provided to the PE in the appropriate order. If a tensor is only used in one way, its values ​​can be statically laid out in memory and easily provided using wide off-chip and on-chip access. However, during training, a layout that adequately serves one of the computations may not adequately serve the other computations. Fortunately, having the ability to transpose tensors as needed allows values ​​to be arranged in memory so that they can be easily fetched for all use cases. In some cases, a tensor layout is used in which values ​​are stored in 4x4 value blocks. The blocks are formed by taking four blocks of adjacent values ​​along the X dimension. Each of these blocks contains four consecutive values ​​along the channel dimension. The starting coordinates for each 4x4 value group are aligned by four along the X and channel dimensions. Finally, the groups that make up the tensor are allocated in memory space, for example, in the order channel, Y, X.

[0067] When fetching values ​​from off-chip, each group can be written directly to multi-bank on-chip memory, so each quad-value block is directly copied to a bank. As a result, a PE can now directly access any block of four contiguous values ​​along the channel dimension in a single step. If transposition is required, on-chip transposers can be used between the on-chip memory banks and the tile scratchpad. The number of transposers can be selected to keep the memory system busy with tiles. Each transposer reads four quad-value blocks from their banks using quad-wide accesses and copies them to its internal 4x4 buffer. The transposer can then provide four blocks of four values, each composed of a single value from each of the four original blocks read from memory, effectively transposing the tensor. For example, it can provide a block containing all values ​​that appear first or all values ​​that appear third in the original block. This transposition approach can be applied to all three computational operations involved in training.

[0068] FIG. 3 illustrates a flowchart of a method 300 for accelerating deep learning networks on processing elements 122 using sparsity, according to one embodiment.

[0069] In block 302, each of the staging buffers 126 receives and stores input tensors from a respective one of the multiple input streams 124.

[0070] In block 304, the scheduler 128 receives the bit vector from each staging buffer 126 to identify which values ​​stored in the respective staging buffer 126 are non-zero.

[0071] In block 306, scheduler 128 instructs staging buffer 126 for each lane which values ​​need to be communicated to MAC unit 130 to arrive at a valid pair. The instructions include communicating one of the following: the current value for the lane, the next value for the same lane if such value is non-zero, the value from one step ahead in time if such value is non-zero, or the value from an adjacent lane if such value is non-zero.

[0072] In block 308, each staging buffer 126 communicates the indicated value to MAC unit 130 to perform the MAC operation.

[0073] In block 310, the MAC unit 130 outputs the value of the MAC operation to, for example, another processing element 122, non-volatile storage 112, database 116, network interface 110, or output interface 108.

[0074] We conducted exemplary experiments to evaluate the benefits of this example on deep neural network (DNN) models across a wide range of applications, as shown in Table 1: Image classification trained on ImageNet (SqueezeNet, VGG, and ResNet-50). Scene Understanding with img2txt, trained on the Microsoft COCO dataset. Natural language modeling, including BERT, a Google Transformer-based model trained on the GLUE dataset, and SNLI, which is trained on the Stanford Natural Language Inference Corpus. Object detection and segmentation: Facebook's Detectron2 model trained on the Microsoft COCO dataset. MovieLens: A recommendation system involving neural collaborative filtering (NCF) trained on a 20M movie ratings dataset.

[0075] To illustrate the benefits of incorporating pruning during dense model training, two variants of ResNet-50 were trained. These variants included the dynamic sparsity reparameterization technique and the sparse momentum technique. Both techniques targeted 90% sparsity.

[0076] In the example experiments, all models were trained on an RTX 2080 Ti GPU using a PyTorch implementation. Each model was trained for the number of epochs necessary to converge to the best reported output accuracy. For each epoch, one randomly selected batch was sampled and the operands of the three operations described herein were traced. Due to different GPU memory requirements, batch sizes varied for each model, ranging from a minimum of 64 samples per batch to a maximum of 143 samples per batch. [Table 1]

[0077] A custom cycle-accurate simulator was used to model performance. Scalable microbenchmark stress tests were run to validate the simulator's fidelity. Table 1 lists the default configuration of the Tensorcore-based baseline, including our extensions. To model area and power consumption, all designs were implemented in Verilog, synthesized with Synopsys Design Compiler, and laid out with Cadence Innovus. Power was estimated by capturing circuit activity through Mentor Graphics ModelSim and then passing it to Innovus. CACTI was used to model the area and energy consumption of on-chip shared SRAM memories. These memories are divided into three large, banked chunks: AM, BM, and CM. CACTI was also used to model the area and energy consumption of the SRAM scratchpad (SP). Finally, the energy and latency of off-chip accesses were modeled via Micron's DRAM model. Both the baseline architecture and the architecture of the present embodiment use CompressingDMA to compress zero values ​​off-chip. [Table 2]

[0078] Figure 14 shows the speedup achieved by incorporating our system into the Tensorcore architecture, broken down by model and by (i) A * W, (ii) A * G, and (iii) W * The chart shows the speedup for each of the three operations, G. The amount and pattern of sparsity in each tensor varies across models, layers, and training phases, so the speedup varies. On average, our system accelerates execution by 1.95x and never introduces any slowdown.

[0079] We determined that this embodiment benefits all models. The advantages of BERT and Dtctron2 derive from their backward passes. Dtcron2 consists of 1) a pre-trained ResNet-50 backbone, 2) a feature pyramid network (FPN) that extracts high-level semantic feature maps at various scales, 3) a region proposal network (RPN) that proposes candidate object bounding boxes, 4) network heads that perform bounding box recognition, classification, and regression, and 5) a network head for mask prediction that is applied separately to each region of interest. The two front-end convolutional layers, FPN and RPN, that dominate the execution time exhibit negligible input activation sparsity. However, the use of ReLU immediately after these dominating layers results in significant (73%–94%) sparsity in the gradients of these layers that system 100 utilizes during the backward pass. BERT's dominance by fully connected-like layers results in almost no sparsity in their weights and activations. However, despite not using ReLU, the sparsity in the gradients during the backward pass is proven to be around 60% sparsity in the gradients of most layers due to the attention mechanism between the encoder and decoder layers.

[0080] ResNet-50, VGG16, and SQZNet are among the models that use ReLU and therefore benefit from the sparsity that ReLU creates. ResNet-50 sees lower gains during the backward pass. This is primarily due to the use of a batch normalization (BatchNorm) layer between each convolutional layer and the subsequent ReLU layer. The BatchNorm layer absorbs almost all of the sparsity in the gradients. Fortunately, however, there is still sparsity in either the activations or weights that are utilized. The use of pruning during training helps the system 100 achieve a low Wattage, especially when the backend layers become smaller. * This results in significant sparsity in the weights used during the G computation.

[0081] SNLI performs natural language inference tasks by recognizing textual entailment between pairs of human-written English sentences. It includes two fully connected (FC) projection layers, two LSTM encoders, and four fully connected classifier layers. Significant sparsity was observed in the gradients and activations, explaining the observed benefits of this implementation. The gradients exhibit greater than 95% sparsity due to the nature of the task and the use of ReLU activations. The input activation sparsity is 63% and 60% for the two front-end projection layers, 0% for the first-class classification layer, and greater than 94% for the other layers.

[0082] The NCF recommendation system consists of four FC layers, with the first layer being the largest. The gradient is W * The sparseness is 83%, which benefits the G operation. In some cases, the first layer has no impact, as it is the most expensive and does not exhibit sparsity in the activations.

[0083] 15 shows the run-time speedup achieved by system 100 during training from the first epoch to convergence. In general, the improvements achieved by system 100 are fairly stable throughout the training process, suggesting that the underlying phenomenon exploited by system 100 is neither transient nor caused by initialization.

[0084] Our measurements reveal two trends: for ResNet50, which uses a pruning-during-training approach, the speedup is slightly higher during the first few epochs, then drops off and stabilizes at around 1.5x. Similarly, we observe slightly more muted behavior for other dynamic sparsity reparameterization techniques. This behavior is due to the pruning algorithm starting by aggressively pruning many weights at the beginning, after which the training process "reclaims" to recover the model's accuracy.

[0085] For dense image classification models where the system 100 utilized sparsity mostly derived from activations and gradients, the speedup tended to follow an inverted U-shaped curve. This behavior was more pronounced for the VGG16 model, where the random initialization of the model resulted in low gains initially. Gains then rapidly increased during the first few epochs as the model rapidly improved by learning which features in the input data were irrelevant to the task. This led to a rapid increase in sparsity in both activations and gradients. The speedup then stabilized until it reached 40%–50% of the training process. It then gradually declined later in the training process as the model began to extract some of the less important features previously discarded to improve accuracy. During the final quarter of the training process, the speedup stabilized as the model parameters were very close to their final values, stabilizing the sparsity of activations and gradients.

[0086] Table 2 provides area and power details. Even without considering on-chip and off-chip memory, the area and power overhead of system 100 is small, requiring only 14% extra silicon area and 9% power for the scheduler and back-end shuffler. Taking into account the speedup, the compute logic alone averages over 1.8x more energy efficient than the bare Tensorcore. Figure 16 shows the energy efficiency of each model for the compute core logic and the entire chip.

[0087] Each of the on-chip AM, BM, and CM memory chunks is 58.6 mm 2 area, but the scratchpad is 3.95 mm from the baseline due to the additional bank. 2 and 5.9mm for System 100 2In total, when considering both the compute and memory area of ​​the entire chip, system 100 had an area overhead of only 4.8%. As shown in Figure 16, when considering accesses to on-chip memory, scratchpad, and off-chip DRAM, system 100 improved the overall energy efficiency of the Tensorcore architecture by 1.5x.

[0088] 17 reports the energy consumed by system 100 relative to the baseline. The measurements also detail the energy consumed across three major components: off-chip data transfer, core logic, and on-chip memory modules. System 100 significantly reduced the energy consumption of the compute core, which dominates the overall energy consumption.

[0089] In general, the depth of the staging buffer limits the maximum number of time steps that can be skipped at one time, thus determining the theoretical peak speedup. For an N-deep buffer, the maximum speedup is N×. However, as the staging buffer becomes deeper, the hardware complexity of the scheduler and the corresponding front-end and back-end shufflers may increase. To investigate the impact of depth configuration, exemplary experiments sweep depths from 2 up to 5. The 2-deep and 3-deep configurations implement four possible promotions per lane instead of eight, for a more balanced implementation of cost vs. performance. Figure 18 shows that the average speedup increases from 1.5× (2-deep) to 2.05× (5-deep), with returns tapering from 4-deep to 5-deep.

[0090] In an exemplary experiment, we investigated the performance behavior of PEs when they are used to configure tiles. To this end, we varied the number of rows and columns of PEs per tile to see how this affected performance. When scaling tile geometry, stalls can occur due to synchronization between PEs, which in turn is caused by work imbalance.

[0091] Figure 18 shows how the performance of system 100 changes when the number of columns is fixed at 4 and the number of rows per tile is varied from 1 to a maximum of 16. The average speedup drops from 2.2x when the tile has one row to 1.8x when the tile has 16 rows. As the number of rows increases, stalls become more frequent due to work imbalances, since all PEs must wait for the slowest one. As the number of rows per tile is scaled up, the data values ​​being processed in parallel exhibit density imbalances among the rows. This can cause some rows to stall because they all must wait for the row with the densest stream of values. In practice, as the number of rows increases, scheduling such a large set of values ​​becomes less likely to skip an entire processing cycle and move on to the next set. This occurs primarily because non-zero activations and gradients tend to cluster in certain 2D feature maps, while other 2D maps become sparser. This clustering phenomenon is fundamental in such models in general, especially towards deeper layers where each filter is trained to extract specific high-level features. In general, this phenomenon is more pronounced in A×G, the second backward convolution, which convolves 2D feature maps of activations and gradients.

[0092] The number of columns per Tensorcore was varied from 4 to 16, while the number of rows remained at 4. Because sparsity was only exploited on one side, the row side, increasing the number of columns did not significantly impact performance compared to an equally scaled baseline. All rows still had to wait for the row with the densest stream. However, increasing the number of columns allowed more windows to be processed in parallel in the convolutional layer while sharing the same schedule along the rows. Negligible degradation was observed in some models, with speedups not exceeding 3%, primarily due to fragmentation caused by the layer dimension.

[0093] In our example experiments, we also experimented with wider Tensorcore configurations with more multiplier lanes. A TensorDash design that still only allows eight promotions per lane was investigated. We found that wider Tensorcores incur less hardware overhead by improving performance even with shallower staging buffers. We achieved nearly the same performance and energy efficiency as narrower Tensorcores with four-deep buffers. Figure 19 shows that TensorDash with three-deep staging buffers on top of 16-wide PEs improves performance by an average of 1.85x, while reducing area overhead by 10% and 3.5% for the compute logic and the entire chip, respectively. As a result of the dedicated sparse connectivity pattern used in this configuration, lanes with non-overlapping connectivity patterns can be grouped and scheduled into a single level, which allows for a single scheduler with only six levels and is not in the critical path.

[0094] To determine whether TensorDash remains effective regardless of the sparsity structure, an example experiment was performed with sparse tensors synthesized with sparsity levels ranging from 10% up to 99.9%. We used the third convolutional layer architecture from SQZNet, but injected tensors using randomly generated values. For each sparsity level, we generated 10 tensor samples. We then used these generated tensors to perform all three operations per sample. We determined the average across all samples for each sparsity level (the maximum deviation in measured speedup was below 5%). Recall that given a 4-deep staging buffer, the maximum speedup possible with TensorDash is 4×, even when the tensor contains only zeros. As Figure 20 shows, performance with TensorDash closely tracked the amount of sparsity in the input. Figure 20 shows that TensorDash approaches ideally possible speedups when the ideal speedup is below 4×. For example, at 20% sparsity, the optimal machine is 1.25x faster. TensorDash is about 1.23x faster. At 90% sparsity, the ideal machine is 10x faster. Example experiments show that TensorDash approaches its ideal 4x speedup, becoming 3.7x faster, and reaching 3.99x at the 99% sparsity level.

[0095] We determined that deep neural networks could be trained using other floating-point data types, such as bfloat16. In our exemplary experiments, we implemented TensorDash and baseline configurations using bfloat16 operations. Even when only the computation logic was considered in the experiments, synthesis and layout results showed that the area and power overhead of TensorDash remained low at 1.16× and 1.1×, respectively. Various components scale differently depending on the data type change. For example, the hardware overhead of the scheduler and shuffler remained unchanged when moving from FP16 to bfloat16, while the multiplier shrank. When considering on-chip memory structures, the area overhead was 4.9%. Regarding energy efficiency, the computation logic using TensorDash was, on average, 1.76× more energy efficient than the baseline. When considering access to on-chip and off-chip memory, the introduction of TensorDash boosted overall energy efficiency by 1.48×.

[0096] In an example experiment, we used GCN, a natural language processing model trained on the Wikitext-2 dataset. There was virtually no sparsity in the activations, gradients, and weights. Even so, with a few layers exhibiting approximately 5% sparsity, TensorDash improved performance by up to 1%. The overall energy efficiency of TensorDash was 0.5% lower than the baseline.

[0097] This embodiment takes advantage of dynamic sparsity and advantageously provides plug-and-play elements that can be used to configure processing tiles. DNN training is typically required in data centers and at the "edge." This embodiment provides a low-level processing element that may be valuable for building accelerators for either segment. This embodiment can interact with several approaches to accelerating training, but because it operates at a very low-level MAC unit, it is primarily complementary to many. We believe this embodiment may be valuable as a replacement PE for many existing hardware accelerators and in conjunction with software-level training acceleration techniques such as pruning.

[0098] The present embodiment advantageously provides a hardware-based technique that enables data-parallel MAC units to exploit sparsity in their input operand streams. When used to construct a hardware accelerator for deep learning, the present embodiment can accelerate the training process while improving energy efficiency. In some cases, a low-cost, sparse input-operand interconnect is combined with an area-efficient hardware scheduler. The scheduler can effectively extract sparsity when it exists in activations, weights, and gradients. A broad set of state-of-the-art models spanning a variety of applications has demonstrated that the training process can be accelerated by up to 1.95x while being more than 1.5x more energy efficient when embedded on top of a Tensorcore-based accelerator with less than 5% area overhead. For the most part, system 100 is data type agnostic, as demonstrated with the Electrical and Electronics Engineers (IEEE) standard mixed-precision floating-point unit and the machine learning-optimized brain floating-point format (BFloat16).

[0099] In one example, in addition to training acceleration, the present embodiments may be used for inference accelerators. In particular, the hardware scheduler described herein can exploit dynamic sparsity in activations while performing forward-pass convolutions between activations and weights. Thus, while this disclosure focuses on training acceleration aspects, the present embodiments can also be used for inference acceleration by exploiting dynamic sparsity in activations and static sparsity in weights. Furthermore, while this disclosure did not specifically describe pre-scheduling input tensors in memory, this is possible to further reduce memory footprint, traffic, and energy during training and inference. Pre-scheduled tensors can be expanded from their initial pre-scheduled form to their corresponding "dense" form in the staging buffer. This can be implemented using a sparse interconnect that mirrors the interconnect described herein to select valid operands. For weights, pre-scheduling can be performed in advance, and possibly in software for inference. For activations and gradients, pre-scheduling can be performed on the output of the preceding layer. Another hardware scheduler, identical to that described herein, pre-schedules the values ​​as they are produced at the output of the layer.

[0100] While the present invention has been described with reference to specific embodiments, various changes and modifications thereof will become apparent to those skilled in the art without departing from the spirit and scope of the invention as set forth in the claims appended hereto.

Claims

1. 1. A computer processor-implemented method for accelerating a deep learning network of processing elements using sparsity, the method comprising: receiving and storing input tensors from a plurality of input streams for subsequent multiply-accumulate (MAC) operations; communicating to a scheduler a bit vector corresponding to the input tensor that identifies which values ​​in the input tensor are non-zero, and determining locations of the non-zero values ​​based on the bit vector; selecting, for each MAC operation, for each corresponding lane of the input tensor, one of the values ​​of the input tensor that form a non-zero value pair and that is communicated for the MAC operation, wherein the selection of the non-zero value comprises: the current value in said lane, the next value in said same lane where such value is non-zero; a value from one step ahead in time where such value is non-zero; and values ​​from adjacent lanes where such values ​​are non-zero; and communicating a selected value for said MAC operation; outputting the value of the MAC operation; A method comprising:

2. 2. The method of claim 1 , wherein the processing element is tiled with a plurality of other processing elements, and wherein receiving and storing the input tensors from the plurality of input streams comprises processing elements along the same row of the input streams sharing an input, and processing elements along the same column of the input streams sharing an input.

3. The method of claim 2 , wherein each processing unit of the plurality of processing units uses a separate scheduler.

4. The method of claim 1 , wherein each processing unit along the same row uses a common scheduler.

5. The method of claim 1 , wherein the lanes are processed such that the last lane of the lanes is arranged in a ring adjacent to the first lane of the lanes.

6. 2. The method of claim 1, wherein communicating a value from an adjacent lane comprises communicating a value from an adjacent lane from one step ahead in time if such value is non-zero.

7. The method of claim 1 , wherein receiving and storing the input tensors from the multiple input streams comprises storing each of the input streams in a respective staging buffer.

8. The method of claim 7 , wherein the staging buffer contains the current step of the input stream and one or more steps ahead in time.

9. The method of claim 1 , wherein the scheduler uses a hierarchical priority scheme.

10. 10. The method of claim 9, wherein the hierarchical priority scheme includes a tight schedule, the tight schedule followed by multiple steps of progressively earlier in time, followed by multiple steps of adjacent lanes progressively earlier in time.

11. 1. A system for accelerating deep learning networks using sparsity, the system including one or more processing units in communication with a data memory for executing processing elements (PEs), each processing element receiving a plurality of input streams; and a staging buffer for receiving and storing input tensors from each of the plurality of input streams for subsequent multiply-accumulate (MAC) operations; a scheduler for receiving a bit vector from each staging buffer to identify which values ​​in a respective one of the input tensors are non-zero and for determining locations of the non-zero values ​​based on the bit vector; Including, For each MAC operation, the scheduler selects, for each corresponding lane of the input tensor, one of the values ​​of the input tensor that forms a non-zero value pair and is communicated to a MAC unit, the selection of the non-zero value being determined by: the current value in said lane, the next value in said same lane where such value is non-zero; a value from one step ahead in time where such value is non-zero; and values ​​from adjacent lanes where such values ​​are non-zero; and The system performs the MAC operation with the MAC unit using the selected value and outputs a value of the MAC operation.

12. 12. The system of claim 11, wherein the processing elements are collectively arranged in a tiled arrangement, with processing elements along the same row of the input stream sharing inputs and processing elements along the same column of the input stream sharing inputs.

13. The system of claim 12 , wherein each processing unit uses a separate scheduler.

14. The system of claim 11 , wherein each processing unit along the same row uses a common scheduler.

15. The system of claim 11 , wherein the lanes are processed such that the last lane of the lanes is arranged in a ring adjacent to the first lane of the lanes.

16. 12. The system of claim 11, wherein communicating a value from an adjacent lane comprises communicating a value from an adjacent lane from one step ahead in time if such value is non-zero.

17. 12. The system of claim 11, wherein the staging buffer contains the current step of the input stream and one or more steps ahead in time.

18. The system of claim 11 , wherein the scheduler uses a hierarchical priority scheme.

19. 20. The system of claim 18, wherein the hierarchical priority scheme includes a tight schedule followed by multiple steps of progressively earlier in time followed by multiple steps of adjacent lanes progressively earlier in time.

20. the scheduler receives two bit vectors from each of the staging buffers associated with values ​​for each lane, and bitwise ORs the two bit vectors to generate an output bit vector indicating the presence of at least one zero; The system of claim 11 , wherein the scheduler uses values ​​in the output bit vector to select lane-by-lane moves such that as many of the non-zero values ​​as possible are processed within a single step.

Images