Systems and methods for accelerating training of deep learning networks

Abstract

A system and method for accelerating a multiply-accumulate (MAC) floating-point unit during training of a deep learning network is disclosed. The method includes receiving a first input data stream A and a second input data stream B, adding pairs of exponents of the first data stream A and the second data stream B to generate a product exponent, identifying a maximum exponent using a comparator, identifying a number of bits to shift for each mantissa of the second data stream before accumulation by adding a product exponent delta to the corresponding term of the first data stream, reducing the operands of the second data stream to a single partial sum using an adder tree, adding the partial sum to a corresponding alignment value using the maximum exponent to identify an accumulation value, and outputting the accumulation value.

Description

[Technical Field]

[0001] The following relates generally to deep learning networks, and more particularly to systems and methods for accelerating the training of deep learning networks. [Background technology]

[0002] The widespread adoption of deep learning and the demise of Dennard scaling have driven efforts to accelerate deep learning inference and training. These efforts span the entire system stack, from algorithms to middleware and hardware architectures. Training is a task that includes inference as a subtask. Training is a computationally and memory-intensive task that often requires weeks of computational time. Summary of the Invention

[0003] In one aspect, a method is provided for accelerating a multiply-accumulate (MAC) floating-point unit during training or inference of a deep learning network, the method including: receiving a first input data stream A and a second input data stream B; adding a pair of exponents of the first data stream A and the second data stream B to generate a product exponent; identifying a maximum exponent using a comparator; identifying a number of bits to shift for each mantissa of the second data stream before accumulation by adding a product exponent delta to a corresponding term of the first data stream; reducing operands of the second data stream to a single partial sum using an adder tree; adding the partial sum to a corresponding aligned value using the maximum exponent to identify an accumulated value; and outputting the accumulated value.

[0004] In a particular instance of the method, identifying the number of bits to shift of each mantissa of the second data stream prior to accumulation includes skipping ineffective terms that map outside the defined accumulator width.

[0005] In another instance of the method, each mantissa comprises a signed power of two.

[0006] In yet another instance of the method, adding the exponents to identify the maximum exponent is shared among multiple MAC floating point units.

[0007] In yet another instance of the method, the exponent is set to a fixed value.

[0008] In yet another instance of the method, the method further includes storing the floating-point values ​​in groups, where the exponent delta is encoded as a difference from the base exponent.

[0009] In yet another instance of the method, the base index is the first index in the group.

[0010] In yet another instance of the method, using the comparator includes comparing the maximum exponent to a threshold value of an accumulator bit width.

[0011] In yet another instance of the method, a threshold is set to ensure that the model converges.

[0012] In yet another example of the method, the threshold is set to within 0.5% of the training accuracy.

[0013] In another aspect, a system for accelerating a multiply-accumulate (MAC) floating-point unit during training or inference of a deep learning network is provided, the system comprising one or more processors in communication with a data memory, the processors performing: an input module receiving a first input data stream A and a second input data stream B; an exponent module adding pairs of exponents of the first data stream A and the second data stream B to generate a product exponent and identifying a maximum exponent using a comparator; a reduction module identifying a number of bits to shift for each mantissa of the second data stream before accumulation by adding a product exponent delta to a corresponding term of the first data stream and reducing operands of the second data stream to a single partial sum using an adder tree; and an accumulation module adding the partial sum to a corresponding alignment value using the maximum exponent to identify an accumulation value and outputting the accumulation value.

[0014] In a specific example of the system, identifying the number of bits to shift of each mantissa of the second data stream prior to accumulation includes skipping ineffective terms that map outside the defined accumulator width.

[0015] In another version of the system, each mantissa contains a signed power of two.

[0016] In yet another example of the system, the index module, the reduction module, and the accumulation module are located in a processing unit, and the summing of the indexes and the identification of the maximum index are shared among multiple processing units.

[0017] In yet another example of a system, the multiple processing units are arranged in a tiled arrangement.

[0018] In yet another example of a system, processing units in the same column share the same output from the exponent module, and processing units in the same row share the same output from the input module.

[0019] In yet another version of the system, the exponent is set to a fixed value.

[0020] In yet another instance of the system, the system further includes storing the floating-point values ​​in groups, where the exponent delta is encoded as a difference from a base exponent, and the base exponent is the first exponent in the group.

[0021] In yet another instance of the system, using a comparator involves comparing the maximum exponent to a threshold of the accumulator bit width, the threshold being set to ensure that the model converges.

[0022] In yet another implementation of the system, the threshold is set to within 0.5% of the training accuracy.

[0023] These and other aspects are contemplated and described herein. It will be understood that the foregoing Summary of the Invention presents exemplary aspects of embodiments to aid those skilled in the art in understanding the detailed description that follows.

[0024] A better understanding of the embodiments may be obtained by reference to the drawings. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a schematic diagram of a system for accelerating training of deep learning networks, according to an embodiment. [Figure 2] FIG. 2 is a schematic diagram illustrating the system of FIG. 1 and an exemplary operating environment. [Figure 3] 1 is a flowchart of a method for accelerating training of a deep learning network, according to an embodiment. [Figure 4] Illustrative examples of zero terms and out-of-range terms are shown. [Figure 5] 2 illustrates an example embodiment of a processing element including an index module, a reduction module, and an accumulation module according to the system of FIG. 1. [Figure 6]We show an example of the exponential distribution for layer Conv2d_8 at epochs 0 and 89 of training ResNet34 on ImageNet. [Figure 7] 2 illustrates another embodiment of a processing element according to the system of FIG. 1. [Figure 8] 2 illustrates an example of a 2×2 tile of processing elements according to the system of FIG. 1. [Figure 9] An example of values ​​blocked for each channel is shown below. [Figure 10] The performance improvement achieved by the system in Figure 1 over the baseline is shown. [Figure 11] We show the total energy efficiency of the system in Fig. 1 relative to the baseline architecture for each model. [Figure 12] The energy consumption of the system in Figure 1, normalized to the baseline, is shown broken down across three major components: computational logic, off-chip data transfer, and on-chip data transfer. [Figure 13] The breakdown of items that the system in Figure 1 can skip is shown below. [Figure 14] Shows acceleration of each of the three phases of training. [Figure 15] The acceleration of the system in Figure 1 relative to the baseline is shown over time and throughout the training process. [Figure 16] We show the speedup of the system in Figure 1 relative to the baseline when varying the number of rows per tile. [Figure 17] The effect of varying the number of rows per cycle is shown. [Figure 18] We show the accuracy of training ResNet18 by emulating the system in Figure 1 in PlaidML. [Figure 19] We show the performance of the system in Figure 1 with layer-by-layer profiled accumulator width and with a fixed accumulator width. DETAILED DESCRIPTION OF THE INVENTION

[0026] Embodiments will now be described with reference to the drawings. For brevity and clarity of description, where considered appropriate, reference numerals may be repeated among the drawings to indicate corresponding or analogous elements. Furthermore, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, those skilled in the art will recognize that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the embodiments described herein. Moreover, this description should not be considered to limit the scope of the embodiments described herein.

[0027] Any instruction-executing module, unit, component, server, computer, terminal, or device illustrated herein may include or have access to a computer-readable medium, such as a storage medium, computer storage medium, or data storage device (removable and / or non-removable), e.g., magnetic disks, optical disks, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other medium usable to store desired information and accessible by an application, a module, or both. Any such computer storage media may be part of the device or may be accessible or connectable to the device. Any of the applications or modules described herein may be implemented using computer-readable / executable instructions, which may be stored or maintained by such computer-readable media.

[0028] During the training of some deep learning networks, a set of annotated inputs, for which the desired output is known, is processed by repeatedly performing forward and backward passes. The forward pass performs inference, where the output is initially inaccurate. However, if the desired output is known, training can calculate a loss, a metric that indicates how far the output is from the desired output. During the backward pass, this loss is used to adjust the network's parameters, which slowly converge to the best accuracy.

[0029] Numerous approaches have been developed to accelerate training, and fortunately, these approaches can often be used in combination. In distributed training, the training workload is split across multiple computing nodes, exploiting data, model, or pipeline parallelism. Timing communication and computation can further reduce training time. Dataflow optimizations to promote data blocking and maximize data reuse reduce on-chip and off-chip access costs within a node and maximize reuse from low-cost components of the memory hierarchy. Another set of methods reduces the intermediate data footprint required during training. For example, in the simplest form of training, all neuron values ​​generated during the forward pass are retained for use during backpropagation. Instead, this cost is reduced by batching and retaining only one or a few samples. Lossless and lossy compression methods further reduce this data footprint. Finally, in selective backpropagation, the backward pass is modified to propagate the loss to only a subset of neurons, thereby reducing work.

[0030] Meanwhile, the need for increased energy efficiency during inference has led to techniques that are computationally and memory-intensive during training. This includes pruning and quantizing networks during training. Pruning creates opportunities to reduce work and model size during inference by zeroing out weights. Quantization produces shorter, more energy-efficient models that compute with data types such as 16b, 8b, or 4b fixed-point values. Parameter Efficient Training and Memorized Sparse Backpropagation are examples of pruning methods. PACT and outlier-aware quantization are training-time quantization methods. Network architecture search techniques also increase training time as they adjust the model architecture.

[0031] Despite the above, the need to further accelerate training at both the data center and the edge persists unabated. Operational costs, maintenance costs, latency, throughput, and number of nodes are major considerations in data centers. At the edge, where training may primarily be used to improve or enhance already-trained models, energy and latency are major considerations. Improving node performance, regardless of the target application, is highly advantageous. Therefore, the present embodiments can complement existing training acceleration methods. Generally, the majority of the computation and data transfer during training is for performing multiply-accumulate (MAC) operations during the forward and backward passes. As noted above, the use of compression methods can significantly reduce the cost of data transfer. Embodiments of the present disclosure target processing elements for these operations and utilize the inefficient work that naturally occurs during training, the frequency of which is expanded through quantization, pruning, and selective backpropagation.

[0032] Some accelerators rely on the natural occurrence of zeros in many model activations, especially when using ReLU. Several accelerators exist that target pruned models. Another class of designs benefits from reduced value ranges, whether they occur naturally or are the result of quantization. This includes bit-serial designs and designs that accommodate many different data types, such as BitFusion. Finally, another class of designs targets bit-sparsity, exposing inefficient work at the bit level by decomposing multiplications into a series of shift-and-add operations.

[0033] While the above accelerates inference, it presents various practical challenges for training. The first is data type. Models during inference work with a relatively limited range of fixed-point values, while training operates on values ​​that tend to be distributed over a wide range. Therefore, training implementations use floating-point arithmetic, with single-precision IEEE floating-point arithmetic (FP32) being sufficient for virtually all models. In training many models, other data types have been successfully used that facilitate the use of more energy- and area-efficient multiply-accumulate units compared to FP32. These include bfloat16 and floating-point formats below 8b. Furthermore, because floating-point arithmetic is significantly more expensive than integer arithmetic, floating-point arithmetic is used sparingly in mixed-data-type training methods. Despite these proposals, FP32 remains the standard fallback format, especially for training large and challenging datasets. The fixed-point representation used during inference, due to its limited range and lack of exponent, results in zero values ​​(values ​​too small to represent), zero-bit prefixes (small values ​​that can be represented), and bit-sparsity (most values ​​tend to be small, with only a few large values) on which the aforementioned inference accelerators rely. FP32 can represent much smaller values, its mantissa is normalized, and the presence or absence of bit-sparsity has not generally been demonstrated.

[0034] Another challenge is the computational structure. Inference operates on two tensors, weights and activations, performing matrix / matrix or matrix / vector multiplications or pairwise vector operations at each layer to generate activations for the next layer in a feedforward fashion. Training involves this computation as a forward pass, followed by a backward pass that includes a third tensor, the gradient. Most importantly, the backward pass uses activation and weight tensors differently than the forward pass, making it more difficult to pack them efficiently in memory and further complicating the zero suppression performed by inference accelerators that target sparsity. Also relevant to the computational structure is the variability and content of values. Weights are static during inference, but not during training. Furthermore, training initializes the network with random values ​​and then slowly adjusts them. Therefore, it is not necessarily possible to expect values ​​processed during training to exhibit similar behavior, such as sparsity or bit-sparsity. This is even less true for gradients, values ​​that never appear during inference.

[0035] The inventors have demonstrated that the majority of the work performed during training can be considered ineffective. To account for this ineffective work, each multiplication was decomposed into a series of single-bit multiply-add operations. This revealed two sources of ineffective work. First, more than 60% of the calculations are ineffective because one of the inputs is zero. Second, the use of extended precision (e.g., 2 -64 2 64 Even when the arithmetic unit attempts to accumulate to a certain value, the combination of high dynamic range (exponent) and limited precision (mantissa) often results in values ​​that are non-zero but too small to affect the accumulation result.

[0036] The above observations led us to consider whether bit-skip (bit-serial, where zero bits are skipped) processing could be used to exploit these two behaviors. For inference, Bit-Pragmatic is a data-parallel processing element that performs such bit-skip on one operand, while Laconic performs such bit-skip on both operands. These methods operate on fixed-point values ​​because they target inference only. Because there is little bit sparsity in the training weights, converting a fixed-point design to floating-point is not a trivial task. Simply converting Bit-Pragmatic to floating-point resulted in an area-expensive unit with poor performance under ISO computation area constraints. Specifically, compared to an optimized Bfloat16 processing element performing eight MAC operations, an optimized accelerator configuration using Bfloat16 Bit-Pragmatic PEs was, on average, 1.72x slower and 1.96x less energy-efficient under ISO computation constraints. In the worst case, the Bfloat16 Bit-Pragmatic PE was 2.86x slower and 3.2x less energy efficient. Although the Bfloat16 BitPragmatic PE is 2.5x smaller than the bit-parallel PE, and more Bfloat16 BitPragmatic PEs can be used in the same area, it is not possible to fit enough Bfloat16 BitPragmatic PEs to improve performance through the parallelism required in all bit-serial and bit-skip designs.

[0037] This embodiment (informally referred to as FPRaker) provides a processing tile for a training accelerator that exploits both bit-sparsity and out-of-range computations. FPRaker, in some cases, comprises several adder-tree-based processing elements organized into a grid to exploit data reuse in both space and time. The processing elements simultaneously multiply pairs of values ​​and accumulate these products in an output accumulator. The processing elements treat one of the input operands for each multiplication as a series of signed powers of two, previously referred to as terms. This conversion of the operands to powers of two can be performed on the fly, and all operands are stored in floating-point format in memory. The processing elements exploit ineffective work due to zero mantissa bits or due to multiplications that are out-of-range given the current accumulator value. The tiles are designed for area efficiency. In some cases of tiles, the processing elements limit the range of powers of two they can process simultaneously, thereby significantly reducing the cost of their shift-and-add components. Additionally, some tile instances use a common exponent processing unit that is time-multiplexed across multiple processing elements. Additionally, some tile instances share a power-of-two encoder along the rows. Additionally, some tile instances have per-processing element buffers to mitigate the effects of work imbalance between processing elements. Additionally, some tile instances have PEs that implement a low-cost mechanism to reject out-of-range intermediate values.

[0038] Furthermore, in some cases, the present embodiments may advantageously provide at least some of the following features: · Respect the floating-point arithmetic used during training without affecting the numerical precision of the results produced. Skip operations that have no effect due to zero mantissa bits and operations that have no effect due to out-of-range intermediate values. Higher computational throughput compared to other floating-point units, despite multiple cycles of individual MAC operations, given the much smaller processing element per area. Training usually does not need to be universally applicable to all models, and using mixed or short data types accommodates shorter mantissa lengths, thus improving training benefits. It is now possible to select tensor inputs to process in a serial manner per layer, allowing for targeting more sparsity tensors depending on the layer and pass (forward or backward).

[0039] This embodiment also advantageously provides low-overhead memory encoding of floating-point values, relying on the value distribution that is typical for deep learning training. The inventors observed that consecutive values ​​across a channel have similar values, and thus exponents. Therefore, the exponent can be encoded as the delta of a group of such values. These encodings can be used when storing and retrieving chip values, further reducing memory transfer costs.

[0040] Through exemplary experiments, the inventors identified the following experimental observations. Some neural networks naturally exhibit zero values ​​(sparsity) during training, but this is usually limited to activations and gradients unless pruning is used. Term sparsity is typically present in all tensors, including weights, and is much higher than sparsity. Compared to an accelerator that uses optimized bit-parallel FP32 processing elements and can perform 4K bfloat16 MAC per cycle, our PE configuration using the same compute area is 1.5 times faster and 1.4 times more energy efficient. The performance advantage of this embodiment is that all three major operations are generally stable throughout the training process. This embodiment can be used in conjunction with a training method that allows you to specify different accumulator precisions for each layer. For ResNet18, this can improve performance by 38% compared to using fixed-width accumulators.

[0041] We measured the work reduction that was theoretically possible with two related approaches: 1) Approaches that remove all MACs where at least one of the operands is zero (value sparsity, or simply sparsity). 2) An approach that processes only the non-zero bits of the mantissa for one of the operands (bit sparsity).

[0042] To investigate the performance of the present embodiment in different applications, exemplary experiments were conducted. Table 1 lists the models studied in the exemplary experiments. ResNet18-Q is a variant of ResNet18 trained using PACT, which quantizes both activations and weights to 4 bits (4b) during training. ResNet50-S2 is a variant of ResNet50 trained using dynamic sparsity reparameterization, targeting sparse learning that maintains high weight sparsity throughout the training process while achieving accuracy levels comparable to baseline training. SNLI performs natural language inference and consists of a fully connected layer, an LSTM encoder layer, a ReLU layer, and a dropout layer. Image2Text is an encoder-decoder model for generating markup from images. Three models for different tasks from the MLPerf training benchmark were experimented: 1) Detectron2: an object detection model based on Mask R-CNN; 2) NCF: a model for collaborative filtering; and 3) Bert: a Transformer-based model using attention. For the measurements, one randomly selected batch was sampled per epoch for the number of epochs required to train the network to its originally reported accuracy (a maximum of 90 epochs was sufficient for all networks). [Table 1]

[0043] Generally, the majority of the computational effort during training comes from three main operations per layer:

number

[0044] For convolutional layers, the above equation (1) represents the convolution of activations (I) and weights (W), which generates the output activation (Z) during forward propagation. The output Z is passed through an activation function before being used as the input to the next layer. The above equations (1) and (3) represent the calculation of the activation (∂E / ∂I) gradient and weight (∂E / ∂W) gradient, respectively, during backward propagation. Only activation gradients are backpropagated between layers. The weight gradient updates the layer weights once per batch. For fully connected layers, the equation represents some matrix-vector operations. For other operations, the equation represents a vector operation or a matrix-vector operation. For clarity, this disclosure refers to the gradient as G. In these measurements, the term term sparsity is used herein to indicate that the mantissa is first encoded into a signed power of two using canonical encoding, a variant of Booth encoding. This is because bit skip processing is performed on the mantissa.

[0045] In the examples, activations for image classification networks show sparsity exceeding 35% in all cases. This result is expected because these networks typically use a ReLU activation function that clips negative values ​​to zero. However, weight sparsity is typically low, and only some classification models exhibit sparsity in their gradients. However, the remaining models, such as those for natural language processing, may exhibit very low value sparsity across all three tensors. Nevertheless, since models typically exhibit some sparsity, we investigated whether such sparsity could be exploited during training. This is not an easy task because training, unlike inference, exhibits dynamic sparsity patterns across all tensors and a different computational structure during the backward pass. We found that typically, all three tensors exhibit high term sparsity in all models, regardless of the target application. Considering that term sparsity is more common than value sparsity and is present in all models, the present embodiment exploits such sparsity during training to improve model training efficiency.

[0046] By skipping zero terms in the serial input, an ideal potential speedup can be realized due to the reduction in multiplication work. The potential speedup over the baseline can be determined as follows:

number

[0047] This embodiment exploits the bit sparsity of one of the operands used in the three operations (equations (1) to (3) above) performed during training, all of which consist of many MAC operations. By decomposing the MAC operations into a series of shift-and-add operations, inefficient work becomes apparent, providing opportunities to save energy and time.

[0048] To account for the inefficient work during a MAC operation, the operation can be decomposed into a series of "shift-and-add" operations. For multiplication, A=2 Ae ×A m and B=2 Be ×B m are two floating-point values, both with an exponent (A e and B e ) and mantissa (A m and B m ), where the mantissa is normalized and contains an implicit "1". Traditional floating-point units perform this multiplication in a single step (the sign bit is XORed).

number

[0049] A m A signed power of 2 m p By decomposing it into a series p, A=Σ p A m p and A m p =±2 i If so, then the multiplication may be performed as follows:

number

[0050] For example, A m =1.0000001b, A e =10b, B m =1.1010011b, and B e If =11b, then A×B is B m <<(10b+11b-0) and B m <<It can be implemented as two shift-and-add operations of (10b + 11b - 111b). A conventional multiplier performs the ineffective work of 6 bits that are zero. m Process all bits of

[0051] However, the above decomposition reveals that floating-point attempts to represent values ​​with a high dynamic range, resulting in traditional units performing additional inefficient work. Informally, some of the work done during multiplication will be out of range when considering the accumulator value. To understand why this is the case, we need to consider accumulation as well as multiplication. A product A × B is accumulated into a running sum S, and S e is A e +B eAssume that S is much larger than B. Given the limited precision of the mantissa, it is impossible to represent the sum S + A × B. In other cases, even considering the increased mantissa length used to perform rounding, i.e., partial swamping, some of the "shift and add" operations are guaranteed to fall outside the mantissa. Figure 4 shows an illustrative example of a zero term and an out-of-range term. At best, a conventional pipelined MAC unit can power gate the multiplier and accumulator only if the entire multiplication result is out of range after comparing the exponents. However, this opportunity cannot be used to reduce the number of cycles. By decomposing the multiplication into several simpler operations, this embodiment can improve performance by completing the operation in a single cycle if the bits are processed from most significant to least significant, thereby starting another MAC sooner. The same applies when processing multiple A × B products in parallel with an adder tree processing element. A conventional adder tree-based MAC unit can potentially power gate the multiplier and adder tree branch corresponding to the out-of-range product. Cycles may still be consumed. Advantageously, in the present embodiment, the shift-and-add based approach allows such products to be completed in a single cycle, allowing other products to proceed instead.

[0052] 1 and 2 , a system 100 (briefly referred to as FPRaker) for accelerating the training of deep learning networks is shown, according to one embodiment. In this embodiment, system 100 executes on a computing device 26 and accesses content located on a server 32 via a network 24, such as the Internet. In further embodiments, system 100 may execute solely on device 26, solely on server 32, or may execute and / or be distributed on any other computing device, such as a desktop computer, laptop computer, smartphone, tablet computer, server, smartwatch, or distributed computing device(s) or cloud computing device(s). In some embodiments, components of system 100 are stored by and executed on a single computer system. In other embodiments, components of system 100 are distributed among two or more computer systems, which may be distributed locally or remotely.

[0053] FIG. 1 illustrates various physical and logical components of an embodiment of system 100. As illustrated, system 100 includes multiple physical and logical components, including a processing unit 102 (comprising one or more processors), random access memory (“RAM”) 104, input interfaces 106, output interfaces 108, network interfaces 110, non-volatile storage 112, and a local bus 114 that enables processing unit 102 to communicate with other components. Processing unit 102 can execute or direct the execution of various modules, as described in more detail below. RAM 104 provides relatively responsive volatile storage for processing unit 102. Input interface 106 allows an administrator or user to provide input via input devices such as a keyboard and mouse. Output interface 108 outputs information to output devices such as a display and / or speakers. Network interface 110 enables system 100 to communicate with other systems, such as other computing devices and servers located remotely from system 100, as in a typical cloud-based access model. Non-volatile storage 112 stores the operating system and programs, including computer-executable instructions for implementing the operating system and modules, as well as any data used by these services. As described below, additional stored data may be stored in database 116. During operation of system 100, the operating system, modules, and associated data may be retrieved from non-volatile storage 112 and placed in RAM 104 to facilitate execution.

[0054] In an embodiment, system 100 includes one or more modules and one or more processing elements (PEs) 122. In some cases, PEs can be combined into tiles. In an embodiment, system 100 includes an input module 120, a compression module 130, and a transposer module 132. Each processing element 122 includes multiple modules, including an exponent module 124, a reduction module 126, and an accumulation module 128. In some cases, some of the above modules may be at least partially implemented in dedicated or separate hardware, while in other cases, at least some of the functionality in some of the modules is implemented in processing unit 102.

[0055] The input module 120 receives two input data streams, A data and B data, respectively, on which the MAC operation is performed.

[0056] PE 122 performs eight Bfloat16 (A, B) value pair multiplications, simultaneously accumulating the results in accumulation module 128. The Bfloat16 format consists of a sign bit followed by a biased 8b exponent and a normalized 7b mantissa (fraction). Figure 5 shows a baseline PE 122 design that performs calculations in three blocks: exponent module 124, reduction module 126, and accumulation module 128. In some cases, all three blocks can be executed in a single cycle. PEs 122 can be combined to build more area-efficient tiles containing several PEs 122. The mantissa in each of the A operands is converted on the fly to a series of terms (signed powers of 2) using canonical encoding; for example, A=(1.1110000) becomes (+2 +1 , -2 -4) before input to PE 122. All values ​​are maintained in bfloat16 format while in memory. PE 122 processes the value of A in a term-serial manner. Accumulation module 128 has an extended 13b (13-bit) mantissa, including 1b for the leading 1 (hidden bit), 9b for extended precision according to a chunk-based accumulation scheme with a chunk size of 64, and 3b for rounding to nearest even. Accumulation module 128 has an additional 3 integer bits after the hidden bit, allowing it to accommodate the worst-case scenario resulting from accumulating eight products. Overall, accumulation module 128 has a 16b, 4-bit integer part, and 12-bit fractional part.

[0057] PE122 is a 8-bit A exponent A e0 ,···,A e7 , their corresponding eight 3-bit mantissa terms t0, . . . , t7 (after canonical encoding), and the sign bit A s0 ,···,A s7 , both of which contain eight 8-bit B exponents B e0 , , B e7 , these mantissas B m0 , , B m7 (Current situation), and their sign bit B s0 , , B s7 Figure 6 shows an example of the exponential distribution for layer Conv2d_8 at epochs 0 and 89 of training ResNet34 on ImageNet. Figure 6 shows only the used portion of the full range [-127:128] of the 8b exponent.

[0058] The exponent module 124 adds a pair of exponents of A and B to obtain the corresponding product exponent ABe. i The comparator tree uses these product exponents and the accumulator exponent to generate the maximum exponent e max The largest exponent is used to align all products so that they can be summed correctly. To determine the proper alignment for each product, the exponent module 124 calculates all product exponents by emax Subtract from the alignment offset δe i The maximum exponent is also used to discard terms that accumulate out of range. PE122 is calculated as e max All terms that fall outside the -12 range are skipped. In any case, the minimum number of cycles to process 8 MACs is 1 cycle, regardless of value. If one of the resulting products has an exponent larger than the current accumulator exponent, the accumulation module 128 is shifted appropriately (acc shift signal) before accumulation. An example of the exponent module 124 is shown in the first block of Figure 5.

[0059] Since multiplication by a term results in a shift, the reduction module 126 determines the number of bits by which each B mantissa should be shifted before accumulation. These are the 4-bit terms K0,...,K7. K i To calculate i ) into the corresponding term t i To skip out-of-range terms, the reduction module 126 places a comparator before each K term, which compares each K term with a threshold for the available accumulator bit width. The threshold can be set to ensure that the model converges to within 0.5% of the FP32 training accuracy on the ImageNet dataset. Alternatively, the threshold can be effectively controlled by implementing a dynamic bit width accumulator, which can improve performance by increasing the number of "out-of-range" bits to skip. The sign bits of A are XORed with their corresponding sign bits of B to produce the product P s0 ,···,P s7The signs of B are identified. The B mantissas are complemented according to their corresponding product signs and then shifted using offsets K0,...,K7. The reduction module 126 performs the multiplication using a shifter for each B mantissa. In contrast, a conventional floating-point unit requires a shifter at the output of the multiplier. Thus, the reduction module 126 effectively reduces the cost of the multiplier. In some cases, bits shifted out of the accumulator from each B operand may be rounded using a round-to-nearest-even (RNE) approach. The adder tree reduces the eight B operands to one partial sum. An example of the reduction module 126 is shown in the second block of FIG. 5.

[0060] In the accumulation module 128, the partial sums resulting from the reduction module 126 are added to the correctly aligned value in the accumulation register. At each accumulation step, the accumulator register is normalized and rounded using the round-to-nearest-even (RNE) scheme. The normalization block updates the accumulator exponent. When the accumulator value is read, only the 7b mantissa is extracted and converted to bfloat16. An embodiment of the accumulation module 128 is shown in the third block of FIG. 5.

[0061] In the worst case, the accumulation module 128 in the embodiment of FIG. 5 has a 12-bit fraction, so two K iThe offsets can vary by up to 12. This means that the baseline PE122 requires a relatively large shifter and an accumulator tree that can accommodate a wide range of inputs. Specifically, the PE122 requires a shifter capable of shifting an 8b (7b mantissa + hidden bit) value up to 12 positions. If this were integer arithmetic, a 12 + 8 = 20b width would need to be accumulated. However, because this is a floating-point unit, only the most significant 14 bits (1b hidden bit, 12b fraction bits, and sign bit) are accumulated. Any bits below this range are included in the sticky bits, which are the least significant bits of each input operand. This cost can be significantly reduced by taking advantage of the expected exponential distribution. The exponential distribution of ResNet34 layers ensures that the majority of the input exponents, weights, and output gradients lie within a narrow range. This suggests that the exponent delta is relatively small in the general case. Furthermore, the MSB of the activations is guaranteed to be 1 (given the lack of support for denormalized numbers). This indicates that the K0,...,K7 offsets are very often within a narrow range. System 100 exploits this behavior to reduce the area of ​​PE 122. In the exemplary configuration, K i The maximum difference between offsets is limited to a maximum of 3. As a result, the shifter must accommodate a maximum shift of 3b, so the adder must handle a 12b input (1b hidden bit, 7b + 3b mantissa bits, and a sign bit). In this case, the term encoder unit is modified to send groups of A terms with a maximum difference of 3.

[0062] In some cases, processing a group of A values ​​requires multiple cycles because some of them are transformed into multiple terms. During this time, the input to the exponent module 124 remains unchanged. To further reduce area, the system 100 can exploit this expected behavior and share the exponent block across multiple PEs 122. The number of PEs 122 that share the exponent module 124 can be determined based on the expected bit sparsity. The lower the bit sparsity, the longer the processing time per PE 122 and the less frequently a new set of exponents is needed. Therefore, more PEs 122 can share the exponent module 124. Some models are highly sparsity, so in such situations, sharing one exponent module 124 for every two PEs 122 may be optimal. Figure 7 shows another embodiment of a PE 122. Collectively, the PE 122 accepts one set of eight A inputs and two sets of B inputs, B and B', as inputs. The exponent module 124 can process either (A, B) or (A, B') at a time. During the cycle in which the exponent module 124 processes (A, B), the multiplexer for PE#1 is max and the exponent delta directly to PE 122. At the same time, these values ​​are latched in a register in front of PE 122 so that these values ​​remain constant while PE 122 processes all terms of input A. When the exponent block processes (A, B'), the process proceeds to PE #2. In this configuration, both PEs 122 must finish processing all A terms before proceeding to process another set of A values. Because the exponent module 124 is shared, each set of eight A values ​​takes at least two cycles to process (even if the set contains zero terms).

[0063] Utilizing a buffer per PE 122 allows for temporal exploitation of data reuse. To spatially exploit data reuse, the system 100 may arrange several PEs 122 into tiles. FIG. 8 shows an example of a 2×2 tile of PEs 122, with each PE 122 performing eight MAC operations in parallel. Each pair of PEs 122 per column shares the exponent module 124 as described above. The B and B′ inputs are shared across PEs 122 in the same row. For example, during the forward pass, there may be different filters processed on each row and different windows processed across columns. Because the B and B′ inputs are shared, every column must wait for the column with the most Ai terms to finish before proceeding to the next set of B and B′ inputs. To reduce these stalls, the tile may include a buffer per B and B′. Having N such buffers per PE 122 allows a column to advance at most N sets of values.

[0064] The inventors studied the spatial correlation of values ​​during training and found that consecutive values ​​across channels have similar values. This applies to activations, weights, and output gradients. Similar floating-point values ​​have similar exponents, and system 100 can exploit this property via a base-delta compression scheme. As shown in the example of FIG. 9, in some cases, values ​​may be blocked into groups of 32 values ​​per channel, with the exponent of the first value in the group serving as the base, and the delta exponents of the remaining values ​​in the group calculated relative to the base. The bit width (δ) of the delta exponent is dynamically determined for each group and set to the maximum precision of the resulting delta exponent for each group. The bit width (3b) of the delta exponent is added to the header of each group as metadata.

[0065] Figure 10 shows the total memory savings of the normalized exponent footprint after base-delta compression. The compression module 130 uses this compression scheme to reduce off-chip memory bandwidth. At the output of each layer, values ​​are compressed before being written off-chip and restored when read back on-chip.

[0066] The inventors have determined that skipping out-of-range terms can be cost-effective. Processing element 122 may use a comparator for each lane to check whether the lane's current K term is within a threshold accumulator precision value. The comparator may be optimized by a synthesis tool to compare to a constant. Processing element 122 may feedback a signal to the corresponding term encoder indicating that, given the current e_acc value, any subsequent terms from the same input pair are guaranteed to have no effect (be out of range). Thus, system 100 can improve its performance and energy efficiency by skipping processing of subsequent out-of-range terms. Feedback signals indicating out-of-range terms for a particular lane across PEs in the same tile column may be synchronized together.

[0067] In deep learning, data transfer typically accounts for a significant portion of the computational overhead, often dominating energy consumption. Therefore, it is useful to consider what the memory hierarchy needs to do to keep the execution units busy. The training challenge is that while processing the three arrays I, W, and G, the order in which elements are grouped is different for the three key calculations (Equations 1-3 above). However, it is possible to rearrange the arrays when they are read off-chip. To this end, system 100 may store the arrays in memory using a "square" container of 32 x 32 bfloat16 values. This size typically aligns with the typical row size of DDR4 memory, allowing system 100 to achieve high bandwidth when reading values ​​off-chip. The container contains values ​​with coordinates (c, r, k) (channel, row, column) to (c + 31, r, k + 31), where c and k are divisible by 32 (using padding if necessary). The container is stored in channel, column, and row order. When read from off-chip memory, the container values ​​may be stored in the exact same order in a multi-banked on-chip buffer. The tile can then access the data directly, reading eight bfloat16 values ​​per access. The weights and activation gradients may need to be processed in a different order depending on the operation being performed. Typically, each array is always accessed in transposed order during one of the operations. To this end, system 100 may include a transposer module 132 on-chip. In an example, transposer module 132 reads eight blocks of eight bfloat16 values ​​from on-chip memory. Each of these eight reads uses a read width of eight values, and the blocks are written as rows inside the transposer buffer. Collectively, these blocks form an 8x8 block of values. Transposer module 132 may read eight blocks of eight values ​​each and send them to PE 122. Each of these blocks may be read as a column from its internal buffer. In this way, the 8x8 value groups are effectively transposed.

[0068] The inventors conducted exemplary experiments to evaluate the advantages of system 100 compared to a comparable baseline architecture using a conventional floating-point unit.

[0069] A custom cycle-accurate simulator was developed to model the execution time of the System 100 (informally referred to as FPRaker) and the baseline architecture. In addition to modeling timing behavior, the simulator faithfully modeled value transfers and calculations in time, ensuring that the generated values ​​were accurate to a gold standard. The simulator was validated with microbenchmarks. For area and power analysis, both the System 100 design and the baseline design were implemented in Verilog and synthesized using Synopsys Design Compiler on TSMC's 65nm technology and commercial libraries for the given technology. Cadence Innovus was used for layout generation. Intel PSG ModelSim was used to generate data-driven activity coefficients, which were fed into Innovus for power estimation. The baseline MAC unit was optimized for area, energy, and latency. While it is usually impossible to optimize all three, it is possible for the MAC unit. An efficient bit-parallel fused MAC unit was used as the baseline PE. The component multipliers are both area- and latency-efficient and are taken from the DesignWare IP library developed by Synopsys. Additionally, the baseline unit was optimized for deep learning training by reducing the precision of its I / O operands to bfloat16 and performing accumulations with chunk-based accumulation at reduced precision. The area and energy consumption of the on-chip SRAM global buffer (GB) is partitioned into activation memory, weight memory, and gradient memory, modeled using CACTI. The global buffer has an odd number of banks to reduce bank contention for layers with strides greater than 1. The configurations of both system 100 (FPRaker) and the baseline are shown in Table 2. [Table 2]

[0070] To evaluate the system 100, traces of one random mini-batch were collected during the forward and backward passes of each training epoch. All models were trained long enough to achieve the maximum reported top-1 accuracy. To collect the traces, each model was trained on an NVIDIA RTX 2080 Ti GPU, and all inputs and outputs of each layer were saved using Pytorch's forward and backward hooks. For BERT, the BERT base training and fine-tuning training on the GLUE task were tracked. The simulator used the traces to model execution time and collected activity statistics to enable energy modeling.

[0071] Because an embodiment of system 100 processes one of the inputs in a term-serial manner, system 100 uses parallel processing to extract more performance. In one approach, an iso-compute area constraint may be used to identify the number of PE 122 tiles that can fit into the same area as a baseline tile.

[0072] The conventional PEs compared simultaneously processed eight pairs of bfloat16 values ​​and accumulated their sums. Buffers can be included for inputs (A and B) and outputs, allowing for temporal data reuse. To spatially exploit data reuse, multiple PEs 122 can be arranged in a grid, sharing buffers and inputs across rows and columns. Both the system 100 and the baseline were configured with enhanced GPU Tensor-Core-like tiles that perform 8x8 vector-matrix multiplications, with 64 PEs 122 organized in an 8x8 grid, each performing eight MAC operations in parallel.

[0073] After layout, considering only the computation area, the tiles of the embodiment of system 100 occupy 0.22% of the area relative to the baseline tiles. Table 3 reports the corresponding area and power per tile. Therefore, to perform an ISO computation area comparison, the baseline accelerator must be configured with 8 tiles, and system 100 must be configured with 36 tiles. The area of ​​the on-chip SRAM global buffers is 344 mm for activations, weights, and gradients, respectively. 2 , 93.6mm 2 , and 334 mm 2 is. [Table 3]

[0074] Figure 10 shows the performance improvement achieved by system 100 over the baseline. On average, system 100 outperforms the baseline by 1.5x. Of the convolution-based models studied, ResNet18-Q benefits most from system 100, improving performance by 2.04x over the baseline. Training of this network incorporates PACT quantization, which allows most of the activations and weights throughout the training process to fall within 4b or less. This results in high term sparsity, which system 100 exploits. This result demonstrates that system 100 can benefit from specialized quantization methods without requiring specialized hardware for this purpose.

[0075] SNLI, NCF, and Bert are dominated by fully connected layers. While fully connected layers do not reuse weights across different output activations, training can utilize batch processing to maximize weight reuse across multiple inputs (e.g., words) of the same input sentence, thereby increasing tile PE utilization. Bit sparsity results in speedup. For example, in the case of SNLI, due to its high bit sparsity, system 100 achieves a 1.8x speedup over the baseline.

[0076] Figure 11 shows the total energy efficiency of system 100 relative to the baseline architecture for each of the models studied. On average, system 100 is 1.4x more energy efficient than the baseline when considering only the compute logic, and 1.36x more energy efficient when considering everything. The energy efficiency improvement closely tracks the performance gain. For example, the gain is approximately 1.7x higher for SNLI and Detectron2. Quantizing ResNet18-Q increases the energy efficiency of the compute logic by 1.97x. Figure 12 shows the energy consumption of system 100, normalized to the baseline, broken down across three major components: compute logic, off-chip data transfer, and on-chip data transfer. System 100 achieves significant energy reductions in compute logic and off-chip memory due to exponential-based delta compression.

[0077] Figure 13 shows the breakdown of terms skipped by the system 100. This includes two cases: 1) skipping zero terms, and 2) skipping non-zero terms that are out of range due to the limited precision of floating-point representations. Skipping out-of-range terms increases the term sparsity of ResNet50-S2 and Detectron2 by approximately 10% and 5.1%, respectively. Highly sparse (zero-valued) networks such as VGG16 and SNLI benefit least from skipping out-of-range terms because most of their term sparsity comes from zero terms. This is because they have fewer terms to start with. For ResNet18-Q, most of the benefit comes from skipping zero terms because activations and weights are effectively quantized to a value of 4b.

[0078] FIG. 14 shows the acceleration for each of the three phases of training, where A×W is forward propagation, and A×G and G×W calculate the weight gradient and input gradient for backpropagation, respectively. System 100 consistently outperforms the baseline for all three phases. The acceleration depends on the amount of term sparsity and the value distribution of A, W, and G across the model, layers, and training phases. The fewer terms a value has, the more likely system 100 will improve performance. However, because PEs 122 can perform limited shifts per cycle (maximum of three positions), how terms are distributed within a value affects the number of cycles required to process the value. This behavior holds true for the same PE 122 across lanes and across PEs 122 within the same tile. Typically, a set of values ​​processed simultaneously translates into a specific term sparsity pattern. In some cases, system 100 may prefer patterns in which terms are numerically close to each other.

[0079] Figure 15 shows the speedup of the system 100 over the baseline over time and throughout the training process for all networks studied. Measurements reveal three distinct trends. For VGG16, the speedup is higher for the first 30 epochs, then drops by about 15% and levels off. For ResNet18-Q, the speedup increases by about 12.5% ​​after epoch 30 and stabilizes. This can be attributed to the PACT clipping hyperparameter being optimized to quantize activations and weights to 4 bits or less. For the remaining networks, the speedup remains stable throughout the training process. Overall, measurements show that the performance of the system 100 is robust, leading to performance improvements across all training epochs. Regarding the effect of tile organization, as shown in Figure 16, increasing the number of rows per tile reduces performance by an average of 6%. This performance degradation is due to synchronization between the large number of PEs 122 per column. With an increased number of rows, more PEs 122 share the same set of A values. A value of A with more terms than others will affect many PEs 122, requiring them to wait for processing to complete. Because each PE 122 processes a different combination of input vectors, each PE 122 may be affected differently by intra-PE 122 stalls, such as "no terms" stalls or "limited shift" stalls. Figure 17 shows the breakdown over time for each configuration. We can see that increasing inter-PE 122 synchronization stalls also increases stalls in other lanes ("no terms").

[0080] FIG. 3 illustrates a flowchart of a method 300 for accelerating a multiply-accumulate unit (MAC) during training of a deep learning network, according to an embodiment.

[0081] At block 302, the input module 120 receives two input data streams, A data and B data, respectively, on which to perform a MAC operation.

[0082] At block 304, the exponent module 124 adds pairs of exponents of the A and B data to generate product exponents and determines the maximum exponent using a comparator.

[0083] At block 306, the reduction module 126 determines the number of bits to shift for each mantissa of B by adding a product exponent delta to the corresponding term of the A data before accumulation, and reduces the operands of B to a single partial sum using an adder tree.

[0084] At block 308, the accumulation module 128 adds the partial sum to the corresponding alignment value using the largest exponent to determine an accumulation value.

[0085] At block 310, the accumulation module 128 outputs the accumulation value.

[0086] To study the effect of training with FPRaker on accuracy, an exemplary experiment emulated the bit-serial processing of PE122 during end-to-end training in PlaidML, a machine learning framework based on the backend OpenCL compiler. PlaidML was forced to use the mad() function for each multiplication and addition during training. To emulate the PE's processing, the mad() function was overridden in the disclosed embodiment. ResNet18 was trained on the CIFAR-10 and CIFAR-100 datasets. The first line shows the top-1 validation accuracy when trained natively in PlaidML with FP32 precision. The baseline performs bit-parallel MAC with bfloat16 I / O operand precision, which is known to converge and is supported in the art. Figure 18 shows that both emulated versions converge within 0.1% accuracy difference compared to the native training version at epoch 60 for both datasets. This result is expected since the system 100 skips ineffective operations, ie, operations that do not affect the final outcome of the baseline MAC processing.

[0087] Traditionally, training uses bfloat16 for all calculations. In some cases, mixed data type 122 operations may be used, and fixed-point may be used instead for some calculations. In other cases, floating-point may be used, with the number of bits used in the mantissa varying from operation to operation and layer to layer. In some cases, the recommended mantissa precision may be used when training AlexNet and ResNet18 on ImageNet. Figure 19 shows the performance of the system 100 following this approach. The system 100 may dynamically utilize variable accumulator widths per layer to skip ineffective terms mapped outside the accumulator, improving overall performance. When training ResNet18 on ImageNet using profiled accumulator widths per layer, the system 100 achieves 1.51x, 1.45x, and 1.22x accelerations for A × W, G × W, and A × G, respectively. This achieves an overall speedup of 1.56x over the baseline, compared to the 1.13x possible when training with a fixed accumulator width. Adjusting the mantissa length while using a bfloat16 container results in a zero-bit suffix appearing in the mantissa.

[0088] Advantageously, system 100 can perform multiple multiply-add floating-point operations, all contributing to a single final value. Processing elements 122 can be used as building blocks of accelerators for training neural networks. System 100 takes advantage of the relatively high term-level sparsity exhibited by all values ​​during training. While the present embodiment describes the use of system 100 for training, it should be understood that system 100 can also be used for inference. System 100 can be particularly advantageous for models that use floating-point, such as models that process language or recommendation systems.

[0089] Advantageously, system 100 enables efficient precision training. Different precisions can be assigned to each layer during training depending on the layer's sensitivity to quantization. Furthermore, training can start with a lower precision and increase the precision per epoch near the conversion. System 100 can enable dynamic adaptation to different precisions, which can improve performance and energy efficiency.

[0090] System 100 can also be used to perform fixed-point arithmetic. Thus, system 100 can be used to perform training, where some of the operations are performed using floating point and some of the operations are performed using fixed point. To perform fixed-point arithmetic, (1) the exponent is set to a known fixed value, typically a value equivalent to zero, and (2) an external override signal indicates that the mantissa does not contain an implicit leading bit that is one. Furthermore, because the operations performed during training can be a superset of the operations performed during inference, system 100 can be used for inference.

[0091] While the present invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the claims appended hereto.

Claims

1. 1. A method for accelerating a multiply-accumulate (MAC) floating-point unit during training or inference of a deep learning network, the method being executed on one or more processors, the method comprising: receiving a first data stream A and a second data stream B; adding pairs of exponents of the first data stream A and the second data stream B to generate a product exponent; Identifying a maximum exponent using a comparator; prior to accumulation, determining the number of bits to shift for each mantissa in the second data stream (B) by adding a product exponent delta to the corresponding term in the first data stream (A), and reducing the operands in the second data stream (B) to a single partial sum using an adder tree, wherein the product exponent delta is determined as a relative value from the maximum exponent; adding the partial sums to corresponding alignment values ​​using the largest exponent to determine a cumulative value; outputting the accumulated value; A method comprising:

2. 2. The method of claim 1, wherein specifying a number of bits to shift each mantissa in the second data stream B prior to accumulation includes skipping ineffective terms that map outside a defined accumulator width.

3. 2. The method of claim 1, wherein each mantissa comprises a signed power of two.

4. 2. The method of claim 1, wherein the adding of the exponents to identify the maximum exponent is shared among multiple MAC floating point units.

5. The method of claim 1 , wherein the exponent is set to a fixed value.

6. The method of claim 1, further comprising storing floating-point values ​​in groups, wherein the product exponent delta is encoded as a difference from a base exponent.

7. The method of claim 6 , wherein the base index is the first index in the group.

8. The method of claim 1 , wherein using the comparator comprises comparing the maximum exponent to a threshold value of an accumulator bit width.

9. The method of claim 8 , wherein the threshold is set to ensure model convergence.

10. The method of claim 9 , wherein the threshold is set to within 0.5% of the training accuracy.

11. 1. A system for accelerating a multiply-accumulate (MAC) floating-point unit during training or inference of a deep learning network, the system comprising one or more processors in communication with a data memory, the one or more processors: an input module receiving a first data stream A and a second data stream B; an exponent module that adds pairs of exponents of the first data stream A and the second data stream B to generate a product exponent and identifies a maximum exponent using a comparator; a reduction module that determines the number of bits to shift for each mantissa in the second data stream (B) prior to accumulation by adding a product exponent delta to the corresponding term in the first data stream (A), and reduces the operands in the second data stream (B) to a single partial sum using an adder tree, the product exponent delta being determined as a relative value from the maximum exponent; an accumulation module that adds the partial sums to corresponding alignment values ​​using the largest exponent to determine an accumulation value, and outputs the accumulation value; To run the system.

12. 12. The system of claim 11, wherein specifying a number of bits to shift each mantissa in the second data stream B prior to accumulation includes skipping ineffective terms that map outside a defined accumulator width.

13. 12. The system of claim 11, wherein each mantissa comprises a signed power of two.

14. 12. The system of claim 11, wherein the index module, the reduction module, and the accumulation module are located in a processing unit, and the adding of the indexes and the identifying of the maximum index are shared among multiple processing units.

15. The system of claim 14 , wherein the plurality of processing units are arranged in a tiled arrangement.

16. 16. The system of claim 15, wherein processing units in the same column share the same output from the index module and processing units in the same row share the same output from the input module.

17. The system of claim 11 , wherein the exponent is set to a fixed value.

18. 12. The system of claim 11, further comprising storing floating-point values ​​in groups, wherein the product exponent delta is encoded as a difference from a base exponent, the base exponent being a first exponent in the group.

19. 12. The system of claim 11, wherein using the comparator comprises comparing the maximum exponent to a threshold of an accumulator bit width, the threshold being set to ensure model convergence.

20. 20. The system of claim 19, wherein the threshold is set to within 0.5% of the training accuracy.

Images