Neural Network Processing Element
By representing activation and weights as bit-levels in neural network accelerator tiles and processing only non-zero terms, the problem of high energy consumption in convolutional neural networks is solved, and the improvement of time and energy efficiency is achieved.
Patent Information
- Application Number
- CN201980031107.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-05-08
- Filing Date
- 2019-04-25
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2039-04-25
AI Technical Summary
Modern computing hardware consumes a lot of energy when performing calculations, especially in convolutional neural networks, a large number of multiplication operations lead to low energy efficiency, and many multiplication operations are invalid, affecting performance.
A neural network accelerator tile is designed to decompose the activation and weights as bit levels, process multiplication operations of only non-zero terms, and use one-hot encoding, histogram, aligner and accumulation sub-elements to generate nerves The product of the network part reduces invalid calculations.
It significantly reduces the computational workload, improves execution time and energy efficiency, and achieves efficient acceleration of convolutional neural networks.
Smart Images

Figure CN112106078B_ABST
Abstract
Description
Technical Field
[0001] This description relates generally to neural networks, and in particular to processing elements that implement neural networks. Background Art
[0002] Modern computing hardware is energy constrained. Reducing the energy required to perform computations is often critical to improving performance.
[0003] For example, many of the computations performed by a convolutional neural network during inference are due to 2D convolutions. 2D convolutions require a large number of multiply-add operations, with most of the work due to multiplications of activations with weights. Many of these multiplications are inefficient.
[0004] The training or operation or other use of a neural network typically involves performing a large number of calculations. Performing fewer calculations typically results in efficiencies, such as time efficiency and energy efficiency. Summary of the invention
[0005] In one aspect of the present invention, a neural network accelerator tile is provided, comprising: an activation memory interface, the activation memory interface being used to connect to an activation memory interface to receive a set of activation representations; a weight memory interface, the weight memory interface being used to connect to a weight memory interface to receive a set of weight representations; and a processing element, the processing element being configured to implement a one-hot encoder, a histogrammer, an aligner, a reducer, and an accumulation sub-element to process the set of activation representations and the set of weight representations to generate a set of output representations.
[0006] In one aspect of the present invention, a method for generating a partial product of a neural network is provided, comprising: receiving a set of activation representations; receiving a set of weight representations, each weight representation corresponding to an activation representation in the set of activation representations; combining the set of weight representations with the set of activation representations to generate a set of partial results by combining each weight representation with its corresponding activation representation; encoding the set of partial results to generate a set of one-hot representations; accumulating the set of one-hot representations into a set of histogram bucket counts; aligning the counts in the set of histogram bucket counts according to the size of the counts in the set of histogram bucket counts; and reducing the aligned counts in the set of histogram bucket counts to generate the neural network partial product.
[0007] Other aspects and features according to the present application will become apparent to those of ordinary skill in the art when reading the following description of embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The principles of the present invention may be better understood with reference to the accompanying drawings, which are provided by way of illustration of one or more exemplary embodiments in conjunction with the principles and aspects of the present invention, and in which:
[0009] Figure 1 is a set of bar graphs comparing average work reduction for eight exemplary computational arrangements across a set of networks;
[0010] Figure 2 is a schematic diagram of an exemplary convolutional layer;
[0011] Figure 3A , Figure 3B and Figure 3C is a schematic diagram of three exemplary processing engines;
[0012] Figure 4A is a schematic diagram of a processing element according to an embodiment;
[0013] Figure 4B is a schematic diagram of a portion of a processing element according to an embodiment;
[0014] Figure 5A and Figure 5B is a schematic diagram of a splicing unit according to an embodiment;
[0015] Figure 6 is a schematic diagram of an accelerator tile according to an embodiment;
[0016] Fig. 7A and Figure 7B is a schematic diagram of an exemplary accelerator tile configuration;
[0017] Figure 8 is a bar graph comparing the performance of a set of exemplary tile configurations across multiple networks;
[0018] Fig. 9 is a schematic diagram of an exemplary accelerator tile configuration;
[0019] Fig.10 is a schematic diagram of an exemplary accelerator tile configuration; and
[0020] Fig.11 is a schematic diagram of the accumulator configuration.
[0021] In the drawings, the same reference numerals designate the same or corresponding elements. Specific implementation plan
[0022] The following description and the embodiments described therein are provided by illustrating one or more examples of specific embodiments having the principles of the present invention. These examples are provided to explain rather than to limit these principles and the purpose of the present invention. In the following description, the same parts are marked with the same corresponding reference numerals throughout the specification and the drawings. The drawings are not necessarily drawn to scale, and in some cases the scale may be exaggerated to more clearly depict certain features of the present invention.
[0023] This description relates to an accelerator for breaking down multiplications into bit levels to reduce the amount of work performed, such as that performed during inference of an image classification model. This reduction can improve execution time and increase energy efficiency.
[0024] The present description also relates to an accelerator that can improve the execution time of deep neural network (DNN) inference and improve energy efficiency. Although in some embodiments, some work reduction potential is given up, resulting in a low-cost, simple and energy-efficient design.
[0025] Since much modern computing hardware is energy constrained, developing techniques for reducing the energy required to perform computations is often critical to improving performance. For example, most convolutional neural networks perform most of the work during inference attributable to 2D convolutions. These convolutions involve a large number of multiple accumulation operations, with most of the work attributable to the multiplication of the activations A with the weights W. Reducing the number of ineffective operations can greatly improve energy efficiency.
[0026] Various computational arrangements have been proposed to decompose A×W multiplications into a series of simpler operations. For example, if A and W are 16b fixed-point numbers, then A×W can be approximated as 256 1b×1b multiplications or 16 16b×1b multiplications.
[0027] Figure 1 Six bar graphs are included, one for each of the six models used in testing a set of eight exemplary computing arrangements. Each bar graph compares the potential work reduction of the eight compared computing arrangements.
[0028] The leftmost bar 1100 in each bar graph represents a first computational arrangement that avoids multiplications with zero activations and represents a first generation of value-based accelerators that are enabled by a relatively large fraction of zero activations that occur in convolutional neural networks (see, e.g., J. Albericio, P. Judd, T. Hetherington, T. Aamodt, N. Enright Jerger, and A. Moshovos, “CNVLUTIN: Ineffectual-Neuron-Free Deep Neural Network Computing,” in Proceedings of the International Symposium on Computer Architecture, 2016, hereby incorporated by reference).
[0029] The second bar 1200 from the left in each figure represents a second computational arrangement that skips those multiplications where the activations or weights are zero and represents an accelerator that targets sparse models in which a large portion of the synaptic connections have been pruned (see, e.g., A. Parashar, M. Rhu, A. Mukkara, A. Puglielli, R. Venkatesan, B. Khailany, J. Emer, SW Keckler, and WJ Dally, “Scnn: An accelerator for compressed-sparse convolutional neural networks,” in Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA '17, (New York, NY, USA), pp. 27–40, ACM, 2017, hereby incorporated by reference).
[0030] The third bar 1300 and the fourth bar 1400 from the left represent the third calculation arrangement and the fourth calculation arrangement targeting accuracy, respectively. The third computational arrangement represented by bar 1300 targets the precision of activations (see, e.g., P. Judd, J. Albericio, T. Hetherington, T. Aamodt, and A. Moshovos, “Stripes: Bit-serial Deep Neural Network Computing,” in Proceedings of the 49th Annual IEEE / ACM International Symposium on Microarchitecture, MICRO-49, 2016, and A. Delmas, P. Judd, S. Sharify, and A. Moshovos, “Dynamic stripes: Exploiting the dynamic precision requirements of activation values in neural networks,” CoRR, vol. abs / 1706.00504, 2017, both of which are hereby incorporated by reference), and the fourth computational arrangement represented by bar 1400 targets the precision of activations and weights (see, e.g., S. Sharify, A. D. Lascorz, P. Judd, and A. Moshovos, “Loom: Exploiting weight and activation precisions to accelerate convolutional neural networks networks,” CoRR, volume abs / 1706.07853, 2017, hereby incorporated by reference).
[0031] If the multiplications are broken down at the bit level, there is the potential to reduce the work further. For example, assuming that these multiplications operate on 16b fixed-point values, the multiplications are given by equation (1) below:
[0032]
[0033] In the above equation (1), A i and W i are the bits of A and W respectively. When decomposed into 256 individual unit multiplications, only A i and W i Only those multiplications where is non-zero are valid.
[0034] The fifth bar 1500 and the sixth bar 1600 from the left represent the fifth and sixth computing arrangements, respectively, which decompose multiplication into unit multiplications. The fifth computing arrangement represented by bar 1500 skips unit multiplications with zero activation bits (see, for example, J. Albericio, A. Delmas, P. Judd, S. Sharify, G. O'Leary, R. Genov, and A. Moshovos, "Bit-pragmatic deep neural network computing", Proceedings of the 50th Annual IEEE / ACM International Symposium on Microarchitecture, MICRO-50'17, pp. 382-394, 2017, hereby incorporated by reference). The sixth arrangement represented by bar 1600 skips unit multiplications with zero activation or weight bits.
[0035] However, in some arrangements, rather than representing A and W as bit vectors, they may instead be Booth encoded as signed powers of 2 or higher cardinality terms. The seventh bar 1700 and the eighth bar 1800 from the left represent the seventh and eighth computational arrangements, respectively, in which the values are Booth encoded as signed powers of 2 or higher cardinality terms rather than being represented as bit vectors. The seventh arrangement represented by bar 1700 Booth encodes the activation values, while the eighth arrangement represented by bar 1800 Booth encodes both the activation values and the weight values. The multiplication of the activation and the weight is then given by the following equation (2):
[0036]
[0037] In the above equation (2), At i and Wt j Has the form ±2 x As with positional representation, only At i and Wt j Only those products where both are non-zero are valid. Therefore, Figure 1 The potential work reduction is shown where 1700 is used to skip invalid terms for Booth-encoded activations (see, e.g., J. Albericio, A. Delmaas, P. Judd, S. Sharify, G. O'Leary, R. Genov, and A. Moshovos, "Bit-pragmatic deep neural network computing," in Proceedings of the 50th Annual IEEE / ACM International Symposium on Microarchitecture, MICRO-50'17, pp. 382-394, 2017, incorporated herein by reference), and the potential work reduction is shown where 1800 is used to skip invalid terms for Booth-encoded activations and Booth-encoded weights.
[0038] like Figure 1 As indicated, a hardware accelerator that computes only the significant terms (such as Booth-encoded activations and weights) has the potential to greatly reduce computational work. In many embodiments, configurations targeting Booth-encoded computations can also be used to compute bit vector representations.
[0039] The computational arrangement may be used in implementations of neural networks such as convolutional neural networks ('CNNs'). CNNs typically consist of several convolutional layers followed by some fully connected layers, and in image processing, most of the operating time is spent processing convolutional layers, where 3D convolution operations are applied to input activations to produce output activations. An example of a convolutional layer is shown in Figure 2 , which shows a c×x×y input activation block 2100 and a set of N c×h×k filters 2200. The layer connects these N filters (denoted as f 0 、f 1 ,……,f N-1 ) 2200 is dot-multiplied with a c×h×k subarray (or 'window') of the input activations (such as window 2300) to generate a single o h ×o k The output activation is 2400. Convolving N filters and activation windows will produce N o h ×o k Outputs, which are passed to the input of the next layer. The convolution of the activation window and the filter occurs in a sliding window manner with a constant step size S. The fully connected layers can be implemented as convolutional layers where the filters have the same size as the input activations (x=h and y=k).
[0040] Data parallel engines (such as using 16b fixed-point activations and weights) have been proposed for implementing neural networks (see, for example, the DaDianNao accelerator disclosed by Y. Chen, T. Luo, S. Liu, S. Zhang, L. He, J. Wang, L. Li, T. Chen, Z. Xu, N. Sun, and O. Temam, "Dadiannao: A machine-learning supercomputer," Microarchitecture (MICRO), 2014 47th Annual IEEE / ACM International Symposium, pp. 609-622, December 2014, hereby incorporated by reference). In an example of such an engine, which will be referred to as the BASE engine, 8 inner product units (IPs) may be provided, each of which accepts 16 input activations and 16 weights as input. In the case of 8 IPs, the 16 input activations can be broadcast to all 8 IPs, and each IP can receive its own 16 weights; each IP multiplies the 16 input activations with their 16 corresponding weights every cycle and reduces them to a single partial output activation using a 16 32b input adder tree. The partial results can be accumulated over multiple cycles to generate the final output activation. The activations can be provided by the activation memory, and the weight memory can provide the weights.
[0041] Variations of data parallel engines may be used to implement the computational arrangements discussed above, such as the example shown in FIG. 3 .
[0042] Figure 3A is a simplified schematic diagram of a bit-parallel engine 3100, which is shown as multiplying two 4b activations with weight pairs, thereby generating a single 4b output activation per cycle. 4b and other bit sizes are used as examples in various parts of this specification, and other sizes may be used in other embodiments. The throughput of the engine 3100 is two 4b×4b products per cycle.
[0043] Figure 3B is a simplified schematic diagram of a bit-serial engine 3200 (see, e.g., S. Sharify, A. D. Lascorz, P. Judd, and A. Moshovos, “Loom: Exploiting weight and activation precisions to accelerate convolutional neural networks,” CoRR, vol. abs / 1706.07853, 2017, incorporated by reference above). To match the throughput of the bit-parallel engine 3100, the engine 3200 processes 8 input activations and 8 weights per cycle, resulting in 32 1b×1b products. Since the activations and weights are processed bit-serially, the engine 3200 will be processed in P a ×Pw 16 output activations are generated in one cycle, of which P a and P w are activation accuracy and weight accuracy respectively. As a result, engine 3200 Better than Engine 3100. Figure 3B As depicted in FIG, since both activations and weights can be represented by three bits, engine 3200 is relatively Figure 3A The speedup of Engine 3100 is 1.78 times. However, Engine 3200 still handles some invalid entries; for example, in the first cycle, 27 of the 32 1b×1b products are zero and therefore invalid.
[0044] Figure 3C is a simplified schematic diagram of engine 3300, where both activations and weights are represented as vectors of base powers of 2 (or 'single bias'). For example, A 0 =(110) represents the vector A with a single offset 0 =(2,1). Each processing element ('PE') accepts a 4b single offset for the input activation and a 4b single offset for the weight per cycle and adds them together to produce the power of the corresponding product term in the output activation. Since the engine 3300 processes activations and weights serially by term, each PE requires t a ×t w cycles to produce the product term of output activation, where t a and t w is the number of single offsets in the corresponding input activations and weights. The engine processes the next set of activations and weight single offsets after T cycles, where T is the maximum value among all PEs. a ×t w .exist Figure 3C In the example, the maximum value T is 6, corresponding to A from PE(1,0) 0 =(2,1) and W 0 1 =(2,1,0). Therefore, engine 3300 can start processing the next set of activations and weights after 6 cycles, thus Figure 3A The 3100-bit parallel engine achieves a 2.67-fold acceleration.
[0045] Some embodiments of the present invention are designed to minimize the required computations for producing the product of input activations and weights by processing only the primitive bits of the input activations and weights. In some embodiments, the input activations and weights are converted on the fly to a representation containing only primitive bits, and a pair of primitive bits is processed per cycle: one activation bit and one weight bit.
[0046] In some implementations, a hardware accelerator may be provided for processing only base bits, whether those base bits are processed in the form of a single offset, a conventional positional representation, a non-fixed point representation, or other base bit representation.
[0047] In an embodiment where the primitives are processed in the form of a single offset, the accelerator may represent each activation or weight as its single offset ( n ,……,o 0 ). Each single offset is represented as a (sign, value) pair. For example, activation A = -2 (10) =1110 (2) (It has 0010 (2) Booth code) will be represented as (-,1), and A=7 (10) =0111 (2) would be presented as ((+,3),(-,0)). A single bit may be used to encode the sign, where, for example, 0 represents "+" and 1 represents "-".
[0048] In some embodiments, the weight W = (W 项 ) and input activation A=(A 项 ) are each represented as a (symbol, value) pair (s i ,t i ) and (s′ i ,t′ i ), and the product is calculated as stated in equation (3) below:
[0049]
[0050] To implement equation (3), instead of processing the complete A×W product in a single cycle, the accelerator processes each product of a single t′ term of the input activation A and a single t term of the weight W separately. Since these terms are powers of 2, their products will also be powers of 2. Thus, an implementation of equation (3) can first add the corresponding exponent t′+t. If a single product is processed per cycle, 2 t′+t In the case of processing more than one item pair per cycle, an implementation may use one decoder per item pair to compute a single 2 t′+t The products are then accumulated using an efficient adder tree as further described below with reference to exemplary embodiments.
[0051] Figure 4A4000 is a schematic diagram of an exemplary processing element (PE) 4000 such as may be used in an engine or computing arrangement implementing a neural network. PE 4000 implements six steps, each of which is implemented using a processing sub-element. In some embodiments, various sub-elements of a processing element (such as the sub-elements of PE 4000 described below) may be combined or split with various hardware and software implementations. Processing element 4000 is configured to convert 16 weights W 0 ,……,W 15 Multiply by 16 input activation A 0 ,……,A 15 .
[0052] The first subelement is an index subelement 4100 which accepts 16 4b weighted single offsets. 0 ,……,t 15 and its 16 corresponding sign bits s 0 ,……,s 15 and 16 4-bit active single offset t′ 0 ,……,t′ 15 and its symbol s′ 0 ,……,s′ 15 And calculate 16 single offset pair products. Since all single offsets are powers of 2, their products will also be powers of 2. Therefore, to multiply the 16 activations by their corresponding weights, the PE 4000 adds up its single offsets to generate a 5-bit exponent (t 0 +t′ 0 ),……,(t 15 +t′ 15 ), and uses 16 XOR gates to determine the sign of the product.
[0053] The second sub-element is the one-hot encoder 4200. For the i-th pair of activation and weight, where i is ∈{0,…,15}, the one-hot encoder 4200 computes via a 5b to 32b decoder The decoder converts the 5-bit exponent result (t i +t′ i ) is converted to its corresponding one-hot format, i.e. a 32-bit number with 1 '1' bit and 31 '0' bits. A single '1' bit in the jth position of the decoder output corresponds to +2 j or -2 j The value of depends on the sign of the corresponding product, such as Figure 4A E shown i symbol.
[0054] The third sub-element is the histogrammer 4300, which generates the equivalent of a histogram of the decoder output values. The histogrammer 4300 accumulates the 16 32b numbers from the one-hot encoder 4200 into 32 buckets N 0 ,……,N 31 , which corresponds to 2 0 ,……,2 31 The values of , since there are 32 powers of 2. The signs of these numbers are also considered, which are obtained from the one-hot encoder 4200 E i sign. Thereafter, each bucket contains a count of the number of inputs with the corresponding value. Since each bucket has 16 signed inputs, the resulting count will be in the range of -16 to 16 and is therefore represented by 6 bits in 2's complement.
[0055] The fourth and fifth sub-elements are aligner 4400 and reducer 4500, respectively. Aligner 4400 shifts the counts according to their weights, converting all of them into 31+6=37 b, and then the reducer uses a 32-input adder tree to reduce the 32 6b counts to the final output, as Figure 4A As indicated in .
[0056] After reduction, the sixth subelement is the accumulation subelement 4600. The accumulation subelement 4600 accepts the partial sums from the reducer 4500. The accumulation subelement 4600 then accumulates the newly received partial sums with any partial sums held in the accumulator. In this way, the complete A×W product can be calculated in multiple cycles, with one valid single offset per cycle.
[0057] In some embodiments, the sub-element design may be able to better exploit the structure of the information being processed. For example, Figure 4B is a schematic diagram of a splicer, aligner, and reducer sub-element 4700. In some embodiments, the aligner 4400 and reducer 4500 of PE 4000 are replaced by a splicer, aligner, and reducer sub-element 4700, which includes a splicer 4710, an aligner 4720, and a reducer 4730.
[0058] Replacing aligner 4400 and reducer 4500 of PE 4000 with splicer, aligner, and reducer sub-element 4700 has the effect of adding a new splicer, which is sub-element 4710, to PE 4000. Adding a new splicer also allows changes to be made to the aligner and reducer, such as making these sub-elements smaller and more efficient; aligner 4400 is reflectively changed to implement aligner sub-element 4720, and reducer 4500 is reflectively changed to implement reducer sub-element 4730.
[0059] Instead of shifting and adding 32 6b counts, the splicer, aligner, and reducer subelement 4700 seeks to reduce cost and energy by taking advantage of the relative weight of each count by grouping and splicing each count, such as Figure 4B For example, N 0 With N 6 Instead of adding them, they are simply concatenated because they are guaranteed to have no overlapping bits of '1'. The concatenated values are then added through a 6-input adder tree to produce a 38b partial sum to be output to the accumulation sub-element 4600.
[0060] The stitcher, aligner and reducer sub-element 4700 implements a more energy and area efficient adder tree than is possible using the aligner 4400 and reducer 4500 and exploits the fact that the output of the histogrammer 4300 contains groups of digits that do not have overlapping bits of '1'.
[0061] For example, consider the sixth 6b input With the 0th 6b input Add. The adder using the aligner 4400 and the reducer 4500 must make the sixth input N 6 Shifting by 6 bits is equivalent to adding 6 zeros as the 6 least significant bits of the result. In this case, there will be no N shifted by 6. 6 And N 0 The bit position of the bit that has 1. Therefore, (N 6 <<6) and N 0 The sum is equal to N 0 The sign bit is concatenated with N 6 With N 0 Or (N 6 -1) and N 0 , as in Figure 5A Schematically depicted as a splicing unit 5100 in and numerically depicted in the following calculation (4):
[0062] N 6 ×2 6 +N 0 =(N 6 <<6)+N 0
[0063] 1) If is zero:
[0064]
[0065] 2) Otherwise if For one:
[0066]
[0067] Therefore, by dividing those N where (i MOD 6) is equal i This process is recursively applied in groups. That is, the i-th input will be concatenated with the (i+6)-th input, the (i+12)-th input, and so on. Figure 5B The exemplary concatenation unit 5200 implements recursive concatenation as those N where (i MOD 6)=0. i The stack of inputs, although other implementations are possible in other embodiments. For the 16 product units described above, the process produces the following depiction of the six packet counts (5):
[0068]
[0069] The final partial sum is then given by the following equation (6):
[0070]
[0071] exist Figure 6 4 . Tile 6000 is a 2D array of PEs (such as PE 4000 of FIG. 4 ) that process 16 input activation windows and K=8 filters per cycle. PEs along the same column share the same input activations, and PEs along the same row receive the same weights. PE(i,j) receives the next single offset from each input activation from the jth window per cycle and multiplies the single offset by the single offset of the corresponding weight from the i-th filter. When all PEs have completed processing the items of the current set of 16 activations and their corresponding weights, tile 6000 begins processing the next set of activations and weights.
[0072] When a tile such as tile 6000 processes activations and weights serially by term, to match the BASE configuration, the tile must process more filters or windows simultaneously. In the worst case, each activation and weight has 16 terms, so a tile such as tile 6000 should process 8×16=128 filters in parallel to match the peak computational bandwidth of BASE. However, if Figure 1 As indicated in , some exemplary implementations using tiles such as tile 6000 improve the performance of BASE by more than two orders of magnitude with more than 16 filters. In addition, embodiments may use modified Booth coding and reduce the worst case of terms to 8 per weight or activation.
[0073] Fig. 7A and Figure 7B is a schematic diagram of several hardware accelerator configurations, the performance of which is Figure 8 A comparison was made in Figure 7BAs indicated in , the BASE configuration 7100 can use approximately 2,000 wires to process 8 filters with 16 weights per filter. Fig. 7A As indicated in FIG. 7 , using tiles such as tile 6000, configuration 7200 may use 128 wires to process 8 filters, configuration 7300 may use 256 wires to process 16 filters, configuration 7400 may use 512 wires to process 32 filters, and configuration 7500 may use approximately 1,000 wires to process 64 filters. In each of the configurations of FIG. 7 , the number of activation wires is set to 256. In other embodiments, other configurations may be employed, such as fixing the number of filter and weight wires and increasing the number of activation windows and wires.
[0074] Figure 8 The performance of configurations 7100 to 7500 shown in FIG. 7 is compared. Figure 8 The bar graphs of show the relative performance improvements of configurations 7200 through 7500 relative to configuration 7100 as a set of four bars for each of the six models, with the six models identified below each set of bars. The relative improvement for configuration 7200 is shown as the leftmost bar of each group, the relative improvement for configuration 7300 is shown as the second bar from the left of each group, the relative improvement for configuration 7400 is shown as the third bar from the left of each group, and the relative improvement for configuration 7500 is shown as the rightmost bar of each group.
[0075] The simulation of the embodiment of the present invention shows that this embodiment achieves the improvement in execution time, energy efficiency and area efficiency. The execution time of the tested embodiment is modeled using a customized cycle-accurate simulator. The layout post-design simulation is used to test energy and area results. The Synopsys design compiler is used to synthesize the design by TSMC 65nm library (see, for example, Synopsys's " Design Compiler " http: / / www.synopsys.com / Tools / Implementation / RTLSynthesis / DesignC ompiler / Pages, which is hereby incorporated by reference). The synthesis result is used to generate layout with Cadence Innovus (see, for example, Cadence, " Encounter rtl compiler " https: / / www.cadence.com / content / cadencewww / global / en_US / home / train ing / all-courses / 84441.html, which is hereby incorporated by reference). Intel PSG ModelSim is used to generate data-driven activity factors to report exponents. The clock frequency of all designs is set to 1GHz. The ABin and About SRAM buffers are modeled using CACTI (see, e.g., N. Muralimanohar and R. Balasubramonian, “Cacti 6.0: A tool to understand large caches,” 2015, hereby incorporated by reference), and the activation memory and weight memory are modeled as eDRAM using Destiny (see, e.g., M. Poremba, S. Mittal, D. Li, J. Vetter, and Y. Xie, “Destiny: A tool for modeling emerging 3d nvm and edram caches,” in Proceedings of the European Design, Automation & Test Conference (DATE), 2015, pp. 1543-1546, March 2015, hereby incorporated by reference).
[0076] Figure 8The performance of configurations 7200 to 7500 relative to the BASE configuration 7100 for convolutional layers with 100% relative TOP-1 accuracy profiles of Table 1 is indicated, and the performance is compared using the indicated network architectures (see, e.g., Yang Tien-Ju and Chen Yu-Hsin and Sze Vivienne, "Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, hereby incorporated by reference).
[0077] Table 1: Overview of activation and weight accuracy of bits in convolutional layers
[0078]
[0079] In some embodiments, additional rewards are generated by using embodiments of the present invention with certain models, such as models designed to have reduced precision, models using alternative digital representations that reduce the number of bits of '1', or models with increased sparsity of weights or activations. However, embodiments of the present invention target both dense and sparse networks and improve performance by processing only primitives.
[0080] like Figure 8 As indicated in , configuration 7200 outperforms configuration 7100 by more than 2 times on average, and for the sparse network AlexNet-Sparse, configuration 7200 achieves 3 times speedup compared to configuration 7100. Figure 8 As indicated in , configurations 7300 to 7500 achieve up to 4.8x, 9.6x, and 18.3x speedups compared to configuration 7100, respectively.
[0081] The energy efficiency of configurations 7200-7500 relative to configuration 7100 is shown below in Table 2. Since configurations using tiles such as tile 6000 require less on-chip memory and communicate fewer bits per weight and activation, the overall energy efficiency is generally higher.
[0082] Table 2: Relative to BASE 2K The energy efficiency of the computing unit.
[0083]
[0084] Post-placement measurements were used to measure the area of configurations 7100 through 7500. Configuration 7200 requires 1.78 times the area of configuration 7100 while achieving an average speedup of 2.4 times. The area overhead of 7300 is 3.56 times, the area overhead of 7400 is 7.08 times, and the area overhead of 7500 is 14.15 times, while the execution time improvements over configuration 7100 are an average of 4.2 times, 8.4 times, and 16 times, respectively. Thus, tiles such as tile 6000 provide better performance and area scaling than configuration 7100.
[0085] The number of activation and weight pairs processed by a tile or accelerator can vary. Some processor elements process 16 activation and weight pairs per cycle, all of which contribute to the same output activation. Some processor elements process more than 16 activation and weight pairs per cycle. Some accelerator implementations combine multiple tiles or processor elements with the same or different configurations. Fig. 9 9 is a schematic diagram of a tile 9000 containing 32 processor elements 9100 organized in an 8×4 grid. An input register (i.e., a small local memory) provides activation and weight inputs, an activation register 9200 provides an activation input and a weight register 9300 provides a weight input. In some layers, the activation register 9200 provides a weight input and the weight register 9300 provides an activation input. A third register (i.e., memory 9400) is used to store partial or complete output neurons.
[0086] A bus 9110 is provided for each row of processing elements 9100 to connect the processing elements to memory 9400. Partial sums are read out from the grid of processing elements and accumulated in accumulators 9500, and then written to memory 9400 one column of processing elements at a time. There is sufficient time to drain the grid of processing elements via the common bus 9800. Since processing of even a single activation and weight pair input group is typically performed in multiple cycles and multiple activation and weight pair groups can typically be processed before a partial sum needs to be read out, this provides sufficient time for each column of processing elements to sequentially access the common bus 9800 to drain its output while processing elements of other columns are still processing their corresponding activations and weights.
[0087] The tile 9000 includes encoders 9600 between the input registers and the processing element grid, one encoder for each input register. The encoder 9600 converts the value into a series of terms. An optional synthesizer column 9700 provides support for spatial synthesis of 16b arithmetic while maintaining 8b processing elements. The tile 9000 allows activations and weights to be reused in space and time, and minimizes the number of connections or wires required to supply the tile 9000 with activations and weights from the rest of the memory hierarchy.
[0088] For example, tile 9000 may perform 4 activation windows and 8 filters per cycle. In this case, weight register 9300 provides 16 weights for each filter, and activation register 9200 provides corresponding 16 activations for each window. Processing elements 9100 along the same column share the same input activation, while processing elements 9100 along the same row share the same input weight. Encoder 9600 converts input values into items at a rate of one item per cycle because each PE can process a single activation item and a single weight item per cycle. The processing elements in row 'i' and column 'j' multiply the input activation from the jth window by the weight from the ith filter per cycle. Once all activation items have been multiplied by the current weight item, the next weight item is generated. The processing element loops through all corresponding activation items again to multiply the corresponding activation item with the new weight item. When all weight items and activation items have been processed, the product is completed. If there are 3 activation items and 4 weight items, at least 12 cycles will be required. Tile 9000 processes a total of 4 windows (16 activations per window) and 8 filters; 4x16x8 activation and weight pairs are processed simultaneously.
[0089] In practice, the number of terms will vary across weight values and activation values, and therefore, some processing elements will require more cycles than other processing elements to process the product of weight values and activation values. Tile 9000 implicitly treats all activation and weight pairs processed simultaneously as groups and processes them synchronously across different groups; when all processing elements have finished processing all terms of the current group, tile 9000 starts processing the next group. However, this gives up some speedup potential.
[0090] In some embodiments, calculations are allowed to be performed in 16 independent groups. For example, the first synchronization group will contain A 0 , A 16 , A 24 ,……,A 48 and weight W 0 , W 16 ,……,W 112 , the second synchronization group will contain A 1 , A 17 , A 25 ,……,A 49 and weight W 1 , W 17 ,……,W 113 , and so on for the remaining 14 groups. This example is called comb synchronization because the groups physically form a comb pattern on the grid. A set of buffers at the input of the Booth encoder 9600 can be used to allow the groups to slide ahead of each other.
[0091] Some neural networks require 16b data width or precision only for some layers. Some neural networks require 16b data width or precision only for activations, with very few values requiring more than 8b. In some embodiments, tiles support the worst-case data width required across all layers and all values. However, in some embodiments, tiles support data type synthesis in space or time, or both space and time.
[0092] For example, a tile design may allow 16b computations to be performed on an 8b processing element for activations and optionally weights. But other bit widths may also be used. A tile design may be useful for a neural network that requires more than 8b for only some of its layers.
[0093] exist Fig.10 A spatial synthesis tile is shown in . Tile 10000 uses multiple 8b processing elements that have not been modified. The spatial synthesis tile requires an additional processing element each time it processes a layer with 16b values. Tile 10000 is a 2 by 2 grid of 8b processing elements 10100, extended to support the following weight and activation combinations: 8b with 8b, 16b with 8b, and 16b with 16b. The normalized peak compute throughputs are: 1, 1 / 2, and 1 / 4, respectively. To support 16b and 8b computations, the activation items are split into activation items corresponding to their low bytes and high bytes, which are processed by two adjacent processing element ("PE") columns, respectively.
[0094] exist Fig.10 In the example indicated in , PE(0,0) and PE(0,1) process A 0 and A 1 The low byte of the data block is read out, and PE(1,0) and PE(1,1) process the high byte. Row 0 and row 1 process filters 0 and 1, respectively, as before. Processing continues until the accumulator subelement 4600 has accumulated the sum of the high byte and the low byte of the data block. Finally, the partial sum registers are consumed one column at a time. A synthesizer column 10200 is added at the output of the grid. When the partial sum of column 0 is read out, the partial sum is captured in the corresponding temporary register ("tmp") 10300. In the next cycle, the partial sum of column 1 appears on the output bus. Each row adder of synthesizer 10200 adds the two halves and produces the final partial sum. Although the example shows a 2 by 2 grid, the concept is applicable to larger processing element grids without modification. A single synthesizer column is sufficient because each row of the grid uses a common bus to output the partial sum one column at a time. Although not shown, the synthesizer can reuse the accumulation column adder instead of introducing a new group. Typically, supporting 16b activation will require two adjacent row-by-row processor elements.
[0095] Tile 10000 also supports 16b weights by splitting the 16b weights along two rows. This requires four processing elements, each of which is assigned to one of the four combinations of low byte and high byte. Fig.10 In this case, PE(0,0), PE(1,0), PE(0,1), and PE(1,1) will be calculated respectively (A L ,W L )、(A H ,W L )、(A L ,W H ) and (A H ,W D ). The second stage adder in the combiner column is responsible for combining the results from the row with the zero-filled row 1 appropriately.
[0096] Temporal Composition Tiles can also be used. Temporal Composition Tiles will use the Temporal Composition Processing Component. Fig.11 An implementation of a temporal synthesis processing element is shown in . The processing element 11000 supports both 8b and 16b operations, albeit at a lower area cost than a native 16b processing element. The temporal synthesis tile requires an extra cycle each time it must process a 16b value.
[0097] Processing element 11000 allows the items of activation and weight to be split into items belonging to high bytes and low bytes, and to be processed separately in time. The output from the previous adder is appropriately filled with zeros, and then added to the extended precision partial sum. Based on the source of the activation item and weight item being processed, there are three cases: both belong to low bytes (L / L), both belong to high bytes (H / H), or one belongs to high bytes and the other belongs to low bytes (H / L or L / H). Multiplexer 11100 selects the appropriate filling value. The selection signal of the multiplexer can be shared among all processing elements in the tile. Processing 8b values does not cause overhead. Processing 16b activation and 8b weights (or processing 8b activation and 16b weights) requires an extra cycle, while processing 16b weights and activations requires 3 extra cycles. However, this time overhead must only be paid when there is a value that really needs 16b in the case of a tile with processing element 11000.
[0098] Temporal synthesis tiles do not reserve resources for the entire layer, and since values greater than 8b may be rare, it is expected to achieve a higher throughput per processing element than spatial synthesis tiles. However, temporal synthesis tiles require larger processing elements and more complex control at the Booth encoder. In an embodiment, an 8b processing element that can be extended to 16b in time is 22% smaller than a local 16b processing element. In some embodiments, a combination of spatial and temporal designs can be used, such as spatial synthesis for weights and temporal synthesis for activations.
[0099] In some networks and for some layers (particularly the first layer), the required data type as determined by profiling sometimes slightly exceeds 8b. For example, 9b or 10b may be found to be required. In some embodiments, these layers or networks may be implemented using unmodified 8b processing elements and minor modifications to the Booth encoder. For example, in the case where the processing requires a value for the 9th bit (i.e., the bit is 1), since the 8b processing element only supports up to +2 7 The Booth encoder can be efficiently synthesized by 8 : Send +2 7 Twice. Importantly, this will only be needed for values where the 9th bit needs to be 1 and for all negative values in 9b. As an added benefit, this flexibility makes quantization of certain networks easier.
[0100] The embodiments presented above utilize bit-level parallelism between values. However, some embodiments may utilize bit-level parallelism within values, and do this in a different way from bit-parallel hardware. This is possible because the processing element will produce the correct result even if the items with one value are processed spatially rather than temporally. For example, if two input channels are allocated for each weight, and the Booth encoder is modified so that it outputs at most two items per cycle. This also enables the accelerator to utilize bit-level parallelism within the value, which may be useful for reducing synchronization overhead. Utilizing bit-level parallelism within the value can also be used to improve the utilization of the following: weights or activations or activations and weights are not fully reused to fill all columns or rows or both rows and columns. For example, this is the case for fully connected layers that do not reuse weights. This optimization contributes to energy efficiency because fully connected layers are usually memory-limited. This is also useful for depth-separable convolutional layers. When there are not enough filters to fill all rows, it may also be useful to utilize bit-level parallelism within the value. This is different from the intra-value bit-level parallelism exploited by conventional bit-parallel units: the conventional bit-parallel units process all bits regardless of value, while an embodiment of the present accelerator will only process significant bits.
[0101] The performance of an 8b accelerator with spatially synthesizable processing elements has been compared to that of an accelerator with native 16b processing elements. The 8b spatially synthesizable processing elements support 8b and 16b weights and activations, and the 8b accelerator is scaled up to use the same compute area as the 16b accelerator. Tests were performed using the GoogleNet-S and Resnet50-S models. The 8b spatially synthesizable processing accelerator uses as many processing elements as are needed only for those layers that require more than 8b precision. The 8b spatially synthesizable processing accelerator is faster than the native 16b accelerator: 1.4 times faster for GoogleNet-S and 1.2 times faster for Resnet50-S.
[0102] In other embodiments, other configurations may be used, such as increasing the number of weight wires. In some embodiments, such as due to the imbalance between filters aggravated by scaling up only by increasing the number of filters, performance is sublinearly improved. In some embodiments, alternatively or in addition to increasing the number of weights, the number of activations processed simultaneously may be increased. Combining configuration design with a minimum buffer configuration (see, for example, J. Albericio, A. Delmas, P. Judd, S. Sharify, G. O'Leary, R. Genov, and A. Moshovos, "Bit-pragmatic deep neural network computing", Proceedings of the 50th Annual IEEE / ACM International Symposium on Microarchitecture, MICRO-50'17, pp. 382-394, 2017, incorporated by reference above) may also reduce cross-activation imbalance. In other embodiments, activation and weight memories may be distributed along tiles or shared between all or groups of them.
[0103] Embodiments of the present invention are compatible with compression methods (see, e.g., A. Delmas, S. Sharify, P. Judd, M. Nikolic, and A. Moshovos, “Dpred: Making typical activation values matter in deeplearning computing,” CoRR, vol. abs / 1804.06732, 2018, hereby incorporated by reference) and can be expected to perform well with practical off-chip memory configurations and interfaces.
[0104] Note that while portions of the above description and associated figures may describe or suggest the use of hardware, some embodiments may be emulated in software on a processor such as a GPU (graphics processing unit) and may yield similar performance enhancements. Additionally, various components may be dedicated, shared, distributed, or combined with other components; for example, in various embodiments, activation and weight memories that supply activation and weight representations may be dedicated, shared, distributed, or combined.
[0105] Various embodiments of the present invention have been described in detail. Since changes and / or additions may be made to the best mode described above without departing from the nature, spirit or scope of the present invention, the invention is not limited to those details, but only by the appended claims.
Claims
1. A neural network accelerator tile, include: an activation memory interface for connecting with the activation memory interface to receive a set of activation representations; a weight memory interface for connecting to the weight memory interface to receive a set of weight representations, each weight representation corresponding to an activation representation in the set of activation representations; as well as a processing element configured to combine the set of activation representations and the set of weight representations by combining each weight representation with its corresponding activation representation, thereby producing a set of partial results, Wherein, the processing element comprises: a one-hot encoder configured to encode the set of partial results to produce a set of one-hot representations; a histogrammer configured to accumulate the set of one-hot representations into a set of histogram bucket counts; an aligner configured to align the counts in the set of histogram bucket counts according to the magnitude of the counts in the set of histogram bucket counts; and A reducer is configured to reduce the aligned counts in the set of histogram bucket counts to produce a partial sum.
2. The neural network accelerator tile of claim 1 , wherein the activation memory interface is configured to provide the set of activation representations to the processing element as a set of activation single offset pairs, and the weight memory interface is configured to provide the set of weight representations to the processing element as a set of weight single offset pairs.
3. The neural network accelerator tile of claim 2, wherein the processing element is further configured to implement an exponential subelement.
4. The neural network accelerator tile of claim 3, wherein the exponential subelement is configured to combine the set of activation single offset pairs with the set of weight single offset pairs to produce a set of single offset pair products.
5. The neural network accelerator tile of claim 4, wherein the exponential subelement comprises a set of numerical adders and a corresponding set of sign gates, a pair of numerical adders and sign gates providing each single offset pair product in the set of single offset pair products.
6. The neural network accelerator tile of claim 1 , wherein the one-hot encoder comprises a set of decoders for performing one-hot encoding on a set of one-hot encoder inputs.
7. The neural network accelerator tile of claim 2, wherein the histogrammer is configured to sort a set of histogrammer inputs by value.
8. The neural network accelerator tile of claim 2, wherein the aligner is configured to shift a set of aligner inputs to provide a set of shifted inputs to be reduced.
9. The neural network accelerator tile of claim 2, wherein the reducer comprises an adder tree to reduce a set of reducer inputs into the partial sums.
10. The neural network accelerator tile of claim 2, wherein the processing element further comprises an accumulation sub-element, The accumulation subelement includes an accumulator and is configured to receive the partial sum and add the partial sum to the accumulator to accumulate products over a plurality of cycles.
11. The neural network accelerator tile of claim 1 , wherein the processing element is further configured to implement a stitcher.
12. A neural network accelerator tile as described in claim 11, wherein the splicer is provided to splice a set of splicer inputs to produce a set of grouped counts, which are to be shifted and reduced by the aligner and the reducer to produce the partial sums.
13. A neural network processing element comprising a set of neural network accelerator tiles according to claim 1.
14. The neural network processing element of claim 13, wherein the neural network processing element is implemented as a convolutional neural network.
15. The neural network processing element of claim 13, wherein a set of neural network accelerator tiles comprises a column of accelerator tiles, the tiles in the column of accelerator tiles being configured to each receive the same set of activation representations and to each receive a unique set of weight representations.
16. The neural network processing element of claim 13, wherein the processing element is configured to process an activation representation in the set of activation representations with a weight representation in the set of weight representations over a plurality of cycles.
17. A method for generating a partial sum of a neural network by a neural network accelerator tile, the neural network accelerator tile comprising an activation memory interface, a weight memory interface and a processing element, the method include: receiving, at the activation memory interface, a set of activation representations; receiving, at the weight memory interface, a set of weight representations, each weight representation corresponding to an activation representation in the set of activation representations; combining, by a processing element, the set of weight representations with the set of activation representations by combining each weight representation with its corresponding activation representation to produce a set of partial results; encoding, by a one-hot encoder of the processing element, the set of partial results to produce a set of one-hot representations; accumulating, by a histogrammer of the processing element, the set of one-hot representations into a set of histogram bucket counts; aligning, by an aligner of a processing element, the counts in the set of histogram bucket counts according to magnitudes of the counts in the set of histogram bucket counts; as well as The aligned counts in the set of histogram bucket counts are reduced by a reducer of the processing element to produce a partial sum of the neural network.
18. The method of claim 17, further comprising outputting the partial sums of the neural network to an accumulator to accumulate products.
19. The method of claim 18, further comprising outputting the product to an activation memory.
20. The method of claim 17, wherein each activation representation is received as an activation pair having an activation value and an activation sign, and each weight representation is received as a weight pair having a weight value and a weight sign.
21. The method of claim 20, further comprising: include: The counts in the set of histogram bucket counts are recursively grouped into a set of grouped counts prior to aligning the counts in the set of histogram bucket counts, and the set of grouped counts to be aligned and reduced is provided in place of the set of histogram bucket counts.
22. The method of claim 17, wherein combining weight representations in the set of weight representations with activation representations in the set of activation representations is performed over a plurality of cycles.
