Multimodal low-precision inner product computation circuit for massively parallel neural inference engines

Abstract

A neural inference chip for computing neural activations is provided. In various embodiments, the neural inference chip is adapted to: receive an input activation tensor comprising a plurality of input activations; receive a weight tensor comprising a plurality of weights; Booth-recode each of the plurality of weights into a plurality of Booth-encoded weights, each Booth-encoded value having an order; multiply the input activation tensor by the Booth-encoded weights to produce a plurality of results for each input activation, each result in the plurality of results corresponding to an order of the Booth-encoded weight; for each order of the Booth-encoded weights, sum the corresponding results to produce a plurality of partial sums, one partial sum for each order; and compute the neural activation from the sum of the plurality of partial sums.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to neural network processing, and more particularly, to a multi-modal low-precision inner product computation circuit for a massively parallel neural inference engine. Summary of the Invention

[0002] According to an embodiment of the present disclosure, a neural inference chip for calculating neural activation is provided. In various embodiments, the neural inference chip is adapted to: receive an input activation tensor comprising a plurality of input activations; receive a weight tensor comprising a plurality of weights; Booth recode each of the plurality of weights into a plurality of Booth-encoded weights, each Booth-encoded value having an order; multiply the input activation tensor by the Booth-encoded weights to produce a plurality of results for each input activation, each result in the plurality of results corresponding to an order of the Booth-encoded weight; for each order of the Booth-encoded weights, sum the corresponding results to produce a plurality of partial sums, one partial sum for each order; and calculate the neural activation from the sum of the plurality of partial sums.

[0003] In some embodiments, the input activation tensor dimension is 1. In some embodiments, the weight tensor dimension is 2.

[0004] In some embodiments, calculating the neural activation comprises shifting each of the plurality of partial sums according to a corresponding order of each of the plurality of partial sums. In some embodiments, calculating the neural activation comprises shifting each of the plurality of partial sums according to a precision of the input activation. In some embodiments, calculating the neural activation comprises applying a nonlinear activation function to the sum of the plurality of partial sums. In some embodiments, summing the corresponding results comprises applying a plurality of carry-save adders.

[0005] According to an embodiment of the present disclosure, a neural inference chip for computing neural activations is provided. In various embodiments, the neural inference chip is adapted to: receive an input activation tensor comprising a plurality of input activations; receive a weight tensor comprising a plurality of weights; Booth-recode each of the plurality of input activations into a plurality of Booth-encoded input activations, each Booth-encoded value having an order; multiply the weight tensor by the Booth-encoded input activations to produce a plurality of results for each weight, each result in the plurality of results corresponding to an order of the Booth-encoded input activation; for each order of the Booth-encoded input activations, sum the corresponding results to produce a plurality of partial sums, one partial sum for each order; and compute the neural activation from the sum of the plurality of partial sums.

[0006] In some embodiments, the input activation tensor dimension is 1. In some embodiments, the weight tensor dimension is 2.

[0007] In some embodiments, calculating the neural activation comprises shifting each of the plurality of partial sums according to a corresponding order of each of the plurality of partial sums. In some embodiments, calculating the neural activation comprises shifting each of the plurality of partial sums according to a precision of the input activation. In some embodiments, calculating the neural activation comprises applying a nonlinear activation function to the sum of the plurality of partial sums. In some embodiments, summing the corresponding results comprises applying a plurality of carry-save adders.

[0008] According to an embodiment of the present disclosure, a method and computer program product for calculating neural activations are provided. An input activation tensor comprising a plurality of input activations is received. A weight tensor comprising a plurality of weights is received. Each of the plurality of weights is Booth-recoded into a plurality of Booth-encoded weights, each Booth-encoded value having an order. The input activation tensor is multiplied by the Booth-encoded weights to produce a plurality of results for each input activation, each of the plurality of results corresponding to an order of the Booth-encoded weight. For each order of the Booth-encoded weight, the corresponding results are summed to produce a plurality of partial sums, one partial sum for each order. A neural activation is calculated from the sum of the plurality of partial sums.

[0009] In some embodiments, the input activation tensor dimension is 1. In some embodiments, the weight tensor dimension is 2.

[0010] In some embodiments, calculating the neural activation comprises shifting each of the plurality of partial sums according to a corresponding order of each of the plurality of partial sums. In some embodiments, calculating the neural activation comprises shifting each of the plurality of partial sums according to a precision of the input activation. In some embodiments, calculating the neural activation comprises applying a nonlinear activation function to the sum of the plurality of partial sums. In some embodiments, summing the corresponding results comprises applying a plurality of carry-save adders.

[0011] According to embodiments of the present disclosure, a method and computer program product for calculating neural activations are provided. An input activation tensor comprising a plurality of input activations is received. A weight tensor comprising a plurality of weights is received. Each of the plurality of input activations is Booth-recoded into a plurality of Booth-encoded input activations, each Booth-encoded value having an order. The weight tensor is multiplied by the Booth-encoded input activations to produce a plurality of results for each weight, each of the plurality of results corresponding to an order of the Booth-encoded input activation. For each order of the Booth-encoded input activation, the corresponding results are summed to produce a plurality of partial sums, one partial sum for each order. A neural activation is calculated from the sum of the plurality of partial sums.

[0012] In some embodiments, the input activation tensor dimension is 1. In some embodiments, the weight tensor dimension is 2.

[0013] In some embodiments, calculating the neural activation includes shifting each of the plurality of partial sums according to a corresponding order of each of the plurality of partial sums. In some embodiments, calculating the neural activation includes shifting each of the plurality of partial sums according to a precision of the input activation. In some embodiments, calculating the neural activation includes applying a nonlinear activation function to the sum of the plurality of partial sums. In some embodiments, summing the corresponding results includes applying a plurality of carry-save adders. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A neural core according to an embodiment of the present disclosure is shown.

[0015] Figure 2 An exemplary inference processing unit (IPU) is shown according to an embodiment of the present disclosure.

[0016] Figure 3 A multi-core inference processing unit (IPU) is shown according to an embodiment of the present disclosure.

[0017] Figure 4 An exemplary Booth recoding according to an embodiment of the present disclosure is shown.

[0018] Figure 5 An exemplary Booth recoding multiplier according to an embodiment of the present disclosure is shown.

[0019] Figure 6 An exemplary Booth recoding multiplier according to an embodiment of the present disclosure is shown.

[0020] Figure 7 An exemplary method for calculating an inner product according to an embodiment of the present disclosure is shown.

[0021] Figure 8 An exemplary method for calculating an inner product according to an embodiment of the present disclosure is shown.

[0022] Figure 9 A method for calculating an inner product according to an embodiment of the present disclosure is shown.

[0023] Figure 10 A multi-precision input data format according to an embodiment of the present disclosure is shown.

[0024] Figure 11A -C shows the parts and generation of various precisions according to an embodiment of the present disclosure.

[0025] Figure 12 A method for calculating a 4-bit inner product according to an embodiment of the present disclosure is shown.

[0026] Figure 13 A method for calculating a 4-bit inner product according to an embodiment of the present disclosure is shown.

[0027] Figure 14 A variable precision method for calculating an inner product according to an embodiment of the present disclosure is shown.

[0028] Figure 15 A method for calculating neural activation according to an embodiment of the present disclosure is shown.

[0029] Figure 16 A computing node according to an embodiment of the present disclosure is depicted. DETAILED DESCRIPTION

[0030] An artificial neuron is a mathematical function whose output is a nonlinear function of its inputs. Two neurons are connected if the output of one neuron is the input of another neuron. A weight is a scalar value that encodes the strength of the connection between the output of one neuron and the input of another neuron.

[0031] A neuron computes its output, called an activation, by applying a nonlinear activation function to the weighted sum of its inputs. A weighted sum is an intermediate result computed by multiplying each input by its corresponding weight and accumulating the products. A partial sum is a weighted sum of a subset of the inputs. The weighted sum of all inputs can be computed in stages by accumulating one or more partial sums.

[0032] A neural network is a collection of one or more neurons. Neural networks are typically divided into groups of neurons, called layers. A layer is a collection of one or more neurons that all receive input from the same layer and all send output to the same layer, typically performing similar functions. An input layer receives input from a source external to the neural network. An output layer sends output to a destination external to the neural network. All other layers are intermediate processing layers. A multilayer neural network is a neural network with more than one layer. A deep neural network is a multilayer neural network with many layers.

[0033] A tensor is a multidimensional array of values. A tensor chunk is a contiguous subarray of elements in a tensor.

[0034] Each neural network layer is associated with a parameter tensor V, a weight tensor W, an input data tensor X, an output data tensor Y, and an intermediate data tensor Z. The parameter tensor includes all the parameters that control the activation function σ of the neurons in the layer. The weight tensor includes all the weights that connect the inputs to the layer. The input data tensor contains all the data consumed as input by the layer. The output data tensor includes all the data calculated as output by the layer. The intermediate data tensor includes any data produced by the layer as an intermediate calculation (such as partial sums).

[0035] The data tensors (input, output, and intermediate values) of a layer can be 3-dimensional, where the first two dimensions can be interpreted as encoding spatial locations, while the third dimension can be interpreted as encoding different features. For example, when the data tensor represents a color image, the first two dimensions encode the vertical and horizontal coordinates within the image, and the third dimension encodes the color at each location. Each element of the input data tensor X can be connected to each neuron via a separate weight, so the weight tensor W typically has 6 dimensions, connecting the 3 dimensions of the input data tensor (input row a, input column b, input feature c) with the 3 dimensions of the output data tensor (output row i, output column j, output feature k). The intermediate data tensor Z has the same shape as the output data tensor Y. The parameter tensor V connects the 3 output data tensor dimensions with an additional dimension o that indexes the parameters of the activation function σ. In some embodiments, the activation function σ does not require additional parameters, in which case the additional dimensions are unnecessary. However, in some embodiments, the activation function σ requires at least one additional parameter, which appears in dimension o.

[0036] The elements of the layer's output data tensor Y can be calculated as in Equation 1, where the neuron activation function σ is configured by a vector of activation function parameters V[i,j,k,:], and the weighted sum Z[i,j,k] can be calculated as in Equation 2.

[0037] Y[i,j,k]=σ(V[i,j,k,:];Z[i,j,k])

[0038] Equation 1

[0039]

[0040] Equation 2

[0041] To simplify the notation, the weighted sum in Equation 2 may be referred to as the output, which is equivalent to using a linear activation function Y[i,j,k]=σ(Z[i,j,k])=Z[i,j,k], with the understanding that the same statement applies without loss of generality when different activation functions are used.

[0042] In various embodiments, the computation of the output data tensor as described above is broken down into smaller problems, each of which can then be solved in parallel on one or more neural cores, or on one or more cores of a traditional multi-core system.

[0043] It is obvious from the above that neural networks are parallel structures. A neuron in a given layer receives input with elements x from one or more layers. i X or other input. Each neuron is based on the input and has elements w iThe weight W is used to calculate its state y∈Y. In various embodiments, the weighted sum of the inputs is adjusted by the bias b, and the result is then passed to the nonlinearity F(·). For example, a single neuron activation can be expressed as y=F(b+∑x i w i )

[0044] Because all neurons in a given layer receive input from the same layer and independently compute their outputs, neuron activations can be computed in parallel. Due to this aspect of the overall neural network, performing computations in parallel distributed cores speeds up the overall computation. Furthermore, within each core vector, operations can be computed in parallel. Even with recurrent inputs, such as when a layer projects back onto itself, all neurons are still updated simultaneously. In effect, recurrent connections are delayed to align with subsequent inputs to the layer.

[0045] Now refer to Figure 1 , depicts a neural core according to an embodiment of the present disclosure. The neural core 100 is a tileable computational unit that computes a block of output tensors. The neural core 100 has M inputs and N outputs. In various embodiments, M=N. To compute the output tensor block, the neural core multiplies the M×1 input tensor block 101 by the M×N weight tensor block 102 and accumulates the products as a weighted sum, which is stored in the 1×N intermediate tensor block 103. The O×N parameter tensor block includes O parameters that specify each of the N neuron activation functions, which are applied to the intermediate tensor block 103 to produce the 1×N output tensor block 105.

[0046] Multiple neural cores can be tiled in a neural core array. In some embodiments, the array is 2-dimensional.

[0047] A neural network model is a set of constants that collectively specify the entire computation performed by the neural network, including the graph of connections between neurons and the weights and activation function parameters for each neuron. Training is the process of modifying the neural network model to perform the desired function. Inference is the process of applying the neural network to inputs to produce outputs without modifying the neural network model.

[0048] An IPU is a type of processor that performs neural network inference. A neural inference chip is a specific physical instance of an IPU.

[0049] refer to Figure 2, an exemplary inference processing unit (IPU) is shown according to an embodiment of the present disclosure. The IPU 200 includes a memory 201 for a neural network model. As described above, the neural network model may include synaptic weights of the neural network to be calculated. The IPU 200 includes an activation memory 202, which may be transient. The activation memory 202 may be divided into input and output areas and stores neuron activations for processing. The IPU 200 includes a neural computing unit 203, which is loaded with the neural network model from the model memory 201. Input activations are provided from the activation memory 202 before each computation step. The output from the neural computing unit 203 is written back to the activation memory 202 for processing on the same or another neural computing unit.

[0050] In various embodiments, the microengine 204 is included in the IPU 200. In this embodiment, all operations in the IPU are directed by the microengine. As described below, in various embodiments, central and / or distributed microengines may be provided. A global microengine may be referred to as a chip microengine, while a local microengine may be referred to as a core microengine or a local controller. In various embodiments, the microengine includes one or more microengines, microcontrollers, state machines, CPUs, or other controllers.

[0051] refer to Figure 3 , shows a multi-core inference processing unit (IPU) according to an embodiment of the present disclosure. IPU 300 includes a memory 301 for neural network models and instructions. In some embodiments, memory 301 is divided into a weight portion 311 and an instruction portion 312. As described above, the neural network model may include synaptic weights of the neural network to be calculated. IPU 300 includes activation memory 302, which may be transient. Activation memory 302 may be divided into an input area and an output area, and stores neuron activations for processing.

[0052] The IPU 300 includes an array 306 of neural cores 303. Each core 303 includes a compute unit 333 that is loaded with a neural network model from the model memory 301 and is operable to perform vector calculations. Each core also includes a local activation memory 332. Input activations are provided from the local activation memory 332 before each computation step. The output from the compute unit 333 is written back to the activation memory 332 for processing on the same or another compute unit.

[0053] The IPU 300 includes one or more network-on-chip (NoC) 305. In some embodiments, a partial sum NoC 351 interconnects the cores 303 and transmits partial sums between them. In some embodiments, a separate parameter distribution NoC 352 connects the cores 303 to the memory 301 to distribute weights and instructions to the cores 303. It will be understood that various configurations of NoCs 351 and 352 are suitable for use in accordance with the present disclosure. For example, broadcast networks, row broadcast networks, tree networks, and switched networks may be used.

[0054] In various embodiments, a global microengine 304 is included in the IPU 300. In various embodiments, a local core controller 334 is included on each core 303. In this embodiment, the direction of operation is shared between the global microengine (chip microengine) and the local core controller (core microengine). Specifically, at 311, computation instructions are loaded from the model memory 301 to the neural compute unit 333 on each core 303 by the global microengine 304. At 312, parameters (e.g., neural network / synaptic weights) are loaded from the model memory 301 to the neural compute unit 333 on each core 303 by the global microengine 304. At 313, neural network activation data is loaded from the activation local activation memory 332 to the neural compute unit 333 on each core 303 by the local core controller 334. As described above, activations are provided to neurons of a particular neural network defined by the model, and the activations can originate from the same or another neural compute unit, or from outside the system. At 314, the neural compute unit 333 performs the computation to generate output neuron activations as directed by the local core controller 334. Specifically, the computation includes applying the input synaptic weights to the input activations. It will be appreciated that various methods may be used to perform such computations, including in silico dendrites and vector multiplication units. At 315, the results from the computation are stored in the local activation memory 332 as directed by the local core controller 334. As described above, these stages may be pipelined to provide efficient use of the neural compute unit on each core. It will also be appreciated that, depending on the requirements of a given neural network, inputs and outputs may be transferred from the local activation memory 332 to the global activation memory 302.

[0055] Thus, the present disclosure provides runtime control of operations in an Inference Processing Unit (IPU). In some embodiments, the microengine is centralized (a single microengine). In some embodiments, IPU computation is distributed (performed by an array of cores). In some embodiments, runtime control of operations is hierarchical, with both the central microengine and the distributed microengines participating.

[0056] The microengine or multiple microengines direct the execution of all operations in the IPU. Each microengine instruction corresponds to several sub-operations (e.g., address generation, load, calculation, storage, etc.). In the distributed case, the core microcode runs on the core microengine (e.g., 334). The core microcode includes instructions to perform complete single tensor operations. For example, a convolution between a weighted tensor and a data tensor. In the context of a single core, the core microcode includes instructions to perform single tensor operations on a subset of locally stored data tensors (and partial sums). The chip microcode runs on the chip microengine (e.g., 304). The microcode includes instructions to perform all tensor operations in the neural network.

[0057] In various embodiments, a vector-matrix multiplier is used to accelerate the computation of synaptic integration. As described above, the activation vector X is multiplied by the weight matrix W. This intermediate result is given as PS=XW. Each column of PS can be calculated as PS j =∑x i w ij In this formula, in low-precision fixed-point calculations, the activation x i and weight w ij It can be, for example, 2 bits, 4 bits, or 8 bits. In an exemplary implementation, the multiplication x is performed i w ij , and sum all the products.

[0058] An exemplary multiplier suitable for this calculation can be implemented in the following manner. Booth recoding is applied to the resulting partial sums, thereby generating n / 2 partial sums for an n-bit multiplier. The partial sums are then compressed by carry-save adders, reducing the number of partial sums from n / 2 to 2. The last 2 partial sums are added to the product using a full carry-propagation adder (or a variant thereof). In these steps, the carry-propagation adder requires complex circuitry. For vector multiplication of an n-element vector X and an n×m-element matrix W, n×m carry-propagation adders are required. To save circuit space, it is desirable to have n×m carry-propagation adders per ∑x i w ij The computation uses only one carry-propagation adder. In general, this can be done by changing ∑x i w ij The order of calculations in the circuit is used to optimize the circuit implementation.

[0059] Furthermore, it is desirable to modify the vector-matrix multiplier to support multiple precisions, such as 2-bit, 4-bit, 8-bit, or higher. It is also desirable to reuse as much circuitry as possible across these multiple computational precisions. In the multiply-first summation-second approach, each multiplier needs to support multiplications of multiple precisions. By changing the order of operations in the circuit, the same data path can be reused for multiple computational precisions.

[0060] In various embodiments, the fixed-point inner product calculation Σx i w ij It is performed by the following steps: Booth recoded partial sum generation; partial sum reduction of Booth recoded partial sums of the same order; and summing all partial sums to obtain the final answer.

[0061] This way, individual multiplier values are not generated. Instead, each multiplier calculation is distributed across the entire inner product calculation. When calculating with different precisions, different amounts of shifting are performed on the partial sums at the summation step. Therefore, a minimal amount of multi-precision circuitry is required.

[0062] refer to Figure 4 , shows an exemplary Booth recoding. Booth recoding can be used by multipliers to generate partial sums. A table lookup (table loop up) is used to recode the values. In this example, Table 1 provides a radix-2 Booth recoding table.

[0063] 3-digit combination 011 +2 010 +1 001 +1 000 0 111 0 110 -1 101 -1 100 -2

[0064] Table 1

[0065] To illustrate the Booth recoding procedure, consider multiplying A by B using radix-4 Booth recoding. First, the Booth recoding table (e.g., Table 1) is checked starting with every other bit to find the overlapping 3 bits of multiplier B. The bits B[1:-1], B[3:1], B[5:3], etc. are used. B[-1] is an extra bit added to the right of the least significant bit of B and is 0. The partial sum vector is generated based on the Booth recoding of the corresponding position of B. For each Booth recoding, the partial sum vector is shifted by two bits. Therefore, {B1, B0, B -1 The partial sum of {B3, B2, B1} starts at bit position 0, while the partial sum of {B3, B2, B1} starts at bit position 2. A carry-save adder is used to compress the number of partial sum vectors to 2. Finally, a carry-propagate adder (or its variant) is used to add the compressed two partial sum vectors to the product.

[0066] refer to Figure 5, an 8-bit Booth recoding multiplier is shown. In this example, a multiplication of multiplicand A (501) and multiplier B (502) is performed. Assume that B is an 8-bit binary number B[7:0]. B[-1]=0 is added to the right of B[0]. To calculate the partial sums 503…506, a Booth table lookup is performed: Booth(B[i+1:i-1]), where i=0,2,4,6. If Booth(B[i+1:i-1])=2, then A*Booth(B[i+1:i-1])=A<<1. If Booth(B[i+1:i-1])=1, then A*Booth(B[i+1:i-1])=A. If Booth(B[i+1:i-1])=0, then A*Booth(B[i+1:i-1])=0. If Booth(B[i+1:i-1])=-1, then A*Booth(B[i+1:i-1])=-A. If Booth(B[i+1:i-1])=-2, then A*Booth(B[i+1:i-1])=-A<<1. Before adding, the partial sum A*Booth(B[i+1:i-1]) is shifted left by i bits.

[0067] For example, the partial sum 503 for i=0 is aligned to positions A and B. The partial sums 504, 505, and 506 are shifted to the left by 2, 4, and 6 bits, respectively. These partial sums 503-506 thus have staggered positions and are said to be of different orders.

[0068] In each case, the partial sum A*Booth(B[i+1:i-1])=A can be calculated from the value A by selecting 0, A or -A or optionally shifting 1 bit to the left. Finally, the four partial sums 503 . . . 506 are added to produce the product A*B 507 .

[0069] refer to Figure 6, showing an example of a Booth recoding multiplier. This example shows a binary calculation of 19*71. Since the Booth recoded value of bit 110 at [B1:B-1] is -1 according to Table 1, the first partial sum 603 is 1111111111101101, which is the 2's complement of the multiplicand 00010011 (601) with sign extension. The second partial sum 000100110 (604) is the multiplicand shifted one position to the left, and the second Boolean recoding of 011 at [B3:B1] is 2. The third partial sum 605 is 000000000, and the third Booth recoded value of 000 at [B5:B3] is 0. The final partial sum 606 is 000010011, and the Booth recoding of 010 at [B7:B5] is 1. The partial sums are 2 bits apart: partial sum 604 is 2 bits to the left of partial sum 603; partial sum 605 is 4 bits to the left of partial sum 603; and partial sum 606 is 6 bits to the left of partial sum 603. Finally, adding all the partial sums produces the correct product 607 in binary format, 19*71=1349.

[0070] refer to Figure 7 , shows an exemplary method for computing the inner product. In this example, for all i, the multiplier computes A i *B i , and then add them together to produce ΣA i *B i Thus, by first calculating each A i *B i and then add them together to get the inner product.

[0071] refer to Figure 8 , showing an exemplary method for calculating the inner product. In this example, instead of calculating individual A i *B i The product of , calculates the sum of the partial sums of the same order, and then adds them together. In particular, for each A i *B i 801…804, the partial sums 811…814, 821…824, 831…834 and 841…844 are calculated as described above. In this example, four partial sums are calculated, corresponding to B i [1:-1], B i [3:1], B i [5:3], B i[7:5]. Partial sums of the same order are summed separately by adder 805 to produce sums 806…809. For example, partial sums 811, 821, 831, and 841 have the same order and are added together by adder 805 to produce sum 806. Partial sums 812, 822, 832, and 842 have the same order, respectively, and are added together to produce sum 807. Partial sums 813, 823, 833, and 843 are added together to produce sum 808. Partial sums 814, 824, 834, and 844 are added together to produce sum 809. Finally, the moving sums 806…809 are separated by two bits and added together to produce the final result ΣA i *B i 810.

[0072] refer to Figure 9 , showing a method for calculating the inner product. In particular, as shown, the inner product ∑A0*B0 is calculated using the sum of partial sums of the same order. At 901, all multiplicands B i Recoded by Booth. At 902, from A i and B i The recoded values of are used to generate partial sums. At 903, the sums of all partial sums of the same order from different multipliers are calculated separately. At 904, the sums of the partial sums are added together via appropriate shifting.

[0073] In this approach, all partial sums of the same order are aligned, and the early summation process is more efficient. For low-precision neural vector-matrix multiplications with large vectors and matrices, each multiplication has a small number of partial sums of different orders. However, a large number of partial sums of the same order come from different multiplications. Therefore, reducing the number of partial sums will lead to a more efficient implementation. For example, a 32×32 matrix with 8-bit precision has 4 partial sums for each multiplication. However, each inner product calculation requires adding 32 partial sums of the same order.

[0074] Another advantage of this approach is that the computational circuitry used in multiple precision modes can be shared. Figure 10 , shows a multi-precision input data format. In this embodiment, the same vector or matrix is interpreted with different precisions. For example, 16-bit data can be used as a 2-element 8-bit vector, a 4-element 4-bit vector, or an 8-element 2-bit vector. Figure 10 As shown in FIG, 8, 8 bits can be used to provide 8-bit pattern activation (1001), 4-bit pattern activation (1002), or 2-bit pattern activation (1003). Similarly, 8 bits can be used to provide 8-bit pattern weight (1004), 4-bit pattern weight (1005), or 2-bit pattern weight (1006). This shows how 8-bit data can be interpreted as 8-bit single, 4-bit double, or 2-bit quad.

[0075] The inner product circuit described above can be used to support multi-precision inner product generation. The Booth recoding and partial sum generation circuits require slight modifications to the input data. The circuit does not need to be modified to add all partial sums of the same order. The final summation circuit simply shifts the sum of the partial sums of the same order by different amounts and then adds them together.

[0076] refer to Figure 11A -C, compare 8 bits ( Figure 11A )、4 digits( Figure 11B ) and 2 digits ( Figure 11C ) mode partial sum generation. This shows how to generate partial sums. Compared to the 8-bit mode, the multiplicand input to the partial sum generator is 4 bits A' i or 2 A's i The multiplicand inputs to the Booth encoder are almost identical bits (because B0[7:5] = B'1[3:1], and B0[7:6] = B"3[1:0]). Only B' i [-1] and B" i [-1] needs to be assumed to be 0.

[0077] Figure 11A It is shown how an 8-bit multiplier calculates an 8-bit product 1117. The partial sums 1113 ... 1116 of the 8-bit multiplier are generated by first calculating the Booth recoding of B0 and then by selecting 0, A0 or -A0 with potential shifts.

[0078] exist Figure 11B In

[15] , 2-way 4-bit multipliers generate A'0*B'0+A'1*B'1. The first B'0 is Booth recoded and used to generate partial sums 1123...1124. These partial sums need to be generated by selecting 0, A'0, or -A'0 with a potential shift to the left by one bit. Similarly, B'1 is Booth recoded and used to generate partial sums 1125...1126 by selecting 0, A'1, or -A'1 with a potential shift of one bit. Unlike the 8-bit multiplier, the partial sums 1123...1124 from A'0 and B'0 are aligned with the partial sums 1125...1126 generated from A'1 and B'1 because they are both Booth recoded at positions from 1 to -1 (which is B'0[1:-1] and B'1[1:-1]). i Finally, all the partial sums 1123…1126 are added together to generate the 4-bit inner product 1127.

[0079] exist Figure 11C, 4-way 2-bit multipliers calculate the inner product A'0*B'0+A'1*B'1+A'2*B'2+A'3*B'3. Partial sum 1133 is generated from A'0 and B'0 by first Booth-recoding B'0 and selecting 0, A'0, or -A'0. Similarly, partial sum 1134 is generated from A'1 and B'1, partial sum 1135 is generated from A'2 and B'2, and partial sum 1136 is generated from A'3 and B'3. All partial sums 1133…1136 are aligned because they are generated from Booth-recoded values at the same bit position. Partial sums 1133…1136 are added together to generate a 2-bit inner product 1137.

[0080] Compared to the 8-bit mode, the multiplicand input to the partial sum generator is 4 bits A' i or 2 A's i The multiplicands input to the Booth encoder are almost the same bits (because B0[7:5]=B'1[3:1], and B0[7:6]=B"3[1:0]). Only B' i [-1] and B" i [-1] needs to be assumed to be 0. The Booth recoding logic can be shared, as can the logic for selecting and shifting the multiplicand.

[0081] refer to Figure 12 , shows a method for calculating a 4-bit inner product. In this embodiment, the same method is used to first add the partial sums. Here, the partial sum summation circuit is the same as Figure 8 . In particular, using the circuitry described for 8-bit mode, the first partial sums (e.g., 1201) from each multiplier are collected to compute sum 1205. Similarly, the second partial sums (e.g., 1202) are added to generate sum 1206, the third partial sums (e.g., 1203) are added to generate sum 1207, and the fourth partial sums (e.g., 1204) are added to generate sum 1208. To accommodate 4-bit mode calculations, a different shift amount is applied to each of sums 1205…1208 before the final summation, producing result 1209.

[0082] For each partial sum calculation, the first partial sum 1201 and the second partial sum 1202 have different orders, so compared with partial sum 1201, partial sum 1202 is shifted to the left by 2 places. However, the third partial sum 1203 has the same order as partial sum 1201, and these two partial sums are aligned before the final addition. Similarly, the fourth partial sum 1204 is aligned with the second partial sum 1202, but is shifted to the left by 2 places compared with partial sum 1203. Sum 1206 is shifted to the left by 2 places compared with the sum of partial sums 1205. However, the third sum 1207 has the same order as sum 1205, and these two sums are aligned before the final addition. Similarly, the fourth sum 1208 is aligned with the second sum 1206, but is shifted to the left by 2 places compared with sum 1207. The shift control of the 4-bit mode is different from the 8-bit mode, but it only needs to be implemented once before the final addition for calculating result 1209, rather than being implemented at each multiplier.

[0083] refer to Figure 13 , showing a method for computing a 2-bit inner product. Similar to Figure 12 4-bit inner product calculation, 2-bit calculation uses the same as 8-bit mode (such as Figure 8 The same data path is used to add the partial sums together. However, the final sum is obtained by adding the sum of the partial sums without any shifting.

[0084] By first summing partial sums of the same order, data paths can be shared for multiple precisions, providing, for example, 8-bit, 4-bit, and 2-bit modes. The Booth recoder and partial sum generator use slightly modified inputs between the various precision configurations. The summation circuitry for partial sums of the same order is identical. The final summation requires varying amounts of shifting, depending on the precision. This results in a more compact design than alternatives. Compared to the 8-bit configuration, the 4-bit mode achieves twice as many multiply-accumulate operations per cycle, and the 2-bit mode achieves four times as many multiply-accumulate operations.

[0085] refer to Figure 14 , shows a method for computing variable precision inner products using sums of partial sums of the same order. At 1401, all multiplicands Bi are Booth recoded. At 1402, partial sums are generated from the recoded values of Ai and Bi. At 1403, all partial sums of the same order are summed. At 1404, the sum of the partial sums is shifted according to the precision of the partial sums. At 1405, the partial sums are added to produce the result.

[0086] refer to Figure 15, a method for calculating neural activations is shown. At 1501, an input activation tensor comprising a plurality of input activations is received. At 1502, a weight tensor comprising a plurality of weights is received. At 1503, each of the plurality of weights is Booth re-encoded into a plurality of Booth-encoded weights, each Booth-encoded value having an order. At 1504, the input activation tensor is multiplied by the Booth-encoded weights, thereby producing a plurality of results for each input activation, each result in the plurality of results corresponding to an order of the Booth-encoded weight. At 1505, for each order of the Booth-encoded weight, the corresponding results are summed to produce a plurality of partial sums, one partial sum for each order. At 1506, a neural activation is calculated from the sum of the plurality of partial sums.

[0087] As described above, various embodiments of the present invention include a chip for computing the inner product of two vectors by Booth recoding each element of the multiplication vector. Partial sums are generated using the elements of the multiplicand vector and the recoded multiplier values. All partial sums of the same order are added. The sum of partial sums of the same order is performed by move addition. In some embodiments, the partial sum addition is performed using a carry-save adder tree. In various embodiments, multiple instances of a vector multiplier are combined to form a vector-matrix multiplier. In various embodiments, multiple instances are combined to form a matrix-matrix multiplier.

[0088] In various embodiments, multiple precisions are supported by first multiplying the Booth recode elements by a vector of precisions. Partial sums can then be generated based on the precisions. All partial sums of the same order are summed. The sum of the partial sums is shifted based on the precisions and then added together.

[0089] Now refer to Figure 16 , a schematic diagram illustrating an example of a computing node. Computing node 10 is merely one example of a suitable computing node and is not intended to impose any limitations on the scope of use or functionality of the embodiments described herein. Regardless, computing node 10 is capable of implementing and / or performing any of the functions set forth above.

[0090] In computing node 10, there is a computer system / server 12, which can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, etc.

[0091] Computer system / server 12 may be described in the general context of computer system-executable instructions, such as program modules, executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer system / server 12 may be practiced in a distributed cloud computing environment, where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0092] like Figure 16 As shown in FIG, computer system / server 12 in computing node 10 is shown in the form of a general-purpose computing device. Components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and bus 18 that couples various system components including system memory 28 to processor 16.

[0093] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

[0094] In various embodiments, one or more inference processing units (not shown) are coupled to bus 18. In this embodiment, the IPU may receive data from or write data to memory 28 via bus 18. Likewise, the IPU may interact with other components via bus 18 as described herein.

[0095] Computer system / server 12 typically includes a variety of computer system readable media. This media can be any available media that can be accessed by computer system / server 12, and it includes volatile and non-volatile media, removable and non-removable media.

[0096] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown and commonly referred to as a "hard drive"). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") may be provided, as well as an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM, or other optical media. In such cases, each may be connected to the bus 18 via one or more data media interfaces. As will be further depicted and described below, the memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present disclosure.

[0097] A program / utility 40 having a set (at least one) of program modules 42, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in memory 28 by way of example and not limitation. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0098] Computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, etc.; one or more devices that enable a user to interact with computer system / server 12; and / or any device that enables computer system / server 12 to communicate with one or more other computing devices (e.g., a network card, modem, etc.). This communication may occur via input / output (I / O) interface 22. However, computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), via network adapter 20. As shown, network adapter 20 communicates with other components of computer system / server 12 via bus 18. It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems.

[0099] The present disclosure may be implemented as a system, method, and / or computer program product.The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present disclosure.

[0100] A computer-readable storage medium can be a tangible device that can retain and store instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted by a wire.

[0101] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0102] The computer-readable program instructions for performing the operation of the present disclosure can be assembly instructions, instruction set architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages (such as Smalltalk, C++, etc.) and traditional procedural programming languages (such as "C" programming language or similar programming languages). The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (such as, using an internet service provider through the internet). In certain embodiments, the electronic circuit comprising, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can execute the computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions, so as to perform aspects of the present disclosure.

[0103] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer-readable program instructions.

[0104] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can direct the computer, programmable data processing device, and / or other equipment to operate in a specific manner, such that the computer-readable storage medium having the instructions stored therein comprises an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0105] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0106] The flow charts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a module, segment or portion of an instruction, which includes one or more executable instructions for realizing a specified logical function. In some alternative implementations, the functions mentioned in the frame may not occur in the order mentioned in the figure. For example, the two frames shown in succession can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order, depending on the functions involved. It will also be noted that each frame of the block diagram and / or flow chart illustration and the combination of the frames in the block diagram and / or flow chart illustration can be implemented by a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0107] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, practical applications, or improvements over existing technologies in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for calculating neural activation, the method comprising: receiving an input activation tensor comprising a plurality of input activations, wherein the input activation tensor represents an image, each of the plurality of input activations corresponding to a value at a location in the image; Receives a weight tensor containing multiple weights; Booth recoding each of the plurality of weights into a plurality of Booth-encoded weights, each Booth-encoded value having an order; multiplying the input activation tensor by the Booth encoding weights to produce a plurality of results for each input activation, each result in the plurality of results corresponding to the order of the Booth encoding weights; For each order of the Booth encoding weights, summing the corresponding results to produce a plurality of partial sums, one partial sum for each order; A neural activation is calculated from the sum of the plurality of partial sums.

2. The method according to claim 1, wherein The input activation tensor has a dimension of one.

3. The method according to claim 1, wherein The weight tensor has a dimension of two.

4. The method according to claim 1, wherein Calculating the neural activation includes shifting each of the plurality of partial sums according to a corresponding order of each of the plurality of partial sums.

5. The method according to claim 1, wherein Computing the neural activation includes shifting each of the plurality of partial sums according to a precision of the input activation.

6. The method according to claim 1, wherein Computing the neural activation includes applying a non-linear activation function to the sum of the plurality of partial sums.

7. The method according to claim 1, wherein Summing the corresponding results includes applying a plurality of carry-save adders.

8. A method for calculating neural activation, the method comprising: receiving an input activation tensor comprising a plurality of input activations, wherein the input activation tensor represents an image, each of the plurality of input activations corresponding to a value at a location in the image; Receives a weight tensor containing multiple weights; Booth re-encoding each of the plurality of input activations into a plurality of Booth-encoded input activations, each Booth-encoded value having an order; multiplying the weight tensor by the Booth-encoded input activation to produce a plurality of results for each weight, each result in the plurality of results corresponding to the order of the Booth-encoded input activation; For each order of activation of the Booth-encoded input, summing the corresponding results to produce a plurality of partial sums, one partial sum for each order; A neural activation is calculated from the sum of the plurality of partial sums.

9. The method according to claim 8, wherein The input activation tensor has a dimension of one.

10. The method according to claim 8, wherein The weight tensor has a dimension of two.

11. The method according to claim 8, wherein Calculating the neural activation includes shifting each of the plurality of partial sums according to a corresponding order of each of the plurality of partial sums.

12. The method according to claim 8, wherein Computing the neural activation includes shifting each of the plurality of partial sums according to a precision of the input activation.

13. The method according to claim 8, wherein Computing the neural activation includes applying a non-linear activation function to the sum of the plurality of partial sums.

14. The method according to claim 8, wherein Summing the corresponding results includes applying a plurality of carry-save adders.

15. A neural inference chip for computing neural activations, the neural inference chip being suitable for: receiving an input activation tensor comprising a plurality of input activations, wherein the input activation tensor represents an image, each of the plurality of input activations corresponding to a value at a location in the image; Receives a weight tensor containing multiple weights; Booth recoding each of the plurality of weights into a plurality of Booth-encoded weights, each Booth-encoded value having an order; multiplying the input activation tensor by the Booth encoding weights to produce a plurality of results for each input activation, each result in the plurality of results corresponding to the order of the Booth encoding weights; For each order of the Booth encoding weights, summing the corresponding results to produce a plurality of partial sums, one partial sum for each order; A neural activation is calculated from the sum of the plurality of partial sums.

16. The neural inference chip according to claim 15, wherein: Calculating the neural activation includes shifting each of the plurality of partial sums according to a corresponding order of each of the plurality of partial sums.

17. The neural inference chip according to claim 15, wherein: Computing the neural activation includes shifting each of the plurality of partial sums according to a precision of the input activation.

18. The neural inference chip according to claim 15, wherein: Computing the neural activation includes applying a non-linear activation function to the sum of the plurality of partial sums.

19. The neural inference chip according to claim 15, wherein: Summing the corresponding results includes applying a plurality of carry-save adders.

20. A neural inference chip for computing neural activations, the neural inference chip being suitable for: receiving an input activation tensor comprising a plurality of input activations, wherein the input activation tensor represents an image, each of the plurality of input activations corresponding to a value at a location in the image; Receives a weight tensor containing multiple weights; Booth re-encoding each of the plurality of input activations into a plurality of Booth-encoded input activations, each Booth-encoded value having an order; multiplying the weight tensor by the Booth-encoded input activation to produce a plurality of results for each weight, each result in the plurality of results corresponding to the order of the Booth-encoded input activation; For each order of activation of the Booth-encoded input, summing the corresponding results to produce a plurality of partial sums, one partial sum for each order; A neural activation is calculated from the sum of the plurality of partial sums.

Images