Neural inference processing unit with flexible precision

Abstract

Neural inference chips are provided. A neural core of a neural inference chip includes a vector-matrix multiplier; a vector processor; and an activation unit, which is operatively coupled to the vector processor. The vector-matrix multiplier, vector processor, and / or activation unit are adapted to operate at variable precision.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to neural network processing units, and more specifically to flexible precision neural inference processing units. BACKGROUND

[0002] An artificial neuron is a mathematical function whose output is a nonlinear function of a linear combination of its inputs. Two neurons are connected if the output of one neuron is an input to the other neuron. A weight is a scalar value that encodes the strength of a connection between the output of one neuron and the input of another neuron.

[0003] A neuron computes its output by applying a nonlinear activation function to a weighted sum of its inputs, called an activation. The weighted sum is an intermediate result computed by multiplying each input by a corresponding weight and accumulating the products. A partial sum is a weighted sum of a subset of the inputs. The weighted sum of all inputs can be computed in stages by accumulating one or more partial sums.

[0004] A neural network is a collection of one or more neurons. Neural networks are often divided into groups of neurons, called layers. A layer is a collection of one or more neurons that all receive input from and all send output to the same layer, and that generally perform similar functions. An input layer is a layer that receives input from a source outside the neural network. An output layer is a layer that sends output to a target outside the neural network. All other layers are intermediate processing layers. A multilayer neural network is a neural network with more than one layer. A deep neural network is a multilayer neural network with multiple layers.

[0005] A tensor is a multidimensional array of numerical values. A tensor block is a contiguous subarray of elements in a tensor.

[0006] Each neural network layer is associated with a parameter tensor V, a weight tensor W, an input data tensor X, an output data tensor Y, and an intermediate data tensor Z. The parameter tensor contains all parameters that control the activation function σ of the neurons in the layer. The weight tensor contains all weights that connect inputs to the layer. The input data tensor contains all data that the layer consumes as input. The output data tensor contains all data that the layer computes as output. The intermediate data tensor contains any data that the layer produces as an intermediate computation, such as partial sums.

[0007] The data tensors of a layer (input, output and intermediate quantities) can be 3-dimensional, where the first two dimensions can be interpreted as encoding spatial positions, and the third dimension can be interpreted as encoding different features. For example, when the data tensors represent a color image, the first two dimensions encode the vertical and horizontal coordinates within the image, and the third dimension encodes the color at each position. Each element of the input data tensor X can be connected to each neuron by a separate weight, so the weight tensor W typically has 6 dimensions, concatenating the 3 dimensions of the input data tensor (input row a, input column b, input feature c) with the 3 dimensions of the output data tensor (output row i, output column j, output feature k). The intermediate data tensor Z has the same shape as the output data tensor Y. The parameter tensor V concatenates the 3 output data tensor dimensions with an additional dimension o indexing the parameters of the activation function σ.

[0008] The elements of the output data tensor Y of a layer can be computed as in Equation 1, where the neuron activation function σ is configured by a vector of activation function parameters V[i,j,k,:], and the weighted sum Z[i,j,k] can be computed as in Equation 2.

[0009] Y[i,j,k] = σ(V[i,j,k,:]; Z[i,j,k])

[0010] Equation 1

[0011]

[0012] Equation 2

[0013] For simplicity of notation, the weighted sum in Equation 2 can be referred to as the output, which is equivalent to using a linear activation function Y[i,j,k] = σ(Z[i,j,k]) = Z[i,j,k], with the understanding that the same statement applies without loss of generality when different activation functions are used. However, there is a need for flexible precision of neural inference. Therefore, there is a need in the art to address the aforementioned problems. SUMMARY

[0014] According to embodiments of the present disclosure, a neural inference chip is provided that includes a neural core. The neural core includes a vector-matrix multiplier adapted to receive a weight matrix having a weight matrix precision, receive an input activation vector having an input activation vector precision, and compute a partial sum vector by multiplying the input activation vector by the weight matrix, the partial sum vector having a partial sum vector precision. The neural core includes a vector processor adapted to receive one or more partial sum vectors from one or more vector sources, and perform one or more vector functions on the one or more partial sum vectors to produce a vector processor output vector, the one or more vector sources including the vector-matrix multiplier, the vector processor output vector having a precision equal to the partial sum vector precision. The neural core includes an activation unit operably coupled to the vector processor and adapted to apply an activation function to the vector processor output vector, thereby producing an output activation vector having an output activation precision. The vector-matrix multiplier, vector processor, and / or activation unit are adapted to operate at variable precision.

[0015] According to embodiments of the present disclosure, methods and computer program products for neural inference with flexible precision are provided. In various embodiments, a weight matrix having a first precision is received. An activation vector having a first precision is received. A vector-matrix multiplication of the weight matrix and the activation vector is computed, thereby producing a partial sum vector of a second precision. One or more vector functions are performed on the partial sum vector to produce a vector processor output vector having the second precision. An activation function is applied to the vector processor output vector, thereby producing an output activation vector having a third precision. At least one of the first precision, the second precision, and the third precision changes at runtime. BRIEF DESCRIPTION OF DRAWINGS

[0016] The application will now be described, by way of example only, with reference to the preferred embodiments illustrated in the following drawings:

[0017] Figure 1 A neural core according to embodiments of the present disclosure is shown;

[0018] Figure 2 An exemplary inference processing unit (IPU) according to embodiments of the present disclosure is shown;

[0019] Figure 3 A multi-core inference processing unit (IPU) according to embodiments of the present disclosure is shown;

[0020] Figure 4 A neural core and associated network according to embodiments of the present disclosure is shown;

[0021] Figure 5 A Boolean function according to embodiments of the present disclosure is depicted;

[0022] Figure 6 Depiction of a ternary function according to embodiments of the application;

[0023] Figure 7 Depiction of a bounded linear unit according to embodiments of the disclosure;

[0024] Figure 8 Illustration of a bounded ReLU according to embodiments of the disclosure;

[0025] Figure 9 Illustration of a bounded shifted ReLU according to embodiments of the disclosure;

[0026] Figure 10 Depiction of a bounded pReLU according to embodiments of the disclosure;

[0027] Figure 11 Illustration of a bounded exp ReLU according to embodiments of the disclosure;

[0028] Figure 12 Depiction of a sigmoid function according to embodiments of the disclosure;

[0029] Figure 13 Depiction of a hyperbolic tangent function according to embodiments of the disclosure;

[0030] Figure 14 Illustration of a flexible precision activation memory according to embodiments of the disclosure;

[0031] Figure 15 Illustration of a flexible precision activation memory according to embodiments of the disclosure;

[0032] Figure 16 Illustration of a flexible precision weight memory according to embodiments of the disclosure;

[0033] Figure 17 Illustration of a method of flexible precision neural processing according to embodiments of the disclosure; and

[0034] Figure 18 Depiction of a compute node according to embodiments of the disclosure. DETAILED DESCRIPTION

[0035] In various embodiments, the computation of the output data tensor as described above is broken down into smaller problems. Each problem can then be solved in parallel on one or more neural cores, or on one or more cores of a conventional multi-core system.

[0036] Reference is now made to Figure 1Figure 1 depicts a neural core according to embodiments of the present disclosure. Neural core 100 is a tileable compute unit that computes one block of an output tensor. Neural core 100 has M inputs and N outputs. In various embodiments M=N. To compute an output tensor block, the neural core multiplies an Mxl input tensor block 101 by an MxN weight tensor block 102 and accumulates the product as a weighted sum, which is stored in a l xN intermediate tensor block 103. The O xN parameter tensor block contains O parameters that specify each of N neuron activation functions that are applied to the intermediate tensor block 103 to produce a l xN output tensor block 105.

[0037] Multiple neural cores can be tiled in a neural core array. In some embodiments, the array is 2-dimensional.

[0038] A neural network model is a set of constants that collectively specify the entire computation performed by a neural network, including the graph of connections between neurons and the weights and activation function parameters for each neuron. Training is the process of modifying a neural network model to perform a desired function. Inference is the process of applying a neural network to an input to produce an output without modifying the neural network model.

[0039] An inference processing unit is a class of processors that perform neural network inference. A neural inference chip is a particular physical instance of an inference processing unit.

[0040] Reference is made to Figure 2 An exemplary inference processing unit (IPU) is shown according to embodiments of the present disclosure. IPU 200 includes a memory 201 for a neural network model. As described above, the neural network model can include synaptic weights of a neural network to be computed. IPU 200 includes an activation memory 202, which can be transient. Activation memory 202 can be partitioned into input and output regions, and stores neuron activations for processing. IPU 200 includes a neural compute unit 203, which is loaded with a neural network model from model memory 201. Input activations are provided from activation memory 202 prior to each compute step. Output from neural compute unit 203 is written back to activation memory 202 for processing on the same or another neural compute unit.

[0041] In various embodiments, a microengine 204 is included in IPU 200. In such embodiments, all operations in the IPU are directed by the microengine. As described below, a central and / or distributed microengine can be provided in various embodiments. A global microengine can be referred to as a chip microengine, while a local microengine can be referred to as a core microengine or local controller. In various embodiments, a microengine includes one or more microengines, microcontrollers, state machines, CPUs, or other controllers.

[0042] Referring to Figure 3 Figure 3 shows a multi-core inference processing unit (IPU) according to embodiments of the present disclosure. The IPU 300 includes a memory 301 for neural network models and instructions. In some embodiments, the memory 301 is divided into a weights portion 311 and an instructions portion 312. As described above, a neural network model can include synaptic weights of a neural network to be computed. The IPU 300 includes an activation memory 302, which can be transient. The activation memory 302 can be divided into input and output regions, and stores neuron activations for processing. The IPU 300 includes a plurality of cores 303. Each core 303 includes a neural compute unit 333 loaded with a neural network model from the model memory 301. Each core also includes a local activation memory 332. Input activations are provided from the local activation memory 332 prior to each compute step. Output from the neural compute unit 333 is written back to the activation memory 332 for processing on the same or another neural compute unit.

[0043] The IPU 300 includes an array 306 of neural cores 303. Each core 303 includes a compute unit 333 loaded with a neural network model from the model memory 301, and is operable to perform vector computations. Each core also includes a local activation memory 332. Input activations are provided from the local activation memory 332 prior to each compute step. Output from the compute unit 333 is written back to the activation memory 332 for processing on the same or another compute unit.

[0044] The IPU 300 includes one or more networks-on-chip (NoCs) 305. In some embodiments, a portion and NoC 351 interconnects the cores 303 and transports portions and between them. In some embodiments, a separate parameter distribution NoC 352 connects the cores 303 to the memory 301 for distribution of weights and instructions to the cores 303. It will be appreciated that various configurations of NoCs 351 and 352 are suitable for use according to the present disclosure. For example, broadcast networks, row broadcast networks, tree networks, and switch networks can be used.

[0045] In various embodiments, a global microengine 304 is included in the IPU 300. In various embodiments, a local core controller 334 is included on each core 303. In such embodiments, the direction of operations is shared between the global microengine (chip microengine) and the local core controller (core microengine). Specifically, at 311, compute instructions are loaded by the global microengine 304 from the model memory 301 to the neural compute units 333 on each core 303. At 312, parameters (e.g., neural network / synaptic weights) are loaded by the global microengine 304 from the model memory 301 to the neural compute units 333 on each core 303. At 313, neural network activation data is loaded by the local core controller 334 from the local activation memory 332 to the neural compute units 333 on each core 303. As noted above, the activation is provided to the axon of the particular neural network defined by the model, and can originate from the same or another neural compute unit, or from outside the system. At 314, the neural compute units 333 perform the computation to generate output neuron activations as directed by the local core controller 334. Specifically, the computation includes applying the input synaptic weights to the input activations. It will be appreciated that various methods can be used to perform such a computation, including in silico dendrites as well as vector multiplication units. At 315, the computation results are stored in the local activation memory 332 as directed by the local core controller 334. As noted above, these stages can be pipelined in order to provide efficient use of the neural compute units on each core. It will also be appreciated that, depending on the requirements of the given neural network, inputs and outputs can be transferred from the local activation memory 332 to the global activation memory 302.

[0046] The compute units 333 perform the computation to generate output neuron activations as directed by the local core controller 334. Specifically, the computation includes applying the input synaptic weights to the input activations. It will be appreciated that various methods can be used to perform such a computation, including in silico dendrites as well as vector multiplication units. The computation results are stored in the local activation memory 332 as directed by the local core controller 334. These stages can be pipelined in order to provide efficient use of the compute units on each core. It will also be appreciated that, depending on the requirements of the given neural network, inputs and outputs can be transferred from the local activation memory 332 to the global activation memory 302.

[0047] Accordingly, the present disclosure provides for runtime control of operations in an inference processing unit (IPU). In some embodiments, the microengine is centralized (single microengine). In some embodiments, the IPU computation is distributed (performed by an array of cores). In some embodiments, the runtime control of operations is hierarchical - both the central microengine and the distributed microengine participate.

[0048] Microengines or microengines direct the execution of all operations in the IPU. Each microengine instruction corresponds to several sub-operations (e.g., address generation, load, compute, store, etc.), in a distributed case, core microcode runs on core microengines (e.g., 334). Core microcode includes instructions for performing a complete single tensor operation. For example, a convolution between a weight tensor and a data tensor. In the context of a single core, core microcode includes instructions for performing a single tensor operation on a subset of the locally stored data tensor (and partial sum). Chip microcode runs on chip microengines (e.g., 304). Microcode includes instructions for performing all tensor operations in a neural network.

[0049] Reference is now made to Figure 4 which shows an exemplary neural core and associated network according to embodiments of the present disclosure. The core can be implemented in the manner described with reference to Figure 1 The core 401 is interconnected with additional cores by networks 402...404. In this embodiment, network 402 is responsible for distributing weights and / or instructions, network 403 is responsible for distributing partial sums, and network 404 is responsible for distributing activations. However, it will be appreciated that various embodiments of the present disclosure can combine these networks, or further separate them into multiple additional networks.

[0050] Input activations (X) are distributed from outside the core to the core 401 via activation network 404 to activation memory 405. Layer instructions are distributed from outside the core to the core 401 via weight / instruction network 402 to instruction memory 406. Layer weights (W) and / or parameters are distributed from outside the core to the core 401 via weight / instruction network 402 to weight memory 407 and / or parameter memory 408.

[0051] A weight matrix (W) is read by vector matrix multiplication (VMM) unit 409 from weight memory 407. An activation vector (V) is read by vector matrix multiplication (VMM) unit 409 from activation memory 405. Vector matrix multiplication (VMM) unit 409 then computes vector matrix multiplication Z = X T W and provides the result to vector unit 410. Vector unit 410 reads additional partial sums from partial sum memory 411 and receives additional partial sums from outside the core over partial sum network 403. Vector-vector unit 410 computes vector-vector operations from these source partial sums. For example, individual partial sums can be summed sequentially. The resulting target partial sum is written to partial sum memory 411, sent outside the core via partial sum network 403, and / or fed back for further processing by vector-vector unit 410.

[0052] After all computations for the input to a given layer are complete, the partial sum results from the vector-vector unit 410 are provided to the activation unit 412 for computation of the output activations. The activation vector (Y) is written to the activation memory 405. The layer activations, including the results written to the activation memory, are redistributed across the cores from the activation memory 405 via the activation network 404. Upon receipt, they are written to the local activation memory destined for each receiving core. Upon completion of processing of a given frame, the output activations are read from the activation memory 405 and sent outside the cores via the network 404.

[0053] Low-precision computations have certain advantages in terms of power consumption, performance, and area. In particular, less energy is required per operation. Higher operation frequencies are achievable (due to fewer logic levels). Smaller circuit implementation areas are required.

[0054] However, low-precision computations also have certain disadvantages. There is a potential loss of accuracy. For example, for a network trained at high precision, a loss can occur when inference is performed at low precision.

[0055] To address these conflicting goals, the present disclosure provides flexible precision in neural inference, combining the advantages of high-precision and low-precision computations. In various embodiments, high-precision computations are performed when / where needed, and low-precision computations are performed when / where low-precision can be used.

[0056] In various embodiments, a flexible precision computation unit is provided, including a flexible precision vector-matrix multiplier (VMM), a flexible prediction vector unit, and a flexible precision activation function unit. In various embodiments, flexible precision data transfer is provided, including activation, weight, and partial sum buses or networks of flexible precision. In various embodiments, flexible prediction storage is provided, including activation, weight, and partial sum memories. In various embodiments, conversion between precisions is provided, including reformatting of values from computation to storage.

[0057] In some example embodiments, the VMM converts from flexible precision (e.g., 8-bit / 4-bit / 2-bit) to high-precision, fixed-precision output (e.g., 32-bit). In such embodiments, the vector unit is high, fixed precision (e.g., 32-bit). The activation function unit converts from high-precision, fixed-precision (e.g., 32-bit) input to flexible precision (e.g., 8-bit / 4-bit / 2-bit) output. The activation function serves as a squashing function / re-ranging function.

[0058] In other example embodiments, the VMM converts from a flexible precision (e.g., 8-bit / 4-bit / 2-bit) to a higher, also flexible precision (e.g., 32-bit / 16-bit / 8-bit) internal representation. In such embodiments, the vector unit is a high flexible precision (e.g., 32-bit / 16-bit / 8-bit), the activation function unit is a high flexible precision (e.g., 32-bit / 16-bit / 8-bit) input to a low flexible precision output (e.g., 8-bit / 4-bit / 2-bit). The activation function serves as a squeeze function / re-ranging function.

[0059] In an example flexible precision VMM, the VMM unit performs the following operation: Z = X T W, supports 8-bit / 4-bit / 2-bit input precision (X, W), and always outputs 32-bit precision (Z). Referring to Table 1, the VMM input and output is shown. Referring to Table 2, the weight size is provided for each configuration.

[0060] Primitive Function Input 1 Input 2 Output Vector-matrix multiplication z j =∑ i x i w i，j ]]> 8-bit x i ]] 8-bit w i，j ]] 32-bit z j ]] Vector-matrix multiplication z j =∑ i x i w i，j ]]> 4-bit x i ]] 4-bit w i，j ]] 32-bit z j ]] Vector-matrix multiplication z j =∑ i x i w i，j ]]> 2-bit x i ]]> 2-bit w i，j ]] 32-bit z j ]]>

[0061] Table 1

[0062] Precision Dimension Period W size (bytes) W size INT8: 8b X, 8b W 32×32 1 8b per byte * 32 * 32 / 8b = 1024 bytes Full block INT4: 4b X, 4b W 64×32 1 4b per byte * 64 * 32 / 8b = 1024 bytes Half block INT8: 2b X, 2b W 128×32 1 2b per byte * 128 * 32 / 8b = 1024 bytes Quarter block

[0063] Table 2

[0064] In an example activation function unit, the input precision (Z) is 32-bit, and supports 8-bit / 4-bit / 2-bit output precision (Y). To provide flexible precision, the activation function computation is performed in two stages. In stage 1, the activation function is computed using 32-bit input precision and 8-bit output precision. In stage 2, if needed, the 8-bit output is rounded to the appropriate precision (keeping the most significant 4-bit or 2-bit of the 8-bit output). Referring to Table 3, the precision conversion in various configurations is illustrated.

[0065]

[0066]

[0067] Table 3

[0068] Quad precision (4x activation memory, index 3 above) selection is used for full precision readout of the partial sum value as the neuron output. This is used for functions such as linear regression. When quad output precision is selected, the activation function computation is ignored. Instead, the full 32-bit z j is assigned directly to the quad (32-bit) y j output. The quad precision readout mode requires 4 cycles to perform the readout, limiting to 4 cycles between back-to-back operations.

[0069] In various embodiments, the activation functions are designed by construction to be saturating functions to avoid overflow and underflow. The saturating functions use the full 32 bits of z j Floor / ceiling checks are performed. They saturate to:

[0070] INT8: 0 and 255 (unsigned)

[0071] INT8: -127 and 127 (signed)

[0072] INT4: 0 and 15 (unsigned)

[0073] INT4: -7 and 7 (signed)

[0074] INT2: 0 and 3 (unsigned)

[0075] INT2: -1 and 1 (signed)

[0076] The output precision conversion functions avoid overflow and underflow by converting after the saturating activation function. The saturating activation function uses the same 8 bits of u j range, then rounding the least significant bits to get the appropriate output precision.

[0077] Exemplary saturating functions include binary, ternary, sigmoid, tanh, bounded ReLU, bounded LU, and others known in the art.

[0078] Referring to Figures 5 to 13 , exemplary activation functions are illustrated in accordance with the present disclosure. These activation transfer function shapes perform re-ranging. They are well suited to multiple precisions. Figure 5 Boolean functions are described. Figure 6 Ternary functions are depicted. Figure 7 Bounded linear units are depicted. Figure 8 Bounded ReLUs are depicted. Figure 9 Bounded shifted ReLUs are depicted. Figure 10 Bounded pReLUs are depicted. Figure 11 Bounded exp ReLUs are depicted. Figure 12 An S-shaped function is depicted. Figure 13 A hyperbolic tangent function is depicted.

[0079] In various embodiments, flexible precision communication and storage is provided. The same network and memory support all precisions (e.g., 8-bit / 4-bit / 2-bit) using the same underlying physical substrate by changing the data format. In various embodiments, the total network bandwidth (e.g., expressed in number of lines on the bus) remains constant while the number of elements and precision varies. In particular, the total network usage is given by the number of elements times the precision. In this way, the precision and number of elements can be continuously adjusted while using the target amount of bandwidth (e.g., all available lines). Likewise, in various embodiments, the total memory utilization remains constant while the number of elements and precision varies.

[0080] In various embodiments, the weight NoC supports 8-bit / 4-bit / 2-bit precision. In some embodiments, a 512-line bus is provided, supporting 8-bit 64 elements, 4-bit 128 elements, and 2-bit 256 elements.

[0081] In various embodiments, the activation NoC supports 8-bit / 4-bit / 2-bit precision. In some embodiments, a 256-line bus is provided, supporting 32 elements with 8-bit, 64 elements with 4-bit, and 128 elements with 2-bit.

[0082] In various embodiments, the weight memory is organized as 4096 read data lines. In such embodiments, the 4096 lines support 32x16 elements with 8-bit, 64x16 elements with 4-bit, and 128x16 elements with 2-bit.

[0083] In various embodiments, the activation memory is organized as 256 read data lines. In such embodiments, the 256 lines support 32 elements with 8-bit, 64 elements with 4-bit, and 128 elements with 2-bit.

[0084] Referring now to Figure 14 , an activation memory is shown in accordance with an embodiment of the present disclosure. The activation memory 1401 supports 8-bit / 4-bit / 2-bit formats. As shown, an activation block with 2-bit precision 1402, an activation block with 4-bit precision 1403, and an activation block with 8-bit precision are supported.

[0085] Referring now to Figure 15 , an activation memory is shown in accordance with an embodiment of the present disclosure, in this example, the activation memory 1501 uses a quad-precision (32-bit) format. In this example, the activation block spans multiple bank word addresses.

[0086] Referring now to Figure 16FIG. 17 illustrates a weight memory according to embodiments of the present disclosure. As shown, a 2-bit format 1601, a 4-bit format 1602, and an 8-bit format 1603 are supported.

[0087] In various embodiments, the activation function unit output is 32 elements x 8 bits, 4 bits, or 2 bits. However, the activation word size is 32 elements in 8 bits, 64 elements in 4 bits, or 128 elements in 2 bits. For low-precision computation, the output words of multiple activation function units are combined as follows: 1 x 32 elements in 8 bits, 2 x 64 elements in 4 bits, or 4 x 128 elements in 2 bits.

[0088] To accommodate this reformatting, in various embodiments, the 4-bit sub-vectors [127:0] or 2-bit [63:0] are stored in registers before the full activation vector [255:0] is written back to memory. In 8 bits, 1 cycle can hold the full activation vector. In 4 bits, 2 cycles hold the full activation vector, with cycle 1 covering bits 127:0 and cycle 2 covering bits 255:128. In 2 bits, 4 cycles hold the full activation vector, with cycle 1 covering bits 63:0, cycle 2 covering bits 127:64, cycle 3 covering bits 191:128, and cycle 4 covering bits 255:192.

[0089] In various embodiments, the partial full vector is zero-padded, with zeros going into the unassigned sub-vector elements. The full vector can be written back to memory even if not all sub-vectors are assigned. The unused sub-vector slots are zeroed. This reduces the computation time for full-dimension, 4-bit 2-bit precision networks that do not need the full activation vector. For example, in 2-bit mode, if the output activation vector has only 64 elements, only two computation cycles are needed.

[0090] Reference Figure 17 FIG. 17 illustrates a method of flexible precision neural processing according to embodiments of the present disclosure. At 1701, a weight matrix having a first precision is received. At 1702, an activation vector having the first precision is received. At 1703, a vector-matrix multiplication of the weight matrix and the activation vector is computed, producing a partial sum vector having a second precision. At 1704, one or more vector functions are performed on the partial sum vector to produce a vector processor output vector having the second precision. At 1705, an activation function is applied to the vector processor output vector, producing an output activation vector having a third precision. At least one of the first, second, and third precisions varies at runtime.

[0091] Reference is now made to Figure 18FIG. 1 shows a schematic diagram illustrating an example of a computing node. The computing node 10 is but one example of a suitable computing node and is not intended to suggest any limitation as to the scope of use or functionality of embodiments described herein. Regardless, the computing node 10 is capable of being implemented and / or performing any of the functionality set forth above.

[0092] Within the computing node 10, there is a computer system / server 12, which is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well- known computing systems, environments, and / or configurations that can be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.

[0093] Computer system / server 12 can be described in the general context of computer system executable instructions, such as program modules being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. Computer system / server 12 can be practiced in distributed cloud computing environments with remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in both local and remote computer system storage media including memory storage devices.

[0094] As shown in FIG. 1, computer system / server 12, in the example of FIG. 1, is shown Figure 18 in the form of a general-purpose computing device. The components of computer system / server 12 can include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including system memory 28 to processor 16.

[0095] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, PCI Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA) O

[0096] The computer system / server 12 generally includes a variety of computer system readable media. Such media can be any available media that is accessible by the computer system / server 12, and it includes both volatile and non-volatile media, removable and non-removable media.

[0097] The system memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or buffer memory 32. Computer system / server 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 can be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Although not specifically shown, a magnetic disk drive can also be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive can be provided for reading from or writing to a removable, non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical media). In such instances, each can be connected to bus 18 by one or more data media interfaces. As will be further depicted and described below, memory 28 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application.

[0098] Program / utility 40 having a set (at least one) of program modules 42, and being implemented by operating system, one or more application programs, other program modules, and program data, can be stored in memory 28 by way of example, and not limitation, as well as an implementation of the networking environment. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, can include an implementation of a networking environment. Program modules 42 generally carry out the functions and / or methodologies of embodiments described herein.

[0099] The computer system / server 12 can also communicate with one or more external devices 14 such as a keyboard, a pointing device, a display 24, etc.; one or more devices that enable a user to interact with the computer system / server 12; and / or any devices (e.g., network card, modem, etc.) that enable the computer system / server 12 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interfaces 22. Still yet, the computer system / server 12 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via network adapter 20. As depicted, network adapter 20 communicates with the other components of the computer system / server 12 via bus 18. It should be appreciated that although not shown, other hardware and / or software components could be used in conjunction with the computer system / server 12. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0100] The present disclosure can be implemented as a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0101] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0102] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adaptation card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions to storage media within the respective computing / processing device for execution.

[0103] Computer readable program instructions for carrying out operations of the present disclosure can be in assembly code, instruction set architecture (ISA) instructions, machine instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0104] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0105] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including

[0106] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0107] The flow and block diagrams in the drawings show the architectural, functional, and operational views of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow and block diagrams can represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0108] The description of various embodiments of the disclosure has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A neural inference chip comprising a neural core, the neural core comprising: a vector-matrix multiplier adapted to receive a weight matrix having a first precision, receive an input activation vector having the first precision, and compute a partial sum vector by multiplying the input activation vector by the weight matrix, the partial sum vector having a second precision; a vector processor adapted to receive one or more partial sum vectors from one or more vector sources, including the vector-matrix multiplier, and perform one or more vector functions on the one or more partial sum vectors to produce a vector processor output vector, the vector processor output vector having the second precision; and an activation unit, operatively coupled to the vector processor, and adapted to apply an activation function to the vector processor output vector, thereby producing an output activation vector having a third precision, wherein the vector-matrix multiplier, vector processor, and / or activation unit are adapted to operate at variable precision; wherein at least one of the first precision, the second precision, and the third precision varies at runtime.

2. The neural inference chip of claim 1, further comprising: at least one network interconnecting the neural core with at least one additional neural core, the at least one network adapted to deliver synaptic weights and / or input activations to the neural core at variable precision. the at least one network further adapted to vary the precision and dimensionality of the weight matrix, the precision and dimensionality of the input activation vector, and / or the precision and dimensionality of the output activation vector, while maintaining constant bandwidth.

3. The neural inference chip of claim 2, wherein, the neural core further comprising:

4. The neural inference chip of any one of claims 1-3, wherein, at least one memory adapted to store weight matrices, input activation vectors, and / or output activation vectors at variable precision. the at least one memory further adapted to vary the precision and dimensionality of the weight matrix, the precision and dimensionality of the input activation vector, and / or the precision and dimensionality of the output activation vector, while maintaining constant storage utilization.

5. The neural inference chip of claim 4, wherein, 6. The neural inference chip of any one of claims 1-3, wherein the vector-matrix multiplier is further adapted to vary the precision and dimensionality of the weight matrix and / or the precision and dimensionality of the input activation vector, while maintaining constant bandwidth. the vector-matrix multiplier is further adapted to compute a variable number of multiplications per cycle at variable precision, wherein the variable number of multiplications per cycle and variable precision are inversely proportional.

7. The neural inference chip of claim 6, wherein, the activation function is adapted to re-range the vector processor output vector.

8. The neural inference chip of any one of claims 1-3, wherein, applying the activation function comprises applying a saturation function.

9. The neural inference chip of claim 8, wherein, the saturation function has at least one boundary corresponding to the output activation precision.

10. The neural inference chip of claim 9, wherein, applying the activation function comprises truncating one or more least significant bits.

11. The neural inference chip of claim 8, wherein, the variable precision is selected from 2 bits, 4 bits, 8 bits, 16 bits, and 32 bits.

12. The neural inference chip of any one of claims 1-3, wherein, the variable precision is selectable at runtime.

13. The neural inference chip of any one of claims 1-3, wherein, the variable precision is selectable for each layer of a neural network.

14. The neural inference chip of any one of claims 1-3, wherein, the weight matrix precision is equal to the activation vector precision.

15. The neural inference chip of any one of claims 1-3, wherein, the partial sum vector precision is not equal to the output activation precision.

16. The neural inference chip of claim 15, wherein, ​ 17. The neural inference chip of any one of claims 1-3, wherein, The partial sum and vector precision is higher than the weight matrix precision and / or the activation vector precision.

18. The neural inference chip of claim 15, wherein, The output activation precision is equal to the weight matrix precision.

19. A method for flexible precision neural inference, comprising: receiving a weight matrix having a first precision; receiving an activation vector having the first precision; computing a vector-matrix multiplication of the weight matrix and the activation vector, resulting in a partial sum vector having a second precision; performing one or more vector functions on the partial sum vector to produce a vector processor output vector having the second precision; and applying an activation function to the vector processor output vector, resulting in an output activation vector having a third precision, wherein at least one of the first precision, the second precision, and the third precision varies at runtime.

20. The method of claim 19, further comprising: varying at least one of the first precision, the second precision, and the third precision for computations of each layer of a neural network.

21. A method of operating a neural inference chip, the neural inference chip comprising a neural core, the method comprising: receiving, by a vector-matrix multiplier of the neural inference chip, a weight matrix having a first precision, receiving an input activation vector having the first precision, and computing, by a vector processor of the neural inference chip, a partial sum vector by multiplying the input activation vector by the weight matrix, the partial sum vector having a second precision; receiving one or more partial sum vectors from one or more vector sources, including the vector-matrix multiplier, and performing one or more vector functions on the one or more partial sum vectors to produce a vector processor output vector, the vector processor output vector having the second precision; and applying, by an activation unit of the neural inference chip, an activation function to the vector processor output vector, resulting in an output activation vector having a third precision, wherein the vector-matrix multiplier, vector processor, and / or activation unit are adapted to operate at variable precision; wherein at least one of the first precision, the second precision, and the third precision varies at runtime.

Images