3D neural inference processing unit architecture

Abstract

Three-dimensional neural inference processing units are provided. A first tier includes a plurality of neural cores. Each core includes a neural compute unit. The neural compute unit is adapted to apply a plurality of synaptic weights to a plurality of input activations to produce a plurality of output activations. A second tier includes a first neural network model memory adapted to store the plurality of synaptic weights. A communication network is operatively coupled to the first neural network model memory and each of the plurality of neural cores and is adapted to provide the synaptic weights from the first neural network model memory to each of the plurality of neural cores.

Description

BACKGROUND

[0001] Embodiments of the present disclosure relate to neural network processing, and more specifically to three-dimensional neural inference processing unit architectures. SUMMARY

[0002] According to the present disclosure, there is provided a neural inference chip as claimed in claim 1.

[0003] There is also provided a method and computer program product for neural inference processing as claimed. BRIEF DESCRIPTION OF DRAWINGS

[0004] Figure 1 A neural core is shown in accordance with embodiments of the present disclosure.

[0005] Figure 2 An exemplary inference processing unit (IPU) is shown in accordance with embodiments of the present disclosure.

[0006] Figure 3 A multi-core inference processing unit (IPU) is shown in accordance with embodiments of the present disclosure.

[0007] Figure 4 A single layer layout of a multi-core inference processing unit (IPU) is shown in accordance with embodiments of the present disclosure.

[0008] Figure 5 A two layer layout of a multi-core inference processing unit (IPU) is shown in accordance with embodiments of the present disclosure.

[0009] Figure 6 A multi-layer layout of a multi-core inference processing unit (IPU) is shown in accordance with embodiments of the present disclosure.

[0010] Figure 7 A multi-layer layout of a multi-core inference processing unit (IPU) is shown in accordance with embodiments of the present disclosure.

[0011] Figure 8 A multi-layer layout of a multi-core inference processing unit (IPU) is shown in accordance with embodiments of the present disclosure.

[0012] Figures 9A to 9G A three-dimensional global memory interconnect is shown in accordance with embodiments of the present disclosure.

[0013] Figure 10 A compute node is depicted in accordance with embodiments of the present disclosure. DETAILED DESCRIPTION

[0014] An artificial neuron is a mathematical function whose output is a nonlinear function of a linear combination of its inputs. Two neurons are connected if the output of one neuron is an input to the other. A weight is a scalar value that encodes the strength of a connection between the output of a neuron and the input of another neuron.

[0015] A neuron computes its output by applying a nonlinear activation function to a weighted sum of its inputs, called an activation. The weighted sum is an intermediate result computed by multiplying each input by a corresponding weight and accumulating the products. A partial sum is a weighted sum of a subset of the inputs. The weighted sum of all inputs can be computed in stages by accumulating one or more partial sums.

[0016] A neural network is a collection of one or more neurons. Neural networks are often divided into groups of neurons called layers. A layer is a collection of one or more neurons that all receive input from and all send output to the same layer and that usually perform similar functions. An input layer is a layer that receives input from a source outside the neural network. An output layer is a layer that sends output to a target outside the neural network. All other layers are intermediate processing layers. A multilayer neural network is a neural network with more than one layer. A deep neural network is a multilayer neural network with many layers.

[0017] A tensor is a multidimensional array of numerical values. A tensor block is a contiguous subarray of elements in a tensor.

[0018] Each neural network layer is associated with a parameter tensor V, a weight tensor W, an input data tensor X, an output data tensor Y, and an intermediate data tensor Z. The parameter tensor contains all the parameters that control the activation function σ of the neurons in the layer. The weight tensor contains all the weights that connect inputs to the layer. The input data tensor contains all the data that the layer consumes as input. The output data tensor contains all the data that the layer computes as output. The intermediate data tensor contains any data that the layer produces as an intermediate computation, such as partial sums.

[0019] The data tensors of a layer (input, output, and intermediate) can be 3-dimensional, where the first two dimensions can be interpreted as encoding spatial positions, and the third dimension can be interpreted as encoding different features. For example, when the data tensors represent a color image, the first two dimensions encode the vertical and horizontal coordinates within the image, and the third dimension encodes the color at each position. Each element of the input data tensor X can be connected to each neuron by a separate weight, so the weight tensor W typically has 6 dimensions, concatenating the 3 dimensions of the input data tensor (input row a, input column b, input feature c) with the 3 dimensions of the output data tensor (output row i, output column j, output feature k). The intermediate data tensor Z has the same shape as the output data tensor Y. The parameter tensor V concatenates the 3 output data tensor dimensions with an additional dimension o indexing the parameters of the activation function σ.

[0020] The elements of the output data tensor Y of a layer can be computed as in Equation 1, where the neuron activation function σ is configured by a vector V[i,j,k, :] of activation function parameters, and the weighted sum Z[i,j,k] can be computed as in Equation 2.

[0021] Equation 1

[0022] Y[i,j,k] = σ(V[i,j,k, :]; Z[i,j,k])

[0023] Equation 2

[0024]

[0025] For simplicity of notation, the weighted sum in Equation 2 can be referred to as the output, which is equivalent to using a linear activation function Y[i,j,k] = σ(Z[i,j,k]) = Z[i,j,k], with the understanding that the same statement applies without loss of generality when a different activation function is used.

[0026] In various embodiments, the computation of the output data tensor as described above is decomposed into smaller problems. Each problem can then be solved in parallel on one or more neural cores or on one or more cores of a regular multi-core system.

[0027] Reference is now made to Figure 1Figure 1 depicts a neural core according to embodiments of the present disclosure. Neural core 100 is a tileable compute unit that computes one block of an output tensor. Neural core 100 has M inputs and N outputs. In different embodiments, M=N. To compute an output tensor block, the neural core multiplies an Mxl input tensor block 101 by an MxN weight tensor block 102 and accumulates the product into a weighted sum stored in a l xN intermediate tensor block 103. An O xN parameter tensor block contains O parameters that specify each of N neuron activation functions applied to the intermediate tensor block 103 to produce a l xN output tensor block 105.

[0028] Multiple neural cores can be tiled in a neural core array. In some embodiments, the array is two-dimensional.

[0029] A neural network model is a set of constants that collectively specify the entire computation performed by a neural network, including the graph of connections between neurons and the weights and activation function parameters of each neuron. Training is the process of modifying a neural network model to perform a desired function. Inference is the process of applying a neural network to an input to produce an output without modifying the neural network model.

[0030] An inference processing unit is a class of processors that perform neural network inference. A neural inference chip is a particular physical instance of an inference processing unit.

[0031] Reference Figure 2 An exemplary inference processing unit (IPU) is shown according to embodiments of the present disclosure. IPU 200 includes a memory 201 for a neural network model. As described above, the neural network model can include synaptic weights of a neural network to be computed. IPU 200 includes an activation memory 202, which can be transient. Activation memory 202 can be partitioned into input and output regions and stores neuron activations for processing. IPU 200 includes a neural compute unit 203 loaded with a neural network model from model memory 201. Input activations are provided from activation memory 202 before each compute step. Output from neural compute unit 203 is written back to activation memory 202 for processing on the same or another neural compute unit.

[0032] In different embodiments, microengines 204 are included in IPU 200. In such embodiments, all operations in the IPU are directed by microengines. As described below, in various embodiments, central and / or distributed microengines can be provided. A global microengine can be referred to as a chip microengine, while a local microengine can be referred to as a core microengine or local controller. In various embodiments, microengines include one or more microengines, microcontrollers, state machines, CPUs, or other controllers.

[0033] Referring to Figure 3 An Inference Processing Unit (IPU) according to embodiments of the present disclosure is shown. The IPU 300 includes a memory 301 for neural network models and instructions. In some embodiments, the memory 301 is divided into a weights portion 311 and an instructions portion 312. As described above, the neural network models can include synaptic weights of a neural network to be computed. The IPU 300 includes an activation memory 302, which can be transient. The activation memory 302 can be divided into input and output regions and stores neuron activations for processing. The IPU 300 includes a plurality of cores 303. Each core 303 includes a neural compute unit 333 loaded with a neural network model from the model memory 301. Each core also includes a local activation memory 332. Input activations are provided from the local activation memory 332 prior to each compute step. Output from the neural compute unit 333 is written back to the activation memory 332 for processing on the same or another neural compute unit.

[0034] The IPU 300 includes an array 306 of neural cores 303. Each core 303 includes a compute unit 333 loaded with a neural network model from the model memory 301 and operable to perform vector computations. Each core also includes a local activation memory 332. Input activations are provided from the local activation memory 332 prior to each compute step. Output from the compute unit 333 is written back to the activation memory 332 for processing on the same or another compute unit.

[0035] The IPU 300 includes one or more Network-on-Chip (NoC) 305. In some embodiments, a portion and NoC 351 interconnects and transports portions and between the cores 303. In some embodiments, a separate parameter distribution NoC 352 connects the cores 303 to the memory 301 for distribution of weights and instructions to the cores 303. It will be appreciated that various configurations of NoCs 351 and 352 are suitable for use in accordance with the present disclosure. For example, broadcast networks, row broadcast networks, tree networks, and switch networks can be used.

[0036] In different embodiments, a global microengine 304 is included in the IPU 300. In different embodiments, a local core controller 334 is included on each core 303. In such embodiments, the indication of the operation is shared between the global microengine (chip microengine) and the local core controller (core microengine). Specifically, at 311, compute instructions are loaded by the global microengine 304 from the model memory 301 to the neural compute units 333 on each core 303. At 312, parameters (e.g., neural network / synaptic weights) are loaded by the global microengine 304 from the model memory 301 to the neural compute units 333 on each core 303. At 313, neural network activation data is loaded by the local core controller 334 from the activation local activation memory 332 to the neural compute units 333 on each core 303. As noted above, the activation is provided to an axon of the particular neural network defined by the model, and can originate from the same or another neural compute unit, or from outside the system. At 314, the neural compute units 333 perform a computation to generate output neuron activations as indicated by the local core controller 334. Specifically, the computation includes applying input synaptic weights to input activations. It will be understood that different methods can be used to perform such a computation, including dendron and vector multiplication units on a computer. At 315, the results of the computation are stored in the local activation memory 332 as indicated by the local core controller 334. As noted above, these stages can be pipelined in order to provide efficient use of the neural compute units on each core. It will also be appreciated that inputs and outputs can be transferred from the local activation memory 332 to the global activation memory 302 as required by the given neural network.

[0037] The compute units 333 perform a computation to generate output neuron activations as indicated by the local core controller 334. Specifically, the computation includes applying input synaptic weights to input activations. It will be understood that different methods can be used to perform such a computation, including dendron and vector multiplication units on a computer. The results of the computation are stored in the local activation memory 332 as indicated by the local core controller 334. These stages can be pipelined in order to provide efficient use of the compute units on each core. It will also be appreciated that inputs and outputs can be transferred from the local activation memory 332 to the global activation memory 302 as required by the given neural network.

[0038] Accordingly, the present disclosure provides for runtime control of operations in an inference processing unit (IPU). In some embodiments, the microengine is centralized (single microengine). In some embodiments, the IPU computation is distributed (performed by an array of cores). In some embodiments, the runtime control of operations is hierarchical - both the central microengine and the distributed microengine participate.

[0039] The microengine(s) direct the execution of all operations in the IPU. Each microengine instruction corresponds to several sub-operations (e.g., address generation, load, compute, store, etc.). In the distributed case, the core microcode runs on the core microengines (e.g., 334). The core microcode includes instructions for performing a complete single tensor operation. For example, a convolution between a weight tensor and a data tensor. In the context of a single core, the core microcode includes instructions for performing a single tensor operation on a locally stored subset of the data tensor (and partial sum). The chip microcode runs on the chip microengines (e.g., 404). The microcode includes instructions for performing all tensor operations in a neural network.

[0040] Referring now to Figure 4 , an exemplary single-tier layout of a multi-core inference processing unit (IPU) is shown in accordance with embodiments of the present disclosure. As in Figure 3 , the IPU 400 includes a memory 401 for neural network models and instructions. As described above, the neural network models can include synaptic weights of a neural network to be computed. The IPU 400 includes an activation memory 402, which can be transient. The activation memory 402 can be partitioned into an input region and an output region, and stores neuron activations for processing.

[0041] The IPU 400 includes an array 406 of neural cores 403. Each core 403 includes a compute unit loaded with a neural network model from the model memory 401 and operable to perform vector computations. In different embodiments, each core also includes a local activation memory. In such embodiments, input activations are provided from the local activation memory prior to each compute step. The output of the compute unit is written back to the activation memory for processing on the same or another compute unit. In different embodiments, each core 403 includes a local model memory, an instruction memory, a controller, and / or an activation memory.

[0042] The IPU 400 includes one or more networks-on-chip (NoCs). In some embodiments, a partial sum NoC 451 interconnects the cores 403 and transports partial sums between them. In some embodiments, a separate parameter distribution NoC 452 connects the cores 403 to the memory 401 for distribution of weights and instructions to the cores 403. In some embodiments, an activation NoC 453 connects the cores to the activation memory 402. It will be appreciated that various configurations of the NoCs 451, 452, and 453 are suitable for use in accordance with the present disclosure. For example, broadcast networks, row broadcast networks, tree networks, and switch networks can be used.

[0043] In various embodiments, a global microengine 404 is included in the IPU 400. In various embodiments, a local core controller is included on each core 403. In such embodiments, operational instructions are shared between the global microengine (chip microengine) and the local core controller (core microengine). Specifically, computational instructions are loaded from model memory 401 by the global microengine 404 into the neural computation units on each core 403. Parameters (e.g., neural network / synaptic weights) are loaded from model memory 401 by the global microengine 404 into the neural computation units on each core 403. Neural network activation data is loaded from activation local activation memory by the local core controller into the neural computation units on each core 403. As described above, activation is provided to the axons of a specific neural network defined by the model and may originate from the same or another neural computation unit, or from outside the system.

[0044] The neural computation unit performs computations to generate output neuron activations as directed by the local kernel controller. Specifically, this computation involves applying input synaptic weights to input activations. It will be understood that different methods can be used to perform such computations, including dendritic and vector multiplication units on a computer. The results of the computation are stored in a local activation memory as directed by the local kernel controller. As described above, these stages can be pipelined to provide efficient use of the neural computation unit on each kernel. It will also be appreciated that inputs and outputs can be transferred from the local activation memory to the global activation memory 402 according to the requirements of a given neural network.

[0045] exist Figure 4 In this embodiment, the components of IPU 400 are laid out on a single layer. In an exemplary embodiment, the layout is approximately distributed as follows: 5% of the total chip area is dedicated to activation I / O, 60% to the core array, 25% to global memory, 5% to global memory interconnect (IC), and 5% to the controller. Within a given core, approximately 25% is dedicated to activation memory, 25% to weighted memory, 25% to computation, and 25% to control.

[0046] This architecture imposes various performance constraints. For example, memory capacity limits network size. Specifically, the size of the global weight memory determines the maximum size of the neural network model that can be computed. The size of the activation memory determines the maximum number of network layers that can be computed. Similarly, line bandwidth limits network distribution (weight distribution) and determines maximum computational parallelism. Data travel distance limits energy efficiency.

[0047] To address these limitations imposed by the physical layout of a given chip, in various embodiments, a 3D layout is employed in which different chip components are spread across multiple layers. In different embodiments, this is achieved through silicon 3D dies and wafer stacking. In some embodiments, inter-layer connections are provided by through-silicon vias.

[0048] A through-silicon via (TSV) or through-chip via is a vertical electrical connection (via) that goes all the way through a silicon wafer or die. TSVs are a high-performance interconnect technology used as an alternative to wire bonding and flip-chip to create 3D packages and 3D integrated circuits. The interconnect and device density is significantly higher compared to alternatives such as package-on-package, and the length of the connections becomes shorter.

[0049] In various embodiments, TSVs have an interconnect pitch of 6-10 pm, achieving a high interconnect density of about 1-3 million interconnects / cm 2 Given an area of 500 pm by 500 pm (250,000 pm 2 ), a 10 pm pitch yields 2500 interconnects (250,000 pm 2 / (10 pm x 10 pm)) and a 6 pm pitch yields 6944 interconnects (250,000 pm 2 / (6 pm x 6 pm)). Alternative implementations can have a 1.6-3.0 pm pitch with a 6-10 um depth.

[0050] TSVs can be implemented in copper, for example, based on a 14 nm process. Such an exemplary embodiment can have a diameter of about 10 pm and a pitch of about 40 pm. TSVs can be implemented in tungsten, for example. Such an exemplary embodiment can have a diameter of about 1.2 pm and a pitch of about 2.5 pm. Another such exemplary embodiment can have a diameter of about 1.0 pm and a pitch of 2.0 pm.

[0051] In various embodiments, the 3D layout is coupled with a 2D global weight distribution bus. In this way, the unique bandwidth to the core (and potentially the total bandwidth to the core) is increased.

[0052] As set forth below, the 3D layout is scalable across two or more layers, achieving increased capacity and minimization of inter-layer routing length.

[0053] Figure 5 An exemplary two-layer layout is shown. In this layout, a first layer 501 includes an array of cores and an activation I / O component. A second layer 502 includes global memory, global memory interconnect, and a controller. A parameter distribution NoC 452 is configured as a mesh network connecting each core for parameter distribution.

[0054] Figure 6An exemplary n-tier layout is shown. In this layout, the first tier 601 includes the core array and the active I / O components. The second tier 602 includes the global memory interconnect. Additional tiers 603...604 include the global memory. It will be appreciated that the global memory can be spread among any number of tiers, scaling capacity without changing the total physical footprint of each tier.

[0055] Figure 7 An exemplary n-tier layout is shown. In this layout, the first tier 701 includes the active I / O components, including the active memory and network 453. The second tier 702 includes the core array. Additional tiers (not shown) include the global memory, global memory interconnect, and controllers (e.g., as shown in Figure 5-6

[0056] Figure 8 An exemplary n-tier layout is shown. In this layout, the cores are divided among multiple tiers. Tier 801 includes the active memory for each core. Tier 802 includes the compute, weight memory, and controllers for each core. Additional tiers (not shown) include the active I / O components (such as shown in Figure 7 Figure 5-6

[0057] In view of the above examples, it will be appreciated that additional multi-tier layouts can be employed in different embodiments. For example, the memory array can be sandwiched between two core arrays. In such an embodiment, the model is in the middle and it distributes weights to the core array tiers above and below. This embodiment is particularly useful in the case of a batch size of 2. Specifically, for a given model, a first image is processed on the top core array tier and a second image is processed on the bottom core array tier. This embodiment is also useful for redundant computation in high reliability settings. A single image can be processed simultaneously on the top and bottom tiers. The results from the two tiers can be compared. If they match, the neural inference computation is successful. If not, the computation can be repeated, or the faulty system can be diagnosed.

[0058] The neural network layers can be split into two strata. For example, neural network layers 1-5 can be processed on a bottom stratum and neural network layers 6-10 are processed on a top stratum. Similarly, different neural networks can be run simultaneously on the same (or different) images on different strata.

[0059] ​​​The above multi-layer approach results in bandwidth improvements relative to a single-layer fabric. Specifically, the wire area increases from ID to 2D. The wire pitch decreases from a minimum wire separation to a TSV separation. The core array weight distribution bus goes from shared to individual. The bus frequency increases due to the short TSV propagation distance (corresponding to the depth). As a result, the activation memory bandwidth and the weight distribution bandwidth increase.

[0060] Referring now to Figure 9A-9G , a 3D global memory interconnect is shown in accordance with embodiments of the present disclosure. A 3D mesh communication network 901 is provided that allows dimensionally ordered routing (see Figure 9A ). Specifically, messages are routed in a first dimension and a second dimension (e.g., X and Y) within a layer, and then in a third dimension (e.g., Z) between layers. Upon receiving a message, a layer destination node (e.g., 911) sends along the third dimension to a core array on another layer (e.g., immediately below). For example, the communication network 452 can be implemented in this way, as shown in Figure 5-6 . It will be appreciated that other NoCs described herein can also be implemented in this way, such as the communication network 453 shown in Figure 7 .

[0061] Referring to Figure 9B , individual bank distribution is shown on the communication network 901. In this example, each node in the communication network 901 sends a personalized weight block along the third dimension from one memory bank to one core.

[0062] Referring to Figure 9C , row broadcast is shown on the communication network 901. In this example, the weight block is distributed from a row memory bank along the row to each node, and then routed along the third dimension to the cores. In this way, each core in a given row receives the same data.

[0063] Referring to Figure 9D , column broadcast is shown on the communication network 901. In this example, the weight block is distributed from a column memory bank along the column to each node, and then routed along the third dimension to the cores. In this way, each core in a given column receives the same data.

[0064] Referring to Figure 9E , single bank broadcast is shown on the communication network 901. In this example, the weight block is distributed from one single memory bank to each other node in the array, and then routed along the third dimension to the cores. In this way, each core receives the same data.

[0065] Referring to Figure 9FA single team broadcast is shown on the communication network 901. In this example, a weight block is distributed from a single internal memory team to every other node in the array, and then routed along the third dimension to the cores. In this way, each core receives the same data.

[0066] Referring to Figure 9G A multiple team broadcast is shown on the communication network 901. In this example, a weight block is distributed from node 971, while a different weight block is distributed from node 972. The respective blocks are routed along the rows and columns to respective subsets of the cores. In this way, concurrent multicasting is achieved.

[0067] In various embodiments, each layer of silicon has a wire stack. The wire stack and wire pitch define the maximum number of wires that can be fabricated within a layer. This, together with the frequency, defines the maximum bandwidth that can be achieved within a layer. By using multiple layers, both the effective number of wires and the effective bandwidth are increased, in direct proportion to the number of layers.

[0068] In various embodiments as described above, the second layer is used for the global memory communication network. This provides more routing capacity for the global memory communication network in the second layer. In addition, this frees up more routing capacity for other communication networks in the first layer (e.g., the partial and active NoCs).

[0069] Various embodiments provide scalable capacity. Specifically, each added layer of memory increases the capacity. This in addition allows for modular manufacturing using the same basic logic layers, e.g., allowing for multiple product lines with different memory capacities.

[0070] In various embodiments, DRAM, FLASH, SRAM, or other memory technology is integrated with standard logic processing layers on separate layers.

[0071] In various embodiments, the total energy reduction can be calculated using Equation 3, where N wires is the number of wires, C is the total wire capacitance, V dd is the supply voltage, b is the average fraction of bits in a packet that are toggled (0→1 or 1→0), U wires is the wire utilization, and f is the clock frequency. In an exemplary embodiment, b = 0.5 and f = 10 3 Hz. Capacitance is proportional to distance, and thus moving from an average distance of 4 mm to 10-100 pm results in a 40-400x weight distribution NoC energy consumption reduction.

[0072] Equation 3

[0073]

[0074] As described herein, in various embodiments, the IPU is partitioned across multiple layers. Specifically, a neural inference chip is provided in which the core array and memory are implemented on different silicon layers. In various embodiments, memory capacity can be scaled by adding (or subtracting) memory layers. In various embodiments, weight or parameter memory is provided on a dedicated memory layer. In various embodiments, activation memory is provided on a dedicated memory layer.

[0075] In various embodiments, instead of a shared row bus, a dedicated memory bus is provided to each core. In various embodiments, weights are distributed to the core array via a dedicated bus to each core. In various embodiments, the dedicated bus is routed to each core via different silicon layers.

[0076] In various embodiments, a global memory interconnect is provided. Such a system provides configurable data broadcasting between a 2D array of memory clusters and a 2D array of computing cores on different silicon layers. In various embodiments, data is routed to the core array via different silicon layers.

[0077] See now Figure 10 The diagram illustrates an example of a computing node. Computing node 10 is merely one example of a suitable computing node and is not intended to impose any limitation on the scope of use or functionality of the embodiments described herein. In any case, computing node 10 is capable of implementing and / or performing any of the functions set forth above.

[0078] Within compute node 10, there exists a computer system / server 12 that can operate alongside many other general-purpose or special-purpose computing system environments or configurations. Examples of known computing systems, environments, and / or configurations that may be suitable for computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the aforementioned systems or devices.

[0079] The computer system / server 12 can be described in the general context of computer system executable instructions (such as program modules) executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. The computer system / server 12 can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules may reside in local and remote computer system storage media, including memory storage devices.

[0080] like Figure 10As shown, the computer system / server 12 in computing node 10 is shown in the form of a general-purpose computing device. The components of computer system / server 12 can include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including system memory 28 to processor 16.

[0081] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, PCI Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA) O

[0082] In embodiments, one or more inference processing units (not shown) are coupled to bus 18. In such embodiments, the IPU can receive data from memory 28 or write data to memory 28 via bus 18. Likewise, the IPU can interact with other components via bus 18, as described herein.

[0083] Computer system / server 12 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by computer system / server 12, and it includes both volatile and non-volatile media, removable and non-removable media.

[0084] The system memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer system / server 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 can be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Although not shown, a magnetic disk drive can also be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive can be provided for reading from or writing to a removable, non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical media). In such instances, each can be connected to bus 18 by one or more data media interfaces. As will be further depicted and described below, memory 28 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the disclosure.

[0085] Program / utility 40 having a set of program modules 42, and operating system software, one or more application programs, other program modules, and program data, can be stored in memory 28 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, can include an implementation of a networking environment. Program modules 42 generally carry out the functions and / or methodologies of embodiments as described herein.

[0086] Computer system / server 12 can also communicate with one or more external devices 14 such as a keyboard, a pointing device, a display 24, etc.; one or more devices that enable a user to interact with computer system / server 12; and / or any devices (e.g., network card, modem, etc.) that enable computer system / server 12 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface(s) 22. Still yet, computer system / server 12 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via network adapter 20. As depicted, network adapter 20 communicates with the other components of computer system / server 12 via bus 18. It should be understood that although not shown, other hardware and / or software components could be used in conjunction with computer system / server 12. Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0087] The present disclosure can be embodied as a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0088] A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, semiconductor, or any other suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0089] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0090] Computer readable program instructions for carrying out operations of the present disclosure can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0091] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0092] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer readable storage medium having no data storage cycles that change state. The instructions can be executed by the computer directly or, alternatively, indirectly, such as through a read only memory or some other medium that stores instructions for execution by the computer.

[0093] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, thereby causing the instructions to be executed on the computer, other programmable apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0094] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to different embodiments of this disclosure. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually execute substantially simultaneously, or these blocks may sometimes execute in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

Claims

1. A neural inference chip, wherein the chip is laid out using a silicon 3D die and wafer stack comprising multiple silicon layers, the neural inference chip comprising: The first silicon layer comprises multiple neural nuclei, each nucleus including: A neural computation unit adapted to apply multiple synaptic weights to multiple input activations to generate multiple output activations; The second silicon layer includes a first neural network model memory adapted to store the plurality of synaptic weights; A communication network, operatively coupled to the first neural network model memory and each of the plurality of neural nuclei, and adapted to provide the synaptic weights from the first neural network model memory to each of the plurality of neural nuclei, wherein the communication network includes a plurality of through-silicon vias providing connectivity between the first silicon layer and the second silicon layer, and wherein the communication network has at least three dimensions, a first dimension of the at least three dimensions extending between the silicon layers of the neural inference chip, and a second dimension of the at least three dimensions extending within a silicon layer of the neural inference chip, wherein the communication network is configured to provide a dedicated bus separate from the plurality of neural nuclei to each of the plurality of neural nuclei.

2. The neural inference chip according to claim 1, further comprising: At least one additional silicon layer, including at least one additional neural network model memory, wherein The communication network is also operatively coupled to the at least one additional neural network model memory and adapted to provide synaptic weights from the at least one additional neural network model memory to each of the plurality of neural nuclei.

3. The neural inference chip of claim 2, wherein, The neural network model is stored across the first neural network model memory and the at least one additional neural network model memory.

4. The neural inference chip of claim 2, wherein, Multiple neural network models are stored across the first neural network model memory and the at least one additional neural network model memory.

5. The neural inference chip of claim 1, wherein, Each core also includes: An activation memory is configured to store the input activation and the output activation. A local controller, the local controller being adapted to load the input activations from the activation memory into the neural computing unit and to store the plurality of output activations from the neural computing unit into the activation memory.

6. The neural inference chip according to claim 1, further comprising: The third silicon layer includes the activation memory, wherein... The communication network is also operatively coupled to the activation memory and adapted to provide activation from the activation memory to each of the plurality of neural nuclei.

7. The neural inference chip according to claim 1, further comprising: The third silicon layer includes the activation memory, wherein... An additional communication network is operatively coupled to the activation memory and adapted to provide activation from the activation memory to each of the plurality of neural nuclei.

8. The neural inference chip according to claim 1, further comprising: The third silicon layer includes multiple neural nuclei, among which The communication network is operatively coupled to the third silicon layer and adapted to provide the synaptic weights from the first neural network model memory to each of the plurality of neural nuclei in the third silicon layer.

9. The neural inference chip of claim 8, configured to provide a first neural network model to both the first silicon layer and the third silicon layer.

10. The neural inference chip of claim 8, configured to provide a different neural network model to each of the first silicon layer and the third silicon layer.

11. The neural inference chip of claim 1, wherein, The communication network is adapted to provide the same synaptic weights to each of the cores.

12. The neural inference chip according to claim 1, wherein, The communication network is adapted to provide the same synaptic weights to a subset of the cores.

Images