Neural Network Processing

Abstract

A contention-free, stall-free broadcast network on a neural inference chip is provided. The neural inference chip includes a plurality of network nodes and a network-on-chip interconnecting the plurality of network nodes. The network includes at least one pair of directed paths. Each pair of paths has opposite directions and a common termination. The network is configured to accept data at any of a plurality of nodes. The network is configured to propagate data along a first path from a source node of the pair of directed paths to the common termination of the pair of directed paths, and along a second path from the common termination of the pair of directed paths to one or more destination nodes of the pair of directed paths.

Description

[Technical Field]

[0001] The present invention relates generally to neural network processing. Summary of the Invention

[0002] According to one aspect of the present invention, a neural inference chip includes a plurality of network nodes and a network-on-chip interconnecting the plurality of network nodes. The network includes at least one pair of directional paths. Each pair of paths has opposite directions and a common end. The network is configured to accept data at any of a plurality of nodes. The network is configured to propagate data along a first path from a source node of the pair of directional paths to the common end of the pair of directional paths, and along a second path from the common end of the pair of directional paths to one or more destination nodes of the pair of directional paths.

[0003] According to another aspect, a method and computer program product are provided for operating a network-on-chip. Data is received at a plurality of network nodes. The plurality of network nodes are interconnected by the network-on-chip. The network includes at least one pair of directed paths. Each pair of paths has opposite directions and a common termination. Data is propagated along a first path from a source node of the pair of directed paths to the common termination of the pair of directed paths, and along a second path from the common termination of the pair of directed paths to one or more destination nodes of the pair of directed paths. [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 illustrates a neural core, according to an embodiment of the present disclosure. [Figure 2] FIG. 1 illustrates an exemplary inference processing unit (IPU), according to an embodiment of the present disclosure. [Figure 3]FIG. 1 illustrates a multi-core inference processing unit (IPU) according to an embodiment of the present disclosure. [Figure 4] FIG. 1 illustrates a neural core and associated network according to an embodiment of the present disclosure. [Figure 5] 1 is a schematic diagram of a one-dimensional broadcast network-on-chip (NOC) according to an embodiment of the present disclosure. [Figure 6] 1 is a schematic diagram of an exemplary baseline of a one-dimensional broadcast network-on-chip (NOC), according to an embodiment of the present disclosure. [Figure 7] 1 is a schematic diagram of an exemplary baseline of a one-dimensional broadcast network-on-chip (NOC), according to an embodiment of the present disclosure. [Figure 8] 1 is a schematic diagram of an exemplary baseline of a one-dimensional broadcast network-on-chip (NOC) employing the J-scheme, according to an embodiment of the present disclosure. [Figure 9] 1 is a schematic diagram of an exemplary baseline of a one-dimensional broadcast network-on-chip (NOC) employing the J-scheme, according to an embodiment of the present disclosure. [Figure 10] 1 is a schematic diagram of an exemplary baseline of a one-dimensional broadcast network-on-chip (NOC) employing the J-scheme, according to an embodiment of the present disclosure. [Figure 11] 1 is a schematic diagram of a two-dimensional broadcast network-on-chip (NOC) according to an embodiment of the present disclosure. [Figure 12] 1 is a schematic diagram of a two-dimensional broadcast network-on-chip (NOC) employing the J scheme, according to an embodiment of the present disclosure. [Figure 13] FIG. 1 is a schematic diagram of a J-scheme overview of a model network-on-chip (NOC), according to an embodiment of the present disclosure. [Figure 14] FIG. 1 is a schematic diagram of a model network-on-chip (NOC) packet format, according to an embodiment of the present disclosure. [Figure 15]FIG. 1 is a schematic diagram of an addressing configuration for a model network-on-chip (NOC), according to an embodiment of the present disclosure. [Figure 16] FIG. 1 is a schematic diagram of a one-dimensional router for a model network-on-chip (NOC), according to an embodiment of the present disclosure. [Figure 17] FIG. 1 is a schematic diagram of a two-dimensional router for a model network-on-chip (NOC), according to an embodiment of the present disclosure. [Figure 18A] 1 is a schematic diagram of a list of router routes for a model network-on-chip (NOC), according to an embodiment of the present disclosure. [Figure 18B] 1 is a schematic diagram of a list of router routes for a model network-on-chip (NOC), according to an embodiment of the present disclosure. [Figure 19] FIG. 1 is a schematic diagram of an external interface of a model network-on-chip (NOC), according to an embodiment of the present disclosure. [Figure 20] 1 is a schematic diagram of a model network-on-chip (NOC) array logic partitioning according to an embodiment of the present disclosure. [Figure 21A] 1 is a schematic diagram of a model network-on-chip (NOC) array logic partitioning according to an embodiment of the present disclosure. [Figure 21B] 1 is a schematic diagram of a model network-on-chip (NOC) array logic partitioning according to an embodiment of the present disclosure. [Figure 22] FIG. 1 illustrates a method for computing neural activations according to an embodiment of the present disclosure. [Figure 23] FIG. 1 illustrates a computing node according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0005] An artificial neuron is a mathematical function whose output is a nonlinear function of a linear combination of its inputs. Two neurons are connected if the output of one is an input to the other. A weight is a scalar value that encodes the strength of the connection between the output of one neuron and the input of the other neuron.

[0006] A neuron calculates an output, called an activation, by applying a nonlinear activation function to a weighted sum of its inputs. A weighted sum is an intermediate result calculated by multiplying each input by its corresponding weight and accumulating the products. A partial sum is a weighted sum of a subset of the inputs. The weighted sum of all inputs can be calculated incrementally by accumulating one or more partial sums.

[0007] A neural network is a collection of one or more neurons. Neural networks are often divided into groups of neurons called layers. A layer is a collection of one or more neurons that all receive input from the same layer, all send output to the same layer, and usually perform a similar function. An input layer is a layer that receives input from sources outside the neural network. An output layer is a layer that sends output to targets outside the neural network. All other layers are intermediate processing layers. A multilayer neural network is a neural network with two or more layers. A deep neural network is a multilayer neural network with many layers.

[0008] A tensor is a multidimensional array of numbers. A tensor block is a contiguous subarray of elements within a tensor.

[0009] Each neural network layer is associated with a parameter tensor V, a weight tensor W, an input data tensor X, an output data tensor Y, and an intermediate data tensor Z. The parameter tensor contains all of the parameters that control the neuron activation function σ within the layer. The weight tensor contains all of the weights that connect inputs to the layer. The input data tensor contains all of the data that the layer consumes as input. The output data tensor contains all of the data that the layer computes as output. The intermediate data tensor contains any data that the layer creates as intermediate computations, such as partial sums.

[0010] The data tensors for a layer (input, output, and intermediate) can be three-dimensional, with the first two dimensions interpreted as encoding spatial location and the third dimension interpreted as encoding a different feature. For example, if the data tensor represents a color image, the first two dimensions encode the vertical and horizontal coordinates within the image, and the third dimension encodes the color at each location. Every element of the input data tensor X can be connected to every neuron by a separate weight; thus, the weight tensor W generally has six dimensions: the three dimensions of the input data tensor (input row a, input column b, input feature c) concatenated with the three dimensions of the output data tensor (output row i, output column j, output feature k). The intermediate data tensor Z has the same shape as the output data tensor Y. The parameter tensor V concatenates the dimensions of the three output data tensors with an additional dimension o that indexes the parameters of the activation function σ. In some embodiments, the activation function σ does not require any additional parameters, in which case no additional dimensions are required. However, in some embodiments, the activation function σ requires at least one additional parameter that appears in dimension o.

[0011] The elements of the layer's output data tensor Y can be calculated as in Equation 1, where the neuron activation function σ is constructed by a vector of activation function parameters V[i,j,k,:], and the weighted sum Z[i,j,k] can be calculated as in Equation 2. Y[i,j,k]=σ(V[i,j,k,:];Z[i,j,k]) formula 1

number

[0012] For notational simplicity, the weighted sum in Equation 2 is sometimes referred to as the output, which is equivalent to using a linear activation function Y[i,j,k]=σ(Z[i,j,k])=Z[i,j,k], with the understanding that the same statements apply without loss of generality when a different activation function is used.

[0013] In various embodiments, the computation of the output data tensor is decomposed into smaller problems, each of which may then be solved in parallel on one or more neural cores, or on one or more cores of a conventional multi-core system.

[0014] From the above, it should be clear that neural networks are parallel structures. A neuron in a given layer receives elements x from one or more layers. i Each neuron receives an input X with an element W i The neural network computes its state y∈Y based on weights W with σ and σ. In various embodiments, the weighted sum of the inputs is adjusted by a bias b, and the result is then passed to a nonlinearity F(·). For example, a single neuron activation can be calculated as y=F(b+Σx i W i ) may also be expressed as

[0015] Because all neurons in a given layer receive input from the same layer and calculate their outputs independently, neuron activations can be calculated in parallel. This aspect of the entire neural network accelerates the overall computation by performing calculations in parallel across distributed cores. Furthermore, within each core, vector operations can be calculated in parallel. Even with recurrent inputs, for example, when a layer projects back onto itself, all neurons are still updated simultaneously. In effect, recurrent connections are delayed to align with subsequent inputs to the layer.

[0016] Referring now to FIG. 1, a neural core according to an embodiment of the present disclosure is shown. Neural core 100 is a tileable computational unit that computes one block of output tensors. Neural core 100 has M inputs and N outputs. In various embodiments, M=N. To compute an output tensor block, the neural core multiplies M×1 input tensor blocks 101 with M×N weight tensor blocks 102 and accumulates the products into a weighted sum, which is stored in a 1×N intermediate tensor block 103. The O×N parameter tensor block contains O parameters that specify each of N neuron activation functions that are applied to the intermediate tensor blocks 103 to create a 1×N output tensor block 105.

[0017] The multiple neural cores may be tiled in a neural core array, hi some embodiments, the array is two-dimensional.

[0018] A neural network model is a set of constants that collectively specify the entire computation performed by a neural network, including the graph of connections between neurons and the weights and activation function parameters for all neurons. Training is the process of modifying a neural network model to perform a desired function. Inference is the process of applying a neural network to inputs to produce outputs without modifying the neural network model.

[0019] An inference processing unit is a category of processor that performs neural network inference. A neural inference chip is a specific physical instance of an inference processing unit.

[0020] Referring to FIG. 2, an exemplary inference processing unit (IPU) according to an embodiment of the present disclosure is shown. The IPU 200 includes a memory 201 for a neural network model. As described above, the neural network model may include synaptic weights for the neural network to be computed. The IPU 200 includes an activation memory 202, which may be transient. The activation memory 202 may be divided into an input domain and an output domain and stores neuron activations for processing. The IPU 200 includes a neural computation unit 203 into which the neural network model is loaded from the model memory 201. Before each computation step, input activations are provided from the activation memory 202. Output from the neural computation unit 203 is written back to the activation memory 202 for processing on the same or another neural computation unit.

[0021] In various embodiments, the IPU 200 includes microengines 204. In such embodiments, all operations in the IPU are directed by the microengines. As described below, various embodiments may provide a central microengine, distributed microengines, or both. A global microengine may be referred to as a chip microengine, and a local microengine may be referred to as a core microengine or local controller. In various embodiments, a microengine comprises one or more microengines, microcontrollers, state machines, CPUs, or other controllers.

[0022] Referring to FIG. 3, a multi-core inference processing unit (IPU) according to an embodiment of the present disclosure is shown. The IPU 300 includes a memory 301 for neural network models and instructions. In some embodiments, the memory 301 is divided into a weight portion 311 and an instruction portion 312. As described above, the neural network model may include synaptic weights for the neural network to be calculated. The IPU 300 includes an activation memory 302, which may be transient. The activation memory 302 may be divided into an input domain and an output domain and stores neuron activations for processing.

[0023] The IPU 300 includes an array 306 of neural cores 303. Each core 303 includes a computation unit 333, which is loaded with a neural network model from the model memory 301 and operates to perform vector computations. Each core also includes a local activation memory 332. Before each computation step, input activations are provided from the local activation memory 332. Outputs from the computation unit 333 are written back to the activation memory 332 for processing on the same or another computation unit.

[0024] The IPU 300 includes one or more networks-on-chip (NoCs) 305. In some embodiments, a partial sum NoC 351 interconnects the cores 303 and transfers partial sums between the cores 303. In some embodiments, a separate parameter distribution NoC 352 connects the cores 303 to memory 301 for distributing weights and instructions to the cores 303. It will be appreciated that various configurations of NoCs 351 and 352 are suitable for use with the present disclosure. For example, a broadcast network, a row broadcast network, a tree network, and a switched network may be used.

[0025] In various embodiments, the IPU 300 includes a global micro-engine 304. In various embodiments, a local core controller 334 is included on each core 303. In such embodiments, the global micro-engine (chip micro-engine) and the local core controller (core micro-engine) cooperate to direct operation. Specifically, at 361, the global micro-engine 304 loads computational instructions from the instruction portion 312 of the model memory 301 into the core controller 334 on each core 303. At 362, the global micro-engine 304 loads parameters (e.g., neural network / synaptic weights) from the weight portion 311 of the model memory 301 into the neural computation unit 333 on each core 303. At 363, the local core controller 334 loads neural network activation data from the local activation memory 332 into the neural computation unit 333 on each core 303. As described above, activations are provided to neurons of a particular neural network defined by the model and may originate from the same or another neural computation unit, or from outside the system. At 364, the neural computation unit 333 performs a computation to generate an output neuron activation as directed by the local core controller 334. Specifically, the computation involves applying input synaptic weights to the input activations. It will be appreciated that various methods are available for performing such computations, including in silico dendrites and vector multiplication units. At 365, results from the computation are stored in the local activation memory 332 as directed by the local core controller 334. As described above, these stages may be pipelined to achieve efficient use of the neural computation units in each core. It will also be appreciated that inputs and outputs may be transferred from the local activation memory 332 to the global activation memory 302 according to the requirements of a given neural network.

[0026] Thus, the present disclosure provides run-time control of operations in an inference processing unit (IPU). In some embodiments, the micro-engines are centralized (a single micro-engine). In some embodiments, the IPU computations are distributed (performed by an array of cores). In some embodiments, the run-time control of operations is hierarchical, involving both central and distributed micro-engines.

[0027] One or more microengines direct the execution of all operations within the IPU. Each microengine instruction corresponds to several sub-operations (e.g., address generation, load, calculation, store, etc.). Core microcode executes on the core microengines (e.g., 334). For local computation, the core microcode contains instructions to perform complete single tensor operations, such as a convolution between a weight tensor and a data tensor. For distributed computation, the core microcode contains instructions to perform single tensor operations on locally stored subsets (and partial sums) of the data tensor. Chip microcode executes on the chip microengines (e.g., 304). The microcode contains instructions to perform all tensor operations in the neural network.

[0028] Referring now to FIG. 4, an exemplary neural core and associated network according to an embodiment of the present disclosure is shown. Core 401, which may be embodied as described with reference to FIG. 1, is interconnected with additional cores by networks 402 through 404. In this embodiment, network 402 is responsible for distributing weights and / or instructions, network 403 is responsible for distributing partial sums, and network 404 is responsible for distributing activations. However, it will be understood that various embodiments of the present disclosure may combine these networks or further separate them into multiple additional networks.

[0029] Input activations (X) are distributed to cores 401 via activation network 404 from off-core to activation memory 405. Layer instructions are distributed to cores 401 via weight / instruction network 402 from off-core to instruction memory 406. Layer weights (W) and / or parameters are distributed to cores 401 via weight / instruction network 402 from off-core to weight memory 407 and / or parameter memory 408.

[0030] A vector matrix multiplication (VMM) unit 409 reads a weight matrix (W) from weight memory 407. A vector matrix multiplication (VMM) unit 409 reads an activation vector (V) from activation memory 405. Then, the vector matrix multiplication (VMM) unit 409 performs a vector matrix multiplication Z=X T The vector-vector unit 410 computes W and provides the result to the vector-vector unit 410. The vector-vector unit 410 reads additional partial sums from the partial sum memory 411 and receives additional partial sums from off-core via the partial sum network 403. The vector-vector unit 410 computes a vector-vector operation from these source partial sums. For example, various partial sums may be summed in sequence. The resulting target partial sum may be written to the partial sum memory 411, sent off-core via the partial sum network 403, or fed back for further processing by the vector-vector unit 410, or a combination thereof.

[0031] After all calculations for a given layer's inputs are completed, the partial sum results from the vector-vector unit 410 are provided to the activation unit 412 for calculation of the output activations. The activation vector (Y) is written to the activation memory 405. The layer activations (including the results written to the activation memory) are redistributed from the activation memory 405 across cores via the activation network 404. As these layer activations are received, they are written to the local activation memory of each receiving core. Once processing for a given frame is complete, the output activations are read from the activation memory 405 and sent off-core via the network 404.

[0032] Thus, in operation, a core control micro-engine (e.g., 413) coordinates the data movement and computation of the core. The micro-engine issues an activation memory address read operation to load an input activation block into the vector matrix multiplication unit. The micro-engine issues a weight memory address read operation to load a weight block into the vector matrix multiplication unit. The micro-engine issues a calculation operation to the vector matrix multiplication unit to cause the vector matrix multiplication unit to calculate a partial sum block.

[0033] The microengine issues one or more of a partial sum read / write memory address operation, a vector calculation operation, or a partial sum communication operation to read the partial sum data from the partial sum source, calculate it using the partial sum arithmetic unit, or write the partial sum data to the partial sum target. Writing the partial sum data to the partial sum target may include communicating external to the core via a partial sum network interface or sending the partial sum data to the activation arithmetic unit.

[0034] The micro-engine issues an activation function calculation operation to cause the activation function arithmetic unit to calculate the output activation block, and issues an activation memory address write, which writes the output activation block to the activation memory through the activation memory interface.

[0035] Thus, various sources, targets, address types, computation types, and control components are defined for a given core.

[0036] The sources for the vector-vector unit 410 include the vector-matrix multiplication (VMM) unit 409, constants from the parameter memory 408, the partial sum memory 411, the partial sum results from the previous cycle (TGT partial sums), and the partial sum network 403.

[0037] The targets of the vector-vector unit 410 include a partial sum memory 411, the partial sum result of the subsequent cycle (SRC partial sum), an activation unit 412, and a partial sum network 403.

[0038] Thus, a given instruction may read or write from activation memory 405, read from weight memory 407, or read or write from partial sum memory 411. Computational operations performed by the core include vector-matrix multiplication by VMM unit 409, vector (partial sum) operations by vector-vector unit 410, and activation functions by activation unit 412.

[0039] The control actions include updating a program counter and a loop counter and / or sequence counter.

[0040] Thus, memory operations are issued to read weights from addresses in weight memory, read parameters from addresses in parameter memory, read activations from addresses in activation memory, and read / write partial sums to / from addresses in partial sum memory. Computation operations are issued to perform vector-matrix multiplication, vector-vector operations, and activation functions. Communication operations are issued to select vector-vector operands, route messages on the partial sum network, and select partial sum targets. Loops over layer outputs and loops over layer inputs are controlled by control operations that specify program counters, loop counters, and sequence counters within the microengines.

[0041] According to an embodiment of the present disclosure, data distribution may be provided from a global memory array. The global memory array includes a plurality of elements, each element comprising a memory element and a buffer. Weights and instructions are provided from the global array to the array of cores over a network. An exemplary configuration of the cores is described above with respect to FIG. 3. The memory 303 in FIG. 3 may be implemented as described with respect to the global memory array.

[0042] As noted above, many core architectures for neural inference offer significant advantages in computational power. However, if neural network weights and parameters are not provided to the computational cores in a timely manner, the cores cannot perform useful computations. As a result, the performance of a neural chip may be limited by the ability to deliver neural network weights and parameters to the computational cores on the chip. On-chip memory significantly improves memory bandwidth compared to typical off-chip memory, such as dynamic random access memory (DRAM) or high-bandwidth memory (HBM). Furthermore, on-chip memory is more energy-efficient than off-chip memory, leading to more power-efficient neural inference systems. In various embodiments, on-chip memory may include static random access memory (SRAM) or other embedded memory. However, delivering neural network weights to the cores fast enough to keep up with the processing speed remains a challenge.

[0043] Certain efficiencies can be achieved by using convolutional neural networks (CNNs). CNNs use the same weight matrix (sometimes called a convolution filter) repeatedly. To minimize the amount of on-chip memory used, it is preferable to store a given weight matrix in a single location without duplication. To store large neural networks, some on-chip memory embodiments consist of a collection of many memory elements. It will also be appreciated that many cores may be targets for memory weights. This creates the problem of many-to-many (many memory elements to many cores) communication. Broadcasting weights can cause congestion in the network-on-chip (NoC), resulting in many contentions and pipeline stalls, leading to reduced broadcast bandwidth.

[0044] As outlined above, various embodiments of a neural inference chip provide a grid of neural inference cores to accelerate neural network inference. In various embodiments, instruction pre-scheduling is provided. Neural network evaluations involve a regular computation pattern, and therefore instructions can be pre-scheduled without stalls to achieve high performance. However, it is preferable that all neural network weights are delivered to the cores just in time as pre-scheduled. If the weight distribution network becomes congested and weight distribution stalls, pre-scheduling of neural network evaluations will fail.

[0045] This disclosure provides a stall-free weight distribution network-on-chip for distributing weight parameters from a grid of memory elements to a grid of compute cores. First, a one-dimensional scheme is presented below, then extended to a two-dimensional grid scheme. These techniques are further extended to support different weight distributions, such as striping (different rows of cores get different weights).

[0046] The techniques described herein function without contention even when the delivery times of instructions to memory elements and data from memory elements to computing cores vary. These techniques accommodate instruction flows that can access any column in any order. The new scheme removes the constraint that all columns must start at the same time.

[0047] According to one aspect of the present disclosure, contention- and stall-free broadcasting is provided for network-on-chip (NoC) applications. In some embodiments, broadcast data is provided from multiple sources over a network-on-chip (NoC) with zero stall cycles on the network such that data is not backpressured on the NoC, data is provided continuously (there are no empty slots on the NoC), and data remains collision-free.

[0048] Thus, the present disclosure provides many advantages, including: - Incorporate multiple sources to allow for distributed memory storage. - Precise and deterministic scheduled behavior. - Full throughput is required (no idle cycles, no data loss). - Simplified router design - minimal design for very wide data paths. - Simplifying the program (e.g. same program for all nodes).

[0049] Referring now to FIG. 5, an exemplary one-dimensional broadcast for a network-on-chip is shown, according to an embodiment of the present disclosure. As shown, at different times t=0 and t=1, a first source 501 ("Source 1") broadcasts to multiple destinations on the bus. Then, at times t=2 and t=3, a second source 510 ("Source 2") broadcasts without overlapping (conflicting) with the first source's broadcast and without introducing empty cycles into the network. In the illustrated exemplary embodiment, there are two broadcast sources and eight destinations (represented by vertical boxes or cells). However, those skilled in the art will understand that the data sources and destinations may be scaled as needed. In the exemplary embodiment, the first source 501 is introduced at the midpoint (or fourth destination), and the second source 510 is introduced at a different location (e.g., one destination below the first source 501). As indicated by arrow 520, the data eventually reaches all locations.

[0050] Referring now to Figure 6, an exemplary baseline for a one-dimensional broadcast network-on-chip is shown. In this example, a first source ("Source 1") transmits four consecutive data words (0:3). With each time increment, the data words propagate vertically in opposite directions. As the data propagates, all destinations receive all data words (0:3) in sequence. At time t=4, the first destination cell into which Source 1 introduced data is erased or emptied by the data propagation. This data erasure continues until each data word (0:3) has propagated beyond all eight vertical destinations, as shown at time t=8.

[0051] FIG. 7 illustrates another exemplary baseline for a one-dimensional broadcast network-on-chip, presenting both empty cells and contention due to a one-clock-cycle latency. Here, a first source ("Source 1") transmits two consecutive data words (0:1) to a fourth cell 701, as shown in columns t=0 and t=1. Then, a second source ("Source 2") transmits two consecutive data words (2:3) to an adjacent cell 720, as shown in columns t=2 and t=3. As previously mentioned, data propagates vertically in opposite directions. As a result, multiple cells (from columns t=2 to t=5, as shown) are empty, reducing efficiency and performance. Furthermore, multiple cells (from columns t=2 to t=5, as shown, or until the data propagates to the destination) have contention introduced by the simultaneous presence of data from both Source 1 and Source 2, further reducing efficiency and performance. This disclosure provides a novel approach to avoid such empty and conflicting data flows.

[0052] FIG. 8 illustrates an exemplary embodiment according to the present disclosure that can incorporate a J-way data flow to avoid the occurrence of empty or conflicting data broadcasts. As shown, data propagates vertically in one direction as Source 1 transmits four consecutive data words (0:3). In this exemplary embodiment, as indicated by the broken arrow data flow, data flows north (N) (or upward) until it reaches the top cell, where it loops back for delivery in the opposite direction (S) (or downward). While the data is on the downward trajectory (i.e., starting at time t=3), it is delivered to its destination address.

[0053] FIG. 9 illustrates another exemplary embodiment according to the present disclosure that can incorporate a J-way data flow to avoid the occurrence of empty or conflicting data broadcasts. Here, Source 1 transmits two consecutive data words (0:1) at times t=0 and t=1, followed by a second, adjacent Source 2 transmitting two consecutive data words (2:3) at times t=2 and t=3. The data words propagate vertically in one direction (N) and are looped back for distribution in the opposite direction (S), with Source 1's data loopback (i.e., reversal of propagation direction) beginning at time t=3 and Source 2's data loopback (i.e., reversal of propagation direction) beginning at time t=5. As shown, there are no conflicts or empty data cells / addresses between Source 1's and Source 2's inputs, and this remains true throughout the entire propagation path.

[0054] FIG. 10 illustrates another exemplary embodiment according to the present disclosure that can incorporate a J-way data flow to avoid empty or conflicting data broadcasts. Here, Source 1 transmits two consecutive data words (0:1) at times t=0 and t=1. Then, at times t=3 and t=4, a second adjacent Source 2 transmits two consecutive data words (2:3). Note that Source 2 does not transmit at time t=2. Therefore, to avoid an empty cell, Source 2 transmits its initial data at a higher location (i.e., the second cell from the top) (at time t=3). Once transmitted, the data word again propagates vertically in one direction (N) and loops back for distribution in the opposite direction (S) as previously described.

[0055] According to one aspect of the present disclosure, the data propagation at time t=8 is the same for each of the examples shown in Figures 8-10. Therefore, the distribution pattern to the eight destinations is the same across all three J-scheme examples shown.

[0056] FIG. 11 illustrates an exemplary embodiment according to the present disclosure that can incorporate J-way data flow in a two-dimensional broadcast network-on-chip. At time t=1, as shown by the dashed line, cores (0:3) begin transmitting, and data begins moving upward, or in the N direction. At time t=2, this data reaches an upper limit, as shown in callout 1101, before turning and reversing direction according to the J-way method disclosed herein. Thus, at time t=3, cores (0:3) transmissions reside in the first two rows, data from cores 1 and 2 coexist in the first row, and data from cores 0 and 3 coexist in the second row.

[0057] At time t=4, a second source begins transmitting data from core (4:7), and the data propagates in a similar J-mode, as indicated by the solid line. At time t=5, the data transmission from the second core begins to change direction and reverse from N to S. Thus, at time t=6, the data from the first core (0:3) has completely changed direction and is propagating entirely in the S-direction, with the initial transmission 0 being delivered to the output, while the data from the second core (4:7) is now located in the first two rows, as shown, with data from cores 5 and 6 coexisting in the first row and data from cores 4 and 7 coexisting in the second row.

[0058] At time t=7, a third source begins sending data from cores (8:11), and the data propagates in a similar J-mode, as shown by the dash-dotted lines. At this point, half of the data from the first cores (0:3) has been delivered to the output, and only data from cores 2 and 3 is still propagating in the S-direction. Data from the second source (i.e., cores 4:7) has almost completely reversed direction, and data from core 7 is at its upper limit and about to switch to the S-direction on the next clock cycle. None of the data from the third source (i.e., cores 8:11) has yet to switch direction.

[0059] According to one aspect of the present disclosure, the J-way data propagation flow described above for the first (vertical) dimension may also be employed in the second (horizontal) dimension, which is shown in callout 1102, where data first propagates in the W (or left) direction, as indicated by the dashed arrow, and then flips to the E (or right) direction upon reaching the W / left boundary.

[0060] Referring to the bottom core shown in Figure 11, at time t=3, the first transmission of data from core (0:3) is shown in the first row, with the data already flipped to the E direction. A second transmission of data from the same core (0:3) can begin transmission in the second row, with none of the data reversing direction. This can continue as data from core (0:3) continues to be multicast to each row as it passes through, as shown at time t=4. Then, as shown at time t=5, data from a second source at core (4:7), following the same W-to-E J-style propagation, can begin transmitting simultaneously. In this exemplary two-dimensional embodiment, data is first propagated in the vertical dimension (N to S) and then in the horizontal dimension (W to E), such that data is fed into the horizontal dimension as it arrives from the vertical dimension. With multiple sources broadcasting simultaneously, as shown at times t=6 and t=7, empty or conflicting data flows in this two-dimensional broadcast still do not occur.

[0061] FIG. 12 illustrates a timing diagram for an exemplary 2D J-scheme according to the present disclosure. As shown, cores within a column transmit data simultaneously, and each column transmits its data out of order, such that Core 3's transmission (in North Row 0, Column 0) overlaps with Core 4's data (in North Row 0, Column 1), but is presented on a different bus. Here, dotted line 1201 indicates a one-cycle offset in timing. According to one aspect of the present disclosure, two different streams of data can be delivered in parallel by using sources from two columns. For example, Source 1 from Column 1 can deliver to the even rows, and Source 2 from Column 2 can deliver to the odd rows (without contention). This feature can be generalized to be true for N streams of data. An advantage of this approach is that it increases the effective bandwidth for distributing unique data.

[0062] According to one aspect of the present disclosure, the structural definition of a model network-on-chip (MNoC) may include an overview of the J-system, routing rules (operations), packet formats with field (signal) definitions, addressing with direct addressing and distributed flags, router structure, router routes (list of router routes), read / write transactions, external interfaces, and latency / skew analysis. The textual representation of the protocols shown in Figures 5-12 can be expressed as follows: "J" Method Protocol Pseudocode: - Go north to the edge of the Core Array - Loop south - Travel south to the edge of the Core Array - During the southward movement, for each row, if the Y distribution flag (or row address in direct addressing mode) matches, distribute to the row - Go west to the edge of the Core Array - Loop east - Travel east to the edge of the Core Array - During the move east, for each column, if the X distribution flag (or column address in direct addressing mode) matches, distribute to core

[0063] 13 illustrates an overview of an exemplary model network-on-chip J-scheme according to the present disclosure. In some embodiments, the YX dimension routing is ordered such that data first propagates in the Y dimension and then in the X dimension. Thus, the data packet follows two "J" shaped routes in sequence. I. Route 1: North then South (as indicated by the solid J-shaped arrows) 1301 II. Route 2: West then East (as indicated by the dashed J-shaped arrows) 1302 The path of the "J" route propagates the data to travel to the end of the array, then the data is looped back. On the second leg of the route (after the loopback), the data travels the entire length of the array (in multicast addressing mode). The data is distributed on the second leg of the route (after the data is looped back). Two addressing modes can also be included: I. Multicast addressing to deliver data to all matching destinations II. Direct addressing, which delivers data to a single matching destination based on the core's row and column index.

[0064] According to one aspect of the present disclosure, a model network-on-chip may include routing rules, such as rules that ensure packets always travel one direction before the other in the Y or X dimension. - Y dimension: North then South - X Dimension: West, then East Additionally, rules can be implemented such that packets are looped from north to south (west to east) at the north (or west) end of the array, and rules can be implemented such that packets are discarded at the south (or east) end of the array. Additionally, rules such as the following can be implemented regarding packet injection: - Within the array, packets are always injected in the north direction - A special boundary case for populating an array from outside the array - In addition, the following rules can be enforced for packet delivery when addresses match: - Eastbound packets are delivered to the core - Southbound packets are delivered westward - Distributed Flag (Multicast) mode: Packets are distributed and continue to propagate in the same direction - Direct Addressing Mode: Packets are delivered and do not continue propagating in the same direction

[0065] FIG. 14 illustrates an example packet format according to the present disclosure. The illustrated example is for a packet in a 16x16 core array. In some embodiments, "ext_in_" and "ext_out_transaction" are mutually exclusive and cannot both be "1" at the same time. Also, some example transaction definitions for this configuration are as follows: ~ADDR_VALID&~DATA_VALID=No transaction (idle) ~ADDR_VALID&DATA_VALID=Invalid state (error) ADDR_VALID&~DATA_VALID=Read transaction ADDR_VALID&DATA_VALID=write transaction

[0066] FIG. 15 illustrates an exemplary model network-on-chip addressing according to the present disclosure. 1501 illustrates direct addressing (north / south direction). 1502 illustrates direct addressing (east / west direction), where the CORE_ROW_ID and CORE_COL_ID are determined by the location of the core in the core array. A packet is delivered to a specific core that matches the 4-bit DISTR_ROW_ID or DISTR_COL_ID in the packet. 1503 illustrates direct flag (multicast) addressing, where the DISTR_FLAGS are compared to the DEST_FLAGS according to a Boolean matching equation. The DEST_FLAGS are set per core in a core control register. A packet is delivered to all matching cores according to a Boolean matching equation that compares the DISTR_FLAGS in the packet with the DEST_FLAGS set in the core. Some exemplary Boolean matching equations employed in NoC addressing include the following: - ANY match=OR_REDUCE(DISTR_FLAGS&DEST_FLAGS) - ALL match=~OR_REDUCE(DISTR_FLAGS&~DEST_FLAGS) - =AND_REDUCE(~DISTR_FLAGS|DEST_FLAGS) / / Alternative DeMorgan equivalent specification - NOT ANY match=~OR_REDUCE(DISTR_FLAGS&DEST_FLAGS) - NOT ALL match=OR_REDUCE(DISTR_FLAGS&~DEST_FLAGS) - where '~' is inversion, '&' is bitwise AND, and '|' is bitwise OR

[0067] FIG. 16 shows an example one-dimensional router for a model network-on-chip according to the present disclosure. In this example model, register values ​​are updated every cycle so that stale data from the previous cycle is not retained. The timing is such that each register incurs a one-clock-cycle delay, and each endpoint also contains one register. Here, data propagation is in the vertical dimension (as indicated by the “RX_North,” “RX_South,” and “TX_North,” “TX_South” arrows). “RX_South” data can be fed upward into a first dynamic merge 1601 and corresponding register 1611 and output as “TX_North,” and “RX_North” data can be fed downward into a second dynamic merge 1602 and corresponding register 1612 and / or Y endpoint 1622 and output as “TX_South.”

[0068] FIG. 17 shows an example two-dimensional router for a model network-on-chip according to the present disclosure. In this example model, register values ​​are updated every cycle so that stale data from the previous cycle is not retained. The timing is such that each register incurs a one-clock-cycle delay, and each endpoint also contains one register. Here, data propagation occurs in the vertical dimension (as indicated by the "RX_North," "RX_South," and "TX_North," "TX_South" arrows) and the horizontal dimension (as indicated by the "RX_West," "RX_East," "TX_West," "TX_East" arrows). The "Rx_South" data may be fed north (or upward) into the first dynamic merge 1701 and corresponding register 1711 and output as "TX_North," and the "RX_North" data may be fed south (or downward) into the second dynamic merge 1702 and corresponding register 1712 and / or Y terminal 1722, and after passing through register 1712, the output may be "TX_South." Distribution from the vertical dimension to the horizontal dimension occurs at Y terminal 1722. Additionally, "RX_East" may be fed west (or left) through the third dynamic merge 1703 and output as "TX_West." "RX_West" may be fed east (or right) through the fourth dynamic merge 1704 and output as "TX_East" and / or X terminal 1723.

[0069] According to another aspect of the present disclosure, dynamic partitioning can be included to manage router operation. For purposes of explanation and not limitation, an exemplary dynamic partitioning can be implemented as follows: [Table 1] The Y endpoint 1722 and X endpoint 1723 shown in Figure 17 can implement address match logic. In some embodiments, in direct addressing mode only, if a packet's direct address matches a core row / column ID, the packet is delivered and does not continue propagating in its original direction. The dynamic merges 1701-1704 transmit valid data packets from either input. Data packets must be scheduled to avoid collisions (two simultaneous valid packets entering the dynamic merge). If a collision occurs, the ERROR flag is asserted and one of the data packets is discarded as follows: North Dynamic Merge 1: Northbound (RX_South) takes precedence over local (i.e., local data is dropped). South Dynamic Merge 2: Southbound (RX_North) takes precedence over loopbackN (i.e., data in loopbackN is discarded). West Dynamic Merge 3: Westbound (RX_East) takes precedence over South (i.e., South data is discarded). East Dynamic Merge 4: Eastbound (RX_West) takes precedence over loopbackW (i.e., data in loopbackW is discarded).

[0070] Figure 18 shows an example list of router paths for a model network-on-chip according to the present disclosure, where "cycle" = cycle within the model network-on-chip router. This example list of router paths relates to the example embodiment shown in Figure 17. When a write or read request is sent from the external interface to the core array, the external control interface injects the data packet onto the southbound path (RX_North) of core (0,0). This can be implemented by setting the "ext_in_transaction" bit = 1. The data packet follows the normal J method and travels south. This can be implemented by setting the Y address and X address as normal to reach the destination. When a write or read request is sent from the core array to the external interface (e.g., returned read data), the core injects the data onto the northbound path using the local core I / F. This can be implemented by setting the "ext_out_transaction" bit = 1. The data propagates north to the edge of the core array, loops back south, and is delivered to row 0. This can be implemented by setting Y Address = Direct Address Mode, Row 0. Also, by setting the X Address to any desired value, data can propagate west to the edge of the core array. Data can be delivered to the external control interface, where the "ext_out_transaction" bit sends the data out of the core array rather than looping it back east at core (0,0). Also, in some scenarios, such as burst reads, "ext_out_transactions" from core (0,0) can collide with "ext_in_transactions." Therefore, the W Bypass in the core (0,0) router overlaps "ext_out_transactions" from all other cores with "ext_in_transactions," avoiding the collision.

[0071] 19 illustrates an example external interface for a model network-on-chip according to the present disclosure. In this example embodiment shown, the core (0,0) configuration may be configured such that split N 1930 and split Wa,b 1931 are configured to loop back. For data traveling from the external interface: - Set "ext_in_transaction" == "1". - Dynamic Merge 2 1902 either passes the external data packet through or loops the data packet back from the northbound path. - Split Wa,b loops ext_in_transactions back onto the westbound path. Regarding data moving to external interfaces: - If "ext_out_transaction" == "1", the data packet goes from TX_West to the external interface and is not looped back by the Wa,b split. Although this exemplary embodiment shows data being distributed from the west side of the core, alternative configurations can be constructed as needed.

[0072] FIG. 20 illustrates an exemplary array-logical partitioning of a model network-on-chip according to the present disclosure. The partitioning is indicated by lines P1 and P2, which intersect in the middle of the array of cores so that data propagates within any given subarray to the edge of that subarray (rather than to the edge of the entire core) and then loops back in a J-way flow pattern. In this exemplary embodiment, array-logical partitioning is achieved by using both link disables, as identified in 2050, and loopback signals at partition boundaries. With respect to the routing protocol, each logical subarray should operate as if it were an independent, complete array. In this way, each subarray J-way is independent and may therefore have a different pattern. Link disable definitions may be implemented such that each directional interface {N, S, E, W} has an disable signal used to partition the array. Additionally, exemplary link behavior based on link disable and loopback settings for both data and external transactions is listed in FIG. 21. For "zero," all signals (addr_valid, addr, data_valid, data, etc.) are set to zero, but "ext_in_transactions" continues to be injected at the top left and can reach any core in the array. While the logical subarray 2025 shown in Figure 20 is shown as a square containing four cores, subarrays of alternative sizes / shapes (e.g., rectangular) can be implemented as desired.

[0073] Referring to Figure 22, a method for computing neural activations is shown. At 2201, data is received at a plurality of network nodes. The plurality of network nodes are interconnected by a network-on-chip. The network includes at least a pair of directed paths. Each pair of paths has opposite directions and a common termination. At 2202, data is propagated along a first path from a source node of the pair of directed paths to a common termination of the pair of directed paths, and along a second path from the common termination of the pair of directed paths to one or more destination nodes.

[0074] Various embodiments of the present disclosure provide instruction and data distribution in one- or two-dimensional memory arrays using a combination of instruction buffers, horizontal buffers, vertical buffers, and layover buffers. It will be appreciated that the present disclosure is applicable to higher-dimensional arrays by adding additional buffers. In these embodiments, the time from instruction issue to data output from the data array is constant, even if each phase takes a different amount of time. Columns may be accessed in random order. When using more than one dimension, two instructions accessing the same column should be separated by the vertical distribution time. In the one-dimensional case, the vertical distribution time is zero and therefore unconstrained.

[0075] In various embodiments, a system is provided that includes a memory array, an instruction buffer, and a horizontal data buffer, wherein the total number of cycles for instruction distribution and memory distribution is constant for all instructions.

[0076] In various embodiments, a two-dimensional memory array is provided. A horizontal buffer is provided for each row of the memory array. A vertical buffer is provided for each column of the memory array. The total number of cycles for instruction distribution, data distribution along the vertical buffers, and data distribution along the horizontal buffers is constant.

[0077] In various embodiments, a two-dimensional memory array is provided. A layover buffer is provided at each location in the memory array. The total number of cycles for instruction distribution, data distribution along the vertical buffers, data distribution along the horizontal buffers, and data transfers to the layover buffers is constant.

[0078] 23, a schematic diagram of an example computing node is shown. Computing node 10 is merely one example of a suitable cloud computing node and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments described herein. Nevertheless, computing node 10 may be implemented and / or perform any of the functionality described herein above.

[0079] Computing node 10 includes computer system / server 12, which may be operable in numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0080] Computer system / server 12 may be described in the general context of computer system-executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system / server 12 may also be practiced in distributed cloud computing environments where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0081] 23, computer system / server 12 within computing node 10 is shown in the form of a general-purpose computing device. Components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 coupling various system components, including system memory 28, to processor 16.

[0082] Bus 18 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures, including, by way of example and not limitation, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe), and an Advanced Microcontroller Bus Architecture (AMBA).

[0083] In various embodiments, one or more inference processing units (not shown) are coupled to bus 18. In such embodiments, the IPU may receive data from or write data to memory 28 via bus 18. Similarly, the IPU may interact with other components via bus 18, as described herein.

[0084] Computer system / server 12 typically includes a variety of computer system-readable media, which can be any available media that can be accessed by computer system / server 12 and includes both volatile and nonvolatile media, removable and non-removable media.

[0085] The system memory 28 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such cases, each may be connected to the bus 18 by one or more data media interfaces. As further illustrated and described below, the memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present disclosure.

[0086] A program / utility 40 having a set (at least one) of program modules 42, including, by way of example and not limitation, an operating system, one or more application programs, other program modules, and program data, may be stored in memory 28. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. The program modules 42 generally perform the functions and / or methodologies of the embodiments described herein.

[0087] Computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, one or more devices that allow a user to interact with computer system / server 12, or any device (e.g., network card, modem, etc.) that allows computer system / server 12 to communicate with one or more other computing devices, or a combination thereof. Such communication may occur via an input / output (I / O) interface 22. Furthermore, computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via a network adapter 20. As shown, network adapter 20 communicates with other components of computer system / server 12 via a bus 18. Although not shown, it should be understood that other hardware and / or software components may be used with computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0088] The present disclosure may be embodied as a system, method, and / or computer program product, which may include a computer-readable storage medium (or multiple computer-readable storage media) having computer-readable program instructions for causing a processor to implement aspects of the present disclosure.

[0089] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction-execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, punch cards, or mechanically encoded devices such as raised structures in grooves with instructions recorded thereon, and any suitable combination of the above. As used herein, computer-readable storage media should not be construed as being ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through fiber optic cable), or electrical signals transmitted over electrical wires.

[0090] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0091] The computer-readable program instructions for carrying out the operations of the present disclosure can be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, C++, and traditional procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, to carry out aspects of the present disclosure, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry.

[0092] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0093] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.The computer-readable program instructions may also be stored on a computer-readable medium, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions implementing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams, and can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.

[0094] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to create a computer-implemented process that causes the computer, other programmable apparatus, or other device to perform a series of operational steps, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0095] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions or a combination of dedicated hardware and computer instructions.

[0096] Although the description of various embodiments of the present disclosure has been presented for illustrative purposes, this description is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond those found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A neural inference chip, comprising: a plurality of network nodes, the plurality of network nodes being organized into a grid of two or more dimensions having at least one row and at least one column; a network-on-chip, said network-on-chip interconnecting said plurality of network nodes, said network-on-chip network including at least one pair of directed paths, each said row and each said column having a pair of directed paths, each pair of said paths having opposite directions and a common termination; Equipped with the network is configured to receive data at any of the plurality of network nodes; the network is configured to propagate data along a first path from a source node of the pair of directed paths to the common termination of the pair of directed paths, and along a second path from the common termination of the pair of directed paths to one or more destination nodes of the pair of directed paths.

2. 2. The neural inference chip of claim 1, wherein the network is configured to deliver data to the one or more destination nodes on a second of the pair of directed paths according to a network address.

3. A neural inference chip as described in claim 1, wherein the common terminal is located at the end of the grid.

4. A neural inference chip as described in claim 1, wherein each network node of the plurality of network nodes comprises a router interconnecting a row network and a column of the network node.

5. 2. The neural inference chip of claim 1, wherein the two or more dimensions are ordered such that, for each dimension, data propagates along each of its pair of directed paths, followed by the subsequent dimension.

6. A neural inference chip as described in claim 5, wherein each network node of the plurality of network nodes comprises a router interconnecting the pair of directed paths in each dimension with the pair of directed paths in the subsequent dimension, and the router is adapted to distribute data from the second path of the pair of directed paths to the first path of the pair of directed paths in the subsequent dimension.

7. A neural inference chip as described in any one of claims 1 to 6, further comprising a router at the common end of each pair of directed paths of the plurality of network nodes, the router configured to receive data along a first path of the pair of directed paths and to transmit data along a second path of the pair of directed paths.

8. 8. The neural inference chip of claim 1, wherein the common termination of each pair is configurable at run time.

9. 9. The neural inference chip of claim 8, wherein the network is partitioned into a plurality of subpartitions, each subpartition operating as an independent network on a subset of the network nodes.

10. 10. The neural inference chip of claim 1, wherein the network is configured to receive one data packet per node per clock cycle and to propagate the data along each directed path to adjacent nodes per clock cycle.

11. 11. The neural inference chip of claim 10, wherein the network is configured to propagate data between adjacent network nodes every clock cycle without stopping or stalling.

12. The neural inference chip of claim 1 , wherein the network is configured to distribute data to all nodes on the second path by broadcasting.

13. The neural inference chip of claim 1 , wherein the network is configured to distribute data to a subset of nodes on the second path by multicasting.

14. The neural inference chip of claim 1 , wherein the network is configured to deliver data to a single node on the second path by unicast.

15. The neural inference chip of claim 1 further comprising an external interface adapted to receive and transmit data from sources and destinations other than the network node.

16. 16. The neural inference chip of claim 15, wherein a router of the network node or a router of the common termination is configured to route data towards the external interface.

17. Receiving data at a plurality of network nodes provided on a neural inference chip, the plurality of network nodes being organized in a two or more dimensional grid having at least one row and at least one column, the plurality of network nodes being interconnected by a network-on-chip provided on the neural inference chip, the network of the network-on-chip including at least one pair of directed paths, each said row and each said column having a pair of directed paths, each pair of said paths having opposite directions and a common termination; propagating data along a first path from a source node of the pair of directed paths to the common end of the pair of directed paths and along a second path from the common end of the pair of directed paths to one or more destination nodes of the pair of directed paths; A method comprising:

18. 20. The method of claim 17, further comprising propagating multiple adjacent data through the network-on-chip, wherein multiple source nodes inject data into the network in a given cycle.

19. 20. The method of claim 17, further comprising propagating a plurality of adjacent data through the network-on-chip, wherein a plurality of source nodes inject the data into the network such that the data is interleaved on the network.

20. 20. The method of claim 17, further comprising propagating multiple adjacent data through the network-on-chip, wherein multiple source nodes inject the data into different networks of a first dimension such that the data is subsequently interleaved on a network of a second dimension.

21. 20. The method of claim 17, wherein the infusion of data into the network from one or more source nodes proceeds according to a predetermined schedule.

22. 22. The method of claim 21, wherein the predetermined schedule ensures that no data collisions occur on the one or more networks.

Images