Neural network processing

Abstract

A collision-free, uninterrupted broadcast network is provided on a neural inference chip. A neural inference chip includes multiple network nodes and an on-chip network interconnecting the multiple network nodes. The network includes at least one directed path pair. The paths in each directed path pair have opposite directions and a common end. The network is configured to accept data at any of the multiple nodes. The network is configured to propagate data from a source node along a first directed path in the directed path pair to the common end of the directed path pair, and from the common end of the directed path pair along a second directed path in the directed path pair to one or more destination nodes.

Description

Background Technology

[0001] This invention generally relates to neural network processing. Summary of the Invention

[0002] According to one aspect of the invention, a neural inference chip includes a plurality of network nodes and an on-chip network interconnecting the plurality of network nodes. The network includes at least one pair of directed paths. Each pair of directed paths has opposite directions and a common end. The network is configured to accept data at any of the plurality of nodes. The network is configured to propagate data from a source node along a first directed path in the pair of directed paths to the common end of the pair of directed paths, and from the common end of the pair of directed paths to one or more destination nodes along a second directed path in the pair of directed paths.

[0003] According to other aspects, a method and computer program product for operating an on-chip network are provided. Data is received at multiple network nodes. The multiple network nodes are interconnected by the on-chip network. The network includes at least one directed path pair. Each directed path pair has paths in opposite directions and a common end. Data is propagated from a source node to the common end of the directed path pair along a first directed path in the directed path pair, and from the common end of the directed path pair along a second directed path in the directed path pair to one or more destination nodes. Attached Figure Description

[0004] Figure 1 A neural core according to an embodiment of the present disclosure is shown.

[0005] Figure 2 An exemplary inference processing unit (IPU) according to an embodiment of the present disclosure is shown.

[0006] Figure 3 A multi-core inference processing unit (IPU) according to an embodiment of the present disclosure is shown.

[0007] Figure 4 A neural core and associated network according to an embodiment of the present disclosure are shown.

[0008] Figure 5 This is a schematic diagram of a one-dimensional broadcast network on chip (NOC) according to an embodiment of the present disclosure.

[0009] Figures 6-7 This is a schematic diagram of an exemplary baseline of a one-dimensional broadcast network on chip (NOC) according to embodiments of the present disclosure.

[0010] Figures 8-10 This is a schematic diagram of an exemplary baseline of a one-dimensional broadcast network on-chip (NOC) employing the J-scheme according to embodiments of the present disclosure.

[0011] Figure 11 This is a schematic diagram of a two-dimensional broadcast network-on-a-chip (NOC) according to an embodiment of the present disclosure.

[0012] Figure 12 This is a schematic diagram of a two-dimensional broadcast network-on-chip (NOC) employing scheme J according to an embodiment of the present disclosure.

[0013] Figure 13 This is a schematic diagram providing an overview of a model Network on-Chip (NOC) scheme according to embodiments of the present disclosure.

[0014] Figure 14 This is a schematic diagram of a model network-on-a-chip (NOC) packet format according to an embodiment of the present disclosure.

[0015] Figure 15 This is a schematic diagram of a model network-on-chip (NOC) addressing configuration according to an embodiment of the present disclosure.

[0016] Figure 16 This is a schematic diagram of a model network-on-a-chip (NOC) one-dimensional router according to an embodiment of the present disclosure.

[0017] Figure 17 This is a schematic diagram of a model network-on-chip (NOC) two-dimensional router according to an embodiment of the present disclosure.

[0018] Figures 18A-18B This is a schematic diagram of a model network-on-chip (NOC) list of router paths according to an embodiment of the present disclosure.

[0019] Figure 19 This is a schematic diagram of the external interface of a model network on-chip (NOC) according to an embodiment of the present disclosure.

[0020] Figure 20 and Figures 21A-21B This is a schematic diagram of a logical partition of a model network-on-chip (NOC) array according to an embodiment of the present disclosure.

[0021] Figure 22 A method for calculating neural activation according to embodiments of the present disclosure is shown.

[0022] Figure 23 A computing node according to an embodiment of the present disclosure is described. Detailed Implementation

[0023] An artificial neuron is a mathematical function whose output is a linear combination of its inputs and which is a nonlinear function. Two neurons are connected if the output of one neuron is the input of another. Weights are scalar values ​​that encode the strength of the connection between the output of one neuron and the input of another.

[0024] A neuron computes its output, called activation, by applying a nonlinear activation function to a weighted sum of its inputs. The weighted sum is an intermediate result computed by multiplying each input by its corresponding weight and summing the products. A partial sum is a weighted sum of a subset of the inputs. The weighted sum of all inputs can be computed in stages by accumulating one or more partial sums.

[0025] A neural network is a collection of one or more neurons. Neural networks are typically divided into groups of neurons called layers. A layer is a collection of one or more neurons that all receive input from the same layer and all send output to the same layer, and typically perform similar functions. The input layer is the layer that receives input from sources outside the neural network. The output layer is the layer that sends output to a target outside the neural network. All other layers are intermediate processing layers. A multilayer neural network is a neural network with more than one layer. A deep neural network is a multilayer neural network with multiple layers.

[0026] A tensor is a multidimensional array of numerical values. A tensor block is a contiguous subarray of elements in a tensor.

[0027] Each neural network layer and its parameter tensor Weight tensors Input data tensors Output data tensors and intermediate data tensors Related. The parameter tensor contains the activation functions of the control neurons in the control layer. All parameters of the layer. The weight tensor contains all the weights that connect the input to the layer. The input data tensor contains all the data consumed by the layer as input. The output data tensor contains all the data that the layer computes as output. The intermediate data tensor contains any data produced by the layer as intermediate computations, such as partial sums.

[0028] The layer's data tensor (input, output, and intermediate values) can be 3-dimensional, where the first two dimensions can be interpreted as encoding spatial locations, and the third dimension can be interpreted as encoding different features. For example, when the data tensor represents a color image, the first two dimensions encode the vertical and horizontal coordinates within the image, and the third dimension encodes the color at each location. Input data tensor Each element can be connected to each neuron via a separate weight, therefore the weight tensor Typically, it has 6 dimensions, which are the 3 dimensions of the input data tensor (input rows). Input column Input features ) and the three dimensions of the output data tensor (output rows) Output column Output features (Connection. Intermediate data tensors) Having output data tensor Same shape. Parameter tensor The three output data tensor dimensions are associated with the index activation function. Additional dimensions of the parameters Connection. In some embodiments, the activation function No additional parameters are needed; in this case, additional dimensions are unnecessary. However, in some embodiments, the activation function... At least one additional parameter is required, which appears in the dimension. middle.

[0029] Layer output data tensor The elements can be calculated as in Equation 1, where the neuron activation function... By activation function parameters The vector configuration, and the weighted sum It can be calculated as shown in Equation 2.

[0030]

[0031] Equation 1

[0032]

[0033] Equation 2

[0034] To simplify the notation, the weighted sum in Equation 2 can be referred to as the output, which is equivalent to using a linear activation function. It should be understood that the same statement applies without loss of generality when different activation functions are used.

[0035] In various embodiments, the computation of the output data tensor, as described above, is decomposed into smaller problems. Each problem can then be solved in parallel on one or more neural cores, or on one or more cores of a conventional multi-core system.

[0036] As can be clearly seen from the above, neural networks are parallel structures. Neurons in a given layer receive element-wise signals from one or more layers. Input Or other inputs. Each neuron is based on input and has elements weight Calculate its state. In various embodiments, by biasing Adjust the weights of the inputs, and then pass the result to a nonlinear circuit. For example, the activation of a single neuron can be represented as .

[0037] Because all neurons in a given layer receive input from the same layer and compute their outputs independently, neuron activations can be computed in parallel. Due to this aspect of the overall neural network, performing computations in the parallel-distributed cores accelerates the overall computation. Furthermore, operations can be computed in parallel within each core vector. Even with recurrent inputs, such as when a layer projects back to itself, all neurons are still updated simultaneously. In effect, recurrent connections are delayed to align with subsequent inputs from that layer.

[0038] Now for reference Figure 1 The diagram depicts a neural core according to an embodiment of the present disclosure. The neural core 100 is a tileable computational unit that computes a block of output tensors. The neural core 100 has... Inputs and One output. In various embodiments, To compute the output tensor block, the neural core will... Input tensor block 101 multiplied by Weighted tensor block 102 is used, and the multiplication accumulation is added to form a weighted sum, which is stored in [the database]. In intermediate tensor block 103. The parameter tensor block contains One parameter, which specifies Each of the neuron activation functions is applied to intermediate tensor block 103 to generate 105 output tensor blocks.

[0039] Multiple neural nuclei can be laid flat in a neural core array. In some embodiments, the array is 2-dimensional.

[0040] A neural network model is a set of constants that collectively specify the entire computation performed by the neural network, including the connection graph between neurons and the weights and activation function parameters of each neuron. Training is the process of modifying the neural network model to perform the desired function. Inference is the process of applying the neural network to inputs to produce outputs without modifying the neural network model.

[0041] An inference processing unit is a type of processor that performs neural network inference. A neural inference chip is a specific physical instance of an inference processing unit.

[0042] refer to Figure 2An exemplary inference processing unit (IPU) is illustrated according to embodiments of the present disclosure. IPU 200 includes a memory 201 for a neural network model. As described above, the neural network model may include synaptic weights of the neural network to be computed. IPU 200 includes an activation memory 202, which may be transient. The activation memory 202 may be divided into input and output regions and stores neuronal activations for processing. IPU 200 includes a neural computation unit 203 loaded with the neural network model from the model memory 201. Input activations are provided from the activation memory 202 prior to each computation step. Outputs from the neural computation unit 203 are written back to the activation memory 202 for processing on the same or another neural computation unit.

[0043] In various embodiments, microengine 204 is included in IPU 200. In such embodiments, all operations in the IPU are directed by the microengine. As described below, central and / or distributed microengines may be provided in various embodiments. A global microengine may be referred to as a chip microengine, while a local microengine may be referred to as a core microengine or a local controller. In various embodiments, a microengine includes one or more microengines, microcontrollers, state machines, CPUs, or other controllers.

[0044] refer to Figure 3 The diagram illustrates a multi-core inference processing unit (IPU) according to an embodiment of the present disclosure. The IPU 300 includes a memory 301 for a neural network model and instructions. In some embodiments, the memory 301 is divided into a weight portion 311 and an instruction portion 312. As described above, the neural network model may include synaptic weights of the neural network to be computed. The IPU 300 includes an activation memory 302, which may be transient. The activation memory 302 may be divided into input and output regions and stores neuron activations for processing.

[0045] The IPU 300 includes an array 306 of neural cores 303. Each core 303 includes a computation unit 333 loaded with a neural network model from model memory 301 and operable to perform vector computations. Each core also includes a local activation memory 332. Input activation is provided from the local activation memory 332 before each computation step. Outputs from the computation unit 333 are written back to the activation memory 332 for processing on the same or another computation unit.

[0046] The IPU 300 includes one or more on-chip networks (NoCs) 305. In some embodiments, part numbers and NoCs 351 interconnect core 303 and transmit part numbers and instructions between them. In some embodiments, separate parameter distribution NoCs 352 connect core 303 to memory 301 for distributing weights and instructions to core 303. It will be understood that various configurations of NoCs 351 and 352 are suitable for use according to this disclosure. For example, broadcast networks, row broadcast networks, tree networks, and switched networks can be used.

[0047] In various embodiments, a global microengine 304 is included in the IPU 300. In various embodiments, a local core controller 334 is included on each core 303. In such embodiments, the global microengine (chip microengine) and the local core controller (core microengine) collaboratively guide operation. Specifically, at 361, computation instructions are loaded from the instruction portion 312 of the model memory 301 to the core controller 334 on each core 303 via the global microengine 304. At 362, parameters (e.g., neural network / synaptic weights) are loaded by the global microengine 304 from the weight portion 311 of the model memory 301 to the neural computation unit 333 on each core 303. At 363, neural network activation data is loaded from the activation local activation memory 332 to the neural computation unit 333 on each core 303 via the local core controller 334. As described above, activation is provided to neurons of a specific neural network defined by the model and may originate from the same or another neural computation unit, or from outside the system. At 364, the neural computation unit 333 performs the computation to generate output neuron activations as directed by the local core controller 334. Specifically, the computation includes applying input synaptic weights to the input activations. It should be understood that various methods can be used to perform this computation, including in-silico dendrite and vector multiplication units. At 365, the computation result is stored in the local activation memory 332 as instructed by the local core controller 334. As mentioned above, these stages can be pipelined to provide efficient use of the neural computation unit on each core. It should also be understood that, depending on the requirements of a given neural network, inputs and outputs can be transferred from the local activation memory 332 to the global activation memory 302.

[0048] Therefore, this disclosure provides runtime control of operations within an inference processing unit (IPU). In some embodiments, the microengine is centralized (a single microengine). In some embodiments, IPU computation is distributed (performed by a core array). In some embodiments, runtime control of operations is hierarchical—both the central microengine and distributed microengines are involved.

[0049] One or more microengines guide the execution of all operations within the IPU. Each microengine instruction corresponds to several sub-operations (e.g., address generation, loading, computation, storage, etc.), and the core microcode runs on a core microengine (e.g., 334). In the case of local computation, the core microcode includes instructions(s) that perform complete single tensor operations, such as convolution between weighted tensors and data tensors. In the case of distributed computation, the core microcode includes instructions(s) that perform single tensor operations on subsets of locally stored data tensors (and partial sums). Chip microcode runs on a chip microengine (e.g., 304). Microcode includes instructions that perform all tensor operations in a neural network.

[0050] Now for reference Figure 4 An exemplary neural core and associated network are illustrated according to embodiments of this disclosure. See references... Figure 1 The core 401, implemented in the described manner, is interconnected with additional cores via networks 402…404. In this embodiment, network 402 is responsible for distributing weights and / or instructions, network 403 is responsible for distributing portions, and network 404 is responsible for distributing activations. However, it will be understood that various embodiments of this disclosure may combine these networks or further separate them into multiple additional networks.

[0051] Input activation ( Layer instructions are distributed from off-core to core 401 via activation network 404 to activation memory 405. Layer instructions are distributed from off-core to core 401 via weight / instruction network 402 to instruction memory 406. Layer weights ( The parameters are distributed from outside the core to the core 401 via the weight / instruction network 402 to the weight memory 407 and / or the parameter memory 408.

[0052] The weight matrix is ​​read from the weight memory 407 by the vector matrix multiplication (VMM) unit 409. The activation vector is read from the activation memory 405 by the vector-matrix multiplication (VMM) unit 409. The Vector Matrix Multiplication (VMM) unit 409 then calculates the vector matrix multiplication. The results are then provided to vector-vector unit 410. Vector-vector unit 410 reads additional partial sums from partial sum memory 411 and receives additional partial sums from outside the core via partial sum network 403. Vector-vector unit 410 performs vector-vector operations from these source partial sums. For example, the partial sums can be summed sequentially. The resulting target partial sum is written to partial sum memory 411, sent outside the core via partial sum network 403, and / or fed back to vector-vector unit 410 for further processing.

[0053] After all computations for the input of a given layer are completed, the partial sums and results from vector-vector unit 410 are provided to activation unit 412 for the computation of output activation. Activation vector ( The layer activations (including the result of writing to the activation memory) are written to the activation memory 405. Layer activations (including the result of writing to the activation memory) are redistributed across the cores from the activation memory 405 via the activation network 404. Upon reception, they are written to the local activation memory of each receiving core. Upon completion of processing of a given frame, the output activation is read from the activation memory 405 and sent outside the core via the network 404.

[0054] Therefore, in operation, the core control microengine (e.g., 413) coordinates the core's data movement and computation. The microengine issues a read operation on the active memory address to load the input active block into the vector-matrix multiplication unit. The microengine issues a read operation on the weighted memory address to load the weighted block into the vector-matrix multiplication unit. The microengine issues computation operations to the vector-matrix multiplication unit, causing the unit to compute parts and blocks.

[0055] The microengine issues one or more of the following operations: partial and read / write memory address operations, vector computation operations, or partial and communication operations, in order to perform one or more of the following operations: reading partial and data from a partial and source; performing computations using the partial and arithmetic unit; or writing partial and data to a partial and target. Writing partial and data to a partial and target may include communicating with external cores via a partial and network interface, or sending partial and data to an active arithmetic unit.

[0056] The microengine issues an activation function calculation operation, causing the activation function computation unit to calculate and output an activation block. The microengine then issues a write activation memory address, and the output activation block is written to the activation memory through the activation memory interface.

[0057] Therefore, various sources, targets, address types, computation types, and control components are defined for a given core.

[0058] The sources of vector unit 410 include vector matrix multiplication (VMM) unit 409, constants from parameter memory 408, partial sum memory 411, partial sum results from previous cycles (TGT partial sum); and partial sum network 403.

[0059] The targets for vector unit 410 include partial sum memory 411, partial sum results for subsequent cycles (SRC partial sum), activation unit 412, and partial sum network 403.

[0060] Therefore, a given instruction can be read from or written to the activation memory 405, from the weight memory 407, or from the partial sum memory 411. The computational operations performed by the core include vector-matrix multiplication in the VMM unit 409, vector (partial sum) operations in the vector-vector unit 410, and activation functions in the activation unit 412.

[0061] Control operations include updating the program counter and the loop and / or sequence counter.

[0062] Therefore, memory operations are issued to read weights from addresses in weight memory, read parameters from addresses in parameter memory, read activations from addresses in activation memory, and read / write partial sums to addresses in partial sum memory. Computation operations are issued to perform vector-matrix multiplication, vector operations, and activation functions. Communication operations are issued to select vector-vector operands, route messages on partial sum networks, and select partial sum targets. Cross-layer loop outputs and cross-layer loop inputs are controlled by specifying control operations of the program counter, loop counter, and sequence counter in the microengine.

[0063] According to embodiments of the present invention, data distribution can be provided from a global memory array. The global memory array includes multiple elements, each including a memory element and a buffer. Weights and instructions are provided from the global array to the core array via a network. The above relates to... Figure 3 The core exemplary configuration was discussed. Figure 3 The memory 303 in the memory can be implemented as described in conjunction with the global memory array.

[0064] As mentioned above, many core architectures used for neural inference offer significant advantages in terms of computational power. However, if neural network weights and parameters are not provided to the computational cores in a timely manner, the cores cannot perform any useful computations. As a result, the performance of a neural chip can be limited by its ability to deliver neural network weights and parameters to the computational cores on the chip. On-chip memory significantly improves memory bandwidth compared to typical off-chip memory such as Dynamic Random Access Memory (DRAM) or High Bandwidth Memory (HBM). Furthermore, on-chip memory is more energy-efficient than off-chip memory, resulting in more energy-efficient neural inference systems. In various embodiments, on-chip memory may include Static Random Access Memory (SRAM) or other embedded memory. However, delivering neural network weights to the cores at a rate commensurate with processing speed remains a challenge.

[0065] Convolutional Neural Networks (CNNs) can achieve certain efficiencies. In CNNs, the same weight matrix (sometimes called convolutional filters) is reused. To minimize the amount of on-chip memory used, it is preferable to store a given weight matrix in one location without repetition. To store large neural networks, some implementations of on-chip memory consist of a collection of many memory elements. It will also be understood that many cores are targets for memory weights. This leads to a many-to-many communication problem (many memory elements to many cores). Broadcasting weights can cause on-chip network (NoC) congestion and can generate many collisions and pipeline stalls, resulting in a degradation of broadcast bandwidth.

[0066] As described above, in various embodiments of the neural inference chip, a grid of neural inference cores is provided to accelerate neural network inference. In various embodiments, instruction pre-scheduling is provided. Neural network evaluation involves rule-based computational patterns, so instructions can be pre-scheduled without any pauses to achieve high performance. However, it is preferable that all neural network weights are delivered to the core in a timely manner according to the pre-scheduling. If the weight delivery network becomes congested and weight delivery pauses, the pre-scheduling of the neural network evaluation fails.

[0067] This disclosure provides a stall-less weight transfer on-chip network for transferring weight parameters from a memory element grid to a computation core grid. A one-dimensional scheme is first shown below, followed by extension to a two-dimensional grid scheme. These methods are further extended to support various weight distributions, such as striping (where cores in different rows receive different weights).

[0068] Even with variations in the timing of instruction transfer to memory elements and data transfer from memory elements to the compute core, the methods described in this paper operate conflict-free. These methods address the instruction stream in any order. The new scheme eliminates the constraint that all columns must start simultaneously.

[0069] According to one aspect of this disclosure, collision-free and uninterrupted broadcasting is provided for Network-on-Chip (NoC) applications. In some embodiments, broadcast data is provided from multiple sources via the Network-on-Chip (NoC), wherein zero latency cycles on the network ensure that: data is not back-pressurized on the NoC; data is provided back-to-back (no empty slots on the NoC); and data remains collision-free.

[0070] Therefore, this disclosure provides many advantages, including:

[0071] - Merge multiple sources to achieve distributed memory storage;

[0072] - Precise, deterministic, scheduled operations;

[0073] - Required full throughput (no empty cycles, no data loss).

[0074] - Router design simplicity – a minimal design for very wide data paths; and

[0075] - Program simplicity (e.g., the same program at each node).

[0076] Now for reference Figure 5 An exemplary one-dimensional broadcast of an on-chip network is illustrated according to embodiments of this disclosure. As shown, a first source 501 (“Source 1”) broadcasts to multiple destinations on the bus at different times t = 0 and t = 1. Then, at times t = 2 and t = 3, a second source 510 (“Source 2”) broadcasts without overlapping (colliding) with the broadcast of the first source and without introducing any empty cycles into the network. In the exemplary embodiment shown, there are two broadcast sources and eight destinations (shown by vertical boxes or cells); however, those skilled in the art will understand that the data sources and destinations can be scaled up / down as needed. In the exemplary embodiment, the first source 501 is introduced at the midpoint (or fourth destination), and the second source 510 is introduced at a different location (e.g., a destination lower than the destination of the first source 501). Data will eventually reach all locations, as indicated by arrow 520.

[0077] Now for reference Figure 6 The diagram illustrates an exemplary baseline of a one-dimensional broadcast-on-chip network. In this example, a first source (“Source 1”) sends four consecutive data words (0:3). With each increment of time, the data words propagate vertically in opposite directions. As the data propagates, all destinations will sequentially receive all data words (0:3). At time t = 4, the first destination unit to which Source 1 introduced the data is cleared or emptied due to the propagation of the data. This data clearing continues until each data word (0:3) has propagated past all eight vertical destinations, as shown at time t = 8.

[0078] Figure 7This illustrates another exemplary baseline of a one-dimensional broadcast on-chip network, where empty cells and collisions occur due to a one-clock-cycle wait time. Here, once the first source (“Source 1”) transmits two consecutive data words (0:1) to the fourth cell 701, as shown in columns t=0 and t=1, the second source (“Source 2”) then transmits two consecutive data words (2:3) to the adjacent cell 720, as shown in columns t=2 and t=3. As previously stated, the data propagates vertically in opposite directions. Therefore, multiple cells are empty (as shown, from column t=2 to column t=5), which degrades efficiency and performance. Furthermore, multiple cells have collisions caused by the concurrent presence of data from both Source 1 and Source 2 (as shown, from column t=2 to column t=5, or until the data propagates to its destination), which further degrades efficiency and performance. This disclosure provides a novel method to avoid such empty and collapsing data streams.

[0079] Figure 8 An exemplary embodiment of the invention is described, in which a J-scheme data stream can be incorporated to avoid empty or conflicting data broadcasts. As shown, once source 1 sends four consecutive data words (0:3), the data propagates vertically in one direction. In this exemplary embodiment, the data flows north (N) (or upward) until it reaches the top cell, where the data is fed back for delivery in the opposite direction (S) (or downward), as indicated by the dashed arrow data stream. While the data is on the downward trajectory (i.e., starting at time t = 3), the data is delivered to the destination address.

[0080] Figure 9 Another exemplary embodiment of the invention is described, in which a J-scheme data stream can be incorporated to avoid empty or conflicting data broadcasts. Here, once source 1 sends two consecutive data words (0:1) at times t=0 and t=1, a second adjacent source 2 sends two consecutive data words (2:3) at times t=2 and t=3. The data words propagate vertically in one direction (N) and are fed back to be transmitted in the opposite direction (S), wherein source 1 data begins to be fed back at time t=3 (i.e., reverses the propagation direction), and source 2 data begins to be fed back at time t=5 (i.e., reverses the propagation direction). As shown, there are neither conflicts nor empty data units / addresses between the inputs of source 1 and source 2, and this remains true throughout the propagation path.

[0081] Figure 10Another exemplary embodiment of the invention is described, in which the J-scheme data stream can be incorporated to avoid empty or conflicting data broadcasts. Here, once source 1 sends two consecutive data words (0:1) at times t=0 and t=1, the second adjacent source 2 sends two consecutive data words (2:3) at times t=3 and t=4. Note that source 2 does not transmit at time t=2. Therefore, source 2 sends its initial data (at time t=3) at a higher position (i.e., the second unit from the top) to avoid empty units. Once transmitted, the data words are again propagated vertically in one direction (N) and are fed back to be delivered in the opposite direction (S), as previously described.

[0082] According to one aspect of the invention, for Figures 8-10 In each example depicted, the data propagation at time t = 8 is the same, and therefore the delivery pattern to the eight destinations is the same in all three J scheme examples shown.

[0083] Figure 11 An exemplary embodiment according to this disclosure is depicted, wherein a J-scheme data stream can be incorporated into a two-dimensional broadcast on-chip network. When cores (0:3) begin transmission, the data begins to travel upwards, or in the N direction, as shown by the dashed line at time t = 1. At time t = 2, after the data reaches its upper limit, the data reverses direction according to the J-scheme disclosed herein, as shown in Figure 1101. Thus, at time t = 3, the transmission of cores (0:3) resides in the first two rows, where data from cores 1 and 2 coexists with the first row, and data from cores 0 and 3 coexists with the second row.

[0084] At time t = 4, the second source begins transmitting data from core (4:7), with the data propagating in a similar J-scheme, as shown by the solid line. At time t = 5, the data transmission from the second core begins to change direction and the direction reverses from N to S. Therefore, at time t = 6, the data from the first core (0:3) has completely changed direction to propagate entirely in the S direction, with the first transmission 0 already delivered to the output; simultaneously, the data from the second core (4:7) is in the first two rows, the data from cores 5 and 6 are in the first row, and the data from cores 4 and 7 are in the second row, as shown in the figure.

[0085] At time t = 7, the third source begins transmitting data from cores (8:11), where the data propagates in a similar J-scheme, as shown by the dashed line. At this point, half of the data from the first core (0:3) has already been delivered to the output, leaving only data from cores 2 and 3 still propagating in the S-direction. Data transmission from the second source (i.e., cores 4:7) has almost the exact opposite direction, with data from core 7 at the top and about to turn to the S-direction in the next clock cycle. None of the data from the third source (i.e., cores 8:11) has already turned.

[0086] According to one aspect of the invention, the J-scheme data propagation process described above for the first (vertical) dimension can also be used in the second (horizontal) dimension. This horizontal dimension is shown in label 1102, where data initially propagates in the W (or left) direction and reverses to the E (or right) direction upon reaching the W / left boundary, as indicated by the dashed arrow.

[0087] refer to Figure 11 The lower core described in the diagram, at time t = 3, shows the first transmission of data from core (0:3) in the first row, where 0 data has been reversed to the E direction, and a second transmission of data from the same core (0:3) can begin in the second row, where no data has been reversed. This can continue to each row as the multicast data from core (0:3) is transmitted, as shown at time t = 4. Next, data from the second core source (4:7) following the same W to E J scheme can begin simultaneous transmission, as shown at time t = 5. In this exemplary two-dimensional embodiment, data first propagates in the vertical dimension (N to S) and then in the horizontal dimension (W to E), such that when data arrives from the vertical dimension, it is fed into the horizontal dimension. As shown at times t = 6 and t = 7, the simultaneous broadcasting from multiple sources ensures that no empty or conflicting data streams are used for this two-dimensional broadcast.

[0088] Figure 12 A timing diagram of an exemplary two-dimensional J-scheme according to this disclosure is depicted. As shown, cores within a column transmit data simultaneously, with each column transmitting its data in an interleaved order, such that the transmission of core 3 (in column 0 of north row 0) overlaps with the data of core 4 (in column 1 of north row 0), but is presented on different buses. Here, dashed line 1201 indicates a cycle offset in timing. According to one aspect of this disclosure, two different data streams can be distributed in parallel by using sources from two columns. For example, source 1 from column 1 can be delivered to even-numbered rows, while source 2 from column 2 can be delivered to odd-numbered rows (without collisions). This feature can be generalized to make it correct for N data streams. The advantage of this approach is that it increases the effective bandwidth for distributing unique data.

[0089] According to one aspect of this disclosure, a Model-on-Chip (MNoC) architecture definition may include: a J-scheme overview; routing rules (operations); packet format with field (signal) definitions; addressing with direct addressing and distribution flags; router architecture; router paths (list); read / write transactions; external interfaces; and latency / skew analysis. Figures 5-12 The textual representation of the protocol shown can be represented as follows:

[0090] "J" scheme protocol pseudocode:

[0091] - Northward to the edge of the core array

[0092] -Southward circulation

[0093] - Proceed south to the edge of the core array

[0094] - When moving south, for each row, if it matches the Y distribution flag (or the row address in direct addressing mode), then the row is delivered.

[0095] - To the west to the edge of the core array

[0096] -Eastward circulation

[0097] - Eastward to the edge of the core array

[0098] - When moving eastward, for each column, if it matches the X distribution flag (or the column address in direct addressing mode), it is passed to the core.

[0099] Figure 13 An overview of an exemplary model on-chip network J scheme according to this disclosure is described. In some embodiments, the YX-dimensional routes are ordered such that data propagates first in the Y dimension and then in the X dimension. Therefore, data packets follow the order of the two "J"-shaped routes:

[0100] I. First route: North then South (as shown by the arrow in the solid J scheme) 1301.

[0101] II. Second route: West then East (as shown by the arrow in the dashed J scheme) 1302.

[0102] The “J” route propagates data along its path to the end of the array, where it is then looped back. On the second segment of the route (after loopback), data traverses the entire length of the array (in multicast addressing mode). Data is delivered on the second segment of the route (after its loopback). Furthermore, two addressing modes can be included:

[0103] I. Multicast addressing, which delivers data to all matching destinations; and

[0104] II. Direct addressing, which uses core row and column indexes to pass data to a matching destination.

[0105] According to one aspect of this disclosure, the on-chip network of the model may include routing rules. For example, rules may be implemented such that, within the Y or X dimension, a group always traverses one direction before the other:

[0106] -Y dimension: South then North

[0107] -X dimension: West then East

[0108] Additionally, rules can be implemented to cycle through the array packets from north to south (or west to east) at the north (or west) end. Furthermore, rules can be implemented to discard packets at the south (or east) end of the array. Additionally, rules regarding packet injection can be implemented such that:

[0109] Within the array, packets are always injected in the north direction.

[0110] - Special boundary cases for injecting data into the array from outside the array.

[0111] - In addition, rules regarding packet delivery can be implemented so that when addresses match:

[0112] - Delivering eastward-bound groups to the core

[0113] - Passing southward groups from west to east

[0114] - Distribution flag (multicast) mode: Packets are delivered and continue to propagate in the same direction.

[0115] - Direct addressing mode: Packets are delivered and do not continue to propagate in the same direction.

[0116] Figure 14 An exemplary grouping format according to this disclosure is depicted. The example shown is for grouping in a 16×16 core array. In some embodiments, “ext_in_” and “ext_out_transaction” are mutually exclusive, and they may not both be “1” at the same time. Furthermore, some exemplary transaction definitions for this configuration are as follows:

[0117] ~ADDR_VALID&~DATA_VALID = No transactions (idle)

[0118] ~ADDR_VALID& DATA_VALID = Invalid state (error)

[0119] ADDR_VALID&~DATA_VALID = Read transaction

[0120] ADDR_VALID & DATA_VALID = Write transaction

[0121] Figure 15 On-chip network addressing according to an exemplary model of this disclosure is depicted. Direct addressing (north / south direction) is shown at 1501. Direct addressing (east / west direction) is shown at 1502, where CORE_ROW_ID and CORE_COL_ID are determined by the core's position in the core array. Furthermore, the packet is delivered to the unique core that matches either the 4-bit DISTR_ROW_ID or DISTR_COL_ID in the packet. Direct flag (multicast) addressing is shown at 1503, where DISTR_FLAGS is compared with DEST_FLAGS according to a Boolean matching equation. Furthermore, DEST_FLAGS is set for each core in the core control register. Additionally, the data packet is delivered to each matching core by comparing the DISTR_FLAGS in the data packet with the DEST_FLAGS set in the core according to a Boolean matching equation. Some exemplary Boolean matching equations used in NoC addressing may include:

[0122] - ANY match=OR_REDUCE(DISTR_FLAGS&DEST_FLAGS)

[0123] - ALL match=~OR_REDUCE(DISTR_FLAGS&~DEST_FLAGS)

[0124] -=AND_REDUCE(~DISTR_FLAGS|DEST_FLAGS) / / Alternative DeMorgan equivalent specification

[0125] - NOT ANY match = ~ OR_REDUCE (DISTR_FLAGS&DEST_FLAGS)

[0126] - NOT ALL match = OR_REDUCE (DISTR_FLAGS&~DEST_FLAGS)

[0127] - Where "~" is the inverse, "&" is the bitwise AND operation, and "|" is the bitwise OR operation.

[0128] Figure 16An exemplary one-dimensional router for a model on-chip network according to this disclosure is depicted. In this exemplary model, register values ​​are updated each cycle such that old data from previous cycles is not retained. This timing causes each register to introduce a 1-clock-cycle delay, and each endpoint also contains 1 register. Here, data propagation is in the vertical dimension (as indicated by the arrows “RX_North”, “RX_South”, and “TX_North”, “TX_South”). “RX_South” data is fed up to a first dynamic merge 1601 and its corresponding register 1611, and can be output as “RX_North”; “RX_North” data is fed down to a second dynamic merge 1602 and its corresponding register 1612 and / or Y endpoint 1622, and can be output as “TX_South”.

[0129] Figure 17 An exemplary two-dimensional router for a model on-chip network according to this disclosure is depicted. In this exemplary model, register values ​​are updated each cycle such that old data from previous cycles is not retained. This timing causes each register to introduce a 1-clock-cycle delay, and each endpoint also contains 1 register. Here, data propagation occurs in the vertical dimension (as indicated by the arrows “RX_North”, “RX_South”, and “TX_North”, “TX_South”) and in the horizontal dimension (as indicated by the arrows “RX_West”, “RX_East”, and “TX_West”, “TX_East”). “Rx_South” data is fed north (or upward) to a first dynamic merge 1701 and the corresponding register 1711, and can be output as “TX_North”; “RX_North” data is fed south (or downward) to a second dynamic merge 1702 and the corresponding register 1712 and / or Y endpoint 1722; when passing through register 1712, the output can be “TX_South”. It is at Y endpoint 1722 that the delivery from the vertical dimension to the horizontal dimension occurs. Similarly, “RX_East” can be fed west (or left) via the third dynamic merge 1703 and output as “TX_West”. “RX_West” can be fed east (or right) via the fourth dynamic merge 1704 and output as “TX_East” and / or X endpoint 1723.

[0130] According to another aspect of this disclosure, dynamic splitting may be included for managing the operation of the router. For illustrative and non-limiting purposes, an exemplary dynamic splitting may be implemented as follows:

[0131] Split N

[0132] if (LoopbackSelNorth)

[0133] 将数据包环回至S总线

[0134] 否则

[0135] 将数据包直通至N总线

[0136] 结束

[0137] Split W

[0138] 如果(~ext_out_transaction)且(LoopbackSelWest)

[0139] 将数据包环回至E总线

[0140] 否则

[0141] 将数据包直通至W总线

[0142] 结束

[0143] Split L

[0144] 如果(~ext_out_transaction)且

[0145] (AddrModeY == Direct)且(TargetRowld == localCoreRowld)且

[0146] (AddrModeX == Direct)且(TargetColld == localCoreColld)

[0147] 将数据包环回至本地总线

[0148] 否则

[0149] 注入到MNoC(北向)

[0150] 结束

[0151] Additionally, an exemplary loopback definition can be as follows:

[0152] LoopbackSelNorth=(ROWID == 0) || (MNOC_Enable_N_Loopback == 1)

[0153] LoopbackSelWest =(CollD == 0)||(MNOC_Enable_W_Loopback == 1)

[0154] Figure 17 The Y endpoint 1722 and X endpoint 1723 shown can implement address matching logic. In some embodiments, when only in direct addressing mode, if the packet's direct address matches the core row / column ID, the packet is delivered and does not continue propagating in its original direction. Dynamic merging 1701-1704 transfers valid data packets from either input. Data packets must be scheduled to avoid collisions (two simultaneously valid packets input into dynamic merging). In the event of a collision, the ERROR flag is asserted, and one of the data packets is discarded as follows:

[0155] Northbound Dynamic Merge 1: The northern boundary (RX_South) has higher priority than the local boundary (i.e., local data is discarded).

[0156] Southbound Dynamic Merge 2: The southern boundary (RX_North) has a higher priority than LoopbackN (i.e., LoopbackN data is discarded).

[0157] Westward Dynamic Merging 3: The western boundary (RX_East) has higher priority than the southward boundary (i.e., southward data is discarded).

[0158] Eastward Dynamic Merge 4: The eastern boundary (RX_West) has a higher priority than LoopbackW (i.e., LoopbackW data is discarded).

[0159] Figures 18A-18B An exemplary list of router paths for a model on-chip network is described according to this disclosure, where "cycle" = cycle within the router of the model on-chip network. This exemplary list of router paths is consistent with... Figure 17As illustrated in the exemplary embodiment, when a Write or Read request is sent from the external interface to the core array, the external control interface injects a data packet into the south path (RX_North) of the core (0,0). This can be achieved by setting the "ext_in_transaction" bit = 1, with the data packet following the normal J scheme, heading south. This can also be achieved by setting the Y and X addresses to normal to reach the destination. When a Write or Read request is sent from the core array to the external interface (e.g., returning read data), the core uses the local core I / F (interface) to inject the data into the north boundary path. This can be achieved by setting the "ext_out_transaction" bit = 1. The data propagates north to the edge of the core array, back south, and is delivered to row 0. This can be implemented by setting the Y address to direct address mode, row 0; similarly, by setting the X address to any desired value, the data can propagate west to the edge of the core boundary. Data can be delivered to the external control interface, where the "ext_out_transaction" bit sends the data out of the core array instead of looping it back to the east of the core (0,0). Furthermore, in some scenarios, "ext_out_transactions" from core 0,0 may conflict with "ext_in_transactions", such as during burst reads. Therefore, W bypass in the core 0,0 router will cause "ext_out_transactions" from all other cores to overlap with "ext_in_transactions", thus avoiding conflicts.

[0160] Figure 19 An exemplary external interface for a model on-chip network according to this disclosure is depicted. In this exemplary embodiment, a core (0,0) configuration can be set such that partition N 1930 and partition 1931 was set to loopback. For data transmitted from the external interface:

[0161] - Set "ext_in_transaction"='1'

[0162] - Dynamic merge 2 1902 will transmit external data packets via the northern boundary path, or send data packets back from the northern boundary path.

[0163] - Division The ext_in_transactions will be sent back to the eastern boundary path.

[0164] For data transmitted to external interfaces:

[0165] - If "ext_out_transaction" = '1', then the data packet will output TX_West to the external interface, and will be partitioned. Send NOT back.

[0166] Although this exemplary embodiment describes data as being transmitted from the west side of the core, alternative configurations can be constructed if needed.

[0167] Figure 20 An exemplary array logical partitioning for a model on-chip network according to this disclosure is depicted. This partitioning is indicated by intersecting lines P1 and P2 in the middle of the core array, such that data propagates within any given subarray in the J-scheme flow mode until the edge of that subarray (rather than the edge of the entire core) before being looped back. In this exemplary embodiment, array logical partitioning is implemented using link disable identified at 2050 and loopback signals at partition boundaries. For routing protocol purposes, each logical subarray should operate as if it were an independent full array. Therefore, each subarray J-scheme is independent, and thus, they can have different modes. A link disable definition can be implemented such that each directed interface {N, S, E, W} has a disable signal for partitioning the array. Furthermore, Figures 21A-21B Exemplary link operations based on link disable and loopback settings for data transactions and external transactions are listed. When "zero" is set, all signals (Addr_valid, Addr, Data_valid, Data, etc.) are set to zero; however, "ext_in_transactions" are still injected into the top left and can reach any core in the array. Despite Figure 20 The logical subarray 2025 shown is depicted as a square comprising four cores, but alternative sizes / shapes of subarrays (e.g., rectangles) can be implemented if needed.

[0168] refer to Figure 22 The diagram illustrates a method for computing neural activation. At 2201, data is received at multiple network nodes. These network nodes are interconnected by an on-chip network. The network includes at least one pair of directed paths. Each pair of directed paths has paths with opposite directions and a common endpoint. At 2202, data is propagated from the source node along a first directed path in the pair to the common endpoint of the pair, and from the common endpoint of the pair along a second directed path to one or more destination nodes.

[0169] Various embodiments of this disclosure use instruction buffers, horizontal buffers, vertical buffers, and combinations of buffers to provide instruction and data distribution in one-dimensional or two-dimensional memory arrays. It should be understood that the invention can be applied to higher-dimensional arrays with added additional buffers. In these embodiments, the time from instruction issuance to data output from the data array is constant, even though each stage may take different amounts of time. Columns can be accessed in a random order. In the case of higher dimensions, two instructions accessing the same column should be separated by vertical distribution time. In the case of one dimension, the vertical distribution time is zero, and therefore there is no constraint.

[0170] In various embodiments, a system is provided that includes a memory array, an instruction buffer, and a horizontal data buffer. The sum of the number of cycles used for instruction allocation and memory allocation is constant for all instructions.

[0171] In various embodiments, a two-dimensional memory array is provided. A horizontal buffer is provided for each row of the memory array. A vertical buffer is provided for each column of the memory array. The sum of the number of cycles for instruction distribution, the number of cycles for data distribution along the vertical buffer, and the number of cycles for data distribution along the horizontal buffer is constant.

[0172] In various embodiments, a two-dimensional memory array is provided. A shelving buffer is provided for each location in the memory array. The sum of the number of cycles for instruction distribution, the number of cycles for data distribution along the vertical buffer, the number of cycles for data distribution along the horizontal buffer, and the number of cycles for data transfer over the covering buffer is constant.

[0173] Now for reference Figure 23 The diagram illustrates an example of a computing node. Computing node 10 is merely one example of a suitable computing node and is not intended to impose any limitation on the scope of use or functionality of the embodiments described herein. In any case, computing node 10 is capable of implementing and / or performing any of the functions set forth above.

[0174] Within compute node 10, there is a computer system / server 12 that can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the aforementioned systems or devices.

[0175] Computer system / server 12 can be described in the general context of computer system executable instructions, such as program modules executed by the computer system. Typically, program modules can include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer system / server 12 can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside in local and remote computer system storage media, including memory storage devices.

[0176] like Figure 23 As shown, the computer system / server 12 in compute node 10 is illustrated in the form of a general-purpose computing device. Components of the computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that couples various system components, including system memory 28, to the processor 16.

[0177] Bus 18 represents one or more of several types of bus architectures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of the various bus architectures. By way of example and not limitation, these architectures include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, the Peripheral Component Interconnect (PCI) bus, the Peripheral Component Interconnect Fast (PCIe) bus, and the Advanced Microcontroller Bus Architecture (AMBA).

[0178] In various embodiments, one or more inference processing units (not shown) are coupled to bus 18. In such embodiments, the IPU can receive data from or write data to memory 28 via bus 18. Similarly, the IPU can interact with other components via bus 18 as described herein.

[0179] Computer system / server 12 typically includes various computer system readable media. Such media can be any available media accessible to computer system / server 12, and it includes volatile and non-volatile media, removable and non-removable media.

[0180] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer system / server 12 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown, and generally referred to as "hard disk drives"). Although not shown, disk drives for reading from and writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks such as CD-ROMs, DVD-ROMs, or other optical media may be provided. In this case, each may be connected to bus 18 via one or more data media interfaces. As will be further described below, memory 28 may contain at least one program product having a set (e.g., at least one) of program modules configured to perform embodiments of the invention.

[0181] A program / utility 40 having a set (at least one) of program modules 42, along with an operating system, one or more applications, other program modules, and program data, may be stored in memory 28 as an example and not as a limitation. Each of the operating system, one or more applications, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. Program modules 42 typically perform the functions and / or methods of the embodiments described herein.

[0182] The computer system / server 12 can also communicate with one or more external devices 14, such as a keyboard, indicating device, display 24, etc.; one or more devices that enable a user to interact with the computer system / server 12; and / or any device that enables the computer system / server 12 to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). This communication can occur via input / output (I / O) interface 22; however, the computer system / server 12 can communicate with one or more networks via network adapter 20, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet). As shown, network adapter 20 communicates with other components of the computer system / server 12 via bus 18. It should be understood that, although not shown, other hardware and / or software components can be used in conjunction with the computer system / server 12, examples including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0183] This disclosure can be implemented as a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of this disclosure.

[0184] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0185] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0186] Computer-readable program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Smalltalk, C++, etc.) and conventional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this disclosure.

[0187] This document describes aspects of the disclosure with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0188] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0189] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0190] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions mentioned in the blocks may occur in a non-linear order as shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0191] Various embodiments of this disclosure have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A neural inference chip, comprising: Multiple network nodes, the multiple network nodes being organized in a grid with at least one row and at least one column in two or more dimensions; An on-chip network interconnects the plurality of network nodes. The network includes directed path pairs for each row and directed path pairs for each column. The paths in each directed path pair have opposite directions and a common end. The common end of each directed path pair is located at the end of the grid. The network is configured to accept data at any of the multiple nodes. The network is configured to propagate the data from the source node to the common end of the directed path pair along a first directed path in the directed path pair, the first directed path in the directed path pair traversing a subset of the plurality of network nodes, and The on-chip network is configured to, in response to the data being propagated to the common end along a first directed path in the directed path pair, propagate the data from the common end of the directed path pair to one or more destination nodes along a second directed path in the directed path pair, the second directed path in the directed path pair traversing the subset of the plurality of network nodes.

2. The neural inference chip of claim 1, wherein the network is configured to transmit data to the one or more destination nodes on a second directed path in the directed path pair according to a network address.

3. The neural inference chip of claim 1, wherein each network node includes a router that interconnects the row networks and columns of the network nodes.

4. The neural inference chip of claim 1, wherein the two or more dimensions are ordered such that, for each dimension, data propagates along each directed path in a pair of directed paths of that dimension, followed by subsequent dimensions.

5. The neural inference chip of claim 4, wherein each network node includes a router that interconnects the directed path pairs of each dimension with the directed path pairs of the subsequent dimensions, the router being adapted to pass data from the second directed path in the directed path pair to the first directed path in the directed path pair of the subsequent dimensions.

6. The neural inference chip according to any one of claims 1 to 5, further comprising a router located at the common end of each directed path pair, the router being configured to receive data along a first directed path in the directed path pair and to transmit data along a second directed path in the directed path pair.

7. The neural inference chip according to any one of claims 1 to 5, wherein the common terminal of each directed path pair is configurable at runtime.

8. The neural inference chip of claim 7, wherein the network is divided into multiple sub-parts, each sub-part operating as an independent network on a subset of the network nodes.

9. The neural inference chip according to any one of claims 1 to 5, wherein the network is configured to receive one data packet per node per clock cycle and to propagate data to adjacent nodes along each directed path per clock cycle.

10. The neural inference chip of claim 9, wherein the network is configured to propagate data between adjacent network nodes in each clock cycle without stopping or pausing.

11. The neural inference chip of claim 1, wherein the network is configured to transmit data to all nodes on the second directed path via broadcast.

12. The neural inference chip of claim 1, wherein the network is configured to transmit data to a subset of nodes on the second directed path via multicast.

13. The neural inference chip of claim 1, wherein the network is configured to transmit data to a single node on the second directed path via unicast.

14. The neural inference chip of claim 1 further includes an external interface adapted to send and receive data from sources and destinations other than the network node.

15. The neural inference chip of claim 14, wherein the node router or public router is configured to route data bound to the external interface.

16. A method comprising: Data is received at multiple network nodes, which are organized in a grid with at least one row and at least one column in two or more dimensions, and the multiple network nodes are interconnected by an on-chip network, the network including directed path pairs for each row and directed path pairs for each column, the paths in each directed path pair having opposite directions and a common end, the common end of each directed path pair being located at the end of the grid; Data is propagated from the source node to the common end of the directed path pair along the first directed path in the directed path pair, the first directed path in the directed path pair passing through a subset of the plurality of network nodes; as well as In response to the data being propagated to the common end along the first directed path in the directed path pair, the data is propagated from the common end of the directed path pair to one or more destination nodes along the second directed path in the directed path pair, the second directed path in the directed path pair traversing the subset of the plurality of network nodes.

17. The method of claim 16, further comprising: Multiple adjacent data segments are propagated through the on-chip network, wherein multiple source nodes inject data into the network in a given period.

18. The method of claim 16, further comprising: Multiple adjacent data segments are propagated through the on-chip network, wherein multiple source nodes inject data into the network, causing the data to be interleaved on the network.

19. The method of claim 16, further comprising: Multiple adjacent data segments are propagated through the on-chip network, where multiple source nodes inject data into different networks in the first dimension, so that the data is subsequently interleaved on the network in the second dimension.

20. The method according to any one of claims 17 to 19, wherein the injection of data from one or more source nodes to the network is performed according to a predetermined schedule.

21. The method of claim 20, wherein the predetermined scheduling ensures that the data does not conflict on the one or more networks.

Images