Parallel computing architecture with reconfigurable core-level and vector-level parallelism

By employing a reconfigurable parallel computing architecture with kernel-level and vector-level parallelism and data reshuffling technology, the problem of low computational efficiency in neural networks in existing technologies is solved, and efficient parallel computing of neural networks is achieved.

CN112219209BActive Publication Date: 2024-11-12INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980036802.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-06-14
Filing Date
2019-06-05
Publication Date
2024-11-12
Estimated Expiration
2039-06-05

AI Technical Summary

Technical Problem

Existing parallel computing architectures struggle to effectively utilize the multi-dimensional parallelism in neural networks, resulting in low computational efficiency. In particular, they fail to fully leverage the parallelism in data and algorithms across different layers of neural networks.

Method used

By providing a parallel computing architecture with reconfigurable kernel-level and vector-level parallelism, the parallelism of kernel and vector units is dynamically adjusted. Combined with data reshuffling technology, it ensures that almost all kernel and vector units are in a computing state at any given time.

Benefits of technology

It achieves efficient utilization of neural network computation, improves computational efficiency, ensures maximum utilization of kernel and vector units, and adapts to the parallelism changes of different levels of neural networks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112219209B_ABST
    Figure CN112219209B_ABST
Patent Text Reader

Abstract

Neural network processing hardware using a parallel computing architecture with reconfigurable core-level parallelism and vector-level parallelism is provided. In various embodiments, a neural network model memory is adapted to store a neural network model comprising multiple layers. Each layer has at least one dimension and comprises multiple synaptic weights. A plurality of neural cores are provided. Each neural core comprises a computation unit and an activation memory. The computation unit is adapted to apply the multiple synaptic weights to multiple input activations to generate multiple output activations. The computation unit has multiple vector units. The activation memory is adapted to store input activations and output activations. The system is adapted to divide the multiple cores into a plurality of partitions based on the dimension of the layer and the vector unit.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Embodiments of the present disclosure relate to neural network processing hardware, and more particularly, to a parallel computing architecture with reconfigurable core-level and vector-level parallelism. Summary of the invention

[0002] According to various embodiments of the present disclosure, a parallel computing architecture is provided. A neural network model memory is suitable for storing a neural network model including multiple layers. Each layer has at least one dimension and includes multiple synaptic weights. A plurality of neural cores are provided. Each neural core includes a computing unit and an activation memory. The computing unit is suitable for applying multiple synaptic weights to multiple input activations to generate multiple output activations. The computing unit has multiple vector units. The activation memory is suitable for storing input activations and output activations. The system is suitable for dividing the multiple cores into multiple partitions based on the dimension and vector unit of the layer.

[0003] According to an embodiment of the present disclosure, a method and a computer program product for configuring an inference processing unit (IPU) are provided. A neural network model including multiple layers is read. Each layer has at least one dimension and includes multiple synaptic weights. For each layer of the neural network model: multiple cores are divided into multiple partitions based on the dimensions of the layer and the vector unit; the multiple cores are configured to implement the layer; the input activations of the layer are provided to the multiple cores; and the synaptic weights associated with the layer are applied to the input activations to generate multiple output activations.

[0004] Figure 1A -D shows an exemplary parallel configuration for a given network on a single multi-core processor.

[0005] Figure 2A A multi-core inference processing unit (IPU) according to an embodiment of the present disclosure is shown.

[0006] Figure 2B A core computing element within a core of an inference processing unit (IPU) according to an embodiment of the present disclosure is shown.

[0007] Figure 3 An exemplary tensor mapping according to an embodiment of the present disclosure is shown.

[0008] Figure 4 Another exemplary tensor mapping according to an embodiment of the present disclosure is shown

[0009] Figure 5 Another exemplary tensor mapping according to an embodiment of the present disclosure is shown.

[0010] Figure 6 Another exemplary tensor mapping according to an embodiment of the present disclosure is shown.

[0011] Figure 7 Another exemplary tensor mapping according to an embodiment of the present disclosure is shown.

[0012] Figure 8 Another exemplary tensor mapping according to an embodiment of the present disclosure is shown.

[0013] Fig. 9 Another exemplary tensor mapping according to an embodiment of the present disclosure is shown.

[0014] Fig.10 Partial sums passed between cores according to an embodiment of the present disclosure are shown.

[0015] Figures 11A-11B Parameter distribution according to an embodiment of the present disclosure is shown.

[0016] Fig.12 is a pipeline timing diagram illustrating core operation according to an embodiment of the present disclosure.

[0017] Fig.13 is a pipeline timing diagram illustrating data reshuffling according to an embodiment of the present disclosure.

[0018] Fig.14 A shuffling operation according to an embodiment of the present disclosure is shown.

[0019] Fig.15 Another shuffle operation according to an embodiment of the present disclosure is illustrated.

[0020] Fig.16 Another shuffle operation according to an embodiment of the present disclosure is illustrated.

[0021] Fig.17 The reshuffling of data for parallel computing utilization according to the present disclosure is shown.

[0022] Fig.18 A method of configuring an inference processing unit (IPU) according to an embodiment of the present disclosure is shown.

[0023] Fig.19 A computing node according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] An artificial neuron is a mathematical function whose output is a nonlinear function of a linear combination of its inputs. Two neurons are connected if the output of one neuron is the input of another neuron. A weight is a scalar value that encodes the strength of the connection between the output of one neuron and the input of another neuron.

[0025] A neuron computes its output, called an activation, by applying a nonlinear activation function to a weighted sum of its inputs. A weighted sum is an intermediate result computed by multiplying each input by its corresponding weight and accumulating the products. A partial sum is a weighted sum of a subset of the inputs. The weighted sum of all inputs can be computed in stages by accumulating one or more partial sums.

[0026] A neural network is a collection of one or more neurons. Neural networks are often divided into groups of neurons called layers. A layer is a collection of one or more neurons that all receive input from the same layer and all send output to the same layer, and generally perform similar functions. An input layer is a layer that receives input from a source external to the neural network. An output layer is a layer that sends output to a target external to the neural network. All other layers are intermediate processing layers. A multi-layer neural network is a neural network with more than one layer. A deep neural network is a multi-layer neural network with multiple layers.

[0027] A tensor is a multidimensional array of values. A tensor block is a contiguous subarray of elements in a tensor.

[0028] Each neural network layer is associated with a parameter tensor V, a weight tensor W, an input data tensor X, an output data tensor Y, and an intermediate data tensor Z. The parameter tensor contains all the parameters that control the activation functions σ of the neurons in the layer. The weight tensor contains all the weights that connect the inputs to the layer. The input data tensor contains all the data consumed as input by the layer. The output data tensor contains all the data computed as output by the layer. The intermediate data tensor contains any data produced by the layer as intermediate computations, such as partial sums.

[0029] The data tensors (inputs, outputs, and intermediate values) of a layer can be three-dimensional, where the first two dimensions can be interpreted as encoding spatial locations and the third dimension can be interpreted as encoding different features. For example, when the data tensor represents a color image, the first two dimensions encode the vertical and horizontal coordinates within the image, and the third dimension encodes the color at each location. Each element of the input data tensor X can be connected to each neuron via a separate weight, so the weight tensor W usually has six dimensions, connecting the three dimensions of the input data tensor (input row a, input column b, input feature c) with the three dimensions of the output data tensor (output row i, output column j, output feature k). The intermediate data tensor Z has the same shape as the output data tensor Y. The parameter tensor V connects the three output data tensor dimensions with an additional dimension o indicating the parameters of the activation function σ.

[0030] The elements of the layer's output data tensor Y can be calculated as in Equation 1, where the neuron activation function σ is given by the activation function parameter vector V[i, j, k:], and the weighted sum Z[i, j, k] can be calculated as in Equation 2.

[0031] Y[i,j,k]=σ(V[i,j,k,:];Z[i,j,k])

[0032] Equation 1

[0033]

[0034] To simplify the notation, the weighted sum in Equation 2 may be referred to as the output, which is equivalent to using a linear activation function Y[i, j, k] = σ(Z[i, j, k]) = Z[i, j, k], it being understood that the same statement applies without loss of generality when different activation functions are used.

[0035] In various embodiments, the computation of the output data tensor as described above is broken down into smaller problems. Each problem can then be solved in parallel on one or more neural cores, or on one or more cores of a conventional multi-core system.

[0036] A neural network model is a set of constants that together specify the entire computation performed by a neural network, including the connection graph between neurons and the weights and activation function parameters of each neuron. Training is the process of modifying a neural network model to perform the desired function. Inference is the process of applying a neural network to inputs to produce outputs without modifying the neural network model.

[0037] An IPU is a type of processor that performs neural network inference. A neural inference chip is a specific physical instance of an IPU.

[0038] As mentioned above, massively parallel computing architectures have the potential to massively accelerate execution over traditional sequential computing architectures. However, actual acceleration is limited by the parallelism present in actual data and algorithms and the efficiency of mapping data and algorithms to parallel architectures. Neural networks are generally subject to parallelism—there is a wealth of parallelism in both activation and weight data along many dimensions including spatial and feature dimensions. However, parallelism and the most effective type of parallelism vary according to individual networks and within a given network. For example, there can be different dimensions from one layer to another. Therefore, parallel architectures with fixed sizes (e.g., fixed array sizes, fixed vector widths, fixed numbers of parallel computing units) and configurations are difficult to utilize all available parallelism in data and / or algorithms.

[0039] To illustrate the variation in network dimensionality, Table 1 includes the layer dimensions (height x width x features) of various exemplary networks. Despite having the same dimensions for the input layer and the final layer (because they solve the same classification problem on the same dataset), there is a wide variation in network layer sizes between implementations (and not all layers are shown). It should be understood that within a network, the shape of the layers changes from the first layer to the last layer. Early layers typically have large spatial dimensions (height, width), while later layers have larger feature dimensions.

[0040] Inception v1 AlexNet VGG-19 ResNet Input Layer 224x224x3 224x224x3 224x224x3 224x224x3 First floor 112x112x64 55x55x48 224x224x64 112x112x64 Middle Layer 14x14x512 13x13x256 28x28x512 14x14x256 Subsequent layers 7x7x1024 6x6x256 7x7x512 7x7x512 Final layer 1x1x1000 1x1x1000 1x1x1000 1x1x1000

[0041] Table 1

[0042] refer to Figure 1A - Figure D shows an exemplary parallel configuration of a given network on a single multi-core processor. Figure 1A In , processor 100 is configured to have 16 core parallelism, which means that each of the 16 cores (e.g., 101) is executing in parallel. The vector parallelism is 2, which means that each core processes 2 input vectors (e.g., 102) in parallel. Figure 1B In FIG. 1 , processor 100 is configured to have a core parallelism of 8 and a vector parallelism of 4. Figure 1C In FIG. 1 , processor 100 is configured to have a core parallelism of 4 and a vector parallelism of 8. Figure 1D In the embodiment, the processor 100 is configured to have a core parallelism of 2 and a vector parallelism of 16. The present disclosure achieves this configuration by solving how to consistently divide and distribute data between cores, how to make arithmetic vector units have different effective vector widths, how to consistently send data to vector units of different widths, and how to reconfigure the processor into different modes.

[0043] refer to Figure 2A , shows a multi-core inference processing unit (IPU) according to an embodiment of the present disclosure. IPU 200 includes a model memory 201 for a neural network model. As described above, the neural network model may include synaptic weights of the neural network to be calculated. In some embodiments, model memory 201 includes one or more physical memories, which may be allocated to synaptic weights 211 and instructions 212, respectively. IPU 200 includes activation memory 202, which may be transient. Activation memory 202 may be divided into input and output areas, and stores neuron activations for processing.

[0044] The IPU 200 includes an array 206 of neural cores 203. Each core 203 includes a computation unit 233 that is loaded with a neural network model from the model memory 201 and is operable to perform vector calculations. Each core also includes a local activation memory 232. Input activations are provided from the local activation memory 232 before each computation step. The output from the computation unit 233 is written back to the activation memory 232 for processing on the same or another computation unit.

[0045] The IPU 200 includes one or more on-chip networks (NoCs) 205. In some embodiments, partial sum NoCs 251 interconnect cores 203 and transmit partial sums between them. In some embodiments, a separate parameter distribution NoC 252 connects the cores 203 to the memory 201 for distributing weights and instructions to the cores 203. It will be appreciated that various configurations of NoCs 251 and 252 are suitable for use in accordance with the present disclosure. For example, broadcast networks, row broadcast networks, tree networks, and switching networks may be used.

[0046] In various embodiments, a global scheduler (chip microengine) 204 is included in the IPU 200. In various embodiments, a local core controller (core microengine) 234 is included on each core 203. In such embodiments, the direction of operation is shared between the global scheduler (chip microengine) and the local core controller (core microengine). For example, the global scheduler 204 can load computational instructions from the model memory 201 to the computational unit 233 on each core 203. Parameters (e.g., neural network / synaptic weights) can be loaded from the model memory 201 to the computational unit 233 on each core 203 by the global scheduler 204. Neural network activation data can be loaded from the local activation memory 232 to the computational unit 233 on each core 203 by the local core controller 234. As described above, activation is provided to the axons of a specific neural network defined by the model, and can originate from the same or another computational unit, or from outside the system.

[0047] The computation unit 233 performs computations to generate output neuron activations as directed by the local core controller 234. Specifically, the computation includes applying the input synaptic weights to the input activations. It should be appreciated that various methods may be used to perform such computations, including in silico dendrites and vector multiplication units. The results of the computations are stored in the local activation memory 232 as directed by the local core controller 234. These stages may be pipelined to provide efficient use of the computation units on each core. It should also be appreciated that inputs and outputs may be transferred from the local activation memory 232 to the global activation memory 202, depending on the requirements of a given neural network.

[0048] refer to Figure 2B , to calculate the 1×M output tensor block 261, the neural core multiplies the 1×N input activation block 262 by the N×M weight block 263. The vector matrix multiplication unit 264 performs the multiplication and passes the resulting product to the 1×N vector arithmetic unit 265. Next, the result of the arithmetic 265 is provided to the M neuron activation functions 266 to produce the 1×M output tensor block 261.

[0049] For purposes of illustration, IPU 200 may include an array of A×B parallel cores, each core having N×M parallel vector or matrix computation elements.

[0050] Each kernel computes a vector-matrix multiplication Z=XW, where: X is the input data row vector [1×N]; W is the weight matrix [N×M]; Z is the output data row vector [1×M]. Each kernel applies an activation function Y=F(Z) to the output of the matrix multiplication, where Y is the output data row vector [1×M] and f(·) is a linear or nonlinear function. This constitutes a single neuron operation. For neural network inference, this operation is repeated over many neurons and many layers of neurons, each operating on different data X and potentially different parameters W.

[0051] As further described herein, the present disclosure provides for parallel exploitation of such cores. In particular, core and vector parallelism are provided. Typically, data-level or thread-level parallelism is accommodated by scaling on cores, while operation-level parallelism is accommodated by scaling on vector units. The present disclosure enables maximizing this mapping so that at any given time, almost all cores are computing (not idle) and almost all vector units are computing (not idle).

[0052] In an exemplary configuration, neural inference data parallelism is provided across spatial and feature dimensions. Data parallelism can be mapped to architectural parallelism as follows: spatial parallelism is provided by scaling across cores, while feature parallelism is provided by scaling across vector units.

[0053] In various embodiments, a parallel computing architecture is provided in which core-level parallelism and vector-level parallelism are reconfigurable. Specifically, the computing architecture can be applied to neural network reasoning. Various examples are expressed in terms of vector-level parallelism. However, it will be understood that the disclosure provided herein is applicable to matrix-level parallelism, or more generally to operation-level parallelism involving parallel arithmetic and / or logic computing units.

[0054] As described herein, tensors of different dimensions (with different spatial and feature dimensions) are mapped to the same core array while maintaining efficient utilization of the core array and core computing elements. Data is partitioned across cores at a configurable granularity (corresponding to core-level parallelism). In various configurations, data spatial parallelism is mapped to core parallelism and data feature parallelism is mapped to vector parallelism. Such a configuration may be particularly suitable for early layers in a network. In some configurations, spatial parallelism and feature parallelism are mapped to core parallelism, and feature parallelism is mapped to vector parallelism. Such a configuration may be particularly suitable for subsequent layers in a network. In various embodiments, NoC is used to distribute data across cores.

[0055] As summarized in Table 2 below, different parallelism strategies can be used depending on the relationship between spatial and core parallelism and feature parallelism and vector parallelism. In typical early layers, feature parallelism is less than or equal to vector parallelism, and spatial parallelism is greater than core parallelism. In typical intermediate layers, feature parallelism is greater than or equal to vector parallelism, and spatial parallelism is greater than or equal to core parallelism. In typical subsequent layers, feature parallelism is greater than vector parallelism, and spatial parallelism is less than core parallelism. It will be understood that where feature parallelism equals vector parallelism and spatial parallelism equals core parallelism, there is a native 1:1 mapping from a given tensor to a given physical core array.

[0056]

[0057]

[0058] Table 2

[0059] refer to Figure 3 , an exemplary tensor mapping is illustrated according to an embodiment of the present disclosure. Parallel core array 301 includes A×B=16 parallel cores, each parallel core having a vector parallelism of N≥1. Activation tensor 302 has spatial parallelism R×C=16, and feature parallelism F=N. Therefore, A=R, B=C, and N=F. Core array 301 can be partitioned into 16 islands 303, each island including one core. Then, A1×1×1 spatial slices of the activation tensor can be mapped to each island.

[0060] refer to Figure 4 , another exemplary tensor mapping is shown according to an embodiment of the present disclosure. Parallel core array 401 includes A×B=16 parallel cores, each parallel core has a vector parallelism of N. Activation tensor 402 has spatial parallelism R×C=64, and feature parallelism F=N. Therefore, A<R, B<C, N=F. Core array 401 can be partitioned into 16 islands 403, each island including one core. Then, A2×2×1 spatial slices of the activation tensor can be mapped to each island.

[0061] Reference Figure 5 , according to an embodiment of the present disclosure, another exemplary tensor mapping is shown. The parallel kernel array 501 includes A×B = 16 parallel kernels, and each parallel kernel has a vector parallelism of N. The activation tensor 502 has a spatial parallelism of R×C = 64 and a feature parallelism of F = 4N. Therefore, A < R, B < C, N < F. The kernel array 501 can be divided into 16 islands 503, and each island contains one kernel. Then, A2×2×4N slices of the activation tensor can be mapped to each island.

[0062] Reference Figure 6 , according to an embodiment of the present disclosure, another exemplary tensor mapping is shown. The parallel kernel array 601 includes A×B = 16 parallel kernels, and each parallel kernel has a vector parallelism of N. The activation tensor 602 has a spatial dimension of R×C = 4 and a feature parallelism of F = 4N. Therefore, A > R, B > C, N < F. The kernel array 601 can be divided into 4 islands 603, and each island includes four kernels. In this example, the islands are square within the kernel array. Then, A1×1×4N slices of the activation tensor can be mapped to each island, and each kernel in the island receives a1×1×N sub-slices.

[0063] Reference Figure 7 , according to an embodiment of the present disclosure, another exemplary tensor mapping is shown. The parallel kernel array 701 includes A×B = 16 parallel kernels, and each parallel kernel has a vector parallelism of N. The activation tensor 702 has a spatial dimension of R×C = 4 and a feature parallelism of F = 4N. Therefore, A > R, B > C, N < F. The kernel array 701 can be divided into 4 islands 703, and each island includes four kernels. In this example, each island spans a column within the kernel array. Then, A1×1×4N slices of the activation tensor can be mapped to each island, and each kernel in the island receives a1×1×N sub-slices.

[0064] Reference Figure 8 , according to an embodiment of the present disclosure, another exemplary tensor mapping is shown. The parallel kernel array 801 includes A×B = 16 parallel kernels, and each parallel kernel has a vector parallelism of N. The activation tensor 802 has a spatial dimension of R×C = 1, and the feature parallelism is F = 16N. Therefore, A > R, B > C, N < F. The kernel array 801 can be divided into one island 803, including 16 kernels. Then, A1×1×1 slices of the activation tensor can be mapped to each kernel in the island.

[0065] Reference Fig. 9, another exemplary tensor mapping is shown according to an embodiment of the present disclosure. Parallel core array 901 includes A×B=16 parallel cores, each parallel core has vector parallelism N. Activation tensor 902 has spatial dimensions R×C=64, and feature parallelism of F=N / 4. Therefore, 2A=R, 2B=C, N=4F. Core array 901 can be partitioned into 16 islands 903, each island including one core. Then, A2×2×F slices of the activation tensor can be mapped to each island. Since the core has vector parallelism N, 2×2×F slices of the activation tensor can be calculated by the core vector unit in a single cycle, thereby maintaining full vector unit utilization.

[0066] It will be appreciated that the above partitions are exemplary and that other dimensions and partitioning schemes are within the scope of the present invention.

[0067] As will be appreciated from the above, different size tensors (having different spatial and feature dimensions) can be mapped to the same core array while maintaining efficient utilization of the core array and core computing elements. In certain partitions, where the available spatial and feature parallelism is greater than or equal to the core and vector parallelism, an island includes one core. In certain partitions, where the available feature parallelism is greater than the vector parallelism, and the available spatial parallelism is less than the core parallelism, an island includes more than one core.

[0068] In certain partitions where available spatial parallelism is greater than core parallelism and available feature parallelism is less than vector parallelism, the core is subdivided using the excess feature parallelism to increase spatial parallelism. Subdivision is achieved by splitting a larger vector unit within the core into multiple smaller vector units. For example, a 64-wide vector unit can also be operated as two 32-wide vector units or four 16-wide vector units. In the case of large summation trees (in the neural inference unit), the results are taken before the last level (or levels) of summation in the tree.

[0069] refer to Fig.10 , according to an embodiment of the present disclosure, partial sums passed between cores are shown. Parallel core array 1001 includes A×B=16 parallel cores, each with vector parallelism N. Each island 1002 calculates vector-matrix multiplication Z=XW, where X and W can be decomposed into sub-vectors and sub-matrices X=[X1, X2, X3, X4] and W=[W1, W2, W3, W4] T. Within an island, the cores compute partial sums Z1=X1W1, Z2=X2W2, Z3=X3W3, and Z4=X4W4. The final result is obtained by transferring the partial sums between the cores within the island and summing the partial sums to obtain the final result Z=Z1+Z2+Z3+Z4. In some embodiments, all partial sums are sent to a single core for the final summation. In some embodiments, the partial sums are sent via an on-chip network. By performing partial summations in this manner, the effective vector size can be configured to optimize overall efficiency.

[0070] In some embodiments, the vector-matrix sub-multiplications, partial sum propagations, and partial sum accumulations are all fully pipelined, so that the entire island appears as a vector-matrix operation.

[0071] refer to Figures 11A-11B , shows the parameter distribution according to an embodiment of the present disclosure. Fig.11A In FIG. 1 , the core array 1101 is divided into 16 islands 1102, each island including one core. Fig. 11B In FIG. 1 , the core array 1101 is divided into four islands 1103, each island including four cores. For these different configurations, different parameters (weights and instructions) are required at each core. In particular, in various neural networks, the islands require the same parameters, while the cores within the islands require different parameters. The NoC 1104 distributes these parameters according to the parallelism configuration, thereby distributing the parallel computation among the cores.

[0072] refer to Fig.12 , a pipeline timing diagram is provided, which illustrates the core operation according to an embodiment of the present disclosure. At 1201, a computation instruction is loaded into a neural computing unit. At 1202, parameters (including neural network weights) are loaded into the neural computing unit. At 1203, operands are loaded into the neural computing unit. At 1204, a vector-matrix multiplication is performed. At 1205, partial sums are transmitted. At 1206, partial sums are accumulated. At 1207, an activation function is calculated. At 1208, the result is stored. Each stage 1211…1214 is a different operation. Therefore, each operation can have an independent parallel configuration.

[0073] In some embodiments, instructions assigned by the chip microengine (204) determine the operations (arithmetic, data addressing, partial and communication, etc.) performed by each core, which can be passed globally to all cores, or can be passed to each core individually using specific instructions for each core. In the latter case, each core only needs to know what it needs to do with its current data, the partial sums passed to it, and any reshuffle operations. It does not need to configure any global knowledge. New instructions can be passed for each operation stage, so the core parallelism configuration can be changed at each operation stage. For pipeline cores, the current parallelism configuration remains consistent throughout the pipeline at each stage, thereby achieving overlapping (pipeline) operations.

[0074] Thus, in various embodiments, the degree of parallelism in the system can be dynamically reconfigured during runtime. In some embodiments, instructions are used to reconfigure the computing architecture. In some embodiments, multiple instructions are used to reconfigure multiple parts of the computing architecture.

[0075] refer to Fig.13 , a pipeline timing diagram is provided, which shows data reshuffling according to an embodiment of the present disclosure. At 1301, the computing instructions are loaded into the neural computing unit. At 1302, the parameters (including the neural network weights) are loaded into the neural computing unit. At 1303, the operands are loaded into the neural computing unit. At 1304, the vector-matrix multiplication is performed. At 1305, the partial sums are transmitted. At 1306, the partial sums are accumulated. At 1307, the activation function is calculated. At 1308, the result is stored. At 1309, the data is reshuffled. The data reshuffling stage occurs between the computing operation stages. Each stage 1211…1214 is a different operation.

[0076] If the activation data X is not in the appropriate core at the end of the operation, it may need to be transferred to another core, for example, via the NoC. This operation may be referred to as a data reshuffle, which occurs as an additional phase between stages of computational operations. Reshuffling may be required when switching between parallel configurations. In various embodiments, the data reshuffle operation is performed using core-level (core microcode) instructions. For example, such instructions may include:<rd_addr> Read the data block length at<block_length> ; and write the data block to the core<core_addr> Address on<wr_addr> .

[0077] Thus, data may be reshuffled in response to changes in system parallelism configuration. In some embodiments, the reshuffling is performed on the NoC. In some embodiments, the data reshuffling is initiated via an instruction.

[0078] refer to Fig.14, shows a shuffle operation according to an embodiment of the present disclosure. Input activation tensor 1401 is provided to core array 1402. Output activation results 1403 are generated. In the shuffle phase, the output activation results are transmitted via the NoC between the cores. In some embodiments, the shuffle operation is explicit. The output activation results are thus distributed between the cores as shown.

[0079] refer to Fig.15 , shows a shuffle operation according to an embodiment of the present disclosure. The input activation tensor 1501 is provided to the core array 1502. The partial sum 1503 is generated and transmitted. Each core in the next layer of islands accumulates different partial sum blocks. In this way, the final accumulation results end up at different cores without sending the output activation results via the NoC.

[0080] refer to Fig.16 , shows a shuffle operation according to an embodiment of the present disclosure. Input activation tensor 1601 is provided to kernel array 1602. Output activation result 1603 is generated. Full output tensor result is calculated at each kernel in the next layer island. Only required data is operated on in the next layer.

[0081] refer to Fig.17 , according to the present disclosure, the reshuffling of data for parallel computing utilization is shown. In an exemplary neural network, layer L has an activation tensor 1701 with dimensions R×C×F=4×4×1. The core array 1702 has dimensions A×B=4×4. Therefore, tensor 1701 is naturally mapped to the core array 1702 configured to have 4×4=16 islands. Layer L+1 has an activation tensor 1703 with dimensions R×C×F=2×2×4. Without data shuffling, tensor 1703 would only occupy 2×2=4 cores (as shown in 1704). However, with data shuffling, core array 1702 can be divided into four islands, each including four cores. Tensor 1703 can then be mapped to the four islands, resulting in 100% core utilization (as shown in 1705).

[0082] In the process of performing inference through different layers of the neural network, the activation data organization naturally shifts from larger spatial parallelism to larger feature parallelism. If the data is not reshuffled, the utilization of parallel cores (the number of cores computing in parallel) will decrease because the spatial parallelism drops below the core-level parallelism. By reshuffling, both spatial parallelism and feature parallelism can be mapped to core-level parallelism, all cores can be kept fully utilized.

[0083] Therefore, the present disclosure provides for improving parallel core array and computing element utilization by reshuffling data between core-level parallelism and vector-level parallelism. In various embodiments, reshuffling of data is provided between spatial parallelism and feature parallelism, mapping to core-level parallelism and vector-level parallelism, respectively.

[0084] Reference now Fig.18 , a method for configuring an inference processing unit (IPU) according to an embodiment of the present disclosure is shown. At 1801, a neural network model including multiple layers is read. Each layer has at least one dimension and includes multiple synaptic weights. For each layer of the neural network model: at 1802, multiple cores are divided into multiple partitions based on the dimension and vector unit of the layer; at 1803, the multiple cores are configured to implement the layer; at 1804, input activations of the layer are provided to the multiple cores; and at 1805, synaptic weights associated with the layer are applied to the input activations to generate multiple output activations.

[0085] Reference now Fig.19 , a schematic diagram of an example of a computing node is shown. Computing node 10 is only one example of a suitable computing node and is not intended to impose any limitations on the scope of use or functionality of the embodiments described herein. Regardless, computing node 10 can be implemented and / or perform any of the functions set forth above.

[0086] In computing node 10, there is a computer system / server 12, which can operate with many other general or special computing system environments or configurations. Examples of well-known computing systems, environments and / or configurations suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems or devices, etc.

[0087] Computer system / server 12 may be described in the general context of computer system executable instructions, such as program modules, executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer system / server 12 may be practiced in a distributed cloud computing environment, where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in local and remote computer system storage media including memory storage devices.

[0088] like Fig.19As shown, the computer system / server 12 in the computing node 10 is shown in the form of a general-purpose computing device. The components of the computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 to the processor 16.

[0089] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

[0090] Computer system / server 12 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer system / server 12, and it includes volatile and nonvolatile media, removable and non-removable media.

[0091] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, and commonly referred to as a "hard drive"). Although not shown, a disk drive for reading from and writing to a removable, non-volatile disk (e.g., a "floppy disk") may be provided, as well as an optical drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media. In this case, each may be connected to the bus 18 via one or more data media interfaces. As will be further shown and described below, the memory 28 may contain at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of an embodiment of the present invention.

[0092] A program / utility 40 having a set (at least one) of program modules 42, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in memory 28 as an example and not by way of limitation. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0093] The computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, etc.; one or more devices that enable a user to interact with the computer system / server 12; and / or any device that enables the computer system / server 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur via an input / output (I / O) interface 22, however, the computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general purpose wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18, and it should be understood that, although not shown, other hardware and / or software components may be used in conjunction with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0094] In various embodiments, one or more inference processing units (not shown) are coupled to bus 18. In such embodiments, the IPU may receive data from or write data to memory 28 via bus 18. Likewise, the IPU may interact with other components via bus 18 as described herein.

[0095] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of the present invention.

[0096] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0097] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0098] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0099] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0100] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0101] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0102] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0103] The description of various embodiments of the present invention has been given for the purpose of illustration, but it is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements existing in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A system, include: a neural network model memory adapted to store a neural network model comprising multiple layers, each layer having at least one dimension and comprising a plurality of synaptic weights; a plurality of neural cores, each neural core comprising a computational unit adapted to apply a plurality of synaptic weights to a plurality of input activations to produce a plurality of output activations, the computational unit having a plurality of vector units, and an activation memory adapted to store said input activations and said output activations; The system is adapted to partition the plurality of cores into a plurality of partitions based on the dimension of the layer and the vector units, and in the partitions where the available spatial parallelism is greater than the core parallelism and the available feature parallelism is less than the vector parallelism, subdivide the core by splitting a larger vector unit into a plurality of smaller vector units.

2. The system according to claim 1, further comprising: include: at least one controller operatively coupled to the neural network model memory and the plurality of cores, the at least one controller being adapted to configure the plurality of cores for each layer of the neural network model to implement the layer, and Input activations for the layer are provided to the plurality of kernels.

3. The system of claim 2, further comprising a network coupled to the plurality of cores.

4. The system of claim 3, wherein input activations are provided to the plurality of cores via the network. 5 . The system of claim 3 , wherein configuring the plurality of cores comprises distributing parameters to the plurality of cores via the network. 6 . The system of claim 5 , wherein configuring the plurality of cores further comprises distributing instructions to the plurality of cores via the network.

7. The system of claim 1, wherein the plurality of partitions of each layer are further determined based on a spatial dimension of the input activations of the layer.

8. The system of claim 1, wherein the plurality of partitions of each layer are further determined based on the spatial dimensions and feature dimensions of the input activations of the layer.

9. The system of claim 1, wherein the plurality of partitions of each layer are further determined based on a spatial dimension of the output activations of the layer.

10. The system of claim 1, wherein the plurality of partitions of each layer are further determined based on a spatial dimension and a feature dimension of the output activations of the layer.

11. The system of claim 1, wherein the plurality of partitions of each layer are further determined based on one or more of the spatial dimension of the input activations, the feature dimension of the input activations, the spatial dimension of the output activations, or the feature dimension of the output activations of the layer.

12. The system of claim 11, wherein the plurality of partitions per layer is further determined by a dimension of the plurality of cores.

13. The system of claim 1, wherein the core within each of the plurality of partitions is configured to compute a partial sum. The system of claim 13 , wherein the partial sums are aggregated to calculate a result for an associated layer.

15. The system of claim 14, wherein the partial sums are sent via a network for aggregation.

16. The system of claim 2, wherein the at least one controller is further adapted to redistribute the output activations of a layer among the plurality of cores when the output activations of the layer are computed.

17. The system of claim 16, wherein the reallocation is via a network.

18. The system of claim 16, wherein the reallocation is determined based on one or more of a spatial dimension of the input activations, a feature dimension of the input activations, a spatial dimension of the output activations, or a feature dimension of the output activations of the layer.

19. A method, include: Reading a neural network model comprising multiple layers, each layer having at least one dimension and comprising a plurality of synaptic weights; For each layer of the neural network model, dividing the plurality of cores into a plurality of partitions based on the dimension and the vector unit of the layer; configuring the plurality of cores to implement the layer; providing input activations of the layer to the plurality of cores; applying the synaptic weights associated with the layer to the input activations to produce a plurality of output activations; as well as In partitions where the available spatial parallelism is greater than the core parallelism and the available feature parallelism is less than the vector parallelism, the core is subdivided by splitting the larger vector unit into multiple smaller vector units.

20. The method according to claim 19, further comprising: include: Calculate the partial sum within each partition; transmitting the partial sums between cores within each partition; The partial sums are summed to compute the output activation.

21. The method of claim 19, wherein configuring the plurality of cores comprises distributing parameters to the plurality of cores via a network.

22. The method of claim 19, wherein configuring the plurality of cores comprises distributing instructions to the plurality of cores via a network.

23. The method of claim 19, wherein the plurality of partitions of each layer are further determined based on one or more of the spatial dimension of the input activations, the feature dimension of the input activations, the spatial dimension of the output activations, or the feature dimension of the output activations of the layer.

24. The method of claim 23, wherein the plurality of partitions per layer is further determined by the dimensions of the plurality of kernels.