MASSIVELY PARALLEL NEURONAL INFERENCE DATA PROCESSING ELEMENTS

Abstract

System, exhibiting: a plurality of multipliers (206; 302; 504; 1202), wherein the plurality of multipliers is arranged in a plurality of equally sized groups, each of the plurality of multipliers being designed to apply a weighting to an input activation in parallel to produce an output; a plurality of adders (204; 304; 506; 1204), each of the plurality of adders being operatively connected to one of the groups of multipliers, each of the plurality of adders being designed to add the outputs of the multipliers within their respective groups in parallel to produce a partial sum (306); a first plurality of function blocks (508; 1206), wherein each from the first plurality of function blocks is operatively connected to one from the plurality of adders, wherein each from the first plurality of function blocks is designed to apply a function in parallel to the partial sum of its associated adder in order to produce an output value; a vector register (116), wherein the vector register is operatively connected to the first plurality of function blocks, wherein the vector register is designed to store the output values ​​of the first plurality of function blocks, wherein the first plurality of function blocks is designed to combine the output values ​​stored in the vector register with subsequently calculated output values ​​of the first plurality of function blocks, wherein output values ​​of this combination are stored in the vector register; a second plurality of function blocks, each of which is operatively connected to the vector register, each of which is designed to apply a function to the stored output values ​​in parallel.

Description

BACKGROUND

[0001] Embodiments of the present disclosure relate to neural network inference and, in particular, to massively parallel neural inference data processing elements.

[0002] The publication "Massively Parallel Coprocessor for Convolutional Neural Networks" describes a massively parallel coprocessor for convolutional neural networks (CNNs). The coprocessor features parallel clusters of vector processing elements, each cluster consisting of hand-optimized 2D convolver units and other hardware specifically designed for CNNs. A key feature of the coprocessor is the use of off-chip memory on the coprocessor board as a scratchpad for intermediate CNN data. This is made possible by the high-bandwidth memory architecture and the reduction of data precision to pack multiple words per memory operation (SANKARADAS, M. et al.: A Massively Parallel Coprocessor for Convolutional Neural Networks. In: 2009 20th IEEE International Conference on Application-specific Systems, Architectures and Processors, 2009, 53-60. https: / / ieeexplore.ieee.org / document / 5200010).

[0003] Publication GB 2 552 243 A relates to a computer-implemented method for configuring a hardware implementation of a CNN, wherein the method comprises: determining, for each of a plurality of layers of the CNN, a first numerical format to represent weight values ​​in the layer based on a distribution of weight values ​​for the layer, wherein the first numerical format comprises a first integer of a first predetermined bit length and a first exponent value that is fixed for the layer; determining a second numerical format for each of a plurality of layers of the CNN to represent data values ​​in the layer based on a distribution of expected data values ​​for the layer, wherein the second numerical format comprises a second integer of a second predetermined bit length and a second exponent value that is fixed for the layer;and storing specific number formats for use when configuring the hardware implementation of a CNN. SUMMARY

[0004] According to embodiments of the present disclosure, systems, methods, and computer program products for massively parallel neural inference data processing are provided. A plurality of multipliers is arranged in a plurality of equally sized groups. Each of the plurality of multipliers is designed to apply a weight to an input activation in parallel to produce an output. A plurality of adders is operatively connected to one of the groups of multipliers. Each of the plurality of adders is designed to add the outputs of the multipliers within their respective group in parallel to produce a partial sum. A plurality of function blocks is operatively connected to one of the plurality of adders. Each of the plurality of function blocks is designed to apply a function to the partial sum of its associated adder in parallel to produce an output value.

[0005] According to one aspect, a system is provided comprising: a plurality of multipliers, wherein the plurality of multipliers are arranged in a plurality of equally sized groups, each from the plurality of multipliers being designed to apply a weight to an input activation in parallel to produce an output; a plurality of adders, wherein each from the plurality of adders is operatively connected to one of the groups of multipliers, each from the plurality of adders being designed to add the outputs of the multipliers within their respective associated groups in parallel to produce a partial sum.

[0006] According to one aspect, a procedure is provided comprising: applying a plurality of weights in parallel to a plurality of input activations by a plurality of equally sized groups of multipliers to generate a plurality of outputs for each group of multipliers; adding the plurality of outputs from each group of multipliers in parallel to generate a partial sum from each group of multipliers.

[0007] According to one aspect, a system is provided comprising: a plurality of multipliers, wherein the plurality of multipliers are arranged in a plurality of equally sized groups; a plurality of adders, wherein each of the plurality of adders is operatively connected to one of the groups of multipliers; a computer-readable storage medium containing program instructions, wherein the program instructions are executable to perform a procedure comprising: applying a weight to an input activation in parallel by the plurality of multipliers to produce an output; adding the outputs of the multipliers within their respective groups in parallel by each of the plurality of adders to produce a partial sum. Brief description of the different views of the drawings

[0008] With reference to the accompanying drawings, embodiments of the invention are now described, which are to be understood as merely exemplary and in which: Fig. 1 represents an inference processor architecture with multiple neural cores according to embodiments of the present disclosure. Fig. 2 represents a massively parallel vector matrix multiplier for calculating partial sums according to embodiments of the present disclosure. Fig. 3 represents a dendritic circuit according to embodiments of the present disclosure. Fig. 4 represents a vector matrix multiplier with an arrangement of adders and a vector register according to embodiments of the present disclosure. Fig. 5 represents a vector matrix multiplier with an activation function according to embodiments of the present disclosure. Fig. 6 represents a reference table for an activation function according to embodiments of the present disclosure. The Fig. 7A and Fig. 7B represents an input activation via a distribution roundup according to embodiments of the present disclosure. Fig. 8 binary trees of adders for a dendritic circuit according to embodiments of the present disclosure. Fig. 9 represents a tree of carry-store adders for a dendritic circuit according to embodiments of the present disclosure. The Fig. 10A and Fig. 10B represents a ternary multiplier realized by a multiplexer according to embodiments of the present disclosure. Fig. 11 represents a shifter for use in a vector matrix multiplication according to embodiments of the present disclosure. Fig. 12 represents a neural circuit according to embodiments of the present disclosure. Fig. 13 represents a mixing operation using a vector matrix multiplication unit according to embodiments of the present disclosure. The Fig. 14A and Fig. 14B illustrates exemplary butterfly network calculations according to embodiments of the present disclosure. Fig. 15 represents a data packing operation using a vector matrix multiplication unit according to embodiments of the present disclosure. Fig. 16 represents a maximum value pooling operation using a vector matrix multiplication unit according to embodiments of the present disclosure. Fig. 17 represents a mean-value pooling operation using a vector matrix multiplication unit according to embodiments of the present disclosure. Fig. 18 represents an element-wise multiplication operation using a vector matrix multiplication unit according to embodiments of the present disclosure. Fig. 19 represents a programmable operation according to embodiments of the present disclosure to combine partial sums using a vector matrix multiplication unit with several element-wise operations. Fig. 20 represents a method for operating a neural processing kernel according to embodiments of the present disclosure. Fig. 21 represents a data processing node according to embodiments of the present disclosure. DETAILED DESCRIPTION

[0009] An artificial neuron is a mathematical function whose output is a non-linear function of a linear combination of its inputs. Two neurons are connected if the output of one is an input of the other. A weight is a single value that encodes the strength of the connection between the output of one neuron and the input of another.

[0010] A neuron computes its output, also known as activation, by applying a non-linear activation function to a weighted sum of its inputs. A weighted sum is an intermediate result calculated by multiplying each input by its corresponding weight and accumulating the products. A partial sum is a weighted sum of a subset of inputs. A weighted sum of all inputs can be computed stepwise by accumulating one or more partial sums.

[0011] A neural network is a collection of one or more neurons. A neural network is often divided into groups of neurons called layers. A layer is a collection of one or more neurons that all receive inputs from the same layers and all send outputs to the same layers, and that typically perform a similar function. An input layer is a layer that receives inputs from a source outside the neural network. An output layer is a layer that sends outputs to a destination outside the neural network. All other layers are intermediate processing layers. A multilayer neural network is a neural network with more than one layer. A deep neural network is a multilayer neural network with many layers.

[0012] Arrangements of neural processing units, also known as neural nuclei, provide an architecture for executing various types of neural networks and deep neural networks. A variety of nucleus architectures are suitable for realizing neural network computations. In general, such computations involve calculating neuron activations based on input values, synaptic weights, and neural activation functions. It should be obvious that neuron activations can be distributed in various ways, e.g., via an on-chip network. Although various examples here refer to neuron activations transmitted via a network, it should be clear that the present disclosure is more broadly applicable to all neurosynaptic systems, such as GPUs and specialized ASIC chips, and is applicable to neuronal activity regardless of the transport mechanism.

[0013] According to various embodiments of the present disclosure, a neural program represents a neural network. A neural program contains information relating to the neural network. In some embodiments, the information includes neural properties and dynamic aspects that define the response of an electronic neuron to input neuron activations. For example, neural properties and dynamic aspects may include a threshold parameter, a loss parameter, a delay parameter, or a reset parameter. In some embodiments, the neural program information includes synaptic connections of the neuron (e.g., synaptic connections established via a synaptic weighting matrix). In some embodiments, the neural program information includes axon properties (e.g.,(Axon types) In some embodiments, the neuronal program information contains one or more targets (e.g., target axons) to which the output activation of the neuron is to be delivered.

[0014] Depending on the specific implementation, a neural network represents an instantiation of a neural program. A neural network can be instantiated in hardware, in a simulation, or both. For example, a neural program can create one or more instances of a neural network, where the instances can reside in a single core, across multiple cores, or on multiple chips.

[0015] According to one embodiment of the present disclosure, a dendritic tree is provided which can be configured with a purpose-built orientation for multiplication-accumulation circuits for each synapse and which can be operated in parallel in some instances. Additionally, according to a preferred embodiment, the present disclosure provides purpose-built partial sum operators and non-linear activation circuits for each neuron, which can also be operated in parallel. In some embodiments, a multi-function circuit is used for adder or maximum / minimum value operations. In some embodiments, the system includes a dendritic tree structured as a pipeline.

[0016] According to embodiments of the present disclosure, a neuron activation function is included, which in some embodiments uses a reference table to represent the neuron activation function. This feature is advantageous because it enables efficient and reconfigurable realizations of discretized models of continuous nonlinearities such as sigmoid, tanh, ReLU (Rectified Linear Unit) functions, etc. In some embodiments, the system distributes activations to all parallel neuron circuits in a kernel, e.g., using a broadcast network or a systolic distribution. Furthermore, according to preferred embodiments, the present disclosure includes a circuit for efficient ternary vector matrix multiplication. In some embodiments, vector multiplication with m entries and n bits is performed using ternary multipliers with m entries and accumulator circuits.In some embodiments, vector multiplication with m entries and n bits is performed directly with a ternary compressed vector with m entries (e.g. 8 bits for five weights) and an accumulation circuit.

[0017] In relation to Fig. Figure 1 presents a reference neural multi-core inference processor architecture according to embodiments of the present disclosure. In some embodiments, the neural core 100 receives a vector of neural activations as axonal inputs and multiplies the input activations by a matrix representing synaptic weights to generate a vector of neural activations as an output. In some embodiments, input and output activations are provided as vectors with 32 elements. In such embodiments, the synaptic weights are represented by a 32 × 32 matrix with 1024 weight elements. In various embodiments, the weights are binary, trinary (-1, 0, 1), or small integers. In some embodiments, a plurality of neural cores are arranged side by side on a chip, as in Fig. 1 shown.

[0018] Overall, the majority of neural inference cores 100 are connected by a Network on Chip (NoC) 102. As in Fig. As shown in Figure 1, each neural inference kernel 100 contains an activation work memory 104, a weighting work memory 106, a broadcast bus 108, an input activation register 110, a massively parallel vector matrix multiplier 112, an output activation register 114, a partial sum work buffer 116 and a routing buffer 118.

[0019] In an exemplary embodiment, the reference kernel behaves according to the following protocol: 1. During initialization, input activities are stored in the core activation memory. 2. Weighting values ​​are distributed to each core via the Network on Chip (NoC) 102 and stored in the weighting memory 106. 3. An activation block X is read from the activation memory 104, a weighting block W is read from the weighting memory 106 and sent to the vector matrix multiplier. 4. The vector matrix multiplier calculates a vector matrix product XW, and the output 7 is stored in the partial sum register 116. 5. The partial sums can be sent to or received from neighboring cores via the routing memory 118 and then added to other matrix multiplication partial results X'W' = Z'. 6. An activation function is applied to the final partial sum, and the result Y is written to the output activation register 114. 7. The output activation Y is written back to the activation memory 104.

[0020] In this way, different core configurations provide distributed logic for computation. In various implementations, multiple distributed neural cores operate in parallel. This parallelism enables faster neural network processing while simultaneously reducing latency between input representation and output computation. Each neural core implements a portion of the larger neural network model for a specific problem. Each neural core receives a portion of the total chip input and a portion of the total neural network model. The resulting modularity of chips and cores streamlines system design, debugging, and testing.

[0021] It should be obvious to the person skilled in the art that the specific core realization disclosed here serves for illustration purposes and is not intended as a limitation, and that embodiments of the present disclosure may contain a variety of core realizations.

[0022] In an exemplary implementation, for each neuron j, an input activation x i and a weighting w ij The following neural network calculation was performed, where the partial sum Z j This can be calculated by: Zj=∑ixiWij

[0023] The output activation is then calculated for each expenditure euro j from the partial sum z. yj=f(zj) where f(z) is a non-linear activation function.

[0024] For f(z), a sigmoid function, a tanh(x) function, or a simpler linear rectifier function can be used.

[0025] Neural accelerators such as GPUs and ASICs, but not limited to these, can perform a distributed multiplication-addition operation across multiple multiply-accumulate (MAC) units. However, such architectures inherently perform a sequence of multiplication-addition operations sequentially, which limits the degree of parallelism, even though the underlying neural network inherently exhibits parallel computation, enabling massive parallelism. Accordingly, the present disclosure preferably provides a low-precision neural network in which the hardware is capable of performing significantly more computations (e.g., several orders of magnitude) in parallel.In particular, the present disclosure preferably provides massively parallel data processing elements that include multiple multiplication-addition operations to compute neurons in parallel.

[0026] With reference to Fig. Section 2 discloses an exemplary massively parallel data processing element for computing neuron activations. The exemplary embodiment includes a massively parallel vector matrix multiplication unit for a low-precision neural inference processor. In the computing unit 202, a plurality of adder trees 204 are provided (where each adder tree can sum n outputs from n multipliers). A single multiplier is represented as 206, where a single column 208 of a VMM is represented as a dendritic tree (and with reference to Fig. 3 is described in more detail).

[0027] The system receives X, an input activation row vector of the n element (in some embodiments, each element consists of 4 bits with n = 32). Additionally, the system receives W, an n × m weighting matrix (in some embodiments, each element consists of 2 bits and n = m = 32). The arithmetic unit 202 has n x m multipliers and adders for combining products. They calculate all multiplications in parallel and sum the results in parallel (although the exact position of a multiplier can vary and is not limited to the specific arrangement shown). In some embodiments, there are 32 x 32 = 1024 small multipliers. The result Z = XW is a vector matrix product with m elements. In one exemplary embodiment, m = 32, but each of the m elements consists of 10 bits.

[0028] As in Fig. As shown in Figure 3, the output element of the vector matrix multiplication unit can be derived from Fig. 2 is calculated using n parallel multipliers, followed by an adder tree. Specifically, the inputs are vector X for n elements and W. j (j-th column of matrix W), distributed to parallel n-way multipliers 302, which can be configured in a parallel orientation. Each multiplier computes a product, the products of which are added by the adder tree 304 to a single sum. The output of the adder tree is the partial sum 306, which can be used to compute the neuron activation. As explained below, this partial sum can be provided to an activation function, which in some embodiments is programmable and can be defined as a non-linear or linear function, such as the identity function.

[0029] In the exemplary embodiment from Fig. 3. n multipliers and an adder tree are provided. Additionally, the input row vector X of the n element and W are given. j (j-th column of matrix W) is distributed to n multipliers; each multiplier calculates a product; the products are added together by the adder tree to form a single sum; and the output Z j = Σ i x i w ijrepresents the j-th vector element of Z = XW. In some embodiments, the multipliers are 4 x 2-bit multipliers, and the adder tree sums the outputs of n = 32 multipliers, with the final output being 10 bits. In other embodiments, the multiplication and addition operations are performed using carry-save adders (CSAs), so that the final output consists of two vectors (sum vector and carry vector). The final response Z is the sum of the sum and carry vectors. The final addition of the sum and carry vectors can be performed within the dendritic tree, or it can be performed by a subsequent processing stage outside the dendritic tree.

[0030] In connection with the in Fig. Figure 4 schematically illustrates an exemplary embodiment in which the result of the vector matrix multiplier is added to a previously calculated (partial sum) result. In such embodiments, the output result of the vector matrix multiplication unit (referred to as a partial sum vector) can be added to a previously calculated partial sum vector. As shown in Figure 4, the output result of the vector matrix multiplication unit (referred to as a partial sum vector) can be added to a previously calculated partial sum vector. Fig. As shown in section 4, a partial sum register can store partial sum vectors of m elements and m parallel adders to store the new partial sums (Z) and previously calculated partial sums (V). t-1 ) to add. Furthermore, previously calculated partial sums (V) can be added. t-1) are received from other cores via a core-to-core transmission channel. In the illustrated embodiment, the partial sum vector addition is performed in parallel using m parallel adders. Further optimization can be achieved by representing the output of the vector matrix multiplier as two sets of vectors (sum vector and carry vector). Subsequently, m parallel adders can be implemented using carry-store adders (similar to a floating-point multiplication adder).

[0031] As in Fig. As shown in section 5, an activation function arrangement is disclosed in which each output element of the vector matrix unit Z j = Σ i x i w ij An activation function f can be applied to obtain the neuronal activation: Y j = f(Σ i x i w ij). Exemplary activation functions include the binary threshold function, the sigmoid function, the ReLU (Rectified Linear Unit) function, and the linear function, as well as non-linear arbitrary functions that are possible using a reference table (LUT). Following this, an arrangement of m activation function calculators are implemented, which are linked to the output of the vector matrix unit. This generates m activation values, each of which can be represented as: y j = f(Σ i x i w ijTo simplify the architectural design, the output of the activation function (Y) can have the same precision as the input activations (X). In some embodiments, both input and output activations have 4 bits. The activation function calculators can be used to reduce the precision of the vectors. The partial sum vectors (Z) generally have higher precision (10 bits), whereas the output activation vectors (Y) have lower precision (4 bits). This reduction in precision can be achieved by a variety of linear and non-linear functions. In one exemplary embodiment, m = n = 32, so the input and output activation vectors have the same length. Accordingly, the output activation vector (Y) can be used directly as an input activation vector (X).

[0032] As shown, a typical neuron activation uses an n-element vector X, an "n × m" weighting matrix W, and calculates Y. j = f(Σ i x i w ij ) and generates an m-element output vector Y. The Z j = Σ i x i w ij Part is calculated by a vector matrix multiplier 502, which consists of arrangements of multipliers 504 and arrangements of adder trees 506. Activation functions 508 are applied in parallel to the output of the vector matrix multiplication. In some embodiments, n = m = 32 or n = m = 64; this results in identical input and output sizes.

[0033] In connection with the in Fig. Figure 6 schematically illustrates an exemplary embodiment in which a programmable, non-linear monotonic activation function is implemented using a reference table. In this exemplary embodiment, programmable table entries T0, T1, T2... specify the partial sum range (PS range) for specific output values: • z j < T0, then f = 0 • < z j < T1, then f = 1 • < z j < T0, then f = 2 etc.

[0034] At the in Fig. The implementation shown in Figure 6 contains the programmable reference table T0, T1, T2, ..., and an arrangement of comparators compares table entries with the partial sum element z. j A selector logic chooses the function value output according to the comparator results. It should be obvious that the configuration consists of Fig. 6 can be applied to a reference table of any size.

[0035] By comparing only the most significant or only the least significant bits, the logic size can be reduced. This works because output activation has limited precision (e.g., 4 bits, while the reference table only needs 15 entries). In this way, non-linear arbitrary functions such as the sigmoid function can be implemented without a complex arithmetic unit.

[0036] According to another embodiment of the disclosure, a vector matrix multiplication is structured as a pipeline, and the combined circuit with the activation function circuit is also structured as a pipeline. Pipeline structuring can be implemented to increase the clock frequency and performance of the logic circuits. Since the vector matrix multiplication circuit consists of comparatively large circuits, it is well suited for pipeline structuring.

[0037] The Fig. 7A and Fig. Figure 7B presents exemplary embodiments where input activation is distributed by broadcasting or a systolic pipeline. The vector matrix multiplication unit, consisting of m dendritic trees (which are medium-sized circuits), can be optimized for input broadcasting. Additionally, input activations can be passed by broadcasting. In some embodiments, all dendritic trees begin the computation together, and all output results are generated simultaneously. Alternatively, or in addition, input activations can be passed via a systolic pipeline. For example, an activation input can be passed from one latch to another on each clock cycle and delivered to the dendritic trees on a subsequent clock cycle.In such scenarios, some delay may occur, but throughput can be improved by providing a faster clock frequency.

[0038] In connection with the in Fig. Figure 8 schematically illustrates an exemplary embodiment in which an adder tree is configured as a binary tree of adders. According to embodiments of the present disclosure, an adder tree can be implemented in various ways. One exemplary implementation of an adder tree uses a binary tree. This is advantageous because current circuit synthesis tools can be used to generate well-optimized circuits. For a low-precision adder tree, this approach provides sufficient results.

[0039] In connection with the in Fig. Figure 9 schematically illustrates an exemplary embodiment in which an adder tree is implemented using carry-memory adders. Carry-memory adders are advantageous because they have a short cycle time, receive three inputs and produce two outputs whose sum equals the sum of the inputs, and because they are frequently used in multiplier circuits. For example, if s0 and c0 are two outputs of a CSA: Sum vector: s0=CSA−S(a0,a1,a2) Carry vector: c0=CSA−C(a0,a1,a2)

[0040] Then s0 + c0 = a0 + a1 + a2

[0041] In the exemplary embodiment shown, a Wallace adder tree is generated using carry-store adders, and the output consists of two vectors whose sum equals the sum of all inputs. To obtain the final summation result, the two output vectors must be added one last time by a normal adder. An additional CSA may be present to add a previously computed partial sum before the final addition is performed. One advantage of this method is that the runtime is shorter than that of a simple adder tree.

[0042] In some implementations, an "n × m" weighting matrix W is used with only {-1, 0, 1} as possible values. A ternary weighting matrix with 4-bit neurons can achieve high precision with suitable training algorithms. An example "4 × 4" matrix is ​​as follows: W=(0−11010−10101−10111)

[0043] The Fig. 10A and Fig. Figure 10B presents exemplary embodiments in which a ternary multiplier is implemented by a multiplexer. A multiplexer can be provided here as a replacement for a multiplier if the weighting is a ternary value: w ij ∈ {-1, 0, 1}. The example shown gives x i w ij from, using a two-way multiplexer. • Expenditures x i if w ij = 1 • Expenditure -x i if w ij = -1 • If no control signal is enabled (“on”), the output consists of the default value 0.

[0044] This can be explained as follows and as in Fig. Optimize as shown in 10B: Use of x l (one's complement of x) i ) instead of -x i (Two's complement). Since -x i = x l+1 means 1 must be added elsewhere. The 1 fits well if a CSA tree is used as the adder tree, since the CSA-C output of a CSA is shifted 1 to the left. Additionally, w ij represented by a 2-bit representation [b1b0]. (1 :

[01] , 0 :

[00] , -1

[10] .) b0 and b1 can then be used as an input to the multiplexer.

[0045] According to another embodiment of the disclosure, a weighting matrix can be stored in a compressed format and decoded as needed. Here, the ternary weight value can be compressed because a ternary representation contains less information (than two binary bits). In some embodiments, an 8-bit binary value can represent five ternary values. For example, a compressed 8-bit value ω is reduced to five ternary values ​​w 0:4 decoded, using the following equation. w0=ω%3−1 w1=(ω / 3)%3−1 w2=(ω / 9)%3−1 w3=(ω / 27)%3−1 w4=(ω / 81)%3−1 where a % b is the remainder from the division of a by b.

[0046] Immediately before the vector matrix multiplication is performed, the compressed weights are decoded. This results in reduced storage and data transmission overhead (e.g., only 8 bits are needed to represent five ternary values ​​instead of 10). It should be clear that this example compression algorithm is provided for illustrative purposes and that other compression algorithms can also be used to compress the weight matrix. Additionally, sparse matrices can be compressed to achieve a better compression ratio.

[0047] Fig. Figure 11 provides an exemplary embodiment in which a shifter is used to obtain vector matrix multiplication with higher precision. In some cases, it is desirable to use a higher-precision computation when evaluating a neural network. For example, input activation data can be represented by 8-bit values. An 8-bit activation value x can be represented by: x[7:0]=xL+(xM<<4) where x L the four least significant bits is and x M The four most significant bits.

[0048] XW can be calculated by repeating the vector matrix multiplication, since: XW=XLW+XMW<<4

[0049] The partial sum V t-1 = X L W is calculated and shifted four bits to the left, resulting in the new partial sum Z = X MW is added. An array of shifters located in the path from the partial sum registers to the adder arrays can be used to add shifted partial sums to the new vector matrix multiplication result.

[0050] Accordingly, there are a number of vector matrix multipliers that apply a multiplication-addition operation element by element and compute sequentially. According to preferred embodiments, the present disclosure addresses a low-precision vector multiplication for which a parallel architecture can be used to compute the entire vector multiplication at once. Thus, the novel parallel aspect disclosed here is particularly advantageous compared to prior art methods.

[0051] In various embodiments, the Fig. 5 circuits shown, as in Fig. Figure 12 is shown. In particular, the inputs, n-element vector X and W, are shown. j (j-th column of matrix W), distributed to n multipliers 1202, which can be configured in a parallel orientation. Each multiplier computes a product, the products of which are added together by the adder tree 1204 to form a single sum. The output of the adder tree is provided to the activation function 1206 to obtain the neuron activation. In some embodiments, this function is programmable and can be defined as a non-linear or linear function, such as the identity function.

[0052] According to one embodiment of the disclosure, the presented systems and methods compute a vector multiplication, which can be followed by a non-linear or linear activation function. In particular, embodiments of the present disclosure provide a computation of neural activations. Additionally, embodiments of the present disclosure provide a pooling operation in neural networks (NNs), which is essential and time-consuming. Furthermore, embodiments of the present disclosure provide a data mixing operation in neural network (NN) inference. These computations are achieved by extending the dendritic tree circuits described above.

[0053] In connection with the in Fig. Figure 13 schematically illustrates an exemplary data mixing operation. For example, a butterfly permutation is provided as follows: For an input vector X = (0 1 2 3), the permuted output result Y = (1 0 3 2). In some embodiments, a mixing operation can be implemented by: defining the activation function as the identity function (id(x) = x); defining the weighting matrix W as a permutation matrix. For example, only one element in a row and column has the value 1, and all other elements have the value 0. An exemplary implementation of a butterfly network is shown in Fig. 14A illustrates this. When the matrix multiplication operation is performed with the one in Fig. When the matrix W shown in Figure 14A is applied, the input vector X = (0,1,2,3) is permuted to an output vector Y = (1,0,3,2). An example second-degree butterfly network is shown in Fig. 14B illustrates this. When the matrix multiplier operation is combined with the one in Fig. When the matrix W shown in 14B is performed, the input vector X = (0,1,2,3) is permuted to an output vector Y = (2,3,0,1).

[0054] In connection with the in Fig. Figure 15 schematically illustrates an exemplary embodiment in which, according to embodiments of the present disclosure, data from several vectors are packed into a single vector. This feature can be particularly useful for tensor downsampling. For the purpose of illustration, and not as a limitation, the vectors X A = (0 1 2 3) and X B = (4 5 6 7) is packed into a single vector Y = (0 2 4 6) by selecting alternative values. In this exemplary embodiment, the following vector matrix multiplication unit 1502 is used, where the activation function is defined as the identity function (id(x) = x). • First, perform: Z = X A W0 and V = Z + 0 • W0=(1000000001000000) • Secondly, perform V = V + X B W1 • W1=(0010000000010000)

[0055] The vector register contains the previously calculated partial sum (V) 1 ). This partial sum is fed back to the array of adders 1504, which calculates the new partial sum (X B W1) then added. The one in Fig. The data stream shown on the right consists of an arrangement of adders, registers, feedback paths and an arrangement of activation functions 1506, which are defined as the identity function.

[0056] In connection with the in Fig. Figure 16 schematically illustrates an exemplary embodiment in which maximum value pooling operations are performed using the vector matrix multiplication unit, as described in embodiments of the present disclosure. This feature can be particularly useful in neural networks for reducing large images to a smaller size. For illustrative purposes, and not as a limitation, a 2 × 2 maximum value pooling operation for X is shown. A = (0 3 2 1), X B = (3 1 2 2), X C = (4 0 1 0) and X D = (0 1 4 0). In the illustrated exemplary embodiment, an element-wise maximum value is used to obtain Y = (4 3 4 2). In this exemplary embodiment, the vector matrix multiplication unit can be used as follows: • Define the activation function as the identity function. • Use the identity matrix Wid=(1000010000100001). • Combine the current partial sum (Z) with the previously calculated partial sum (V) using the feedback path and the element-wise maximum value operation. • These operations can be described as follows: 1. V = max (X A W id , 0) 2. V = max (X B W id , V) 3. V = max (X C W id , V) 4. V = max (X D W id , V) 5. Y = id(V t-1 )

[0057] In connection with the in Fig. Figure 17 schematically illustrates an exemplary embodiment in which mean-pooling operations are performed using the vector matrix multiplication unit, as described in embodiments of the present disclosure. This feature can also be particularly useful in neural networks for reducing large images to a smaller size. For illustrative purposes, and not as a limitation, a 2x2 mean-pooling operation for X is shown. A = (0 3 2 1), X B = (3 1 2 2), X C = (4 0 1 0) and X D = (1 0 3 1). An element-wise average is used to obtain Y = (2 1 2 1). In this exemplary embodiment, the vector matrix multiplication unit can be used as follows: • Define the activation function as multiplication by 1 / 4. • Use the identity matrix W id . • Simply add the previously calculated partial sum (V) to the new vector matrix product (Z = XW) using the feedback path. • Expressed as a formula, the following is done: 1. V = X A W id + 0 2. V = X B W id + V 3. V = X C W id + V 4. V = X D W id + V 5. Y = (V * 1 / 4)

[0058] In connection with the in Fig. Figure 18 schematically illustrates an exemplary embodiment in which element-wise multiplication operations are performed using the vector matrix multiplication unit. This feature is crucial in some neural networks, such as LSTM (Long Short-Term Memory) networks. For illustrative purposes, and not as a limitation, an element-wise multiplication operation for X is shown. A = (0 3 2 1) and XB = (3 1 2 2) is performed, the result of which is Y = (0 9 4 2). In this exemplary embodiment, the vector matrix multiplication unit can be used as follows: • Define the activation function as the identity function. • Use the identity matrix W id . • The feedback is combined by an arrangement of multiplier units to calculate the new vector matrix product (Z = XW) from the product of the previously calculated partial sums (V). • Expressed as a formula, the following is done: 1. V = X A W id * 1 2. V = X B Wi d * V 3. Y = id(V)

[0059] In connection with the in Fig. Figure 19 schematically illustrates an exemplary embodiment in which a programmable operation combines previous partial sums with several element-wise operations. For illustrative purposes, and not as a limitation, the vector unit can perform element-wise functions such as addition, maximization, multiplication, translation, and bypassing. Additionally or alternatively, these functions can be provided in conjunction with a translation (of a single input), e.g.: • add(shift(A), B), max(shift(A), B), mult(shift(A), B), shift(A) • SIMD (Single Instruction, Multiple Data) operation • The same operation is applied to all inputs into the vector unit in parallel. • The vector unit is programmable to perform any operation in each cycle.

[0060] Accordingly, the vector unit has several selectable element-wise operators, and the input path has a selectable move operator.

[0061] Accordingly, embodiments of the present disclosure provide a unit comprising a vector matrix multiplication unit with programmable element-wise operators and element-wise shift operations. In some embodiments, the data logic is structured as a pipeline.

[0062] In relation to Fig. Figure 20 illustrates a method for operating a neural processing kernel according to embodiments of the present disclosure. In Figure 20, a plurality of equally sized groups of multipliers apply a plurality of weights to a plurality of input activations in parallel to generate a plurality of outputs for each group of multipliers. In Figure 20, the plurality of outputs from each group of multipliers are summed in parallel to generate a partial sum from each group of multipliers. In some embodiments, Figure 20, a function is applied to the partial sums in parallel to generate a plurality of output values. In some embodiments, the function applied to the partial sums includes vector addition, multiplication, shift, and maximum value operations that generate composite partial sums.As shown, in some implementations in 2001, additional partial sums can be calculated. In 2004, an activation function is applied to the partial sums in parallel to generate multiple output activations.

[0063] In relation to Fig. Figure 21 shows a schematic representation of an example of a data processing node. Data processing node 10 is merely an example of a suitable data processing node and is not intended to limit the scope or functionality of the embodiments described herein. Regardless, data processing node 10 can be implemented with and / or perform any of the functionality described herein.

[0064] Data processing node 10 contains a computer system / server 12 that can be operated with numerous other general-purpose or specialized data processing system environments or configurations. Examples of known data processing systems, environments, and / or configurations that may be suitable for use with a computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop units, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, mini-computer systems, mainframe computer systems, and distributed cloud computing environments that incorporate any of the above systems or units, and the like.

[0065] The computer system / server 12 can be generally described in terms of instructions that can be executed by a computer system, such as program modules that are run by a computer system. In general, program modules can contain routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement certain abstract data types. The computer system / server 12 can be used in distributed cloud computing environments, where tasks are performed by remotely located processing units connected via a data transmission network. In a distributed cloud computing environment, program modules can reside in both local and remotely located computer system storage media, such as RAM units.

[0066] As in Fig.As shown in Figure 21, the computer system / server 12 is depicted in the data processing node 10 as a general-purpose data processing unit. The components of the computer system / server 12 can include one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components, such as the system memory 28, to the processor 16, but are not limited to these.

[0067] Bus 18 represents at least one of any several types of bus structures, such as a memory bus or memory controller, a peripheral bus, an Accelerated Graphics Port (AGP), and a processor or local bus, where any one of a variety of bus architectures can be used. By way of example, and not intended as a limitation, such architectures include an ISA bus (Industry Standard Architecture), an MCA bus (Micro Channel Architecture), an EISA bus (Enhanced ISA), a local VESA bus (Video Electronics Standards Association), a PCI bus (Peripheral Component Interconnect), Peripheral Component Interconnect Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

[0068] The computer system / server 12 typically contains a variety of media that are readable by a computer system. Such media can be any available media that the computer system / server 12 can access, and can include both volatile and non-volatile, removable and non-removable media.

[0069] The system memory 28 can contain a computer system-readable medium in the form of volatile memory such as RAM 30 (Random Access Memory) and / or cache memory 32. The computer system / server 12 can also contain other removable / non-removable, volatile / non-volatile computer system storage media. For example, a storage system 34 can be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, commonly referred to as a "hard disk"). Although not shown here, a magnetic disk drive can be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), as well as an optical disk drive for reading from or writing to a removable, non-volatile optical disk, such as a floppy disk.A CD-ROM, a DVD-ROM, or another optical medium may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. As further detailed below, the memory 28 may contain at least one program product with a set of (e.g., at least one) program modules configured to perform the functions of embodiments of the disclosure.

[0070] For example, and without being intended as a limitation, a program / utility 40 with a set of (at least one) program modules 42 can be stored in the main memory 28, as can an operating system, one or more application programs, other program modules, and program data. The operating system, the one or more application programs, the other program modules, and the program data, or a combination thereof, can each contain an implementation of a network environment. The program modules 42 generally perform the functions and / or procedures of the embodiments described herein.

[0071] The computer system / server 12 can also exchange data with one or more external units 14, e.g., a keyboard, a pointing device, a display 24, etc.; with one or more units that allow a user to interact with the computer system / server 12; and / or with any units (e.g., network card, modem, etc.) that enable the computer system / server 12 to exchange data with one or more other data transmission units. Such data transmission can take place via input / output interfaces (I / O interfaces) 22. Furthermore, the computer system / server 12 can exchange data with one or more networks, e.g., a local area network (LAN), a wide area network (WAN), and / or a public network (e.g., the Internet), via a network adapter 20. As shown, the network adapter 20 exchanges data with the other components of the computer system / server 12 via the bus 18.It should be clear that – although not shown here – other hardware and / or software components could also be used in conjunction with the computer system / server 12. Examples include, but are not limited to, microcode, unit drivers, redundant processing units, external disk drive stacks, RAID systems, tape drives, and data archiving storage systems, etc.

[0072] In various embodiments, one or more (not shown) neurosynaptic nuclei are connected to bus 18. In such embodiments, a neurosynaptic nucleus can receive data from or write data to the main memory 28 via bus 18. Similarly, a neurosynaptic nucleus can exchange data with other components via bus 18, as described here. In various embodiments, a neurosynaptic nucleus can contain one or more local controllers, main memory, or clock generators, as described, for example, elsewhere herein.

[0073] The present disclosure may include a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium (or media) containing computer-readable program instructions to induce a processor to execute aspects of the present disclosure.

[0074] A computer-readable storage medium can be a physical unit capable of containing and storing instructions for use by an instruction execution unit. For example, a computer-readable storage medium can be an electronic storage unit, a magnetic storage unit, an optical storage unit, an electromagnetic storage unit, a semiconductor storage unit, or any suitable combination thereof, without limitation. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, RAM, ROM, and erasable programmable read-only memory (EPROM).Flash memory), static random-access memory (SRAM), a portable CD-ROM, a DVD (Digital Versatile Disc), a USB flash drive, a floppy disk, a mechanically coded unit such as punched cards or raised structures in a groove on which instructions are stored, and any suitable combination thereof. For the purposes of this usage, a computer-readable storage medium shall not be understood as volatile signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses guided by an optical fiber cable), or electrical signals transmitted by a wire.

[0075] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to individual data processing units or, via a network such as the internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission lines, wireless transmission, routing computers, firewalls, switching units, gateway computers, and / or edge servers. A network adapter card or network interface in each data processing unit receives computer-readable program instructions from the network and forwards them for storage on a computer-readable storage medium within the respective data processing unit.

[0076] Computer-readable program instructions for performing operations of the present disclosure may be assembly instructions, ISA (Instruction Set Architecture) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., as well as conventional procedural programming languages ​​such as the programming language "C" or similar programming languages.The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, via the internet using an internet service provider).In some embodiments, electronic circuits, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can execute computer-readable program instructions by using state information from the computer-readable program instructions to personalize the electronic circuits to perform aspects of the present disclosure.

[0077] Aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams or diagrams of processes, devices (systems), and computer program products according to embodiments of the disclosure. It is noted that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be executed by means of computer-readable program instructions.

[0078] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or any other programmable data processing device to create a machine such that the instructions executed by the processor of the computer or other programmable data processing device will generate a means of implementing the functions / steps specified in the block(s) of the flowchart and / or block diagram.These computer-readable program instructions may also be stored on a computer-readable storage medium capable of controlling a computer, programmable data processing device and / or other units to function in a particular manner, such that the computer-readable storage medium on which instructions are stored has a manufactured product, including instructions that implement aspects of the function / step specified in the block(s) of the flowcharts and / or block diagrams.

[0079] The computer-readable program instructions can also be loaded onto a computer, other programmable data processing device or other unit to cause the execution of a series of process steps on the computer or other programmable device or other unit in order to produce a computer-implemented process, such that the instructions executed on the computer, other programmable device or other unit implement the functions / steps specified in the block(s) of the flowcharts and / or block diagrams.

[0080] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, processes, and computer program products according to various embodiments of the present disclosure. In this context, each block in the flowcharts or block diagrams can represent a module, segment, or part of instructions that includes one or more executable instructions for performing the specific logical function(s). In some alternative embodiments, the functions specified in the block may occur in a different order than shown in the figures. For example, two blocks shown consecutively may in reality be executed essentially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the corresponding functionality.It should also be noted that each block of the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, can be implemented by special hardware-based systems that perform the specified functions or steps, or execute combinations of special hardware and computer instructions.

[0081] The descriptions of the various embodiments of this disclosure are provided for illustrative purposes and are not to be understood as complete or limited to the disclosed embodiments. Those skilled in the art know that numerous modifications and adaptations are possible without deviating from the scope and spirit of the described embodiments. The terminology used here has been chosen to explain, as clearly as possible, the principles of the embodiments, their practical application, or technical improvements over commercially available technologies, and to enable other skilled persons to understand the embodiments disclosed herein.

Claims

System comprising: a plurality of multipliers (206; 302; 504; 1202), wherein the plurality of multipliers is arranged in a plurality of equally sized groups, each of the plurality of multipliers being designed to apply a weight to an input activation in parallel to produce an output; a plurality of adders (204; 304; 506; 1204), each of the plurality of adders being operatively connected to one of the groups of multipliers, each of the plurality of adders being designed to add the outputs of the multipliers within their respective groups in parallel to produce a partial sum (306); a first plurality of function blocks (508;1206), wherein each of the first plurality of function blocks is operatively connected to one of the plurality of adders, wherein each of the first plurality of function blocks is designed to apply a function in parallel to the partial sum of its associated adder to produce an output value; a vector register (116), wherein the vector register is operatively connected to the first plurality of function blocks, wherein the vector register is designed to store the output values ​​of the first plurality of function blocks, wherein the first plurality of function blocks is designed to combine the output values ​​stored in the vector register with subsequently calculated output values ​​of the first plurality of function blocks, wherein output values ​​of this combination are stored in the vector register;a second plurality of function blocks, each of which is operatively connected to the vector register, each of which is designed to apply a function to the stored output values ​​in parallel. System according to claim 1, which is designed to receive a matrix of weights and a vector of activations. System according to claim 1, wherein each of the plurality of adders comprises a tree of adders. System according to claim 3, wherein the tree of adders is a binary tree. System according to claim 3, wherein the tree of adders comprises a plurality of carry-store adders. System according to claim 2, wherein each activation of the vector of activations is broadcast to all groups of multipliers. System according to claim 2, further comprising a systolic pipeline that is operatively connected to each of the groups of multipliers. System according to claim 1, wherein the groups of multipliers are structured as a pipeline. System according to claim 1, wherein the weights are balanced ternary values. System according to claim 1, wherein each of the plurality of multipliers comprises a multiplexer. System according to claim 2, wherein the matrix of weights is compressed and wherein the system is designed to decompress the compressed matrix of weights. System according to claim 1, wherein each of the plurality of multipliers comprises a ternary multiplier realized by a multiplexer. System according to claim 1, further comprising: a plurality of shifters, wherein each shifter is operatively connected to one of the first plurality of function blocks, wherein each shifter is designed to shift the output value of its corresponding function block in parallel, and wherein the first plurality of function blocks is designed to combine the shifted values ​​with subsequently calculated output values. System according to claim 1, wherein the function of each of the first plurality of function blocks is an activation function. System according to claim 1, wherein the function of each of the first plurality of function blocks is programmable. System according to claim 1, wherein the function of each of the first plurality of function blocks is an addition. System according to claim 1, wherein the function of each of the first plurality of function blocks is a multiplication. System according to claim 1, wherein the function of each of the first plurality of function blocks is an identity function. System according to claim 1, further comprising a reference table, wherein the function of each of the first plurality of function blocks has a reference from the reference table. System according to claim 19, wherein the reference table is programmable. System according to claim 1, wherein the function of each of the first plurality of function blocks is a maximum value function. System according to claim 1, wherein the function of each of the first plurality of function blocks is a minimum value function. System according to claim 1, wherein the function of each of the second plurality of function blocks is the identity function. System according to claim 1, wherein the function of each of the second plurality of function blocks is an activation function. System according to claim 1, which is designed to perform a permutation operation. System according to claim 1, which is designed to perform a vector packing operation. System according to claim 1, which is designed to perform a maximum value pooling operation. System according to claim 1, which is designed to perform mean pooling operations. System according to claim 1, which is designed to perform an element-wise multiplication operation. Method comprising: parallel application (2001) of a plurality of weights to a plurality of input activations by a plurality of equally sized groups of multipliers (206; 302; 504; 1202) to generate a plurality of outputs for each group of multipliers; parallel addition (2002) of the plurality of outputs from each group of multipliers to generate a partial sum from each group of multipliers; parallel application (2003) of a function to the partial sums to generate a plurality of output values; storage of the plurality of output values ​​by a vector register (116); combination of the stored plurality of output values ​​with a subsequently generated plurality of output values ​​and storage of the combined plurality of output values ​​in the vector register; and parallel application (2004) of a function to the stored output values. Method according to claim 30, wherein the combining comprises: parallel shifting of the output value of each of the first plurality of function blocks (508; 1206); and combining the shifted values ​​with the subsequently calculated output values. System comprising: a plurality of multipliers (206; 302; 504; 1202), wherein the plurality of multipliers is arranged in a plurality of equally sized groups; a plurality of adders (204; 304; 506; 1204), wherein the plurality of adders is operatively connected to one of the groups of multipliers; a first plurality of function blocks (508; 1206), wherein each from the first plurality of function blocks is operatively connected to one from the plurality of adders; a vector register (116), wherein the vector register is operatively connected to the first plurality of function blocks; a second plurality of function blocks, wherein each from the second plurality of function blocks is operatively connected to the vector register;A computer-readable storage medium containing program instructions, wherein the program instructions are executable to perform a procedure, comprising: applying a weight to an input activation in parallel by each of the plurality of multipliers to produce an output; adding the outputs of the multipliers within their respective groups in parallel by each of the plurality of adders to produce a partial sum (306); applying a function to the partial sum of its associated adder in parallel by each of the plurality of function blocks to produce an output value; storing the output values ​​of the first plurality of function blocks by the vector register; combining the stored output values ​​with subsequently calculated output values ​​and storing the combined output values ​​by the vector register;and parallel application of a function to the stored output values ​​by each of the second plurality of function blocks. System according to claim 32, further comprising: a plurality of shifters, wherein each shifter is operatively connected to one of the first plurality of function blocks, wherein the combining comprises: parallel shifting, by each shifter, of the output value of its corresponding function block; and combining the shifted values ​​with the subsequently calculated output values ​​by the first plurality of function blocks.

Images