Hardware for parallel layer norm computation

Abstract

A system and method for performing layer normalization are described. A circuit can receive a sequence of input data over multiple clock cycles, where the sequence of input data represents a portion of an input vector. The circuit can determine multiple sums and multiple sums of squares corresponding to the sequence of input data. The circuit can determine a first scalar representing the inverse square root of the variance of the vector elements in the input vector based on the multiple sums of squares. The circuit can determine a second scalar representing the negation of the product of the first scalar and the mean of the vector elements in the input vector. The circuit can determine an output vector that is a normalization of the input vector based on the first scalar, the second scalar, and the received sequence of input data.

Description

[Background technology]

[0001] This application relates generally to analog memory-based artificial neural networks, and more particularly to techniques for computing layer normalizations to normalize distributions of hidden layers in analog memory-based artificial neural networks.

[0002] An artificial neural network (ANN) can include multiple layers of nodes, such as an input layer, one or more hidden layers, and an output layer. Each node can be connected to another node and has an associated weight and threshold. If the output of any individual node exceeds a specified threshold, that node is activated and sends data to the next layer of the network; otherwise, the data is not passed to the next layer of the network. ANNs rely on training data to learn and improve their accuracy over time. Once an ANN is fine-tuned for accuracy, it can be used to classify and cluster data.

[0003] Analog memory-based neural networks, for example, may utilize the storage capacity and physical properties of memory devices to implement artificial neural networks. This type of in-memory computing hardware increases speed and energy efficiency, offering potential performance improvements. Rather than moving data from a memory device to a processor to perform a calculation, an analog neural network chip can perform the calculation in the same location where the data is stored (e.g., in analog memory). Without data movement, tasks can be performed faster and require less energy. Summary of the Invention

[0004] This summary of the present disclosure is provided to aid in understanding systems and methods of dedicated digital computing hardware for precision-reduced layer norm computation in analog memory-based artificial neural networks that can provide efficiencies, without intending to limit the disclosure or the invention. It should be understood that various aspects and features of the present disclosure can be advantageously used separately in some cases or in combination with other aspects and features of the present disclosure in other cases. Accordingly, variations and modifications may be made to the systems and / or their methods of operation to achieve different results.

[0005] In one embodiment, an integrated circuit for performing layer normalization is generally described. The integrated circuit may include a plurality of circuit blocks and a digital circuit. Each circuit block of the plurality of circuit blocks may be configured to receive a sequence of input data over a plurality of clock cycles. The sequence of input data may represent a portion of an input vector, and each input data of the sequence may include data elements representing a subset of vector elements in the portion of the input vector. Each circuit block of the plurality of circuit blocks may be further configured to determine a plurality of sums corresponding to the sequence of input data. Each sum of the plurality of sums may be a sum of a subset of the vector elements in a corresponding input data. Each circuit block of the plurality of circuit blocks may be further configured to determine a plurality of sums of squares corresponding to the sequence of input data. Each sum of squares of the plurality of sums of squares may be a sum of the squares of a subset of the vector elements in a corresponding input data. Each circuit block of the plurality of circuit blocks may be further configured to output the plurality of sums and the plurality of sums of squares to the digital circuit. The digital circuit may be configured to determine an average of the vector elements in the input vector based on the plurality of sums. The digital circuit may be further configured to determine a first scalar representing the inverse square root of the variance of the vector elements in the input vector based on the multiple sums of squares. The digital circuit may be further configured to determine a second scalar representing the negation of the product of the first scalar and the inverse square root mean of the variance of the vector elements in the input vector based on the multiple sums of squares. The digital circuit may be further configured to output the first scalar and the second scalar to the multiple circuit blocks. Each circuit block of the multiple circuit blocks may be further configured to determine vector elements of an output vector based on the first scalar, the second scalar, and the sequence of the received input data. The output vector may be a normalization of the input vector.

[0006] Advantageously, an integrated circuit in one aspect can utilize parallel processing to perform layer normalization to reduce latency and improve throughput of layer normalization operations in artificial neural network applications.

[0007] In one embodiment, a system for performing layer normalization is generally described. The system may include an integrated circuit having a first crossbar array of memory elements, a second crossbar array of memory elements, a plurality of circuit blocks, and the digital circuit. Each circuit block of the plurality of circuit blocks may be configured to receive a sequence of input data from the first crossbar array of memory elements over a plurality of clock cycles. The sequence of input data may represent a portion of an input vector, and each input data of the sequence may include data elements representing a subset of vector elements in the portion of the input vector. Each circuit block of the plurality of circuit blocks may be configured to determine a plurality of sums corresponding to the sequence of input data. Each sum of the plurality of sums may be a sum of a subset of the vector elements in a corresponding input data. Each circuit block of the plurality of circuit blocks may be configured to determine a plurality of sums of squares corresponding to the sequence of input data. Each sum of squares of the plurality of sums of squares may be a sum of the squares of a subset of the vector elements in a corresponding input data. Each circuit block of the plurality of circuit blocks can be configured to output the plurality of sums and the plurality of sums of squares to the digital circuit. The digital circuit can be configured to determine the mean of the vector elements in the input vector based on the plurality of sums. The digital circuit can be further configured to determine a first scalar representing the inverse square root of the variance of the vector elements in the input vector based on the plurality of sums of squares. The digital circuit can be further configured to determine a second scalar representing the negation of the product of the first scalar and the inverse square root mean of the variance of the vector elements in the input vector based on the plurality of sums of squares. The digital circuit can be further configured to output the first scalar and the second scalar to the plurality of circuit blocks.Each circuit block of the plurality of circuit blocks can be further configured to determine vector elements of an output vector based on the first scalar, the second scalar, and the sequence of received input data. The output vector can be a normalization of the input vector. Each circuit block of the plurality of circuit blocks can be further configured to output the output vector to a second crossbar array of memory elements.

[0008] Advantageously, a system in one aspect can utilize parallel processing to perform layer normalization to reduce latency and improve throughput of layer normalization operations in artificial neural network applications. In one embodiment, a method for performing layer normalization is generally described. The method can include receiving a sequence of input data over multiple clock cycles from a first crossbar array of memory elements. The sequence of input data can represent a portion of an input vector, with each input data in the sequence including data elements representing a subset of vector elements in the portion of the input vector. The method can further include determining a plurality of sums corresponding to the sequence of input data. Each sum of the plurality of sums can be a sum of a subset of the vector elements in a corresponding input data. The method can further include determining a plurality of sums of squares corresponding to the sequence of input data. Each sum of squares of the plurality of sums of squares can be a sum of the squares of a subset of the vector elements in a corresponding input data. The method can further include determining an average of the vector elements in the input vector based on the plurality of sums. The method may further comprise determining a first scalar representing the inverse square root of the variance of the vector elements in the input vector based on the multiple sums of squares. The method may further comprise determining a second scalar representing the negation of the product of the first scalar and the mean of the vector elements in the input vector based on the multiple sums of squares. The method may further comprise determining vector elements of an output vector based on the first scalar, the second scalar, and the sequence of the received input data. The output vector may be a normalization of the input vector. The method may further comprise outputting the output vector to a second crossbar array of memory elements.

[0009] Advantageously, in one aspect, the method can utilize parallel processing to perform layer normalization to reduce latency and improve throughput of layer normalization operations in artificial neural network applications.

[0010] Further features, as well as the structure and operation of various embodiments, are described in detail below with reference to the accompanying drawings, where like reference numbers indicate identical or functionally similar elements. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 illustrates an analog memory-based device implementing a hardware neural network in one embodiment.

[0012] [Figure 2] FIG. 1 illustrates details of an analog memory-based device capable of implementing dedicated digital computing hardware for parallel layer norm computation in one embodiment.

[0013] [Figure 3A] FIG. 10 illustrates details of a digital circuit capable of implementing the first stage of a layer normalization operation performed by dedicated digital computing hardware for parallel layer norm computation in one embodiment.

[0014] [Figure 3B] FIG. 3B is a timing diagram of the first stage shown in FIG. 3A in one embodiment.

[0015] [Figure 4A] FIG. 10 illustrates details of a digital circuit capable of implementing the second stage of the layer normalization operation performed by dedicated digital computing hardware for parallel layer norm computation in one embodiment.

[0016] [Figure 4B]FIG. 10 illustrates another implementation of the second stage of the layer normalization operation performed by dedicated digital computing hardware for parallel layer norm computation in one embodiment.

[0017] [Figure 4C] FIG. 4B is a timing diagram of the second stage shown in FIG. 4A in one embodiment.

[0018] [Figure 4D] 4D is a continuation of the timing diagram shown in FIG. 4C in one embodiment.

[0019] [Figure 5A] FIG. 10 illustrates details of a digital circuit capable of implementing the third stage of the layer normalization operation, implemented by dedicated digital computing hardware for parallel layer norm computation, in one embodiment.

[0020] [Figure 5B] FIG. 5B is a timing diagram of the third stage shown in FIG. 5A in one embodiment.

[0021] [Figure 6] FIG. 10 is a timing diagram of a layer normalization operation performed by dedicated digital computing hardware for parallel layer norm computation in one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] A deep neural network (DNN) can be an ANN with a relatively large number of hidden or intermediate layers between an input layer and an output layer. Due to the large number of intermediate layers, training a DNN can involve a relatively large amount of parameters. Layer normalization ("LayerNorm") is a technique for normalizing the distribution of intermediate layers in a deep neural network (DNN). In one aspect, layer normalization can be an operation performed in a transformer (e.g., a neural network that transforms a sequence into another sequence) of a DNN. Layer normalization can enable smoother gradients, faster training, and better generalization accuracy.

[0023] Layer normalization can normalize the output vector from a particular DNN layer across vector elements using the mean and standard deviation of the vector elements. Because the length of the vector to be normalized can be relatively large (e.g., 256 to 1024 elements or more), it is desirable to provide fast end-to-end latency and high throughput in the layer normalization operation. Large latency can delay processing in subsequent layers, and throughput can be constrained by the limits imposed by layer normalization.

[0024] Some conventional solutions for performing layer normalization on large vectors may involve microprocessors or multiprocessors and instruction set architectures that utilize memory space. However, such utilization of memory devices may be relatively energy inefficient. To provide fast end-to-end latency and high throughput for layer normalization operations, the systems and methods described herein may provide dedicated computational hardware that can efficiently compute means and standard deviations over relatively large vectors and exchange intermediate sum information to handle even larger vectors. The computed means and standard deviations can be used in layer normalization operations with reasonable throughput and energy efficiency.

[0025] FIG. 1 illustrates an analog memory-based device implementing a hardware neural network in one embodiment. An analog memory-based device 114 (“device 114”) is shown in FIG. 1. Device 114 may be a coprocessor or accelerator, and device 114 may sometimes be referred to as an analog fabric (AF) engine. One or more digital processors 110 may communicate with device 114 to facilitate the operation or functionality of device 114. In one embodiment, digital processor 110 may be a field programmable gate array (FPGA) board. Device 114 may also interface to components, such as digital-to-analog converters (DACs), that can provide power, voltage, and current to device 114. Digital processor 110 may implement digital logic to interface with device 114 and other components, such as DACs.

[0026] In one embodiment, device 114 may include multiple multiply accumulate (MAC) hardware having a crossbar structure or array. These may be multiple crossbar structures or arrays that may be configured as multiple tiles, such as tile 102. While FIG. 1 shows two MAC hardware (two tiles), there may be additional (e.g., more than two) MAC tiles integrated into device 114. By way of example, tile 102 may include electronic devices such as multiple memory elements 112. The memory elements 112 may be located at the intersections of the crossbar array. At each intersection or junction of the crossbar structure or crossbar array, there may be at least one memory element 112, including analog memory elements such as resistive random access memory (ReRAM), conductive bridge RAM (CBRAM), NOR flash, magnetic random access memory (MRAM), and phase change memory (PCM). In one embodiment, such analog memory elements may be programmed to store synaptic weights of an artificial neural network (ANN).

[0027] In one aspect, each tile 102 can represent a layer of an ANN. Each memory element 112 can be connected to a respective one of a plurality of input lines 104 and a respective one of a plurality of output lines 106. The memory elements 112 can be configured in an array on the surface of the substrate with a fixed distance between crossing points in the horizontal and vertical dimensions. Each tile 102 can perform vector-matrix multiplication. By way of example, the tile 102 can include peripheral circuitry such as a pulse width modulator at 120 and a readout circuit 122.

[0028] Electrical pulses 116 or voltage signals can be input (or applied) to the input lines 104 of the tiles 102. Based on the input pulses or voltage signals 116 applied to the input lines 104 and the values ​​(synaptic weights) stored in the memory elements 112, output currents can be obtained from the output lines 106 of the crossbar structure, for example, according to a multiply-accumulate (MAC) operation.

[0029] The tile 102 may include n input lines 104 and m output lines 106. A controller 108 (e.g., a global controller) may program memory elements 112 to store synaptic weight values ​​of the ANN, e.g., to have conductivity (or resistance) representative of such values. The controller 108 may include (or be connected to) a signal generator (not shown) to couple input signals (e.g., to apply pulse durations or voltage biases) to the input lines 104 or directly to the outputs.

[0030] In one embodiment, readout circuitry 122 can be connected or coupled to read out m output signals (currents) obtained from the m output lines 106. Readout circuitry 122 can be implemented with multiple analog-to-digital converters (ADCs). Readout circuitry 122 can read the currents output directly from the crossbar array, which can be provided to another hardware or circuitry 118 that can process the currents, such as to perform compensation or determine errors.

[0031] The processor 110 can be configured to input a set of input activation vectors to the crossbar array (e.g., via the controller 108). In one embodiment, the set of input activation vectors input to the tile 102 are encoded as electrical pulse durations. In another embodiment, the set of input activation vectors input to the tile 102 can be encoded as voltage signals. The processor 110 can also be configured to read output activation vectors from the plurality of output lines 106 of the tile 102 via the controller 108. The output activation vectors can represent the output of an operation (e.g., a MAC operation) performed on the crossbar array based on the set of input activation vectors and synaptic weights stored in the memory elements 112. In one aspect, the input activation vectors are multiplied by values ​​(e.g., synaptic weights) stored in the memory elements 112 of the tile 102, and the resulting products are accumulated (added) column by column to generate an output activation vector for each one of the columns (output lines 106).

[0032] FIG. 2 illustrates details of an analog memory-based device capable of implementing dedicated digital computation hardware for efficient parallel layer norm computation in one embodiment. In the embodiment shown in FIG. 2, device 114 may further include multiple compute cores (CCs) 200. The compute cores 200 may be inserted between tiles 102 and configured to perform auxiliary operations not easily performed on an analog crossbar structure (e.g., the array of memory elements 112 in FIG. 1). Some examples of auxiliary operations include, but are not limited to, rectifier linear activation functions (ReLUs), element-wise addition, element-wise multiplication, mean pooling, max pooling, batch normalization, layer normalization, lookup tables, and other types of operations not easily performed on an analog crossbar structure. Each compute core 200 may be a digital circuit composed of multiple integrated circuits (ICs), and each IC within a compute core 200 may be assigned to perform a specific auxiliary operation.

[0033] In one embodiment, each CC 200 located between tiles 102 in device 114 may include a vector processing unit (VPU) 210 configured to perform a layer normalization auxiliary operation. Layer normalization may be an auxiliary operation for normalizing the distribution of hidden layers in a deep neural network (DNN). The VPU 210 may be an IC including digital circuit components such as adders, multipliers, static random access memory (SRAM) and registers (e.g., accumulators), and / or other digital circuit components that may be used to perform the auxiliary operation.

[0034] The VPU 210 may receive an input vector 202 from a tile of tiles 102. The VPU 210 may normalize the input vector 202 across the vector elements in the input vector 202 using the mean and standard deviation of the vector elements. The normalized vector may be the output vector 230. In one embodiment, the input vector 202 may be a vector output from a layer of a DNN, and the output vector 230 may be a vector input to the next layer of the DNN. The input vector 202 may be a vector consisting of multiple vector elements x k , the output vector 230 is

number

[0035] The VPU 210 may be implemented as a pipelined vector computation engine having three stages, such as stage 1, stage 2, and stage 3 shown in FIG. 2 . Stage 1 may be implemented by digital circuitry 212. The digital circuitry 212 may include a plurality of circuit blocks 214 (e.g., W circuit blocks 214), including, for example, circuit blocks 214-1, 214-2, ..., 214-W. Each one of the circuit blocks 214 may be identical to one another (e.g., include identical components), and each one of the circuit blocks 214 may be configured to implement a processing pipeline for P cycles (e.g., clock cycles). In each of the P cycles, each one of the circuit blocks 214 may perform a processing pipeline for a vector element x kThe vector elements of Q can be received in parallel, and each one of circuit blocks 214 can generate partial sums A and partial sums B based on the received vector elements of Q. In one aspect, W (selection of the number of circuit blocks 214), Q (selection of how many elements to process in parallel), and P (selection of the number of time-multiplexed calculations that each circuit block 214-1, 214-2 can be expected to start for input vector 202) are such that the product W * Q * P can be arbitrarily selected to match the width of input vector 202 so as to ensure that all vector elements x k are properly processed.

[0036] The partial sums B can be made values that can be used by the VPU 210 of compute core 200 to estimate a scalar C representing the inverse square root of the variance of the vector elements x k and the partial sums A can be made values that can be used by the VPU 210 of compute core 200, together with the scalar C, to estimate a scalar D representing the negation of the product of the average of the vector elements x k and the scalar C. In each one of the P cycles, each one of circuit blocks 214 can output respective partial sums A and respective partial sums B to stage 2. For example, in each one of the P cycles, circuit block 214-1 can output partial sum A1 and partial sum B1 to stage 2, and circuit block 214-2 can output partial sum A2 and partial sum B2 to stage 2. Therefore, after performing stage 1 over the P cycles, stage 1 can output the total of (P × W) partial sums A and (P × W) partial sums B to stage 2.

[0037] In one embodiment, for example, if input vector 202 includes 512 vector elements (e.g., N=512), digital circuit 212 includes eight circuit blocks (e.g., W=8), each one of circuit blocks 214 configured to receive and process four vector elements in parallel (e.g., Q=4), in which case stage 1 can be performed by circuit blocks 214 over 16 cycles (e.g., P=16), and each one of circuit blocks 214 can process a total of 64 vector elements after 16 cycles (Q=4 per cycle). After performing stage 1 over 16 cycles, stage 1 can output a sum of 128 first partial sums and 128 second partial sums to stage 2. In one embodiment, the Q vector elements received at circuit blocks 214 can be in half-precision floating-point (FP16) format.

[0038] Stage 2 may be implemented by digital circuitry 216. Digital circuitry 216 may be configured to implement a processing pipeline over P cycles. In each of the P cycles, digital circuitry 216 may receive a partial sum A of W and a partial sum B of W. In each of the P cycles, digital circuitry 216 may sum the partial sum B of W, which may be calculated as a vector element x k can be used to estimate a scalar C that represents the inverse square root of the variance of m. In each of the m cycles, digital circuitry 216 estimates the vector element x k The partial sum A of W can be summed, which can be used to estimate a scalar D corresponding to the negation of the product of the mean μ of W and the scalar C. Digital circuit 216 can output scalars C and D to circuit block 214 of digital circuit 212.

[0039] Stage 3 may be implemented by circuit blocks 214 of digital circuit 212. Each one of circuit blocks 214 may receive scalars C, D from digital circuit 216. In each of the P cycles, each one of circuit blocks 220 may receive vector element X' of Q of the vector elements. k can be determined in parallel, and the vector element X' of Q k may be vector elements of the output vector 230. The output vector 230 may be a normalized version of the input vector 202, and the output vector 230 may have the same number of vector elements as the input vector 202.

[0040] The values ​​of W and P may be adjustable depending on the size (e.g., number of vector elements) of the input vector 202 (e.g., value of N). In one embodiment, if the input vector 202 includes 1024 vector elements (e.g., N=1024), the digital circuit 212 includes eight circuit blocks (e.g., W=8), and each one of the circuit blocks 214 is configured to receive and process Q vector elements in parallel (e.g., Q=4), then two VPUs 210 (or two compute cores 200) can perform stages 1, 2, and 3. Each one of the two VPUs can perform stage 1 over 16 cycles (e.g., P=16). In stage 2, the digital circuits 216 in the two VPUs can exchange intermediate values ​​that can be used to determine scalars C and D (described further below). The digital circuits 216 for the two VPUs can determine the same scalars C and D because they correspond to the same input vector. In stage 3, the two VPUs may determine respective sets of vector elements for the output vector 230. For example, one of the two VPUs may determine the 1st through 512th vector elements of the output vector 230, and the other of the two VPUs may determine the 513th through 1024th vector elements of the output vector 230.

[0041] 3A illustrates details of a digital circuit capable of implementing the first stage of a layer normalization operation performed by dedicated digital computing hardware for parallel layer norm computation in one embodiment. An example of an implementation of one circuit block 214 in digital circuit 212 of FIG. 2 is shown in FIG. 3A. In stage 1 of the layer normalization process described herein, circuit block 214 may receive a time-multiplexed sequence of input data, labeled as sequence 302. Each input data in sequence 302 may include at least one vector element (e.g., Q vector elements) of a portion of input vector 202 (e.g., 64 vector elements out of 512 vector elements). In one embodiment, each input data in sequence 302 may be in FP16 format.

[0042] In one example shown in FIG. 3A, circuit block 214 may receive vector elements x1, x2, x3, x4 in cycle 1, then x5, x6, x7, x8 in cycle 2, and the final four vector elements x 61 , x 62 , x 63 , x 64 In response to receiving x1, x2, x3, and x4, circuit block 214 may store x1, x2, x3, and x4 in memory device 304 and input x1, x2, x3, and x4 to fused-multiply-add (FMA) circuit 306. In one embodiment, memory device 304 may be a dual-port static random access memory (SRAM). FMA circuit 306 ... 2 , x2 2 , x3 2 , x4 2 We can determine the square of each vector element, such as x1 2 , x2 2 , x3 2 , x4 2to a floating-point addition (FADD) circuit 310. In one embodiment, the FMA circuit 306 can take three inputs X, Y, and Z and perform X*Y+Z, such that the digital circuit 212 can input a zero "0.0" as the Z input, allowing the FMA circuit 306 to determine the square of the vector element using the vector element as the X and Y inputs. The FADD circuit 310 squares x1 2 , x2 2 , x3 2 , x4 2 and output the sum of the squares as partial sum B. Circuit block 214 may wait a predetermined number of cycles before transferring or loading vector elements x1, x2, x3, and x4 from memory device 304 to FADD circuit 308. FADD circuit 310 may determine the sum of vector elements x1, x2, x3, and x4 and output the sum as partial sum A. Circuit block 214 may output partial sums A and B to digital circuit 216. While specific FMA, FADD, and SRAM units are shown here, other implementations for performing these same mathematical operations may be used or envisioned.

[0043] In one embodiment, the predetermined number of cycles that the circuit block 214 waits is the number of cycles that the FMA circuit 306 will wait before multiplying by 1. 2 , x2 2 , x3 2 , x4 2 The squared x1 2 , x2 2 , x3 2 , x4 2If it takes three cycles to determine x, then circuit block 214 can wait three cycles before transferring vector elements x, x, x, x, and x from memory device 304 to FADD circuit 308. By setting the predetermined number of cycles equal to the number of cycles it takes FMA circuit 306 to determine the square, FADD circuits 308, 310 can determine partial sums A and B in parallel and can parallelize or synchronize the output of partial sums A and B to the digital circuitry. Other implementations can be envisioned that may take more or fewer clock cycles.

[0044] FIG. 3B is a timing diagram of the first stage shown in FIG. 3A in one embodiment. In the timing diagram shown in FIG. 3B, FMA circuit 306 may take three cycles to output the squares. Input data received in cycle 1 may be stored in memory device 304 and processed by FMA circuit 306 during cycles 1-3, and in cycle 4, FMA circuit 306 may output the squares of the input data received in cycle 1 to FADD circuit 310. Input data received in cycle 2 may be processed by FMA circuit 306 during cycles 2-4, and in cycle 5, FMA circuit 306 may output the squares of the input data received in cycle 2 to FADD circuit 310. The final set of input data received in cycle 16 may be processed by FMA circuit 306 during cycles 16-18, and in cycle 19, FMA circuit 306 may output the squares of the input data received in cycle 19 to FADD circuit 310.

[0045] Circuit block 214 (FIGS. 2 and 3A) may wait three cycles to transfer or load input data representing vector elements from memory device 304 to FADD circuit 308. FADD circuit 308 may receive input data from memory device 304, and in the same cycle, FADD circuit 310 may receive the squares of the input data from FMA circuit 306. FADD circuits 308 and 310 may take three cycles to determine and output partial sums A and B. As shown in FIG. 3B, the squares output by FMA circuit 306 in cycle 4 may be processed by FADD circuit 308 during cycles 4-6 to determine partial sum A1, which corresponds to the input data received in cycle 1. Additionally, input data received in cycle 1 and stored in memory device 304 may be transferred or loaded to FADD circuit 310 in cycle 4. The input data transferred from memory device 304 may be processed by FADD circuit 310 during cycles 4-6 to determine B1, which corresponds to the input data received in cycle 1. FADD circuits 308, 310 may output partial sums A1, B1 to digital circuit 216 in cycle 7. As a result of implementing stage 1 as a pipelined process, in cycle 19, FMA circuit 306 may output the square of the last set of input data received in cycle 16. FADD circuits 308, 310 may output partial sums A16, B16 to digital circuit 216 in cycle 22, which correspond to the input data received in cycle 16.

[0046] FIG. 4A illustrates details of a digital circuit capable of implementing the second stage of a layer normalization operation implemented by dedicated digital computing hardware for efficient parallel layer norm computation in one embodiment. In stage 2 shown in FIG. 4A, digital circuit 216 (see FIGS. 2 through 3B) can receive a sequence of partial sums A, B from circuit blocks 214 (see FIGS. 2 and 3A). If there are eight circuit blocks 214, digital circuit 216 can receive eight partial sums A and eight partial sums B per cycle. In the example shown in FIG. 4A, the partial sums A received in the first cycle (or cycle 22 in FIG. 3B) are labeled A11, ... A18, and the partial sums B received in the first cycle (or cycle 22 in FIG. 3B) are labeled B11, ... B18. In the final cycle (e.g., after 16 cycles), digital circuit 216 can receive partial sums A 161, ... A 168 and partial sums B 161, ... B 168. After 16 cycles, digital circuit 216 can receive the sum of 128 partial sums A and 128 partial sums B.

[0047] In response to receiving partial sums B in each cycle, digital circuit 216 can determine an intermediate sum of the received partial sums B. In the example shown in FIG. 4A , FADD circuit 402 can sum partial sums B11, ... B14 received from a first set of four circuit blocks 214 to determine intermediate sum S1 = B11 + B12 + B13 + B14. FADD circuit 404 can sum partial sums B15, ... B18 received from a second set of four circuit blocks 214 to determine intermediate sum S2 = B15 + B16 + B17 + B18. S1 and S2 can be provided to FADD circuit 406, which can determine intermediate sum S12 = S1 + S2.

[0048] B11 is B11=x1 2 +x2 2 +x3 2 +x42 The sum of the first four vector elements squared, B18, is B18=x 449 2 +x 450 2 +x 451 2 +x 452 2 Note that intermediate sum S12 determined based on partial sums B11 through B18 is the sum of another set of squares of vector elements, etc. Thus, intermediate sum S12 determined based on partial sums B11 through B18 is the sum of 32 squares of the 512 vector elements of input vector 202 (see FIG. 2). Intermediate sum S12 determined based on partial sums B21 through B28 is the sum of another 32 squares of the 512 vector elements of input vector 202.

[0049] Intermediate sum S12 may be input to FADD 408. FADD 408 may be a looped accumulator, such as a two-wide FADD unit with a loopback, allowing FADD 408 to determine the sum between intermediate sum S12 and a previous value of S12. For example, if intermediate sum S12 determined based on partial sums B11 to B18 is input from FADD circuit 406, FADD circuit 408 may determine the sum of S12 and zero (since there is no previous value of S12). FADD circuit 408 may feed back output S12 determined based on partial sums B11 to B18 to FADD circuit 408 rather than feeding output S12 determined based on partial sums B11 to B18 to a subsequent circuit (e.g., multiplier circuit 410). If the FADD circuit 406 determined input S12 based on partial sums B21 to B28, the FADD circuit 408 may sum S12 determined based on partial sums B21 to B28 with S12 determined based on partial sums B11 to B18, and this updated value of S12 may be fed back to the FADD circuit 408. In one embodiment, additional mantissa bits may be allocated within the FADD circuit 408 to avoid rounding errors in the least significant bits of the mantissa bits. In one embodiment, the multiplier circuit 410 may be a custom divider that uses either a right shift if N is a power of 2, or right shift scaling plus some logic for other values ​​of N (e.g., N=384 or 768). Alternatively, a lookup table or other implementation of a division operation by N may be implemented to cover all possible values ​​of N.

[0050] When FADD circuit 406 determines input S12 based on the last set of partial sums B161 to B168, FADD circuit 408 may sum S12 determined based on partial sums B161 to B168 with S12 determined based on partial sums B151 to B158, and output this updated value of S12 to multiplier circuit 410 rather than feeding it back to FADD circuit 408. After determining the last S12, the final cumulative sum S is the sum of partial sums B11 to B168, where S is the sum of all vector elements x of input vector 202. k The sum of the squares of (e.g., x k 2 Multiplier circuit 410 may receive the final cumulative sum S and multiply S by 1 / N, where N is the number of vector elements in input vector 202 (e.g., if input vector 202 has 512 vector elements, then 1 / N=1 / 512). Multiplier circuit 410 may output the product of 1 / N and S as intermediate value V.

[0051] In response to receiving partial sum A in each cycle, digital circuit 216 can determine an intermediate sum of the received partial sums A. In the example shown in FIG. 4A , FADD circuit 422 can sum partial sums A11, ...A14 received from a first set of four circuit blocks 214 to determine intermediate sum T1 = A11 + A12 + A13 + A14. FADD circuit 424 can sum partial sums A15, ...A18 received from a second set of four circuit blocks 214 to determine intermediate sum T2 = A15 + A16 + A17 + A18. T1 and T2 can be provided to FADD circuit 426, which can determine intermediate sum T12 = T1 + T2.

[0052] A11 is the sum of the first four vector elements, such as A11=x1+x2+x3+x4, and A18 is the sum of the first four vector elements, such as A18=x 449 +x 450 +x 451 +x 452Note that intermediate sum T12 determined based on partial sums A11 through A18 is the sum of another set of vector elements, such as A11 through A18. Thus, intermediate sum T12 determined based on partial sums A11 through A18 is the sum of 32 of the 512 vector elements of input vector 202 (see FIG. 2). Intermediate sum T12 determined based on partial sums A21 through A28 is the sum of another 32 of the 512 vector elements of input vector 202.

[0053] The intermediate sum T12 may be input to the FADD circuit 428. The FADD circuit 428 may be a looped accumulator, such as a two-wide FADD unit with a loopback, so that the FADD circuit 428 can determine the sum between the intermediate sum T12 and a previous value of T12. For example, if the intermediate sum T12 determined based on the partial sums A11 to A18 is input from the FADD circuit 426, the FADD circuit 428 may determine the sum of T12 and zero (because there is no previous value of T12). The FADD circuit 428 may feed back the output T12 determined based on the partial sums A11 to A18 to the FADD circuit 428 rather than feeding the output T12 determined based on the partial sums A11 to A18 to a subsequent circuit (e.g., the multiplier circuit 430). If the FADD circuit 426 determined input T12 based on partial sums A21 to A28, the FADD circuit 428 may sum T12 determined based on partial sums A21 to A28 with T12 determined based on partial sums A11 to A18, and this updated value of T12 may be fed back to the FADD circuit 428. In one embodiment, additional mantissa bits may be allocated in the FADD circuit 428 to avoid rounding errors in the least significant bits of the mantissa bits. In one embodiment, the multiplier circuit 430 may be a custom divider using either a right shift if N is a power of 2, or right shift scaling plus some logic for other values ​​of N (e.g., N=384 or 768).

[0054] When FADD circuit 426 determines input T12 based on the last set of partial sums A161 through A168, FADD circuit 428 may sum T12 determined based on partial sums A161 through A168 with T12 determined based on partial sums A151 through A158, and this updated value of T12 is output to multiplier circuit 430 rather than being fed back to FADD circuit 428. After determining the last T12, the final cumulative sum T is the sum of partial sums A11 through A168, where T is the sum of all vector elements x of input vector 202. k The sum of (e.g., x k Multiplier circuit 410 may receive the final running sum T and multiply T by 1 / N, where N is the number of vector elements in input vector 202. Multiplier circuit 410 may output the product of 1 / N and T as the mean μ, where μ is the mean of the N vector elements of input vector 202.

[0055] The multiplier circuit 410 can output the intermediate value V to the FMA circuit 412, and the multiplier circuit 410 can output the mean μ to the FMA circuit 412. The FMA circuit 412 can receive three inputs, where the intermediate value V can be the first input and the mean μ can be the second and third inputs. The FMA 412 can multiply (μ*μ) by −1 to obtain the variance σ of the N vector elements. 2 = -(μ*μ) + V. Variance σ 2 may be used as an input key to a look-up table (LUT) 414, which may output a scalar C, where the scalar C may be the inverse square root of the variance,

number

[0056] In one embodiment, the LUT 414 may be an FP16 lookup table that includes data bins, each of which may include a range of values. 2 can be input to LUT414 as an input key, and σ 2 Compare σ with the bin edge (e.g., the boundary of the range of bin values) 2 In response to identifying the bin, the digital circuit 216 can obtain a slope value (SLOPE) and an offset value (OFFSET) corresponding to the identified bin and input the SLOPE and OFFSET to the FMA circuit 416. The FMA circuit 416 can calculate a slope value (SLOPE*σ) corresponding to the identified bin. 2 +OFFSET can be determined to estimate the scalar C. Using a lookup table, σ 2 In one embodiment, the digital circuit 216 may prevent the scalar C from approaching infinity when scalar C approaches zero.

number

number

[0057] The FMA circuit 416 can output the scalar C to the FMA circuit 418 of the digital circuit 216. The multiplier circuit 430 can also output the mean μ to the FMA circuit 418. The FMA circuit 418 can determine the product of the mean μ and the scalar C and multiply the product by −1 to determine the scalar D. In one embodiment, the FMA circuit 418 can take three inputs, X, Y, and Z, and perform X*Y+Z, such that the digital circuit 216 can input zero “0.0” as the Z input, allowing the FMA circuit 418 to determine the product D using −μ and the scalar C as the X and Y inputs. The FMA circuit 416 can output the scalar C to the digital circuit 212, and the FMA circuit 418 can output the scalar D to the digital circuit 212 to perform stage 3.

[0058] 4B illustrates another implementation of the second stage of the layer normalization operation performed by dedicated digital computing hardware for efficient parallel layer norm computation in one embodiment. When compute cores 200 (see FIG. 2) are configured to process N vector elements and input vector 202 (see FIG. 2) contains more than N vector elements, more than one compute core 200 can be utilized to perform the layer normalization and generate output vector 230. In one example shown in FIG. 4B, after FADD circuit 408 determines the final intermediate sum S, FADD circuit 408 can provide S to a neighboring VPU labeled VPU1. Furthermore, after FADD circuit 408 determines S, FADD circuit 408 can provide the final intermediate sum S from VPU1. VPU1 Receive S VPU1 and S. If N=1024 (e.g., input vector 202 contains 1024 vector elements) and each compute core 200 can process 512 vector elements, then S is the sum of vector elements x1 through x2. 512 can be expressed as the sum of the squares of S VPU1 is the vector element x 513 From x 1024Therefore, S and S VPU1 The sum of may be the sum of the squares of the 1024 vector elements in the input vector 202.

[0059] Furthermore, after the FADD circuit 428 determines the final intermediate sum T, the FADD circuit 428 can provide T to VPU1. Furthermore, after the FADD circuit 428 determines T, the FADD circuit 428 can provide the final intermediate sum T from VPU1. VPU1 Receive and T VPU1 and T. If N=1024 and each compute core 200 can process 512 vector elements, then T is the sum of vector elements x1 through x2. 512 can be expressed as the sum of T VPU1 is the vector element x 513 From x 1024 Therefore, T and T VPU1 The sum of may be the sum of the 1024 vector elements in the input vector 202.

[0060] Figure 4C is a timing diagram of the second stage shown in Figure 4A in one embodiment. In the timing diagram shown in Figure 4C, each one of the FADD circuits 402, 404, 422, 424 can take three cycles to accumulate four partial sums (four partial sums A or four partial sums B) to determine the intermediate sums S1, S2, T1, T2 in Figure 4A. The partial sums received in cycle 7 can be accumulated by the FADD circuits 402, 404, 422, 424, and the resulting sums from the accumulation can be output to the FADD circuits 406, 426 in cycle 10. Each one of the FADD circuits 406, 426 can take three cycles to determine the intermediate sums S12 and T12 in Figure 4A. The intermediate sums received in cycle 10 may be accumulated by the FADD circuits 406, 426, and in cycle 13, the intermediate sums S12, T12 may be output to the FADD circuits 408, 428.

[0061] Each one of the FADD circuits 408, 428 may take at least three cycles to determine the final values ​​S and T of the intermediate sums S and T, respectively, shown in FIG. 4A . In one or more embodiments, the number of feedback loops used by the FADD circuits 408, 428 to update S and T may determine the number of cycles required for the FADD circuits 408, 428 to determine S and T. For example, if stage 2 receives partial sums A and B over 16 cycles, the FADD circuits 408, 428 may take 16 cycles to complete updating S and T to determine S and T. Furthermore, if more than one VPU is used (e.g., if N is greater than the number of vector elements that can be processed by one compute core 200), the FADD circuits 408, 428 may require additional cycles to swap S and T and add any incoming values ​​of S and T to their own S and T values. 4C, the FADD circuits 408, 428 may take 16 cycles to obtain S and T and may output S and T to the multiplier circuits 410, 430 in cycle 29. Each one of the multiplier circuits 410, 430 may take 1 cycle to multiply the S and T values ​​by 1 / N to obtain the mean value V and the mean μ.

[0062] 4D is a continuation of the timing diagram shown in FIG. 4C in one embodiment. The FMA circuit 412 can receive the mean value V and the mean μ, and the variance σ 2 In cycle 35, the FMA circuit 412 determines the variance σ 2to LUT 414. Digital circuit 216 can use LUT 414 to identify a slope and offset that can be input to FMA circuit 416 and take three cycles to implement FMA circuit 416 to determine scalar C. Scalar C can be output to digital circuit 212 and to FMA circuit 418 in cycle 38. FMA circuit 418 can take three cycles to determine scalar D, which can be output to digital circuit 212 in cycle 41.

[0063] 5A illustrates details of a digital circuit capable of implementing the third stage of the layer normalization operation implemented by dedicated digital computing hardware for efficient parallel layer norm computation in one embodiment. Circuit blocks 214 can implement stage 3 of the layer normalization process described herein. In stage 3, each circuit block 214 can receive scalars C and D from digital circuit 216. The vector elements x of the received sequence 302 of input data (see FIG. 3A) stored in memory device 304 in stage 1 can be used to calculate the layer normalization operation. k can be loaded or transferred to the FMA circuit 502 of the circuit block 214. The FMA circuit 502 calculates the vector element x k and vector element X of output vector 230 based on scalars C and D k can be determined for each vector element X k is x k *C+D. In one embodiment, the FMA circuit 502 calculates the vector element X k can be output from register 504 as a time multiplexed sequence. In one embodiment, vector elements X k can be output to register 504 in FP16 format.

[0064] In one embodiment, stage 3 and a new instance of stage 1 for a new sequence of input data 510 can be performed simultaneously in response to a predetermined condition. As an example, in response to multiplier circuits 410, 430 generating the variance V and mean μ, digital circuit 216 can notify digital circuit 212 that circuit block 214 has received a new sequence 510 so that normalization for the new input vector can begin.

[0065] 5B is a timing diagram of the third stage shown in FIG. 5A in one embodiment. Continuing from stage 2 in FIG. 4D, in cycle 41, circuit block 214 can access scalars C, D and a first set of vector elements x1, x2, x3, and x4 from memory device 304. FMA circuit 502 can take three cycles to generate the corresponding vector elements X1, X2, X3, and X4. In cycle 44, FMA circuit 502 can output vector elements X1, X2, X3, and X4 to register 504. In cycle 56 (e.g., after 16 cycles), FMA circuit 502 can access scalars C, D, and vector elements x1, x2, x3, and x4. 61 , x 62 , x 63 , x 64 Based on the vector element X 61 , X 62 , X 63 , X 64 In cycle 59, the FMA circuit 502 generates the vector element X 61 , X 62 , X 63 , X 64 can be output to register 504.

[0066] In the example embodiment shown herein, it takes approximately 60 cycles to normalize a 512-element input vector using eight circuit blocks 214. The number of vector elements in the input vector, the number of compute cores 200, and the number of circuit blocks 214 in the digital circuit 212 can affect the total amount of time or cycles to normalize the input vector. For example, an input vector with more than 512 vector elements may utilize another compute core 200, with intermediate sums being swapped between different compute cores 200, potentially increasing the amount of time to normalize the input vector. Additionally, the FADD circuits in the digital circuits 212, 216 can be configurable. For example, an FADD circuit that sums four elements can take three cycles to generate the sum, while FADD circuits that sum a different number of elements may use a different number of cycles to generate the sum. Thus, the systems and methods described herein can provide flexibility to normalize vectors of various sizes using different combinations of hardware components.

[0067] Furthermore, the pipelined process in stages 1, 2, and 3, the use of memory device 304 for temporary storage of input vector elements, and the use of lookup tables to estimate scalars can improve the calculation of layer normalization in ANN applications. The parallel computation resulting from the pipelined process can improve throughput and energy efficiency. The compute cores and digital circuits within the compute cores are customized for normalizing vectors having a relatively large number of vector elements, and these customized hardware can be more energy efficient when compared to conventional systems using microprocessors or multiprocessors with conventional memory space and instruction set architectures. Furthermore, by using a dual-port SRAM (e.g., memory device 304), a new set of inputs can enter circuit block 214 to perform a new instance of stage 1 while stage 3 is simultaneously being performed.

[0068] 6 is a flow diagram illustrating a process 600 implemented by dedicated digital computing hardware for efficient parallel layer norm computation in one embodiment. Process 600 in FIG. 6 may be implemented using, for example, device 114 described above. Process 600 may include one or more operations, actions, or functions, such as those illustrated by one or more of blocks 602, 604, 606, 608, 610, 612, 614, and / or 616. Although shown as separate blocks, various blocks may be divided into additional blocks, combined into fewer blocks, eliminated, performed in a different order, or performed in parallel, depending on the desired implementation.

[0069] Process 600 may begin at block 602. In block 602, a circuit may receive a sequence of input data from a first crossbar array of memory elements over multiple clock cycles. The sequence of input data may represent a portion of an input vector, and each input data of the sequence may include data elements representing a subset of vector elements in the portion of the input vector. Process 600 may proceed from block 602 to block 604. In block 604, the circuit may determine a plurality of sums corresponding to the sequence of input data, each sum being the sum of a subset of the vector elements in a corresponding input data.

[0070] Process 600 may proceed from block 604 to block 606. At block 606, circuitry may determine a plurality of sums of squares corresponding to a sequence of input data. Each sum of squares in the plurality of sums of squares may be a sum of squares of a subset of the vector elements in the corresponding input data. In one embodiment, circuitry may determine in parallel a sum of the corresponding input data in the plurality of sums and a sum of the squares of the corresponding input data in the plurality of sums of squares.

[0071] Process 600 may proceed from block 606 to block 608. At block 608, the circuitry may determine the mean of the vector elements in the input vector. Process 600 may proceed from block 608 to block 610. At block 610, the circuitry may determine a first scalar representing the inverse square root of the variance of the vector elements in the input vector based on multiple sums of the squares. In one embodiment, the circuitry may determine the first scalar by using a lookup table. Process 600 may proceed from block 608 to block 612. At block 612, the circuitry may determine a second scalar representing the negation of the product of the first scalar and the mean of the vector elements in the input vector.

[0072] In one embodiment, the circuitry can further receive an intermediate sum of the squares from a neighboring integrated circuit. The circuitry can determine a first scalar based on the plurality of sums of the squares and the received intermediate sum of the squares. The circuitry can receive the intermediate sum of the squares from a neighboring integrated circuit. The circuitry can determine a second scalar based on the plurality of sums and the received intermediate sum of the squares.

[0073] Process 600 may proceed from block 612 to block 614. At block 614, the circuit may determine vector elements of an output vector based on the first scalar, the second scalar, and the sequence of received input data, where the output vector may be a normalization of the input vector. Process 600 may proceed from block 614 to block 616. At block 616, the circuit may output the output vector to a second crossbar array of memory elements. In one embodiment, the circuit may store the sequence of input data in a memory device. The circuit may further obtain the sequence of input data from the memory device to determine the vector elements of the output vector. In one embodiment, the memory device may be a dual-port static random access memory (SRAM).

[0074] In one embodiment, the input vector can be a vector output from a first layer of a neural network implemented by a first crossbar array. The output vector can be a vector that can be input to a second layer of a neural network implemented by a second crossbar array. In one embodiment, the sequence of input data can be a time-multiplexed sequence, and the vector elements of the output data can be output as another time-multiplexed sequence.

[0075] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially simultaneously, or the blocks may even be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or acts or executes a combination of dedicated hardware and computer instructions.

[0076] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the present invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. As used herein, the term "or" is an inclusive operator and can mean "and / or" unless the context expressly or clearly dictates otherwise. It will be further understood that as used herein, the terms "comprise," "comprises," "comprising," "include," "including," and / or "having" can specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the phrase "in one embodiment" may, but does not necessarily, refer to the same embodiment. As used herein, the phrase "in one embodiment" may, but does not necessarily refer to the same embodiment. As used herein, the phrase "in another embodiment" does not necessarily refer to a different embodiment, but may. Furthermore, embodiments and / or components of embodiments may be freely combined with each other unless they are mutually exclusive.

[0077] As used herein, a "module" or "unit" may include hardware (e.g., circuitry such as an application-specific integrated circuit), firmware and / or software executable by the hardware (e.g., by a processor or microcontroller), and / or combinations thereof for performing various operations disclosed herein. For example, the processor or hardware may include one or more integrated circuits configured to perform function mapping or polynomial fitting based on reading currents output from one or more of the output lines of the crossbar array at different times and / or apply the function to subsequent outputs to correct or compensate for temporal conductivity variations in the crossbar array. The same or another processor may include circuitry configured to input activation vectors encoded as electrical pulse durations and / or voltage signals across the input lines for the crossbar array to perform the operations.

[0078] The corresponding structure, material, acts, and equivalents of all means or step-plus-function elements (if any) in the following claims are intended to include any structure, material, or acts for performing a function as specifically claimed in combination with other claimed elements. While the description of the present invention has been presented for purposes of illustration and description, it is not intended to be exhaustive or to limit the invention to the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the invention. These embodiments were chosen and described to best explain the principles and practical applications of the invention, and to enable those skilled in the art to understand the invention in various embodiments with various modifications suited to the particular uses contemplated.

Claims

1. a plurality of circuit blocks; Digital circuits; Equipped with Each circuit block among the plurality of circuit blocks is receiving a sequence of input data over a plurality of clock cycles, wherein the sequence of input data represents a portion of an input vector, and each input data of the sequence includes data elements representing a subset of vector elements in the portion of the input vector; determining a plurality of sums corresponding to the sequence of input data, wherein each sum of the plurality of sums is the sum of a subset of the vector elements in a corresponding input data; determining a plurality of sums of squares corresponding to the sequence of input data, wherein each sum of squares of the plurality of sums of squares is a sum of squares of a subset of the vector elements in a corresponding input data; outputting the plurality of sums and the plurality of sums of squares to the digital circuit; It is configured as follows: The digital circuit determining an average of the vector elements in the input vector based on the plurality of sums; determining a first scalar representing an inverse square root of the variance of the vector elements in the input vector based on the plurality of sums of squares; determining a second scalar representing the negation of the product of the first scalar and the average of the vector elements in the input vector; outputting the first scalar and the second scalar to the plurality of circuit blocks; It is configured as follows: Each circuit block of the plurality of circuit blocks is further configured to determine vector elements of an output vector based on the first scalar, the second scalar, and the sequence of received input data, where the output vector is a normalization of the input vector. Integrated circuit.

2. Each circuit block of the plurality of circuit blocks includes a memory device, and each circuit block of the plurality of circuit blocks includes: storing the sequence of input data in the memory device; Obtaining the sequence of input data from the memory device to determine the vector elements of the output vector.

10. The integrated circuit of claim 1, configured as follows:

3. 3. The integrated circuit of claim 2, wherein the memory device is a dual-port static random access memory (SRAM).

4. the input vector is a vector output from a first layer of a neural network implemented by a first crossbar array of memory elements in an analog memory device; The output vector is a vector that is input to a second layer of the neural network implemented by a second crossbar array of memory elements in the analog memory device.

10. The integrated circuit of claim 1.

5. 5. The integrated circuit of claim 4, wherein each circuit block of the plurality of circuit blocks is configured to determine, in parallel, a sum of corresponding input data of the plurality of sums and a sum of squares of the corresponding input data of the plurality of sums of squares.

6. The integrated circuit of claim 1 , wherein the digital circuitry is configured to determine the first scalar by using a lookup table.

7. The sequence of input data received at each circuit block is a time multiplexed sequence. The vector elements of the output data are output as separate time multiplexed sequences.

10. The integrated circuit of claim 1.

8. The digital circuit receiving intermediate sums of squares from neighboring integrated circuits; determining the first scalar based on the plurality of sums of squares and the received intermediate sums of squares; receiving intermediate sums from the neighboring integrated circuits; determining the second scalar based on the plurality of sums and the received intermediate sums; 10. The integrated circuit of claim 1, configured as follows:

9. a first crossbar array of memory elements; a second crossbar array of memory elements; an integrated circuit having a plurality of circuit blocks and the digital circuit, wherein each circuit block of the plurality of circuit blocks: receiving a sequence of input data from the first crossbar array of memory elements over a plurality of clock cycles, wherein the sequence of input data represents a portion of an input vector, and each input data of the sequence includes data elements representing a subset of vector elements in the portion of the input vector; determining a plurality of sums corresponding to the sequence of input data, wherein each sum of the plurality of sums is the sum of a subset of the vector elements in a corresponding input data; determining a plurality of sums of squares corresponding to the sequence of input data, wherein each sum of squares of the plurality of sums of squares is a sum of squares of a subset of the vector elements in a corresponding input data; outputting the plurality of sums and the plurality of sums of squares to the digital circuit; It is configured as follows: The digital circuit determining an average of the vector elements in the input vector based on the plurality of sums; determining a first scalar representing an inverse square root of the variance of the vector elements in the input vector based on the plurality of sums of squares; determining a second scalar representing the negation of the product of the first scalar and the average of the vector elements in the input vector; outputting the first scalar and the second scalar to the plurality of circuit blocks; It is configured as follows: Each circuit block among the plurality of circuit blocks is determining vector elements of an output vector based on the first scalar, the second scalar and the sequence of received input data, wherein the output vector is a normalization of the input vector; outputting the output vector to a second crossbar array of the memory elements; It is further configured as follows: system.

10. Each circuit block of the plurality of circuit blocks includes a memory device, and each circuit block of the plurality of circuit blocks includes: storing the sequence of input data in the memory device; Obtaining the sequence of input data from the memory device to determine the vector elements of the output vector. The system of claim 9 , configured to:

11. 11. The system of claim 10, wherein the memory device is a dual-port static random access memory (SRAM).

12. the first crossbar array of memory elements implementing a first layer of a neural network; The second crossbar array of memory elements implements a first layer of a neural network. The system of claim 9.

13. 13. The system of claim 12, wherein each circuit block of the plurality of circuit blocks is configured to determine, in parallel, a sum of corresponding input data of the plurality of sums and a sum of squares of the corresponding input data of the plurality of sums of squares.

14. 10. The system of claim 9, wherein the digital circuitry is configured to determine the first scalar by using a lookup table.

15. The sequence of input data received at each circuit block is a time multiplexed sequence; The vector elements of the output data are output as separate time multiplexed sequences. The system of claim 9.

16. The digital circuit receiving intermediate sums of squares from neighboring integrated circuits; determining the first scalar based on the plurality of sums of squares and the received intermediate sums of squares; receiving intermediate sums from the neighboring integrated circuits; determining the second scalar based on the plurality of sums and the received intermediate sums; The system of claim 9 , configured to:

17. receiving a sequence of input data from a first crossbar array of memory elements over a plurality of clock cycles, wherein the sequence of input data represents a portion of an input vector, and each input data of the sequence includes data elements representing a subset of vector elements in the portion of the input vector; determining a plurality of sums corresponding to the sequence of input data, wherein each sum of the plurality of sums is the sum of a subset of the vector elements in a corresponding input data; determining a plurality of sums of squares corresponding to the sequence of input data, wherein each sum of squares of the plurality of sums of squares is a sum of squares of a subset of the vector elements in a corresponding input data; determining a mean of the vector elements in the input vector based on the plurality of sums; determining a first scalar representing the inverse square root of the variance of the vector elements in the input vector based on the plurality of sums of squares; determining a second scalar representing the negation of the product of said first scalar and said average of said vector elements in said input vector; determining vector elements of an output vector based on the first scalar, the second scalar, and the sequence of received input data, wherein the output vector is a normalization of the input vector; and outputting the output vector to a second crossbar array of memory elements. A method for providing the above.

18. storing the sequence of input data in a memory device; and obtaining the sequence of input data from the memory device and determining the vector elements of the output vector; The method of claim 17 further comprising:

19. 18. The method of claim 17, further comprising: determining in parallel sums of corresponding input data among the plurality of sums and sums of squares of the corresponding input data among the plurality of sums of squares.

20. 18. The method of claim 17, further comprising determining the first scalar by using a lookup table.

Images