Semi-programmable and reconfigurable co-accelerators for deep neural networks with normalization or nonlinearity

Abstract

This disclosure relates to a device for using a configurable stacked architecture for a fixed-function data path, having an accelerator for accelerating operations or layers of a deep neural network (DNN). The stacked architecture may have a fixed-function data path comprising one or more configurable micro-execution units that perform a series of vector, scalar, reduction, broadcast, and normalization operations for DNN layer operations. The fixed-function data path may be customized based on the DNN or the operations.

Description

Background Technology

[0001] With the proliferation of deep neural networks (DNNs) for natural language processing (NLP) applications in the cloud, the demand for computational acceleration has grown exponentially. Hardware accelerators have been used to meet this growing demand by offloading most of the computation to custom hardware accelerators. To maximize performance and minimize cost and power consumption, the architecture of these accelerators can be customized to closely match the dataflow graphs of common DNN patterns. However, the performance of existing DNN accelerators is limited when performing computations for NLP. Summary of the Invention

[0002] This summary is provided to introduce some concepts in a simplified form, which will be further described in the detailed implementations below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0003] One example implementation relates to a configurable stacked architecture for a fixed-function data path used in conjunction with an accelerator to accelerate deep neural network (DNN) computations. The stacked architecture may include multiple configurable micro-scalar processing units (SPUs) that perform at least one scalar operation on vector values ​​from received vectors. The stacked architecture may also include multiple configurable micro-multifunctional units (MFUs) that perform vector operations on vector values, wherein multiple configurable micro-SPUs and multiple configurable micro-MFUs are sequentially placed to perform DNN computations, wherein the output of one of the multiple configurable micro-SPUs is provided as the input of one of the multiple configurable micro-MFUs.

[0004] Another example implementation involves an accelerator. The accelerator may include multiple vector register files (VRFs) that provide the accelerator with one or more vectors containing data. The accelerator may include multiple programmable multifunction units (MFUs) that communicate with the VRFs to perform vector operations on vector values ​​from the one or more vectors. The accelerator may include at least one programmable scalar processing unit (SPU). The accelerator may include a configurable stacked architecture with fixed-function data paths communicating with the multiple programmable MFUs, wherein the stacked architecture performs nonlinear operations on vector values ​​to accelerate layers of a DNN.

[0005] Another example implementation involves a configurable stacked architecture for a fixed-function data path used in conjunction with an accelerator to accelerate softmax or layer normalization operations in a deep neural network (DNN). The stacked architecture may include a first micro-scalar processing unit (SPU) that performs maximum reduction, scalar inverse, and broadcast operations on vector values ​​received from a programmable multifunction unit (MFU) of the accelerator. The stacked architecture may also include a first micro-multifunction unit (MFU) that receives vector values ​​from a first micro-SPU and performs vector multiplication on the vector values. The stacked architecture may further include a second micro-SPU that receives vector values ​​from the first micro-MFU and performs reduction, scalar multiplication, and broadcast operations on the vector values. Finally, the stacked architecture may include a second micro-MFU that receives vector values ​​from a second micro-SPU and performs vector subtraction, square root, and exponentiation on the vector values. The stacked architecture may include a third micro SPU that receives vector values ​​from the second micro MFU and performs reduction, scalar multiplication, scalar addition, scalar square root, scalar inverse, and broadcast operations on the vector values. The stacked architecture may also include a third micro MFU that receives vectors from the third micro SPU and performs vector multiplication on the vector values.

[0006] Additional features and advantages will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practice of the teachings herein. The features and advantages of this disclosure may be realized and obtained by the means and combinations particularly pointed out in the appended claims. The features of this disclosure will become more apparent from the following description and the appended claims, or may be learned by practice of this disclosure as set forth below. Attached Figure Description

[0007] To describe in more detail the ways in which the above and other features of this disclosure can be obtained, reference will be made to the specific implementations illustrated in the accompanying drawings. For better understanding, similar elements are indicated by similar reference numerals throughout the drawings. While some of the drawings may be schematic or exaggerated representations of concepts, at least some of the drawings are drawn to scale. It is to be understood that the drawings depict example implementations that will be described and explained with additional features and details using the drawings, in which:

[0008] Figure 1 The diagram illustrates the existing architecture of the accelerator.

[0009] Figure 2The illustration shows an example stacking architecture with fixed-function data paths for use with accelerators, based on some implementations.

[0010] Figure 3 The diagram illustrates an example microarchitecture based on fixed functional data paths implemented in some way.

[0011] Figure 4 The illustration shows example configurable fixed-function instructions for use with accelerators, based on some implementations.

[0012] Figure 5 The illustration shows an example architecture of an accelerator based on some implementations, which uses a stacked architecture with fixed-function data paths.

[0013] Figure 6 The diagram illustrates a block diagram of a system, based on some implementations, comprising nodes interconnected via a data center network for use with an accelerator.

[0014] Figure 7 The diagram illustrates a block diagram of a system, based on some implementations, that includes distributed nodes for use with an accelerator.

[0015] Figure 8 The diagram illustrates a block diagram of a hardware node used in conjunction with an accelerator, based on some implementations. Detailed Implementation

[0016] This disclosure generally relates to accelerators. With the proliferation of deep neural networks (DNNs) for natural language processing (NLP) applications in the cloud, the demand for computational acceleration is growing exponentially. Hardware accelerators are being used to meet this growing demand by offloading most of the computation to custom hardware accelerators. To maximize and minimize cost and power consumption, the architecture of these accelerators can be customized to closely match the dataflow graph of common DNN patterns. A common DNN pattern popular in the NLP field is a normalized nonlinear DNN layer, which typically limits the performance of DNN accelerators.

[0017] A well-known example of normalization and / or nonlinear operations is the softmax function, which is common in many DNN models. The softmax function performs many different atomic operations: element-wise vector addition, multiplication, exponentiation, reduction of vector elements (e.g., summation, maximization), and scalar operations such as inversion and broadcasting the scalar result to a vector. Running atomic operations in a general-purpose vector / scalar processor, the softmax function must be computed using many instructions, increasing code size and decoding complexity. Furthermore, data dependencies between instructions make it difficult to fill the execution pipeline without sacrificing efficiency. This problem is particularly severe in general-purpose processors, where processors execute many different types of instructions, and normalization operations (such as mean and standard deviation calculations) are very expensive due to frequent memory accesses across multiple instructions, thus degrading the performance of domain-specific processors used for DNN acceleration.

[0018] exist Figure 1 The figure illustrates an example of performance degradation of a DNN accelerator when the softmax function is executed. Figure 1 The illustration shows a current architecture 100 of a DNN accelerator with a Multifunctional Unit (MFU) and a Scalar Processing Unit (SPU). The softmax operation executes multiple chained vector / scalar instructions via MFU 1 / MFU 2 and the SPU, using multiple iterations to produce multiple instruction chains. The execution pipeline between data-dependent chains remains idle, leading to low hardware utilization and degrading the overall performance of the DNN accelerator. Thus, the performance of current DNN accelerators is limited when performing computations on any DNN model with softmax and / or any normalization layers. Transformer-based DNN accelerators also suffer from performance limitations when performing computations for NLP.

[0019] The device described in this paper provides a semi-programmable co-accelerator that enables the execution of Softmax and layer normalization operations by leveraging pipeline and single-instruction, multiple-data (SIMD) parallelism in an area-efficient and high-performance architecture. The device described in this paper can be used for any nonlinear DNN layer operation involving vector, scalar, and / or reduction operations, rather than conventional tensor and activation neural layers.

[0020] The device described in this paper maximizes the utilization of vector / scalar processing data paths in DNN acceleration by directly transferring data between pipelined execution units without requiring round trips to memory, and by customizing the number, type, and order of execution units to the nonlinear kernel so that most execution units can be utilized simultaneously during execution.

[0021] The device described in this paper stacks programmable vector / scalar processing units and hardwired but configurable vector / scalar processing units in a pipeline. The device architecture described in this paper maximizes pipeline utilization by increasing pipeline depth (as opposed to other techniques that primarily or exclusively rely on data parallelism), ensuring that current DNN layers can execute at maximum throughput within the complete pipeline without inefficiency between instructions. In some implementations, the architecture of the device described in this paper maximizes pipeline utilization by increasing both pipeline depth and width through parallelism to scale performance by applying SIMD parallelism. This avoids the waste of resources associated with data path control, which rarely changes, but minimizes control path complexity by adding fixed-function instructions and data paths that can execute a given DNN layer (such as softmax and / or layer normalization) within a single instruction.

[0022] Compared to approaches using Reduced Instruction Set Computer (RISC) instruction architectures for vector / scalar processing, the architecture of the described device employs a hybrid of Complex Instruction Set Computer (CISC) and RISC, providing not only general vector programmability with element-wise vector instructions (and further, the addition of programmable scalar units to the execution pipeline), but also fixed-function instructions that optimally execute the required DNN layer operations (such as softmax and layer normalization). The architecture of the described device requires significantly lower area / resources because a series of micro-operations are optimally pre-scheduled in the hardware execution unit pipeline regardless of the software / firmware code. Furthermore, the architecture of the described device can utilize serial and bypass microarchitectures for fixed-function data paths. The structure of the architecture of the described device can have configurable finite vector processing units (e.g., micro-MFUs or u-MFUs) and configurable finite vector-to-scalar, scalar-to-scalar, and scalar-to-vector processing units with forwarding paths (e.g., micro-SPUs or u-SPUs). Micro-MFU and micro-SPU modules can be stacked with this structure to implement a given nonlinear DNN layer to be accelerated, which can then be further optimized through flattening.

[0023] The architecture of the device described in this paper provides programmability and configurability to fully utilize the data paths used for various model and inference parameters. The architecture can execute general element-wise vector instructions (e.g., adding a mask in a scaled dot product operation, adding an arbitrary vector before a layer normalization operation). Furthermore, the accelerator data path of the device described in this paper has its own programmability, enabling the execution of different DNN layer acceleration instructions. For example, the accelerator data path of the device described in this paper is programmable in its connections to a series of vector addition, multiplication, exponentiation, maximum reduction, summation reduction, and broadcast execution units, allowing a given data path to be used to its fullest extent for softmax and layer normalization DNN layers, which can be dynamically configured using instructions. One technical advantage of this programmability is that the architecture of the device described in this paper allows it to support multiple transformer-based models (e.g., models with different numbers of heads) and different queries for inference (e.g., different sequence lengths of input tokens).

[0024] Furthermore, the fixed-function data path of the device architecture described in this paper is a microarchitecture, which includes a series of vector micro-execution units and vector-to-scalar reduction, scalar, and scalar-to-vector micro-execution units with forwarding paths. A technical advantage of this device architecture is its ease of reconfiguration, and its ability to cover a wide range of nonlinear DNN layers that can be implemented using vector / scalar and reduction / broadcast operations.

[0025] Another technical advantage of the device described in this paper is that it improves the performance of DNN accelerators by addressing the problem of low hardware utilization, which occurs particularly at the vector / scalar processing data path when performing normalization and / or nonlinear operations.

[0026] Another technical advantage is that the described architecture is both software-programmable and hardware-configurable, which is particularly helpful in Field Programmable Gate Array (FPGA) platforms. The MFU retains its programmability and can be used for many different purposes (e.g., element-wise vector addition, multiplication, passthrough, masked addition such as in softmax, and / or input addition in layer normalization). The fixed-function data path is also programmable, so it can perform a series of vector / scalar / normalization operations to run softmax, layer normalization, and / or passthrough. The fixed-function data path can also be configured at the hardware level using the proposed microarchitecture, which has a fixed-function data path that can be easily customized depending on the microarchitecture-accelerated DNN model. This is particularly helpful in FPGA platforms, where the fixed-function data path can be reconfigured and optimized to accelerate the DNN model.

[0027] See now Figure 2The illustration shows the accelerator 500 ( Figure 5 This is an example of a programmable and / or configurable stacked architecture 200 for vector operations. The stacked architecture 200 consists of modular general-purpose vector processing blocks (MFU_IVRF / TMM 10, MFU 14), scalar processing blocks with reduction and broadcasting capabilities, and / or one or more vector register files (VRFs) 12 communicating with the MFU 14. The MFU 14 can be programmable and can perform any element-wise vector operation (e.g., addition or multiplication). DNN layer operations can use the MFU 14 to perform general element-wise vector instructions (adding a mask in a scaled dot product operation, adding arbitrary vectors) prior to layer normalization operations. Thus, the MFU 14 can be used for many different purposes (e.g., element-wise vector addition, multiplication, pass-through, such as masked addition in softmax, and / or input addition in layer normalization).

[0028] Additionally, the stacked architecture 200 may include a fixed-function data path 16. The fixed-function data path 16 may include one or more configurable micro-execution units (e.g., configurable micro-SPUs and First-In-First-Out (FIFO) 18, 26 and / or configurable micro-MFUs 20, 28) that perform a series of vector, scalar, reduction, broadcast, and / or normalization operations for DNN layer operations in a single instruction. DNN layer operations may include any nonlinear operations involving vector, scalar, and / or reduction operations, rather than conventional tensor and activation neural layers. Examples of DNN layer operations may include softmax and / or layer normalization. The fixed-function data path 16 is a customizable data path that performs a series of vector, scalar, reduction, and / or broadcast operations for a specific DNN nonlinear layer.

[0029] Fixed-function data path 16 may include a series of micro-operations pre-scheduled in a hardware execution unit pipeline. Multiple configurable micro-execution units (e.g., micro SPUs and FIFOs 18, 26, and micro MFUs 20, 28) perform a series of micro-operations for DNN layer operations. Micro SPUs 18, 26 perform scalar operations, ensuring that vector values ​​have the same scalability. Micro SPUs 18, 26 also perform reduction, broadcast, and / or normalization instructions on vector values. Micro SPUs 18, 26 also support forwarding using FIFOs. Micro MFUs 20, 28 perform vector operations, such as, but not limited to, addition, multiplication, square root, and / or exponentiation.

[0030] The micro-execution units included in the fixed-function data path 16 can support only a limited subset of operations, such as nonlinear operations. By limiting the operations supported by the micro-execution units or restricting the scope of operations performed by the micro-execution units to necessary operations, the fixed-function data path 16 can be further optimized, resulting in better resource and / or area utilization. Micro-MFUs 20 and 28 can include finite vector processing units. Additionally, micro-SPUs 18 and 26 can be finite vector-to-scalar, scalar-to-scalar, and scalar-to-vector processing units with forwarding paths. Thus, the micro-execution units can support only a finite subset of operations performed for the DNN layer, rather than performing all operations.

[0031] Stacked architecture 200 can increase the depth of the accelerator pipeline by layering the operations used to perform vector, scalar, reduction, broadcast, and / or normalization operations. Stacked architecture 200 may include serial and bypass microarchitectures for fixed-function data paths 16.

[0032] Micro-execution units (e.g., micro-SPUs and FIFOs 18, 26, and micro-MFUs 20, 28) can be stacked sequentially to implement a specific nonlinear DNN layer to be accelerated. The fixed-function data path 16 may have a forwarding path, wherein the outputs of micro-SPUs 18, 26 are provided as inputs to micro-MFUs 20, 26 in sequence, and the outputs of micro-MFUs 20, 26 are provided as inputs to micro-SPUs 18, 26. Additionally, the fixed-function data path 16 may include multiple FIFO structures that receive vector outputs from micro-MFUs 20, 26 and forward the vector outputs sequentially to the next micro-MFU 20, 26.

[0033] Micro-execution units (e.g., micro-SPUs and FIFOs 18, 26, and micro-MFUs 20, 28) can be mapped to different operations performed for DNN layer operations. In some implementations, micro-MFUs 20, 28 perform different operations. In some implementations, micro-MFUs 20, 28 perform the same operation. In some implementations, micro-SPUs and FIFOs 18, 26 perform different operations. In some implementations, micro-SPUs and FIFOs 18, 26 perform the same operation.

[0034] The fixed-function data path 16 is programmable to perform a range of vector, scalar, normalization operations, such as, but not limited to, softmax, layerNorm, and / or passthrough. The fixed-function data path 16 can be modified when different DNN models and / or different DNN layer operations need to be supported. The modules used in the fixed-function data path 16 can be optimized depending on the DNN model and / or DNN layer operations to be accelerated.

[0035] One example includes adding one or more micro SPU modules 18, 26 and / or one or more micro MFU modules 20, 28 to the fixed-function data path 16. Another example includes removing one or more micro SPU modules 18, 26 and / or one or more micro MFU modules 20, 28 from the fixed-function data path 16. Another example includes removing one or more micro SPU modules 18, 26 and / or one or more micro MFU modules 20, 28 from the fixed-function data path 16 and adding one or more micro SPU modules 18, 26 and / or one or more micro MFU modules 20, 28. Another example includes skipping one or more micro SPU modules 18, 26 and / or one or more micro MFU modules 20, 28. Another example includes rearranging the order of one or more micro MFU modules 20, 28 and / or micro SPU modules 18, 26. Therefore, the number, type, and / or order of configurable micro-execution units in the fixed-function data path 16 can be changed and / or customized based on DNN layer operations and / or DNN models.

[0036] Therefore, the stacked architecture 200 provides sufficient programmability and / or configurability to fully utilize the fixed-function data path 16 for a variety of model and inference parameters. Architecture 200 maximizes the utilization of vector and / or scalar processing data paths in the DNN accelerator 500 by directly passing data between pipelined execution units without requiring round trips to memory. For example, architecture 200 can perform operations without intermediate memory accesses. Furthermore, architecture 200 customizes the number, type, and / or order of micro-execution units to the nonlinear kernel, allowing most micro-execution units to be used simultaneously during execution. Additionally, modular units (e.g., micro MFUs 20, 28 and micro SPU modules and FIFOs 18, 26) make the design of accelerator 500 easier, more readable, scalable, and easier to debug. The fixed-function data path 16 requires significantly lower area and / or resources because a series of micro-operations are optimally pre-scheduled in the hardware execution unit pipeline, regardless of the software and / or firmware code.

[0037] See now Figure 3 The illustration shows an example microarchitecture 300 for a fixed-function data path 16 used to accelerate softmax operations and / or layer normalization operations. Microarchitecture 300 may include multiple VRFs 12 communicating with an MFU 14 that performs vector-to-vector addition operations. The MFU 14 may be programmable and may perform any vector-by-vector element-wise operation (e.g., addition or multiplication). The output of the MFU 14 is the input to a micro SPU module 18 of the fixed-function data path 16.

[0038] Micro SPU module 18 receives the vector from MFU 14 and performs a maximum reduction operation 30 on the vector to remove the largest element. After performing the maximum reduction operation 30, micro SPU module 18 performs an inverse scalar operation 32 on the vector. Micro SPU module 18 performs a broadcast operation 34 to broadcast the inverse scalar value to the vector. Additionally, a first-in-first-out (FIFO) structure 36 runs in parallel with the reduction and broadcast paths performed by micro SPU module 18. FIFO structure 36 is used to receive the vector from MFU 14 and forward it to micro MFU 20.

[0039] The output from micro SPU module 18 (e.g., a vector with inverse scalar values ​​broadcast by micro SPU module 18) is provided as input to micro MFU module 20. Furthermore, FIFO structure 36 is provided as input to micro MFU module 20. Micro MFU module 20 performs vector-to-vector multiplication on the received vector inputs (e.g., the broadcast vector from micro SPU module 18 and the vector from FIFO structure 36). The output of micro MFU module 20 is provided as input to FIFO structure 44.

[0040] Additionally, the output of micro MFU module 20 is provided as input to micro SPU module 22. Micro SPU module 22 performs a reduction operation 38 on the vector. Micro SPU module 22 performs a scalar multiplication operation 40 on the vector and performs a broadcast operation 42 that broadcasts the scalar multiplication value to the vector. FIFO structure 44 runs in parallel with the reduction and broadcast paths performed by micro SPU module 22. FIFO structure 44 receives the vector output from micro MFU 20 and forwards the vector output to micro MFU module 24.

[0041] The outputs of micro SPU module 22 and FIFO structure 44 are provided as inputs to micro MFU module 24. Micro MFU module 24 performs vector-to-vector subtraction on the vector inputs (e.g., the broadcast vector from micro SPU module 22 and the vector from FIFO structure 44). Additionally, micro MFU module 24 performs element-wise square root function 46 and element-wise exponentiation function 48 on the vectors. The output of micro MFU module 24 is provided as input to micro SPU module 26. Furthermore, the output of micro MFU module 24 is provided to FIFO structure 62.

[0042] Micro SPU module 26 performs a reduction operation 50 on the vector. Micro SPU module 26 also performs scalar multiplication 52 and scalar addition 54 on the vector. Additionally, micro SPU module 26 performs a scalar square root operation 56, a scalar inverse operation 58, and a broadcast operation 60 to broadcast the values ​​to the vector. Thus, micro SPU module 26 performs multiple scalar operations on the vector and broadcasts these values ​​to the vector. FIFO structure 62 runs in parallel with the reduction and broadcast paths performed by micro SPU module 26. FIFO structure 62 receives the vector output from micro MFU 24 and forwards the vector output to micro MFU module 28.

[0043] The outputs of the micro SPU module 26 and the FIFO structure 62 are provided as inputs to the micro MFU module 28, which performs element-wise vector multiplication.

[0044] The output of micro MFU module 28 is provided as input to MFU 14, which performs vector-to-vector multiplication, and MFU 14, which performs vector-to-vector addition. MFU 14 can be programmable and can perform any element-wise vector operation (e.g., addition or multiplication).

[0045] The microarchitecture 300 with fixed-function data path 16 can accelerate softmax operations and layer normalization operations.

[0046] See now Figure 4 The illustration shows the device used with the accelerator 500 ( Figure 5 The example used together can be a fixed-function instruction 400. The fixed-function instruction 400 includes a general format to provide customized operations for the accelerator 500. In the example, the fixed-function instruction 400 provides information for softmax and layer normalization operations in a single instruction.

[0047] Fixed-function instructions 400 can use extended instruction (EI) formants comprising multiple bits 402 and corresponding multiple fields 404. For example, fixed-function instructions include a 64-bit structure with five fields. The first set of eight bits 406 (bits 0-7) can use the OpCode field 408 with the opcode to identify the type of instruction. For example, the opcode could be “v_func”, used to implement any accelerator function. The next set of eight bits 410 (bits 8-15) identifies the specific type of layer of the DNN model used to accelerate the DNN model using the Opd0 field 411. For example, the Opd0 field 411 could be “SoftMax / Layernorm” (softmax operation and Layernorm operation) identifying two sub-layers of a bidirectional encoder representation (BERT) model from a transformer to be accelerated. The Opd0 field 411 can be used to identify additional and / or different layers of the DNN model used for acceleration.

[0048] The next set of eight bits 412 (bits 16-31) includes an Opd1 field 414 containing data to be used with the operation. For example, the Opd1 field 414 may include an epsilon value used in the softmax operation and an fp16 format to be used with the operation. The next set of eight bits 416 (bits 32-47) includes an Opd2 field 418 containing data to be used with the operation. For example, the Opd2 field 418 may include data reserved for 1 / vecLen and floating-point (fp)16 format to be used with the operation. The next set of eight bits 420 (bits 48-63) includes an Opd3 field 422. In some implementations, only the OpCode field 408, Opd0 field 411, Opd1 field 414, and Opd2 field 418 are used. In some implementations, only the OpCode field 408 and Opd0 field 411 are used.

[0049] An example use case for fixed-function instruction 400 is the Extended Instruction (EI) format for accelerating softmax and / or layer normalization. For example, OpCode field 408 is “v_func”. Opd0 field 411 “v_softmax()” b2 is used for the softmax instruction, and Opd0 field 411 “v_layernorm()” is used for the layer normalization instruction. The parameters required for the softmax and / or layer normalization instructions include “VectorLength”, “Epsilon”, and “[1 / VectorLength]” (used to shift 1 / x calculation from SLU). For the softmax instruction, Opd1 field 414, Opd2 field 418, and Opd3 field 422 are not used. An example fixed-function instruction 400 for softmax includes “v_softmax():=v_func(ISA_SubOpCode_Softmax,DONTCARE,DONTCARE)”. For the layer normalization instruction, Opd1 field 414 is "epsilon", Opd2 field 418 is "[1 / VectorLength]", and Opd3 field 422 is not used. An example fixed function instruction 400 for layer normalization includes "v_layernorm(eps,vecLenInv):=v_func(ISA_SubOpCode_LayerNorm,eps,vecLenInv)".

[0050] Example program code used with the softmax directive may include:

[0051] / / seqLen = sequence length;

[0052] / / Iterations = seqLen;

[0053] / / columns=ceil(seqLen / NativeDim);

[0054] SetIterationsCols(bs,iterations,columns);

[0055] SetVectorLength(seqLen);

[0056] v_rd_inc(MfuInitialVrf,ivrf_initial,columns);

[0057] vv_add_inc(AddSubVrf_0,mask,0);

[0058] v_softmax();

[0059] v_wr_inc(MfuInitialVrf,ivrf_result,columns);

[0060] Where “v_softmax()” is OpCode field 408.

[0061] Example program code used with layer normalization directives may include:

[0062] SetIterationsCols(numVecs,hidden_size);

[0063] SetVectorLength(hidden_size*native_dim);

[0064] v_rd_inc(MfuInitialVrf,mfuvrf_input_vectors,hidden_size);

[0065] vv_add_inc(ASVRF0,hidden_state,hidden_size);

[0066] v_layerNorm(epsilon,1 / (hidden_size*native_dim));

[0067] vv_mul(MultiplyVrf,mulvrf_weight);

[0068] vv_add(AddSubVrf_1,asvrf_bias);

[0069] v_wr_inc(MfuInitialVrf,mfuvrf_input_vectors,hidden_size);

[0070] Where “v_layerNorm” is OpCode field 408, “epsilon” is OpCode field 414, and “1 / (hidden_size*native_dim)” is Opd2 field 418.

[0071] Thus, fixed-function instruction 400 can be used to perform operations for fixed-function data path 16 within a single instruction. Figure 2 Instead of using multiple instructions to perform the operation (e.g., softmax and / or layer normalization), a single instruction can be used. This reduces the code size for the operation.

[0072] See now Figure 5 The illustration shows an example architecture of a DNN accelerator 500, which has a programmable and / or configurable stacking architecture for a fixed-function data path 16 to accelerate one or more layers of the DNN. In some implementations, the accelerator 500 receives data from a network 502 for use with a bidirectional encoder representation (BERT) model from a transformer to be accelerated. The accelerator 500 includes multiple VRFs 12 that receive data from vectors for processing. The multiple VRFs 12 may include one or more of single-port VRFs, multi-port VRFs, and / or multiple grouped VRFs. The multiple VRFs 12 communicate with one or more programmable MFUs 14 that perform vector operations. Different MFUs 14 may perform different vector operations. The MFUs 14 may communicate with a Tensor Memory Manager (TMM) 10.

[0073] Additionally, one or more MFUs 14 can communicate with fixed-function data path 16. Fixed-function data path 16 may include one or more configurable micro-execution units (e.g., micro SPUs and FIFOs 18, 26, and micro MFUs 20, 28) that perform a series of vector, scalar, reduction, broadcast, and / or normalization operations on the DNN layer in a single instruction. For example, accelerator 500 receives a fixed-function instruction 400 identifying the DNN layer operation to be performed. Figure 4 ).

[0074] Fixed-function data path 16 is a customizable data path that performs a series of vector, scalar, reduction, and / or broadcast operations for a specific DNN nonlinear layer. Configurable micro-execution unit (MFU) modules (e.g., micro SPUs and FIFOs 18, 26, and micro MFUs 20, 28) can be stacked and / or layered sequentially to implement the specific nonlinear DNN layer to be accelerated.

[0075] By using fixed-function data path 16, the performance of accelerator 500 can be improved by addressing the low hardware utilization problem, which occurs particularly at the vector / scalar processing data path when performing normalization and / or nonlinear operations. Furthermore, by using fixed-function data path 16, accelerator 500 maximizes the utilization of the vector / scalar processing data path in DNN acceleration by directly transferring data between pipelined execution units without needing to travel to and from memory.

[0076] Additionally, Accelerator 500 can provide depth to the pipeline using fixed-function data path 16. The architecture of Accelerator 500 can achieve higher throughput and better efficiency with limited resources and / or area because it can be highly optimized toward the actual DNN layers, i.e., optimally pre-scheduling a series of data operations to be executed in the hardware, regardless of the software / firmware code. Therefore, the architecture of Accelerator 500 achieves high utilization of the entire data path without the inefficiencies that would otherwise occur between many micro-operations and instructions performed in a general-purpose vector / scalar processor, resulting in a significantly reduced width of the vector / scalar data path.

[0077] In some implementations, the Accelerator 500 architecture has been used in Stratix 10GX 2800 FPGA devices, resulting in resource / area reductions of more than 5x compared to previous general-purpose vector / scalar processors. Furthermore, the Accelerator 500 architecture uses a smaller code size, and a given DNN layer can be executed with only a few instructions. For example, for softmax, the number of instructions in the Accelerator 500 is reduced from 20 to 8, and for layer normalization, the number of instructions is reduced from 43 to 10.

[0078] Figure 6 This is a block diagram of an example system 600, which includes nodes interconnected via a data center network 610. For example, as... Figure 6As shown, multiple nodes 602, 604, and 606 can be coupled via a data center network. Such nodes can be instantiated and used to parallelize multiple layers of a neural network, such as a Long Short-Term Memory (LSTM) network. In one example, each node is implemented as a server and may also include at least one hardware node (e.g., an FPGA). Thus, node 602 may include FPGA 622, node 604 may include FPGA 624, and node 606 may include FPGA 626. FPGAs can be interconnected via a system based on an optical transport layer protocol. In one example, a first instance of FPGA 622 is coupled to a first instance of FPGA 624 via transport link 632, and the first instance of FPGA 622 can also be coupled to a second instance of FPGA 624 via transport link 634. A first instance of FPGA 624 can be coupled to a first instance of FPGA 626 via transport link 636, and the first instance of FPGA 624 can also be coupled to the first instance of FPGA 626 via transport link 640. Similarly, a second instance of FPGA 624 can be coupled to a first instance of FPGA 626 via transmission link 642, and a second instance of FPGA 624 can also be coupled to a second instance of FPGA 626 via transmission link 638. The optical transport layer protocol can provide the FPGAs with the ability to transmit or receive packets or other such data from each other via data center network 610. FPGAs can also be interconnected in other configurations. For example, several instances of FPGA 622 can be coupled to several instances of FPGA 624 via multiple transmission links 652. Similarly, several instances of FPGA 624 can be coupled to several instances of FPGA 626 via transmission link 654. Although... Figure 6 A certain number and arrangement of nodes (including FPGAs) are shown, but there may be more or fewer nodes arranged in different ways.

[0079] Figure 7This is a block diagram of a system 700 including distributed nodes, based on an example. In this example, multiple nodes are implemented as server racks in a data center. Each server can be coupled to a top-of-rack (TOR) switch. Other racks, though not shown, may have similar configurations. Each server may include at least one or more nodes. Each node may include a server (e.g., server 704, server 706, or server 708) and each server can be coupled to a TOR switch (e.g., TOR switch 710). Server 704 may include host components including central processing units (CPUs), such as CPU 714 and CPU 716, which can be coupled to hardware nodes, such as FPGA 718, via a local link (e.g., PCIe) 720. Each hardware node can also be coupled via a network interface controller 722 (e.g., used for communication across the data center's network infrastructure). Figure 7 The system illustrated allows nodes to process messages received from (and / or sent to) TOR switches or other switches. Using this example system, individual nodes can send packets of messages directly to each other, and this can therefore allow communication across multiple FPGA partitions or even a single neural network without incurring unacceptable latency. For communication, nodes can use lightweight protocols, including, for example, RDMA. Although... Figure 7 A certain number of components of a system arranged in a certain way are shown, but there may be more or fewer components arranged in different ways.

[0080] Parallelization within a neural network layer can also be performed by partitioning neural weights across multiple nodes. As an example, a single recurrent neural network (RNN) model (e.g., including an LSTM weight matrix) can be partitioned and fixed across multiple nodes. In this example implementation, the RNN model can be distributed across the memory (e.g., BRAM) of each of multiple FPGAs. In this example configuration, each individual FPGA in the multi-stage pipeline can store a portion of the LSTM weight matrix in fast on-chip memory (e.g., BRAM). This can advantageously result in a system with high throughput and low latency. At service startup, the LSTM weight matrix can be decomposed into matrices of a certain size (e.g., an N×M matrix, where each of N and M is an integer equal to or greater than 8) and then loaded into the FPGA's on-chip memory. The runtime management layer can implement the allocation, scheduling, and management of the FPGAs. In one example, each node can be implemented as an LSTM-focused vector processor based on an additional HaaS of one or more FPGAs. Each node can be designed to run neural network evaluations either as a PCIe-connected FPGA or as part of a HaaS pool of FPGAs.

[0081] Figure 8 This is a block diagram based on an example hardware node 800. Each hardware node 800 may include an Input Message Processor (IMP) 810 for receiving messages from other nodes and an Output Message Processor (OMP) 840 for processing outgoing messages destined for other nodes or components. Each node may also include a Control / Scalar Processor (CSP) 820 and a Neural Function Unit (NFU) 830. Although not shown, messages received by a node may be stored in at least two different queues: (1) an IMP-to-CSP auxiliary queue and (2) an IMP-to-NFU data queue. Although not shown, outgoing messages may be stored in at least two different queues: (1) a CSP-to-IMP auxiliary queue and (2) an NFU-to-OMP data queue. In this example, a node may accept an off-chip message containing both auxiliary information (such as control and scalar data) and payload data (e.g., vectors, matrices, or other tensor data structures). The auxiliary information may include a request to perform computationally intensive operations on the payload and then return the results as an output message. In this example, the input message is processed by a lightweight input message processor (IMP) 810, which sends auxiliary information to a control / scalar processor (CSP) 820 (which may be a NIOS-based control processor) and payload data (e.g., an input tensor) to a neural function unit (NFU) 830, which may be implemented as a matrix vector processor. As an example, the CSP 820 can then interpret the request and, based on its firmware, send a series of instructions to the NFU 830. After a certain processing delay, the NFU can produce the requested result, which can be combined with auxiliary data generated by the CSP 820 in a lightweight output message processor (OMP) 840 and then sent off-chip. The CSP firmware can provide instructions to the NFU 830. Further details of the example instructions are discussed as part of the instruction set architecture (ISA). A runtime reload of the CSP 820's firmware can also be performed. Therefore, in this example, the architecture is primarily event-driven. Input messages can arrive from many sources, including via a network. IMP can inspect the head of the message queue and dequeue any instructions that need to be executed, feeding them back to the system. Although Figure 8 The example node is shown with a certain number of components arranged in a certain way, but there may be more or fewer components arranged in different ways.

[0082] In one example, the NFU is implemented as a matrix vector processor designed to scale to most of the FPGA's resources. In this example, the primary hardware acceleration goal of the NFU is to perform matrix vector multiplication with high throughput and low latency by applying thousands of multiply-adders and utilizing its matrix vector units (MVUs). The NFU can receive a matrix of coefficients (e.g., constants) and can be used to multiply these coefficients with dynamic input vector data. Therefore, instead of storing the coefficients in dynamic random access memory (DRAM) corresponding to a CPU / graphics processing unit (GPU), the coefficients can be preloaded into the on-chip memory corresponding to the NFU (e.g., the FPGA's BRAM) at service startup. In one example, once the coefficients are loaded, they may never be reloaded again unless the neural network model being used is modified or the service is restarted. As part of this example, the model can be partitioned in a distributed manner and pinned to the on-chip memory of multiple nodes (e.g., the FPGA) so that they can directly transmit messages or packets to each other without relying on assistance from CPU resources.

[0083] In one example, the MFU is fully pipelined and capable of performing matrix-vector multiplications with O(n) complexity in O(n.sup.2) time, achieving 400-1800 billion fixed-point operations per second. While matrix-vector multiplication can represent most of the fixed-point operations required to evaluate an LSTM layer, evaluation can also include various additions of vector reduction, transcendence, and bias vectors. The NFU can also implement pipelined Multifunction Units (MFUs) to process these O(n) complexity vector functions in O(n) time. These MFUs can be organized into a chained architecture, where the MVU passes data to the first MFU, the first MFU passes data to the second MFU, and so on. In one example implementation of the NFU, one MVU and five MFUs can be used.

[0084] Based on the observation that vector functions may potentially dominate LSTM evaluation time in architectures where vector and matrix-vector operations take approximately the same amount of computation time, the chained architecture allows NFU to leverage large-scale pipeline parallelism between a single matrix-vector multiplication and several vector functions. The NFU's memory subsystem can also be configured to support high throughput. For example, the memory subsystem can support matrix-value throughput up to 1.8 TB / s and support simultaneous loading and storage of 6 vectors.

[0085] The techniques described herein can be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a particular manner. Any features described as modules, components, etc., can also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part through a non-transitory processor-readable storage medium comprising instructions that, when executed by at least one processor, perform one or more of the methods described herein. Instructions can be organized into routines, programs, objects, components, data structures, etc., which can perform specific tasks and / or implement specific data types, and can be combined or distributed as needed in various embodiments.

[0086] Computer-readable media can be any available medium that can be accessed by a general-purpose or special-purpose computer system. A computer-readable medium storing computer-executable instructions is a non-transitory computer-readable storage medium (device). A computer-readable medium carrying computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of this disclosure may include at least two distinct types of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

[0087] As used herein, a non-transitory computer-readable storage medium (device) may include RAM, ROM, EEPROM, CD-ROM, solid-state drive (“SSD”) (e.g., RAM-based), flash memory, phase-change memory (“PCM”), other types of memory, other optical disc storage, disk storage or other magnetic storage devices, or any other medium or data structure that may be used to store desired program code in the form of computer-executable instructions and that may be accessed by a general-purpose or special-purpose computer.

[0088] The steps and / or actions of the methods described herein may be interchanged without departing from the scope of the claims. Unless the correct operation of the described methods requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0089] The term "determine" encompasses a wide variety of actions; therefore, "determine" can include calculation, operation, processing, deduction, investigation, searching (e.g., looking in a table, database, or other data structure), ascertaining, and so on. Furthermore, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and so on. Additionally, "determine" can include resolving, selecting, choosing, establishing, and so on.

[0090] The articles “a,” “an,” and “the” are intended to indicate the presence of one or more elements in the foregoing description. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that additional elements may exist in addition to those listed. Furthermore, it should be understood that references to “an implementation” or “implementation” in this disclosure are not intended to be construed as excluding the existence of additional implementations that also incorporate the listed features. For example, any element described in connection with an implementation herein may be combined with any element of any other implementation described herein. Numbers, percentages, ratios, or other values ​​stated herein are intended to include that value, and also include other values ​​“about” or “approximately” as would be understood by one of ordinary skill in the art as covered by the implementations of this disclosure. Therefore, the values ​​should be interpreted broadly enough to cover values ​​that are at least close enough to perform the desired function or achieve the desired result. The values ​​include at least the variations expected in a suitable manufacturing or production process and may include values ​​within 5%, 1%, 0.1%, or 0.01% of the stated value.

[0091] In view of this disclosure, those skilled in the art should recognize that equivalent constructions do not depart from the spirit and scope of this disclosure, and that various modifications, substitutions, and alterations can be made to the implementations disclosed herein without departing from the spirit and scope of this disclosure. Equivalent structures including functional “device plus function” clauses are intended to cover structures described herein that perform the listed functions, including structural equivalents that operate in the same manner and equivalent structures that provide the same function. The applicant’s explicit intent is not to invoke device plus function or other functional claims for any claim unless the word “device” appears with the relevant function. Every addition, deletion, and modification to the implementation that falls within the meaning and scope of the claims is covered by the claims.

[0092] Industrial applications

[0093] The device's architecture provides a semi-programmable co-accelerator that enables the execution of Softmax and layer normalization operations through pipelined and single-instruction multiple-data (SIMD) parallelism in an area-efficient and performance-effective architecture. The device can be used for any nonlinear DNN layer operation involving vector, scalar, and / or reduction operations, rather than conventional tensor and activation neural layers.

[0094] The device maximizes the utilization of vector / scalar processing data paths in DNN acceleration by directly passing data between pipelined execution units without requiring round trips to memory, and by customizing the number, type, and order of execution units to the nonlinear kernel so that most execution units can be utilized simultaneously during execution.

[0095] These devices stack programmable vector / scalar processing units and hardwired but configurable vector / scalar processing units in a pipeline. The device architecture described in this paper maximizes pipeline utilization by increasing pipeline depth (as opposed to other techniques that rely primarily or exclusively on data parallelism), ensuring that the current DNN layer can execute at maximum throughput in its entirety without inefficiency between instructions. In some implementations, the architecture of the devices described in this paper maximizes pipeline utilization by increasing both pipeline depth and width through parallelism to scale performance by applying SIMD parallelism. Therefore, the resource waste of data path control, which rarely changes, can be avoided, but the complexity of the control path is minimized by adding fixed-function instructions and data paths that can execute a given DNN layer (such as softmax and / or layer normalization) within a single instruction.

[0096] The architecture of the device disclosed herein employs a hybrid of Complex Instruction Set Computer (CISC) and RISC, providing not only general vector programmability with element-wise vector instructions (and further, the addition of programmable scalar units to the execution pipeline), but also fixed-function instructions that optimally execute the required DNN layer operations (such as softmax and layer normalization). This architecture requires significantly lower area / resources because a series of micro-operations are optimally pre-scheduled in the hardware execution unit pipeline regardless of the software / firmware code. Furthermore, the architecture can utilize serial and bypass microarchitectures for fixed-function data paths. The architecture can have configurable finite vector processing units (e.g., micro-MFUs or u-MFUs) and configurable finite vector-to-scalar, scalar-to-scalar, and scalar-to-vector processing units with forwarding paths (e.g., micro-SPUs and FIFOs, or u-SPUs and FIFOs). Configurable micro-MFU and configurable micro-SPU modules can be stacked with this architecture to implement a given nonlinear DNN layer to be accelerated, which can then be further optimized through flattening.

[0097] This architecture provides programmability and configurability to fully utilize the data paths used for various model and inference parameters. The architecture can execute general element-wise vector instructions (e.g., adding a mask during scaling dot product operations, adding arbitrary vectors before layer normalization operations). Furthermore, the accelerator data path of the device described herein has its own programmability, enabling the execution of different DNN layer acceleration instructions. For example, the accelerator data path of the device described herein is programmable in its connections to a series of vector addition, multiplication, exponentiation, maximum reduction, summation reduction, and broadcast execution units, allowing a given data path to be maximized for softmax and layer normalization DNN layers, which can be dynamically configured using instructions. One technical advantage of this programmability is that it allows the device architecture to support various transformer-based models (e.g., models with different numbers of heads) and different queries for inference (e.g., different sequence lengths of input tokens).

[0098] Furthermore, the fixed-function data path of this device's architecture is a microarchitecture, comprising a series of vector micro-execution units and vector-to-scalar reduction, scalar, and scalar-to-vector micro-execution units with forwarding paths. A technical advantage of this architecture is its ease of reconfiguration and its ability to cover a wide range of nonlinear DNN layers that can be implemented using vector / scalar and reduction / broadcast operations.

[0099] Another technical advantage of this device is that it improves the performance of DNN accelerators by addressing the problem of low hardware utilization, which occurs particularly in the vector / scalar processing data path when performing normalization and / or nonlinear operations.

[0100] Another technical advantage is that the device's architecture is both software-programmable and hardware-configurable, which is particularly helpful in Field Programmable Gate Array (FPGA) platforms. The MFU retains its programmability and can be used for many different purposes (e.g., element-wise vector addition, multiplication, pass-through, masked addition such as in softmax, and / or input addition in layer normalization). The fixed-function data path is also programmable, so it can perform a series of vector / scalar / normalization operations to run softmax, layerNorm, and / or pass-through. The fixed-function data path can also be configured at the hardware level using the proposed microarchitecture, which features a fixed-function data path that can be easily customized depending on the microarchitecture-accelerated DNN model. This is particularly helpful in FPGA platforms, where the fixed-function data path can be reconfigured and optimized based on the DNN model for acceleration.

[0101] An example of an existing solution is a wide SIMD vector processor that increases the width of the data path. Compared to existing solutions that increase the width of the data path, the architecture of the device disclosed herein provides pipeline depth. This device architecture offers advantages over existing solutions because it can achieve higher throughput and better efficiency with limited resources and / or area, as it can be highly optimized toward the actual DNN layer, i.e., optimally pre-scheduling a series of data operations to be executed in the hardware, regardless of the software / firmware code. Therefore, the architecture of the device disclosed herein easily achieves high utilization of the entire data path without the inefficiencies that would otherwise occur between many micro-operations and instructions performed in a general-purpose vector / scalar processor, resulting in a significant reduction in the width of the vector / scalar data path. For example, in the Stratix 10GX 2800 FPGA device, the described architecture reduces resources / area by more than 5 times compared to previous general-purpose vector / scalar processors.

[0102] Another technical advantage of the architecture of the device disclosed herein is the smaller code size and simpler decoding logic and / or control path, enabling a given DNN layer to be executed with only a few instructions. For example, the number of instructions for softmax is reduced from 20 to 8, and the number of instructions for layer normalization is reduced from 43 to 10.

[0103] Another example of an existing solution is the use of High-Level Synthesis (HLS) approach. Compared to HLS, the device described herein offers a compact, efficient microarchitecture that is readable and configurable. The architecture of the device disclosed herein is a stacked architecture consisting of modular general-purpose vector processing blocks (e.g., MFUs), scalar processing blocks with reduction and broadcasting (e.g., SPUs), and custom data paths (e.g., fixed-function data paths), which perform a series of vector / scalar / reduction / broadcast operations for the specific DNN nonlinear layer they accelerate. One of the technical advantages of the modular programmable vector / scalar units (e.g., MFUs and SPUs) and modular micro-execution units (e.g., u-MFUs and u-SPUs) in the fixed-function units is the optimization of the modules and the ability to maintain the use of modules and change the custom data paths only when necessary.

[0104] (A1) Some implementations include a configurable stacked architecture (e.g., architecture 200) for a fixed-function data path (e.g., data path 16), used in conjunction with an accelerator (e.g., accelerator 500) to accelerate deep neural network (DNN) operations. The configurable stacked architecture includes multiple configurable microscalar processing units (SPUs) (e.g., micro SPUs 18, 22, 26) that perform at least one scalar operation (e.g., inverse scalar operation 32, scalar multiplication operation 40, scalar multiplication operation 52, scalar addition operation 54, scalar square root operation 56, scalar inverse operation 58) on vector values ​​from received vectors. The configurable stacked architecture also includes multiple configurable micro-multifunctional units (MFUs) (e.g., micro-MFU 20, micro-MFU 28, micro-MFU 28) that perform vector operations on vector values, wherein multiple configurable micro-SPUs and multiple configurable micro-MFUs are placed sequentially to perform DNN operations, wherein the output of one of the multiple configurable micro-SPUs is provided as an input to the multiple configurable micro-MFUs.

[0105] (A2) In some implementations of the stacked architecture of A1, each of the multiple configurable micro SPUs also performs a reduction operation on the vector value, performs at least one scalar operation on the vector value, and performs a broadcast operation to broadcast the vector value to a vector.

[0106] (A3) In some implementations of the stacked architecture of A1 or A2, the operation is a softmax operation or a layer normalization operation.

[0107] (A4) In some implementations of the stacked architecture of any of A1-A3, the operation is a non-linear operation involving at least one of the following: vector operation, scalar operation, or reduction operation.

[0108] (A5) In some implementations of the stacking architecture of any of A1-A4, the number of microSPUs of multiple configurable microSPUs and the number of microMFUs of multiple configurable microMFUs are selected based on computation.

[0109] (A6) In some implementations of the stacked architecture of any of A1-A5, multiple configurable micro SPUs and multiple configurable micro MFUs are selected based on computation.

[0110] (A7) In some implementations of the stacking architecture of any of A1-A6, the order of multiple configurable micro SPUs and multiple configurable micro MFUs is selected based on computation.

[0111] (A8) In some implementations of the stacked architecture of any of A1-A7, multiple configurable micro SPUs and multiple configurable micro MFUs perform operations without intermediate access to the accelerator’s memory.

[0112] (A9) In some implementations, the stacking architecture of any of A1-A8 includes multiple first-in-first-out (FIFO) structures (e.g., FIFO structure 36, FIFO structure 44, FIFO structure 62) that sequentially provide vector values ​​output from the previous micro MFU to the next micro MFU, wherein one of the multiple FIFO structures runs in parallel with each of the multiple micro SPUs.

[0113] (A10) In some implementations, the stacking architecture of any one of A1-A9 includes at least one programmable MFU that communicates with one or more of the plurality of configurable micro SPUs, wherein the output of the at least one programmable MFU is provided as an input to one or more micro SPUs.

[0114] (A11) In some implementations, the stacking architecture of any one of A1-A10 includes a first programmable MFU that communicates with a first micro SPU among a plurality of configurable micro SPUs; a second programmable MFU that communicates with the last micro SPU among a plurality of configurable micro SPUs; and a third programmable MFU that communicates with the second programmable MFU.

[0115] (A12) In some implementations, the stacking architecture of any one of A1-A11 includes at least one programmable SPU.

[0116] (B1) Some implementations include an accelerator (e.g., accelerator 500). The accelerator includes multiple vector register files (VRFs) (e.g., VRF 12) that provide the accelerator with one or more vectors having data. The accelerator includes multiple programmable multifunction units (MFUs) (e.g., programmable MFU 14) that communicate with the VRFs to perform vector operations on vector values ​​from one or more vectors. The accelerator also includes at least one programmable scalar processing unit (SPU). The accelerator also includes a configurable stacked architecture (e.g., architecture 200) having fixed-function data paths (e.g., fixed-function data path 16) communicating with the multiple programmable MFUs, wherein the stacked architecture performs nonlinear operations on vector values ​​to accelerate layers of a DNN.

[0117] (B2) In some implementations, the accelerator of B1 includes a plurality of configurable finite scalar processing units (SPUs) (e.g., micro SPUs 18, 22, 26) that perform at least one scalar operation on vector values; and a plurality of configurable finite vector processing units (e.g., micro MFUs 20, 28, 28) that perform vector operations on vector values.

[0118] (B3) In some implementations of the B1 or B2 accelerators, multiple configurable finite SPUs and multiple configurable finite vector processing units are stacked sequentially to perform computations.

[0119] (B4) In some implementations of any of the accelerators in B1-B3, the output of a first finite SPU among a plurality of configurable finite SPUs is provided as input to a first finite vector processing unit among a plurality of configurable finite vector processing units, and the output of the first finite vector processing unit is provided as input to a second finite SPU among a plurality of configurable finite SPUs.

[0120] (B5) In some implementations, any one of B1-B4 accelerators includes multiple first-in-first-out (FIFO) structures (e.g., FIFO structure 36, FIFO structure 44, FIFO structure 62) that sequentially provide vector values ​​output from the previous finite vector processing unit to the next finite vector processing unit, wherein one of the multiple FIFO structures runs in parallel with each of the multiple configurable finite SPUs.

[0121] (B6) In some implementations of any of the accelerators in B1-B5, each of the plurality of configurable finite SPUs performs a reduction operation, at least one scalar operation, and a broadcast operation to broadcast vector values ​​to vectors.

[0122] (B7) In some implementations of any of the accelerators in B1-B6, multiple configurable finite SPUs and multiple configurable finite vector processing units are selected for acceleration based on nonlinear operations.

[0123] (B8) In some implementations of any of the accelerators in B1-B7, the accelerator uses fixed-function instructions to identify the nonlinear operation to be accelerated.

[0124] (B9) In some implementations of any of the accelerators in B1-B8, the DNN is a bidirectional encoder representation (BERT) model derived from the transformer, and the nonlinear operation is either a softmax operation or a layer normalization operation.

[0125] (C1) Some implementations include a configurable stacked architecture (e.g., architecture 200) for a fixed-function data path (e.g., fixed-function data path 16), used in conjunction with an accelerator (e.g., accelerator 500) to accelerate softmax operations or layer normalization operations of deep neural networks (DNNs). The stacked architecture includes a first microscalar processing unit (SPU) (e.g., micro SPU 18) that performs maximum reduction on received vector values ​​from a programmable multifunction unit (MFU) (e.g., programmable MFU 14) of the accelerator, performs scalar inverse operations (e.g., inverse scalar operation 32) on the received vector values, and performs a broadcast operation that broadcasts the vector values ​​to a vector. The stacked architecture also includes a first micromultifunction unit (MFU) (e.g., micro MFU module 20) that receives vector values ​​from the first micro SPU and performs vector multiplication on the vector values. The stacked architecture also includes a second micro SPU (e.g., micro SPU 22) that receives vector values ​​from the first micro MFU and performs reduction operations, scalar multiplication operations (e.g., scalar multiplication operation 40), and broadcast operations that broadcast the vector values ​​to vectors. The stacked architecture also includes a second micro MFU (e.g., micro MFU 24) that receives vector values ​​from the second micro SPU and performs vector subtraction, vector square root operations (e.g., element-wise square root function 46), and vector exponentiation operations (e.g., element-wise exponentiation function 48) on the vector values ​​of the vectors. The stacked architecture also includes a third micro SPU (e.g., micro SPU 26) that receives vector values ​​from the second micro MFU and performs reduction operations, scalar multiplication operations (e.g., scalar multiplication 52), scalar addition operations (e.g., scalar addition 54), scalar square root operations (e.g., scalar square root operation 56), scalar inverse operations (e.g., scalar inverse operation 58), and broadcast operations that broadcast vector values ​​to vectors. The stacked architecture also includes a third micro MFU (e.g., micro MFU 28) that receives vectors from the third micro SPU and performs vector multiplication operations on the vector values ​​of the vectors.

[0126] (C2) In some implementations, the stacking architecture of C1 includes a first-in-first-out (FIFO) structure (e.g., FIFO structure 36) that receives vector values ​​from a programmable MFU and provides vector values ​​from the programmable MFU to a first micro MFU, wherein the FIFO structure runs in parallel with the first micro SPU; a second FIFO structure (e.g., FIFO structure 44) that receives vector values ​​modified by the first micro MFU and provides vector values ​​modified by the first micro MFU to a second micro MFU, wherein the second FIFO structure runs in parallel with the second micro SPU; and a third FIFO structure (e.g., FIFO structure 62) that receives vector values ​​modified by the second micro MFU and provides vector values ​​modified by the second micro MFU to a third micro MFU, wherein the third FIFO structure runs in parallel with the third micro SPU.

[0127] This disclosure may be implemented in other specific forms without departing from its spirit or characteristics. The described embodiments should be considered illustrative rather than restrictive. Therefore, the scope of this disclosure is indicated by the appended claims rather than the foregoing description. Changes falling within the meaning and scope of equivalents of the claims will be included within its scope.

Claims

1. A configurable stacked architecture for a fixed function data path for use with an accelerator to accelerate an operation of a deep neural network (DNN), comprising: a plurality of configurable micro scalar processing units (SPUs) that perform at least one scalar operation on vector values from a received vector; and a plurality of configurable micro multi-function units (MFUs) that perform a vector operation on the vector values, wherein the plurality of configurable micro SPUs and the plurality of configurable micro MFUs are placed in a sequence to perform the operation of the DNN, wherein the sequence indicates an alternating coupled arrangement of each configurable micro SPU with each configurable micro MFU in the fixed function data path, and comprising: an output of a first micro SPU of the plurality of configurable micro SPUs is provided as an input to a first micro MFU of the plurality of configurable micro MFUs, and an output of the first micro MFU is provided as an input to a second micro SPU of the plurality of configurable micro SPUs.

2. The stacked architecture of claim 1, wherein each micro SPU of the plurality of configurable micro SPUs further performs a reduction operation on the vector values, performs the at least one scalar operation on the vector values, and performs a broadcast operation to broadcast the vector values to a vector.

3. The stacked architecture of claim 1, wherein the operation is a softmax operation or a layer normalization operation.

4. The stacked architecture of claim 1, wherein the operation is a non-linear operation that involves at least one of a vector operation, a scalar operation, or a reduction operation.

5. The stacked architecture of claim 1, wherein a number of micro SPUs for the plurality of configurable micro SPUs and a number of micro MFUs for the plurality of configurable micro MFUs are selected based on the operation.

6. The stacked architecture of claim 1, wherein the plurality of configurable micro SPUs and the plurality of configurable micro MFUs are selected based on the operation.

7. The stacked architecture of claim 1, wherein the sequence of the plurality of configurable micro SPUs and the plurality of configurable micro MFUs is selected based on the operation.

8. The stacked architecture of claim 1, wherein the plurality of configurable micro SPUs and the plurality of configurable micro MFUs perform the operation without an intermediate access to a memory of the accelerator.

9. The stacked architecture of claim 1, further comprising: a plurality of first-in-first-out (FIFO) structures that provide the vector values output from a previous micro MFU to a next micro MFU in the sequence, wherein one of the plurality of FIFO structures is in parallel with each micro SPU of the plurality of micro SPUs.

10. The stacked architecture of claim 1, further comprising: at least one programmable MFU in communication with one or more micro-SPUs of the plurality of configurable micro-SPUs, wherein the output of the at least one programmable MFU is provided as an input to the one or more micro-SPUs.

11. The stacked architecture of claim 1, further comprising: a first programmable MFU in communication with a first micro-SPU of the plurality of configurable micro-SPUs; a second programmable MFU in communication with a last micro-SPU of the plurality of configurable micro-SPUs; and a third programmable MFU in communication with the second programmable MFU.

12. An accelerator, comprising: a plurality of vector register files (VRFs) providing one or more vectors having data for the accelerator; a plurality of programmable multi-function units (MFUs) in communication with the VRFs to perform vector operations on vector values from the one or more vectors; at least one programmable scalar processing unit (SPU); and a configurable stacked architecture having fixed function data paths in communication with the plurality of programmable MFUs, wherein the stacked architecture performs non-linear operations on the vector values to accelerate a layer of a DNN, and the stacked architecture includes a plurality of configurable finite SPUs that perform scalar operations on the vector values as part of the non-linear operations, a plurality of configurable finite vector processing units that perform vector operations on the vector values as part of the non-linear operations, and the plurality of configurable finite SPUs and the plurality of configurable finite vector processing units are stacked in a sequence to perform the non-linear operations, wherein the sequence indicates an alternating coupled arrangement of each configurable finite SPU with each configurable finite vector processing unit in the fixed function data paths, and comprising: an output of a first finite SPU of the plurality of configurable finite SPUs is provided as an input to a first finite vector processing unit of the plurality of configurable finite vector processing units, and the output of the first finite vector processing unit is provided as an input to a second finite SPU of the plurality of configurable finite SPUs.

13. The accelerator of claim 12, wherein the stacked architecture further comprises: a plurality of first-in-first-out (FIFO) structures providing the vector values output from a previous finite vector processing unit to a next finite vector processing unit in the sequence, wherein one of the plurality of FIFO structures is in parallel with each finite SPU of the plurality of configurable finite SPUs.

14. The accelerator of claim 12, wherein each finite SPU of the plurality of configurable finite SPUs performs a reduction operation, the at least one scalar operation, and a broadcast operation to broadcast the vector values to a vector. ​ 15. The accelerator of claim 12, wherein the plurality of configurable finite SPUs and the plurality of configurable finite vector processing units are selected based on the non-linear operation to be accelerated.

16. The accelerator of claim 12, wherein the accelerator uses fixed function instructions to identify the non-linear operation to be accelerated.

17. The accelerator of claim 12, wherein the DNN is a Bidirectional Encoder Representations from Transformers (BERT) model and the non-linear operation is one of a softmax operation or a layer normalization operation.

18. A configurable stacked architecture for a fixed function data path for use with an accelerator to accelerate a softmax operation or a layer normalization operation of a deep neural network (DNN), each micro scalar processing unit (SPU) in the fixed function data path being alternately coupled with each programmable multi-function unit (MFU), the configurable stacked architecture comprising: a first micro SPU to perform a max-reduce operation on a received vector value from a programmable multi-function unit (MFU) of the accelerator, to perform a scalar inverse operation on the received vector value, and to perform a broadcast operation to broadcast the vector value to a vector; a first micro MFU to receive the vector value from the first micro SPU and to perform a vector multiplication operation on the vector value; a second micro SPU to receive the vector value from the first micro MFU and to perform a reduce operation on the vector value, to perform a scalar multiplication operation on the vector value, and to perform the broadcast operation to broadcast the vector value to the vector; a second micro MFU to receive the vector value from the second micro SPU and to perform a vector subtraction operation, a vector square root operation, and a vector power operation on the vector value of the vector; a third micro SPU to receive the vector value from the second micro MFU and to perform a reduce operation on the vector value, to perform a scalar multiplication operation on the vector value, to perform a scalar addition operation on the vector value, to perform a scalar square root operation on the vector value, to perform a scalar inverse operation on the vector value, and to perform the broadcast operation to broadcast the vector value to the vector; and a third micro MFU to receive the vector from the third micro SPU and to perform a vector multiplication operation on the vector value of the vector.

19. The stacked architecture of claim 18, further comprising: a first-in-first-out (FIFO) structure to receive the vector value from the programmable MFU and to provide the vector value from the programmable MFU to the first micro MFU, wherein the FIFO structure is in parallel with the first micro SPU. ​ a second FIFO structure that receives the vector value modified by the first micro-MFU and provides the vector value modified by the first micro-MFU to the second micro-MFU, wherein the second FIFO structure is in parallel with the second micro-SPU; and a third FIFO structure that receives the vector value modified by the second micro-MFU and provides the vector value modified by the second micro-MFU to the third micro-MFU, wherein the third FIFO structure is in parallel with the third micro-SPU.

Images