Vision transformer accelerator

Abstract

A vision transformer accelerator is provided, including: a plurality of kernel blocks, each kernel block configured to compute operations of a vision transformer model, including matrix multiplication operations and different types of non-linear functions; a weight buffer configured to store weight data associated with the vision transformer model; a kernel input bus configured to transfer input data of a current encoder layer of the vision transformer model to each of the plurality of kernel blocks; a kernel output bus configured to transfer first output data of each of the plurality of kernel blocks to a next encoder layer of the vision transformer model; and a weight bus configured to transfer weight data from the weight buffer to each of the plurality of kernel blocks. Furthermore, each of the plurality of kernel blocks includes: a plurality of matrix buffers, each matrix buffer configured to store one or more matrices; a matrix-multiplication unit communicatively coupled to the plurality of matrix buffers and the weight buffer, the matrix-multiplication unit being configured to compute a matrix multiplication operation based on at least one matrix stored in one or more of the plurality of matrix buffers; and a non-linear function block configured to compute each of the different types of non-linear functions. There is also provided a corresponding method of operating the vision transformer accelerator.

Description

VISION TRANSFORMER ACCELERATORCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority of Singapore Patent Application No. 10202400322Q, filed on 5 February 2024, the content of which being hereby incorporated by reference in its entirety for all purposes.TECHNICAL FIELD

[0002] The present invention generally relates to a vision transformer accelerator and a method of operating the vision transformer accelerator for computing operations of a vision transformer model.BACKGROUND

[0003] Transformer-based models have achieved remarkable results in various Natural Language Processing (NLP) and Computer Vision (CV) tasks, such as machine translation, sentiment analysis, text summarization, question answering, image classification, object detection, semantic segmentation, image colorization, action recognition, and image generation. Some Transformer models, such as GPT-4, can even perform both NLP and CV tasks in the multitask setting. Transformers have become the dominant neural architecture for both NLP and CV domains. However, the scaling up of Transformer models up to trillions of parameters and trillions of Multiply-Accumulate (MAC) operations, as in the case of GPT-4, during both training and inference has made them both compute and data-intensive. This poses a significant challenge for the deployment of these models in an area- and power-efficient manner on both cloud and edge devices.

[0004] Operations in Transformer models include matrix multiplications and non-linear functions For example, matrix multiplication consumes over 60% of the execution time of Vision Transformer (ViT) on NVIDIA 2080 Ti GPU. Nevertheless, thanks to both scalable hardware (e.g., GPUs and Transformer accelerators on ASIC or FPGAs) and algorithmic optimizations (e.g., quantization, sparsity and function approximation), this operation has seen significant improvements. The implementation of the non-linear functions, e.g., Gaussian error linear unit (GELU), Softmax, and LayerNorm, has been a bottleneck in terms of both area and power efficiency. For example, non-linear functions occupy nearly 40% and 60% of the execution time and memory usage, respectively, and occupy 45% of the total area in SwiftTron(disclosed in Marchisio, et al., “Swifttron: An efficient hardware accelerator for quantized transformers,” arXiv preprint arXiv:2304.03986, 2023 (herein referred to as the Marchisio reference)) even if the matrix multipliers are dedicated. However, they have not witnessed significant improvement and remain bottlenecks due to their computational complexity and high throughput requirements.

[0005] Moreover, the computational dataflow of Transformer models is less explored, and many Transformer accelerators simply follow the original software implementation dataflow, which can be further optimized to reduce both memory footprint and data movement for achieving higher area- and power-efficiency. For example, FIG. 1 shows the memory capability requirements of the Multi-Head Self Attention (MHA) layer in the ViT-Base models, with a 224x224 and VGA resolution RGB image as input in INT8 mode. As shown in FIG. 1, considering the LayerNorm operation before the MHA and the shortcut adding after the Project GEMM (General Matrix Multiply), the memory requirements of the MHA layer in the ViT- Base models to support 224x224 and VGA image resolutions are 1.45 MB and 22.66 MB, respectively, which requires a significant chip area reserved for the memory. As also shown in FIG. 1, to support VGA images, 22.66 MB and 20.03 MB memory are required if dedicated memories are used for middle results or memories are reused, respectively.

[0006] A need therefore exists to provide a vision transformer accelerator, as well as a method of operating the vision transformer accelerator, that seek to overcome, or at least ameliorate, one or more deficiencies in existing processors (e.g., GPUs) and transformer accelerators in performing or computing the workload of vision transformer models, and more particularly, for accelerating the vision transformer workload with improved area and power efficiency. It is against this background that the present invention has been developed.SUMMARY

[0007] According to a first aspect of the present invention, there is provided a vision transformer accelerator comprising: a plurality of kernel blocks, each kernel block configured to compute operations of a vision transformer model, including matrix multiplication operations and different types of nonlinear functions; a weight buffer configured to store weight data associated with the vision transformer model;a kernel input bus configured to transfer input data of a current encoder layer of the vision transformer model to each of the plurality of kernel blocks; a kernel output bus configured to transfer first output data of each of the plurality of kernel blocks to a next encoder layer of the vision transformer model; and a weight bus configured to transfer weight data from the weight buffer to each of the plurality of kernel blocks, wherein each of the plurality of kernel blocks comprises: a plurality of matrix buffers, each matrix buffer configured to store one or more matrices; a matrix-multiplication unit communicatively coupled to the plurality of matrix buffers and the weight buffer, the matrix-multiplication unit being configured to compute a matrix multiplication operation based on at least one matrix stored in one or more of the plurality of matrix buffers; and a non-linear function block configured to compute each of the different types of non-linear functions.

[0008] According to a second aspect of the present invention, there is provided a method of operating the vision transformer accelerator according to the first aspect of the present invention, comprising, for each of the plurality of kernel blocks of the vision transformer accelerator: computing, using the matrix-multiplication unit of the kernel block, a matrix multiplication operation based on at least one matrix stored in one or more of the plurality of matrix buffers of the kernel block; and computing, using the non-linear function block of the kernel block, a non-linear function of the different types of non-linear functions.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments of the present invention will be better understood and readily apparent to one of ordinary skill in the art from the following written description, by way of example only, and in conjunction with the drawings, in which:FIG. 1 shows the memory capability requirements of the MHA layer in the ViT-Base models, with a 224x224 and VGA resolution RGB image as input in INT8 mode;FIG. 2 depicts a schematic drawing showing an architecture overview of a vision transformer accelerator (ViTA), according to various embodiments of the present invention;FIG. 3 depicts a schematic diagram of a method of operating the vision transformer accelerator, according to various embodiments of the present invention;FIG. 4 depicts a schematic drawing of a Transformer encoder;FIG. 5 shows a table (Table I) providing a summary of the data formats of the basic kernels (operations) of ViT in the FasterTransformer and the present ViTA according to various example embodiments of the present invention;FIG. 6A depicts a schematic drawing showing an architecture overview of a ViTA, according to various example embodiments of the present invention;FIG. 6B depicts a schematic drawing showing an architecture overview of the kernel block of the ViTA, according to various example embodiments of the present invention;FIG. 7 depicts a schematic drawing illustrating the matrix-multiplication unit configured as a VMU (vector-multiplication unit) lane comprising a series of VMUs for performing a matrix multiplication operation, according to various example embodiments of the present invention;FIG. 8 illustrates an example computation data path of the VMU, whereby the output is the inner proj ect of two input vectors, according to various example embodiments of the present invention;FIG. 9A depicts a schematic drawing showing an architecture overview of the special function module (non-linear function block) of the kernel block of the ViTA, according to various example embodiments of the present invention,FIG. 9B depicts a schematic drawing showing a computation data path for a normal dedicated Softmax function module;FIG. 9C depicts a schematic drawing showing example ports and parameters of LUT- PLF-Softmax (Softmax implemented with LUTs using piecewise linear function (PLF) for approximating Softmax);FIG. 9D depicts a schematic drawing showing a computation data path for a normal dedicated LayerNorm function module;FIG. 9E depicts a schematic drawing showing example ports and parameters of LUT- PLF-LayerNorm (LayerNorm implemented with LUTs using PLF);FIG. 9F depicts a schematic drawing showing example ports and parameters of LUT- PLF-GeLU (GeLU implemented with LUT s using PLF for approximating GeLU);FIG. 9G depicts a schematic drawing showing a computation data path of a normal dedicated GeLU function module;FIG. 9H depicts a schematic drawing showing example ports of a regfile, or a RAM, where the chip selection signal CEN is active low;FIG. 10 shows a table (Table II) providing a summary of the usages of the PLF units and the two adder trees of the special function module (non-linear function block) in different non-linear function modes, according to various example embodiments of the present invention;FIG. 11 shows a table (Table III) presenting the rough area and power for arithmetic modules in different data types in 28 nm 1.05 V;FIG. 12 shows a table (Table IV) providing a summary of the dataflows of each ViT layer, including the name of ViT operations, the corresponding data source and destination of the GEMM computation, the GEMM dataflow, and the special functions (non-linear functions) before and after the GEMM operation, according to various example embodiments of the present invention;FIG. 13 shows an example MKN-RS GEMM dataflow (pseudocode thereof), according to various example embodiments of the present invention;FIG. 14 shows a table (Table V) providing a summary of dataflows, optimum conditions, memory and bandwidth bottlenecks in GEMM, as well as the throughput requirements of special functions (non-linear functions), and whether they are supported in ViTA, according to various example embodiments of the present invention;FIG. 15 depict plots showing the on-chip memory requirements (KB) for an accelerator to support the GEMM operations in ViT-base with a VGA image resolution as input in different dataflows;FIG. 16 depict plots showing the bandwidth requirements (GB / s) of matrices A, B and C, and the throughput requirements (GOPS) of the special functions (non-linear functions) for an accelerator to support the GEMM operations in ViT-base with a VGA image resolution as input in different dataflows;FIG. 17 depicts a schematic drawing showing tiling adopted for large-scale GEMM operations, according to various example embodiments of the present invention;FIG. 18A shows a fused multi -head attention algorithm (herein referred to as Algorithm 1) from the input matrix X to the input of the MLP layers, according to various example embodiments of the present invention;FIG. 18B illustrates the corresponding fused multi-head attention dataflow of the fused multi-head attention algorithm, according to various example embodiments of the present invention;FIG. 19 depicts a schematic drawing illustrating the dataflow of the PerHeadQKV function in Algorithm 1 , according to various example embodiments of the present invention;FIG. 20 depicts a schematic drawing illustrating the dataflow of the PerHeadA and PerHeadZ functions in Algorithm 1, according to various example embodiments of the present invention;FIG. 21 shows atable (Table VI) providing a summary of experimental setup fortesting the ViTA, according to various example embodiments of the present invention;FIG. 22 shows the performance (GOPS), the performance (GOPS), area efficiency, and power efficiency of the ViTA Huge at different clock frequencies;FIG. 23 shows a table (Table VII) presenting the experimental results of the ViTA huge at different clock frequencies, supporting a 224x224 RGB image as input;FIG. 24 shows a table (Table VIII) providing a summary of the memory bandwidth requirements for the ViTA with various hardware configurations operating at 300 MHz for supporting a 224x224 RGB image as input;FIG. 25 shows plots of the performance (GOPS), area efficiency, and power efficiency of the ViTA with different hardware configurations at 300 MHz;FIGs. 26A and 26B show the area (FIG. 26A) and power (FIG. 26B) of the ViTA Kernel in the ViTA Huge (m = 32, k= n = 16) at 300 MHz, with a kernel area of 0.211 mm2and power of 83.84 mW;FIGs. 26C and 26D show the area (FIG. 26C) and power (FIG. 26D) in the ViTA Tiny (m = k = n = 8) at 300 MHz, with a kernel area of 0.248 mm2and power of 40.21 mW at 300 MHz;FIG. 27 shows plots of the memory bandwidth requirements for the ViTA with various hardware configurations operating at 300 MHz for supporting a 224x224 RGB image as input;FIG. 28 shows a table (table IX) presenting detailed comparisons between the related works and the introduced ViTA architectures according to various example embodiments of the present invention for area- and power-constraint applications and high-performance applications, respectively; andFIG. 29 shows the performance, power efficiency, and area efficiency of related works compared with the ViTA Huge.DETAILED DESCRIPTION

[0010] Various embodiments of the present invention relate a vision transformer accelerator and a method of operating the vision transformer accelerator for computing operations of a vision transformer model

[0011] As discussed in the background, existing processors (e.g., GPUs) and transformer accelerators suffer from various deficiencies in performing or computing the workload of vision transformer models, including area and power inefficiencies. In this regard, various embodiments of the present invention provide a vision transformer accelerator, as well as a method of operating the vision transformer accelerator, that seek to overcome, or at least ameliorate, one or more deficiencies in conventional processors and transformer accelerators in performing or computing the workload of vision transformer models, and more particularly, for accelerating the vision transformer workload with improved area and power efficiency.

[0012] FIG. 2 depicts a schematic drawing showing an architecture overview of a vision transformer accelerator 200, according to various embodiments of the present invention. The vision transformer accelerator 200 comprises: a plurality of kernel blocks 210 (e.g., 32 kernel blocks shown in FIG. 2 as an example only), each kernel block 210 configured to compute operations of a vision transformer model, including matrix multiplication operations and different types of non-linear functions (e g., Gaussian error linear unit (GELU) function, softmax function and layer normalization function), a weight buffer 220 configured to store weight data associated with the vision transformer model; a kernel input bus 230 configured to transfer input data of a current encoder layer of the vision transformer model to each of the plurality of kernel blocks 210; a kernel output bus 232 configured to transfer first output data (e g., output data (e g., final results of a current encoder layer) for a next encoder layer) of each of the plurality of kernel blocks 210 to a next encoder layer of the vision transformer model; and a weight bus 234 configured to transfer weight data from the weight buffer 220 to each of the plurality of kernel blocks 210. Furthermore, each of the plurality of kernel blocks 210 comprises: a plurality of matrix buffers 240, each matrix buffer configured to store one or more matrices; a matrix-multiplication unit 244 communicatively coupled to the plurality of matrix buffers 240 and the weight buffer 220, the matrix-multiplication unit 244 being configured to compute a matrix multiplication operation based on at least one matrix stored in one or more of the plurality of matrix buffers 240, and a non-linear function block 248 configured to compute each of the different types of non-linear functions.

[0013] Accordingly, the vision transformer accelerator 200 according to various embodiments of the present invention advantageously has a scalable and highly efficient architecture for accelerating the workload of vision transformer models with high area and power efficiency. In particular, the vision transformer accelerator 200 is advantageously configured with multiple kernel blocks 210, each capable of computing operations in parallel, including matrix multiplication operations and different types of non-linear functions. In addition, with multiple matrix buffers 240 in each kernel block 210 for supporting operations, the vision transformer accelerator 200 is advantageously configured to enable highly efficient memory-centric dataflow for computing operations of the vision transformer model. Furthermore, the non-linear function block 248 is configured to be capable of computing different types of non-linear functions (e.g., GELU function, softmax function and layer normalization function) integrally, thereby optimizing hardware resource sharing for further improving area and power efficiency. Accordingly, the vision transformer accelerator 200 according to various embodiments of the present invention is advantageously able to accelerate the workload of vision transformer models while achieving high area and power efficiency. These advantages or technical effects, and / or other advantages or technical effects, will become more apparent to a person skilled in the art as the vision transformer accelerator 200, as well as the corresponding method of operating the vision transformer accelerator 200, is described in more detail according to various embodiments and example embodiments of the present invention.

[0014] In various embodiments, the matrix-multiplication unit 244 is configured as a vector-multiplication unit lane comprising a series of vector-multiplication units. The vectormultiplication unit lane is configured to compute the matrix multiplication operation (e.g., vector-matrix multiplication) based on the at least one matrix stored in the one or more of the plurality of matrix buffers 240. In this regard, each vector-multiplication unit is configured to perform a respective vector multiplication operation (e.g., vector-vector multiplication) associated with the matrix multiplication operation.

[0015] In various embodiemnts, for a matrix multiplication operation (e.g., a matrix-matrix multiplication to be computed by the vision transformer accelerator 200) above a predefined size (e.g., an input matrix for the matrix multiplication operation exceeding a predefined size), the vision transformer accelerator 200 is configured to split an input matrix for the matrix multiplication operation into multiple input matrix chunks and map the multiple input matrix chunks to multiple kernel blocks of the plurality of kernel blocks 210, respectively, for themultiple kernel blocks to compute matrix multiplication operations (e g., vector-matrix multiplication) based on the multiple input matrix chunks, respectively, in parallel. It will be appreciated by a person skilled in the art that the predefined size may be set or determined as desired or as appropriate and is not limited to any specific size.

[0016] In various embodiments, the weight buffer 220 is a ping-pong buffer comprising a first weight buffer and a second weight buffer. Furthermore, the vision transformer accelerator 200 further comprises: a multiplexer (not shown in FIG. 2); and a key and value bus (not shown in FIG. 2) configured to transfer second output data (e g., output data (e.g., key and value data or matrix) for a current encoder layer) of each of the plurality of kernel blocks 210 to the multiplexer. In this regard, the multiplexer is arranged between the weight buffer 220 and each of the plurality of kernel blocks 210 and is configured to multiplex the weight data from the weight buffer 220 with the second output data received via the key and value bus to produce an output (i .e., select one of the weight data and the second output data) for transfer to each of the plurality of kernel blocks 210 via the weight bus 234.

[0017] In various embodiments, for each of the plurality of kernel blocks 210: the kernel block 210 further comprises a first multiplexer configured to multiplex an output of one of the plurality of matrix buffers 240 and a first output of the non-linear function block 248 to produce an output, and the matrix-multiplication unit 244 is configured to receive the output of the first multiplexer as a first input thereto and the output of the multiplexer of the vision transformer accelerator 200 via the weight bus 234 as a second input thereto.

[0018] In various embodiments, for each of the plurality of kernel blocks 210: the plurality of matrix buffers 240 comprises a first matrix buffer, a second matrix buffer and a third matrix buffer, and the kernel block 210 further comprises a partial sum buffer configured as a ping- pong buffer comprising a first partial sum buffer and a second partial sum buffer.

[0019] In various embodiments, the vision transformer accelerator 200 further comprises a bias bus configured to transfer bias data to each of the plurality of kernel blocks 210. For each of the plurality of kernel blocks 210, the kernel block 210 further comprises: an accumulator; a second multiplexer; a dequantization unit; and a third multiplexer. In this regard, the second multiplexer is configured to multiplex an output of the matrix-multiplication unit 244, the bias data received via the bias bus and an output of the first matrix buffer to produce an output. The dequantization unit is configured to receive and dequantize the output of the second multiplexer to produce an output. The accumulator is configured to receive the output of the dequantization unit as a first input thereto and a first output of the partial sum buffer as a second input thereto.The third multiplexer is configured to multiplex an output of the accumulator and the output of the dequantization unit to produce an output.

[0020] In various embodiments, for each of the plurality of kernel blocks 210: the nonlinear function block 248 is configured to receive a second output of the partial sum buffer as a first input thereto and the output of the first matrix buffer as a second input thereto. Furthermore, the kernel block 210 further comprises a fourth multiplexer configured to multiplex the first output of the non-linear function block 248, a second output of the non-linear function block 248 and the output of the third multiplexer to produce an output.

[0021] In various embodiments, for each of the plurality of kernel blocks 210: the kernel block 210 further comprises a fifth multiplexer configured to multiplex the output of the fourth multiplexer and the input data of the current encoder layer received via the kernel input bus 230 to produce an output. Furthermore, the second matrix buffer is configured to receive the output of the fifth multiplexer as an input thereto, and the first and third matrix buffers are each configured to receive the output of the fourth multiplexer as an input thereto.

[0022] In various embodiments, for each of the plurality of kernel blocks 210: the kernel block 210 further comprises a sixth multiplexer, a seventh multiplexer and a demultiplexer. In this regard, the demultiplexer is configured to receive the output of the third multiplexer to produce a first output for input to the first partial sum buffer and a second output for input to the second partial sum buffer. The sixth multiplexer is configured to multiplex an output of the first partial sum buffer and an output of the second partial sum buffer to produce the first output of the partial sum buffer. The seventh multiplexer is configured to multiplex the output of the first partial sum buffer and the output of the second partial sum buffer to produce the second output of the partial sum buffer.

[0023] In various embodiments, for each of the plurality of kernel blocks 210, the nonlinear function block 248 comprises a plurality of piecewise linear function (PLF) units and a plurality of adder trees configured to compute the different types of non-linear functions. In various embodiments, the different types of non-linear functions comprise a GELU function, a softmax function and a layer normalization function.

[0024] In various embodiments, for computing the softmax function: a set of PLF units of the plurality of PLF units is configured to compute exponential values of input elements, respectively; a first adder tree of the plurality of adder trees and a first adder are configured to compute a sum of the exponential values of the input elements; a further PLF unit of the plurality of PLF units is configured to compute a reciprocal of the sum of the exponential valuesof the input elements; and a first multiplier of the non-linear function block 248 is configured to, for each of the input elements, multiply the exponential value of the input element with the reciprocal of the sum of the exponential values of the input elements to obtain the softmax value of the input element. In various embodiments, the parallelism of the non-linear function block 248 may be set to n per cycle such that the first multiplier determines the softmax value for n input elements simultaneously, while computing other non-linear functions for n input elements in turn. In various embodiments, the parallelism of the non-linear function block 248 may be reduced from n per cycle to nl(Klk) per cycle as will be described later below (e.g., with reference to Table V in FIG. 14) according to various example embodiments of the present invention.

[0025] In various embodiments, for computing the layer normalization function: the first adder tree and the first adder are further configured to compute a sum of input elements; a second adder tree of the plurality of adder trees and a second adder are configured to compute a sum of squares of the input elements; a second multiplier of the non-linear function block 248 is configured to compute a mean of input elements based on the sum of the input elements; a third multiplier, a fourth multiplier and a first subtractor of the non-linear function block 248 are configured to compute a square of a variance of the input elements based on the mean of the input elements and the sum of the squares of the input elements computed; a second subtractor of the non-linear function block 248 is configured to, for each of the input elements, perform a subtraction between the mean and the input element to obtain a difference value associated with the input element; the further PLF unit of the plurality of PLF units is configured to compute a reciprocal of the square root of the square of the variance of the input elements to obtain a reciprocal of the variance of the input elements; and the first multiplier of the nonlinear function block 248 is further configured to, for each of the input elements, multiply the difference value associated with the input element and the reciprocal of the variance of the input elements to obtain the layer normalization value of the input element. In various embodiments, the parallelism of the non-linear function block 248 may be set to n per cycle for n input elements.

[0026] In various embodiments, the vision transformer accelerator 200 is further configured to compute the layer normalization function in two stages. In this regard, at a first stage, the vision transformer accelerator 200 is configured to compute the mean of input elements and the square of the variance of the input elements, and the input elements are written to one of the plurality of matrix buffers 240 after quantization. At a second stage, the vision transformeraccelerator 200 is configured to read the input elements from the above-mentioned one of the plurality of matrix buffers 240 after dequantization and compute, for each of the input elements, the layer normalization value of the input element based on the mean of input elements and the square of the variance of the input elements computed at the first stage.

[0027] Tn various embodiments, for computing the GELU function, the set of PLF units of the plurality of PLF units is configured to compute the GELU function based on input elements. |0028| In various embodiments, the vision transformer accelerator 200 is configured to compute matrix multiplication operations according to a first dataflow or a second dataflow. In this regard, each matrix multiplication operation involves a first input matrix thereto having a row dimension and a column dimension (e.g., matrix A with dimensions of M K), a second input matrix thereto having a row dimension and a column dimension (e.g., matrix B with dimensions ofand an output matrix therefrom having a row dimension and a column dimension (e.g., matrix C with dimensions of M*N). The first dataflow is a row stationary (RS) dataflow whereby executions of for-loops of the matrix multiplication operation, from inner to outer, is according to an order of the column dimension of the second input matrix (e.g., along N dimension), the column dimension of the first input matrix (e.g., along K dimension) and the row dimension of the first input matrix (along M dimension) (e.g., may be referred to as the MKN-RS dataflow). The second dataflow is a column stationary (CS) dataflow wherein executions of for-loops of the matrix multiplication operation, from inner to outer, is according to an order of the row dimension of the first input matrix (e g., along AL dimension), the column dimension of the first input matrix (e.g., along K dimension) and the column dimension of the second input matrix (e.g., along N dimension) (e.g., may be referred to as the NKM-CS dataflow).

[0029] In various embodiments, the vision transformer accelerator 200 is further configured to compute non-linear functions according to a third dataflow or a fourth dataflow. In this regard, each non-linear function involves an input matrix thereto having a row dimension and a column dimension. The third dataflow is an output stationary (OS) dataflow whereby executions of for-loops of the non-linear function, from inner to outer, is according to an order of the column dimension of the input matrix and the row dimension of the input matrix (e.g., may be referred to as the MNK-OS dataflow). The fourth dataflow is an output stationary (OS) dataflow whereby executions of for-loops of the non-linear function, from inner to outer, is according to an order of the row dimension of the input matrix and the column dimension of the input matrix (e.g., may be referred to as the NMK-OS dataflow).

[0030] In various embodiments, for each of the plurality of kernel blocks 210, the partial sum buffer comprising the first partial sum buffer and the second partial sum buffer is configured to convert a dataflow for computing a matrix multiplication operation to a dataflow for computing a non-linear function for the non-linear function block 248.

[0031] In various embodiments, the partial sum buffer is configured to convert the first dataflow (e.g., the MKN-RS dataflow) for computing the matrix multiplication operation to the third dataflow (e g., the MNK-OS dataflow) for computing the non-linear function, or convert the second dataflow (e.g., the NKM-CS dataflow) for computing the matrix multiplication operation to the fourth dataflow (e.g., the NMK-OS dataflow) for computing the non-linear function.

[0032] In various embodiments, the input data transferred to each of the plurality of kernel blocks 210 via the kernel input bus 230 and the weight data transferred to each of the plurality of kernel blocks 210 via the weight bus 234 are synchronized to the plurality of kernel blocks 210 through asynchronous first-in-first-out (FIFO) buffers, respectively.

[0033] In various embodiments, the operations of the vision transformer model, include matrix multiplication operations and different types of non-linear functions relating to multihead self attention (MHA) layers and fully connected feed-forward network (FFN) layers of the vision transformer model.

[0034] FIG. 3 depicts a schematic diagram of a method 300 of operating the vision transformer accelerator 200, according to various embodiments of the present invention, comprising, for each of the plurality of kernel blocks of the vision transformer accelerator: computing (at 302), using the matrix-multiplication unit 244 of the kernel block 210, a matrix multiplication operation based on at least one matrix stored in one or more of the plurality of matrix buffers 240 of the kernel block 210; and computing (at 304), using the non-linear function block 248 of the kernel block 210, a non-linear function of the different types of nonlinear functions.

[0035] In various embodiments, the method 300 of operating the vision transformer accelerator 200 corresponds to the vision transformer accelerator 200 as described herein according to various embodiments with reference to FIG. 2, therefore, various operations, functions or steps of the method 300 may correspond to various operations, functions or steps in which various components (or elements) of the vision transformer accelerator 200 are configured to perform, as described herein according to various embodiments of the present invention. In other words, various embodiments described herein in context of the visiontransformer accelerator 200 are analogously valid for the method 300 of operating the vision transformer accelerator 200, and vice versa.

[0036] In various embodiments, the above-mentioned computing (at 302), using the matrixmultiplication unit 244 of the kernel block 210, the matrix multiplication operation comprises computing, using the vector-multiplication unit lane, the matrix multiplication operation based on the at least one matrix stored in the one or more of the plurality of matrix buffers 240. In this regard, each vector-multiplication unit is configured to perform a respective vector multiplication operation (e g., vector-vector multiplication) associated with the matrix multiplication operation.

[0037] In various embodiments, for a matrix multiplication operation (e g , a matrix-matrix multiplication to be computed by the vision transformer accelerator 200) above a predefined size (e g., an input matrix for the matrix multiplication operation exceeding a predefined size), the method 300 further comprises: splitting an input matrix for the matrix multiplication operation into multiple input matrix chunks; and mapping the multiple input matrix chunks to multiple kernel blocks of the plurality of kernel blocks 210, respectively, for the multiple kernel blocks to compute matrix multiplication operations (e g., vector-matrix multiplication) based on the multiple input matrix chunks, respectively, in parallel.

[0038] In various embodiments, the method 300 further comprises multiplexing, using the multiplexer of the vision transformer accelerator 200, the weight data from the weight buffer 220 with the second output data received via the key and value bus to produce an output for transfer to each of the plurality of kernel blocks 210 via the weight bus 234.

[0039] In various embodiments, for each of the plurality of kernel blocks 210, the method 300 further comprises: multiplexing, using the first multiplexer of the kernel block 210, an output of one of the plurality of matrix buffers 240 and a first output of the non-linear function block 248 to produce an output, and receiving, by the matrix-multiplication unit 244, the output of the first multiplexer as a first input thereto and the output of the multiplexer of the vision transformer accelerator 200 via the weight bus 234 as a second input thereto.

[0040] In various embodiments, for each of the plurality of kernel blocks 210, the method 300 further comprises: multiplexing, using the second multiplexer of the kernel block 210, an output of the matrix-multiplication unit 244, the bias data received via the bias bus and an output of the first matrix buffer to produce an output; receiving and dequantizing, using the dequantization unit of the kernel block 210, the output of the second multiplexer to produce an output; receiving, by the accumulator of the kernel block 210, the output of the dequantizationunit as a first input thereto and a first output of the partial sum buffer as a second input thereto; and multiplexing, using the third multiplexer of the kernel block 210, an output of the accumulator and the output of the dequantization unit to produce an output.

[0041] In various embodiments, for each of the plurality of kernel blocks 210, the method 300 further comprises: receiving, by the non-linear function block 248 of the kernel block 210, a second output of the partial sum buffer as a first input thereto and the output of the first matrix buffer as a second input thereto, and multiplexing, using the fourth multiplexer of the kernel block 210, the first output of the non-linear function block 248, a second output of the nonlinear function block 248 and the output of the third multiplexer to produce an output.

[0042] In various embodiments, for each of the plurality of kernel blocks 210, the method 300 further comprises: multiplexing, using the fifth multiplexer of the kernel block 210, the output of the fourth multiplexer and the input data of the current encoder layer received via the kernel input bus to produce an output; receiving, by the second matrix buffer of the kernel block 210, the output of the fifth multiplexer as an input thereto; and receiving, by each of the first and third matrix buffers of the kernel block 210, the output of the fourth multiplexer as an input thereto.

[0043] In various embodiments, for each of the plurality of kernel blocks 210, the method 300 further comprises: receiving, by the demultiplexer of the kernel block 210, the output of the third multiplexer to produce a first output for input to the first partial sum buffer and a second output for input to the second partial sum buffer, multiplexing, using the sixth multiplexer of the kernel block 210, an output of the first partial sum buffer and an output of the second partial sum buffer to produce the first output of the partial sum buffer; and multiplexing, using the seventh multiplexer of the kernel block 210, the output of the first partial sum buffer and the output of the second partial sum buffer to produce the second output of the partial sum buffer.

[0044] In various embodiments, for computing the softmax function, the above-mentioned computing (at 304), using the non-linear function block 248 of the kernel block 210, the nonlinear function comprises: computing, using a set of PLF units of the plurality of PLF units, exponential values of input elements, respectively; computing, using a first adder tree of the plurality of adder trees and a first adder, a sum of the exponential values of the input elements; computing, using a further PLF unit of the plurality of PLF units, a reciprocal of the sum of the exponential values of the input elements; and for each of the input elements, multiply, using a first multiplier of the non-linear function block 248, the exponential value of the input elementwith the reciprocal of the sum of the exponential values of the input elements to obtain the softmax value of the input element.

[0045] In various embodiments, for computing the layer normalization function, the above- mentioned computing (at 304), using the non-linear function block 248 of the kernel block 210, the non-linear function comprises: computing, using the first adder tree and the first adder, a sum of input elements, computing, using a second adder tree of the plurality of adder trees and a second adder, a sum of squares of the input elements; computing, using a second multiplier of the non-linear function block 248, a mean of input elements based on the sum of the input elements; computing, using a third multiplier, a fourth multiplier and a first subtractor of the non-linear function block 248, a square of a variance of the input elements based on the mean of the input elements and the sum of the squares of the input elements computed; for each of the input elements, performing, using a second subtractor of the non-linear function block 248, a subtraction between the mean and the input element to obtain a difference value associated with the input element; computing, using the further PLF unit of the plurality of PLF units, a reciprocal of the square root of the square of the variance of the input elements to obtain a reciprocal of the variance of the input elements; and for each of the input elements, multiplying, using the first multiplier of the non-linear function block 248, the difference value associated with the input element and the reciprocal of the variance of the input elements to obtain the layer normalization value of the input element.

[0046] In various embodiments, the layer normalization function is computed in two stages.At a first stage, the mean of input elements and the square of the variance of the input elements are computed, and the input elements are written to one of the plurality of matrix buffers 240 after quantization. At a second stage, the input elements are read from the above-mentioned one of the plurality of matrix buffers 240 after dequantization and for each of the input elements, the layer normalization value of the input element is computed based on the mean of input elements and the square of the variance of the input elements computed at the first stage.

[0047] In various embodiments, for computing the GELU function, the above-mentioned computing (at 304), using the non-linear function block 248 of the kernel block 210, the nonlinear function comprises: computing, using the set of PLF units of the plurality of PLF units, the GELU function based on input elements.

[0048] In various embodiments, the method 300 computes matrix multiplication operations according to the above-mentioned first dataflow (e.g., the MKN-RS dataflow) or the above- mentioned second dataflow (e g., the NKM-CS dataflow).

[0049] In various embodiments, the method 300 computes non-linear functions according to the above-mentioned third dataflow (e.g., the MNK-OS dataflow) or the above-mentioned fourth dataflow (e g., the NMK-OS dataflow).

[0050] In various embodiments, for each of the plurality of kernel blocks 210, the method 300 further comprises converting, using the partial sum buffer comprising the first partial sum buffer and the second partial sum buffer, a dataflow for computing a matrix multiplication operation to a dataflow for computing a non-linear function for the non-linear function block 248.

[0051] In various embodiments, the first dataflow for computing the matrix multiplication operation is converted to the third dataflow for computing the non-linear function, or the second dataflow for computing the matrix multiplication operation is converted to the fourth dataflow for computing the non-linear function.

[0052] It will be appreciated by a person skilled in the art that the terminology used herein is for the purpose of describing various embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0053] Any reference to an element or a feature herein using a designation such as “first”, “second” and so forth does not limit the quantity or order of such elements or features, unless stated or the context requires otherwise. For example, such designations may be used herein as a convenient way of distinguishing between two or more elements or instances of an element. Thus, a reference to first and second elements does not necessarily mean that only two elements can be employed, or that the first element must precede the second element, unless stated or the context requires otherwise. In addition, a phrase referring to “at least one of’ a list of items refers to any single item therein or any combination of two or more items therein.

[0054] In order that the present invention may be readily understood and put into practical effect, various example embodiments of the present invention will be described hereinafter by way of examples only and not limitations. It will be appreciated by a person skilled in the art that the present invention may, however, be embodied in various different forms or configurations and should not be construed as limited to the example embodiments set forthhereinafter. Rather, these example embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present invention to those skilled in the art.

[0055] Transformer-based deep neural networks (DNNs) have dominated several artificial intelligence (AT) fields with remarkable performance. However, the scaling up of Transformer models up to trillions of parameters and computation operations has made them both compute- and data-intensive. This poses a significant challenge to utilize Transformer models, e.g., in the area- and power-constrained systems. In this regard, according to various example embodiments, there is provided a vision transformer accelerator (ViTA) (which may herein be referred to as the present ViTA) with a scalable and highly efficient architecture, as well as a highly efficient dataflow, for vision transformers (ViT) to accelerate the entire ViT workload with high area and power efficiency. According to various example embodiments, the present ViTA adopts a memory-centric, hardware-efficient dataflow to reduce memory footprint and data movement, and exploits the parallelism and locality of the computation. For example, the present ViTA reduces 76.71% memory capability requirements for the Multi-head Self Attention (MHA) with VGA images as input compared to the original dataflow. Additionally, a fused configurable special function module (e g., corresponding to the non-linear function block 248 of the kernel block 210 described hereinbefore according to various embodiments of the present invention) is introduced to compute different types of non-linear functions in ViT, e.g., GELU, Softmax, and LayerNorm, optimizing hardware resource sharing, and further improving area and power efficiency. As will be discussed later below, experimental results show that the present ViTA achieves 16.384 TOPS with area and power efficiencies of 2.13 TOPS / mm2and 1.57 TOPS / W at 1 GHz targeting 28 nm node, surpassing state-of-the-art Transformer accelerators by 27.85 times and 1.40 times, respectively.

[0056] In particular, according to various example embodiments, there are provided:• a ViTA with a scalable architecture for the entire ViT workload, exploiting a memorycentric, hardware-efficient dataflow to reduce memory requirement and data movement.• a fused special function module to compute different types of nonlinear functions in ViT, e g., GELU, Softmax, and Layer-Norm, optimizing hardware resource sharing and further improving area and power metrics.• a comprehensive design space exploration of the number of ViTA Kernels, and the number of vector-multiplication units (VMUs) in each VMU lane, which is used to trade-off the area and power.• a performance analysis of the present ViTA architecture synthesized in the 28 nm FD- SOI technology node as an example, and comparison with the state-of-the-art Transformer accelerators.

[0057] For better understanding, different conventional methods for accelerating the Transformer models and optimizations on the non-linear functions will now be described.Transformer Models

[0058] The Transformer network, introduced by Vaswani etal., “Attention is all you need,” Advances in Neural Information Processing Systems, vol. 30, 2017 (hereinafter referred to as the Vaswani reference), serves as a pivotal reference for both related works and subsequent models It encompasses attention mechanisms, a fundamental aspectthat was initially presented in the same work. The Transformer architecture comprises two core structures: the encoder and the decoder. These components include two sub-layers each, namely Multi-Head Self Attention (MHA, or MSA) mechanisms and position-wise fully connected Feed-Forward Network (FFN) layers (also known as Multilayer Perceptron (MLP) hidden layers). The MHA mechanism allows the network to learn the dependencies and relationships between different elements in the input or output sequence, while the FFN layer applies a non-linear transformation to each element independently. Layer Normalization (LayerNorm, or LN) and residual connections are universally applied across all sublayers to facilitate the learning process. ViTA only focuses on the encoder. FIG. 4 depicts a schematic drawing of the Transformer encoder, adopted from NVIDIA FasterTransformer (“Fastertransformer,” GitHub, 12 2022. [Online], Available: https: / / github.com / NVIDIA / FasterTransformer / tree / d2923c035blee7f8d05b883f052361a673 b0e9ad)

[0059] The MHA layer is described by the Vaswani reference as follows:MHA(Q, K, V) = Concatfhead- ••• headh) ■ W°(Equation 1) where head; is the output of Scaled Dot-Product Attention, which completes the following operations:[Q, K, V] = X ■ [Wq, Wk, Wv](Equation 2)A = Softmax(Equation 3)Z = 4 - V(Equation 4) where dkrepresents the dimension of both queries and keys.

[0060] Vision Transformer (ViT) (disclosed in Dosovitskiy, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020 (herein referred to as the Dosovitskiy reference)) is a model proposed for vision tasks that build upon the core concepts of BERT (Bidirectional Encoder Representations from Transformers), using raw image patches as input. Instead of processing an entire image as a single entity, ViT divides it into a sequence of smaller patches, each consisting of 16x 16 pixels. These patches are then treated as tokens, similar to how BERT handles natural language. ViT has achieved remarkable results on various vision tasks, inspiring other Transformer-based models, e.g., DeiT (disclosed in Touvron, et al., “Training data-efficient image transformers & distillation through attention,” in International conference on machine learning, PMLR, 2021, pp. 10347-10357), PVT (disclosed in Wang, et al., “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in Proceedings of the IEEE / CVF international conference on computer vision, 2021, pp. 568-578), TNT (disclosed in Han, etal., “Transformer in transformer,” Advances in Neural Information Processing Systems, vol. 34, pp. 15908-15919, 2021) and Swin (disclosed in Liu, et al., “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE / CVF international conference on computer vision, 2021, pp. 10012-10022) to follow its footsteps. The core operation of these models is the MHA, which is applied to the sequence of image patches, followed by fully connected layers that perform non-linear transformations.Accelerations of Entire Transformers

[0061] Most Transformer models still rely on the general-purpose platform GPU for training and inference, which is not energy -efficient. To reduce the memory footprint, inference latency, and power consumption, several works have been proposed to accelerate the Transformer models on GPUs. I-BERT (disclosed in Kim, et al., “Lbert: Integer-only bert quantization,” in International conference on machine learning. PMLR, 2021, pp. 5506-5518 (herein referred to as the Kim reference)) uses lightweight integer-only approximation methods for non-linear operations, such as GELU, Softmax, and Layer Normalization, performing an end-to-end integer-only BERT inference without any floating-point calculation on GPUs. It approximated the GELU (I-GELU) and the experimental (LEXP) function by second-orderpolynomials and computed the square-root function (I-SQRT) by the iterative algorithm proposed in Crandall et al., “Prime numbers: a computational perspective”, Springer Science & Business Media, 2006, vol. 182 (herein referred to as the Crandall reference). Similarly, I- ViT (Li et al. , “I-vit: integer-only quantization for efficient vision transformer inference,” arXiv preprint arXiv:2207.01405, 2022 (hereinafter referred to as the Li reference)) performs an end- to-end integer-only ViT inference on GPUs with approximated Shiftmax, ShiftGELU, and I- LayerNorm.

[0062] NVIDIA FasterTransformer is a de-facto framework that contains highly optimized Transformer layers for many Transformer models. It offers two quantization modes, where the input and output of GEMM operations are INT8 and INT32 / INT8 respectively, while other non-linear functions are FP32. Various example embodiments adopt the GEMM quantizations from the FasterTransformer and extend the non-linear functions to INT32. The data formats of the basic kernels (operations) of ViT in the FasterTransformer and the present ViTA according to various example embodiments of the present invention are summarized in Table 1 shown in FIG. 5.

[0063] FlashAttention (disclosed in Dao, etal., “Flashattention: Fast and memory-efficient exact attention with io-awareness,” Advances in Neural Information Processing Systems, vol. 35, pp. 16344 - 16359, 2022) uses a data layout that optimizes memory access patterns on GPUs, reducing the number of cache misses and improving memory efficiency. It also introduces block-sparse attention, which reduces the number of computations required.

[0064] Accelerating the entire Transformer workload on ASIC was also explored. SwiftTron introduced an architecture for the Transformer models with very large chip size, i.e., 273 mm2, and high power, i.e., 33.64 W. Although this architecture adopted the optimized approach for the non-linear functions in the above-mentioned Kim reference, the area and power are still very high, as it directly maps the whole Transformer models to the hardware. Besides, no special synchronizations are adopted in SwiftTron, hence SwiftTron is only executed at low frequency.

[0065] Nag et al. , “ Vita: A vision transformer inference accelerator for edge applications,” arXiv preprint arXiv:2302.09108, 2023 introduced an architecture for ViT on FPGAs, adopting the non-linear functions units from the former study (disclosed in Lu, et al., “Hardware accelerator for multi-head attention and position-wise feed-forward in the transformer”, in 2020 IEEE 33rd International System-on-Chip Conference (SOCC), pp. 84-89). The MHA is computed head by head, reducing the memory requirements on-chip. However, five dedicatedPE blocks are needed to compute the MHA, decreasing the flexibility of the architecture and the utilization of those PEs, for supporting both MHA and MLP layers.

[0066] Approximation schemes are introduced to accelerate the entire Transformer workload. NN-LUT (disclosed in Yu, et al., “Nn-lut: neural approximation of non-linear operations for efficient transformer inference,” in Proceedings of the 59th ACM / TEEE Design Automation Conference, 2022, pp. 577-582 (herein referred to as the Yu reference)) introduced a framework that employs a simple neural network as a universal approximator with its structure equivalently transformed into a Lookup Table (LUT). It approximated GELU, experimental function, division, and the reciprocal of the square root function by the LUT.

[0067] In Wang et al., “A 28nm 27.5 tops / w approximate-computing-based transformer processor with asymptotic sparsity speculating and out-of-order computing,” in 2022 IEEE International Solid-State Circuits Conference (IS SCC), vol 65. IEEE, 2022, pp 1-3 (hereinafter referred to as the Wang 2022 reference), and Wang etal.,“ A 28nm 77.35 tops / w similar vectors traceable transformer processor with principal-component-prior speculating and dynamic bitwise stationary computing,” in 2023 IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits). IEEE, 2023, pp. 1-2 (the Wang 2023 reference). Wang et al. introduced several techniques to utilize similarity and sparsity to achieve high energy efficiency. In the Wang 2022 reference, approximate computing is adopted to compute small values with large errors for energy-saving, while computing large values exactly, matching the error tolerance of attention. The PCSU is introduced in the Wang 2023 reference to remove the redundant multiplications whose input vectors are similar.

[0068] However, re-training is required in works with dedicated approximated Transformer models to map the general existing models to the corresponding accelerators, which is not flexible and time-consuming for large modelsAccelerations of Individual Operations in Transformers

[0069] Acceleration on MHA Layer'. A1(disclosed in Ham et al., “A’: Accelerating attention mechanisms in neural networks with approximation,” in 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2020, pp. 328-341 (herein referred to as the Ham 2020 reference)) first introduced an approximate candidate generation method to reduce the amount of computation in the attention mechanism. ELSA (disclosed in Ham et al., “Elsa: Hardware-software co-design for efficient, lightweight selfattention mechanism in neural networks,” in 2021 ACM / TEEE 48'1' Annual InternationalSymposium on Computer Architecture (ISC A), IEEE, 2021, pp. 692-705 (herein referred to as the Ham 2021 reference)) adopted an approximate self-attention mechanism to reduce the number of computations. Besides filtering out the unimportant tokens, it also introduced hash values and Hamming distances to compute the approximate similarity between a query and each key instead of the dot product. Although the above works have achieved significant performance improvement on MHA layers, they cannot accelerate the entire workload of Transformer models with optimized dataflow.

[0070] Acceleration on GELU'. GELU is introduced in Hendrycks et al., “Gaussian error linear units (gelus),” arXiv preprint arXiv: 1606.08415, 2016 (herein referred to as theHendrycks reference), whose original equation is Equation (5) below and can be approximated with Equation (6) below. From the Hendrycks reference, the error function can be approximated as a sigmoid function, as shown in Equation (7) below. Tn FasterTransformer, GELU is implemented as Equation (6), replacing- with a constant value.(Equation 6)« X ■ (j(1.702x)(Equation 7) where a is the sigmoid function, and erf is the error function, as shown in Equation (8) below. erf((Equation 8)

[0071] In I-BERT, the above-mentioned Kim reference proposed I-GELU, which approximates the error function by a second-order polynomial. The above-mentioned Marchisio reference extended this work from GPU to ASIC. In LViT, ShiftGELU is proposed based on Equation (7), replacing the sigmoid function with ShiftExp and IntDiv. In the above-mentioned Yu reference and Lu et al., “An efficient piecewise linear approximation of non-linear operations for transformer inference,” in 2023 IEEE 31st Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM). IEEE, 2023, pp. 206-206, GELU is approximated as a whole via Piecewise Linear approximation.

[0072] Acceleration on Softmax Softmax is a general operation to predict the class probability in many tasks, which may be expressed as follows: Softmax((Equation 9)(Equation 10) denotes the z-th element of the input vector x. C denotes the number of classes. Softmax involves expensive non-linear operations such as exponential operations and divisions, and its computational cost is expensive when dealing with numerous classes and the bit width of the input data is large

[0073] The hardware acceleration of the Softmax algorithm can be classified into three categories:A. Direct optimization, optimize the hardware implementation of exponential and division units directly.B. Mathematical transformation Apply mathematical transforming (e g., logarithmic transforming) to replace the exponential operations.C. Mathematical reformulation'. Reformulate the de-facto Softmax equation into other hardware-friendly equations, such as replacing the exponent base e to 2, new Softmax layer based on Integral Stochastic Computation (SC).

[0074] Acceleration on LayerNornr. LayerNorm is a normalization operation that is widely used in Transformer models. During inference, it requires the on-the-fly computation of the mean and variance of the input data across the channel dimension, as shown in Equation (12) and Equation (13) or Equation (14) respectively.(Equation 12)(Equation 13)(Equation 14)

[0075] Equation (13) and Equation (14) are two different ways to compute the variance. In Fang et al., “Turbotransformers: an efficient gpu serving system for transformer models,” in Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 2021, pp. 389-402, the authors proposed to compute the variances of the input data in a parallel manner on GPUs, as formulated in Equation (14) to increase the instruction execution efficiency and reduce half of the synchronizations. I-SQRT is proposed in the above- mentioned Kim reference to approximate the square root function via the iterative algorithm proposed in the Crandall reference. While in I-LayerNorm (disclosed in the above-mentionedLi reference), the square root function is computed via bit-shifting based on the integer iterative algorithm.

[0076] The computation of ViT may be formulated in Equations (15) to (16) below:{(Equation 17) y = LN(z°)(Equation 18) where L is the number of Transformer encoders, z;and z(are outputs of two sub-layers in the Transformer encoder. MHA denotes a multihead self-attention layer. FFN denotes a positionwise feed-forward networks layer. LN denotes layer normalization operation.

[0077] In various example embodiments, the forward operation of ViT include:1. pre-process: patches embedding, concat class token, and add position embedding;2. computing each layer:(a) computing layer normalization;(b) computing attention layer;(c) adding bias residual and computing pre-layer normalization;(d) computing Feed-Forward Network (FFN), i. computing full connection layer 1 ; ii. adding bias and doing Gaussian Error Linear Unit (GELU); iii. computing full connection layer 2; iv. adding bias residual;3. general layer normalization.

[0078] In various example embodiments, the forward operation of the patch and position embedding in ViT include:1 . performing conv 2D;2. adding bias;3. concatenating class-token (if with class-token);4. adding position embedding.

[0079] A vision transformer accelerator (ViTA), including an example architecture thereof, according to various example embodiments of the present invention will now be further described.ViTA Architecture and Bus Overview

[0080] FIG. 6A depicts a schematic drawing showing an architecture overview of a ViTA 600 (which may herein be referred to as the present ViTA), according to various example embodiments of the present invention (e.g., corresponding to the vision transformer accelerator 200 as described hereinbefore with reference to FIG 2 according to various embodiments of the present invention). As illustrated in FIG. 6A, the ViTA 600 comprises a plurality kernels 610 (e.g., 32 ViTA kernels as an example only) for supporting all the operations in the ViT workload, and the ViTA kernels 610 are connected by buses, including input bus 630, weight bus 634, output bus 632, and bias bus 636.

[0081] The input bus 630 is used to unicast the input data of the current encoder layer to every ViTA kernel 610. The weight bus 634 and the bias bus 636 are used to broadcast the weight and bias to every ViTA kernel 610 respectively. The output bus 632 is used to collect the output data (e.g., output data (e.g., final results of the current encoder layer) for the next encoder layer) of every ViTA kernel 610 and send it to the next encoder layer. The key andvalue bus 638 is used to collect and broadcast the key and value data to every ViTA kernel 610. The ViTA 600 further comprises two weight buffers 620-1, 620-2 that act as a pingpong buffer, whose weight data are multiplexed to the key and value bus 638 and further broadcast to every ViTA kernel 610. As shown in FIG. 6A, in various example embodiments, the output bus 632 may also be, or may be shared with, the key and value bus 638, which is multiplexed with the weight from the weight buffer 620. In various example embodiments, the ViTA kernels 610 are asynchronous.

[0082] Therefore, the ViTA 600 comprises: a plurality of kernel blocks 610, each kernel block 610 configured to compute operations of a ViT model, including matrix multiplication operations and different types of non-linear functions (e g., GELU function, softmax function and layer normalization function); and a weight buffer 620 configured as a ping-pong buffer, comprising a first weight buffer 620-1 (e.g., weight buffer A) and a second weight buffer 620- 2 (e.g., weight buffer B), for storing weight data associated with the ViT model. The ViTA 600 further comprises: a kernel input bus 630 connected to an input of each of the plurality of kernel blocks 610 and configured to transfer input data of a current encoder layer of the ViT model to each of the plurality of kernel blocks 610; a kernel output bus 632 connected to an output of each of the plurality of kernel blocks 610 and configured to transfer first output data (e.g., output data (e.g., final results of a current encoder layer) for a next encoder layer) of each of the plurality of kernel blocks 610 to a next encoder layer of the ViT model; and a weight bus 634 connected to the first weight buffer 620-1, the second weight buffer 620-2 and each of the plurality of kernel blocks 610 and is configured to transfer weight data from the first weight buffer 620-1 / the second weight buffer 620-2 to an input of each of the plurality of kernel blocks 610. In this regard, as shown in FIG. 6A, the ViTA 600 further comprises: a multiplexer 650; and a key and value bus 638 connected to the output of each of the plurality of kernel blocks 610 and an input of the first multiplexer 650. The key and value bus 638 is configured to transfer second output data (e.g., output data (e.g., key / value matrix) for a current encoder layer) of each of the plurality of kernel blocks 610 to the first multiplexer 650. The multiplexer 650 is arranged between the weight buffer 620 and each of the plurality of kernel blocks 610 and is configured to multiplex the weight data from the weight buffer 620 with the second output data received via the key and value bus 638 to produce an output (i.e.., select one of the weight data from the first weight buffer 620-1 , the weight data from the second weight buffer 620-2 and the second output data) for transfer to each of the plurality of kernel blocks 610 viathe weight bus 634. The multiplexer 650 may be controlled by a controller for performing the selection depending on the operation being performed by the ViTA 600.Vi TA Kernel 610

[0083] FIG. 6B depicts a schematic drawing showing an architecture overview of the kernel block 610 (which may also be herein referred to as a ViTA kernel or a kernel), according to various example embodiments of the present invention (e.g., corresponding to the kernel block 210 as described hereinbefore with reference to FIG. 2 according to various embodiments of the present invention). As illustrated in FIG. 6B, each ViTA kernel 610 comprises a vectormultiplication unit (VMU) lane 644 (e.g., corresponding to the matrix-multiplication unit 244 as described hereinbefore with reference to FIG. 2 according to various embodiments of the present invention), three matrix buffers 640 (e g., collectively corresponding to the matrix buffers 240 as described hereinbefore with reference to FIG. 2 according to various embodiments of the present invention), an accumulator 660, two partial sum buffers 664-1, 664-2 that act as a ping-pong buffer 664, and a special function module (or a non-linear function module) 648 (e.g., corresponding to the non-linear function block 248 described hereinbefore according to various embodiments of the present invention). Furthermore, as shown in FIG. 6B, various input data, e.g., input matrix (via the kernel input bus 630), Key and Value (KV) matrix (via the weight bus 634), weight matrix (via the weight bus 634), and bias (via the bias bus 636) are synchronized to the ViTA kernel 610 through asynchronous FIFOs 680-1, 680-2, 680-3. For example, the kernel input data, the weight data, the Key and Value (KV) data, and the bias data that cross a clock domain crossing are passed into the ViTA kernel 610 via the asynchronous FIFOs 680-1, 680-2, 680-3.

[0084] Therefore, each kernel block 610 comprises: a plurality of matrix buffers (e.g., a first matrix buffer (e.g., matrix buffer A) 640-1, second matrix buffer (matrix buffer B) 640-2 and a third matrix buffer (e.g., matrix buffer C) 640-3), each matrix buffer configured to store one or more matrices; a matrix-multiplication unit 644 communicatively coupled to the plurality of matrix buffers 640 and the weight buffer 620. The matrix-multiplication unit 644 is configured to compute a matrix multiplication operation based on at least one matrix stored in one or more of the plurality of matrix buffers 640; and a non-linear function block 648 configured to compute each of the different types of non-linear functions.

[0085] In various example embodiments, for each of the plurality of kernel blocks 610: the kernel block 610 further comprises a first multiplexer 670-1 configured to multiplex an outputof one of the plurality of matrix buffers 640 and a first output of the non-linear function block 648 to produce an output. Furthermore, the matrix-multiplication unit 644 is configured to receive the output of the first multiplexer 670-1 as a first input thereto and the output of the multiplexer 650 of the vision transformer accelerator 600 via the weight bus 634 as a second input thereto.

[0086] In various example embodiments, for each of the plurality of kernel blocks 610, the kernel block 210 further comprises a partial sum buffer 664 configured as a ping-pong buffer comprising a first partial sum buffer 664-1 and a second partial sum buffer 664-2.

[0087] In various example embodiments, the vision transformer accelerator 600 further comprises a bias bus 636 configured to transfer bias data to each of the plurality of kernel blocks 610. For each of the plurality of kernel blocks 610, the kernel block 610 further comprises: an accumulator 660; a second multiplexer 670-2; a dequantization unit 672; and a third multiplexer 670-3. In this regard, the second multiplexer 670-2 is configured to multiplex an output of the matrix-multiplication unit 644, the bias data received via the bias bus 636 and an output of the first matrix buffer 640-1 to produce an output. The dequantization unit 672 is configured to receive and dequantize the output of the second multiplexer 670-2 to produce an output. The accumulator 660 is configured to receive the output of the dequantization unit 672 as a first input thereto and a first output of the partial sum buffer 664 as a second input thereto. The third multiplexer 670-3 is configured to multiplex an output of the accumulator 660 and the output of the dequantization unit 672 to produce an output.

[0088] In various example embodiments, for each of the plurality of kernel blocks 610: the non-linear function block 648 is configured to receive a second output of the partial sum buffer 664 as a first input thereto and the output of the first matrix buffer 640-1 as a second input thereto. Furthermore, the kernel block 610 further comprises a fourth multiplexer 670-4 configured to multiplex the first output of the non-linear function block 648, a second output of the non-linear function block 648 and the output of the third multiplexer 670-3 to produce an output.

[0089] In various example embodiments, for each of the plurality of kernel blocks 610: the kernel block 610 further comprises a fifth multiplexer 670-5 configured to multiplex the output of the fourth multiplexer 670-4 and the input data of the current encoder layer received via the kernel input bus 630 to produce an output. Furthermore, the second matrix buffer 640-2 is configured to receive the output of the fifth multiplexer 670-5 as an input thereto, and the firstand third matrix buffers 640-1, 640-3 are each configured to receive the output of the fourth multiplexer 670-4 as an input thereto.

[0090] In various example embodiments, for each of the plurality of kernel blocks 610: the kernel block 610 further comprises a sixth multiplexer 670-6, a seventh multiplexer 670-7 and a demultiplexer 674. In this regard, the demultiplexer 674 is configured to receive the output of the third multiplexer 670-3 to produce a first output for input to the first partial sum buffer 664- 1 and a second output for input to the second partial sum buffer 664-2. The sixth multiplexer 670-6 is configured to multiplex an output of the first partial sum buffer 664-1 and an output of the second partial sum buffer 664-2 to produce the above-mentioned first output of the partial sum buffer 664. The seventh multiplexer 670-7 is configured to multiplex the output of the first partial sum buffer 664-1 and the output of the second partial sum buffer 664-2 to produce the above-mentioned second output of the partial sum buffer 664.

[0091] In various example embodiments, the input data transferred to each of the plurality of kernel blocks 610 via the kernel input bus 630, the weight data or the KV data transferred to each of the plurality of kernel blocks 610 via the weight bus 634 and the bias data transferred to each of the plurality of kernel blocks 610 vie the bias bus 636 are synchronized to the plurality of kernel blocks 210 through asynchronous first-in-first-out (FIFO) buffers, 680-1, 680-2, 680- 3, respectively.

[0092] Therefore, as illustrated in FIG. 6B according to various example embodiments, the asynchronous FIFO outputs from the asynchronous FIFO 680-1 are directed to the fifth multiplexer 670-5, which also receives input from the fourth multiplexer 670-4 positioned before the quantization module 690, and together they feed into matrix buffer B 640-2. Matrix buffers A 640-1, B 640-2, and C 640-3 are collectively multiplexed to serve as a single input source for the matrix-multiplication unit (also referred to as the Vector Matrix Unit (VMU) Lane) 644. The VMU Lane 644 receives another input from the first output of the non-linear block 648. These two inputs to the VMU Lane 644 are subject to multiplexing by the first multiplexer 670-1. KV or weight values are transmitted to the VMU Lane 644 via the asynchronous FIFO 680-2, while bias values are also directed through the asynchronous FIFO 680-3. The output of the VMU Lane 664, along with the output from matrix buffer A 640-1, is multiplexed by the second multiplexer 670-2 prior to entering the de-quantization module 672. One input to the accumulator 660 originates from the second multiplexer 670-2 that selects between the outputs of partial sum buffers A 664-1 and B 664-2, and the other input comes from the de-quantization module 672 The outputs of the accumulator 660 and the de-quantization module 672 are further multiplexed to serve as inputs to the ping-pong buffer 664. The non-linear function block 648 receives its first input from the multiplexed results of the ping-pong buffer 664, its second input from matrix buffer A 640-1, and its third input from other kernel blocks 610. The outputs from the third multiplexer 670-3 following the dequantization module 672 and the accumulator 660, and the first and second outputs from the non-linear function block 648, are multiplexed by the fourth multiplexer 670-4. The resultant output is directed to matrix buffer A 640-1 and the quantization module 690. The output of the quantization module 690 is transmitted to the VK bus.

[0093] As described hereinbefore, the patch embedding layer includes conv 2D, bias adding, class-token concatenating, and position embedding adding. In various example embodiments, image data is loaded into matrix buffer B 640-2. Meanwhile, the weight data is loaded into weight buffers A 620-1 and B 620-2 from a memory chip via the reading TSVs. Weight buffers A 620-1 and B 620-2 act as ping-pong buffering. The bias is loaded into the partial sum buffer 664 from the memory chip via the reading TSVs. After the data is ready, matrix A (i.e., the image data) is read from matrix buffer B 640-2 and matrix B (i.e., weight data) is read from weight buffers A 620-1 and B 620-2 into the matrix-multiplication unit 644 to do the GEMM computation. The partial sum is accumulated with the accumulator 660 iteratively with the pre-loaded bias. Class concatenation and position embedding adding are also completed with the accumulator 660 and partial sum buffer 664. The final results are written into matrix buffer A 640-1 for shortcut adding.

[0094] In various example embodiments, for the attention layer, multihead attention is computed in head iterations. In every iteration, there is only one of the h heads of the attention operation computed until they are contacted together right before the project GEMM.Step 1: data is read from matrix buffer A 640-1 to the layer normalization module and the results are sent to the matrix-multiplication unit 644 as input matrix A.Step 2: to compute the Q, K, and V matrices, their weight data is loaded into weight buffer A 620-1 and B 620-2 from the memory chip via the reading TSVs. The bias is loaded into the partial sum buffer 664 from the memory chip via the reading TSVs. After the data is ready, matrix A, i.e., the input data X, is read from matrix buffer A via the processing of the layer normalization module, and matrix B, i.e., the combination of weight data Wq, Wk , and Wv, is read from weight buffer A 620-1 and B 620-2 into the matrix-multiplication unit 644 to do the GEMM computation. The partial sum is accumulated with the accumulator 660iteratively with the pre-loaded bias. The K and V matrices are stored in matrix buffer C 640-3, while the Q matrix is stored in matrix buffer B 640-2.Step 3 : to compute A matrix, matrix A (i.e., Q matrix) is read from matrix buffer B 640- 2, matrix B (i.e., K matrix), is read from matrix buffer C 640-3, into the matrix-multiplication unit 644 to do the GEMM computation. The partial sum is accumulated with the accumulator 660 iteratively. The final results A (A matrix) are stored in matrix buffer A 640-1 after processing in the Softmax module.Step 4: to compute the Z matrix (i.e., Z = A • V in Equation (4)), matrix A (i.e., A matrix) is read from matrix buffer C 640-3, matrix B (i.e., V matrix) is read from matrix buffer B 640-2 into the matrix-multiplication unit 644 to do the GEMM computation. The partial sum is accumulated with the accumulator 660 iteratively. The final results are one head of Z, which are stored in matrix buffer B 640-2 in each head iteration.Step 1 to step 4 are repeated for head iterations until all heads of Z are computedStep 5: to compute the project GEMM, the weight data is loaded into weight buffers A 620-1 and B 620-2 from the memory chip via the reading TSVs. The bias is loaded into the partial sum buffer 664 from the memory chip via the reading TSVs. After the weight buffer A 620-1 is ready, matrix A (i.e., the input data) is read from matrix buffer B 640-2, and matrix B (i.e., the weight data) is read from weight buffer A 620-1 and B 620-2 into the matrixmultiplication unit 644 to do the GEMM computation. The partial sum is accumulated with the accumulator 660 iteratively with the pre-loaded bias. The shortcut adding is done by an external iteration of accumulation, where the shortcut data is read from matrix buffer A 640-1. The final results are written back to matrix buffer A 640-1 .

[0095] In various example embodiments, for the Feed-Forward Network (FFN) layers:Step 1: data is read from matrix buffer A 640-1 to the layer normalization module and the results are sent to the matrix-multiplication unit 644 as input matrix A.Step 2: to compute the first full connection layer, the weight data is loaded into weight buffers A 620-1 and B 620-2 from the memory chip via the reading TSVs. The bias is loaded into the partial sum buffer 664 from the memory chip via the reading TSVs. After the data is ready, matrix A (i.e., the input data) is read from matrix buffer A 640-1 via the processing of the layer normalization module, and matrix B (i.e., the weight data) is read from weight buffers A 620-1 and B 620-2 into the matrix-multiplication unit 644 to do the GEMM computation. The partial sum is accumulated with the accumulator 660 iteratively with the pre-loaded bias. The final results are stored in matrix buffers B 640-2 and C 640-3 after the processing of theGeLU module. Here the matrix buffers B 640-2 and C 640-3 are regarded as a larger virtual memory.Step 3: to compute the second full connection layer, the weight data is loaded into weight buffers A 620-1 and B 620-2 from the memory chip via the reading TSVs. The bias is loaded into the partial sum buffer 664 from the memory chip via the reading TSVs. After the data is ready, matrix A (i.e., the input data) is read from matrix buffers B 640-2 and C 640-3, and matrix B (i.e., the weight data) is read from weight buffers A 620-1 and B 620-2 into the multiplication unit 644 to do the GEMM computation. The partial sum is accumulated with the accumulator 660 iteratively with the pre-loaded bias. The shortcut adding is done by an external iteration of accumulation, where the shortcut data is read from matrix buffer A 640-1. The final results are written back to matrix buffer A 640-1.

[0096] In various example embodiments, to compute the final Classifier MLP Head GEMM, the weight data is loaded into weight buffers A 620-1 and B 620-2 from the memory chip via the reading TSVs. The bias is loaded into the partial sum buffer 664 from the memory chip via the reading TSVs. After the data is ready, matrix A (i.e., the input data) is read from matrix buffer A 640-1, and matrix B (i.e., the weight data) is read from weight buffers A 620- 1 and B 620-2 into the multiplication unit 644 to do the GEMM computation. The partial sum is accumulated with the accumulator 660 iteratively with the pre-loaded bias. The final results are output directly.VMU Lane 644

[0097] In various example embodiments, the matrix-multiplication unit 644 is configured as a VMU lane comprising a series of VMUs 645 (e g., see FIG. 7). Each VMU 645 is configured to perform a respective vector multiplication operation (vector-vector multiplication) associated with the matrix multiplication operation. As illustrated in FIG. 7, in temporal, the VMU Lane 644 computes a single GEMV operation, namely, c — a X b, where a is a 1 X k vector, and b is a k X n matrix. Each VMU Lane 644 comprises n VMUs 645. In various example embodiments, the matrix B may be mapped into n VMUs 645. For example, FIG. 8 illustrates the data path of the VMU 645, which outputs the inner product result of two vectors, whose length is k. In particular, FIG. 8 illustrates the computation data path of VMU 645, whereby the output is the inner project of two input vectors. The left column indicates the bit width of each value and all the values are signed In various example embodiments, every two levels of the adder tree may be piped for better timing, i.e., the pipeline stages is (log K) / 2.Special Function Module (non-linear function block) 648

[0098] FIG. 9A depicts a schematic drawing showing an architecture overview of the special function module (which may also be referred to as the non-linear function block) 648, according to various example embodiments of the present invention. Non-linear functions in the ViT model, e.g., GELU, Softmax, and LayerNorm, are computed by the special function module 648. Accordingly, FIG. 9A shows the computation data path of the special function module 648, which is used to compute each of the different types of non-linear functions, including GELU, Softmax, and LayerNorm. The special function module 648 comprises a plurality of piecewise linear function (PLF) units 910-1 (three PLF units 0-3), 910-2 (PLF unit 4) and a plurality of adder trees 912-1, 912-2 configured to compute the different types of nonlinear functions. The usages of the PLF units 910-1, 910-2 and the two adder trees 912-1, 912- 2 in different non-linear function modes are summarized in Table II shown in FIG. 10. For example, the design may utilize the PLF unit from the above-mentioned Yu reference, which is adopted in BERT without accuracy degradation. In various example embodiments, the GELU function, exponential function, reciprocal function, and the reciprocal of the square root function are approximated by the PLF unit. For illustration purpose and better understanding, arrows labelled with “GELU” indicate the data path used only for the GELU, arrows labelled with “Softmax” indicate the data path used only for the Softmax, and arrows labelled with “LayerNorm” indicate the data path used only for the LayerNorm.

[0099] An example operation of the special function module 648 for computing the Softmax function (i.e., under the Softmax mode) will now be described according to various example embodiments of the present invention. In various example embodiments, the special function module 648 is configured to compute the Softmax according to Equation (9) above, with the approximated exponential function and reciprocal function. In this regard, for computing the softmax function: a set of PLF units 910-1 of the plurality of PLF units is configured to compute exponential values eXiof input elements xt, respectively; a first adder tree 912-1 of the plurality of adder trees and a first adder 913-1 are configured to compute a sum of the exponential values 2 eXiof the input elements xt; a further PLF unit 910-2 of the plurality of PLF units is configured to compute a reciprocal of the sum of the exponentialvalues of the input elements xt; and a first multiplier 916-1 of the non-linear function block 648 is configured to, for each of the input elements x;, multiply the exponential value ex’ of the1 input element x,- with the reciprocal —7- of the sum of the exponential values of the input 2-1elements xtto obtain the softmax value of the input element X .

[0100] For illustration purpose and better understanding, as mentioned above, the data paths used only in the Softmax mode are denoted by arrows labelled with “softmax” in FIG. 9A, according to various example embodiments of the present invention. For better understanding in computing the Softmax function and as a comparative reference, FIG. 9B depicts a schematic drawing showing a computation data path for a normal dedicated / standalone Softmax function module (i.e., can only be used to compute the Softmax and it is not configureable and there is no resource reuse) prior to the Softmax function module being integrated in the non-linear function block 648 with other non-linear function modules. As shown in FIG. 9A, the exponential values eXiare computed by the PLF Units 0 to 3 910-1, and they are stored in the exponential buffer 920-1 for further use. Meanwhile, the exponential values are also summed up by the adder tree 912-1 and the accumulator (or adder) 913-1 to compute the sum of the exponential values ' eXi. Instead of using several expensive dividers, learned from Table III shown in FIG. 11, the vector division is converted to one reciprocal function and one relatively cheaper vector multiplication. In particular, Table III shows the rough area and power for arithmetric modules in different data types in 28nm 1.05V. After the whole row is processed, the reciprocal — 7 f of the sum of the exponential values is computed by PLF Units 4 910-2,1 which acts as the reciprocal function. Then the reciprocal — r of the sum of the exponential values is multiplied by the exponential values eXi, which are read from the exponential buffer920-1, to compute the Softmax values r- 7-. As the Softmax is applied row-wise, according to 2^G 1various example embodiemnts, the corresponding GEMM dataflow is also row-wise to reduce the memory requirement of the exponential buffer 920-1. Table IV shown in FIG. 12 summarizes the dataflows of each ViT layer, including the name of ViT operations, the corresponding data source and destination of the GEMM computation, the GEMM dataflow, and the special functions before and after the GEMM operation. As shown in Table IV, the GEMM dataflow may be MKN when the Softmax is applied.

[0101] In various example embodiments, to reduce the computational complexity and minimize the area while obtaining good accuracy, LUT-based Softmax implementations are utilized based on PLF for approximating Softmax (LUT-PLF- Softmax). The function of Softmax is expressed in Equation (9) above. For illustration purpose, example ports and parameters of LUT-PLF- Softmax are shown in FIG. 9C. The sum of inputs is accumulated inseveral clock cycles due to the computation limitations of the matrix multiplication unit 644. The number of parallel inputs equals the number of parallel outputs in one row from the multiplication unit 644. The adder tree sums those inputs together after the exponential LUT and the partial sum is accumulated with the previous partial sums by the following adder. Meanwhile, Xi or exp( i) are parallel written into the memory After the accumulation finishes, Xi or exp(xt) are parallel read out for the final computation. The number of parallel outputs also equals the number of parallel outputs in one row from the matrix multiplication unit 644 To enable the pipeline, one register is inserted to store 2 ex>. In this case, the accumulation register is ready for accumulating the next row.

[0102] An example operation of the special function module (non-linear function block) 648 for computing the layer normalization (LayerNorm) function (i.e., under the LayerNorm mode) will now be described according to various example embodiments of the present invention. For computing the layer normalization function: the first adder tree 912-1 and a first adder 913-1 is further configured to compute a sum of input elements xp, a second adder tree 912-2 of the plurality of adder trees and a second adder 913-2 is configured to compute a sum of squares of the input elements x;2; a second multiplier 914-2 of the non-linear function block 648 is configured to compute a mean of input elements based on the sum of the input elements X xp, a third multiplier 914-3, a fourth multiplier 914-4 and a first subtractor 922-1 of the nonlinear function block 648 are configured to compute a square of a variance cr2of the input elements based on the mean of the input elements and the sum of the squares of the input elements computed; a second subtractor 922-2 of the non-linear function block 648 is configured to, for each of the input elements x;, perform a subtraction between the mean and the input element to obtain a difference value (p — Xj) associated with the input element xp, the further PLF unit 910-2 of the plurality ofPLF unitsis further configured to compute a reciprocal of the square root of the square of the variance of the input elements to obtain the reciprocal of the variance - of the input elements; and the first multiplier 916-1 of the non-linear function block 648 is further configured to, for each of the input elements x£, multiply the difference value (jti — x associated with the input element xtand the reciprocal of the variance - of the input elements to obtain the layer normalization value of the input element x,.

[0103] For illustration purpose and better understanding, as mentioned above, the data paths used only in the LayerNorm mode are denoted by arrows labelled with “LayerNorm” in FIG. 9A according to various example embodiments of the present invention. For betterunderstanding in computing the LayerNorm function and as a comparative reference, FIG. 9D depicts a schematic drawing showing a computation data path for a normal dedicated / standalone LayerNorm function module prior to the LayerNorm function module being integrated in the non-linear function block 648 with other non-linear function modules.

[0104] As described hereinbefore, there are two formulas to compute the variance cr in Equation (11), which are formulated in Equation (13) and Equation (14). To reduce the latency and improve the throughput, the ViTA 600 according to various example embodiments of the present invention adopts Equation (14) to compute the variance (J . In various example embodiments, as shown in FIG. 9A, the sum of the input and the sum of the squares of the input x are computed simultaneously. However, in various example embodiments, as the layer normalization is computed across the whole matrix while each ViTA kernel 610 only computes a chunk of the matrix, the sum of the input and the sum of the squares of the input x2are accumulated across the ViTA kernels 610 for the final result.

[0105] In various example embodiments, the ViTA 600 is configured to compute the LayerNorm function in two stages to reduce the memory requirements of the middle results. In the first stage, the mean / J. and the variance a (or more specifically, the square of the variance <72) are computed, while the input x;is written back to one of the matrix buffers 640 (e.g., matrix buffer A 640-1) after quantization. In the second stage, the input X; is read from that matrix buffer (e.g., matrix buffer A 640-1) after dequantization, then the layer normalization results are computed following Equation (1 1) with the mean fl and the variance a computed in the first stage. In this regard, as the LayerNorm is computed in two stages, there is no special constraint on the GEMM dataflow.

[0106] In various example embodiments, layer normalization is implemented with LUTs. For illustration purpose, example ports and parameters of LUT-PLF -LayerNorm are shown in FIG. 9E. As shown in FIG. 9D, by computingxtand 2f=ixt simultaneously, variance o is computed once (.1 is ready. Hence, the results of LayerNorm can be computed with only one accumulation time. To enable the pipeline, registers are inserted to store and n

[0107] An example operation of the special function module 648 for computing the GELU function (i.e., under the GELU mode) will now be described according to various example embodiments of the present invention. For computing the GELU function, the set of PLF units 0-3 910-1 of the plurality of PLF units is configured to compute the GELU function based on input elements. In this regard, the set of PLF units 0-3 910-1 is configured to compute the GELU function based on Lookup Tables (LUTs) for approximating GELU, whereby coefficients inthe LUTs are configurable to compute the GELU function, as indicated in Table II shown in FIG. 10. For illustration purpose and better understanding, as mentioned above, the data paths used only in the GELU mode are denoted by arrows labelled with “GELU” in FIG. 9A according to various example embodiments of the present invention. To compute the GELU function, the input data vector is sent to the PLF Units 0 to 3 910-1, which act as the GELU function. Then the output from the PLF Units 0 to 3 910-1 is output. As the GELU function is applied element-wise according to various example embodiments, there is no special constraint on the GEMM dataflow. That is, there is no data dependency, and the output of each element can be computed once any of the input data is obtained. In contrast, operations such as softmax is row-wise, which means the output cannot be computed unless the data of a whole row is obtained. Therefore, for the softmax function, the preferred dataflow may be row-wise. Otherwise, if data comes in column-wise, the memory has to store the whole matrix (instead of just one row) for the computation.

[0108] In various example embodiments, GELU is implemented with LUTs using PLF for approximating GELU (LUT-PLF-GeLU). For illustration purpose, example ports and parameters of LUT-PLF-GeLU are drawn in FIG. 9F. The computation data path of LUT-PLF- GeLU is shown in FIG. 9G, and there are two potential implementations for the linear part, as described in Equations (19) and Equation (20) below. f 0, ifx < —3GELU(x) = |sj X x + tj, if di-i < x < di, for 0 < i < MEM_CNT — 1(. x, if x > 3(Equation 19) ifx < -3 if df-i < x < d[, for 0 < i < MEM_CNT — 1ifx > 3(Equation 20)For better understanding in computing the GELU function and as a comparative reference, FIG. 9G shows the computation data path for a normal dedicated / standalone GELU function module prior to the GELU function module being integrated in the non-linear function block 648 with other non-linear function modules. The ports of a regfile, or a RAM, are shown in FIG. 9H, where the chip selection signal CEN is active low.

[0109] As summarized in Table IV shown in FIG. 12, the GELU function and the second stage of the LayerNorm function are computed simultaneously in the Special function module 648.

[0110] ViTA dataflows, including the supported GEMM dataflows and the fused multihead attention dataflow, will now be described according to various example embodiments of the present invention. As described hereinbefore, the ViTA 600 is advantageously configured to enable highly efficient memory-centric dataflow for computing operations of the ViT model. In this regard, various example embodiments introduce methods or approaches of operating the ViTA 600 to optimize the dataflow within the ViTA architecture to reduce the memory bandwidth and capability requirements. According to various example embodiments, Row Stationary (RS) and Column Stationary (CS) dataflows are adopted for General Matrix Multiply (GEMM) operations. Meanwhile, two Output Stationary (OS) dataflows are introduced to compute special functions (non-linear functions) while decreasing throughput requirements. A fused multi-head attention dataflow according to various example embodiments is also presented for further optimization Table TV shown in FIG. 12 summarizes the dataflows of each ViT layer, including the name of ViT operations, the corresponding data source and destination of the GEMM computation, the GEMM dataflow, and the special functions (nonlinear functions) before and after the GEMM operation.Dataflow Discussion

[0111] GEMM is a foundational operation in Transformer models and is responsible for matrix multiplications and additions A GEMM operation is formulated as:CMXN=^MxKX^KxN(Equation 21) where M, N, and K are the number of rows and columns of the matrix C and the number of columns of the matrix^ respectively.

[0112] Accordingly, there are six ways to implement the GEMM operation in software and hardware using three nested for loops, namely, MNK, NMK, KMN, MKN, KNM, and NKM. In the above six categories, the sequence of M, N, and K represent the order of the for-loop in the GEMM operation, from outer to inner. Based on the data reuse, MNK and NMK are Output Stationary (OS), KMN and KNM are Input Stationary (IS) and Weight Stationary (WS), respectively. Based on both data reuse and movements, MKN and NKM are Row Stationary (RS) and Column Stationary (CS), respectively. An example MKN-RS GEMM dataflow (pseudocode thereof) is shown in FIG. 13. In the example MKN-RS GEMM dataflow, the partial sums are computed firstly via the . dimension and then accumulated via the K dimension for the final outputs of one row. Finally, the outputs of other rows (M dimension) are computed.

[0113] FIG. 14 shows a table (Table V) providing a summary of dataflows, optimum conditions, memory and bandwidth bottlenecks in GEMM, as well as the throughput requirements of special functions, and whether they are supported in ViTA. As summarized in Table V, different permutations impact both performance and energy efficiency with different access patterns for matrix A, matrix B, and the partial sums. More specifically, FIGs 15 and 16 illustrate the on-chip memory requirements and bandwidth requirements for an accelerator to support the GEMM operations in ViT-base in different dataflows. In particular, FIG. 15 shows the on-chip memory requirements (KB) for an accelerator to support the GEMM operations in ViT-base with a VGA image resolution as input in different dataflows. In FIG. 15, the values are shown in the log scale The computation ability of this accelerator on AT, K, and N dimensions is 16, i.e., 8. 192 TOPS at 1 GHz. The data format of matrix A and matrix B is INT8 while that of the partial sum is INT32. Relatively large memories are required for matrix B in MNK-OS dataflow, for matrix A in NMK-OS dataflow, and for partial sums in KMN-IS and KNM-WS dataflows, respectively to maximize on-chip data reuse. FIG. 16 shows the bandwidth requirements (GB / s) of matrices A, B and C, and the throughput requirements (GOPS) of the special functions for an accelerator to support the GEMM operations in ViT- base with a VGA image resolution as input in different dataflows. The computation ability of this accelerator on M, K, and N dimensions is 16, i.e., 8.192 TOPS at 1 GHz. The data format of matrices A, B and C is INT8. Matrices A and B are read off-chip and the matrix C is written back to the off-chip memory once the partial sums are accumulated. Relatively large bandwidths are required for matrix B in both MNK-OS and MKN-RS dataflows and for matrix A in both NMK-OS and NKM-CS dataflows. Only in MNK-OS and NMK-OS dataflows, the throughput requirement of the special functions is relatively small.

[0114] To reduce both memory footprint and bandwidth, MKN-RS and NKM-CS are preferred for the GEMM operations. However, MNK-OS and NMK-OS dataflows are preferred for the special functions to reduce the throughput requirements hence reducing the computation resources for the special function module.GEMM Dataflows in ViTA

[0115] For the GEMM dataflow, according to various example embodiments, ViTA supports both MKN (i.e., RS) and NKM (i.e., CS) dataflows for higher energy efficiency and less memory footprint in their optimum conditions. In this regard, the GEMM operation involves a first input matrix thereto having a row dimension and a column dimension (e g ,matrix A with dimensions of M K), a second input matrix thereto having a row dimension and a column dimension (e.g., matrix B with dimensions of K*N) and an output matrix therefrom having a row dimension and a column dimension (e.g., matrix C with dimensions of M*N). A first dataflow for the GEMM operation is a row stationary (RS) dataflow whereby executions of for-loops of the GEMM operation, from inner to outer, is according to an order of the column dimension of the second input matrix (e.g., along A dimension), the column dimension of the first input matrix (e.g., along K dimension) and the row dimension of the first input matrix (along M dimension) (i.e., corresponding to the MKN-RS dataflow). A second dataflow for the GEMM operation is a column stationary (CS) dataflow wherein executions of for-loops of the matrix multiplication operation, from inner to outer, is according to an order of the row dimension of the first input matrix (e g., along M dimension), the column dimension of the first input matrix (e.g., along K dimension) and the column dimension of the second input matrix (e.g., along A dimension) (i.e., corresponding to the NKM-CS dataflow).100116] In various example embodiments, tiling is adopted for large-scale GEMM operations, which is illustrated in FIG. 17. In this regard, for a matrix-matrix multiplication to be computed by the vision transformer accelerator 600 above a predefined size (e.g., an input matrix (e.g., matrix A) for the matrix multiplication operation exceeding a predefined size), the vision transformer accelerator 600 is configured to split an input matrix for the matrix multiplication operation into multiple input matrix chunks and map the multiple input matrix chunks to multiple kernel blocks of the plurality of kernel blocks 610, respectively, for the multiple kernel blocks 610 to compute matrix multiplication operations (e.g., vector-matrix multiplication) based on the multiple input matrix chunks, respectively, in parallel. For example, the matrix A (IRMxK) is split into m chunks, which are mapped into m ViTA kernels 610. In temporal, the ViTA 600 computes a c = a X b GEMM operation, where a G Kmxk, b G IRfcxn. Accordingly, each GEMM operation is distributed across multiple ViTA kernels 610 to enhance parallelism and performance. More specifically, in temporal, the ViTA 600 computes a c = a X b GEMM operation, where a is an m X k matrix, and b is a k X n matrix. The GEMM operation is further mapped to m ViTA kernels 610, where each kernel 610 computes a m X k GEMV operation, where a is a 1 X k vector, and b is a k X n matrix.Special Functions (Non-Linear Functions) Dataflows in ViTA

[0117] For the special function dataflow, non-linear operations, critical to Transformer computations, are resource-intensive, occupying 60% of the memory usage in GPUs and 45%area of the accelerator. Therefore, the dataflow for special functions are carefully designed according to various example embodiments to improve the area-efficiency and powerefficiency of the special functions.

[0118] In various example embodiments, two OS dataflows, i.e., MNK and NMK, have been adopted in the special function (non-linear function) module 648 of ViTA to reduce the throughput requirements of the special functions from n per cycle to n / (K / k~) per cycle, as summarized in Table V shown in FIG 14. Tn this regard, the non-linear function involves an input matrix thereto having a row dimension and a column dimension (e.g., dimensions of M*N). A third dataflow is an output stationary (OS) dataflow whereby executions of for-loops of the non-linear function, from inner to outer, is according to an order of the column dimension of the input matrix (e.g., along N dimension) and the row dimension of the input matrix (e.g., along AT dimension) (e.g., corresponding to the MNK-OS dataflow). The fourth dataflow is an output stationary (OS) dataflow whereby executions of for-loops of the non-linear function, from inner to outer, is according to an order of the row dimension of the input matrix (e g , along M dimension) and the column dimension of the input matrix (e.g., along N dimension) (e.g., corresponding to the NMK-OS dataflow).

[0119] In various example embodiments, to convert the GEMM dataflow to the corresponding special function dataflow, two partial sum buffers 664-1, 664-2 are introduced in each ViTA kernel 610, acting as ping-pong buffers. Accordingly, for each kernel block 610, the partial sum buffer 664 comprising the first partial sum buffer 664-1 and the second partial sum buffer 664-2 is configured to convert a dataflow for computing a matrix multiplication operation to a dataflow for computing a non-linear function for the special function (non-linear function) module 648. In various example embodiments, the partial sum buffer 644 is configured to convert the first dataflow (e.g., the MKN-RS dataflow) for computing the GEMM operation to the third dataflow (e.g., the MNK-OS dataflow) for computing the non-linear function, or convert the second dataflow (e.g., the NKM-CS dataflow) computing the GEMM operation to the fourth dataflow for computing the non-linear function (e.g., the NMK-OS dataflow).

[0120] As an illustrative example, the partial sum buffer 644 may be configured to convert the MKN-RS dataflow for computing the GEMM operation to the NKM-CS dataflow for computing the non-linear function in the following manner. Assuming that the dimensions of the matrices associated with the GEMM operation is M, K, and N, the parallelism of the hardware on the three dimensions are m, k, and n respectively, m ViTA kernels 610 eachN operates on a GEMV operation. The GEMV output matrix is computed with a vector n in a -NK continuously cycles, every — cycles. In this regard, vector n is stored in the first partial sum buffer 664-1 continuously. When the N values are in the first partial sum buffer 664-1 , the special function module 648 starts to read n values every cycles. Meanwhile, the another row (the M for-loop) of the GEMV output is stored in the second partial sum buffer 664-2.Fused Multi-Head Attention Dataflow

[0121] In various example embodiments, a memory-centric, hardware-efficient fused Multi-Head Attention dataflow is introduced in the ViTA 600 to reduce the memory capability requirements across the Multi-Head Attention layer. In this dataflow, MHA is computed per head, and memories are reused during the computation of every self-attention. As an illustrative example, FIG. 18A shows a fused multi-head attention algorithm (herein referred to as Algorithm 1) from the input matrix X to the input of the MLP layers, according to various example embodiments of the present invention, and FIG. 18B illustrates the corresponding fused multi-head attention dataflow. In particular, FIG. 18B shows the fused multi -head attention dataflow in the view of each module, where MB stands for Matrix Buffer 640, WB stands for Weight Buffer 620, PSum A and B are Partial Sum Buffer A 644-1 and B 644-2, which are acting as a ping-pong buffer. The GEMMs in PerHeadQKV, PerHeadA, PerHeadZ are MKN dataflow, while the GEMM in Project GEMM in NKM dataflow.

[0122] FIG. 19 depicts a schematic drawing illustrating the dataflow of the PerHeadQKV function in Algorithm 1 FIG. 20 depicts a schematic drawing illustrating the dataflow of the PerHeadA and PerHeadZ functions in Algorithm 1.Gains of the ViTA Dataflows

[0123] FIG. 1 demonstrates that the original MHA that supports VGA images requires 22.66 MB when dedicated memories are used for each matrix. In contrast, with the fused multihead attention according to various example embodiments, this requirement drops to 5.28 MB, marking a 76.71% reduction This reduction translates to the potential for deploying larger or more complex models on constrained hardware. In the MLP layers, while FC 1 results require 3.52 MB, the weights demand 2.25 MB. By adopting dual GEMM dataflows, 48 KB is needed for the weight buffers while maintaining the weight reuse. The bandwidth requirements for theweights are reduced from 256 GB / s to an affordable 56 GB / s when our ViTA operates at 1 GHz. Such savings increase both area efficiency and power efficiency.Experimental Setup

[0124] Tn experiments conducted, the ViTA 600 was implemented in Chisel in a configurable manner and evaluate it for ViT-base with 224x224 RGB images as input. Memory IPs (Regfiles and SRAMs) are generated by ARM memory compilers. The generated Verilog from Chisel and memory IPs are synthesized by Cadence Genus with Foundry 28 nm FD-SOI technology node for both area and power estimation. The experimental setup is summarized in Table VI shown in FIG 21Experimental Results

[0125] The performance (GOPS, throughput, and latency), area, power, area efficiency, power efficiency, and memory bandwidth of the ViTA 600 at different clock frequencies and hardware configurations were evaluated. The parameters m, k, and n, selected from 8, 16, and 32, are tailored to the matrix dimensions in ViT and ViTA’s computational capacity, optimizing the trade-off between performance and area-efficiency and power-efficiency for specific applications: ViTA Tiny and Huge target low-power edge devices and high-performance cloud applications, respectively, with Base and Large models illustrating the trends of metrics.

[0126] 1) ViTA at Different Clock Frequencies'. ViTA Huge ( = 32, k = n = 16) was synthesized at different clock frequencies, namely, 200 MHz, 300 MHz, 500 MHz, and 1 GHz, and the experimental results are summarized in Table VII shown in FIG. 23. The supported input image resolution is 224x224 (i.e., a 224x224 RGB image as input).

[0127] FIG. 22 shows the performance (GOPS), the performance (GOPS), area efficiency, and power efficiency of the ViTA Huge at different clock frequencies. The performance is on the secondary y-axis. As can be seen from FIG. 22, the higher the operation frequency, the larger the area efficiency. ViTA Huge achieves a relatively high area efficiency of 2,131.85 GOPS / mm2at 1 GHz and achieves a relatively high power efficiency of 1,929.49 GOPS / W at 200 MHz.

[0128] 2) ViTA with Different Hardware Configurations'. ViTA was synthesized with different hardware configurations (different computation capabilities), namely, m = k = n = 8, m = 16, k = n = 8, m = k = n = 16, and m = 32, k = n = 16, with the same clock frequency, i.e., 300 MHz, and the experimental results are summarized in Table VIII shown in FIG. 24. Thesupported input image resolution is 224><224. The area and power of a Tiny ViTA (m = k = n = 8) are 2.00 mm2and 325.45 mW, respectively, which is 3.40 times smaller than the area of a Huge ViTA ( / « = 32, k = n = 16) and 8.26 times smaller than the power of a Huge ViTA (m = 32, k= n = 6).

[0129] FIG. 25 shows the performance (GOPS), area efficiency, and power efficiency of the ViTA with different hardware configurations at 300 MHz. The performance is on the secondary y-axis. As can be seen from FIG. 22, as computational capability increases, both area and power efficiencies improve. Specifically, the area efficiency rises from 153.45 to 724.05 GOPS / mm2and the power efficiency from 943.94 to 1828.13 GOPS / W, marking increases of 4.72 times and 1.94 times, respectively.Area and Power Breakdown

[0130] The area and power breakdown of the ViTA Kernel in ViTA Huge (m = 32, k = n = 16) and that in ViTA Tiny (m = k = n = 8) at 300 MHz are shown in FIGs. 26A to 26D. In particular, FIGs. 26 A and 26B show the area (FIG. 26 A) and power (FIG. 26B) of the ViTA Kernel in the ViTA Huge (m = 32, k = n = 16) at 300 MHz, with a kernel area of 0.211 mm2and power of 83.84 mW. FIGs. 26C and 26D show the area (FIG. 26C) and power (FIG. 26D) in the ViTA Tiny ( = k = n = 8) at 300 MHz, with a kernel area of 0.248 mm2and power of 40.21 mW at 300 MHz. In all sub-figures, the sections depicted with a lighter shade indicate modules associated with memory.

[0131] As shown in FIGs. 26A and 26B, memories occupy 44% area and 38% power in the ViTA kernel in ViTA Huge, due to the efficient ViTA dataflow, achieving a high area efficiency of 724.05 GOPS / mm2and a high power efficiency of 1,828. 13 GOPS / W at 300 MHz. With the number of ViTA kernels decreasing, the capability of the matrix buffers increases, which leads to an increased percentage in the area and power of memories. As shown in FIGs. 26C and 26D, memories occupy 84% area and 57% power in the ViTA kernel in ViTA Tiny. Even with the optimized special function module, it occupies 22% area in ViTA Huge, which is 66% of the VMU lane.Memory Bandwidth

[0132] Table VIII shown in FIG 24 also summarizes the memory bandwidth requirements for the ViTA with various hardware configurations operating at 300 MHz for supporting a 224x224 RGB image as input and FIG. 27 shows a plot thereof As computational capabilityincreases, the required bandwidth correspondingly expands from 0.9 GB / s to 11. 1 GB / s, making a rise of 12.3 times. However, even for the most demanding ViTA Huge configuration, the bandwidth remains well within the capabilities of a 64-lane DDR 3 interface, which offers a transfer rate of 19.9 GB / s.Comparison with Related Works|00133| Detailed comparisons between the related works and the introduced ViTA architectures according to various example embodiments of the present invention, namely, ViTA Tiny at 200 MHz and ViTA Huge at 1 GHz, for area- and power-constraint applications and high-performance applications, respectively, are summarised in Table IX shown in FIG. 28. The performance, power efficiency, and area efficiency of related works compared with the ViTA Huge are illustrated in FIG. 29 as well, where the size of the bubble corresponds to the performance (GOPS). The bubble at the right top corner with a large bubble size owns the best metrics. Only the introduced ViTA architectures according to various example embodiments of the present invention and SwiftTron support the entire workload while others only focus on MHA or do not support the Softmax operation. However, SwiftTron is directly implemented from the original software implementation and dataflow, resulting in a large area of 273 mm2and a power of 33.64 W. In contrast, the ViTA Tiny not only supports the entire ViT workload but also achieves a power efficiency of 943.42 GOPS / W with only an area of 2 mm2and a power of 0.217 W. Its power is only 155 times and 4.05 times less than that of SwiftTron and a conventional ViTA (disclosed in Nag el al., “Vita: A vision transformer inference accelerator for edge applications,” arXiv preprint arXiv:2302.09108, 2023), respectively.

[0134] Works introduced by the above-mentioned Ham 2020 reference and the above- mentioned Wang 2022 reference achieve good power efficiencies of 2,00.41 GOPS / W and 1,913.49 GOPS / W, respectively, while relatively low area efficiencies of 106.25 G0PS / mm2 and 76.54 GOPS / mm2, respectively. Worse still, retraining is required in works (the above- mentioned Ham 2020 reference, the above-mentioned Ham 2021 reference and the above- mentioned Wang 2022 reference) which is not flexible and time-consuming for large models. The introduced ViTA architectures according to various example embodiments of the present invention, instead, offers more balanced area efficiency and power efficiency for both ViTA Tiny and ViTA Huge and retraining is not necessary. ViTA Huge achieves the best area efficiency of 2.13 TOPS / mm2at 1 GHz, which is 20.06 times and 27.85 times higher than the above-mentioned Ham 2020 reference and the above-mentioned Wang 2022 reference,respectively. It is also 2.46 times higher when benchmarked against the best reported by ELSA, which is 865.08 GOPS / mm2at 1 GHz. Moreover, unlike ELSA, the introduced ViTA architectures according to various example embodiments of the present invention support the end-to-end workload, while also delivering a power efficiency of 1,574.91 GOPS / W that surpasses ELSA’s 1 ,124.45 GOPS / W. Overall, the introduced ViTA architecture according to various example embodiments of the present invention achieves a competitive area and power efficiency for both resource-constraint and highperformance applications while supporting the entire ViT workload.

[0135] Accordingly, various example embodiments advantageously introduced ViTA 600 - a highly efficient and scalable hardware accelerator for the entire workload of ViT with optimized dataflows and a configurable special function module. Various example embodiments present a unique memory-centric fused Multi-Head Attention dataflow that reduces the memory requirements of MHA with 224 -- 224 and VGA images as input by, for example, 40.5% and 76.71%, from 1.45 MB to 0.87 MB and from 22.66 MB to 5.28 MB, respectively. By supporting both MKN-RS and NKM-CS GEMM dataflows, the introduced ViTA reduces the weight buffer size from 2.25 MB to 48 KB while maintaining the weight reuse, and decreases the bandwidth requirements from 256 GB / s to 56 GB / s at 1 GHz. With a ping-pong buffering scheme on the two partial sum buffers 664-1, 664-2 to convert the MKN- RS or NKM-CS dataflows for computing the GEMM operation to MNK-OS or NMK-OS dataflows for computing the special functions (non-linear functions), ViTA 600 reduces the throughput requirement of the special functions from n per cycle to n / K / k per cycle.

[0136] Various example embedments also introduce a configurable special function module (non-linear function block) 648, that supports the approximated non-linear functions, e g., Softmax, LayerNorm, and GELU, in the ViT model with reused hardware resources to improve the power efficiency and area efficiency. Empowered by both optimized dataflows and dedicated hardware modules, ViTA Huge achieves a high area efficiency of 2.13 TOPS / mm2and a high power efficiency of 1.57 TOPS / W at 1 GHz while satisfying the end-to-end computation ability of the ViT model. Compared with A3disclosed in the above-mentioned Ham 2020 reference and the above-mentioned Wang 2022 reference, ViTA is 20.06 times and 27.85 times better in terms of area efficiency. ViTA outperforms ELSA by a factor of 2.46 times and 1 .40 times in terms of area efficiency and power efficiency, respectively.

[0137] Various example embodiments also conduct a comprehensive design space exploration of ViTA, and the results show that ViT As with different hardware configurationsachieve a wide range of computation capabilities, from 204.8 GOPS to 16.384 TOPS, with the area from 2.00 mm2to 7.69 mm2and power consumption from 0.217 W to 10.403 W, respectively, suitable for a wide range of applications.

[0138] While embodiments of the invention have been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced.

Claims

CLAIMS1. A vision transformer accelerator comprising: a plurality of kernel blocks, each kernel block configured to compute operations of a vision transformer model, including matrix multiplication operations and different types of nonlinear functions; a weight buffer configured to store weight data associated with the vision transformer model; a kernel input bus configured to transfer input data of a current encoder layer of the vision transformer model to each of the plurality of kernel blocks; a kernel output bus configured to transfer first output data of each of the plurality of kernel blocks to a next encoder layer of the vision transformer model; and a weight bus configured to transfer weight data from the weight buffer to each of the plurality of kernel blocks, wherein each of the plurality of kernel blocks comprises: a plurality of matrix buffers, each matrix buffer configured to store one or more matrices; a matrix-multiplication unit communicatively coupled to the plurality of matrix buffers and the weight buffer, the matrix-multiplication unit being configured to compute a matrix multiplication operation based on at least one matrix stored in one or more of the plurality of matrix buffers; and a non-linear function block configured to compute each of the different types of non-linear functions.

2. The vision transformer accelerator according to claim 1, wherein the matrix-multiplication unit is configured as a vector-multiplication unit lane comprising a series of vector-multiplication units, the vector-multiplication unit lane being configured to compute the matrix multiplication operation based on the at least one matrix stored in the one or more of the plurality of matrix buffers, and each vector-multiplication unit being configured to perform a respective vector multiplication operation associated with the matrix multiplication operation.

3. The vision transformer accelerator according to claim 1 or 2, wherein for a matrix multiplication operation above a predefined size, the vision transformer accelerator is configured to split an input matrix for the matrix multiplication operation into multiple input matrix chunks and map the multiple input matrix chunks to multiple kernel blocks of the plurality of kernel blocks, respectively, for the multiple kernel blocks to compute matrix multiplication operations based on the multiple input matrix chunks, respectively, in parallel.

4. The vision transformer accelerator according to any one of claims 1 to 3, wherein the weight buffer is a ping-pong buffer comprising a first weight buffer and a second weight buffer, and the vision transformer accelerator further comprises: a multiplexer; and a key and value bus configured to transfer second output data of each of the plurality of kernel blocks to the multiplexer, wherein the multiplexer is arranged between the weight buffer and each of the plurality of kernel blocks and is configured to multiplex the weight data from the weight buffer with the second output data received via the key and value bus to produce an output for transfer to each of the plurality of kernel blocks via the weight bus.

5. The vision transformer accelerator according to claim 4, wherein for each of the plurality of kernel blocks: the kernel block further comprises a first multiplexer configured to multiplex an output of one of the plurality of matrix buffers and a first output of the non-linear function block to produce an output, and the matrix-multiplication unit is configured to receive the output of the first multiplexer as a first input thereto and the output of the multiplexer of the vision transformer accelerator via the weight bus as a second input thereto.

6. The vision transformer accelerator according to claim 5, wherein for each of the plurality of kernel blocks: the plurality of matrix buffers comprises a first matrix buffer, a second matrix buffer and a third matrix buffer, andthe kernel block further comprises a partial sum buffer configured as a ping-pong buffer comprising a first partial sum buffer and a second partial sum buffer.

7. The vision transformer accelerator according to claim 6, wherein the vision transformer accelerator further comprises a bias bus configured to transfer bias data to each of the plurality of kernel blocks, and for each of the plurality of kernel blocks, the kernel block further comprises: an accumulator; a second multiplexer; a dequantization unit; and a third multiplexer, the second multiplexer is configured to multiplex an output of the matrixmultiplication unit, the bias data received via the bias bus and an output of the first matrix buffer to produce an output, the dequantization unit is configured to receive and dequantize the output of the second multiplexer to produce an output, the accumulator is configured to receive the output of the dequantization unit as a first input thereto and a first output of the partial sum buffer as a second input thereto, and the third multiplexer is configured to multiplex an output of the accumulator and the output of the dequantization unit to produce an output.

8. The vision transformer accelerator according to claim 7, wherein for each of the plurality of kernel blocks: the non-linear function block is configured to receive a second output of the partial sum buffer as a first input thereto and the output of the first matrix buffer as a second input thereto, and the kernel block further comprises a fourth multiplexer configured to multiplex the first output of the non-linear function block, a second output of the non-linear function block and the output of the third multiplexer to produce an output.

9. The vision transformer accelerator according to claim 8, wherein for each of the plurality of kernel blocks:the kernel block further comprises a fifth multiplexer configured to multiplex the output of the fourth multiplexer and the input data of the current encoder layer received via the kernel input bus to produce an output, the second matrix buffer is configured to receive the output of the fifth multiplexer as an input thereto, and the first and third matrix buffers are each configured to receive the output of the fourth multiplexer as an input thereto.

10. The vision transformer accelerator according to claim 8 or 9, wherein for each of the plurality of kernel blocks: the kernel block further comprises a sixth multiplexer, a seventh multiplexer and a demultiplexer, the demultiplexer is configured to receive the output of the third multiplexer to produce a first output for input to the first partial sum buffer and a second output for input to the second partial sum buffer, the sixth multiplexer is configured to multiplex an output of the first partial sum buffer and an output of the second partial sum buffer to produce the first output of the partial sum buffer, and the seventh multiplexer is configured to multiplex the output of the first partial sum buffer and the output of the second partial sum buffer to produce the second output of the partial sum buffer.

11. The vision transformer accelerator according to any one of claims 6 to 10, wherein for each of the plurality of kernel blocks, the non-linear function block comprises a plurality of piecewise linear function (PLF) units and a plurality of adder trees configured to compute the different types of non-linear functions.

12. The vision transformer accelerator according to claim 11, wherein the different types of non-linear functions comprise a Gaussian error linear unit (GELU) function, a softmax function and a layer normalization function.

13. The vision transformer accelerator according to claim 12, wherein, for computing the softmax function:a set of PLF units of the plurality of PLF units is configured to compute exponential values of input elements, respectively; a first adder tree of the plurality of adder trees and a first adder are configured to compute a sum of the exponential values of the input elements; a further PLF unit of the plurality of PLF units is configured to compute a reciprocal of the sum of the exponential values of the input elements, and a first multiplier of the non-linear function block is configured to, for each of the input elements, multiply the exponential value of the input element with the reciprocal of the sum of the exponential values of the input elements to obtain the softmax value of the input element.

14. The vision transformer accelerator according to claim 13, wherein, for computing the layer normalization function: the first adder tree and the first adder are further configured to compute a sum of input elements; a second adder tree of the plurality of adder trees and a second adder are configured to compute a sum of squares of the input elements; a second multiplier of the non-linear function block is configured to compute a mean of input elements based on the sum of the input elements; a third multiplier, a fourth multiplier and a first subtractor of the non-linear function block are configured to compute a square of a variance of the input elements based on the mean of the input elements and the sum of the squares of the input elements computed; a second subtractor of the non-linear function block is configured to, for each of the input elements, perform a subtraction between the mean and the input element to obtain a difference value associated with the input element; the further PLF unit of the plurality of PLF units is configured to compute a reciprocal of the square root of the square of the variance of the input elements to obtain a reciprocal of the variance of the input elements; and the first multiplier of the non-linear function block is further configured to, for each of the input elements, multiply the difference value associated with the input element and the reciprocal of the variance of the input elements to obtain the layer normalization value of the input element.

15. The vision transformer accelerator according to claim 14, wherein the vision transformer accelerator is further configured to compute the layer normalization function in two stages, wherein at a first stage, the vision transformer accelerator is configured to compute the mean of input elements and the square of the variance of the input elements, and the input elements are written to one of the plurality of matrix buffers after quantization, and wherein at a second stage, the vision transformer accelerator is configured to read the input elements from said one of the plurality of matrix buffers after dequantization and compute, for each of the input elements, the layer normalization value of the input element based on the mean of input elements and the square of the variance of the input elements computed at the first stage.

16. The vision transformer accelerator according to any one of claims 13 to 15, wherein, for computing the GELU function, the set of PLF units of the plurality of PLF units is configured to compute the GELU function based on input elements.

17. The vision transformer accelerator according to any one of claims 6 to 16, wherein the vision transformer accelerator is configured to compute matrix multiplication operations according to a first dataflow or a second dataflow, each matrix multiplication operation involves a first input matrix thereto having a row dimension and a column dimension, a second input matrix thereto having a row dimension and a column dimension and an output matrix therefrom having a row dimension and a column dimension, the first dataflow is a row stationary dataflow wherein executions of for-loops of the matrix multiplication operation, from inner to outer, is according to an order of the column dimension of the second input matrix, the column dimension of the first input matrix and the row dimension of the first input matrix, and the second dataflow is a column stationary dataflow wherein executions of for-loops of the matrix multiplication operation, from inner to outer, is according to an order of the row dimension of the first input matrix, the column dimension of the first input matrix and the column dimension of the second input matrix.

18. The vision transformer accelerator according to claim 17, whereinthe vision transformer accelerator is further configured to compute non-linear functions according to a third dataflow or a fourth dataflow, each non-linear function involving an input matrix thereto having a row dimension and a column dimension, the third dataflow is an output stationary dataflow wherein executions of for-loops of the non-linear function, from inner to outer, is according to an order of the column dimension of the input matrix and the row dimension of the input matrix, and the fourth dataflow is an output stationary dataflow wherein executions of for-loops of the non-linear function, from inner to outer, is according to an order of the row dimension of the input matrix and the column dimension of the input matrix.

19. The vision transformer accelerator according to claim 18, wherein for each of the plurality of kernel blocks, the partial sum buffer comprising the first partial sum buffer and the second partial sum buffer is configured to convert a dataflow for computing a matrix multiplication operation to a dataflow for computing a non-linear function for the non-linear function block.

20. The vision transformer accelerator according to claim 19, wherein the partial sum buffer is configured to convert the first dataflow for computing the matrix multiplication operation to the third dataflow for computing the non-linear function, or convert the second dataflow for computing the matrix multiplication operation to the fourth dataflow for computing the nonlinear function.

21. The vision transformer accelerator according to any one of claims 6 to 20, wherein the input data transferred to each of the plurality of kernel blocks via the kernel input bus and the weight data transferred to each of the plurality of kernel blocks via the weight bus are synchronized to the plurality of kernel blocks through asynchronous first-in-first-out (FIFO) buffers, respectively.

22. The vision transformer accelerator according to any one of claims 6 to 21, wherein the operations of the vision transformer model, include matrix multiplication operations and different types of non-linear functions relating to multi-head self attention (MHA) layers and fully connected feed-forward network (FFN) layers of the vision transformer model.

23. A method of operating the vision transformer accelerator according to claim 1, comprising, for each of the plurality of kernel blocks of the vision transformer accelerator: computing, using the matrix-multiplication unit of the kernel block, a matrix multiplication operation based on at least one matrix stored in one or more of the plurality of matrix buffers of the kernel block; and computing, using the non-linear function block of the kernel block, a non-linear function of the different types of non-linear functions.

24. The method according to claim 23, wherein the matrix-multiplication unit is configured as a vector-multiplication unit lane comprising a series of vector-multiplication units, and said computing, using the matrix-multiplication unit of the kernel block, the matrix multiplication operation comprises computing, using the vector-multiplication unit lane, the matrix multiplication operation based on the at least one matrix stored in the one or more of the plurality of matrix buffers, each vector-multiplication unit being configured to perform a respective vector multiplication operation associated with the matrix multiplication operation.

25. The method according to claim 23 or 24, wherein for a matrix multiplication operation above a predefined size, the method further comprises: splitting an input matrix for the matrix multiplication operation into multiple input matrix chunks; and mapping the multiple input matrix chunks to multiple kernel blocks of the plurality of kernel blocks, respectively, for the multiple kernel blocks to compute matrix multiplication operations based on the multiple input matrix chunks, respectively, in parallel.

26. The method according to any one of claims 23 to 25, wherein the weight buffer is a ping-pong buffer comprising a first weight buffer and a second weight buffer, the vision transformer accelerator further comprises: a multiplexer; and a key and value bus configured to transfer second output data of each of the plurality of kernel blocks to the multiplexer,wherein the multiplexer is arranged between the weight buffer and each of the plurality of kernel blocks and is configured to multiplex the weight data from the weight buffer with the second output data received via the key and value bus to produce an output, and the method further comprises multiplexing, using the multiplexer, the weight data from the weight buffer with the second output data received via the key and value bus to produce an output for transfer to each of the plurality of kernel blocks via the weight bus.

27. The method according to claim 26, wherein for each of the plurality of kernel blocks: the kernel block further comprises a first multiplexer, and the method further comprises: multiplexing, using the first multiplexer, an output of one of the plurality of matrix buffers and a first output of the non-linear function block to produce an output; and receiving, by the matrix-multiplication unit, the output of the first multiplexer as a first input thereto and the output of the multiplexer of the vision transformer accelerator via the weight bus as a second input thereto.

28. The method according to claim 27, wherein for each of the plurality of kernel blocks: the plurality of matrix buffers comprises a first matrix buffer, a second matrix buffer and a third matrix buffer, and the kernel block further comprises a partial sum buffer configured as a ping-pong buffer comprising a first partial sum buffer and a second partial sum buffer.

29. The method according to claim 28, wherein the vision transformer accelerator further comprises a bias bus configured to transfer bias data to each of the plurality of kernel blocks, and for each of the plurality of kernel blocks, the kernel block further comprises: an accumulator; a second multiplexer; a dequantization unit; and a third multiplexer, and the method further comprises:multiplexing, using the second multiplexer, an output of the matrixmultiplication unit, the bias data received via the bias bus and an output of the first matrix buffer to produce an output; receiving and dequantizing, using the dequantization unit, the output of the second multiplexer to produce an output; receiving, by the accumulator, the output of the dequantization unit as a first input thereto and a first output of the partial sum buffer as a second input thereto; and multiplexing, using the third multiplexer, an output of the accumulator and the output of the dequantization unit to produce an output.

30. The method according to claim 29, wherein for each of the plurality of kernel blocks: the kernel block further comprises a fourth multiplexer, and the method further comprises: receiving, by the non-linear function block, a second output of the partial sum buffer as a first input thereto and the output of the first matrix buffer as a second input thereto, and multiplexing, using the fourth multiplexer, the first output of the non-linear function block, a second output of the non-linear function block and the output of the third multiplexer to produce an output.

31. The method according to claim 30, wherein for each of the plurality of kernel blocks: the kernel block further comprises a fifth multiplexer, and the method further comprises: multiplexing, using the fifth multiplexer, the output of the fourth multiplexer and the input data of the current encoder layer received via the kernel input bus to produce an output; receiving, by the second matrix buffer, the output of the fifth multiplexer as an input thereto; and receiving, by each of the first and third matrix buffers, the output of the fourth multiplexer as an input thereto.

32. The method according to claim 30 or 31, wherein for each of the plurality of kernel blocks:the kernel block further comprises a sixth multiplexer, a seventh multiplexer and a demultiplexer, and the method further comprises: receiving, by the demultiplexer, the output of the third multiplexer to produce a first output for input to the first partial sum buffer and a second output for input to the second partial sum buffer; multiplexing, using the sixth multiplexer, an output of the first partial sum buffer and an output of the second partial sum buffer to produce the first output of the partial sum buffer; and multiplexing, using the seventh multiplexer, the output of the first partial sum buffer and the output of the second partial sum buffer to produce the second output of the partial sum buffer.

33. The method according to any one of claims 28 to 32, wherein the non-linear function block comprises a plurality of piecewise linear function (PLF) units and a plurality of adder trees configured to compute the different types of non-linear functions.

34. The method according to claim 33, wherein the different types of non-linear functions comprise a Gaussian error linear unit (GELU) function, a softmax function and a layer normalization function.

35. The method according to claim 34, wherein, for computing the softmax function, said computing, using the non-linear function block of the kernel block, the non-linear function comprises: computing, using a set of PLF units of the plurality of PLF units, exponential values of input elements, respectively; computing, using a first adder tree of the plurality of adder trees and a first adder, a sum of the exponential values of the input elements; computing, using a further PLF unit of the plurality of PLF units, a reciprocal of the sum of the exponential values of the input elements; and for each of the input elements, multiply, using a first multiplier of the non-linear function block, the exponential value of the input element with the reciprocal of the sum of the exponential values of the input elements to obtain the softmax value of the input element.

36. The method according to claim 35, wherein, for computing the layer normalization function, said computing, using the non-linear function block of the kernel block, the non-linear function comprises: computing, using the first adder tree and the first adder, a sum of input elements; computing, using a second adder tree of the plurality of adder trees and a second adder, a sum of squares of the input elements; computing, using a second multiplier of the non-linear function block, a mean of input elements based on the sum of the input elements; computing, using a third multiplier, a fourth multiplier and a first subtractor of the nonlinear function block, a square of a variance of the input elements based on the mean of the input elements and the sum of the squares of the input elements computed; for each of the input elements, performing, using a second subtractor of the non-linear function block, a subtraction between the mean and the input element to obtain a difference value associated with the input element; computing, using the further PLF unit of the plurality of PLF units, a reciprocal of the square root of the square of the variance of the input elements to obtain a reciprocal of the variance of the input elements; and for each of the input elements, multiplying, using the first multiplier of the non-linear function block, the difference value associated with the input element and the reciprocal of the variance of the input elements to obtain the layer normalization value of the input element.

37. The method according to claim 36, wherein the layer normalization function is computed in two stages, wherein at a first stage, the mean of input elements and the square of the variance of the input elements are computed, and the input elements are written to one of the plurality of matrix buffers after quantization, and wherein at a second stage, the input elements are read from said one of the plurality of matrix buffers after dequantization and for each of the input elements, the layer normalization value of the input element is computed based on the mean of input elements and the square of the variance of the input elements computed at the first stage.

38. The method according to any one of claims 35 to 37, wherein, for computing the GELU function, said computing, using the non-linear function block of the kernel block, the non-linear function comprises: computing, using the set of PLF units of the plurality of PLF units, the GELU function based on input elements.

39. The method according to any one of claims 28 to 38, wherein matrix multiplication operations are computed according to a first dataflow or a second dataflow, each matrix multiplication operation involves a first input matrix thereto having a row dimension and a column dimension, a second input matrix thereto having a row dimension and a column dimension and an output matrix therefrom having a row dimension and a column dimension, the first dataflow is a row stationary dataflow wherein executions of for-loops of the matrix multiplication operation, from inner to outer, is according to an order of the column dimension of the second input matrix, the column dimension of the first input matrix and the row dimension of the first input matrix, and the second dataflow is a column stationary dataflow wherein executions of for-loops of the matrix multiplication operation, from inner to outer, is according to an order of the row dimension of the first input matrix, the column dimension of the first input matrix and the column dimension of the second input matrix.

40. The method according to claim 39, wherein non-linear functions are computed according to a third dataflow or a fourth dataflow, each non-linear function involving an input matrix thereto having a row dimension and a column dimension, the third dataflow is an output stationary dataflow wherein executions of for-loops of the non-linear function, from inner to outer, is according to an order of the column dimension of the input matrix, and the row dimension of the input matrix, and the fourth dataflow is an output stationary dataflow wherein executions of for-loops of the non-linear function, from inner to outer, is according to an order of the row dimension of the input matrix and the column dimension of the input matrix.

41. The method according to claim 40, wherein for each of the plurality of kernel blocks,the method further comprises converting, using the partial sum buffer comprising the first partial sum buffer and the second partial sum buffer, a dataflow for computing a matrix multiplication operation to a dataflow for computing a non-linear function for the non-linear function block.

42. The method according to claim 41, wherein the first dataflow for computing the matrix multiplication operation is converted to the third dataflow for computing the non-linear function, or the second dataflow for computing the matrix multiplication operation is converted to the fourth dataflow for computing the non-linear function.

43. The method according to any one of claims 28 to 42, wherein the input data transferred to each of the plurality of kernel blocks via the kernel input bus and the weight data transferred to each of the plurality of kernel blocks via the weight bus are synchronized to the plurality of kernel blocks through asynchronous first-in-first-out (FIFO) buffers, respectively.

44. The method according to any one of claims 28 to 43, for performing operations of the vision transformer model, including matrix multiplication operations and different types of nonlinear functions relating to multi-head self attention (MHA) layers and fully connected feedforward network (FFN) layers of the vision transformer model.

Images