Electronic device and method for accelerating neural network computation

Abstract

In general, the present disclosure generally relates to an electronic device for accelerating machine learning ML model calculations. The electronic device includes a first processor configured to generate a query matrix, a key matrix, and a value matrix by performing a fast Fourier transform (FFT) and a butterfly linear transform on at least one input matrix, and a second processor configured to perform a first matrix multiplication between the query matrix and the key matrix, a softmax operation is performed on a result of the first matrix multiplication, and a second matrix multiplication is performed between a result of the softmax operation and the value matrix.

Description

Technical Field

[0001] The present application generally relates to a device for accelerating neural network calculations. In particular, the present application relates to a hardware accelerator, an electronic device or a processing unit for accelerating machine learning ML model calculations. Background Art

[0002] Advances in deep learning have led to various neural networks (NNs) that are able to achieve state-of-the-art accuracy on many AI tasks. Among them, recurrent neural networks (RNNs) such as gated recurrent units (GRUs) and long short-term memory (LSTM) networks have demonstrated their advantages in solving sequencing transformation and modeling problems.

[0003] However, the recursive nature of these models introduces two disadvantages: (1) the limited ability to exploit past or external knowledge leads to loss of long-range information; and (2) the sequential nature of RNNs hinders their parallelization in hardware and slows down their processing speed.

[0004] Recently, the attention mechanism has been proposed to address these shortcomings. It can process long data sequences in parallel to capture long-range information. Based on this principle, different attention-based NNs (such as BERT and GPT) have been proposed to achieve state-of-the-art accuracy in various natural language processing (NLP) tasks.

[0005] However, the benefits of attention-based NNs are achieved at this cost: the high degree of parallelism significantly increases the amount of parameters and computations, resulting in a large overhead in their speed and power consumption. To alleviate this situation, various hardware accelerators have been proposed. However, there are several problems in these designs. For example, these accelerators focus only on optimizing FFNs or attention mechanisms. Without jointly optimizing the two parts, these hardware designs lack scalability when accelerating end-to-end attention-based NNs with different input lengths. In addition, while optimizing the attention mechanism, most existing designs dynamically detect and prune redundant calculations at run time to achieve high sparsity on specific data sets and networks. However, the generalization of these dynamic methods needs to be further tested because their performance gains may vary between different data sets and network architectures. In addition, since the sparsity patterns introduced by these dynamic methods are unstructured, dynamic hardware controllers are required in order to exploit sparsity. However, such complex controllers usually contain a large number of clock elements, the cost of which increases as the size of transistors decreases. Therefore, the performance or energy improvements brought by these dynamic methods may be weakened due to the hardware overhead of these dynamic controllers.

[0006] Accordingly, the present applicants have recognized a need for improved hardware acceleration mechanisms for machine learning computations. Summary of the invention

[0007] Technical Solution

[0008] The demand for data-intensive applications such as machine learning and the need for high-speed computing has led to a need for "accelerators" that offload work from general-purpose processors. An accelerator is a hardware device that works with a general-purpose processor (such as a CPU) to increase the speed at which data is processed and thus improve the performance of the processor. This is particularly useful in situations where a larger, more powerful processor cannot be used within the device (such as within a mobile communication device, a smart phone, a home assistant device, a robotic device, a battery-powered device, etc.).

[0009] In an embodiment of the present disclosure, an electronic device for accelerating machine learning ML model calculation is provided. The electronic device includes: a first processor configured to generate a query matrix, a key matrix, and a value matrix by performing a fast Fourier transform (FFT) and a butterfly linear transform on at least one input matrix, and a second processor configured to perform a first matrix multiplication between the query matrix and the key matrix, perform a softmax operation on the result of the first matrix multiplication, and perform a second matrix multiplication between the result of the softmax operation and the value matrix.

[0010] In an embodiment of the present disclosure, a method for accelerating the calculation of a machine learning ML model is provided. The method includes: generating a query matrix, a key matrix, and a value matrix by performing a fast Fourier transform (FFT) and a butterfly linear transform on at least one input matrix, performing a first matrix multiplication between the query matrix and the key matrix, performing a softmax operation on the result of the first matrix multiplication, and performing a second matrix multiplication between the result of the softmax operation and the value matrix. In an embodiment of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium includes instructions that, when executed by a processor, cause the processor to perform the method of the present disclosure.

[0011] In an embodiment of the present disclosure, a processing unit for accelerating the calculation of a machine learning ML model is provided, also referred to herein as an "electronic device", and the processing unit includes: a first processor for accelerating calculations involving sparse matrices having a butterfly sparsity pattern. The first processor can be used to accelerate the calculation of any layer of the ML model as long as the calculation involves a matrix with a butterfly sparsity pattern. For example, when the butterfly sparsity pattern is adopted / applied to the attention layer and / or linear layer of the ML model, the first processor can be used to accelerate the calculation of these layers. The ML model may include one or more additional layers that do not adopt butterfly sparsity, but the first processor is not used to accelerate the calculation of these additional layers.

[0012] Because the first processor operates on data using a butterfly sparsity pattern, the first processor is also referred to herein as a “butterfly processor.” That is, the first processor accesses data from the input matrix according to a butterfly sparsity pattern to reduce the number of calculations that need to be performed without losing precision.

[0013] The first processor may include a plurality of engines, also referred to herein as "butterfly engines," for accelerating computations involving sparse matrices.

[0014] Each butterfly engine may include an adaptive memory system and a plurality of adaptive butterfly units, wherein each butterfly unit may be configured to perform a specific calculation.

[0015] Each butterfly unit can be configured to perform one or both of a fast Fourier transform and a linear transform. This is advantageous because many conventional techniques can perform either an attention-based operation (i.e., a linear transform) or a fast Fourier transform-based operation, but not both. The butterfly units of the present technique can be configured at runtime to perform either operation based on which layer of the ML model the processing unit is being used for.

[0016] Each butterfly unit may include a plurality of multiplexers arranged to select the inputs required to perform a particular calculation. This enables the butterfly unit to be adapted / configured to perform a particular calculation.

[0017] In an embodiment of the present disclosure, each butterfly unit may include: four real multipliers arranged to multiply inputs to perform a specific calculation; two real adders or subtractors arranged to operate on the outputs of the real multipliers; and two complex adders or subtractors arranged to operate on the outputs of the real adders or subtractors. For example, to perform a linear transform calculation, all four real multipliers can be used to perform four real multiplications, and to perform an FFT calculation, all four real multipliers can be used to perform four complex multiplications. Therefore, the same hardware can be configured in real time to adapt to the calculation.

[0018] Each butterfly unit may include: eight multiplexers arranged to receive inputs for specific calculations and select inputs required to perform the specific calculations; and two demultiplexers arranged between the two real adders or subtractors and the two complex adders or subtractors to provide outputs of the butterfly unit.

[0019] The memory system of the butterfly engine may include a serial-to-parallel module arranged to access data for a particular calculation based on the butterfly sparsity pattern.

[0020] The processing unit may further include: a third processor for receiving the output from the first processor and for performing post-processing using the received output. The third processor is also referred to herein as a "post-processor".

[0021] As described above, the first processor can be used to accelerate the calculation of any layer of the ML model as long as the calculation involves a matrix with a butterfly sparsity pattern. For example, when the butterfly sparsity pattern is adopted / applied to the attention layer and / or linear layer of the ML model, the first processor can be used to accelerate the calculation of these layers. However, when the ML model includes at least one attention layer that requires calculations that do not involve or use sparse matrices with a butterfly sparsity pattern, the processing unit may also include: a second processor for performing some operations required by the attention mechanism, that is, for performing operations required for this / these attention layers. The second processor is also referred to as an "attention processor" in this article. That is, when the ML model even includes a single attention layer (which is vanilla or conventional (that is, does not use butterfly sparsity)), the second processor is used to execute the attention layer when the model is executed.

[0022] The second processor may include multiple attention engines, wherein each attention engine may include: a query key QK unit for implementing matrix multiplication between a query matrix and a key matrix; and a score value SV unit for receiving an output from the QK unit and multiplying the output with a value vector to output a result of the attention layer of the ML model.

[0023] As mentioned above, multiple butterfly engines and butterfly processors can be configured in real time to perform specific calculations. The butterfly processor can be reconfigurable and adaptive. Therefore, the butterfly processor can be configured at runtime to use a single hardware to accelerate different layers. This makes this hardware accelerator efficient because it can be used to accelerate different multiple layers of the ML model.

[0024] During runtime, the processing unit may receive input for processing by the ML model and may use the first processor to perform a fast Fourier transform calculation using the received input.

[0025] In an embodiment of the present disclosure, a computer-implemented method for accelerating machine learning ML model calculations using the processing unit described herein is provided, the method comprising: receiving an input matrix for ML model processing; and using a first processor of the processing unit to perform the following operations: accessing data from the input matrix based on a butterfly sparsity pattern; configuring the plurality of butterfly units of the first processor to perform fast Fourier transform calculations; and performing fast Fourier transform calculations using the accessed data.

[0026] The method may further include: using a second processor of the processing unit to perform the following operations:

[0027] An attention mechanism computation not involving a sparse matrix having a butterfly sparsity pattern is performed using the input matrix and the output of the first processor.

[0028] In a related method of the present technology, a computer-readable storage medium is provided that includes instructions that, when executed by a processor, cause the processor to perform the method described herein.

[0029] As will be appreciated by those skilled in the art, the present technology can be implemented as a system, method or computer program product.Therefore, the present technology can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects.

[0030] In addition, the present disclosure may take the form of a computer program product embodied in a computer-readable medium having a computer-readable program code embodied thereon. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or apparatus, or any suitable combination of the foregoing.

[0031] Computer program code for performing operations of the present technology may be written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. Code components may be implemented as procedures, methods, etc., and may include subcomponents that may take the form of instructions or sequences of instructions at any level of abstraction, from direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs.

[0032] Embodiments of the present disclosure also provide a non-transitory data carrier carrying code, which, when implemented on a processor, causes the processor to perform any of the methods described herein.

[0033] The techniques also provide processor control code to implement the above methods, for example, on a general purpose computer system or a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code, the processor control code being used to implement any of the above methods at runtime, particularly on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, a CD-ROM or DVD-ROM, a programming memory such as a non-volatile memory (e.g., flash memory) or a read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. The code (and / or data) for implementing the embodiments of the techniques described herein may include source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C or assembly code, code for setting or controlling an ASIC (application specific integrated circuit) or FPGA (field programmable gate array), or code for a hardware description language such as Verilog (RTM) or VHDL (very high speed integrated circuit hardware description language). As will be appreciated by the skilled person, such code and / or data may be distributed between multiple connected components that communicate with each other. These techniques may include a controller including a microprocessor, working memory, and program memory coupled to one or more components of the system.

[0034] It will also be clear to those skilled in the art that all or part of the logic methods according to embodiments of the present technology may be suitably implemented in a logic device comprising logic elements to perform the steps of the above methods, and such logic elements may include components such as logic gates in, for example, a programmable logic array or an application specific integrated circuit. Such a logic arrangement may also be implemented in an enabling element for temporarily or permanently establishing a logic structure in such an array or circuit using, for example, a virtual hardware descriptor language, wherein the virtual hardware descriptor language may be stored and transmitted using a fixed or transmittable carrier medium.

[0035] In an embodiment, the present disclosure may be implemented in the form of a data carrier having functional data thereon, wherein the functional data comprises a functional computer data structure, so that when loaded into a computer system or network and operated by it, the computer system is able to perform all the steps of the above method.

[0036] The above method can be performed in whole or in part on a device (i.e., an electronic device) using machine learning or an artificial intelligence model. The model can be processed by a dedicated artificial intelligence processor designed in a hardware structure designated for artificial intelligence model processing. The artificial intelligence model can be obtained by training. Here, "obtained by training" means obtaining a predefined operating rule or artificial intelligence model configured to perform a desired function (or purpose) by training a basic artificial intelligence model with multiple training data through a training algorithm. The artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers includes multiple weight values, and the neural network calculation is performed by calculation between the calculation result of the previous layer and the multiple weight values.

[0037] As described above, the present disclosure can be implemented using an AI model. Functions associated with AI can be performed by non-volatile memory, volatile memory, and a processor. The processor may include one or more processors. At this time, one or more processors may be a general-purpose processor (such as a central processing unit (CPU), an application processor (AP), etc.), a graphics processing unit (such as a graphics processing unit (GPU), a visual processing unit (VPU)), and / or an AI-specific processor (such as a neural processing unit (NPU)). One or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. Predefined operating rules or artificial intelligence models are provided by training or learning. Here, providing by learning means: generating an AI model of predefined operating rules or desired characteristics by applying a learning algorithm to multiple learning data. Learning can be performed in the device itself in which the AI ​​according to the embodiment is performed, and / or can be implemented by a separate server / system.

[0038] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and the layer operation is performed by the calculation of the previous layer and the operation of multiple weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recursive deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0039] A learning algorithm is a method for training a predetermined target device (e.g., a robot) using multiple learning data to enable, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0041] Figure 1 is a diagram showing the structure of a Transformer network;

[0042] Figure 2 shows the algorithm used by the multi-head attention module to perform linear transformation and attention;

[0043] Figure 3 is the key to the symbols used in this article;

[0044] Figure 4 Shows latency breakdown for Transformer with different input lengths on GPU and CPU.

[0045] Figure 5 is a diagram showing basic sparsity patterns in existing efficient attention-based NNs;

[0046] Figure 6 It is a schematic diagram of the network structure of FABNet;

[0047] Figure 7 is a table showing combinations of sparsity patterns in different variations;

[0048] Figure 8 is a hardware overview of the adaptive butterfly accelerator described in this paper;

[0049] FIG9 a shows the architecture of a butterfly unit;

[0050] FIG9b shows the data flow of the butterfly linear transformation;

[0051] Figure 9c shows the data flow of the Fast Fourier Transform;

[0052] Fig.10 A butterfly memory system with an adaptive address mapping strategy is shown;

[0053] Fig.11 An adaptive fine-grained pipelining strategy is shown;

[0054] Fig.12 shows bank conflicts in column-major and row-major order;

[0055] Fig.13 The data layout scheme and hardware design of the S2P module are shown;

[0056] Fig.14 An example of a 16-input butterfly is shown;

[0057] Fig.15The hardware architecture of the index merge module is shown;

[0058] Fig.16 is a table showing the accuracy performance of different models on LRA;

[0059] Fig.17 Experimental data showing accelerated collapse compared to a baseline design are shown;

[0060] Fig.18 The compression ratio of the applicant's optimized FABNet over the vanilla Transformer and FNet in terms of floating point operations (FLOPs) and model size (number of parameters) is shown.

[0061] Fig.19 shows experimental data comparing performance for (a) high-end and (b) edge GPUs and CPUs; and

[0062] Fig. 20 shows a comparison of the applicant's technology with existing Transformer accelerators in terms of latency, power, and energy efficiency;

[0063] Fig.21 shows experimental data on the latency achieved for different input sequence lengths when varying the available off-chip memory bandwidth;

[0064] Fig. 22 A flow chart describing a method for hardware acceleration is shown;

[0065] Fig.23 A flow chart describing a method for hardware acceleration is shown; and

[0066] Fig.24 is a block diagram of a device 200 incorporating an accelerator of the present disclosure. DETAILED DESCRIPTION

[0067] In general, the present disclosure generally relates to a device for accelerating neural network calculations. In particular, the present application relates to a hardware accelerator or processing unit for accelerating machine learning ML model calculations.

[0068] To address the above problems, the applicant uses butterfly sparsity to accelerate attention-based models, with at least three novel aspects: i) fine-grained structured regularity with regular data access to optimize both memory and computational efficiency; ii) static sparsity patterns, avoiding the need to design dynamic controllers in hardware; iii) sparsity utilization on attention layers and linear layers, which allows scalable end-to-end acceleration of attention-based NNs. Therefore, the applicant proposes FABNet, a hardware-friendly model for FFT, attention, and butterfly networks. In order to fully exploit the sparsity in hardware, the applicant proposes an adaptive butterfly accelerator that can be configured at runtime via dedicated hardware control to use a unified engine to accelerate different layers, thereby significantly improving hardware efficiency. In order to push performance limits, the applicant jointly optimizes models and hardware via a collaborative design approach.

[0069] Based on their network structures, attention-based NNs can be divided into three categories: (i) encoder-decoder networks, (ii) encoder-only networks, and (iii) decoder-only networks. Encoder-decoder NNs are mainly designed for sequence-to-sequence tasks such as machine translation.

[0070] Figure 1 1 is a diagram showing the structure of the Transformer 100. One of the most widely used encoder-decoder networks is a Transformer constructed by stacking an encoder 110 and a decoder 120, such as Figure 1 shown. Figure 1 The structure of Transformer 100 is shown, where N 1 , D hid and R ffi denote the input length, hidden size, and FFN expansion ratio, respectively. Each encoder 110 starts with a multi-head attention module 111, followed by a feed-forward network (FFN) 112 consisting of two linear (fully connected) layers. Finally, residual addition and layer normalization (LN) are used after FFN 112.

[0071] Figure 2 The algorithm used by the multi-head attention module to perform linear transformation and attention is shown, and the pseudocode of the multi-head attention module is presented.

[0072] Figure 3 is the key to the notation used in this article and summarizes the notation used throughout this disclosure. The input is first mapped to a query (Q) matrix, a key (K) matrix, and a value (V) matrix through three different linear layers. The query matrix is ​​then TMultiply them, and then perform a softmax operation to obtain the score (S) matrix. The generated S is multiplied with V, and the resulting matrix flows into another linear layer to generate the final output matrix of the multi-head attention. Similarly, the decoder adopts two multi-head attention modules and one FFN, where the difference is that the input of the query vector and key vector in the second attention module comes from the output of the last encoder.

[0073] Based on the original encoder-decoder structure of Transformer, different variants have been proposed. Encoder-only networks (such as BERT and XLM) are also called autoencoding models and have been widely used in NLP tasks such as sequence classification. Visual Transformer (ViT) also belongs to this category and introduces an additional linear projection layer at the beginning. Their encoder layer corresponds to the encoder part of the original Transformer. Finally, the decoder-only network represents an autoregressive model designed for NLP tasks such as language modeling. GPT is a typical decoder-only model, corresponding to the decoder part of the original Transformer. Although the applicant focuses on the encoder-only network, the hardware design is flexible and also applicable to the decoder.

[0074] Butterfly matrices and FFTs. Despite impressive accuracy achieved using attention-based NNs, these models are expensive and non-scalable. The self-attention mechanism in the Transformer scales quadratically in computation and memory as a function of the input sequence length. Therefore, many works employ structured linear mappings (such as sparse and low-rank matrices) to approximate the attention matrix in the attention component and / or the weight matrix in the feedforward layer. However, choosing the appropriate structure for each linear mapping is application-dependent, often requires domain expertise, and because different structures have different trade-offs in performance and speed, a laborious process of hand-picking solutions is required.

[0075] In general, a sparse matrix is ​​a matrix that contains mostly zero values, i.e., most of the values / elements are zero. These are useful because they reduce the amount of computational processing that needs to be performed when performing matrix-based calculations (such as Fourier transforms or fast Fourier transforms). Since only the non-zero elements of a sparse matrix are stored, the amount of memory required to store the matrix is ​​reduced (compared to a dense matrix). It is desirable to transform data into a sparse matrix to make the calculations less complex / time consuming. As described below with respect to Figure 5As mentioned above, there are many different types of sparse matrices, each with their advantages and disadvantages. For example, a random sparsity pattern is a situation where random values ​​in the matrix are zero. Although this is sparse, it may be difficult for the processor to handle because the random values ​​of the matrix will need to be accessed and read. For example, in order to fully utilize the sparsity in the random pattern, a complex dynamic controller is required to achieve load balancing execution on different hardware engines. The additional overhead of such a controller may offset the improvement brought by skipping sparse operations, especially when the transistor size decreases.

[0076] To counteract this, many works have exploited butterfly matrices, which are a common representation of structured matrices and are efficient in practice due to their simple recursive structure. Specifically, each butterfly matrix W of size N Bfly The recursive divide-and-conquer structure of the Fast Fourier Transform (FFT) is encoded and can therefore be represented as a product of sparse butterfly factor matrices as follows:

[0077]

[0078] Among them, each W′ is called a butterfly factor N is a 2x2 block matrix, B i The size of is N / 2, and its entries can be trained via gradient-based methods:

[0079]

[0080] In other words, the butterfly matrix is ​​produced by the Fast Fourier Transform algorithm. The FFT algorithm works by performing the following steps: (1) separating the odd and even indices of the input, (2) performing the FFT on each half of the input, and (3) recombining the index pairs using a 2×2 matrix. The sparsity pattern of this FFT algorithm is analyzed, which shows that a recursive factorization of the FFT matrix occurs. The sparsity pattern of the factorization is called a butterfly matrix, and each individual sparse matrix in the product (see the equation above) is a butterfly factor. As Figure 5 As shown, each butterfly matrix W Bfly The sparsity patterns are different but fixed.

[0081] Due to their expressiveness in representing structured matrices and approximating unstructured data, butterfly matrices and their variants have been successful in replacing attention and weight matrices, greatly improving the accuracy and efficiency of attention-based NNs.

[0082] In addition to the attention and weight matrices, some works have explored replacing the entire attention mechanism with a more efficient counterpart. A prominent example is FNet, where the self-attention module is replaced by a 2D discrete Fourier transform (DFT) operation. Specifically, for each input, a 1D DFT is applied along the sequence dimension, and another 1D DFT is applied along the hidden dimension, keeping only the real component of the resulting output. Since the use of DFT helps information flow across all embeddings, it leads to similar performance compared to the use of ordinary self-attention layers, but with significantly reduced latency and memory.

[0083] In terms of algorithms, the proposed FABNet utilizes a hybrid of these techniques (FFT and butterfly matrix) to outperform existing works in terms of accuracy. It is worth noting that since the FFT matrix is ​​a special case of the butterfly matrix, is the identity matrix, is a rotation factor, therefore, both FFT and butterfly matrix have a recursive butterfly structure. Therefore, the computation and data access pattern can be unified, and then a single hardware engine can be designed to accelerate both FFT-based and butterfly-based operations with high hardware efficiency.

[0084] Latency Collapse and Motivation. Most previous Transformer-based accelerators focus on optimizing subcomponents of the entire model, leading to suboptimal performance gains. Furthermore, execution time depends heavily on the input length and thus varies across different subcomponents, reducing scalability with respect to input length and narrowing the deployability of these solutions. Figure 4 Shown is the latency breakdown of the Transformer’s subcomponents for different input lengths on both the GPU and CPU with batch sizes of 8 and 1, respectively.

[0085] Specifically, the Transformer architecture is divided into three main subcomponents: attention layers, linear layers, and other operations such as softmax, layer normalization, residual connections, matrix transformations, and IO operations. It is worth noting that on the CPU, the linear layer occupies a large portion of the execution time, up to 71.61%, for all input lengths. In contrast, on the GPU, executing the linear layer only occupies a large portion of the total execution time when the input layer is small, for example, 40.98% and 37.9% for input lengths 128 and 256, respectively. Therefore, the work focused on optimizing the attention layer can only be effective in limited cases of small input lengths.

[0086] On the other hand, accelerators that focus on optimizing only linear layers also suffer from the same shortcomings in certain common scenarios when the input length is large. For example, when the input length is 512, executing the attention layer on the GPU consumes up to 46.15% of the total execution time.

[0087] However, naively adopting a combination of previous works to optimize both the linear subcomponent and the attention subcomponent would require the instantiation of two separate engines, resulting in prohibitively high resource consumption. Thus, there is a gap in designing accelerators for scalable and general Transformer-based models.

[0088] The present applicants address this challenge by proposing an adaptive engine that accelerates both attention and linear layers through its runtime reconfigurability, resulting in a significant reduction in resource consumption with negligible loss in accuracy.

[0089] Algorithm optimization

[0090] Figure 5 is a diagram showing the basic sparsity patterns in existing efficient attention-based NNs. Various pruning schemes have been proposed to reduce the computational complexity of attention-based NNs, leading to different efficient models. By analyzing the computational patterns of these variants, five basic sparsity patterns are defined, such as Figure 5 As shown, (i) low-rank, (ii) sliding window, (iii) butterfly, (iv) random, and (v) block patterns. Observe that the sparsity in these variants can be represented as different combinations of the basic sparsity patterns, such as Figure 7 The rationale behind using multiple sparsity patterns in each variant is mainly to compensate for the loss of accuracy. For example, Pixelfly introduces additional low-rank sparsity patterns to increase the expressiveness of its flat block butterfly pattern and improve accuracy (B. Chen et al., "Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models" International Conference on Learning Representations (ICLR), 2022).

[0091] Different sparsity patterns exhibit different data access patterns, which require customized hardware support. However, supporting multiple sparsity patterns can complicate hardware design. For example, to fully exploit the sparsity in random patterns, complex dynamic controllers are required to achieve load-balanced execution on different hardware engines. The additional overhead of such controllers can offset the improvement brought by skipping sparse operations, especially as transistor size decreases.

[0092] This disclosure aims to find a hardware-friendly sparsity pattern that: (1) has a structured data access pattern to simplify memory design, (2) achieves satisfactory algorithmic performance without the help of other sparsity patterns, and (3) is applicable to both attention mechanisms and FFNs to achieve scalable improvements.

[0093] exist Figure 5 Among the five sparsity patterns in , the random pattern is excluded for two reasons: (1) the random pattern requires a complex controller with excessive hardware overhead, and (2) the performance gain of the random pattern cannot be guaranteed because the sparsity may vary significantly between different datasets and tasks. Figure 7 As summarized in , because the sliding window only captures the local relations within each window, the sliding window pattern always requires another sparsity to compensate for the accuracy loss. In addition, although some variants adopt a single low-rank or block sparsity pattern with satisfactory algorithmic performance, they require additional algorithmic operations and dedicated computational kernels (e.g., locality sensitive hashing (LSH) in the reformer) during inference, which brings significant overhead at the hardware level. Therefore, in the next section, the focus is on leveraging the butterfly sparsity pattern to accelerate attention-based NNs.

[0094] Figure 6 Figure 1 is a schematic diagram of the network structure of FABNet. The butterfly pattern has demonstrated its effectiveness and versatility in approximating linear transformations. In addition, the potential for simplifying computation by replacing the entire attention layer with Fourier transforms has been shown. In order to maximize the power of approximate computation with acceptable algorithmic performance, two basic building blocks for scalable inference are proposed: (1) Attention Butterfly (ABfly) block, and (2) Fourier Butterfly (FBfly) block.

[0095] In the ABfly block, the multi-head attention mechanism is maintained, but butterfly factorization is used to compress all linear layers. To further reduce the amount of computation and parameters, the FBfly block replaces the entire attention mechanism with a 2D Fourier transform layer. Although FBfly is less computationally and memory intensive than ABfly, the use of Fourier transform layers may also affect accuracy. To restore algorithm performance when needed, a butterfly-based network called FABNet is proposed, which introduces a mixture of ABfly and FBfly blocks, such as Figure 6 As shown in . There are NFBfly FBfly blocks at the beginning and NABfly ABfly blocks at the bottom. With this setup, both NFBfly and NABfly are exposed as user-defined hyperparameters, forming a trade-off between algorithm and hardware performance. To balance this trade-off, a co-design framework is proposed to explore the design space of both neural architecture and hardware design according to the user's requirements.

[0096] Hardware Accelerator

[0097] Figure 8The proposed hardware accelerator, electronic device or processing unit 800 for accelerating machine learning model calculations is shown. The electronic device 800 includes a first processor 810, also referred to herein as a butterfly processor (BP). The first processor 810 is used to accelerate calculations involving sparse matrices having a butterfly sparsity pattern.

[0098] The first processor 810 includes PBE butterfly engines (BE) 812 for accelerating calculations involving butterfly patterns (including both FFT and butterfly linear transforms). That is, the first processor 810 may include multiple butterfly engines 812 for accelerating calculations involving sparse matrices. Each butterfly engine 812 may include a memory system and multiple butterfly units 814, wherein each butterfly unit 814 may be configured to perform a specific calculation.

[0099] Each butterfly unit 814 may include a plurality of multiplexers arranged to select the inputs required to perform a particular calculation, as explained in more detail below.

[0100] In an embodiment, the first processor 810 generates a query matrix, a key matrix, and a value matrix by performing a Fast Fourier Transform (FFT) and a butterfly linear transform on at least one input matrix.

[0101] In an embodiment, the first processor 810 includes at least one butterfly engine 812, and the at least one butterfly engine 812 is configured to accelerate the calculation involving sparse matrices about FFT and butterfly linear transformation. At least one butterfly engine 812 includes an (adaptive) memory system 816 and multiple (adaptive) butterfly units 814, wherein each butterfly unit 814 in the multiple butterfly units includes at least one real multiplier, at least one real adder, at least one complex adder, at least one multiplexer and at least one demultiplexer. The configuration functions of at least one real multiplier, at least one real adder, at least one complex adder, at least one multiplexer and at least one demultiplexer are described in detail in FIG. 9.

[0102] In an embodiment, each butterfly unit in the plurality of butterfly units 814 performs FFT and butterfly transformation based on at least one input matrix and a plurality of rotation factors. In the case of performing butterfly linear transformation, the rotation factors are asymmetric real numbers. In the case of performing FFT, the rotation factors are complex numbers and symmetric numbers.

[0103] In an embodiment, the first processor 810 may generate the key matrix and the value matrix prior to the query matrix. When at least a portion of the query matrix becomes available, the second processor 820 may start the first matrix multiplication.

[0104] In an embodiment, the second processor 820 may start the second matrix multiplication when at least a portion of the result of the softmax operation becomes available.

[0105] The electronic device 800 may include a second processor 820, also referred to herein as an attention processor (AP). When the ML model includes at least one attention layer that requires computations that do not involve a sparse matrix having a butterfly sparsity pattern, the second processor 820 is configured to perform operations required by the at least one attention layer. The AP 820 includes a processor (AP) and a processor (CPU). head Attention engines (AE) 822 are provided, and each AE 822 includes a QK unit 824 and a SV unit 826. The QK unit 824 is designed to implement softmax and matrix multiplication between the query matrix and the key matrix (i.e., the transposed key matrix). The SV receives the output from the QK unit and multiplies the result with the value vector to generate the final result of the attention layer.

[0106] In an embodiment, the QK unit 824 of the second processor 820 performs a first matrix multiplication between the query matrix and the key matrix. The QK unit 824 of the second processor 820 performs a softmax operation on the result of the first matrix multiplication. The SV unit 826 of the second processor 820 performs a second matrix multiplication between the result of the softmax operation and the value matrix.

[0107] The electronic device 800 includes a third processor 830 (also referred to herein as a post-processing processor (PostP)), an off-chip memory, and different on-chip buffers. That is, the electronic device 800 includes a third processor 830, wherein the third processor 830 is used to receive the output from the first processor 810 and to perform post-processing using the received output. The third processor 830 is responsible for performing layer normalization and shortcut (SC) addition. In order to reduce on-chip memory consumption, intermediate results between different FFT and butterfly linear transform operations are transferred back to the off-chip memory. All on-chip buffers utilize double buffering so that data transfer and calculation overlap.

[0108] In an embodiment, the third processor 830 receives the query matrix, the key matrix, and the value matrix from the first processor 810. The third processor 830 performs at least one of layer normalization or shortcut addition based on the query matrix, the key matrix, and the value matrix.

[0109] Each BE 812 mainly includes a (adaptive) memory system 816 and a P BUThe BE 812 is designed to focus on adaptability in order to improve hardware efficiency and enable the use of a single unified engine. Therefore, it can be configured at run time via programmable multiplexers and demultiplexers to perform either FFT or butterfly linear transform.

[0110] FIG9a shows the architecture of the butterfly unit 814. Figure 8 9, each adaptive BU 814 includes four real multipliers and two real adders / subtractors, followed by two complex adders / subtractors. The inputs and rotation factors of both FFT and butterfly linear transform are connected to the multipliers, where eight multiplexers are used to select the correct input for each operation. Two demultiplexers are placed after the real adders / subtractors to control the output flow.

[0111] In an embodiment, four real number multipliers are arranged to multiply at least one input matrix and a plurality of rotation factors. Two real number adders or subtractors are arranged to add or subtract the outputs of two real number multipliers in the four real number multipliers. Two complex number adders or subtractors are arranged to add or subtract the outputs of two real number adders or subtractors. Eight multiplexers are arranged to select at least one input matrix required for performing FFT or butterfly linear transformation. Two demultiplexers are arranged to control output streams, including outputting data from two real number adders or subtractors, or providing data from two real number adders or subtractors to two complex number adders or subtractors.

[0112] In an embodiment, control signals for eight multiplexers or two demultiplexers are set before performing the FFT and butterfly linear transform.

[0113] When performing a butterfly linear transformation (see Figure 9b), the rotation factors are asymmetric real numbers. Therefore, the output of each rotation multiplication can be calculated as:

[0114]

[0115] in, and Denote the input factors and rotation factors, respectively. To perform the butterfly linear transformation, the four multipliers in each BE are configured to perform the four real multiplications in the above equation. The multiplexer selects the value and As the operand of the multiplier. At the same time, the result generated from the real adder / subtractor Directly from the demultiplexer output.

[0116] For FFT (see Figure 9c), since the rotation factors of FFT are complex and symmetric, it only requires one complex multiplication per rotation multiplication. Therefore, by choosing the complex input and the rotation factor w fft_r +w fft_i i, the four real multipliers in each BE are used to perform the required complex multiplication. The demultiplexer is then configured to output the result to the complex adder / subtractor to obtain the final result The control signals of the multiplexer and demultiplexer are set by the user before running each layer. Therefore, the proposed adaptive BE can be used to speed up both FFT and butterfly linear transform by reusing multipliers, adders and subtractors to improve hardware efficiency. The control signals of the multiplexer and demultiplexer are set by the user before running each layer. Therefore, the proposed adaptive BE can be used to speed up both FFT and butterfly linear transform by reusing multipliers, adders and subtractors to improve hardware efficiency.

[0117] Fig.12 Memory bank conflicts in column-major and row-major order are shown. The applicant's adaptive memory system includes an input manager, a serial-to-parallel (S2P) module, a parallel-to-serial (P2S) module, and an input buffer. In order to reduce memory consumption, the same input buffer is used for both FFT and butterfly linear transform. In addition, ping-pong buffering is used to reduce data transfer time. However, since the data width of FFT is twice that of butterfly linear transform, different address mapping and fine-grained pipeline strategies are used to improve hardware efficiency and performance.

[0118] Fig.10 The present address mapping strategy for FFT and butterfly linear transform is shown. Assuming the bit width of real numbers is 16 bits, each input buffer is 16 bits wide. When processing butterfly linear transform, input buffers A and B are used as two independent ping-pong memory banks with separate read ports and write ports. In this way, when input buffer A is used for calculation, buffer B can start the next batch of input data transfer, resulting in Fig.11 The fine-grained pipeline strategy shown in . When processing FFT, since the data includes real and imaginary parts that require 32-bit read and write ports, the lower parts of input buffers A and B are connected as the first ping-pong memory bank for storing complex numbers. To improve hardware efficiency, the upper parts of the two buffers are reused as the second ping-pong memory bank. Fig.11 As shown in Fig. 2, since the computation requires both read access and write access, another fine-grained pipeline strategy is adopted to overlap the output data transfer with the input data loading of the next batch. The adaptive pipeline strategy of FFT and butterfly linear transform is used to maximize the hardware efficiency and performance.

[0119] like Fig.12 As shown in (a), the butterfly pattern requires different data access at different levels. The traditional column-major order or row-major order will cause memory bank conflicts when reading data. For example, Fig.12 As shown in (b), accessing the first-level index pair x o and x 8 This causes a read conflict in column-major order. When reading x in the third level o and x 2 The behavior-major order suffers from the same problem.

[0120] In order to avoid storage conflicts, a custom data layout strategy is introduced ( Fig.13 (a)) and use Fig.13 The S2P module shown in (b) is used to implement the i Permute each column i, starting at position P i Indicates how many rows the first element in the current column should be shifted down. The starting position is defined using the following formula:

[0121]

[0122] For every 2 n -2 n-1 Columns, such as Fig.13 As shown in a, by Move down one position to get the starting position Use a counter with bit counting and addition operations to generate the starting position ( Fig.13 b). After packing the serial data together, S2P permutes the packed data based on the starting position to generate the final output.

[0123] In an embodiment, Fig.13 The functions and structures shown in Figure 8 The memory system 816 shown in FIG. Figure 8 as well as Fig.13 , the memory system 816 may calculate the starting position of the data layout corresponding to the input matrix. The starting position may indicate how many rows the first element in the current column should be shifted down. The memory system 816 may permute at least one input matrix based on the starting position, and the memory system 816 may provide data access to the butterfly unit 814 based on the permuted input matrix. Fig.14 An example with 16 inputs is shown, where the data required for the first and second stages of the butterfly pattern are read from the buffer without bank conflicts. However, as the butterfly unit receives paired data, additional pairing is required after the SP2 module. An example is Fig.14 The second output column of the first level in b <x 11 , x 1, x 9 , x 3 >. In order to pair the index, an index merge module is designed before the butterfly unit ( Fig.15 ). Based on the index of each input, the corresponding shift position is calculated using bit counting and addition operations. Then, the crossbar merges the index pairs based on the index and shift position. To ensure that the output produced from the butterfly unit remains in the original order, a recovery module is used before writing back the data.

[0124] When executing the ABfly block, BP and AP are used to perform butterfly linear transformation and attention matrix multiplication respectively. In order to further improve the performance when executing the ABfly block, the applicant adopts a fine-grained pipeline between BP and AP.

[0125] Fig.11 The adaptive fine-grained pipeline strategy is shown, and the data flow of BP and AP is shown. In the original implementation, the key (K) matrix, value (V) matrix, and query (Q) matrix are generated sequentially from BP. After all Q, K, and V are calculated, AP starts Q×K T To optimize this process, the applicant reorders the execution sequence of the linear layer so that BP calculates K and V at the beginning, see Fig.11 b. Due to Q×K T can be decomposed into the different rows of Q and the entire matrix K T Multiple vector-matrix multiplications are multiplied together, so once the first few rows of Q are available, we can start Q×K T Therefore, Q×K in AP T can be pipelined together with the computation of Q in BP. At the same time, since S is generated row by row from the QK unit, the computation of S×V can start once the first few rows of S are generated from the QK unit, so Q×K T It can be further pipelined with S×V. Assume that there are M rows and L rows in the Q matrix and K matrix, and that SV units and QK units are required, respectively. and to compute a row, the total latency is reduced to

[0126] The overall design space of the applicant's end-to-end system is formed by the hyperparameters of FABNet and the hardware parameters of the butterfly accelerator. Specifically, the joint design space includes: 1) algorithm parameters, i.e., hidden size (D hid )、FFN(R ffn ) expansion ratio, the total number of blocks in FABNet (N tota l) and the number of ABfly blocks (N ABfly ), and 2) hardware parameters, i.e., BU (N ABfly ) and BE(Pbe ) and the parallelism of QK(P qk ) and SV(P SV ) units’ parallelism.

[0127] In order to evaluate the tradeoffs offered by each design point, it is necessary to evaluate its algorithmic performance (e.g., accuracy metrics), its latency, and its resource consumption. During the search, algorithmic performance was obtained by training and evaluating FABNet, while latency was estimated by utilizing a custom cycle-accurate simulator built for the Applicant's butterfly accelerator. To verify whether the target FPGA device can adapt to the design, the Applicant developed an analytical model to estimate the consumption of DSP blocks and on-chip memory (BRAM), and used a set of placement and routing measurements to fit a linear regression model of the consumption of lookup tables (LUTs) and registers.

[0128] In order to evaluate the performance of this algorithm and hardware under long sequence loads, this paper selected 6 tasks in the long-range field, including hierarchical data classification (ListOPs), byte-level text classification (Text), byte-level document retrieval (Retrieval), image classification of pixel sequences (Image), and long-range spatial dependency classification (Pathfinder). The input sequences of these datasets range from 1024 to 4000.

[0129] Software implementation. The vanilla Transformer, FNet, and FABNet models are implemented using the Pytorch 1.10 framework. The pre-trained models are obtained from Huggingface 4.16. The batch size is 256 for both the Image task and the Pathfinder task, while the batch size is 32 for the rest of the datasets during training. The learning rate is 0.0001, except for the Image task which is 0.01 and the Pathfinder task which is 0.0005. Multiple Nvidia A100 and V100 GPUs are used for training. In order to use the FFT core on Nvidia GPUs, the PyTorch API “rfft2” is used to implement the required FFT operations in both FNet and FABNet. A high-performance CUDA implementation of the butterfly linear transform is adopted to accelerate both GPU training and inference. Two models are defined with different default settings: FABNet-Base (D hia =768, R ffn =4,N total＝12 , N ABfly =0) and FABNet-Large(D hid =1024, R ffn =4,N total＝24 , N ABfly =0).

[0130] The applicants implemented their hardware accelerator using Verilog. To evaluate the performance in different scenarios, two Xilinx FPGA boards were used in the experiments: VCU128 for cloud / server scenarios and Zynq7045 for edge / mobile settings. Xilinx Vivado 2019.1 was used for synthesis and implementation. The clock frequency of this design varies with different FPGA boards and resource consumption. All FPGA designs are clocked at 200MHz, which is lower than the maximum. Power consumption is obtained using the XilinxPower Estimator (XPE) tool provided by Vivado. A cycle-accurate performance model was developed to evaluate the speed performance, which was cross-validated with the RTL simulation results. The applicant's hardware design uses 16-bit floating point.

[0131] Effectiveness of Co-design. The effectiveness of the applicant's co-design method in finding the optimal algorithm and hardware design is evaluated. For demonstration, LRA-Text is used as the target dataset and VCU128 FPGA is used as the target device. hid , R ffn , N total , and N ABfly are selected from {64, 128, 256, 512, 1024}, {1, 2, 4}, {0, 1} and {1, 2} respectively. Hardware parallelization (P be , P bu , P qk , and N SV Selected from {4, 8, 16, 32, 64, 128}. Among the design points that meet the accuracy constraints, the points with the lowest latency in the Pareto front are selected as comparison points. The selected points are up to 10% more accurate than points in the same latency range and up to 130 times faster than points in the same accuracy range, emphasizing the advantages of the applicant's collaborative design approach. To obtain configurations for the remaining data sets in LRA, the overall accuracy loss is constrained to be less than 0.5% compared to the vanilla Transformer. The final model and design are selected as the configuration with the highest hardware performance without violating the accuracy constraints. Unless otherwise mentioned, the rest reports the algorithm and hardware performance using the configuration optimized by the applicant's hardware configuration.

[0132] Fig.16The accuracy of the vanilla Transformer FNet and the applicant's FABNet is compared. On three of the five tasks (including ListOPs, Retrieval, and Image), FABNet achieves higher accuracy than Transformer and FNet. FAB-Net has the same accuracy performance as Transformer on average. In order to study the efficiency of FABNet, Fig.18 The compression ratio of the applicant's optimized FABNet in terms of floating point operations (FLOPS) and model size (number of parameters) over the vanilla Transformer and FNet is shown. Compared with the vanilla Transformer, FABNet reduces FLOP and model size by 10 to 60 times and 2 to 22 times, respectively. In addition, compared with FNet, FABNet reduces FLOP and model size by 2 to 10 times and 2 to 32 times, respectively.

[0133] To demonstrate the advantages of both the co-design algorithm (FABNet) and hardware (butterfly accelerator), a baseline design is implemented to accelerate the vanilla BERT. The baseline hardware design has multiple multiplier-accumulator (MAC) units to accelerate linear transformations and different matrix multiplications between query vectors, key vectors, and value vectors. Each MAC consists of a bundle of multipliers and a tree of adders following it. Fine-grained intra-layer and inter-layer pipelining techniques are employed to optimize the hardware performance. The parallelism of each MAC unit is allocated according to its workload to achieve load-balanced execution between different pipeline stages. For fair comparison, both the baseline accelerator and the butterfly accelerator are implemented on a VCU128 FPGA using a 2048 DSP. High Bandwidth Memory (HBM) is used as external memory. Both designs are clocked at 200MHz. Both the base and large versions of each model are evaluated using four different input sequences (128, 256, 512, and 1024). The base version contains 12 layers and the large version has 24 layers. The acceleration collapses at Fig.17 Compared with the BERT-Base baseline design, the butterfly accelerator can achieve a 21-30 times acceleration on FFN. The acceleration is increased to 37-53 times on the large version. At the same time, the butterfly accelerator is 57-120 times and 98-201 times faster than the baseline design in the attention part of Bert-Base and Bert-Large, respectively. The butterfly accelerator shows a higher acceleration in attention than FFN, which is due to the higher compression rate brought by FFT.

[0134] Comparisons with GPU and CPU are performed in both server and edge scenarios. In the edge setting, the butterfly accelerator is implemented on Xilinx Zynq7045 FPGA. DDR4 is used as external memory, and 512 DSPs are used for computation. NvidiaJetson Nano GPU and Raspberry Pi4 are used as GPU platform and CPU platform, respectively. In the server scenario, the butterfly accelerator is implemented using Xilinx VCU128 FPGA. HBM is used as external memory, and the design consumes 2048 DSPs. Nvidia V100 and TITAN Xp GPUs are used for comparison. Highly optimized CUDA code is used for GPU implementation. The FPGA design is clocked at 200MHz. Both FABNet-Base and FABNet-Large are evaluated using 128, 256, 512, and 1024 input sequences. Fig.19 Results in terms of acceleration and energy efficiency are shown. Energy efficiency is expressed in giga operations per second per watt (GOPS / Watt). In edge scenarios, this design on Zynq 7045FPGA achieves 4.5 to 10 times acceleration over Jetson Nano GPU and 41.2 to 394.8 times acceleration over Raspberry Pi4. At the same time, this design also shows energy efficiency that is 5.0 to 12.5 times and 21.3 to 197.4 times higher than Jetson Nano and Raspberry Pi4, respectively. In server scenarios, this design on VCU128 is up to 10.9 times and 9.7 times faster than V100 and TITAN Xp GPU, respectively, and has energy efficiency of up to 160.6 times and 149 times. In summary, the end-to-end acceleration and energy efficiency gains on both edge scenarios and server scenarios under different input sequences highlight the scalability of this butterfly accelerator.

[0135] In edge scenarios, this design on Zynq 7045FPGA achieves 4.5-10 times acceleration compared to Jetson Nano GPU, and 41.2-394.8 times acceleration compared to Raspberry Pi4. On FABNet-Large with long input sequences greater than 768, Raspberry Pi4 suffers from insufficient memory (OOM) problems. At the same time, this design also shows energy efficiency that is 5.0-12.5 and 21.3-197.4 higher than Jetson Nano and Raspberry Pi4, respectively. In a server setting, this design on VCU128 is 10.9 and 756 times faster than RTX 2080Ti GPU and Golden 6145CPU. Energy efficiency that is 168.0 and 7624.9 times higher than RTX 2080Ti GPU and Golden 6145CPU, respectively, is achieved. End-to-end acceleration in both edge scenarios and server scenarios under different input sequences also demonstrates the scalability of this butterfly accelerator.

[0136] Fig. 20 The butterfly accelerator is compared with other state-of-the-art (SOTA) accelerators in terms of speed performance and power consumption. Instead of comparing effective throughput, end-to-end latency is used to represent the actual execution speed. Energy efficiency is represented by the number of predictions per joule (Pred. / J). Among these accelerators, only SpAtten and DOTA report end-to-end performance. For the remaining accelerators with only attention support, their performance is estimated by reusing available multipliers in the design to accelerate FFN. For a fair comparison, it is assumed that all ASIC designs use 128 multipliers for calculations and the clock frequency is 1GHz. For designs with more than 128 multipliers, the throughput is linearly scaled down to obtain end-to-end performance. For example, DOTA achieves an 11.41x acceleration over Nvidia V100 with 12TOPS throughput using 12,000 multipliers. The throughput is scaled down by 12,000 / 128=93.75, which results in a 0.123x acceleration. Power consumption is obtained using the same linear scaling method. For example, Sanger reports the power consumption of a design with 1024 multipliers. The power consumption of their systolic array (2243mW) is divided by 1024 / 128=8, which results in 280.375mW. Together with the power of other modules such as preprocessing and memory, their total power consumption is 0.801W. In order to match the computational power of the ASIC design, 640 DSPs in the VCU128FPGA are used. Since this FPGA-based design is clocked at 200MHz, this ensures the same theoretical peak performance of 620×200M=128 GOPs as the ASIC design 128×1G=128 GOPs.

[0137] like Fig. 20 As shown, the butterfly accelerator achieves a 25.6-fold acceleration over the previous FPGA-based design while using nearly 10 times less DSP. At the same time, it achieves 62.3 times higher energy efficiency than FTRANS. Compared with other ASIC designs, the present disclosure can achieve 14.2 to 23.2 times acceleration at the same computing power. Although the power consumption of the FPGA-based butterfly design is higher than all other ASIC designs, the energy efficiency is 1.2 to 4.7 times higher than other SOTA ASIC accelerators. When the present design is implemented on an ASIC, higher acceleration and energy efficiency are expected. There are two reasons for the better performance than the ASIC design: (1) The use of FFT and butterfly factorization significantly reduces the computational complexity at the algorithm level; the adaptive butterfly design uses a unified hardware engine to accelerate both FFT and butterfly linear transform, which greatly improves hardware efficiency.

[0138] To study the sensitivity of this design to the off-chip memory bandwidth, the bandwidth was varied from 6, 12, 25, 50, 100, and 200 Gb / s and its latency was evaluated based on this custom cycle-accurate simulator. For this experiment, five different designs with 16, 32, 64, and 128 BEs were used when executing a large FABNet-Large with 24 layers. To understand the bandwidth requirements at short and long input lengths, each design was evaluated using three input sequences (128, 1024, and 4096). The results are shown in Fig.21 For the small design with 16 BEs, a bandwidth of 50 Gb / s is sufficient for the design to reach its peak performance under different input sequences. For the largest design with 128 BEs, the achieved performance saturates once the bandwidth reaches 100 GB / s.

[0139] Power and Resource Analysis. Table 6 shows the breakdown of power consumption based on the reports generated from the Vivado XPE tool. Two designs of 128 BEs (BE-128) and 40 BEs (BE-40) were implemented on a VCU128 FPGA. In both designs, dynamic power accounts for more than 70% of the total power consumption. Memory resources, including both BRAM and HBM, consume more than 40% of the dynamic power consumption. In addition, when the number of BEs is scaled from 40 to 128, the power of clock, logic and signal, and DSP increases from 0.724W, 0.844W, and 0.129W to 2.143W, 2.693W, and 0.412W, respectively. Table 7 presents the resource consumption of both BE40 and BE-128 on the same VCU128 FPGA. Due to the use of FFT and butterfly matrices, the present FABNet becomes less memory intensive than ordinary attention-based NNs. Since the theoretical memory bandwidth of a single HBM (450Gb / s) can already meet the requirements of this accelerator, one HBM is used in both designs to reduce resources and power consumption. When the number of BEs is reduced from 128 to 40, the BRAM utilization is reduced from 1042 to 338. Reductions can also be observed in LUT resources and register resources.

[0140] Fig. 22 A flowchart describing a method for accelerating machine learning ML model calculations using a processing unit described herein is shown. The method includes: receiving an input matrix for ML model processing (step S100); and using a first processor of the processing unit to perform the following operations: accessing data from the input matrix based on a butterfly sparsity pattern (step S102); configuring multiple butterfly units of the first processor to perform fast Fourier transform calculations (step S104); and performing fast Fourier transform calculations using the accessed data (step S106). The same method can be used to accelerate calculations of any layer involving a matrix with a butterfly sparsity pattern. That is, instead of FFT calculations, any other calculations (e.g., linear transformations) of any layer of the ML model can be performed using the first processor, as long as the layer uses butterfly sparsity.

[0141] The method may further include: using a second processor of the processing unit to perform the following operations: using the input matrix and the output of the first processor to perform an attention mechanism calculation, wherein the attention mechanism calculation does not involve a sparse matrix with a butterfly sparsity pattern. That is, when the ML model includes even a single attention layer (which is vanilla or conventional (i.e., does not use butterfly sparsity)), the second processor is used to perform the attention layer when the model is executed.

[0142] Fig.23A flowchart describing a method for accelerating machine learning ML model calculations using the processing unit described herein is shown. The method includes: generating a query matrix, a key matrix, and a value matrix by performing a fast Fourier transform (FFT) and a butterfly linear transform on at least one input matrix (step S210); performing a first matrix multiplication between the query matrix and the key matrix (step S220); performing a softmax operation on the result of the first matrix multiplication (step S230); and performing a second matrix multiplication between the result of the softmax operation and the value matrix (step S240).

[0143] The method may further include: receiving a query matrix, a key matrix, and a value matrix; and performing at least one of layer normalization or shortcut addition based on the query matrix, the key matrix, and the value matrix.

[0144] The method may also include: generating a key matrix and a value matrix before querying the matrix; and starting a first matrix multiplication when at least a portion of the query matrix becomes available. Herein, "matrix available" means that in an operation using a matrix as an operand, at least one row or at least one column of the matrix required for the operation is in a completed or ready state.

[0145] The method may further include starting the second matrix multiplication when at least a portion of a result of the softmax operation becomes available.

[0146] Fig.24 1 is a block diagram of a device 1000 incorporating an accelerator of the present disclosure. Device 1000 may be any device that will implement an ML model and benefit from an accelerator to achieve high-speed computing. For example, device 1000 may be a resource-constrained device, but it has minimal hardware capabilities to use a trained neural network / ML model. Device 1000 may be: a smartphone, a tablet, a laptop, a computer or computing device, a virtual assistant device, a vehicle, a drone, an autonomous vehicle, a robot or robotic device, a robotic assistant, an image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, a smart consumer device, a smart watch, a fitness tracker, and a wearable device. It should be understood that this is a non-exhaustive and non-limiting list of example devices.

[0147] The device 1000 includes at least one processor 1100. The at least one processor 1100 may include one or more of the following: a microprocessor, a microcontroller, and an integrated circuit. The at least one processor 1100 may be, for example, a CPU or a GPU.

[0148] Device 1100 includes this document (including reference Figure 8) is an electronic device 800 of the type described in the foregoing description. Therefore, the description of the processing unit 800 is not provided here. The electronic device 800 accelerates the machine learning ML model calculation. Specifically, the electronic device 800 cooperates with the general processor 1100 of the device 1000 to increase the speed of processing data, thereby improving the performance of the processor 1100.

[0149] The device 1000 may include a memory 1200. For example, the memory 1200 may include a volatile memory such as a random access memory (RAM) used as a temporary memory and / or a nonvolatile memory such as a flash memory, a read-only memory (ROM), or an electrically erasable programmable ROM (EEPROM) for storing data, programs, or instructions.

[0150] The apparatus 1000 includes an ML model 1400. The electronic device 800 accelerates the calculation of the ML model involving a sparse matrix, and the sparse matrix has a butterfly sparsity pattern. As described above, the first processor 810 of the electronic device 800 accelerates the calculation of any layer of the ML model 1400 as long as the calculation involves a matrix with a butterfly sparsity pattern. That is, when the ML model 1400 starts to process some data using the processor 1100, and (because of, for example, latency requirements) accelerated processing is required, the processor 1100 can offload the task to the electronic device 800.

[0151] The device 1000 includes at least one interface 1300 for receiving data / data items processed by the ML model 1400. The interface 1300 may be a camera for capturing images or a microphone for capturing audio. It will be understood that these are non-limiting example interfaces.

[0152] In an embodiment of the present disclosure, an electronic device for accelerating machine learning ML model calculation is provided. The electronic device includes: a first processor configured to generate a query matrix, a key matrix, and a value matrix by performing a fast Fourier transform (FFT) and a butterfly linear transform on at least one input matrix, and a second processor configured to perform a first matrix multiplication between the query matrix and the key matrix, perform a softmax operation on the result of the first matrix multiplication, and perform a second matrix multiplication between the result of the softmax operation and the value matrix.

[0153] In an embodiment of the present disclosure, the first processor includes at least one butterfly engine, which is configured to accelerate calculations involving sparse matrices regarding FFT and butterfly linear transform, wherein the at least one butterfly engine includes a memory system and a plurality of butterfly units, wherein each of the plurality of butterfly units includes at least one real multiplier, at least one real adder, at least one complex adder, at least one multiplexer and at least one demultiplexer.

[0154] In an embodiment of the present disclosure, each of the plurality of butterfly units is configured to perform FFT and butterfly transform based on the at least one input matrix and the plurality of rotation factors.

[0155] In an embodiment of the present disclosure, in the case of performing butterfly linear transformation, the rotation factor is an asymmetric real number, and in the case of performing FFT, the rotation factor is a complex number and a symmetric number.

[0156] In an embodiment of the present disclosure, each of the plurality of butterfly units includes: four real number multipliers, arranged to multiply at least one input matrix and a plurality of rotation factors; two real number adders or subtractors, arranged to add or subtract outputs of two real number multipliers among the four real number multipliers; two complex number adders or subtractors, arranged to add or subtract outputs of the two real number adders or subtractors; eight multiplexers, arranged to select at least one input matrix required to perform FFT or butterfly linear transformation; and two demultiplexers, arranged to control output flows, including outputting data from two real number adders or subtractors, or providing data from two real number adders or subtractors to two complex number adders or subtractors.

[0157] In an embodiment of the present disclosure, control signals for eight multiplexers or two demultiplexers are set before performing FFT and butterfly transform.

[0158] In an embodiment of the present disclosure, the memory system is configured to: calculate a starting position of a data layout corresponding to the input matrix, wherein the starting position indicates how many rows the first element in the current column should be shifted down; permute the at least one input matrix based on the starting position; and provide data access to the butterfly unit based on the permuted input matrix.

[0159] In an embodiment of the present disclosure, the electronic device includes: a third processor configured to: receive a query matrix, a key matrix, and a value matrix from the first processor, and perform at least one of layer normalization or shortcut addition based on the query matrix, the key matrix, and the value matrix.

[0160] In an embodiment of the present disclosure, the first processor is configured to generate a key matrix and a value matrix before a query matrix, and the second processor is configured to start the first matrix multiplication when at least a portion of the query matrix becomes available.

[0161] In an embodiment of the present disclosure, the second processor is configured to start the second matrix multiplication when at least a portion of the result of the softmax operation becomes available.

[0162] In an embodiment of the present disclosure, a method for accelerating machine learning ML model calculation is provided. The method includes: generating a query matrix, a key matrix, and a value matrix by performing a fast Fourier transform (FFT) and a butterfly linear transform on at least one input matrix, performing a first matrix multiplication between the query matrix and the key matrix, performing a softmax operation on the result of the first matrix multiplication, and performing a second matrix multiplication between the result of the softmax operation and the value matrix.

[0163] In an embodiment of the present disclosure, the method includes receiving a query matrix, a key matrix, and a value matrix, and performing at least one of layer normalization or shortcut addition based on the query matrix, the key matrix, and the value matrix.

[0164] In an embodiment of the present disclosure, the method includes generating a key matrix and a value matrix before a query matrix, and starting a first matrix multiplication when at least a portion of the query matrix becomes available.

[0165] In an embodiment of the present disclosure, the method includes starting the second matrix multiplication when at least a portion of the result of the softmax operation becomes available.

[0166] In an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes instructions, and when the instructions are executed by a processor, the processor executes the method of the present disclosure.

[0167] Those skilled in the art will appreciate that, although the foregoing has described what is considered to be the best mode and other modes of carrying out the present disclosure where appropriate, the present disclosure should not be limited to the specific configurations and methods disclosed in the description of the preferred embodiments. Those skilled in the art will recognize that the present disclosure has a wide range of applications, and that the embodiments can be extensively modified without departing from any inventive concept defined in the appended claims.

Claims

1. An electronic device (800) for accelerating machine learning (ML) model calculation, the electronic device comprising: The first processor (810) is configured to: generating a query matrix, a key matrix, and a value matrix by performing a fast Fourier transform (FFT) and a butterfly linear transform on at least one input matrix; as well as The second processor (820) is configured to: performing a first matrix multiplication between the query matrix and the key matrix; Perform a softmax operation on the result of the first matrix multiplication; as well as A second matrix multiplication is performed between the result of the softmax operation and the value matrix.

2. The electronic device (800) according to claim 1, wherein: The first processor (810) includes at least one butterfly engine (812), the at least one butterfly engine (812) being configured to accelerate computations involving sparse matrices regarding the FFT and the butterfly linear transform, wherein the at least one butterfly engine (812) includes a memory system (816) and a plurality of butterfly units (814), wherein each of the plurality of butterfly units (814) includes at least one real multiplier, at least one real adder, at least one complex adder, at least one multiplexer, and at least one demultiplexer.

3. The electronic device (800) according to claim 2, wherein: Each butterfly unit of the plurality of butterfly units (814) is configured to perform the FFT and the butterfly transform based on the at least one input matrix and a plurality of rotation factors.

4. The electronic device (800) according to claim 3, wherein: In the case of performing a butterfly linear transformation, the rotation factors are asymmetric real numbers; and In the case of performing an FFT, the twiddle factors are complex and symmetric numbers.

5. The electronic device (800) according to any one of claims 2 to 4, wherein: Each of the plurality of butterfly units (814) comprises: four real multipliers arranged to multiply at least one input matrix with a plurality of twiddle factors; two real adders or subtractors arranged to add or subtract outputs of two of the four real multipliers; two complex adders or subtractors arranged to add or subtract the outputs of the two real adders or subtractors; eight multiplexers arranged to select at least one input matrix required to perform an FFT or butterfly linear transform; and Two demultiplexers are arranged to control the output flow, including outputting the data from the two real adders or subtracters, or providing the data from the two real adders or subtracters to the two complex adders or subtracters.

6. The electronic device (800) according to claim 5, wherein: The control signals for the eight multiplexers or the two demultiplexers are set before performing FFT and butterfly linear transform.

7. The electronic device (800) according to any one of claims 2 to 6, wherein: The memory system (816) is configured to: Calculate the starting position of the data layout corresponding to the input matrix, the starting position indicating how many rows the first element in the current column should be shifted down; permuting the at least one input matrix based on the starting position; and Data access to the butterfly unit is provided based on the permuted input matrix.

8. The electronic device (800) according to any one of claims 1 to 7, further comprising: The third processor (830) is configured to: receiving the query matrix, the key matrix, and the value matrix from a first processor; as well as At least one of layer normalization or shortcut addition is performed based on the query matrix, the key matrix, and the value matrix.

9. The electronic device (800) according to any one of claims 1 to 8, wherein: The first processor (810) is further configured to generate the key matrix and the value matrix prior to the query matrix; and The second processor (820) is further configured to start the first matrix multiplication when at least a portion of the query matrix becomes available.

10. The electronic device (800) according to claim 9, wherein: The second processor (820) is further configured to start the second matrix multiplication when at least a portion of the result of the softmax operation becomes available.

11. A method for accelerating machine learning (ML) model calculation, the method comprising: Generate a query matrix, a key matrix, and a value matrix by performing a fast Fourier transform (FFT) and a butterfly linear transform on at least one input matrix (S210); performing a first matrix multiplication between the query matrix and the key matrix (S220); Performing a softmax operation on the result of the first matrix multiplication (S230); as well as A second matrix multiplication is performed between the result of the softmax operation and the value matrix (S230).

12. The method according to claim 11, further comprising: receiving the query matrix, the key matrix, and the value matrix; as well as At least one of layer normalization or shortcut addition is performed based on the query matrix, the key matrix, and the value matrix.

13. The method according to claim 11, further comprising: generating the key matrix and the value matrix before the query matrix; as well as When at least a portion of the query matrix becomes available, a first matrix multiplication is initiated.

14. The method according to claim 11, further comprising: When at least a portion of the result of the softmax operation becomes available, a second matrix multiplication is started.

15. A computer-readable storage medium comprising instructions which, when executed by a processor, cause the processor to perform the method according to claims 11 to 14.

Images