A swin-transformer inference hardware accelerator

Abstract

The application provides a Swin-Transformer reasoning hardware accelerator, comprising a three-layer storage hierarchy structure of external storage, static random access memory (SRAM) and register (Reg), two quantization units, one outer product matrix multiplication array and three nonlinear calculation units; the application designs a special hardware acceleration architecture applied to an end side by means of the unique calculation characteristics of Swin Transformer, and verifies the acceleration architecture on an FPGA platform; the energy efficiency ratio of the accelerator is improved by 15.14 times compared with an NVIDIARTX 2080Ti GPU, and is improved by 2.02 to 3.11 times compared with an existing Transformer accelerator.

Description

Technical Field

[0001] This invention relates to a Swing-Transformer inference hardware accelerator. Background Technology

[0002] In recent years, Transformer-type neural networks have been successfully applied to computer vision tasks and have achieved state-of-the-art performance. However, their excellent performance comes at the cost of a huge number of parameters and computational burden, making the deployment of their inference process very expensive and resulting in high runtime latency when deployed on resource-constrained edge platforms. By developing a dedicated Transformer accelerator based on an FPGA platform through customized processing unit arrays and sophisticated dataflow design, high energy efficiency can be provided for the inference deployment of Transformer models on edge platforms.

[0003] Early versions of the Vision Transformer (ViT) suffered from computational complexity that was quadratic with the image size due to the need to compute global self-attention. The Swing Transformer solved this problem by constructing a hierarchical network architecture with a moving window, achieving computational complexity linear with the image size. The most crucial computational component is the Swing Transformer Block, whose architecture is as follows: Figure 1 As shown. However, existing Transformer accelerator designs do not take advantage of the hardware-friendly features of SwinTransformer.

[0004] The Swin Transformer follows the ViT approach of segmenting the input image into multiple non-overlapping patches. Building upon this, it further divides a fixed number of patches into windows, restricting the computation of self-attention to within a single window. This window partitioning, as a natural data segmentation method, ensures that the data between windows is independent, allowing for separate computation of each window. Compared to computing global attention, self-attention computation confined to a window requires less computation and fewer parameters.

[0005] To ensure information exchange between windows, the Swin Transformer moves the windows to the bottom right corner, allowing self-attention to be calculated between patches that were not originally in the same window. However, moving windows increases the number of windows and computational complexity. Using cyclic shifting and mask matrices can solve this problem while maintaining computational correctness, and also introduces sparsity.

[0006] Due to the data dependencies within each window, nonlinear computations can block matrix multiplication, increasing the overall network latency. Furthermore, complex nonlinear computations also lead to high hardware complexity. Employing quantization methods to compress nonlinear computations and approximate calculations to simplify complex operations is an effective solution.

[0007] ViA (Reference: T. Wang, L. Gong, C. Wang, Y. Yang, Y. Gao, X. Zhou, and H. Chen, “Via: A novel vision-transformer accelerator based on FPGA,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 41, no. 11, pp. 4088–4099, 2022) designed an FPGA-based ViT hardware accelerator with two processing units corresponding to the two half-layer computation processes in each layer of ViT. Using a suitable partitioning strategy for each layer allows for pipelined deployment of computations across different layers of ViT on ViA. However, the strong data dependency between the two half-layers causes idle states in the computation units during processing, reducing computational efficiency. Simultaneously, multiple accesses to external memory increase hardware power consumption. All of these factors reduce the energy efficiency of the architecture, hindering its deployment and widespread adoption on edge devices. Summary of the Invention

[0008] Purpose of the invention: The technical problem to be solved by the present invention is to provide a Swin-Transformer inference hardware accelerator that addresses the shortcomings of the prior art. The accelerator includes a three-layer storage hierarchy consisting of external storage, static random access memory (SRAM) and registers (Reg), two quantization units, an outer product matrix multiplication array, and three nonlinear computation units.

[0009] The external storage is used to store off-chip data;

[0010] The static random access memory (SRAM) is used to store on-chip data;

[0011] The register Reg is used to store intermediate calculation result data;

[0012] The two quantization units are used to implement 8-bit quantization and 16-bit quantization of data, respectively;

[0013] The outer product matrix multiplication array is used to compute all types of matrix multiplication in the Swing Transformer, including Sampled Dense-Dense Matrix Multiplication (SDDMM), Sparse-Dense Matrix Multiplication (SpMM), and dense matrix multiplication.

[0014] The three nonlinear computation units are the Softmax unit for normalized exponential function, the GELU unit for Gaussian error linearity, and the LayerNorm unit for layer normalization. The Softmax unit is used to perform normalized exponential function computation, the GELU unit is used to perform Gaussian error linearity computation, and the LayerNorm unit is used to perform layer normalization computation.

[0015] The static random access memory (SRAM) comprises three parts: parameter SRAM, first input SRAM, and second input SRAM. The parameter SRAM is used to store the parameters required for calculation. The first input SRAM is used to store the input data read from external storage and 16-bit intermediate calculation results. The second input SRAM is used to store 8-bit intermediate calculation results.

[0016] In this context, the multi-head self-attention (MSA) portion and the multi-layer perceptron (MLP) portion in the parameter SRAM are used as ping-pong caches to reduce the latency of accessing external storage.

[0017] The accelerator processes data in a window-to-window order. Data loaded onto a window on the chip will have its result written back to external storage after the computation of an entire Swing Transformer block is completed.

[0018] When calculating multi-head self-attention MSA, the self-attention calculation in multi-head self-attention MSA is performed in the order of one head after another;

[0019] The register Reg includes a first register, a second register, a Softmax register, and a GELU register. The first register and the second register are used to store intermediate results of matrix multiplication calculations. The Softmax register is used to store input data for Softmax unit calculations and the calculation results of the Softmax unit. The GELU register is used to store input data for GELU unit calculations.

[0020] The Q matrix and the result matrix S of the Softmax unit in the self-attention calculation are stored in the first register, and the K matrix and V matrix in the self-attention calculation are written to the second register;

[0021] The computation of the Softmax unit is local and independent of other computations; the Softmax unit only exchanges data with the Softmax register.

[0022] After completing the multi-head self-attention MSA calculation for one window, the result is quantized to 16 bits and written back to the first input SRAM, awaiting subsequent residual connection calculation.

[0023] When computing matrix multiplication in a multilayer perceptron (MLP), the left and right inputs of the outer product matrix multiplication array come from the second input SRAM and the parameter SRAM, respectively.

[0024] In a multilayer perceptron (MLP), each part of the computation result generated by the first fully connected layer is quantized into 16 bits, processed by the GELU unit, and then written back to the second input SRAM.

[0025] The computation result of the second fully connected layer in the multilayer perceptron (MLP) is quantized by 16 bits and written into the first input SRAM. The residual is added to the MSA output data, thus completing the computation of one window and one Swing Transformer block.

[0026] External storage reads the calculation results of one window from the first input SRAM and then writes the data of the next window into the first input SRAM.

[0027] The following block-based strategy is adopted for matrix computation in Swin-Transformer:

[0028] The four sparsity modes of the attention matrix Attn in multi-head self-attention MSA computation are denoted as mask modes and represented by 2 bits: the highest bit represents the sparsity between computation blocks generated after block partitioning according to the block partitioning strategy. The highest bit being 0 indicates that the distribution of computation blocks within a window is dense, and the highest bit being 1 indicates that the distribution of computation blocks within a window is structured sparse; the lowest bit represents the sparsity within a computation block. The lowest bit being 0 indicates that the distribution of elements within each non-empty computation block is dense, and the lowest bit being 1 indicates that the distribution of elements within each non-empty computation block is sparse.

[0029] In calculating Attn = QK T A 7x7 block partitioning strategy is used.

[0030] The linear mapping of the input matrix, the linear mapping used to merge the different headers of the MSA output, and all matrix multiplications in the MLP are all performed in a 7×4 block format.

[0031] The outer product matrix multiplication array is configurable for performing sampled dense dense matrix multiplication (SDDMM) and sparse dense matrix multiplication (SpMM) operations caused by sparsity in moving window multi-head self-attention, and also for performing dense matrix multiplication computations.

[0032] The outer product matrix multiplication array includes 49×32 multiply-accumulate units (MACs), and every 7×4 multiply-accumulate units (MACs) constitute a processing unit (PE). Each processing unit (PE) is responsible for the multiplication calculation of one computation block. The sparsity between computation blocks is handled by the configuration between different processing units (PEs): the different configurations between processing units (PEs) are controlled by a 2-bit mask mode. The processing unit (PE) corresponding to each non-empty computation block is in the working state, and the processing unit (PE) corresponding to each completely empty computation block is not in the working state.

[0033] The multiply-accumulate (MAC) units are connected in a way that processes matrix outer products, that is, the left input data of the processing unit PE is broadcast row by row, and the right input data is broadcast column by column;

[0034] The processing unit PE includes three configurable computing modes: a configurable first computing mode, a configurable second computing mode, and a configurable third computing mode;

[0035] The configurable first calculation mode is used to generate an output matrix of size 7×4. The left input data of each row multiply-accumulate unit MAC unit in the processing unit PE comes from the corresponding row of the left input matrix, and the right input data of each column multiply-accumulate unit MAC unit in the processing unit PE comes from the corresponding column of the right input matrix.

[0036] The configurable second calculation mode is used to generate an output matrix of size 7×3. The left input data of each row of the multiply-accumulate unit MAC unit in the processing unit PE comes from the corresponding row of the left input matrix. The right input data of the first three columns of the multiply-accumulate unit MAC unit in the processing unit PE comes from the corresponding columns of the right input matrix. The right input data of the last column is 0.

[0037] The configurable third calculation mode is used to generate an output matrix of size 7×7. The left input data of each row of the multiply-accumulate unit MAC unit in the processing unit PE comes from the corresponding row of the left input matrix. The right input data of the top four rows of the multiply-accumulate unit MAC unit in the processing unit PE comes from the first four columns of the right input matrix. The right input data of the bottom three rows of the multiply-accumulate unit MAC unit in the processing unit PE comes from the last three columns of the right input matrix. The right input data of the bottom three rows of the last column of the multiply-accumulate unit MAC unit in the processing unit PE is 0.

[0038] When calculating Attn = QK TAt this time, one processing unit PE is responsible for a 7×7 computation block, where T represents the matrix transpose; if the data distribution within the computation block is dense, i.e., the least significant bit of the mask mode is 0, the processing unit PE operates sequentially in a configurable first computation mode and a configurable second computation mode, completing the computation of a 7×7 computation block in two stages; each column of the Q matrix serves as the left input of the processing unit PE, K T Each row of the matrix serves as the right input of the processing unit PE. If the data distribution within the computation block is sparse, i.e., the least significant bit of the mask pattern is 1, the processing unit PE operates in the configurable third computation mode to complete the SDDMM computation. The upper 4×4 MAC units in the processing unit PE are responsible for the computation of the upper left 4×4 elements in the computation block, and the lower 3×4 MAC units are responsible for the computation of the lower right 3×3 elements in the computation block. The non-zero elements in the original 7×7 computation block are retained and compressed into a 7×4 computation block.

[0039] When calculating O = SV, each processing unit PE is responsible for calculating a 7×4 computation block in the O matrix; if the computation block in the S matrix is ​​dense, i.e. the least significant bit of the mask mode is 0, the processing unit PE operates in the configurable first computation mode; the left and right inputs of the processing unit PE are each column of the S matrix and each row of the V matrix, respectively; when the least significant bit of the mask mode is 1, the processing unit PE operates in the configurable third computation mode to perform the SpMM calculation.

[0040] The generation of Q, K, and V matrices, as well as dense matrix multiplication in the MLP, can be directly deployed onto the outer product matrix multiplication array, at which point the processing unit PE operates in the configurable first computation mode.

[0041] The Softmax unit includes a comparator array, two or more exponentiation units (iexp), two or more accumulators, two shift registers, a lookup table (LUT), an exponentiation register, and two or more multipliers.

[0042] The Softmax unit calculates the normalized exponential function Softmax(x) of the i-th element in vector x using the following formula. i :

[0043]

[0044] Where exp is an exponential function with base e, and x is the vector for which the normalized exponential function needs to be calculated. i It is the i-th element in vector x, x max It is the largest element in vector x;

[0045] The calculation process of the Softmax unit includes:

[0046] Step 1: For each row of the Attn matrix, the data fed into the comparator array is divided into two or more pairs. The data in each pair is compared, and the largest data is retained to find x. max ;

[0047] In step 2, x i and x max It is fed into the exponentiary unit iexp to obtain exp(x) i -x max The calculation of the exponent iexp is performed according to formulas (2) and (3):

[0048]

[0049] L1(p) = 0.3585(p + 1.353) 2 +0.344 (3)

[0050] Where >> is a shift operation, L1(·) is a second-order polynomial approximated by the exponent e; non-positive real numbers Decomposed into z is a non-negative integer, and p is a real number in (-ln2, 0].

[0051] Transforming constant multiplication into addition and shifting further reduces hardware complexity;

[0052] Step 3: The output exp_out of the exponentiation calculation unit iexp is fed into the accumulator to obtain ∑exp(x j -x max ), and simultaneously stored in the e-index register;

[0053] Step 4: Use a lookup table (LUT) to perform the division calculation: the size of the lookup table (LUT) is 2. 14 ×8bit, a shift register is inserted between the parallel accumulator and the lookup table LUT, and the connection between the lookup table LUT and the subsequent multiplier is also inserted into the shift register;

[0054] Step 5: After completing a series of divisions, the exp_out retrieved from the exponent register is multiplied by the output of the lookup table LUT to obtain the final Softmax result.

[0055] In the GELU unit, the GELU function GELU(x) is defined as:

[0056]

[0057] Where erf(·) is the error function; the second-order polynomial L2(·) is used to estimate the erf(·) function:

[0058] L2(x)=sgn(x)[a(min(|x|,-b)+b 2 )+1] (5)

[0059] The GELU unit includes two constant multipliers, an error function approximation unit int_erf, a two-way selector, and an adder.

[0060] The error function approximation unit int_erf is used to calculate the second-order polynomial L2(·), in which constant multiplication is transformed into addition and shift, further reducing hardware complexity.

[0061] The LayerNorm unit calculates the layer normalized LayerNorm(x) of the i-th element in vector x using the following formula. i :

[0062]

[0063] The square root and division operations are combined into a lookup table (LUT) with a size of 2. 16 ×9bit; The method of inserting shift registers before and after the lookup table LUT is used to solve the problem of multiple parallel computing components accessing the lookup table LUT simultaneously in the LayerNorm unit.

[0064] Design pipelined interleaving-based data flows for the Softmax, GELU, and LayerNorm units respectively:

[0065] The pipelined interleaved data flow of the Softmax unit includes: during MSA computation, the outer product matrix multiplication array and the Softmax unit process the data of one head in one window at a time; the GH (GH=2 or 3) heads in one window are grouped together, and the computation order of matrix multiplication in each group is rearranged, that is, the generation of Q and K matrices and Attn=QK for the GH heads are completed first. T The calculation of generating V matrices for GH heads in a set and the calculation of O=SV are placed at the end of the calculation process; matrix multiplications that are not related to the Softmax data of the i-th head will be merged with the Softmax calculation of the i-th head.

[0066] The pipelined interleaved data flow of the GELU unit includes: when matrix multiplication in the MLP is deployed to the accelerator, the outer product matrix multiplication array generates a portion of the output matrix of the first fully connected layer and the second fully connected layer at one time, with the output matrix size being 49×32; the already generated portion of the first fully connected layer output matrix is ​​fed into the GELU unit to obtain the corresponding calculation result; at the same time, the outer product matrix multiplication array continues to calculate the next portion of the first fully connected layer output matrix;

[0067] The pipelined interleaved data flow of the LayerNorm unit includes: rearranging the computation order of the multi-head self-attention MSA and multilayer perceptron MLP for the two windows to cover the computation time of LayerNorm, that is, completing the multi-head self-attention MSA computation for the two windows first, and then performing the corresponding multilayer perceptron MLP computation; at the same time, the on-chip memory stores the data of the two windows simultaneously; after completing the computation of the entire Swing Transformer block for one window, the result is written back to external storage, and then the data of the third window is loaded from external storage;

[0068] The LayerNorm unit is fused with the output linear mapping OL in MSA or the second fully connected layer in MLP; the self-attention in MSA and the first fully connected layer in MLP are used to cover the time of data transfer between on-chip memory and off-chip memory;

[0069] The parallelism N of the nonlinear computing unit is determined by the pipelined interleaved data flow and is calculated using the following formula:

[0070]

[0071] Among them, #of cycles non-linear It is the number of computation cycles in which a nonlinear computational unit obtains a complete set of results. mat_mul It is the number of computational cycles for matrix multiplication, which is integrated with nonlinear computation.

[0072] This invention proposes a block-based strategy for all matrix operations in the Swing Transformer and designs a configurable reusable multiplication array based on this strategy. This array can be configured to implement various operations within the Swing Transformer. Secondly, the proposed pipelined interleaved data flow breaks down data dependencies, enabling parallel processing of matrix and nonlinear computations, thereby reducing network latency while using nonlinear units with lower parallelism. Simultaneously, this data flow reduces access to external storage and SRAM, resulting in lower power consumption.

[0073] Beneficial Effects: Currently, the Swing Transformer is widely used due to its excellent performance; however, its excessive computational load and parameter count make it difficult to deploy in resource-constrained edge scenarios. This invention addresses this problem by designing a dedicated hardware acceleration architecture for edge applications, leveraging the unique computational characteristics of the Swing Transformer. This architecture has been validated on an FPGA platform, demonstrating an energy efficiency 15.14 times higher than the NVIDIA RTX 2080 Ti GPU and 2.02 to 3.11 times higher than existing Transformer accelerators. Attached Figure Description

[0074] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0075] Figure 1 This is a calculation diagram of the Swing Transformer block.

[0076] Figure 2 This is a schematic diagram of the overall hardware architecture of the present invention.

[0077] Figure 3a This is a schematic diagram of the shift window in stage 3 of Swing-T.

[0078] Figure 3b This is a schematic diagram of the mask corresponding to each window.

[0079] Figure 3c This is a schematic diagram of the three data distribution patterns in each computation block.

[0080] Figure 4a This is a schematic diagram of the configurable first computing mode of PE.

[0081] Figure 4b This is a schematic diagram of the configurable second computing mode of PE.

[0082] Figure 4c This is a schematic diagram of the configurable third computing mode of PE.

[0083] Figure 5a This is a schematic diagram comparing the original calculation process of O=SV with the calculation process of this invention.

[0084] Figure 5b This is a schematic diagram of the hardware architecture of PE when calculating O=SV.

[0085] Figure 6 This is a schematic diagram of the overall architecture of the Softmax unit.

[0086] Figure 7 This is a schematic diagram of the hardware architecture of the GELU unit.

[0087] Figure 8a The data flow is directly implemented using the Softmax algorithm of the Swing Transformer.

[0088] Figure 8b Data flow directly implemented using GELU based on the Swing Transformer algorithm.

[0089] Figure 8cData flow directly implemented using LayerNorm based on the Swing Transformer algorithm.

[0090] Figure 9a It is a Softmax pipeline interleaved data stream.

[0091] Figure 9b It is a GELU pipeline interleaved data stream.

[0092] Figure 9c It is a LayerNorm pipeline interleaved data stream. Detailed Implementation

[0093] This invention provides a Swin-Transformer inference hardware accelerator, the overall hardware architecture of which is as follows: Figure 2 As shown, it includes a three-layer storage hierarchy consisting of external storage, static random-access memory (SRAM), and registers (Reg), two quantization units, an outer product matrix multiplication array, and three nonlinear computation units: a normalized exponential function Softmax unit, a Gaussian error linearization unit (GELU) unit, and a layer normalization unit (LayerNorm). It is worth noting that because multi-head self-attention (MSA) and multi-layer perception (MLP) are two relatively independent computational processes, the MSA and MLP portions of the parameter SRAM can be used as ping-pong caches to further reduce the latency of accessing external storage.

[0094] This invention processes data in a window-to-window sequence. Data loaded onto a window on the chip will have its result written back to external storage after the calculation of an entire Swing Transformer block is completed.

[0095] During MSA computation, self-attention calculations are performed sequentially, one after another. The Q matrix and the result matrix S of Softmax are stored in the first register, while the K and V matrices are written to the second register. This storage method, which stores the intermediate result matrices of self-attention computation in registers, reduces SRAM access and effectively lowers dynamic power consumption. The computation of the Softmax unit is local and independent of other computations; it only exchanges data with the Softmax register. After completing the MSA computation for one window, the result is quantized to 16 bits and written back to the first input SRAM, awaiting subsequent residual concatenation computation.

[0096] When calculating matrix multiplication in the MLP, the left and right inputs of the outer product matrix multiplication array come from the second input SRAM and the parameter SRAM, respectively. Each portion of the computation result generated by the first fully connected layer is quantized to 16 bits, processed by GELU, and then written back to the second input SRAM. The computation result of the second fully connected layer is quantized to 16 bits and written to the first input SRAM, where it is added to the residual with the output data of the MSA. External storage reads the computation result of one window from the first input SRAM and then writes the data of the next window into the first input SRAM.

[0097] Block partitioning strategy and matrix multiplication array

[0098] Moving window multi-head self-attention introduces sparsity by applying a mask to the attention matrix. Although hierarchical feature maps exist in the SwingTransformer, it can be observed that the sparsity pattern in the attention matrix is ​​static if the window size is fixed.

[0099] Taking phase 3 in Swin-T as an example, such as Figure 3a As shown. Figure 3b The corresponding mask applied to each window is shown. By dividing the attention matrix with the mask applied into multiple computational blocks, sparsity between and within computational blocks is discovered. For the sparsity between computational blocks, there are two distributions of computational blocks within each window. Figure 3b Window 0 and Window 1 in the diagram show a dense distribution, while windows 2 and Window 3 show a sparse distribution. The sparse distribution has a structured characteristic: non-empty computational blocks are concentrated in the top left 4×4 and bottom right 3×3 positions, and the values ​​in other computational blocks are all 0.

[0100] Regarding sparsity within a computation block, the data distribution within a computation block can be of three types: dense, sparse, and completely empty, such as... Figure 3cAs shown. Except for the completely empty case, the sparsity pattern within a computation block is similar to the sparsity pattern between computation blocks, but it is concentrated on computation blocks of size 7×7. Here, this sparsity pattern is called the hierarchical 4-3 sparsity pattern. It is worth mentioning that all attention matrices in different stages of the SwinTransformer can be described using the hierarchical 4-3 sparsity pattern.

[0101] Four sparsity patterns are denoted as mask patterns and represented by 2-bit numbers. The most significant bit represents the sparsity between computation blocks: 0 indicates that the distribution of computation blocks within a window is dense, and 1 indicates that the distribution is structured sparse. The least significant bit represents the sparsity within a computation block: 0 indicates that the distribution of elements within each non-empty computation block is dense, and 1 indicates that the distribution is sparse. The number of windows with different mask patterns varies with the stage, as shown in Table 1 below:

[0102] Table 1

[0103] Mask pattern (sparseness) Phase 1 Phase 2 Phase 3 Phase 4 00(0％) 49 9 1 1 01(48.98％) 7 3 1 0 10(48.98％) 7 3 1 0 11(73.97％) 1 1 1 0

[0104] In the original calculation process of the Swing Transformer, Attn = QK T The computation is redundant because some results are set to zero after applying the mask. By making good use of the sparsity in the masked attention matrix, the computation of generating the attention matrix can be transformed into Sampled Dense–Dense Matrix Multiplication (SDDMM). Simultaneously, the multiplication of the S and V matrices can be transformed into Sparse-dense Matrix Multiplication (SpMM). This transformation can reduce computation and intermediate storage. Except in the computation of Attn = QK T The system uses a 7×7 block strategy, while other matrix multiplications, such as linear mapping of the input matrix, linear mapping used to merge different headers of the MSA output, and all matrix multiplications in the MLP, use a 7×4 block strategy.

[0105] To better handle the hierarchical 4-3 sparse pattern and various matrix multiplications in the Swing Transformer, this invention proposes an outer product matrix multiplication array suitable for the block-based strategy presented herein. This outer product matrix multiplication array is configurable and can efficiently perform not only Sampled Dense Dense Matrix Multiplication (SDDMM) and Sparse Dense Matrix Multiplication (SpMM) operations caused by the sparsity in moving window multi-head self-attention, but also general dense matrix multiplication computations.

[0106] The outer product matrix multiplication array consists of 49×32 multiplication-and-accumulation (MAC) units, with each 7×4 MAC units forming a processing element (PE). Each PE is responsible for the multiplication calculation of one computational block, and the sparsity between computational blocks is handled through the configuration of different PEs. The MAC units in the outer product matrix multiplication array are connected in a manner that processes matrix outer products; that is, the left input data of a PE is broadcast row-wise, and its right input data is broadcast column-wise. The working process of the proposed outer product matrix multiplication array is illustrated using different matrix multiplication methods in the Swin Transformer as examples.

[0107] Figure 4a , Figure 4b , Figure 4c Three configurable computation modes for PE are demonstrated. Specifically, when calculating Attn = QK... T At this time, one PE is responsible for a 7×7 computation block. If the data distribution within the computation block is dense (the least significant bit of the mask mode is 0), the PE will sequentially operate in the first configurable computation mode (e.g., ...). Figure 4a (as shown) and the second calculation mode (such as) Figure 4b As shown), the computation of a 7×7 computation block is completed in two stages. Each column of the Q matrix serves as the left input of PE, and K... T Each row of the matrix serves as the right input to the PE. Conversely, if the data distribution within the computation block is sparse (the least significant bit of the mask mode is 1), the PE operates in a configurable third computation mode (e.g., ...). Figure 4c As shown), the SDDMM calculation is completed. The top 4×4 MAC units in the PE are responsible for calculating the top-left 4×4 elements in the calculation block; the bottom 3×4 MAC units are responsible for calculating the bottom-right 3×3 elements in the calculation block. The non-zero elements in the original 7×7 calculation block are retained and compressed into a 7×4 calculation block, as shown. Figure 5a The matrix S in the diagram is shown. This data storage method conforms to the logic of reading and writing data during the computation process, while also saving storage space.

[0108] When computing O = SV, each PE is responsible for computing a 7×4 block of computation within the O matrix. If the computational blocks in the S matrix are dense (the least significant bit of the mask mode is 0), the PE operates in the configurable first computation mode. The left and right inputs of the PE are each column of the S matrix and each row of the V matrix, respectively. When the least significant bit of the mask mode is 1, the PE operates in the configurable third computation mode to perform SpMM computation. Figure 5a , Figure 5b As shown, the connection between the data and the PE changes to match the computational characteristics of SpMM. Figure 5aThe left side shows the original calculation process, while the right side shows the calculation process of this invention. Figure 5b The corresponding hardware architecture is shown. The 4×4 MAC units at the top and the 3×4 MAC units at the bottom are each responsible for calculating a part of the O matrix, saving nearly half the computation time compared to direct calculation.

[0109] The generation of Q, K, and V matrices, as well as dense matrix multiplication computations in the MLP, can be directly deployed onto the outer product matrix multiplication array. In this case, the PE operates in a configurable first computation mode. In summary, this invention designs an outer product matrix multiplication array. This array can operate in different configurable computation modes based on a mask mode, performing SDDMM and SpMM operations with simple control logic and high hardware utilization.

[0110] Nonlinear operation unit

[0111] This invention simplifies complex nonlinear calculations by using approximate calculations and performs customized quantization for each nonlinear calculation in the Swing Transformer to achieve a balance between model accuracy and hardware complexity.

[0112] (1) Softmax unit

[0113] The formula for calculating Softmax is as follows:

[0114]

[0115] The overall architecture of the Softmax unit designed according to formula (1) is as follows: Figure 6 As shown, the calculation process can be divided into the following five steps. In step 1, for each row of the Attn matrix, the data fed into the comparator array is divided into multiple pairs. The data in each pair are compared, and the largest data is retained to quickly find x. max In step 2, x i and x max It is fed into the e-exponential calculation unit to obtain exp(x) i -x max According to the formulas in the reference "S. Kim, A. Gholami, Z. Yao, M.M. Mahoney, and K. Keutzer, "I-bert: Integer-only bert quantization," in International conference on machine learning, pp. 5506–5518, PMLR, 2021", as shown in formulas (2) and (3), the iexp unit was implemented. (Non-positive real numbers) It can be decomposed into Where z is a non-negative integer and p is a real number in (-ln2,0].

[0116]

[0117] L1(p) = 0.3585(p + 1.353) 2 +0.344 (3)

[0118] The >> symbol represents a shift operation.

[0119] This invention also adopts the algorithm intensity reduction strategy in the reference "M. Wang, S. Lu, D. Zhu, J. Lin, and Z. Wang, "A high-speed and low-complexity architecture for softmax function in deep learning," in 2018 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS), pp. 223–226, IEEE, 2018", transforming constant multiplication into addition and shifting, further reducing hardware complexity.

[0120] In step 3, the output exp_out of the iexp unit is fed into the accumulator to obtain ∑exp(x j -x max The result is simultaneously stored in the exponent register (e). In step 4, a lookup table (LUT) is used to perform the division calculation. The LUT has a size of 2. 14 The 8-bit value is determined by the results of precision experiments. Because the Softmax unit's computation involves parallel processing of multiple rows of data, the problem arises where multiple parallel computing components access the LUT simultaneously. This problem is solved by inserting a shift register between the parallel accumulator and the LUT. A similar approach is used for the connection between the LUT and subsequent multipliers. In step 5, after a series of divisions, the exp_out value retrieved from the exponent register is multiplied by the LUT's output to obtain the final Softmax result.

[0121] In addition, to adapt to the sparse pattern described above, Figure 6 The calculation component within the dashed box can adjust the number of calculation cycles based on the number of elements in each row of the Attn matrix to further improve computational efficiency.

[0122] (2) GELU unit

[0123] The GELU function is defined as follows:

[0124]

[0125] Where erf(·) is the error function. This invention uses the second-order polynomial L2(·) proposed in the reference “S. Kim, A. Gholami, Z. Yao, M.M. Mahoney, and K. Keutzer, “I-bert: Integer-only bert quantization,” in International conference on machine learning, pp. 5506–5518, PMLR, 2021” to estimate the erf(·) function. The expression for L2(·) is as follows:

[0126] L2(x)=sgn(x)[a(min(|x|,-b)+b 2 )+1] (5)

[0127] The specific hardware architecture of GELU is as follows: Figure 7 As shown. The optimization of constant multiplication is similar to the implementation in the iexp unit.

[0128] (3) LayerNorm unit

[0129] For LayerNorm, the following formula is used:

[0130]

[0131] The square root and division operations are combined into a single LUT of size 2. 16 ×9bit. The method for handling multiple parallel computing components accessing the LUT simultaneously in the LayerNorm cell is similar to step 4 in the Softmax cell computation.

[0132] Pipeline interwoven data stream

[0133] Data streams directly implemented based on the Swing Transformer algorithm, such as... Figure 8a , Figure 8b , Figure 8c As shown, the computation time of the entire network is the sum of the matrix multiplication computation time and the nonlinear computation time. Furthermore, the nonlinear computation time (the gray area in the diagram) is related to the parallelism of the nonlinear units. Reducing the nonlinear computation time requires achieving high parallelism at the cost of very high hardware costs, which is extremely expensive. However, by rearranging the computation order, merging matrix multiplication and nonlinear computation, and allocating sufficient time for nonlinear computation, the aforementioned problem can be solved.

[0134] Considering the varying data dependencies between matrix multiplication and different nonlinear computations in the Swing Transformer, pipelined interleaving-based dataflows were designed for Softmax, GELU, and LayerNorm. The proposed dataflows can cover the nonlinear computation time with matrix multiplication computation time without increasing hardware costs, thereby reducing overall computational latency.

[0135] Figure 9a This demonstrates a pipelined interleaved data flow for Softmax. During MSA computation, the outer product matrix multiplication array and the Softmax unit process data from one head in a window at a time. The GH heads in a window are grouped together, and the order of matrix multiplications within each group is rearranged. The Softmax unit takes an Attn matrix as input, and its output matrix S is used as input for calculating O = SV. Therefore, the generation of the V matrix and the calculation of O = SV for the GH heads in a group are placed at the end of the computation. Matrix multiplications unrelated to the Softmax data of the i-th head are merged with the Softmax calculation of the i-th head. Based on analysis of the SwingTransformer, the parameter GH = 3 is used for Swing-T, Swing-S, and Swing-L, and the parameter GH = 2 is used for Swing-B.

[0136] For GELU data streams, such as Figure 9b As shown. When matrix multiplication in MLP is deployed on the accelerator of this invention, the outer product matrix multiplication array generates a portion of the output matrices of the first and second fully connected layers at a time, with a size of 49×32. Because GELU computation is element-wise, there is no data dependency in the row and column directions of the matrix. The already generated portion of the first fully connected layer output matrix is ​​fed into the GELU unit to obtain the corresponding computation result. At the same time, the outer product matrix multiplication array continues to compute the next portion of the first fully connected layer output matrix.

[0137] Reorder the computation order of MSA and MLP in the two windows to cover the computation time of LayerNorm, such as Figure 9c As shown. At any given moment, on-chip memory stores data from two windows simultaneously. After completing the computation of the entire Swing TransformerBlock for one window, the result is written back to external storage, and then data for the third window is loaded from external storage. LayerNorm is fused with the Output Linear (OL) in MSA or the Fully Connected Layer 2 (FC2) in MLP. Self-Attention (SA) in MSA and the first fully connected layer in MLP are used to cover the time required for data transfer between on-chip and off-chip storage.

[0138] The nonlinear units in this invention are plug-and-play. Specifically, the nonlinear units in this invention can be replaced with other nonlinear unit designs whose computation time can be covered by the corresponding matrix multiplication computation time. This makes the hardware architecture of this invention more flexible. Furthermore, the matrix multiplication computation integrated with nonlinear computation has a sufficiently long computation time, meaning that the design of this invention does not require the nonlinear computation units to have high parallelism.

[0139] The parallelism of the nonlinear unit is determined by the pipelined interleaved data flow and can be calculated using the following formula:

[0140]

[0141] Among them, #of cycles non-linear It is the number of computation cycles in which the nonlinear unit obtains a complete set of results, #ofcycles mat_mul This is the number of computation cycles for matrix multiplication, which is integrated with nonlinear computation. The minimum values ​​for parallelism (NS, NG, and NL) for Softmax, GELU, and LayerNorm are 25, 16, and 49, respectively. As long as the chosen parallelism is greater than or equal to these minimum values, the latency of nonlinear computation can be fully covered.

[0142] Example

[0143] This embodiment implements the proposed hardware accelerator on the Xilinx Alveo U50 development board, and evaluates the accelerator's performance on the ImageNet dataset using the Swin-T version of the Swin Transformer. The resource consumption and power consumption are the results after placement and routing using Vivado 2020.2, with a clock frequency of 200MHz.

[0144] In this invention, 7×8 physical exponents (PEs) are used, each PE contains 7×4 MAC units, and the computational precision is 8-bit fixed-point. The parallelism of the nonlinear units is selected.<NS,NG,NL> =<25,49,49>. For the GELU implementation, we use a value larger than the minimum parallelism mentioned above to simplify the control logic.

[0145] The accuracy experiment is based on 8-bit quantized FQ-ViT[4] and uses the implementation method adopted in this invention to replace the nonlinear calculations. The Top-1 and Top-5 accuracies of the overall model after introducing three nonlinear approximations with different quantization bit widths are shown in Table 2 below (Top-1 accuracy loss represents the Top-1 accuracy loss relative to 8-bit FQ-ViT, "+" indicates an increase in accuracy relative to FQ-ViT, and "-" indicates a decrease in accuracy). For the two calculations Softmax and LayerNorm containing LUT, the size of LUT is determined through experiments.

[0146] Table 2

[0147]

[0148] Experimental results show that the accuracy loss caused by the approximation calculations of the three nonlinear functions is negligible. It is noteworthy that when the approximation methods from the reference “S. Kim, A. Gholami, Z. Yao, M.M. Mahoney, and K. Keutzer, “I-bert: Integer-only bert quantization,” in International conference on machine learning, pp. 5506–5518, PMLR, 2021” are used to replace Softmax and GELU respectively, in some cases, the overall model accuracy even surpasses that of the original FQ-ViT. This suggests that the nonlinear function approximation from the reference “S. Kim, A. Gholami, Z. Yao, M.M. Mahoney, and K. Keutzer, “I-bert: Integer-only bert quantization,” in International conference on machine learning, pp. 5506–5518, PMLR, 2021” may be more suitable for the inference of the Swin Transformer.

[0149] The bolded results in the table above represent the accuracy configuration of the nonlinear units ultimately selected in this embodiment. With an appropriate quantization bit width, the accuracy loss of the model due to the three nonlinear approximations relative to the original FQ-ViT is negligible: Top-1 accuracy loss is 0.148, and Top-5 accuracy loss is 0.138.

[0150] This embodiment selects an NVIDIA RTX 2080Ti GPU with 4352 CUDA cores, 544 Tensor cores, and a base clock frequency of 1350MHz as the GPU benchmark for comparison. Simultaneously, it uses `torch.cuda.Event()` to measure processing latency and Thermal Design Power (TDP) to estimate its power consumption. Compared to the GPU benchmark, this invention achieves a 15.14x improvement in energy efficiency.

[0151] The performance comparison of this invention with other existing Transformer accelerators is shown in Table 3 below:

[0152] Table 3

[0153]

[0154] In Table 3, Document 1 is "B.Li, S.Pandey, H.Fang, Y.Lyv, J.Li, J.Chen, M.Xie, L.Wan, H.Liu, and C.Ding, "Ftrans: energy-efficient acceleration of transformers using fpga," in Proceedings of the ACM / IEEE International Symposium on Low Power Electronics and Design,pp.175–180,2020.”

[0155] Document 2 is "P.Qi, EH-M.Sha, Q.Zhuge, H.Peng, S.Huang, Z.Kong, Y.Song, andB.Li, "Accelerating framework of transformer by hardware design and modelcompression co-optimization," in 2021IEEE / ACM International Conference OnComputer Aided Design (ICCAD), pp.1–9, IEEE, 2021."

[0156] Document 3 is "H.Khan, A.Khan, Z.Khan, LBHuang, K.Wang, and L.He, "Npe: an fpga-based overlay processor for natural language processing," arXiv preprintarXiv:2104.06535, 2021."

[0157] Document 4 is "T.Wang, L.Gong, C.Wang, Y.Yang, Y.Gao,

[0158] As can be seen from the table above, compared with other existing Transformer accelerators, the present invention achieves a throughput improvement of 1.27 to 6.23 times per DSP and a power efficiency improvement of 2.02 to 3.11 times.

[0159] This invention provides a Swin-Transformer inference hardware accelerator. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A Swing-Transformer inference hardware accelerator, characterized in that, It includes a three-layer storage hierarchy consisting of external storage, static random access memory (SRAM) and register Reg, two quantization units, an outer product matrix multiplication array and three nonlinear computation units; The external storage is used to store off-chip data; The static random access memory (SRAM) is used to store on-chip data; The register Reg is used to store intermediate calculation result data; The two quantization units are used to implement 8-bit quantization and 16-bit quantization of data, respectively; The outer product matrix multiplication array is used to compute all types of matrix multiplication in the Swing Transformer, including Sampled Dense Dense Matrix Multiplication (SDDMM), Sparse Dense Matrix Multiplication (SpMM), and Dense Matrix Multiplication. The three nonlinear computation units are the Softmax unit for normalized exponential function, the GELU unit for Gaussian error linearity, and the LayerNorm unit for layer normalization. The Softmax unit is used to perform normalized exponential function computation, the GELU unit is used to perform Gaussian error linearity computation, and the LayerNorm unit is used to perform layer normalization computation. The static random access memory (SRAM) comprises three parts: parameter SRAM, first input SRAM, and second input SRAM. The parameter SRAM is used to store the parameters required for calculation. The first input SRAM is used to store the input data read from external storage and 16-bit intermediate calculation results. The second input SRAM is used to store 8-bit intermediate calculation results. In this context, the multi-head self-attention (MSA) portion and the multilayer perceptron (MLP) portion in the parameter SRAM are used as ping-pong caches to reduce the latency of accessing external storage. Design pipelined interleaving-based data flows for the Softmax, GELU, and LayerNorm units respectively: The pipelined interleaved data flow of the Softmax unit includes: during MSA computation, the outer product matrix multiplication array and the Softmax unit process the data of one head in one window at a time; dividing the GH heads in one window into a group and rearranging the computation order of matrix multiplication in each group, that is, first completing the generation of Q and K matrices and Attn = QK for the GH heads. T The calculation of generating V matrices for GH heads in a set and the calculation of O=SV are placed at the end of the calculation process; matrix multiplications that are not related to the Softmax data of the i-th head will be merged with the Softmax calculation of the i-th head. The pipelined interleaved data flow of the GELU unit includes: when matrix multiplication in the MLP is deployed to the accelerator, the outer product matrix multiplication array generates a portion of the output matrix of the first fully connected layer and the second fully connected layer at one time, with the output matrix size being 49×32; the already generated portion of the first fully connected layer output matrix is ​​fed into the GELU unit to obtain the corresponding calculation result; at the same time, the outer product matrix multiplication array continues to calculate the next portion of the first fully connected layer output matrix; The pipelined interleaved data flow of the LayerNorm unit includes: rearranging the computation order of the multi-head self-attention MSA and multilayer perceptron MLP for the two windows to cover the computation time of LayerNorm, that is, completing the multi-head self-attention MSA computation for the two windows first, and then performing the corresponding multilayer perceptron MLP computation; at the same time, the on-chip memory stores the data of the two windows simultaneously; after completing the computation of the entire Swing Transformer block for one window, the result is written back to external storage, and then the data of the third window is loaded from external storage; The LayerNorm unit is fused with the output linear mapping OL in MSA or the second fully connected layer in MLP; the self-attention in MSA and the first fully connected layer in MLP are used to cover the time of data transfer between on-chip and off-chip memory; The parallelism N of the nonlinear computing unit is determined by the pipelined interleaved data flow and is calculated using the following formula: Among them, #of cycles non-linear #ofcycles is the number of computation cycles required for a nonlinear computational unit to obtain a complete set of results. mat_mul It is the number of computational cycles for matrix multiplication, which is integrated with nonlinear computation.

2. The Swing-Transformer inference hardware accelerator according to claim 1, characterized in that, The accelerator processes data in a window-to-window order. Data loaded onto a window on the chip will have its result written back to external storage after the computation of an entire Swing Transformer block is completed.

3. The Swing-Transformer inference hardware accelerator according to claim 2, characterized in that, When calculating multi-head self-attention MSA, the self-attention calculation in multi-head self-attention MSA is performed in the order of one head after another; The register Reg includes a first register, a second register, a Softmax register, and a GELU register. The first register and the second register are used to store intermediate results of matrix multiplication calculations. The Softmax register is used to store input data for Softmax unit calculations and the calculation results of the Softmax unit. The GELU register is used to store input data for GELU unit calculations. The Q matrix and the result matrix S of the Softmax unit in the self-attention calculation are stored in the first register, and the K matrix and V matrix in the self-attention calculation are written to the second register; The computation of the Softmax unit is local and independent of other computations; the Softmax unit only exchanges data with the Softmax register. After completing the multi-head self-attention MSA calculation for one window, the result is quantized to 16 bits and written back to the first input SRAM, awaiting subsequent residual connection calculation.

4. A Swing-Transformer inference hardware accelerator according to claim 3, characterized in that, When computing matrix multiplication in a multilayer perceptron (MLP), the left and right inputs of the outer product matrix multiplication array come from the second input SRAM and the parameter SRAM, respectively. In a multilayer perceptron (MLP), each part of the computation result generated by the first fully connected layer is quantized into 16 bits, processed by the GELU unit, and then written back to the second input SRAM. The computation result of the second fully connected layer in the multilayer perceptron (MLP) is quantized by 16 bits and written into the first input SRAM. The residual is added to the MSA output data, thus completing the computation of one window and one Swing Transformer block. External storage reads the calculation results of one window from the first input SRAM and then writes the data of the next window into the first input SRAM.

5. A Swing-Transformer inference hardware accelerator according to claim 4, characterized in that, The following block-based strategy is adopted for matrix computation in Swin-Transformer: The four sparsity modes of the attention matrix Attn in multi-head self-attention MSA computation are denoted as mask modes and represented by 2 bits: the highest bit represents the sparsity between computation blocks generated after block partitioning according to the block partitioning strategy. The highest bit being 0 indicates that the distribution of computation blocks within a window is dense, and the highest bit being 1 indicates that the distribution of computation blocks within a window is structured sparse; the lowest bit represents the sparsity within a computation block. The lowest bit being 0 indicates that the distribution of elements within each non-empty computation block is dense, and the lowest bit being 1 indicates that the distribution of elements within each non-empty computation block is sparse. In calculating Attn = QK T A 7×7 block partitioning strategy is adopted at this time; The linear mapping of the input matrix, the linear mapping used to merge the different headers of the MSA output, and all matrix multiplications in the MLP are all performed in a 7×4 block format. The outer product matrix multiplication array is configurable for performing sampled dense dense matrix multiplication (SDDMM) and sparse dense matrix multiplication (SpMM) operations caused by sparsity in moving window multi-head self-attention, and also for performing dense matrix multiplication computations. The outer product matrix multiplication array includes 49×32 multiply-accumulate units (MACs), and every 7×4 multiply-accumulate units (MACs) constitute a processing unit (PE). Each processing unit (PE) is responsible for the multiplication calculation of one computation block. The sparsity between computation blocks is handled by the configuration between different processing units (PEs): the different configurations between processing units (PEs) are controlled by a 2-bit mask mode. The processing unit (PE) corresponding to each non-empty computation block is in the working state, and the processing unit (PE) corresponding to each completely empty computation block is not in the working state. The multiply-accumulate (MAC) units are connected in a way that processes matrix outer products, that is, the left input data of the processing unit PE is broadcast row by row, and the right input data is broadcast column by column; The processing unit PE includes three configurable computing modes: a configurable first computing mode, a configurable second computing mode, and a configurable third computing mode; The configurable first calculation mode is used to generate an output matrix of size 7×4. The left input data of each row multiply-accumulate unit MAC unit in the processing unit PE comes from the corresponding row of the left input matrix, and the right input data of each column multiply-accumulate unit MAC unit in the processing unit PE comes from the corresponding column of the right input matrix. The configurable second calculation mode is used to generate an output matrix of size 7×3. The left input data of each row of the multiply-accumulate unit MAC unit in the processing unit PE comes from the corresponding row of the left input matrix. The right input data of the first three columns of the multiply-accumulate unit MAC unit in the processing unit PE comes from the corresponding columns of the right input matrix. The right input data of the last column is 0. The configurable third calculation mode is used to generate an output matrix of size 7×7. The left input data of each row of the multiply-accumulate unit MAC unit in the processing unit PE comes from the corresponding row of the left input matrix. The right input data of the top four rows of the multiply-accumulate unit MAC unit in the processing unit PE comes from the first four columns of the right input matrix. The right input data of the bottom three rows of the multiply-accumulate unit MAC unit in the processing unit PE comes from the last three columns of the right input matrix. The right input data of the bottom three rows of the last column of the multiply-accumulate unit MAC unit in the processing unit PE is 0. When calculating Attn = QK T At this time, one processing unit PE is responsible for a 7×7 computation block, where T represents the matrix transpose; if the data distribution within the computation block is dense, i.e., the least significant bit of the mask mode is 0, the processing unit PE operates sequentially in a configurable first computation mode and a configurable second computation mode, completing the computation of a 7×7 computation block in two stages; each column of the Q matrix serves as the left input of the processing unit PE, K T Each row of the matrix serves as the right input of the processing unit PE. If the data distribution within the computation block is sparse, i.e., the least significant bit of the mask pattern is 1, the processing unit PE operates in the configurable third computation mode to complete the SDDMM computation. The upper 4×4 MAC units in the processing unit PE are responsible for the computation of the upper left 4×4 elements in the computation block, and the lower 3×4 MAC units are responsible for the computation of the lower right 3×3 elements in the computation block. The non-zero elements in the original 7×7 computation block are retained and compressed into a 7×4 computation block. When calculating O = SV, each processing unit PE is responsible for calculating a 7×4 computation block in the O matrix; if the computation block in the S matrix is ​​dense, i.e. the least significant bit of the mask mode is 0, the processing unit PE operates in the configurable first computation mode; the left and right inputs of the processing unit PE are each column of the S matrix and each row of the V matrix, respectively; when the least significant bit of the mask mode is 1, the processing unit PE operates in the configurable third computation mode to perform the SpMM calculation. The generation of Q, K, and V matrices, as well as dense matrix multiplication in the MLP, can be directly deployed onto the outer product matrix multiplication array, at which point the processing unit PE operates in the configurable first computation mode.

6. A Swing-Transformer inference hardware accelerator according to claim 5, characterized in that, The Softmax unit includes a comparator array, two or more exponentiation calculation units (iexp), two or more accumulators, two shift registers, a division lookup table (LUT), an exponentiation register, and two or more multipliers. The Softmax unit calculates the normalized exponential function Softmax(x) of the i-th element in vector x using the following formula. i : Where exp is an exponential function with base e, and x is the vector for which the normalized exponential function needs to be calculated. i It is the i-th element in vector x, x max It is the largest element in vector x; The calculation process of the Softmax unit includes: Step 1: For each row of the Attn matrix, the data fed into the comparator array is divided into two or more pairs. The data in each pair is compared, and the largest data is retained to find x. max ; In step 2, x i and x max It is fed into the exponentiary unit iexp to obtain exp(x) i -x max The calculation of the exponent iexp is performed according to formulas (2) and (3): L1(p)=0.3585(p+1.353) 2 +0.344 (3) Where >> is a shift operation, L1(·) is a second-order polynomial approximated by the exponent e; non-positive real numbers Decomposed into z is a non-negative integer, and p is a real number in (-ln2, 0]. Transforming constant multiplication into addition and shifting further reduces hardware complexity; Step 3: The output exp_out of the exponentiation calculation unit iexp is fed into the accumulator to obtain ∑exp(x j -x max ), and simultaneously stored in the e-index register; Step 4: Use a lookup table (LUT) to perform the division calculation: the size of the lookup table (LUT) is 2. 14 ×8bit, a shift register is inserted between the parallel accumulator and the lookup table LUT, and the connection between the lookup table LUT and the subsequent multiplier is also inserted into the shift register; Step 5: After completing a series of divisions, the exp_out retrieved from the exponent register is multiplied by the output of the lookup table LUT to obtain the final Softmax result.

7. A Swing-Transformer inference hardware accelerator according to claim 6, characterized in that, In the GELU unit, the GELU function GELU(x) is defined as: Where erf(·) is the error function; the second-order polynomial L2(·) is used to estimate the erf(·) function: L2(x)=sgn(x)[a(min(|x|, -b)+b 2 )+1] (5) The GELU unit includes two constant multipliers, an error function approximation unit int_erf, a two-way selector, and an adder. The error function approximation unit int_erf is used to calculate the second-order polynomial L2(·), in which constant multiplication is transformed into addition and shift, further reducing hardware complexity.

8. A Swing-Transformer inference hardware accelerator according to claim 7, characterized in that, The LayerNorm unit calculates the layer normalized LayerNorm(x) of the i-th element in vector x using the following formula. i : The square root and division operations are combined into a lookup table (LUT) with a size of 2. 16 ×9bit; The method of inserting shift registers before and after the lookup table LUT is used to solve the problem of multiple parallel computing components accessing the lookup table LUT simultaneously in the LayerNorm unit.

Images