Data processing methods and accelerators based on fusion attention and quantization operations

Abstract

This invention provides a data processing method and accelerator based on fusion attention and quantization operations, applied in the field of computer technology. It includes: multiple computing cores based on a heterogeneous multi-core architecture in the accelerator, performing attention processing operations on multiple input data according to a target model to obtain output results; a normalization core for normalizing the input data to obtain normalized input data; a linear layer core for performing linear calculations on the normalized input data to obtain target input data; and a fusion attention core for performing exponential calculation operations on the target input data based on a shift exponential approximation algorithm using a shift exponential approximation unit to obtain exponential calculation results. The processing time periods corresponding to multiple correlated exponential calculation results partially overlap, and these multiple correlated exponential calculation results are multiple exponential calculation results output by the same fusion attention core. The processing time period represents the duration of converting input data into exponential calculation results.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a data processing method and accelerator based on fusion attention and quantization operations. Background Technology

[0002] While the Transformer model has made groundbreaking progress in fields such as natural language processing and computer vision, it has also brought more severe performance and energy efficiency challenges to computing devices.

[0003] The model quantization methods in related technologies are not suitable for nonlinear calculations such as Softmax in Transformer models. Furthermore, when mapping Transformer models to computational cores in the construction of hardware accelerators, the cyclic data dependencies are only applicable to linear calculations and not to Transformer models, resulting in low efficiency of accelerators when processing nonlinear calculations. Summary of the Invention

[0004] In view of the above problems, the present invention provides a data processing method and accelerator based on the fusion of attention and quantization operations.

[0005] According to a first aspect of the present invention, a data processing method based on a quantization factor is provided, comprising: performing attention processing operations on multiple input data according to a target model based on multiple computing cores of a heterogeneous multi-core architecture in an accelerator, thereby obtaining an output result; wherein, the computing cores include a normalization core, a linear layer core, and a fusion attention core; the normalization core is used to normalize the input data to obtain normalized input data; the linear layer core is used to perform linear calculations on the normalized input data to obtain target input data; the fusion attention core is used to perform exponential calculation operations on the target input data based on a shift exponential approximation algorithm of a shift exponential approximation unit to obtain an exponential calculation result; the output result is determined based on the exponential calculation result; the processing time periods corresponding to the multiple associated exponential calculation results partially overlap; the multiple associated exponential calculation results are multiple exponential calculation results output by the same fusion attention core; the processing time period represents the duration of converting input data into exponential calculation results; and the shift exponential approximation unit is obtained by updating the exponential calculation unit of the initial model based on a quantization factor.

[0006] According to an embodiment of the present invention, a shift exponential approximation algorithm based on a shift exponential approximation unit performs an exponential calculation operation on target input data, including: using a fusion attention core to perform the following operations: performing shift calculations on the input integer value and the input quantization factor based on the shift exponential approximation unit to obtain the output integer value and the output quantization factor; performing logarithmic transformations on the output integer value and the output quantization factor respectively to obtain the exponential calculation result.

[0007] According to an embodiment of the present invention, the shift calculation of the input integer value and the input quantization factor using the shift exponent approximation unit to obtain the output integer value and the output quantization factor includes: determining the shift value of the input integer value based on the quantization value of the input quantization factor; shifting the input integer value based on the shift value and a specified integer to obtain the output integer value; and determining the output quantization factor based on the sum between the specified integer and the quantization value of the input quantization factor.

[0008] According to an embodiment of the present invention, the input data includes a query matrix, a key matrix, and a value matrix; wherein, the above method further includes: multiplying the query matrix and the key matrix to obtain a length matrix, wherein the length matrix is ​​used as a first input integer value, and the maximum value of each row of the length matrix is ​​used as a second input integer value; and the first input integer value and the second input integer value are used as input integer values.

[0009] According to an embodiment of the present invention, the exponent calculation result includes a first exponent calculation result and a second exponent calculation result, wherein the first exponent calculation result corresponds to a first input integer value and the second exponent calculation result corresponds to a second input integer value; based on multiple computing cores in an accelerator including a heterogeneous multi-core architecture, attention processing operations are performed on multiple input data according to the target model to obtain an output result, including: using multiple computing cores to perform the following operations: performing a multiplication operation on the first exponent calculation result to obtain a first multiplication operation result; performing a row summation operation on the second exponent calculation result based on a preset first specified quantization factor to obtain a row summation operation result; performing a multiplication operation on the second exponent calculation result and the value matrix based on a preset second specified quantization factor to obtain a second multiplication operation result; and determining the output result based on the first multiplication operation result, the row summation operation result, and the second multiplication operation result.

[0010] According to an embodiment of the present invention, the output result includes at least one of image recognition result, text generation result, and speech recognition result.

[0011] A second aspect of the present invention provides an accelerator, comprising: a normalization core for normalizing input data to obtain normalized input data; a linear layer core for performing linear calculations on the normalized input data to obtain target input data; and a fusion attention core for performing exponential calculation operations on the target input data based on a shift exponential approximation algorithm of a shift exponential approximation unit to obtain an exponential calculation result. The output result is determined based on the exponential calculation result. The processing time periods corresponding to multiple related exponential calculation results partially overlap. The multiple related exponential calculation results are multiple exponential calculation results output by the same fusion attention core. The processing time period represents the duration of converting input data into exponential calculation results. The shift exponential approximation unit is obtained by updating the exponential calculation unit of the initial model based on a quantization factor.

[0012] According to an embodiment of the present invention, the fusion attention core includes multiple computing units; wherein, the fusion attention core is used to split the target input data to obtain multiple data slices; the multiple computing units are used to process the multiple data slices, wherein the multiple data slices are input to the multiple computing units in a specified order at different specified time periods, and for any specified time period, each computing unit processes different data slices in parallel.

[0013] According to an embodiment of the present invention, the linear layer core includes an input buffer; wherein, a first-in-first-out (FIFO) buffer is provided between the input buffer and the input data stream corresponding to the input buffer, and the FIFO buffer is used to control the data interaction operation of the input buffer.

[0014] According to an embodiment of the present invention, the accelerator optimizes the performance parameters of the fusion attention core, linear layer core and normalization core of the accelerator based on the resource consumption data generated by the accelerator performing attention processing operations, so as to obtain an optimized accelerator.

[0015] According to an embodiment of the present invention, based on multiple computing cores of a heterogeneous multi-core architecture in an accelerator, attention processing operations are performed on multiple input data according to the target model to obtain output results; a normalization core is used to normalize the input data to obtain normalized input data; a linear layer core is used to perform linear calculations on the normalized input data to obtain target input data; a fusion attention core is used to perform exponential calculation operations on the target input data based on the shift exponential approximation algorithm of the shift exponential approximation unit to obtain exponential calculation results. The processing time periods corresponding to multiple related exponential calculation results overlap to some extent. Multiple related exponential calculation results are multiple exponential calculation results output by the same fusion attention core. The processing time period represents the time it takes for the input data to be converted into exponential calculation results. The shift exponential approximation unit is obtained by updating the exponential calculation unit of the initial model based on the quantization factor. By using a shift exponent approximation unit built based on quantization factors, the floating-point number output by the exponent calculation unit in the initial model is converted, transforming multiplication calculation into hardware-efficient shift calculation, thus achieving efficient data processing. Furthermore, the parallel and interleaved execution of data processing operations by multiple computing cores in the heterogeneous multi-core architecture avoids the problem of data dependencies with cycles, allowing the accelerator to overlap and execute different operations, thereby improving data processing efficiency. Attached Figure Description

[0016] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0017] Figure 1A flowchart of a data processing method based on fused attention and quantization operations according to an embodiment of the present invention is shown.

[0018] Figure 2 A schematic diagram of an initial module according to an embodiment of the present invention is shown.

[0019] Figure 3 A schematic diagram of the target module according to an embodiment of the present invention is shown.

[0020] Figure 4 A schematic diagram of an accelerator according to an embodiment of the present invention is shown.

[0021] Figure 5 A schematic diagram of an encoder according to an embodiment of the present invention is shown.

[0022] Figure 6 A schematic diagram of data processing for an accelerator according to an embodiment of the present invention is shown.

[0023] Figure 7 A schematic diagram of a fusion attention core according to an embodiment of the present invention is shown.

[0024] Figure 8 A schematic diagram of data sharding for a fused attention core according to an embodiment of the present invention is shown.

[0025] Figure 9 A schematic diagram of the fusion attention core processing according to an embodiment of the present invention is shown.

[0026] Figure 10 A schematic diagram of a linear layer core according to an embodiment of the present invention is shown.

[0027] Figure 11 A schematic diagram of accelerator construction according to an embodiment of the present invention is shown.

[0028] Figure 12 The diagram illustrates the accelerator throughput and computational efficiency according to an embodiment of the present invention.

[0029] Figure 13 A comparison chart of throughput according to an embodiment of the present invention is shown.

[0030] Figure 14 A comparison chart of energy efficiency according to an embodiment of the present invention is shown. Detailed Implementation

[0031] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0033] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0034] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0035] In related technologies, the optimization of Transformer models includes model quantization and data flow optimization methods.

[0036] Model quantization transforms the high-precision floating-point representation of a network model into a low-bit-width fixed-point representation, thereby compressing the model's storage space and utilizing efficient fixed-point computation units on hardware. While analog quantization compresses the model's weights and activations into low-bit-width data, it still requires dequantizing this low-bit-width data back to floating-point numbers for computation, thus not utilizing efficient fixed-point computation units. Pure integer quantization can directly perform fixed-point computation on the input low-bit-width fixed-point data without dequantization, but it requires operators to satisfy a homogeneity condition. Nonlinear computations such as Softmax in Transformers do not meet this condition.

[0037] Data flow optimization targets the characteristics of computing platforms, such as on-chip cache and bandwidth. It enhances computational locality and eliminates unnecessary intermediate variables through methods such as data sharding, adjusting loop order, and operator fusion, thereby improving computational efficiency and reducing storage overhead.

[0038] The quantization schemes of related technologies follow a matrix multiplication-Softmax-matrix multiplication computation order when quantizing attention computation, and quantize the three operators separately. Fine-grained attention operator fusion changes the computational flow of attention computation, employing online Softmax computation, making the quantization schemes of related technologies no longer applicable.

[0039] Related technologies also propose a semi-layer mapping method to address path dependency issues in visual models. On the hardware side, two heterogeneous processing units are designed based on this semi-layer mapping. However, this method does not solve the storage bottleneck of the attention mechanism, making it difficult to support longer input sequence lengths. Secondly, related technologies generally map attention computation and linear layer computation to the same computational core, failing to fully consider the differences in their computational characteristics. This results in low computational efficiency in some cases. The computational flow after the fusion of fine-grained attention operators exhibits cyclical data dependencies, requiring a reconsideration of pipeline partitioning and parallelization design.

[0040] In view of this, embodiments of the present invention provide a data processing method based on the fusion of attention and quantization operations, comprising: updating the exponential calculation unit of an initial model using a shift exponential approximation unit constructed based on a quantization factor to obtain a target model including a target module, wherein the target module includes a shift exponential approximation unit; performing attention processing operations on multiple input data based on multiple computing cores in an accelerator including a heterogeneous multi-core architecture, according to the target model, to obtain an output result, wherein the computing cores are used to perform exponential calculation operations based on the shift exponential approximation algorithm of the shift exponential approximation unit to obtain an exponential calculation result, and the output result is determined based on the exponential calculation result.

[0041] The following will be through Figures 1-3 The data processing method based on quantization factors according to the disclosed embodiments will be described in detail.

[0042] Figure 1 A flowchart of a data processing method based on a quantization factor according to an embodiment of the present invention is shown.

[0043] like Figure 1 As shown, the data processing method based on quantization factors in this embodiment includes operation S110.

[0044] When operating the S110, based on multiple computing cores of the heterogeneous multi-core architecture in the accelerator, attention processing operations are performed on multiple input data according to the target model to obtain the output results.

[0045] The computational core comprises a normalization core, a linear layer core, and a fusion attention core. The normalization core normalizes the input data to obtain normalized input data. The linear layer core performs linear calculations on the normalized input data to obtain the target input data. The fusion attention core performs exponential calculations on the target input data based on the shift exponential approximation algorithm of the shift exponential approximation unit to obtain the exponential calculation result. The output result is determined based on the exponential calculation result. The processing time periods corresponding to multiple related exponential calculation results overlap to some extent. Multiple related exponential calculation results are multiple exponential calculation results output by the same fusion attention core. The processing time period represents the duration of converting input data into exponential calculation results. The shift exponential approximation unit is obtained by updating the exponential calculation unit of the initial model based on the quantization factor.

[0046] According to an embodiment of the present invention, the initial model includes an initial module for performing fusion attention calculation. The calculation method of the initial module is shown in formulas (1) to (6), which are the main iterative formulas for fine-grained attention fusion calculation.

[0047] Where j is the inner loop iteration variable and i is the outer loop iteration variable. It is a partition of matrix Q. Size is Q is of size , It is a matrix A shard, Size is , Xiaowei Here, N and d represent the length of the input sequence of the module and the dimension of the scaling dot product attention, respectively, while Br and Bc represent the size of the fusion slice parameters in the row and column dimensions, respectively.

[0048]

[0049] Where j is the inner loop iteration variable and i is the outer loop iteration variable. It is a query matrix. It is a key matrix. It is a value matrix. It is a length matrix. yes transpose, It is the j-th largest row vector. It is the jth The largest row vector in the array. For attention score, The attention score matrix, It is the exponent of the (j-1)th largest row vector and the jth largest row vector. It is the jth And the exponent of the j-th largest row vector, Indicates the summation of rows. Represents a diagonal matrix, where the exponent calculation unit of the initial module is used to obtain... .

[0050] According to an embodiment of the present invention, the quantization factor is a hyperparameter used to map floating-point weights / activation values ​​to integers.

[0051] According to an embodiment of the present invention, the shift exponential approximation unit constructed based on the quantization factor combines the power of 2 quantization factor, Log2 quantization, and shift exponential approximation method, which can realize the quantization of fine-grained attention fusion calculation to obtain the target module, and then obtain the target model based on the target module.

[0052] According to an embodiment of the present invention, since the target module is quantized compared to the initial module, the target module can convert most of the multiplication calculations in attention calculation into efficient shift calculations and realize the inference process of full integer calculation.

[0053] According to an embodiment of the present invention, the obtained target model is mapped to an accelerator, the heterogeneous multi-core architecture of which includes multiple computing cores, and the computing cores are capable of performing attention processing operations on different input data of the target model simultaneously.

[0054] It should be noted that the computing core can be a component in any type of computing unit used to perform computing functions, such as a central processing unit (CPU) or a graphics processing unit (GPU). The embodiments of the present invention do not limit the specific device type of the computing unit.

[0055] According to an embodiment of the present invention, the normalization core can normalize the input data, the linear layer core can perform linear calculations on the input data, and the fusion attention core performs exponential calculations on the input data based on the target module to output the exponential calculation results.

[0056] Furthermore, the aforementioned normalization core, linear layer core, and fusion attention core can be configured in a preset linear order, first normalizing the input data, then performing linear calculations, and finally performing exponential calculations. Alternatively, the order can be adjusted according to actual needs without limitation.

[0057] The processing time periods corresponding to the calculation results of multiple correlation indices overlap to some extent. For example, different input data are input at different times. The normalization core, linear layer core and fusion attention core mentioned above can process different data in the same time period, so that the normalization core, linear layer core and fusion attention core are all in working state.

[0058] According to an embodiment of the present invention, the output result includes at least one of image recognition result, text generation result, and speech recognition result.

[0059] According to an embodiment of the present invention, based on multiple computing cores of a heterogeneous multi-core architecture in an accelerator, attention processing operations are performed on multiple input data according to the target model to obtain output results; a normalization core is used to normalize the input data to obtain normalized input data; a linear layer core is used to perform linear calculations on the normalized input data to obtain target input data; a fusion attention core is used to perform exponential calculation operations on the target input data based on the shift exponential approximation algorithm of the shift exponential approximation unit to obtain exponential calculation results. The processing time periods corresponding to multiple related exponential calculation results overlap to some extent. Multiple related exponential calculation results are multiple exponential calculation results output by the same fusion attention core. The processing time period represents the time it takes for the input data to be converted into exponential calculation results. The shift exponential approximation unit is obtained by updating the exponential calculation unit of the initial model based on the quantization factor. By using a shift exponent approximation unit built based on quantization factors, the floating-point number output by the exponent calculation unit in the initial model is converted, transforming multiplication calculation into hardware-efficient shift calculation, thus achieving efficient data processing. Furthermore, the parallel and interleaved execution of data processing operations by multiple computing cores in the heterogeneous multi-core architecture avoids the problem of data dependencies with cycles, allowing the accelerator to overlap and execute different operations, thereby improving data processing efficiency.

[0060] According to an embodiment of the present invention, a shift exponential approximation algorithm based on a shift exponential approximation unit performs an exponential calculation operation on target input data, including: using the fusion attention core to perform the following operations: performing shift calculation on the input integer value and the input quantization factor based on the shift exponential approximation unit to obtain the output integer value and the output quantization factor; performing logarithmic transformation on the output integer value and the output quantization factor respectively to obtain the exponential calculation result.

[0061] According to an embodiment of the present invention, the input integer value and the input quantization factor are shifted to obtain the output integer value and the output quantization factor, wherein the input quantization factor and the output quantization factor are both Power-of-Two Scaling Factors (PoT) with a constraint of 2.

[0062] According to an embodiment of the present invention, the output integer value and the output quantization factor are quantized using Log2, as shown in formula (7).

[0063]

[0064] in, and These represent the output integer value and the output quantization factor, respectively. It is a 32-bit integer value. Let L be the power of the quantization factor of matrix L. This is the result of the exponent calculation.

[0065] According to an embodiment of the present invention, since The exponent calculated from the Log2 quantization output is always a positive number, represented as an unsigned 8-bit integer. In this case, the result of the e-exponent calculation is... Approximately. Therefore, multiplication calculations after Log2 quantization can also be performed by shifting.

[0066] According to an embodiment of the present invention, the shift calculation of the input integer value and the input quantization factor using the shift exponent approximation unit to obtain the output integer value and the output quantization factor includes: determining the shift value of the input integer value based on the quantization value of the input quantization factor; shifting the input integer value based on the shift value and a specified integer to obtain the output integer value; and determining the output quantization factor based on the sum between the specified integer and the quantization value of the input quantization factor.

[0067] According to an embodiment of the present invention, the input quantization factor can be expressed as: ,in, To input the quantization factor's quantization value, based on... and specified integer The output integer value can be determined. The displacement value, in response to the input integer value Perform the shift, as shown in formula (8).

[0068] (8)

[0069] Where ≫ represents a signed right shift operation, for example Will Shift the bit to the right by 1 position. Indicates interception From a to b-1, and perform a signed extension. This is an unsigned extension. All are intermediate integer values. To output integer values, To output the quantization factor, It is a 32-bit integer value.

[0070] The input for calculating the exponent e must be a negative number, so that the above formula... Since the value is negative, a sufficiently large positive integer M is introduced into the output to avoid truncation error caused by right shift. Since the output must be non-negative, an unsigned 32-bit integer value is used.

[0071] According to an embodiment of the present invention, the input data includes a query matrix, a key matrix, and a value matrix; wherein, the above method further includes: multiplying the query matrix and the key matrix to obtain a length matrix, wherein the length matrix is ​​used as a first input integer value, and the maximum value of each row of the length matrix is ​​used as a second input integer value; and the first input integer value and the second input integer value are used as input integer values.

[0072] According to an embodiment of the present invention, multiplying the query matrix and the key matrix to obtain the length matrix can correspond to the above formula (1), and taking the maximum row value of the length matrix as the second input integer value can correspond to the above formula (2). The quantization of the Q, K, V and L matrices can be based on the minimum maximum value (Minmax) of the matrix to determine their respective quantization factors, and the quantization factor is rounded to the power of 2 according to minimizing the quantization error.

[0073] According to an embodiment of the present invention, the exponent calculation result includes a first exponent calculation result and a second exponent calculation result, wherein the first exponent calculation result corresponds to a first input integer value and the second exponent calculation result corresponds to a second input integer value; based on multiple computing cores in an accelerator including a heterogeneous multi-core architecture, attention processing operations are performed on multiple input data according to the target model to obtain an output result, including: using multiple computing cores to perform the following operations: performing a multiplication operation on the first exponent calculation result to obtain a first multiplication operation result; performing a row summation operation on the second exponent calculation result based on a preset first specified quantization factor to obtain a row summation operation result; performing a multiplication operation on the second exponent calculation result and the value matrix based on a preset second specified quantization factor to obtain a second multiplication operation result; and determining the output result based on the first multiplication operation result, the row summation operation result, and the second multiplication operation result.

[0074] According to an embodiment of the present invention, a multiplication operation is performed on the result of the first exponent calculation, since... and and With the same scaling factor, this multiplication operation can be performed by formula (9) instead of the above formulas (3) and (5), where formula (9) is used to calculate the result of the first multiplication operation. For example, and All can be calculated based on formula (9).

[0075]

[0076] According to an embodiment of the present invention, the calculation result of the second exponent and the value matrix are multiplied, that is, the above formula (6) can be quantized as shown in formula (10).

[0077] (10)

[0078] The second specified quantification factor includes and , It is preset The quantification factor will Set the value to a large 16 and move the corresponding shift operation before the summation to avoid truncation errors caused by arithmetic right shift before accumulation. Second multiplication result. Represented using a signed 32-bit integer value, and having the same... Using the same quantization factor facilitates subsequent... addition calculation, The quantization factor of the V-value matrix powers of, All are matrix coordinates, for example Represents the V-value matrix Array element.

[0079] According to an embodiment of the present invention, the result of the second index calculation is subjected to row summation, that is, the above formula (4) can be quantized as shown in formula (11).

[0080] (11)

[0081] The first specified quantization factor includes and , It is preset The quantification factor will Set to a larger value of 16 to avoid truncation errors before accumulation, and to calculate the row summation result. Represented using a signed 32-bit integer value, and having the same... Using the same quantization factor facilitates subsequent... Addition calculations.

[0082] Figure 2 A schematic diagram of an initial module according to an embodiment of the present invention is shown; Figure 3 A schematic diagram of the target module according to an embodiment of the present invention is shown.

[0083] According to embodiments of the present invention, such as Figure 2As shown, after the Q, K, and V matrices are input into the initial module, the Q and K matrices are multiplied in the MatMul unit to obtain the L matrix. The L matrix is ​​then input into the RowMax unit to select the largest row m, and another MatMul unit. m is calculated by two exponent calculation modules, and the outputs are input into the Elem-Mul unit, MatMul unit, and RowSum unit for multiplication and row summation. The outputs are then summed by the Add unit to obtain the attention score. Attention score matrix .

[0084] After quantizing the initial module based on the quantization factor, the following is obtained: Figure 3 The target module shown is updated to Shift-Exp, Shift-Mul and Shift-MatMul units by introducing a quantization factor, compared to the initial module. At the same time, a Log2-Q unit is added. The Shift-Exp and Log2-Q units are the shift exponent approximation units pointed out in this invention.

[0085] According to embodiments of the present invention, the quantization operation can convert 50% of the multiplication calculations in attention computation into hardware-efficient shift calculations, and realizes a fully integer-based inference process. Furthermore, this quantization operation has good universality, with average precision losses of only 0.98% and 0.90% on the ImageNet dataset in computer vision (CV) and the GLUE dataset in natural language processing (NLP) respectively.

[0086] According to embodiments of the present invention, representative encoder-only models and datasets from NLP and CV were selected for testing, and the present invention was compared with other related technologies. The tested models were BERT-base from the NLP domain; and DeiT-tiny, DeiT-small, DeiT-base, ViT-base, and ViT-large from the CV domain. The test dataset for the NLP models was the General Language Understanding Evaluation (GLUE); the test dataset for the CV models was ImageNet-1K. Experimental results are shown in Tables 1 and 2.

[0087] Table 1 shows the comparison results of the quantization accuracy of the CV model after training with related works.

[0088]

[0089] Compared to the full-precision 32-bit floating-point model, the quantization accuracy impact of this invention is between -1.55% and -0.51%, which falls within the normal range of post-training quantization loss. Compared to related technologies such as FQ-ViT and P2-ViT, the quantization accuracy improvement of this invention is between -0.02% and 0.46%. Furthermore, the e-exponent calculation of this invention can be performed using more hardware-friendly shift operations.

[0090] Table 2 shows the results of the post-training quantized BERT-base model of the present invention on the GLUE dataset and compares them with related techniques.

[0091]

[0092] The CoLA and STS-B tasks use Matthews correlation coefficient and Pearson correlation coefficient as evaluation metrics, respectively; the MRPC and QQP tasks use F1 score as evaluation metrics; and the remaining tasks use accuracy as evaluation metrics. It is important to note that this invention focuses on the precision loss of quantization after training, rather than the absolute precision value. Therefore, the fine-tuned full-precision model of this invention may differ from other works. To fairly compare the impact of quantization precision, this invention compares the quantization precision loss with related works. Compared to the full-precision model with 32-bit floating-point values, the quantization method of this invention has an average precision loss of 0.90% on the GLUE dataset. CoLA and QNLI show larger precision losses, at 4.60% and 2.05%, respectively. Outside of the CoLA and QNLI tasks, the quantization precision impact of this invention ranges from -0.83% to +0.36%, with an average of -0.21%. Overall, the quantization of this invention is significantly better than PTQ and Q8BERT. MP-PTQ only outperforms the method of this invention on the CoLA and QNLI tasks, but it uses a hybrid quantization bit width setting. ZeroQuant achieved optimal quantization performance using FP16 precision in self-attention computation, but the method of this invention supports hardware-friendly PoT quantization factors and shift exponents. Compared with related technologies, the post-training quantization method of this invention requires only a few minutes of calibration time and achieves better accuracy.

[0093] A second aspect of the present invention provides an accelerator, comprising: a normalization core for normalizing input data to obtain normalized input data; a linear layer core for performing linear calculations on the normalized input data to obtain target input data; and a fusion attention core for performing exponential calculation operations on the target input data based on a shift exponential approximation algorithm of a shift exponential approximation unit to obtain an exponential calculation result. The output result is determined based on the exponential calculation result. The processing time periods corresponding to multiple related exponential calculation results partially overlap. The multiple related exponential calculation results are multiple exponential calculation results output by the same fusion attention core. The processing time period represents the duration of converting input data into exponential calculation results. The shift exponential approximation unit is obtained by updating the exponential calculation unit of the initial model based on a quantization factor.

[0094] Figure 4 A schematic diagram of an accelerator according to an embodiment of the present invention is shown.

[0095] According to embodiments of the present invention, such as Figure 4 As shown, the accelerator comprises a Central Processing Unit (CPU) and a Programmable Logic (PL) platform consisting of a Field-Programmable Gate Array (FPGA). The CPU is primarily responsible for data preparation, accelerator configuration, and managing the accelerator's operating mode. The PL side includes the aforementioned heterogeneous multi-core architecture, comprising multiple computational cores: a Fusion Attention Engine (FAE), a Linear Layer Engine (LNE), and a Residual and Normalization Engine (ANE). The FAE performs the fusion computation of scaled dot product attention; the LNE executes all activation-weight matrix multiplication computations; and the ANE is responsible for residual connections and layer normalization computations.

[0096] According to embodiments of the present invention, the aforementioned computing cores access off-chip memory (DRAM / HBM) through a memory controller, utilizing memory interleaving technology to cache data across multiple BRAM groups or lookup tables to provide sufficient on-chip bandwidth for the computing array. Each computing core acts as an AXI master with its own independent AXI bus. This architecture, combined with off-chip memory, enables task-level parallelism, significantly improving throughput.

[0097] Figure 5 A schematic diagram of an encoder according to an embodiment of the present invention is shown.

[0098] According to embodiments of the present invention, such as Figure 5As shown, the QKV matrix generation (QKV Gen), attention output mapping (Concat), and feedforward layer (FFN) in the accelerator are computed by the LNE core, the scaled dot product attention (SPDA) is computed by the FAE, and the residuals and layer normalization are computed by A&N.

[0099] Figure 6 A schematic diagram of data processing for an accelerator according to an embodiment of the present invention is shown.

[0100] According to embodiments of the present invention, in order to keep the FAE and LNE computationally busy and prevent idle waiting, multiple received input data are executed in an interleaved and parallel manner. For example, for input data... Within one time step, FAE calculates Scaling dot product attention; LNE can be computed in parallel. All linear layer calculations. Residual and normalization calculations between linear layer calculations are performed via... The staggered execution is used to cover it up, such as Figure 6 As shown, in LNE calculation When performing attention output mapping, ANE calculates... Residuals and normalization; in LNE calculations When using the feedforward layer, ANE calculates... The residuals and normalization.

[0101] According to an embodiment of the present invention, the fusion attention core includes multiple computing units; wherein, the fusion attention core is used to split the normalized input data to obtain multiple data slices; the multiple computing units are used to process the multiple data slices, wherein the multiple data slices are input to the multiple computing units in a specified order at different specified time periods, and for any specified time period, each computing unit processes different data slices in parallel.

[0102] Figure 7 A schematic diagram of a fusion attention core according to an embodiment of the present invention is shown.

[0103] According to embodiments of the present invention, such as Figure 7 As shown, the fusion attention core comprises three computational units: CU1, CU2, and CU3. CU1 utilizes the memory management units MU and AT of the KReg core to perform matrix multiplication of Q and K and calculation of the row maximum value, i.e., the calculation of formulas (1) and (2). CU2 performs special processing for piecewise Softmax calculation, including... and The scaling update for each iteration round is calculated using formula (9). CU3 utilizes the memory management units MU and AT of the VReg core to perform multiplication calculations with matrix V. The summation is the calculation of formulas (10) and (11). CU1, CU2 and CU3 are pipelined in parallel through on-chip double buffering.

[0104] According to an embodiment of the present invention, the above-described computational partitioning results in a cyclic data dependency between CU2 and CU3, as shown in formulas (9)-(11). and and These are different iteration rounds of the same data partition. Therefore, CU2 and CU3 cannot be directly executed in a pipelined manner. To address this, a pipelined operation with interleaved execution of dual data partitions was designed, enabling the three computational units to be executed in a pipelined overlapping manner to improve computational efficiency.

[0105] Figure 8 A schematic diagram of data fragmentation according to an embodiment of the present invention is shown; Figure 9 A schematic diagram of the fusion attention core processing according to an embodiment of the present invention is shown.

[0106] like Figure 8 As shown, the data is first split into multiple data shards, such as... Figure 8 Obtain two independent input matrix slices and Then, in the prescribed order, such as Figure 9 As shown, the input is sequentially processed in a timeline consisting of three computational units, which are executed in an interleaved manner. and The corresponding fusion computation. In this way, CU2 computation and... Related and CU3 calculation and Related and ; CU2 calculation and Related and CU3 calculation and Related and Therefore, CU2 and CU3 will not operate on the same data shards within the same specified time period, thus resolving the dependency issue.

[0107] According to an embodiment of the present invention, the linear layer core includes an input buffer; wherein, a first-in-first-out (FIFO) buffer is provided between the input buffer and the input data stream corresponding to the input buffer, and the FIFO buffer is used to control the data interaction operation of the input buffer.

[0108] According to an embodiment of the present invention, in order to reduce unnecessary off-chip memory access, GELU activation can be integrated into the linear layer core.

[0109] Figure 10 A schematic diagram of a linear layer core according to an embodiment of the present invention is shown.

[0110] According to an embodiment of the present invention, as shown in the figure, the linear layer core consists of a computation array, input / output streams, an input buffer, an output buffer, and GELU activation units. The computation array is composed of a broadcast array of two-dimensional processing elements (PEs), performing calculations in the form of outer products. Each processing element consists of a multiplier, an adder, and a register, performing multiplication and accumulation operations. Off-chip data interaction of this core uses HLS Stream, where the weight input stream directly inputs weights into the processing units, while input and output activation are both handled through buffers. The number of banks in the input buffer is related to the parallelism of the computation array. Similarly, the parallelism of the weight input stream and the parallelism of the computation array are the same. The input activation data is broadcast in the same column of the computation array, while the weight data is broadcast in the same row. GELU activation units are integrated into the output module, allowing for dynamic selection of whether to perform GELU activation operations when the computation results are written.

[0111] According to an embodiment of the present invention, the maximum off-chip bandwidth required by the computation array is the sum of the data volume of the input activation column vector and the weight row vector, i.e. Setting up double buffering for the input buffer can reduce off-chip bandwidth requirements, but it will significantly increase buffer usage. In order to reduce off-chip bandwidth while minimizing buffer overhead, a depth-configurable First-in-First-out (FIFO) buffer is added between the input data stream and the input buffer. The data bandwidth of this FIFO is the same as that of the input buffer, and it blocks writing when the FIFO is full and blocks reading when the FIFO is empty. The minimum depth of the FIFO is shown in formula (12).

[0112] (12)

[0113] Where W represents the size of the common dimension of the matrix multiplication. This represents the number of input data that can be read from outside the chip per cycle, which needs to meet the following conditions: .

[0114] According to an embodiment of the present invention, the accelerator optimizes the performance parameters of the fusion attention core, linear layer core and normalization core of the accelerator based on the resource consumption data generated by the accelerator performing attention processing operations, so as to obtain an optimized accelerator.

[0115] According to embodiments of the present invention, the length of the input sequence of the target model varies, and the computational cost of attention calculation also varies. In order to balance the throughput among cores, hardware modeling of the resource consumption and performance of the computing cores is performed, and a search algorithm based on genetic algorithm is used to determine the performance parameters such as parallelism parameters, sharding parameters, and off-chip bandwidth of each computing core.

[0116] According to embodiments of the present invention, the search algorithm incorporates model configuration (R) and hardware constraints ( The input is , and the output is the performance parameters of each computing core. And the throughput of the accelerator. The objective function and constraints of the search algorithm are as shown in equations (13) and (14):

[0117]

[0118]

[0119] in, and For the computational latency of FAE and LNE, Resource represents the weighted term for resource consumption. These represent the total DSP, BRAM, and off-chip bandwidth consumed by the accelerator, respectively.

[0120] Figure 11 A schematic diagram of accelerator construction according to an embodiment of the present invention is shown.

[0121] like Figure 11 As shown, the target model is obtained after optimizing the initial model. The target model is mapped to a heterogeneous multi-core architecture. The heterogeneous multi-core architecture is used to perform attention processing operations on multiple input data based on the target model to obtain resource consumption data generated during the attention processing operation. Based on the input model load / configuration, the resource consumption data is analyzed by a genetic algorithm to optimize the performance parameters of the fusion attention core, linear layer core and normalized core of the heterogeneous multi-core architecture to obtain the optimized accelerator.

[0122] According to embodiments of the present invention, the accelerator based on a heterogeneous multi-core architecture achieves task-level parallelism and load balancing among cores. Furthermore, to address the data dependency problem caused by fine-grained attention fusion algorithms, the present invention interleaves the execution of multiple data shards, allowing computing units to overlap in executing different data shards.

[0123] According to embodiments of the present invention, the performance of the accelerator was evaluated in cloud scenarios (Xilinx Alveo U50) and edge scenarios (Xilinx ZCU102). Experimental results show that, compared with related technologies, the accelerator designed in this invention achieves a throughput improvement of 1.71 to 9.95 times and an energy efficiency improvement of 1.83 to 8.78 times. Furthermore, the accelerator of this invention supports input sequence lengths up to 350K, far exceeding other works.

[0124] According to an embodiment of the present invention, a hardware design space search algorithm is first used to search and deploy the ViT-base model under different input sequence length loads, verifying the throughput and computational efficiency of the accelerator in different task scenarios. Then, the performance of the searched accelerator is compared with that of CPU, GPU, and related FPGA accelerators.

[0125] Two representative FPGAs were selected as target platforms: the Xilinx Zynq ZCU102 for edge computing and the Xilinx Alveo U50 for cloud computing. The ZCU102 is an embedded platform based on the Zynq MPSoc architecture, comprising a Processing System (PS) and Programmable Logic (PL). The ZCU102's PS side includes a quad-core ARM Cortex A53 general-purpose processor and 4GB of DDR4 memory, while the PL side is equipped with an UltraScale+ XCZU9EG FPGA chip. The Alveo U50 is an accelerator card designed for data center applications, connecting to the host via a PCIe interface. The U50 accelerator card only includes the PL portion, featuring an UltraScale+ xcu50-fsvh2104 FPGA chip and 8GB of HBM memory. Compared to the ZCU102, the Alveo U50 offers richer programmable logic resources and higher memory bandwidth. We designed and deployed a prototype accelerator using Xilinx Vitis HLS and Xilinx Vitis (v2022.2).

[0126] Figure 12 The diagram illustrates the accelerator throughput and computational efficiency according to an embodiment of the present invention.

[0127] like Figure 12As shown, the accelerator's throughput and computational efficiency in executing ViT-based models on the Alveo U50 platform cover task workloads with input sequence lengths ranging from 128 to 16K. The results demonstrate that the accelerator of this invention achieves an average throughput of 2759 GOPS and a computational efficiency of 88.1%. Because the attention computation quantization method of this invention optimizes 50% of multiplications into shift operations, the attention computation core can achieve higher throughput with the same resource consumption. Furthermore, the proportion of attention computation increases with increasing sequence length; therefore, the accelerator of this invention achieves higher throughput when the sequence length is greater than 2K.

[0128] The performance and energy efficiency of FPGA accelerators searched on Alveo U50 for inference of ViT-small models were compared with those of CPU and GPU. This invention uses the PyTorchv2.0.1 framework with CUDA 11.7 and cuDNN 8.5.0 to run model inference on both CPU and GPU platforms, with a batch size of 4 for fair comparison. The CPU platform used an Intel Core i9-14900K processor with 32 MB of cache, 24 physical cores, 32 threads, a base frequency of 3.2 GHz, and a base power consumption of 125 W. It also features 48 GB of DDR5 memory. The GPU platform used an NVIDIA Tesla V100S PCIe GPU with 32 GB of HBM2 memory, 5120 hardware threads, and a clock speed of 1.245 GHz.

[0129] Figure 13 A comparison chart of throughput according to an embodiment of the present invention is shown; Figure 14 A comparison chart of energy efficiency according to an embodiment of the present invention is shown.

[0130] like Figure 13 and Figure 14 The figure shows a comparison of the throughput and energy efficiency of the accelerator of this invention on the FPGA platform compared to CPU and GPU platforms. Compared to the CPU, our accelerator achieved a 5x and 17.31x throughput improvement and a 19.69x and 10.25x energy efficiency improvement, respectively, at sequence lengths of 256 and 2048. Compared to the GPU, our accelerator achieved a 2.13x throughput improvement and a 6.17x energy efficiency improvement at a sequence length of 256; at a sequence length of 2048, the throughput reached 56% of the GPU, but still achieved a 2.84x energy efficiency improvement.

[0131] Table 3 lists the comparison results of the accelerator of the present invention with other Transformer accelerators.

[0132]

[0133] Due to different hardware platforms and model configurations, this invention uses throughput (GOPS), energy efficiency (GOPS / W), and DSP computing density (GOPS / DSP) as comparative metrics. Experimental results show that the accelerator of this invention achieves high throughput and high efficiency on different platforms. The accelerator of this invention on the ZCU102 board achieved a throughput of 942 GOPS, a maximum energy efficiency of 69.8 GOPS / W, and a DSP computing density of 0.4748. Compared with NPE and Edge-MoE, which are also deployed on embedded platforms, the throughput is improved by 0.86 times and 11.21 times, the energy efficiency by 1.28 times and 7.19 times, and the DSP computing density by 1.76 times and 11.17 times, respectively. The accelerator of this invention on the U50 board achieved a maximum throughput of 3222 GOPS, an energy efficiency of 66.85 GOPS / W, and a maximum DSP computing density of 0.5953. Compared with ViA, TECS'23, and Unified, which are also deployed on cloud platforms, the throughput is increased by 10.39 times, 1.79 times, and 5.58 times, the energy efficiency is increased by 8.41 times, 2.86 times, and 1.96 times, and the DSP computing density is increased by 2.32 times, 2.77 times, and 2.60 times.

[0134] Furthermore, related technologies employ coarse-grained attention fusion or no attention fusion at all, failing to address the storage bottleneck of the attention mechanism. They either cannot support longer input sequence lengths or are relatively inefficient. However, the accelerator of this invention, through fine-grained attention fusion, can efficiently support long sequence inference, with the length of the input sequence that can be processed limited only by the size of the activated off-chip memory. Taking U50 as an example, the output of the first fully connected layer in the feedforward layer occupies the most storage space and is allocated 8Gb of storage space across 4 HBM channels, therefore... If configured as in ViT / BERT-base The maximum input sequence length N can be 350K, far exceeding previous research.

[0135] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0136] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A data processing method based on the fusion of attention and quantization operations, characterized in that, include: Based on multiple computing cores in the heterogeneous multi-core architecture of the accelerator, attention processing operations are performed on multiple input data according to the target model to obtain the output result; The computational core includes a normalization core, a linear layer core, and a fusion attention core. The normalization core normalizes the input data to obtain normalized input data. The linear layer core performs linear calculations on the normalized input data to obtain target input data. The fusion attention core performs exponential calculations on the target input data based on a shift exponential approximation algorithm using a shift exponential approximation unit. The fusion attention core splits the normalized input data into multiple data slices. The fusion attention core processes different data slices in parallel at different specified time periods according to a specified order to obtain exponential calculation results. The output result is determined based on the exponential calculation results. The shift exponential approximation unit updates the exponential calculation unit of the initial model based on a quantization factor. The shift exponent approximation algorithm based on the shift exponent approximation unit performs an exponent calculation operation on the target input data to obtain the exponent calculation result, including: The following operations are performed using the fusion attention core: The shift value of the input integer value is determined based on the quantization value of the input quantization factor. The input integer value is shifted according to the displacement value and the specified integer to obtain the output integer value; The output quantization factor is determined based on the sum of the quantization values ​​of the specified integer and the input quantization factor. The output integer value and the output quantization factor are logarithmically transformed respectively to obtain the exponent calculation result.

2. The method according to claim 1, characterized in that, The input data includes a query matrix, a key matrix, and a value matrix; The method further includes: Multiply the query matrix and the key matrix to obtain a length matrix, wherein the length matrix is ​​used as the first input integer value and the maximum value of each row of the length matrix is ​​used as the second input integer value; The first input integer value and the second input integer value are used as the input integer value.

3. The method according to claim 2, characterized in that, The index calculation result includes a first index calculation result and a second index calculation result, wherein the first index calculation result corresponds to the first input integer value, and the second index calculation result corresponds to the second input integer value; The multiple computing cores based on the heterogeneous multi-core architecture in the accelerator perform attention processing operations on multiple input data according to the target model to obtain output results, including: using the multiple computing cores to perform the following operations: Perform a multiplication operation on the first exponent calculation result to obtain the first multiplication result; Based on a preset first specified quantization factor, the calculation result of the second index is subjected to a row summation operation to obtain the row summation result; Based on a preset second specified quantization factor, a multiplication operation is performed on the second index calculation result and the value matrix to obtain the second multiplication result; The output result is determined based on the result of the first multiplication operation, the result of the row summation operation, and the result of the second multiplication operation.

4. The method according to claim 1, characterized in that, The output results include at least one of image recognition results, text generation results, and speech recognition results.

5. An accelerator, characterized in that, The accelerator includes: The normalization core is used to normalize the input data to obtain normalized input data; The linear layer core is used to perform linear calculations on the normalized input data to obtain the target input data; The fusion attention core is used to perform exponential calculation operations on the target input data based on the shift exponential approximation algorithm of the shift exponential approximation unit. The fusion attention core splits the normalized input data to obtain multiple data slices. The fusion attention core processes different data slices in parallel at different specified time periods according to a specified order to obtain the exponential calculation result. The output result is determined based on the exponential calculation result. The shift exponential approximation unit is obtained by updating the exponential calculation unit of the initial model based on the quantization factor. The shift exponent approximation algorithm based on the shift exponent approximation unit performs an exponent calculation operation on the target input data to obtain the exponent calculation result, including: The following operations are performed using the fusion attention core: The shift value of the input integer value is determined based on the quantization value of the input quantization factor. The input integer value is shifted according to the displacement value and the specified integer to obtain the output integer value; The output quantization factor is determined based on the sum of the quantization values ​​of the specified integer and the input quantization factor. The output integer value and the output quantization factor are logarithmically transformed respectively to obtain the exponent calculation result.

6. The accelerator according to claim 5, characterized in that, The linear layer core includes an input buffer; The input buffer and the corresponding input data stream are provided with a first-in-first-out (FIFO) buffer, which is used to control the data interaction operation of the input buffer.

7. The accelerator according to claim 5, characterized in that, The accelerator optimizes the performance parameters of its fusion attention core, linear layer core, and normalized core based on the resource consumption data generated by the accelerator performing attention processing operations, in order to obtain an optimized accelerator.

Images