Adaptive hierarchical top-k pruning sparse attention acceleration method and system

Abstract

The application discloses a sparse attention acceleration method and system of self-adaptive hierarchical Top-K pruning. low and the key vector K low of all rows are used to perform matrix multiplication to obtain the approximate attention score of a row, and then the approximate attention score is normalized into an attention probability distribution through Softmax normalization, is divided into continuous G groups, and a group importance score is calculated; in combination with a preset double threshold, three-level decision is made on each group g in the G groups to dynamically determine each group as an unimportant group, a medium important group or an important group, and the corresponding number of non-zero elements K g is output. g The K g non-zero elements screened out in each group are sorted, the index position of each probability value in the global index of the original one-row attention probability distribution is generated, and the index position is output. The application aims to realize the acceleration of sparse attention, give consideration to the self-adaptation and flexibility of the dynamic sparse attention method, and realize efficient execution on hardware.

Description

TECHNICAL FIELD

[0001] The present application relates to an acceleration technique of a Transformer model, in particular to a sparse attention acceleration method and system based on adaptive hierarchical Top-K pruning. BACKGROUND

[0002] In recent years, the Transformer model has shown excellent performance in both natural language processing and computer vision fields, such as machine translation, text classification, image recognition, etc. The success of the Transformer is largely due to its attention mechanism, which achieves powerful context modeling by dynamically focusing on all positions in the sequence. However, the computational complexity of this mechanism grows quadratically with the sequence length, becoming the main bottleneck for processing long sequence tasks. For example, when processing a sequence of length 16K in the BERT-Base model, the computational load of a single attention head can reach hundreds of GFLOPS, not only bringing high latency, but also requiring storage of an LxL-sized attention matrix (L is the sequence length), causing huge memory pressure.

[0003] To reduce the computational and memory overhead of the attention mechanism, sparse attention has become a mainstream solution, which can be divided into two categories: static sparse attention methods and dynamic sparse attention methods. Among them, static sparse attention methods use predefined, input-independent fixed sparse patterns, such as the sliding window mechanism used in the Longformer model, the combination of random, local and global attention used in the BigBird model, and the block-based sparse attention based on fixed arrangement used in the BlockBERT model. Although such regularized sparse patterns are suitable for parallel computation on GPUs, they lack the ability to adapt to different models, tasks and inputs, and there is an inherent contradiction between sparsity and model accuracy: excessive sparsity will compromise model accuracy, while overly conservative sparsity will not be able to sufficiently reduce computational overhead. Dynamic sparse attention methods can dynamically select key computation positions based on input, and have the potential to achieve higher sparsity. However, such methods have paid a high performance price for their adaptability. Specifically, there are two points: first, the dynamically generated sparse patterns are mostly unstructured, for example, some existing methods directly generate irregular sparse patterns that need to be processed by additional packing and segmentation to convert them into a hardware-friendly format. Such irregular access severely restricts memory efficiency; second, the hardware implementation complexity of the dynamic prediction mechanism is high, for example, the large-scale Top-K sorting engine used in some methods has a very complex hardware implementation.

[0004] A traditional Transformer block is stacked by multi-head self-attention modules and feed-forward neural network modules. Among them, the multi-head self-attention mechanism is the core component of the Transformer block, such asFigure 1 As shown in (a), it enables the model to capture information from multiple different subspaces in parallel by uniformly dividing the query vector Q, key vector K, and value vector V of the input sequence into H attention heads along the feature dimension. Specifically, each attention head independently performs scaled dot product attention calculation, as shown in Equation (1):

[0005] ;

[0006] in, For any h-th attention head, the attention features are obtained. Let Q be the query vector, K be the key vector, and V be the input to the h-th attention head, respectively. The softmax activation function is used. This process allows the model to simultaneously focus on dependencies of different positions and types within the sequence. Finally, the outputs of all heads are concatenated and recombined through a linear projection layer to form the final attention result. This separate head processing mechanism enables the model to effectively capture long-distance dependencies, fundamentally improving its ability to model sequence data. The computation process of each single-head self-attention mechanism (or attention mechanism for short) can be understood as a content-based retrieval. Figure 1 Figures (b) and (c) describe the mechanism, which takes three matrices—query (Q), key (K), and value (V)—as input, and first proceeds according to S=QK. T The attention score matrix S is calculated as follows: Figure 1 As shown in (b) of the diagram. Subsequently, the attention score matrix S is scaled and Softmax normalization is applied row-by-row to convert it into an attention weight matrix Softmax(S). Finally, the attention weight matrix Softmax(S) is multiplied by V to achieve weighted aggregation of contextual information into the final output, as shown in the diagram. Figure 1 As shown in (c) in the figure.

[0007] While the attention mechanism endows Transformer with powerful sequence modeling capabilities, its computational and memory overhead has become a major bottleneck in practical deployments. This issue can be analyzed in depth from the perspectives of computational complexity and memory requirements. From a computational complexity perspective, the computational burden of the attention mechanism mainly comes from two core matrix multiplication operations. In the attention score calculation stage, the multiplication operation of the query vector Q and the key vector K has a time complexity of O(L...). 2 The complexity of d) is O(L), where L is the sequence length and d is the feature dimension. In the subsequent context aggregation stage, i.e., the multiplication operation of the attention weight matrix and the value vector V, it is also O(L). 2d) of the calculation complexity. Therefore, the overall calculation amount increases in square level with the sequence length. For example, taking the BERT-Base model as an example, when the sequence length increases from 512 to 768, the calculation time of the attention module accounts for about 40% and sharply rises to nearly 80%, becoming the key factor restricting the reasoning efficiency. In terms of memory requirement, the intermediate matrix with a size of LxL needs to be stored in the calculation process of the attention mechanism, and the space complexity is O(L 2 ). Taking the sequence length of 512 and the precision of FP16 as an example, the intermediate matrix of a single attention head needs about 0.5MB of storage space, and the Transformer model usually contains multiple attention heads, and the inner product of these intermediate results will significantly increase the on-chip memory pressure and may cause frequent off-chip memory access.

[0008] It is worth noting that the attention matrix itself presents significant inherent sparsity characteristics, which provides an important breakthrough for optimizing the calculation efficiency. Taking a typical attention probability matrix extracted in the BERT-Base model as an example, for most rows (i.e. query positions), only a small part of columns (i.e. key positions) correspond to higher attention probabilities. This phenomenon intuitively reveals the inherent implicit sparsity of the attention matrix: when processing information, the model will only focus on a few key semantic units. In summary, even in the medium sequence length scenario, the attention mechanism still faces severe calculation and memory challenges. This challenge is particularly prominent in resource-constrained environments such as edge computing devices. The inherent sparsity characteristics of the attention matrix lay a theoretical foundation for improving the calculation efficiency through algorithm-hardware collaborative design: how to effectively tap and utilize this sparsity has become a key research direction to promote the actual deployment of the Transformer model.

[0009] To overcome the problem of quadratic computational complexity in attention mechanism, and make full use of its inherent implicit sparsity, researchers have proposed various sparse attention methods, which can be generally divided into two categories: static sparse attention method and dynamic sparse attention method. The static sparse attention method uses a predefined fixed pattern to limit the range of attention calculation. Its regular sparse pattern is suitable for GPU parallel computing, but it lacks the ability to adapt to data characteristics. In addition, this method has an inherent contradiction between sparsity and model accuracy: over-sparsity will damage performance, while conservative sparsity will be difficult to reduce computational overhead. The dynamic sparse attention method can select important elements in real time according to input data, and achieves a better balance between accuracy and sparsity than the static method. However, this method pays a significant performance price for adaptability. Specifically, the existing dynamic sparse attention method generally faces two major bottlenecks: first, the dynamically generated sparse pattern usually has irregularity, leading to frequent random memory access; second, the hardware implementation complexity of the dynamic prediction mechanism is high. Therefore, the existing sparse attention method has not solved the fundamental contradiction between algorithm adaptability and hardware friendliness, which seriously restricts its practical application. SUMMARY

[0010] The technical problem solved by the present application: In view of the above problems of the prior art, the present application provides a sparse attention acceleration method and system based on adaptive hierarchical Top-K pruning, which aims to solve the core contradiction between adaptability and hardware friendliness in dynamic attention mechanism through a dynamic structured sparsification method, realize efficient acceleration of sparse attention, and balance the adaptability and flexibility of dynamic sparse attention method, and finally realize efficient execution on hardware.

[0011] To solve the above technical problems, the technical scheme adopted by the present application is:

[0012] A sparse attention acceleration method based on adaptive hierarchical Top-K pruning, comprising the following steps:

[0013] S101, for the current processing query row, using the query vector Q low and the key vector K low of all rows, performing matrix multiplication to obtain the approximate attention score of a row; the approximate attention score is normalized by Softmax to convert it into the attention probability distribution of a row;

[0014] S102, divide the obtained attention probability distribution P=[P1,P2,...,P L of each row with length L into continuous G groups, each group containing M probability values, where P1~P L are the attention probability distributions of the 1st~Lth rows, and the group size M is a hyperparameter;

[0015] S103, calculate group importance score for each group g in G group

[0016]

[0017] wherein, is the length of attention probability distribution P of each row, is the group size, is the attention probability of the i-th position of the g-th group;

[0018] S104, according to the group importance score , combined with the preset double threshold, make a three-level decision for each group g in G group, dynamically determine each group as "unimportant", "medium important" or "important" group, and output the corresponding number of non-zero elements K g , sort the probability values of K g non-zero elements screened out by each group to generate the index position of each probability value in the global index of the original row of attention probability distribution and output.

[0019] Optionally, the query vector Q low is a low-precision query vector of 4-bit, and when the approximate attention score is subjected to Softmax normalization, the exponential function in the Softmax normalization is realized by a lookup table.

[0020] Optionally, the realization by the lookup table includes: decomposing the exponential operation of an 8-bit integer into the product of the corresponding exponential values of high four bits MSB and low four bits LSB, obtaining the exponential operation result of the high four bits MSB by a lookup table corresponding to the high four bits MSB, obtaining the exponential operation result of the low four bits LSB by a lookup table corresponding to the low four bits LSB, and multiplying the exponential operation result of the high four bits MSB and the exponential operation result of the low four bits LSB to obtain the exponential operation result of the 8-bit integer.

[0021] Optionally, the function expression for making a three-level decision for each group g in G group in step S104 is:

[0022]

[0023] wherein, is the lower limit threshold of one of the double thresholds, is the upper limit threshold of the double thresholds, is the number of normalized elements, represents that the group g is "unimportant", represents that the group g is "medium important", represents that the group g is "important".

[0024] ​​​Furthermore, this invention also provides a sparse attention acceleration system for applying the aforementioned adaptive hierarchical Top-K pruning sparse attention acceleration method, comprising a dynamic pruning engine DPE and a sparse attention computation engine SACE, wherein the dynamic pruning engine DPE includes...

[0025] The query register is used to cache the query vector Q from the memory DRAM;

[0026] The first key buffer is used to cache the key vector K from the memory DRAM;

[0027] The PE computation array is used to compute an approximate attention score for a row of query vectors Q and all key vectors K using query vectors Q in the query register and key vectors K in the first key buffer.

[0028] The approximate attention score register is used to temporarily store the obtained approximate attention scores;

[0029] The first Softmax calculation module is used to normalize the temporarily stored approximate attention scores using Softmax and transform them into an attention probability distribution for one row.

[0030] The probability distribution register is used to temporarily store the attention probability distribution P=[P1,P2,...,P] of each row of length L. L ];

[0031] The pruning module includes a group controller, an importance score calculation unit, a threshold comparator, an intra-group Top-K sorter, and a global index generator. The group controller is used to process the attention probability distribution P=[P1,P2,...,P] of each row of a temporarily stored length L. L The data is divided into G consecutive groups, each containing M probability values, where P1 to P2 are... L These represent the attention probability distributions for rows 1 to L, with group size M being a hyperparameter. The importance score calculation unit is used to calculate the group importance score for each group g in group G. The threshold comparator is used to determine the importance score of the group. Using a preset dual threshold, a three-level decision is made for each group g in group G, dynamically classifying each group as "unimportant," "moderately important," or "important," and outputting the corresponding number of non-zero elements K. g The in-group Top-K sorter is used to sort the K groups selected from each group. g The probability values ​​of the non-zero elements are sorted; the global index generator is used to generate the index position of each probability value selected by the Top-K sorter within the group in the global index of the original row attention probability distribution and output it to the sparse attention computing engine SACE.

[0032] Optionally, the sparse attention computing engine SACE comprises:

[0033] a data fetcher configured to fetch the key vectors K corresponding to the data stored in the second key buffer, and fetch the data of the value vectors V stored in the value buffer, and fetch the data of the query vectors Q stored in the double-buffered query register,

[0034] a second key buffer configured to buffer the key vectors K from the data fetcher;

[0035] a value buffer configured to buffer the value vectors V from the data fetcher;

[0036] a double-buffered query register comprising two query registers, one of which is configured to buffer the current query vector Q fetched from the data fetcher, and the other of which is configured to buffer the next query vector Q fetched from the data fetcher;

[0037] a MAC computing array configured to perform inner product operation on the query vectors Q in the double-buffered query register and the key vectors K in the second key buffer to obtain attention scores and temporarily store the attention scores in a score register;

[0038] a score register configured to temporarily store the attention scores;

[0039] a second Softmax computing module configured to normalize the attention scores by Softmax to obtain an attention probability distribution;

[0040] a multiply-add unit configured to multiply the probability matrix formed by the attention probability distribution with the filtered value vectors V to generate a final attention output;

[0041] The sparse attention computing engine SACE adopts a dynamic sparse mode-based incremental loading strategy for data loading for each attention head of the sparse attention: according to the index positions provided by the dynamic pruning engine DPE, the key vectors K required by the first query vector Q are loaded into the second key buffer, and the data of the value vectors V required by the first query vector Q are loaded into the value buffer; in the process of processing each current query vector Q by the MAC computing array, the key vectors K and the value vectors V required by the next query vector Q are analyzed in advance, and it is detected whether the corresponding contents in the second key buffer and the value buffer are cached, and only when the key vectors K and the value vectors V required by the next query vector Q are not cached, the missing key vectors K and value vectors V required by the next query vector Q are loaded from the off-chip memory.

[0042] Optionally, the sparse attention acceleration system employs a two-level pipeline design of a head-level execution pipeline between multiple attention heads and a single-head execution pipeline within a single attention head when executed, the head-level execution pipeline refers to when the current attention head is performing attention computation, the key vector K and value vector V loading operation of the next attention head has started in advance; the single-head execution pipeline refers to when the sparse attention computation engine SACE is processing attention operation of any query vector Q, the dynamic pruning engine DPE has already executed pruning and prefetching operation in parallel for the next query vector Q, and the required data is loaded in advance into the idle buffer of the double-buffer query register.

[0043] The application also provides a sparse attention acceleration system with adaptive hierarchical Top-K pruning, comprising a microprocessor and a memory connected to each other, the microprocessor being programmed or configured to execute the sparse attention acceleration method with adaptive hierarchical Top-K pruning.

[0044] The application also provides a computer readable storage medium having a computer program or instructions stored therein, the computer program or instructions being programmed or configured to execute the sparse attention acceleration method with adaptive hierarchical Top-K pruning by a processor.

[0045] The application also provides a computer program product comprising a computer program or instructions, the computer program or instructions being programmed or configured to execute the sparse attention acceleration method with adaptive hierarchical Top-K pruning by a processor.

[0046] Compared with the prior art, the application mainly has the following beneficial effects: an efficient algorithm is the basis for fully exerting hardware performance. Attention matrix generally presents the characteristics of long-tail distribution sparsity, indicating that most attention weights contribute weakly to the calculation result, which provides a theoretical possibility for calculation acceleration. However, how to effectively utilize this sparsity is a big challenge. The existing dynamic sparse method generally relies on fine-grained element-level decision, resulting in the generation of unstructured sparse mode. This irregularity not only causes high-frequency random memory access, but also makes it difficult for hardware to effectively utilize data locality and parallel computing resources, ultimately resulting in the failure to realize the theoretical acceleration potential. To fundamentally resolve the above-mentioned contradiction, the method of the application comprises the following steps for the query row being processed: using the query vector Q low performing matrix multiplication with the key vector K of all rows low obtaining the approximate attention score of a row, and then performing Softmax normalization to obtain the attention probability distribution, dividing the attention probability distribution into continuous G groups, and calculating the group importance score combining the preset double threshold value to make a three-level decision for each group g in the G groups, dynamically determining each group as an “unimportant”, “medium important” or “important” group, and outputting the corresponding number of non-zero elements Kg The K g non-zero elements of each group are sorted and the index position of the global index of each probability value in the original one row of attention probability distribution is generated and output, and the application proposes a group-level hierarchical decision mechanism (combining the preset double threshold value to make a three-level decision on each group g in the G groups, dynamically determining each group as an "unimportant", "medium important" or "important" group, and outputting the corresponding number of non-zero elements K g ), which improves the decision granularity from the element level to the predefined group level unit, systematically evaluates the importance of the whole group, and restricts the decision to a limited discrete level, thereby directly generating a hardware-friendly regular sparse mode at the algorithm level. This design, while retaining dynamic adaptability, realizes the internal transformation of sparse mode from unstructured to structured, laying a solid foundation for subsequent efficient hardware architecture. Through the dynamic structured sparse method, the core contradiction between adaptability and hardware friendliness in dynamic attention mechanism is solved, realizing efficient acceleration of sparse attention, balancing the adaptability and flexibility of dynamic sparse attention method, and finally achieving efficient execution on hardware. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 Figure (a) is a schematic diagram of the working principle of the multi-head self-attention mechanism, (b) is a schematic diagram of calculating the attention score matrix S, and (c) is a schematic diagram of calculating the final result Output.

[0048] Figure 2 Figure is a schematic diagram of the basic flow of the method of the application.

[0049] Figure 3 Figure is a schematic diagram of the basic principle of the method of the application.

[0050] Figure 4 Figure is a schematic diagram of the structure of the sparse attention acceleration system in the embodiment of the application.

[0051] Figure 5 Figure is a schematic diagram of the structure of the pruning module in the embodiment of the application.

[0052] Figure 6 Figure is a schematic diagram of the two-stage pipeline in the embodiment of the application, wherein (a) is a schematic diagram of the working principle of the head execution pipeline, and (b) is a schematic diagram of the single-head execution pipeline.

[0053] Figure 7 Figure is the accuracy and sparsity performance of the model in different tasks in the embodiment of the application.

[0054] Figure 8The speedup ratio of the sparse attention acceleration system of the embodiment of the present application on the BERT benchmark relative to GPU and CPU.

[0055] Figure 9 The comparison of the embodiment of the present application with the existing sparse attention accelerator.

[0056] Figure 10 The precision comparison of the hierarchical approximation method (the method of the present application) and the Taylor approximation method (the traditional method) in the embodiment of the present application, wherein (a) is the absolute error comparison, and (b) is the relative error comparison. DETAILED DESCRIPTION

[0057] In order to enable the person skilled in the art to better understand the technical solutions of the present application, the technical solutions of the present application will be further described in detail below with reference to the drawings in the embodiments of the present application.

[0058] The complete calculation process of the sparse attention acceleration method of the adaptive hierarchical Top-K pruning of the present embodiment takes the low-precision query vector Q and the key vector K as input, and outputs the normalized sparse key-value index. The whole process is executed independently by row, and the core process includes four stages of low-precision score calculation, grouping, evaluation, and decision-making. As shown in Figure 2 The sparse attention acceleration method of the adaptive hierarchical Top-K pruning of the present embodiment includes the following steps:

[0059] S101, low-precision score calculation: for the current processed query row, use the 4-bit low-precision query vector Q low and the key vector K low of all rows to perform matrix multiplication to obtain the approximate attention score of a row; the approximate attention score is normalized by Softmax to convert into the attention probability distribution of a row;

[0060] S102, grouping: divide the obtained attention probability distribution P=[P1, P2,..., P L of each row with a length of L into continuous G groups, and each group contains M probability values, wherein P1~P L are the attention probability distributions of the 1st~Lth rows, the group size M is a hyperparameter, and the value is optimized in the training and evaluation stages, aiming to maintain the model accuracy rate in the range close to the baseline level;

[0061] S103, evaluation: calculate the group importance score for each group g in the G groups.

[0062] ;

[0063] wherein, is the length of the attention probability distribution P of each row, is the group size, is the attention probability of the i-th position in the g-th group; to quantify the overall importance of a group, a group importance score is defined for each group g in this embodiment, which directly measures the importance degree of the overall attention probability carried by the group by calculating the sum of all attention probability scores within the group and multiplying a scaling factor determined by the sequence length and the group size. The higher the value of the group importance score, the greater the contribution of the group in the overall attention distribution, and the higher the overall importance.

[0064] S104, decision: according to the group importance score , combined with the preset double threshold, three-level decision is made for each group g in G group, and each group is dynamically determined as "unimportant", "medium important" or "important" group, and the corresponding number of non-zero elements K g , the K g non-zero elements screened out by each group are sorted to generate the index position of each probability value in the global index of the original row of attention probability distribution and output.

[0065] As Figure 3 shown, the adaptive hierarchical Top-K pruning sparse attention acceleration method 4-bit low-precision query vector Q low and all row key vectors K low perform matrix multiplication to get a row of approximate attention score S (dense attention), through the steps S101-S104 executed by the adaptive hierarchical Top-K pruning sparse attention acceleration method of this embodiment, which is simply called DARS (Dynamic Attention with Regularized Sparsity) pruning, so that the dense attention is converted to structured sparse attention, Figure 3 K g takes the value of 0-2, which is a group of sparse attention of "unimportant", "medium important" or "important" three groups g. The adaptive hierarchical Top-K pruning sparse attention acceleration method of this embodiment first divides the attention calculation of each row into multiple fine-grained groups, and quickly evaluates the group importance score of each group through low-precision calculation. Based on this, a double threshold mechanism is used to make a three-level adaptive decision for each group, which is dynamically classified as "unimportant", "medium important" or "important" group. The key innovation of this process is that the decision result endogenously regularizes the originally unstructured dynamic sparse mode into a structured block containing only a fixed number of non-zero elements. This design not only preserves the high precision and high sparsity of input adaptivity, but also lays a solid foundation for efficient hardware processing.

[0066] The query vector Qlow For the low-precision query vector of 4-bit, the exponential function in the Softmax normalization is realized by lookup table. Benefiting from the query vector Q low The extremely low precision of 4-bit is adopted, and the exponential function in the Softmax normalization can be realized by an efficient lookup table, thereby avoiding complex exponential operation and ensuring extremely low calculation and hardware overhead of the pre-step.

[0067] In the embodiment, the implementation by the lookup table includes: decomposing the exponential operation of the 8-bit integer into the product of the high four MSB and the low four LSB corresponding exponential values, obtaining the exponential operation result of the high four MSB by the lookup table of the high four MSB corresponding lookup table, obtaining the exponential operation result of the low four LSB by the lookup table of the low four LSB corresponding lookup table, and multiplying the exponential operation result of the high four MSB and the exponential operation result of the low four LSB to obtain the exponential operation result of the 8-bit integer.

[0068] The function expression of the three-level decision for each group g in the G group in step S104 of the embodiment is:

[0069] ;

[0070] wherein, is the lower limit threshold of one of the double thresholds, is the upper limit threshold of the double thresholds, is the regularized element number, represents that the group g is "unimportant", represents that the group g is "medium important", represents that the group g is "important". The advantage of the above decision mechanism is that: first, the hierarchical reservation mechanism as a kind of fine trade-off strategy avoids excessive pruning of important groups and excessive reservation of unimportant groups, thereby better maintaining model accuracy under the same sparsity. Secondly, the mechanism constrains the reservation number of non-zero elements in each group to the three regular levels {0, k / 2, k}, and the regularity of the data block greatly optimizes the transmission efficiency of the memory system, bringing significant performance gain for hardware execution.

[0071] As Figure 4As shown, the embodiment also provides a sparse attention acceleration system for applying the adaptive hierarchical Top-K pruning sparse attention acceleration method, including a dynamic pruning engine DPE and a sparse attention calculation engine SACE. The dynamic pruning engine DPE and the sparse attention calculation engine SACE constitute a double-engine system, the dynamic pruning engine DPE as the front-end decision, uses a low-precision arithmetic unit to approximately calculate the attention score, and executes the dynamic adaptive hierarchical Top-K sparsification algorithm proposed in this paper, and finally outputs a set of normalized key-value indexes. The sparse attention calculation engine SACE as the back-end execution, receives the key-value index from the dynamic pruning engine DPE, only loads the high-precision key K and value V data corresponding to it, and completes the subsequent high-precision calculation. The dynamic pruning engine DPE is responsible for executing the adaptive hierarchical Top-K pruning sparse attention acceleration method: it uses low-precision query Q and key K vectors to calculate the approximate attention score of a row of query Q and all keys K, and completes grouping and importance decision based on double thresholds in a pipelined manner, and finally passes the screened important key-value index to the sparse attention calculation engine. The sparse attention calculation engine SACE then accurately loads the high-precision corresponding key K and value V from the off-chip memory into the second key buffer and value buffer of its core. Subsequently, the processor uses its parallel MAC calculation array to efficiently complete high-precision core attention probability calculation and weighted summation of numerical values, thereby ensuring model accuracy while completely avoiding the huge memory overhead caused by global loading of key-value pairs in traditional schemes, and realizing efficient calculation.

[0072] The dynamic pruning engine DPE undertakes the key task of generating a normalized sparse key-value index for each row of query vector. Its complete execution process is as follows: after the pre-quantized 4-bit low-precision query Q and key K vectors are loaded from the off-chip memory to the dedicated buffer, the low-precision score calculation unit performs QK T matrix multiplication to generate an approximate attention score matrix. The result is then sent to the lookup table Softmax unit to complete the normalization processing through the lookup table operation, and the corresponding attention probability distribution is output. Finally, the complete probability value of this row is sent to the pruning module to obtain the key-value index of the important position through pruning operation. As Figure 4 As shown, the dynamic pruning engine DPE in the embodiment includes

[0073] a query register for caching the query vector Q from the memory DRAM;

[0074] a first key buffer for caching the key vector K from the memory DRAM;

[0075] The PE computation array, comprising Xm computation units PE, is used to compute an approximate attention score for a row by using the query vector Q in the query register and the key vector K in the first key buffer to compute a row of query vector Q and all key vectors K.

[0076] The approximate attention score register is used to temporarily store the obtained approximate attention scores;

[0077] The first Softmax calculation module is used to normalize the temporarily stored approximate attention scores using Softmax and transform them into an attention probability distribution for one row.

[0078] The probability distribution register is used to temporarily store the attention probability distribution P=[P1,P2,...,P] of each row of length L. L ];

[0079] Pruning modules, such as Figure 5 As shown, it includes a group controller, an importance score calculation unit, a threshold comparator, an intra-group Top-K sorter, and a global index generator. The group controller is used to process the attention probability distribution P=[P1,P2,...,P] of each row of a temporarily stored length L. L The data is divided into G consecutive groups, each containing M probability values, where P1 to P2 are... L These represent the attention probability distributions for rows 1 to L, with group size M being a hyperparameter. The importance score calculation unit is used to calculate the group importance score for each group g in group G. The threshold comparator is used to determine the importance score of the group. Using a preset dual threshold, a three-level decision is made for each group g in group G, dynamically classifying each group as "unimportant," "moderately important," or "important," and outputting the corresponding number of non-zero elements K. g The in-group Top-K sorter is used to sort the K groups selected from each group. g The probability values ​​of the non-zero elements are sorted; the global index generator is used to generate the index position of each probability value selected by the Top-K sorter within the group in the global index of the original row attention probability distribution and output it to the sparse attention computing engine SACE.

[0080] like Figure 5As shown, the pruning module is responsible for executing the pruning algorithm, the core of which includes five components: group controller, GIS calculation unit, threshold comparator, intra-group Top-K sorter, and global index generator. The group controller first divides the attention probability distribution row of length L into several consecutive groups of size M. The GIS calculation unit then calculates the group importance score GIS of each group, quickly evaluating the overall importance of each group through accumulation and scaling operations. The threshold comparator receives the GIS calculation results, compares them with the on-chip stored double threshold (τ0, τ1), completes the fine three-level decision, dynamically determines each group as an "unimportant", "medium important", or "important" group, and outputs the corresponding retention quantity signal K g ∈{0, k / 2, k}. Based on this decision, the intra-group Top-K sorter performs fixed-size Top-k sorting on each group. This process sorts the M attention probability values in each group and simultaneously records their corresponding position information, generating the intra-group local index. After sorting is completed, the system dynamically selects the top K g g largest probability values according to the decision signal Koutput by the threshold comparator, and outputs the corresponding local index to the global index generator. In the global index generator, the group number maintained by the group controller is combined with the intra-group local index output by the sorter to generate the final index position of each probability value in the original attention matrix row. These regularized indices accurately point to the important elements in the original key K, value V data that need to be retained, forming a hardware-friendly data access pattern.

[0081] To maximize hardware efficiency, the dynamic pruning engine DPE adopts a carefully designed group-level pipeline architecture. This architecture organizes the index generation process into three logically clear and closely connected processing stages: first, in the decision stage, the system reads the calculated approximate probability values from the buffer, completes group importance evaluation and dynamic threshold judgment; then, in the sorting stage, Top-K sorting is performed on the elements within the group; finally, in the index generation stage, based on the sorting results and the retention number output calculated in the dynamic threshold judgment stage, the regularized sparse index is output. This three-stage pipeline design ensures that the dynamic pruning engine DPE can continuously and efficiently generate hardware-friendly sparse patterns, providing stable data flow for subsequent computing units.

[0082] The sparse attention computing engine SACE is the core execution unit of the DARS architecture of the sparse attention acceleration system of the embodiment to perform high-precision attention computation. The sparse attention computing engine SACE receives the key-value index from the dynamic pruning engine (DPE) and obtains the corresponding key K and value V data from the external memory and stores them into the key buffer and the value buffer respectively. The engine is equipped with a double-buffered query register inside to save the high-precision query vector, adopts a MAC computing array to perform inner product operation, and completes attention probability computation through a special second Softmax computing module. As shown in Figure 4 The sparse attention computing engine SACE comprises:

[0083] a data obtainer configured to obtain the data of the corresponding key vector K and store it into the second key buffer, obtain the data of the value vector V and store it into the value buffer, and obtain the data of the query vector Q and store it into the double-buffered query register,

[0084] a second key buffer configured to buffer the key vector K from the data obtainer;

[0085] a value buffer configured to buffer the value vector V from the data obtainer;

[0086] a double-buffered query register comprising two query registers, one of which is configured to buffer the current query vector Q obtained from the data obtainer and the other of which is configured to buffer the next query vector Q obtained from the data obtainer;

[0087] a MAC computing array comprising Xn MAC computing units, configured to perform inner product operation on the query vector Q from the double-buffered query register and the key vector K from the second key buffer to obtain attention scores and temporarily store them in the score register;

[0088] a score register configured to temporarily store the attention scores;

[0089] a second Softmax computing module configured to normalize the attention scores through Softmax to obtain an attention probability distribution;

[0090] a multiply-add unit configured to multiply the probability matrix formed by the attention probability distribution with the screened value vector V to generate a final attention output;

[0091] The sparse attention computing engine SACE adopts a dynamic sparse mode-based incremental loading strategy for data loading for each attention head of sparse attention: according to an index position provided by the dynamic pruning engine DPE, the key vector K required by the first query vector Q is stored into the second key buffer, and the data of the value vector V required by the first query vector Q is stored into the value buffer; in the process of processing each current query vector Q by the MAC computing array, the key vector K and the value vector V required by the next query vector Q are analyzed in advance, and it is detected whether the corresponding content is cached in the second key buffer and the value buffer, and only when the key vector K and the value vector V required by the next query vector Q are not cached, the missing key vector K and value vector V required by the next query vector Q are loaded from the off-chip memory.

[0092] As an optional implementation, the exponential function calculation in the Softmax computing module (the first Softmax computing module and the second Softmax computing module) is optimized in the embodiment. The traditional implementation method usually relies on a single large lookup table (LUT) to store the exponential function value, although it is simple to implement, but when facing high precision requirements, it will bring significant storage resource overhead. For example, if 16-bit representation is used, the lookup table needs 65536 entries, and such SRAM implementation will generate huge power consumption and area overhead. If the Taylor approximation method is completely used, although the storage requirement can be reduced, a large error will be caused in a large input range, which is difficult to meet the accuracy requirement. In order to achieve a better balance between storage resources and calculation accuracy, a hierarchical approximation optimization strategy is proposed in the embodiment. The core of the strategy is to decompose the input value into an integer part and a decimal part by using the decomposable characteristics of the exponential function, and the exponential values of the integer part and the decimal part are calculated by different approximation methods, and the complete result can be obtained by multiplication, that is:

[0093] ;

[0094] wherein, is the integer part, is the decimal part [0, 1).

[0095] In the specific implementation, the double lookup table structure is used to replace the traditional single large lookup table for the integer part, and for example, the exponential operation of the 8-bit integer is decomposed into the product of the exponential values corresponding to the high four bits MSB and the low four bits LSB:

[0096] ;

[0097] The method uses two small lookup tables with 16 entries and a multiplier to replace a lookup table with 256 entries, effectively reducing the storage resource requirement. In the decimal exponent operation in the interval [0, 1), a six-order Taylor expansion is performed with a = 0.5 10 = 0.12 as the expansion point, and a high approximation effect can be obtained. The optimization strategy of this hierarchical approximation effectively solves the dual problems of excessive size of the lookup table and insufficient approximation accuracy in high-precision exponent operation, and provides a feasible scheme for efficient hardware implementation of the high-precision Softmax module.

[0098] In terms of data loading, the sparse attention computing engine SACE adopts an incremental loading strategy based on a dynamic sparse mode, aiming to reduce the huge memory (DRAM) access overhead caused by loading the key vector and value vector in full. Specifically, for each attention head, the system first loads the key and value data required for the first query according to the index provided by the dynamic pruning engine (DPE). During the processing of the current query, the system will analyze the key and value data required for the next query in advance and detect whether the corresponding content is cached in the buffer. Only when the target key and value data are not cached, the system loads the missing part from the off-chip memory. This mechanism effectively avoids unnecessary data transmission and improves the overall access efficiency. Through the deep cooperation of the computing pipeline and the data incremental loading mechanism, the sparse attention computing engine SACE maintains the computing accuracy while achieving excellent memory access efficiency, providing a key guarantee for the overall performance of the DARS architecture.

[0099] The sparse attention acceleration system in this embodiment adopts a two-level pipeline design of a head-level execution pipeline between multiple attention heads and a single-head execution pipeline within a single attention head during execution, Figure 6 The principle diagram of the two-level pipeline of the embodiment of the application, wherein (a) is a working principle diagram of the head-level execution pipeline, and (b) is a principle diagram of the single-head execution pipeline. The sparse attention acceleration system de DARS architecture in this embodiment realizes the high overlap of computing and memory access operations through the two-level pipeline design of the head-level and the single-head, thereby maximizing the system throughput. Figure 6 In the figure, the low-precision query vector Q and the high-precision query vector Q are query vectors Q in the dynamic pruning engine DPE and the sparse attention computing engine SACE, respectively. In this embodiment, the head-level execution pipeline refers to the key vector K and the value vector V loading operation of the next attention head starting in advance when the current attention head is performing attention calculation; for example, Figure 6As shown in (a), the execution processes of multiple attention heads are tightly coupled in a pipelined manner. While one head (e.g., Head0) is performing high-precision attention calculations, the key-value data loading operation for the next head (Head1) has already begun in advance. This inter-head pipeline mode effectively utilizes computing resources and ensures continuous and efficient processing across multiple heads. In this embodiment, single-head execution pipeline means that while the Sparse Attention Calculation Engine (SACE) is processing the attention operation of any query vector Q, the Dynamic Pruning Engine (DPE) is already performing pruning and prefetching operations in parallel for the next query vector Q, preloading the required data into the free buffer in the double-buffered lookup register. Figure 6 As shown in (b), efficient collaboration between the dynamic pruning engine and the sparse attention computation engine is achieved at the single-head execution pipeline level. Each head employs a double-buffering mechanism, ensuring complete overlap between query loading and computation operations, while the dynamic pruning and attention computation processes form a continuous pipeline. Specifically, when the SACE engine is processing Q... n During high-precision attention calculations, the DPE engine is already performing parallel Q operations. n+1 The "pruning-prefetching" operation loads the required data into the free buffer in advance. This sophisticated pipeline design ensures that the processing unit always operates efficiently, effectively hides data access latency, and guarantees a continuous data supply to the computing unit.

[0100] To evaluate the effectiveness of the pruning function in the adaptive hierarchical Top-K pruning sparse attention acceleration method of this embodiment, the evaluation was performed on a BERT model. The datasets used included eight tasks from the GLUE benchmark: MNLI, MRPC, QQP, SST-2, STS-B, QNLI, CoLA, and RTE, as well as SQuAD and CLOTH. For the software pruning implementation, the code for the pruning function in this embodiment is based on NVIDIA's BERT, and the evaluation code comes from Hugging Face's Transformers library. All models were implemented and executed using PyTorch v2.1.0. MKL-DNN and CuDNN were used as the operation libraries for Intel CPU and NVIDIA GPU, respectively, in this experiment. Since the pruning function in this embodiment does not require changing the model's pre-training weights, it directly performs fine-tuning on downstream tasks based on pre-training checkpoints. In this embodiment, the adaptive hierarchical Top-K pruning sparse attention acceleration method independently adjusts the group size and hierarchical decision threshold for different tasks to achieve the best accuracy-sparseness balance. All models use the same training configuration as their baseline models, including hyperparameters such as batch size and learning rate, and are trained for a sufficient number of epochs until convergence on the validation set. The sparsity metrics for each task reported are evaluated on the corresponding test set. In terms of hardware implementation, this embodiment adopts a high-level synthesis design method to implement an accelerator architecture consisting of a dynamic pruning engine and a sparse attention computation engine. First, optimized Verilog RTL code is generated through high-level synthesis, and logic synthesis and implementation are performed in the Vivado 2023.1 design suite at a target frequency of 500MHz. Through this process, we estimate the chip area and total power consumption of the overall hardware architecture under the target process technology. At the same time, to evaluate the system performance of this embodiment, a cycle-accurate simulator was developed, which runs under an off-chip bandwidth constraint of 25.6GB / s. The hardware baseline was used to evaluate the performance of the proposed framework on a modern hardware platform, including an NVIDIA GeForce RTX 4060 GPU and an Intel Core i7-13700F CPU. For measurement methods, GPU testing used PyTorch and CuDNN 8.9.2, while CPU evaluation used PyTorch and MKL-DNN. All latency measurements were performed using platform-specific high-precision timing mechanisms to ensure accuracy and consistency across platforms. Furthermore, the proposed method was compared with advanced sparse attention accelerators such as SpAtten, Sanger, and Energon. To ensure fairness, the hardware configuration of all compared accelerators was standardized to 128 multipliers and a 1GHz operating frequency.

[0101] To evaluate the software pruning results in the method of the embodiment, the proposed pruning method is comprehensively evaluated on the BERT model in the embodiment, and is compared with the current advanced sparse attention method Sanger. It should be noted that the sparsity in the embodiment is defined as the proportion of the removed calculation amount in the total calculation amount in the attention calculation process, and a higher sparsity means more calculation is saved. Figure 7 The accuracy and sparsity performance of the model in different tasks are shown, where the baseline represents the dense attention model without pruning. To ensure fairness, DARS adjusts the threshold and group size parameters to keep the accuracy of each task basically flat with Sanger, and then compares the sparsity of the two. The experimental results show that the average accuracy of the BERT model of DARS on MNLI, MRPC and other tasks reaches 81.62%, which is only about 0.13% lower than the baseline, and is comparable to Sanger (81.27%). In terms of sparsity, the average sparsity of DARS reaches 87.32%, which is 1.06 times that of Sanger (82.71%). In summary, the method maintains similar accuracy to the original dense model while achieving significant computational savings. It is worth noting that in some tasks our method shows unique advantages. In the MNLI and SST-2 tasks, the accuracy of the pruned model even slightly surpasses the dense baseline, which can be attributed to our hierarchical pruning mechanism that can effectively filter out noisy attention, making the model pay more attention to important semantic information. It is also observed that tasks with longer sequence lengths usually exhibit higher redundancy, for example, SQuAD and CLOTH tasks have higher compressibility than CoLA and STS-2 tasks.

[0102] To evaluate the acceleration performance of the sparse attention acceleration system of the embodiment, the acceleration performance is evaluated on the BERT model in the embodiment, with throughput (GOP / s) as the core evaluation index, and compared with modern hardware platforms (GPU and CPU). The experimental results are as follows: Figure 8As shown, our accelerator achieves significant performance improvements. Specifically, compared to an NVIDIA GeForce RTX 4060 GPU (FP32 precision), our accelerator achieves a 7.48x increase in throughput on BERT models, and compared to an Intel Core i7-13700F CPU, it achieves a 39.05x increase in throughput. This performance improvement stems primarily from two key factors: First, the dynamic pruning method effectively reduces computational complexity, reducing computation by an average of over 87.32% by removing redundant attention calculations; second, the specially designed sparse attention computation engine efficiently handles the regular sparse computation patterns generated by dynamic pruning, resulting in a significant improvement in computational efficiency.

[0103] Furthermore, the sparse attention acceleration system (DARS) of this embodiment is compared with current state-of-the-art sparse attention accelerators, including the SpAtten, Sanger, and Energon accelerators. Similarly, throughput (GOP / S) is used as the core evaluation metric. Experimental results are as follows: Figure 9 As shown. From Figure 9 As can be seen, the throughput of the Sparse Attention Acceleration System (DARS) in this embodiment is 2.53, 1.72, and 1.49 times that of the SpAtten, Sanger, and Energon accelerators, respectively. This advantage stems from the inherent limitations of each comparative architecture: Although the SpAtten accelerator achieves high model sparsity through cascaded pruning, its operation of globally sorting the attention scores of the entire row introduces a large amount of fixed decision overhead, significantly crowding out computational resources and limiting actual throughput. The Energon accelerator's Mixed Precision Multi-Round Filtering (MP-MRF) algorithm generates unstructured sparse patterns due to row-by-row independent decision-making, resulting in irregular memory access; at the same time, its process of sequentially executing multi-round filtering judgments cannot fully utilize the hardware's parallel computing capabilities. Although the Sanger accelerator achieves high sparsity through thresholding, the preprocessing steps such as packing and segmentation required for unstructured sparse patterns introduce significant additional overhead. Experimental results show that the Sparse Attention Acceleration System (DARS) in this embodiment generates hardware-friendly regular sparse patterns while maintaining high sparsity through a dynamic adaptive hierarchical Top-K pruning mechanism. This effectively avoids irregular memory access and complex preprocessing overhead in the above methods, thus significantly outperforming existing design schemes in terms of throughput.

[0104] The power consumption of the sparse attention acceleration system (DARS) in this embodiment is 0.903W, and the core area is 6.08mm². 2The power consumption and area overhead mainly come from the computing engine SACE and the on-chip cache. Specifically, the on-chip storage system includes two 128KB high-precision Key / Value buffers (the second key buffer + the value buffer), a 32KB low-precision Key buffer (the first key buffer), and the like. The sparse attention computing engine SACE needs to perform high-precision matrix multiplication QK and matrix multiplication Prob x V, which become the main source of power consumption in the architecture.

[0105] To evaluate the comprehensive performance of the hierarchical approximate exponential calculation method proposed in the method of the embodiment, comparative experiments are performed on the calculation accuracy and hardware resource consumption. Figure 10 For the accuracy comparison between the hierarchical approximation method (the method of the application) and the Taylor approximation method (the traditional method) in the embodiment, (a) is the absolute error comparison, and (b) is the relative error comparison. As shown in Figure 10 As shown in (a), the relative error of the traditional Taylor method increases sharply when the input value is greater than 4, showing an exponential growth trend, which leads to completely unreliable calculation results. In contrast, as shown in Figure 10 (b), the hierarchical approximation method (the method of the application) in the embodiment maintains a low error throughout the [0, 8] input range, successfully solving the large input value calculation inaccuracy problem caused by local convergence of the traditional method. In terms of storage resources, this method reduces the number of storage entries of the traditional single lookup table from 256 to 32 through a double lookup table structure when the input is an 8-bit integer, significantly reducing the occupation of BRAM and other on-chip storage resources. This design guarantees the stability of the calculation in the whole input range at the premise of introducing a small number of multiplication operations, and saves considerable storage space, providing a more feasible implementation scheme for efficient deployment of the Softmax calculation module on embedded platforms such as FPGA.

[0106] In summary, the embodiment proposes a hardware-software co-design method for attention mechanism acceleration. At the software algorithm level, the dynamic adaptive hierarchical Top-K pruning proposed by the method of the embodiment not only effectively reduces the calculation complexity, but also generates a regular sparse structure that is friendly to hardware, thereby avoiding the additional overhead caused by irregular memory access. At the hardware level, the embodiment designs a special hardware architecture, including a dynamic pruning engine and a sparse attention computing engine, which effectively supports efficient calculation in the dynamic sparse mode. Experimental results show that the embodiment can achieve significant performance improvement on the BERT model while maintaining model accuracy, providing an effective solution for the optimization of the attention mechanism.

[0107] In addition, the embodiment further provides a sparse attention acceleration system with adaptive hierarchical Top-K pruning, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the sparse attention acceleration method with adaptive hierarchical Top-K pruning. The embodiment further provides a computer readable storage medium, wherein a computer program or instructions are stored in the computer readable storage medium, and the computer program or instructions are programmed or configured to execute the sparse attention acceleration method with adaptive hierarchical Top-K pruning by a processor. The embodiment further provides a computer program product, comprising a computer program or instructions, which are programmed or configured to execute the sparse attention acceleration method with adaptive hierarchical Top-K pruning by a processor.

[0108] Those skilled in the art should understand that the technical solutions provided by the present application can be in the form of a method, a system or a computer program product. Therefore, the present application can be in the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code. The present application is described with reference to flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device implemented in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks Figure 1steps of the functions specified in the one or more blocks.

[0109] The above merely describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solution falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application shall also be considered as falling within the protection scope of the present application.

Claims

1. A system for sparse attention acceleration, comprising: The dynamic pruning engine DPE comprises a query register for caching a query vector Q from a memory DRAM; a first key buffer for caching a key vector K from the memory DRAM; a PE computing array for computing an approximate attention score of a row of the query vector Q and all the key vectors K using the query vector Q in the query register and the key vector K in the first key buffer; an approximate attention score register for temporarily storing the obtained approximate attention score; a first Softmax computing module for normalizing the temporarily stored approximate attention score by Softmax to convert the approximate attention score into an attention probability distribution of a row; the global index generator is configured to generate an index position of each probability value filtered by the in-group Top-K sorter in the original attention probability distribution of a row and output the index position to the sparse attention computing engine SACE; a probability distribution register for temporarily storing the resulting attention probability distributions P = [P1, P2,..., P L ] of each row of length L; The pruning module comprises a group controller, an importance score calculation unit, a threshold comparator, an intra-group Top-K sorter and a global index generator. The group controller is used to divide the temporarily stored attention probability distribution P=[P1, P2,..., P L ] of each row of length L into continuous G groups, each group containing M probability values, wherein P1~P L are the attention probability distributions of the 1st to Lth rows respectively, and the group size M is a hyperparameter. The importance score calculating unit is configured to calculate a group importance score for each group g in the G groups ; The threshold comparator is configured to determine, according to the group importance score , a three-level decision for each group g in the G groups in combination with a preset double threshold, dynamically determine each group as a "non-important", "medium-important" or "important" group, and output a corresponding number of non-zero elements K g ; and the intra-group Top-K sorter is configured to sort the probability values of the K g non-zero elements screened out in each group. the sparse attention computing engine SACE comprises: a data obtainer configured to obtain data of the corresponding key vector K and store the data into a second key buffer, obtain data of the value vector V and store the data into a value buffer, and obtain data of the query vector Q and store the data into a double-buffered query register, the second key buffer for caching the key vector K from the data obtainer; the value buffer for caching the value vector V from the data obtainer; the double-buffered query register comprising two query registers, one of which is configured to cache a current query vector Q obtained from the data obtainer, and the other of which is configured to cache a next query vector Q obtained from the data obtainer; a MAC computing array configured to perform an inner product operation on the query vector Q from the double-buffered query register and the key vector K from the second key buffer to obtain an attention score and temporarily store the attention score in a score register; the score register for temporarily storing the attention score; a second Softmax computing module for normalizing the attention score by Softmax to obtain an attention probability distribution; a multiply-add unit configured to multiply a probability matrix formed by the attention probability distribution with the filtered value vector V to generate a final attention output; The sparse attention computing engine SACE adopts an incremental loading strategy based on a dynamic sparse mode for data loading for each attention head of the sparse attention: according to the index position provided by the dynamic pruning engine DPE, the key vector K required by the first query vector Q is stored into the second key buffer, and the data of the value vector V required by the first query vector Q is stored into the value buffer; in the process of processing each current query vector Q by the MAC computing array, the key vector K and the value vector V required by the next query vector Q are analyzed in advance, and it is detected whether the corresponding contents in the second key buffer and the value buffer are cached, and only when the key vector K and the value vector V required by the next query vector Q are not cached, the missing key vector K and value vector V required by the next query vector Q are loaded from an off-chip memory. ​ The sparse attention acceleration system adopts a two-stage pipeline design of a head-level execution pipeline between multiple attention heads and a single-head execution pipeline within a single attention head when executed, the head-level execution pipeline refers to when the current attention head is performing attention calculation, the key vector K and value vector V loading operation of the next attention head has started in advance, and the single-head execution pipeline refers to when the sparse attention calculation engine SACE is processing attention operation of any query vector Q, the dynamic pruning engine DPE has already executed pruning and prefetching operation in parallel for the next query vector Q, and the required data is loaded in advance into the idle buffer area in the double-buffer query register.

2. The sparse attention acceleration system of claim 1, wherein, An importance score calculation unit calculates a group importance score for each group g in the G groups The function expression of the function is: ； wherein, is the length of the attention probability distribution P for each row, is the group size, is the attention probability for the i-th position of the g-th group.

3. The sparse attention acceleration system of claim 2, wherein, The exponential function in the Softmax normalization is realized by a lookup table.

4. The sparse attention acceleration system of claim 3, wherein, The realization by a lookup table includes: decomposing the exponential operation of an 8-bit integer into the product of high four bits MSB and low four bits LSB corresponding exponential values, obtaining the exponential operation result of the high four bits MSB by a lookup table corresponding to the high four bits MSB, obtaining the exponential operation result of the low four bits LSB by a lookup table corresponding to the low four bits LSB, and multiplying the exponential operation result of the high four bits MSB and the exponential operation result of the low four bits LSB to obtain the exponential operation result of the 8-bit integer.

5. The sparse attention acceleration system of claim 1, wherein, The function expression of the three-level decision for each group g in the G groups in combination with the preset double threshold value is: ； wherein is a lower threshold value of one of the double thresholds, is an upper threshold value of one of the double thresholds, is a preset number of elements, denotes that the group g is "not important", denotes that the group g is "medium important", denotes that the group g is "important".

Images