X:m dynamic attention sparsity method and device

Abstract

The application discloses an X:M dynamic attention sparsification method and device, the method comprises the following steps: predicting an attention score matrix, and performing X:M structured pruning based on a prediction result to obtain a pruning result; updating a minimum row number and an index row number in a cache and adjusting a scheduling threshold based on the pruning result, and storing an index block number meeting a condition into a priority queue to map a corresponding index to a PEA processing unit array for execution. The application greatly reduces the storage and calculation overhead of attention prediction, and combines X:M with a scheduling algorithm, so that processing units in an application-specific integrated circuit can maintain a very high utilization rate under different sparsities.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sparse attention acceleration task, and particularly relates to an X:M dynamic attention sparsity method and device. BACKGROUND

[0002] Transformers based on attention mechanisms have shown excellent performance in various domains of artificial intelligence, such as natural language processing (NLP), computer vision, and multi-modal tasks. In the NLP domain, large language models (LLMs) have demonstrated impressive performance on a variety of tasks, such as question answering, text generation, and logical reasoning assistance. The superior performance of LLMs is mainly attributed to the self-attention mechanism, which enables the model to capture global contextual information from the entire sequence. The self-attention block involves two matrix multiplication operations on three matrices, namely query (Q), key (K), and value (V). By multiplying Q and K, self-attention first obtains a score matrix (S), which is then normalized by the softmax function. Subsequently, the normalized score matrix is multiplied by V to generate the output. Therefore, self-attention has a quadratic complexity and incurs a significant computational overhead, especially when dealing with long input sequences. As a result, the self-attention block is considered a bottleneck for long-sequence Transformers, making deployment on end devices difficult.

[0003] To address the high cost of the self-attention mechanism when scaling to long sequences, sparse attention is a widely used technique in accelerators. Unlike traditional dense Q and K computation of the entire score matrix, sparse attention selects a subset of Q and K pairs for sampling dense-dense matrix multiplication (SDDMM), resulting in a sparse score matrix. Then, sparse-dense matrix multiplication (SpMM) is performed by multiplying the normalized sparse score matrix with V. Existing sparse attention mechanisms mainly include static sparsity and dynamic sparsity. Static sparsity determines the sparsity pattern in advance, ignoring the differences between different sequences, resulting in limited computational savings at the same accuracy level. In contrast, dynamic sparsity dynamically generates sparse patterns based on the input query and key, achieving a better trade-off between accuracy and computational savings. Therefore, the present work also focuses on dynamic sparsity.

[0004] However, dynamic sparsity also faces two challenges when employing hardware accelerators. First, dynamic sparsity may learn arbitrary sparse patterns that are typically highly unstructured. Due to the lack of regularity, it can lead to workload imbalance and random memory access in SDDMM and SpMM operations, resulting in low hardware resource utilization. Second, when scaling to long sequences in LLMs, there is inevitably a non-negligible overhead in predicting sparse patterns, including prediction hardware overhead (e.g., computation and memory resources) and prediction latency overhead.

[0005] Existing dynamic sparse attention accelerators mainly adopt three sparse methods. The first is row-wise selection, which performs dynamic sparsification on the entire row range of the attention matrix through row-wise sorting. Since it involves selecting the entire row, it has high time complexity, is difficult to extend to longer sequences, and when this type of extension is extended to multiple rows for row parallelization, it will inevitably suffer from the heavy memory overhead of prediction. The second is global thresholding, which dynamically selects non-zero elements by comparing attention scores with a threshold. This accelerator only needs one round of comparison, so it has lower prediction complexity. However, since the attention scores of different rows share the same threshold, it needs to perform a softmax operation to normalize the scores into probabilities for comparison, which involves saving the entire row of scores in on-chip memory. When multiple rows are processed simultaneously, this pruning will generate a large memory overhead due to prediction. In addition, this type of pruning method can generate arbitrary sparse patterns to achieve high sparsity, but this will also result in poor regularity, making it difficult to achieve high hardware utilization. The third is block-wise selection, which performs value pruning by sorting values within blocks of the attention matrix. Since the generated sparse patterns maintain regularity to some extent, it can be efficiently accelerated by hardware. However, since each block retains similar patterns, even those that are not important, it can only achieve a moderate level of sparsity at the same level of accuracy. Therefore, achieving a good balance between accuracy, sparsity, and hardware friendliness is a challenging task for sparse attention. SUMMARY

[0006] The present application aims to at least partially solve one of the technical problems in the related art.

[0007] In view of the problems existing in the prior art, the present application proposes an algorithm-hardware co-design framework to accelerate sparse attention through X:M fine-grained structured pruning. The present application proposes an X:M dynamic attention sparsification method that improves the regularity of attention scores through X:M fine-grained structured pruning, and combines a scheduling algorithm to reorganize score blocks into hardware blocks that can perfectly match the size of processing element arrays (PEA) for SDDMM and SpMM operations, ultimately achieving extremely high PE utilization.

[0008] Another object of the present application is to propose an X:M dynamic attention sparsification device.

[0009] To achieve the above object, the present application proposes, in one aspect, an X:M dynamic attention sparsification method, comprising:

[0010] predicting the attention score matrix and performing X:M structured pruning based on the prediction result to obtain a pruning result;

[0011] Update the minimum row number and index row number in the cache based on the pruning result and adjust the scheduling threshold, and store the index block number meeting the condition in the priority queue to map the corresponding index to the PEA processing unit array for execution.

[0012] The X:M dynamic attention sparsification method of the embodiment of the application can further have the following additional technical features:

[0013] In an embodiment of the application, the attention score matrix is predicted, and X:M structured pruning is performed based on the prediction result to obtain a pruning result, including:

[0014] Fine-grained structured pruning is performed on the original Q matrix to generate a sparse low-bit Q matrix and related index information;

[0015] The sparse low-bit Q matrix is multiplied with the K matrix using a 4-bit multiplication array to generate a low-precision attention score matrix;

[0016] N:M pruning and X:N pruning are performed using the generated low-precision score matrix and corresponding index information to obtain a sparse attention score matrix, and a final index block is generated based on the pruned attention score matrix; wherein each index block contains the sparse score and corresponding index information.

[0017] In an embodiment of the application, the minimum row number and index row number in the cache are updated based on the pruning result and the scheduling threshold is adjusted, and the index block number meeting the condition is stored in the priority queue to map the corresponding index to the PEA processing unit array for execution, including:

[0018] The minimum row number and index row number of each block index stored in the cache of the X:M index are updated according to the index block generated in the current round, and the scheduling threshold is updated according to the current round number;

[0019] The index block number with a minimum index number less than the scheduling threshold is stored in the first priority queue, and the index block number with an index row number not less than the row number threshold is stored in the second priority queue;

[0020] The corresponding index is mapped to the corresponding PEA processing unit array according to the index number in the priority queue.

[0021] To achieve the above purpose, another aspect of the application provides an X:M dynamic attention sparsification device, which is integrated into an ASIC chip, and the modules on the ASIC chip include a prediction array, an N:M selector, an X:M generator, a block scheduler, a PEA, a QKV cache, an output cache, an N index cache, and an X index cache; wherein,

[0022] The prediction array performs 1:2 / 1:4 sparsification on the Q matrix, and generates prediction attention scores using a Q fixed pulsation array;

[0023] The N:M selector and the X:N generator generate X:M sparse indexes according to the prediction attention scores, and store them in the N index cache and the X index cache;

[0024] The block scheduler reads sparse indexes from the N index cache and the X index cache according to a scheduling algorithm, determines the execution order of each round of sparse indexes, and maps the sparse indexes to the PEA processing unit array;

[0025] The PEA reads the data of the Q, K, and V matrices from the QKV cache, and performs SDDMM and SpMM calculations to store the calculation results in the output cache.

[0026] The X:M dynamic attention sparsification method and device of the embodiment of the application is oriented to the sparse attention acceleration task in the Transformer, greatly reduces the storage and calculation overhead of attention prediction through a novel dynamic X:M sparse format, and combines X:M with a scheduling algorithm, so that the processing units (PEs) in an application-specific integrated circuit (ASIC) can maintain a very high utilization rate under different sparsities.

[0027] Additional aspects and advantages of the application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0028] The above and / or additional aspects and advantages of the application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0029] Figure 1 is a flowchart of the X:M dynamic attention sparsification method according to the embodiment of the application;

[0030] Figure 2 is a schematic diagram of a sparse attention accelerator that flexibly supports different sparsities according to the embodiment of the application;

[0031] Figure 3 is a schematic diagram of a block scheduler according to the embodiment of the application;

[0032] Figure 4 is a schematic diagram of the PEA and the internal data path of the PE according to the embodiment of the application;

[0033] Figure 5is a sparse attention calculation schematic diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0035] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0036] The X:M dynamic attention sparsification method and device according to the embodiments of the present application will be described below with reference to the accompanying drawings.

[0037] Figure 1 is a flowchart of an X:M dynamic attention sparsification method according to an embodiment of the present application, as shown in Figure 1 , the method comprises:

[0038] S1, predicting the attention score matrix, and performing X:M structured pruning based on the prediction result to obtain a pruning result.

[0039] In an embodiment of the present application, fine-grained structured pruning is performed on the original Q matrix to generate a sparse low-bit Q matrix and related index information; a sparse low-bit Q matrix is multiplied with a K matrix by using a 4-bit multiplication array to generate a low-precision attention score matrix; N:M pruning and X:N pruning are performed by using the generated low-precision score matrix and the corresponding index information to obtain a sparse attention score matrix, and a final index block is generated according to the pruned attention score matrix, that is, a pruning result; wherein each index block contains the sparse score of this segment and the corresponding index information.

[0040] In particular, the present application predicts the use of 4-bit multiplication array implementation, but this is still a part of the huge overhead. In order to reduce the pre-computation cost of the fraction, the present application also introduces a 1:2 / 1:4 fine-grained structured pruning for low-bit Q matrix, including N:M pruning and X:N pruning, to reduce the number of low-bit PEs required. After the prediction score is generated, the present application proposes a two-step X:M structured pruning method to solve the challenge of prediction memory overhead when using threshold-based pruning. In the N:M pruning step, the present application performs Top-N sorting and selects N largest scores from M scores, which can be quickly implemented because the group size (M) is relatively small. After that, the present application does not store N largest scores as in the traditional N:M pruning, but only stores the relative index within the group corresponding to the descending order of the score, which can significantly reduce the memory occupancy, because the bit width of the relative index (e.g., 6 bits) is much smaller than the bit width of the score (e.g., 12 bits). In addition, the present application also stores the partial exponent sum of the M scores of each group to facilitate further pruning. After the generation of the entire row of scores, the present application starts to perform X:N pruning to further improve the sparsity.

[0041] S2, update the minimum row number and index row number in the cache based on the pruning result and adjust the scheduling threshold, and store the index block number that meets the condition into the priority queue to map the corresponding index to the PEA processing unit array for execution.

[0042] In an embodiment of the present application, the minimum row number and index row number of each block index stored in the cache where the X:M index is located are updated according to the index block generated in the current round, and the scheduling threshold is updated according to the current round number; the index block number with the minimum index number less than the scheduling threshold is stored into the first priority queue, and the index block number with the index row number not less than the row number threshold is stored into the second priority queue; the corresponding index is mapped to the corresponding PEA processing unit array according to the index number in the priority queue.

[0043] In particular, after X:M pruning, the height of the attention score matrix will change, in order to make it adapt to PEA, the present application proposes a block scheduling algorithm to schedule sparse index blocks according to the size stored on-chip. First, the minimum row number and index row number of each block index stored in the cache where the X:M index is located are updated according to the index block generated in the current round, and the scheduling threshold is updated according to the current round number. Then, the index block number with the minimum index number less than the scheduling threshold is stored into the first priority queue, and the index block number with the index row number not less than H is stored into the second priority queue. Finally, the corresponding index is mapped to the corresponding PEA for execution according to the index number in the priority queue.

[0044] The X:M dynamic attention sparsity method of this invention accelerates sparse attention through X:M fine-grained structured pruning. This invention improves the regularity of attention scores through fine-grained structured pruning of X:M, and combines a scheduling algorithm to reorganize score blocks into hardware blocks that perfectly match the size of the Processing Unit Array (PEA) for SDDMM and SpMM operations, ultimately achieving extremely high PE utilization.

[0045] To implement the above embodiments, this embodiment also provides an X:M dynamic attention sparse device. The device integrates an ASIC chip, and the modules on the ASIC chip include a prediction array, an N:M selector, an X:M generator, a block scheduler, a PEA, a QKV cache, an output cache, an N-index cache, and an X-index cache; wherein...

[0046] The prediction array is 1:2 / 1:4 sparsed on the Q matrix, and a Q-fixed systolic array is used to generate prediction attention scores.

[0047] The N:M selector and the X:N generator generate an X:M sparse index based on the predicted attention score and store it in the N index cache and the X index cache.

[0048] The block scheduler reads sparse indexes from the N-index cache and X-index cache according to the scheduling algorithm, determines the execution order of sparse indexes in each round, and maps the sparse indexes to the PEA processing unit array.

[0049] PEA reads the Q, K, and V matrix data from the QKV buffer and performs SDDMM and SpMM calculations to store the results in the output buffer.

[0050] Furthermore, such as Figure 2 As shown, firstly, the prediction array is sparsed on Q in a 1:2 / 1:4 ratio, and a fixed-systolic array of Q is used to generate prediction attention scores, reducing the number of multipliers by half and lowering startup latency. Then, an X:M sparse index is generated using an N:M selector and an X:N generator, and stored in the index cache. Secondly, to enable efficient mapping of the sparse index to the hardware array, a block scheduler is designed based on a scheduling algorithm to determine the execution order of each round of sparse indexing. Finally, the PEA retrieves data from the QKV cache to perform SDDMM and SpMM calculations, and stores the results in the output cache. Since the positions of the X sparse indices within a block of M elements are random, this invention uses a unidirectional systolic PEA to avoid the overhead of Q position selection, further improving the accelerator's energy efficiency.

[0051] After the prediction attention score of each row is generated, it is streamed into the corresponding N:M selector and accumulator simultaneously. Each group of selectors consists of N comparators, each of which is used to retain the maximum value and the corresponding index of the incoming stream, and the position index is generated by the counter in the head of the selector. Each group of accumulators contains two registers, which store the M partial sum and the corresponding row sum, respectively, for subsequent X:M index generation. After every M elements are streamed in, the index in the selector and the partial sum in the accumulator are stored in the N index buffer. The width of the N index buffer is H groups, and the depth is G, which can accommodate 1 batch of N:M pruning results, where each group includes a 24-bit partial sum and N 5 / 6-bit intra-group indexes. After the prediction of a round of attention scores is completed, the N:M index generated in this round is streamed into the X:N generator along with the corresponding M partial sum and row sum, and the X value of the current N:M index is determined according to the threshold. Finally, the indexes with larger X values are split, and the indexes with X value of 0 are skipped to generate the final X:M index format.

[0052] As shown in Figure 3 , the hardware of the block scheduler consists of a forced hardware block detector, a full hardware block detector, a counter, two first-in-first-out queues (FIFO0 and FIFO1), and some multiplexers (MUXs). First, the counter iterates from 0 to G-1 to generate FIFO IDs. For each FIFO ID, the forced hardware block detector compares the corresponding block count with the maximum allowed block number to generate the write enable signal (WE0) of FIFO0, which has higher priority. At the same time, the full hardware block detector compares the corresponding group count with the block height (H) to generate the write enable signal (WE1) of the issuing FIFO1. If WE0 is valid, this FIFO ID will be pushed into the issuing FIFO0. Otherwise, if WE1 is valid, this FIFO ID will be pushed into the issuing FIFO1. If neither is valid, no FIFO ID is pushed in. If FIFO0 becomes non-empty, FIFO IDs will be read from it first, and the corresponding hardware block will be read from the corresponding X-FIFO for further execution. If FIFO0 is empty and FIFO1 is non-empty, FIFO1 will be read for further processing.

[0053] As shown in Figure 4 , (a) in Figure 4 is a PEA schematic diagram; Figure 4 (b) is a PE data path of the first-stage SDDMM; Figure 4 (c) is a PE data path for exponential operation of S; Figure 4 (d) is a PE data path of the second-stage SpMM.

[0054] PEA consists of H*(N / 2) reconfigurable PEs, H adder trees and H rows of registers for latching K or V. Through these register banks, K / V groups flow into PEA in a pulsatile manner. For Q, since PEA working on SDDMM adopts output (fractional) fixed data flow, PEs in the same row will consume the same Q value, thus a query is sent from a ping-pong Q buffer to N / 2 PEs within the same row in broadcast form, saving the position control overhead on Q. The invention does not need to consider the scalability problem of broadcast because the width of PEA is relatively small (4 or 8). In order to obtain partial sums quickly, each row contains an adder tree for adding the products of a row. In the first stage of SDDMM, based on the index generated by the block scheduler, PEs select a K from the register bank, multiply it with Q, and finally store the partial sum into the fractional register. In the second stage, it performs the exponential operation on the partial sum and updates the fractional register with the generated result. In the last stage of SpMM, PEs select a V from the register bank, multiply it with the latched exponential fraction, and store the product into the output register.

[0055] Further, as shown in the figure, the invention is based on X:M pruning algorithm and scheduling algorithm and combined with the hardware architecture of the corresponding accelerator, and the following is an embodiment of a specific application scenario of the invention: Figure 5

[0056] Step one: the dense low-bit Q of this round is input into the prediction array for 1:2 / 1:4 pruning and generates the corresponding sparse index, and then the low-bit Q that is not pruned and the index are stored into the prediction PE for subsequent Q-fixed matrix multiplication.

[0057] Step two: the low-bit K block flows into the prediction array and performs matrix multiplication with Q in the PE to generate the corresponding low-bit S matrix of the partial, and the S matrix immediately flows into the ordered N:M selector and partial sum accumulator after being generated.

[0058] Step three: the ordered N:M selector immediately prunes a group of S of each row in N:M for each block of S prediction in step two, a software block is generated for each block of S, and the accumulator also calculates the partial sum of the group of S, and the pruning index and the partial sum are stored into the N index cache. When a round of prediction in step two is completed, a round of step three is also completed at the same time.

[0059] Step four: when step two and step three are completed, X:N pruning is started. Each group of N:M index and partial sum flows into the X:N generator, and the generator determines the specific value of the group of X according to the threshold and sequence length set in advance. For the group of X less than N, the redundant index will be discarded to further save space, and because N:M has been sorted, the subsequent index can be directly truncated.

[0060] ​Step five: for the index generated in step four, the generator will determine whether the current group needs to be split and skipped according to the specific size of X, and generate the corresponding row ID for it and store it in the X index cache. Specifically, the group of X greater than N / 2 (X=N) will be split into two groups, and the two groups have the same row number. The group less than N / 2 (X=0) will be skipped.

[0061] Step six: when the final X:M index is generated, the block scheduler will schedule according to the index information in the X index cache. Specifically, the scheduler will preferentially schedule the index block that reaches the maximum round size, regardless of whether it reaches the hardware height, and then the block whose height is sufficient for the size of the hardware block.

[0062] Step seven: after block scheduling, the index is mapped to PEA for SDDMM and SpMM. In the SDDMM stage, Q flows into all PEs in the same row through broadcast, K flows into the register group through pulsation, and each PE selects Q*K through the index. The result S of Q*K exists in the PE for subsequent calculation. When Q*K is completed, S performs exponential operation in the PE and is accumulated through the adder tree for the final softmax calculation. Then in the SpMM stage, V and K flow into the PE in the same way, and are multiplied by the exponential result of S in the PE. The result is accumulated through the adder tree. The partial sum of the final input is generated and stored in the partial sum cache for accumulation with the partial sum generated in the subsequent round to generate the final output.

[0063] The X:M dynamic attention sparse device according to the embodiments of the present application improves the regularity of the attention score through X:M fine-grained structured pruning, and reorganizes the score block into a hardware block that can perfectly match the size of the processing unit array (PEA) through a scheduling algorithm, so as to perform SDDMM and SpMM operations, and finally realize extremely high PE utilization.

[0064] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0065] Furthermore, the terms "first", "second", "third", "fourth", "fifth" and "sixth" are used herein for descriptive purposes only and are not to be construed as indicating or implying relative importance or a significant nature of so described technical features. It is to be understood that a technical feature described with the "first", "second", "third", "fourth", "fifth" or "sixth" can implicitly or explicitly include at least one of the technical features described with the "first", "second", "third", "fourth", "fifth" or "sixth". In the description of the present application, the meaning of "a plurality" is at least two, for example, two, three, etc., unless otherwise specifically defined.

Claims

1. An X:M dynamic attention sparsity method, characterized in that, include: The attention score matrix is ​​predicted, and X:M structured pruning is performed based on the prediction results to obtain the pruning results; Based on the pruning results, the minimum row number and index row number in the cache are updated and the scheduling threshold is adjusted. The index block numbers that meet the conditions are stored in the priority queue so that the corresponding indexes can be mapped to the PEA processing unit array for execution.

2. The method according to claim 1, characterized in that, The attention score matrix is ​​predicted, and X:M structured pruning is performed based on the prediction results to obtain the pruning results, including: Fine-grained structure pruning is performed on the original Q matrix to generate a sparse low-bit Q matrix and related index information; A low-precision attention score matrix is ​​generated by multiplying the sparsed low-bit Q matrix with the K matrix using a 4-bit multiplication array. The generated low-precision attention score matrix and its corresponding index information are used to perform N:M pruning and X:N pruning to obtain a sparse attention score matrix, and the final index block is generated based on the sparse attention score matrix; wherein each index block contains the sparse score and its corresponding index information.

3. The method according to claim 2, characterized in that, Based on the pruning results, the minimum row number and index row number in the cache are updated and the scheduling threshold is adjusted. The index block numbers that meet the conditions are stored in the priority queue so that the corresponding indexes are mapped to the PEA processing unit array for execution, including: Update the minimum row number and number of index rows of each block index stored in the cache where the X:M index is located based on the index blocks generated in the current round, and update the scheduling threshold based on the current round number; Store the index block numbers whose minimum row number is less than the scheduling threshold into the first priority queue, and store the index block numbers whose number of index rows is not less than the row number threshold into the second priority queue. The corresponding index is mapped to the corresponding PEA processing unit array based on the index block number in the priority queue.

4. An X:M dynamic attention sparse device, characterized in that, The device integrates an ASIC chip, and the modules on the ASIC chip include a prediction array, an N:M selector, an X:M generator, a block scheduler, a PEA, a QKV cache, an output cache, an N-index cache, and an X-index cache; wherein, The prediction array is sparsed by the Q matrix in a 1:2 / 1:4 ratio and a Q-fixed systolic array is used to generate prediction attention scores. The N:M selector and the X:N generator generate an X:M sparse index based on the predicted attention score and store it in the N index cache and the X index cache. The block scheduler reads sparse indexes from the N-index cache and X-index cache according to the scheduling algorithm, determines the execution order of each round of sparse indexes, and maps the sparse indexes to the PEA processing unit array. The PEA reads the data of the Q, K, and V matrices from the QKV cache and performs SDDMM and SpMM calculations to store the calculation results in the output cache.

5. The apparatus according to claim 4, characterized in that, Also used for: The predicted attention score for each row is input into the corresponding N:M selector and accumulator; each selector consists of N comparators to retain the maximum value of the incoming data and its corresponding index, and the position index is generated by the counter at the head of the selector; each accumulator contains two registers to store the M part and the sum of the corresponding row, respectively. After inputting M elements, the indices in the selector and the partial sums in the accumulator are stored in the N-index buffer; where the N-index buffer has a width of H groups and a depth of G, and each group includes a 24-bit partial sum and N 5 / 6-bit in-group indices; After a round of attention score prediction is completed, the N:M index generated in this round, the corresponding M part sum and row sum are input into the X:N generator, and the X value of the current N:M index is determined according to the threshold. Indexes with X greater than a preset threshold are split, and indexes with X equal to 0 are skipped to generate the final X:M index format.

6. The apparatus according to claim 4, characterized in that, The block scheduler hardware consists of a forced hardware block detector, a full hardware block detector, a counter, two first-in-first-out queues, and a multiplexer.

7. The apparatus according to claim 6, characterized in that, The forced hardware block detector compares the corresponding block count with the maximum allowed number of blocks to generate the write enable signal WE0 for FIFO0; The full hardware block detector compares the corresponding group count with the block height to generate a write enable signal WE1 for issuing FIFO1; If WE0 is valid, FIFO ID is pushed into FIFO0 for issuance; otherwise, if WE1 is valid, FIFO ID is pushed into FIFO1 for issuance. If both are invalid, no FIFO ID is pushed into any queue; If FIFO0 becomes non-empty, the FIFO ID is read first from FIFO0, and the corresponding hardware block is read from the corresponding X-FIFO; if FIFO0 is empty and FIFO1 is not empty, the FIFO ID is read from FIFO1.

8. The apparatus according to claim 4, characterized in that, The PEA consists of H*(N / 2) reconfigurable PEs, H adder trees, and H row registers; the register group and K / V group are input to the PEA in a pulsed manner.

Images