X: M dynamic attention sparse method and device

Abstract

The invention discloses an X: M dynamic attention sparseness method and device, and the method comprises the steps: carrying out the prediction of an attention score matrix, and carrying out the X: M structured pruning based on a prediction result, so as to obtain a pruning result; and updating the minimum line number and the index line number in the cache based on a pruning result, adjusting a scheduling threshold, and storing the index block number meeting the condition into a priority queue so as to map the corresponding index to a PEA processing unit array for execution. According to the method, the storage and calculation overhead of attention prediction is greatly reduced, and meanwhile, X: M is combined with a scheduling algorithm, so that a processing unit in an application-specific integrated circuit can keep an extremely high utilization rate under different sparseness.

Description

Technical Field

[0001] The present invention relates to the technical field of sparse attention acceleration tasks, and in particular to an X:M dynamic attention sparse method and device. Background Art

[0002] The attention-based Transformer has performed well in many fields of artificial intelligence, such as natural language processing (NLP), computer vision, and multimodal tasks. In the field of NLP, large language models (LLMs) have shown amazing performance in various tasks, such as question answering, text generation, and logical reasoning assistance. The superior performance of LLMs mainly comes from the self-attention mechanism, which enables the model to capture global contextual information from the entire sequence. The self-attention block involves two matrix multiplication operations on three matrices, namely query (Q), key (K), and value (V). By multiplying Q and K, self-attention first obtains a score matrix (S), which is then normalized by a softmax function. Next, the normalized score matrix is ​​multiplied with V to generate the output. Therefore, self-attention has quadratic complexity and is computationally expensive, especially when processing tasks with long input sequences. Therefore, the self-attention block is considered to be the bottleneck of the Transformer for long sequences, resulting in difficulties in deployment on terminal devices.

[0003] In order to cope with the high cost of the self-attention mechanism when expanding to long sequences, sparse attention is a technique widely used in accelerators. Unlike the traditional dense Q and K to calculate the entire score matrix, sparse attention selects part of Q and K pairs for sampled dense-dense matrix multiplication (SDDMM), thereby producing a sparse score matrix. Then, sparse-dense matrix multiplication (SpMM) is performed by multiplying the normalized sparse score matrix with V. Existing sparse attention mechanisms mainly include static sparsity and dynamic sparsity. Static sparsity ignores the differences between different sequences by predetermining the sparsity pattern, resulting in limited computational savings at the same accuracy level. In contrast, dynamic sparsity dynamically generates a sparse pattern based on the input query and key, achieving a better trade-off between accuracy and computational savings. Therefore, the work of the present invention also focuses on dynamic sparsity.

[0004] However, dynamic sparsity also faces two challenges when adopting hardware accelerators. First, dynamic sparsity may learn arbitrary sparsity patterns that are usually highly unstructured. Due to the lack of regularity, it can cause workload imbalance and random memory access in SDDMM and SpMM operations, resulting in poor utilization of hardware resources. Second, when extending to long sequences in LLMs, it is inevitable to incur non-negligible overheads of predicting sparsity patterns, including prediction hardware overhead (e.g., computational and memory resources) and prediction latency overhead.

[0005] Existing dynamic sparse attention accelerators mainly use three sparsification methods. The first is row sorting selection, which dynamically sparsifies the entire row range of the attention matrix through row sorting. Since it involves selecting the entire row, it has high time complexity and is difficult to extend to longer sequences, and when this category is extended to multiple rows for row parallelization, it will inevitably suffer from heavy memory overhead for prediction. The second is global thresholding, which dynamically selects non-zero elements by comparing the attention score with the threshold. This accelerator only requires one round of comparison and therefore has low prediction complexity. However, since the attention scores of different rows share the same threshold, it requires a softmax operation to normalize the scores to probabilities for comparison, which involves saving the scores of the entire row in on-chip memory. When processing multiple rows at the same time, this pruning will incur large memory overhead due to prediction. In addition, this type of pruning method can generate arbitrary sparse patterns to achieve high sparsity, but this also leads to poor regularity, making it difficult to achieve high hardware utilization. The third is intra-block sorting selection, which performs value pruning by sorting values ​​within the block of the attention matrix. Since the generated sparsity patterns maintain regularity to a certain extent, it can be efficiently accelerated by hardware. However, since each block retains similar patterns, even those unimportant blocks, it can only achieve a moderate level of sparsity at the same accuracy level. Therefore, achieving a good balance between accuracy, sparsity, and hardware friendliness is a challenging task for sparse attention. Summary of the invention

[0006] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0007] In view of the problems existing in the current technology, this paper proposes an algorithm-hardware co-design framework to accelerate sparse attention through X:M fine-grained structured pruning. This paper proposes an X:M dynamic attention sparse method, which improves the regularity of attention scores through X:M fine-grained structured pruning, and reorganizes the score blocks into hardware blocks that can perfectly match the size of the processing unit array (PEA) in combination with the scheduling algorithm to perform SDDMM and SpMM operations, ultimately achieving extremely high PE utilization.

[0008] Another object of the present invention is to propose an X:M dynamic attention sparse device.

[0009] To achieve the above object, the present invention proposes an X:M dynamic attention sparse method, comprising:

[0010] Predict the attention score matrix and perform X:M structured pruning based on the prediction results to obtain the pruning results;

[0011] Based on the pruning result, the minimum row number and the number of index rows in the cache are updated and the scheduling threshold is adjusted, and the qualified index block numbers are stored in the priority queue to map the corresponding indexes to the PEA processing unit array for execution.

[0012] The X:M dynamic attention sparse method of the embodiment of the present invention may also have the following additional technical features:

[0013] In one embodiment of the present invention, the attention score matrix is ​​predicted, and X:M structured pruning is performed based on the prediction result to obtain a pruning result, including:

[0014] Perform fine-grained structured pruning on the original Q matrix to generate a sparse low-bit Q matrix and related index information;

[0015] Use a 4-bit multiplication array to multiply the sparse low-bit Q matrix with the K matrix to generate a low-precision attention score matrix;

[0016] The generated low-precision score matrix and the corresponding index information are used to perform N:M pruning and X:N pruning to obtain a sparse attention score matrix, and the final index block is generated based on the pruned attention score matrix; wherein each index block contains the sparse score and corresponding index information of the segment.

[0017] In one embodiment of the present invention, based on the pruning result, the minimum row number and the number of index rows in the cache are updated and the scheduling threshold is adjusted, and the qualified index block number is stored in the priority queue to map the corresponding index to the PEA processing unit array for execution, including:

[0018] Update the minimum row number and number of index rows of each block index stored in the cache where the X:M index is located according to the index block generated in the current round, and update the scheduling threshold according to the current round number;

[0019] The index block number whose minimum index number is less than the scheduling threshold is stored in the first priority queue, and the index block number whose index row number is not less than the row number threshold is stored in the second priority queue;

[0020] According to the index number in the priority queue, the corresponding index is mapped to the corresponding PEA processing unit array.

[0021] To achieve the above object, the present invention proposes an X:M dynamic attention sparse device on the other hand, the device integrates an ASIC chip, the modules on the ASIC chip include a prediction array, an N:M selector, an X:M generator, a block scheduler, a PEA, a QKV cache, an output cache, an N index cache and an X index cache; wherein,

[0022] The prediction array performs 1:2 / 1:4 sparsification on the Q matrix and generates predicted attention scores using the Q fixed systolic array;

[0023] The N:M selector and the X:N generator generate an X:M sparse index according to the predicted attention score and store it in the N index cache and the X index cache;

[0024] The block scheduler reads sparse indexes from the N index cache and the X index cache according to the scheduling algorithm, determines the execution order of each round of sparse indexes, and maps the sparse indexes to the PEA processing unit array;

[0025] The PEA reads the data of the Q, K, and V matrices from the QKV buffer, and performs SDDMM and SpMM calculations to store the calculation results in the output buffer.

[0026] The X:M dynamic attention sparse method and device of the embodiment of the present invention is aimed at the sparse attention acceleration task in Transformer. Through a novel dynamic X:M sparse format, the storage and computing overhead of attention prediction are greatly reduced. At the same time, X:M is combined with the scheduling algorithm, so that the processing element (PE) in the application-specific integrated circuit (ASIC) can maintain extremely high utilization under different sparsity.

[0027] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0029] Figure 1 is a flow chart of an X:M dynamic attention sparse method according to an embodiment of the present invention;

[0030] Figure 2 is a schematic diagram of a sparse attention accelerator that flexibly supports different sparsities according to an embodiment of the present invention;

[0031] Figure 3 is a schematic diagram of a block scheduler according to an embodiment of the present invention;

[0032] Figure 4 is a schematic diagram of a data path within a PEA and a PE according to an embodiment of the present invention;

[0033] Figure 5is a schematic diagram of sparse attention calculation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] The following describes the X:M dynamic attention sparse method and device proposed according to an embodiment of the present invention with reference to the accompanying drawings.

[0037] Figure 1 is a flow chart of the X:M dynamic attention sparse method according to an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0038] S1, predict the attention score matrix, and perform X:M structured pruning based on the prediction results to obtain the pruning results.

[0039] In one embodiment of the present invention, fine-grained structured pruning is performed on the original Q matrix to generate a sparse low-bit Q matrix and related index information; the sparse low-bit Q matrix is ​​multiplied with the K matrix using a 4-bit multiplication array to generate a low-precision attention score matrix; the generated low-precision score matrix and the corresponding index information are used to perform N:M pruning and X:N pruning to obtain a sparse attention score matrix, and the final index block, i.e., the pruning result, is generated based on the pruned attention score matrix; wherein each index block contains the sparse score of the segment and the corresponding index information.

[0040] Specifically, the present invention predicts that it is implemented using a 4-bit multiplication array, but this is still a huge overhead. In order to reduce the pre-calculation cost of the score, the present invention also introduces 1:2 / 1:4 fine-grained structured pruning for the low-bit Q matrix, including N:M pruning and X:N pruning, to reduce the number of low-bit PEs required. After the prediction score is generated, the present invention proposes a two-step X:M structured pruning method to address the challenge of predicting memory overhead when using threshold-based pruning. In the N:M pruning step, the present invention performs Top-N sorting and selects N maximum scores from M scores, which can be implemented quickly because the group size (M) is relatively small. Afterwards, the present invention does not store the N maximum scores like traditional N:M pruning, but only stores the relative index within the group corresponding to the scores in descending order, which can significantly reduce memory usage because the bit width of the relative index (e.g., 6 bits) is much smaller than the bit width of the score (e.g., 12 bits). In addition, the present invention also stores the partial exponential sum of the M scores of each group for further pruning. After the scores of the entire row are generated, the present invention starts to perform X:N pruning to further improve sparsity.

[0041] S2, based on the pruning result, updates the minimum row number and index row number in the cache and adjusts the scheduling threshold, and stores the qualified index block number into the priority queue to map the corresponding index to the PEA processing unit array for execution.

[0042] In one embodiment of the present invention, the minimum row number and the number of index rows of each block index stored in the cache where the X:M index is located are updated according to the index block generated in the current round, and the scheduling threshold is updated according to the current round number; the index block number whose minimum index number is less than the scheduling threshold is stored in the first priority queue, and the index block number whose index row number is not less than the row number threshold is stored in the second priority queue; the corresponding index is mapped to the corresponding PEA processing unit array according to the index number in the priority queue.

[0043] Specifically, after X:M pruning, the height of the attention score matrix will change. In order to make it compatible with PEA, the present invention proposes a block scheduling algorithm to perform sparse index block scheduling according to the size of on-chip storage. First, the minimum row number and number of index rows of each block index stored in the cache where the X:M index is located are updated according to the index block generated in the current round, and the scheduling threshold is updated according to the current round number. Then, the index block number with the minimum index number less than the scheduling threshold is stored in the first priority queue, and the index block number with the number of index rows not less than H is stored in the second priority queue. Finally, the corresponding index is mapped to the corresponding PEA for execution according to the index number in the priority queue.

[0044] The X:M dynamic attention sparse method of the embodiment of the present invention accelerates sparse attention through X:M fine-grained structured pruning. The present invention improves the regularity of attention scores through X:M fine-grained structured pruning, and reorganizes the score blocks into hardware blocks that can perfectly match the size of the processing unit array (PEA) in combination with the scheduling algorithm to perform SDDMM and SpMM operations, ultimately achieving extremely high PE utilization.

[0045] In order to implement the above embodiment, this embodiment also provides an X:M dynamic attention sparse device, which integrates an ASIC chip. The modules on the ASIC chip include a prediction array, an N:M selector, an X:M generator, a block scheduler, a PEA, a QKV cache, an output cache, an N index cache, and an X index cache; wherein,

[0046] The prediction array performs 1:2 / 1:4 sparseness on the Q matrix and uses the Q fixed systolic array to generate the predicted attention scores;

[0047] The N:M selector and the X:N generator generate an X:M sparse index according to the predicted attention score and store it in the N index cache and the X index cache;

[0048] The block scheduler reads the sparse index from the N-index cache and the X-index cache according to the scheduling algorithm, determines the execution order of each round of sparse index, and maps the sparse index to the PEA processing unit array;

[0049] PEA reads the data of Q, K, and V matrices from the QKV cache and performs SDDMM and SpMM calculations to store the calculation results in the output cache.

[0050] Furthermore, if Figure 2 As shown. First, the prediction array performs 1:2 / 1:4 sparseness on Q, and uses the Q fixed pulsation array to generate the predicted attention score, which reduces the multipliers by half and reduces the startup delay. Then, the X:M sparse index is generated by the N:M selector and the X:N generator, and stored in the index cache. Secondly, in order to enable the sparse index to map the hardware array efficiently, the present invention designs a block scheduler according to the scheduling algorithm to determine the execution order of each round of sparse indexing. Finally, PEA obtains data from the QKV cache to calculate SDDMM and SpMM, and stores the results in the output cache. Since the positions of the X sparse indexes within the block composed of M elements are random, the present invention uses the unidirectional pulsating PEA to avoid the position selection overhead of Q, further improving the energy efficiency of the accelerator.

[0051] After the predicted attention score of each row is generated, it flows into the corresponding N:M selector and accumulator at the same time. Each group of selectors consists of N comparators, each of which is used to retain the maximum value and corresponding index of the inflow, and the position index is generated by the counter in the head of the selector. Each group of accumulators contains two registers, which store the M partial sum and the corresponding row sum respectively, for subsequent X:M index generation. After each M elements flow in, the index in the selector and the partial sum in the accumulator are stored in the N index buffer. The width of the N index buffer is H groups and the depth is G, which can accommodate a batch of N:M pruning results, where each group includes a 24-bit partial sum and N 5 / 6-bit in-group indexes. When a round of attention score prediction is completed, the N:M index generated in this round and the corresponding M partial sum and row sum flow into the X:N generator, and the X value of the current N:M index is determined according to the threshold. Finally, the index with a larger X is split, and the index with X of 0 is skipped to generate the final X:M index format.

[0052] like Figure 3 As shown, the hardware of the block scheduler consists of a forced hardware block detector, a full hardware block detector, a counter, two first-in-first-out queues (FIFO0 and FIFO1), and some multiplexers (MUXs). First, the counter iterates from 0 to G-1 to generate FIFO IDs. For each FIFO ID, the forced hardware block detector compares the corresponding block count with the maximum number of blocks allowed to generate a write enable signal (WE0) for FIFO0, which has a higher priority. At the same time, the full hardware block detector compares the corresponding group count with the block high (H) to generate a write enable signal (WE1) for the issue FIFO1. If WE0 is valid, this FIFO ID will be pushed into the issue FIFO0. Otherwise, if WE1 is valid, this FIFO ID will be pushed into the issue FIFO1. If both are invalid, no FIFO ID is pushed. If FIFO0 becomes non-empty, the FIFO ID will be read from it with priority, and the corresponding hardware block will be read from the corresponding X-FIFO for further execution. If FIFO0 is empty and FIFO1 is non-empty, FIFO1 will be read for further processing.

[0053] like Figure 4 As shown, Figure 4 (a) is a schematic diagram of PEA; Figure 4 (b) in the figure is the PE data path of the first stage SDDMM; Figure 4 (c) in the figure is the PE data path for S to perform exponential operations; Figure 4 (d) in the figure is the PE data path of the second stage SpMM.

[0054] PEA consists of H*(N / 2) reconfigurable PEs, H adder trees, and H row registers, which are used to latch K or V. Through these register groups, the K / V group flows into the PEA in a pulsating manner. For Q, since the PEA of SDDMM operation adopts a data stream with a fixed output (fraction), the PEs in the same row will consume the same Q value, so a query is sent from a ping-pong Q buffer to N / 2 PEs in the same row, saving the position control overhead of Q in the form of broadcast. The present invention does not need to consider the scalability problem of broadcasting because the width of PEA is relatively small (4 or 8). In order to quickly obtain partial sums, each row contains an adder tree for adding the products of a row. In the first stage of SDDMM, based on the index generated by the block scheduler, the PE selects a K from the register group, multiplies it with Q, and finally stores the partial sum in the fraction register. In the second stage, it performs an exponential operation on the partial sum and updates the fraction register with the generated result. In the final stage of SpMM, the PE selects a V from the register bank, multiplies it with the latched exponential fraction, and stores the product into the output register.

[0055] Furthermore, if Figure 5 As shown, the present invention is based on the X:M pruning algorithm and the scheduling algorithm and combined with the corresponding accelerator hardware architecture. The following is an embodiment of the present invention in a specific application scenario:

[0056] Step 1: The dense low-bit Q of this round is input into the prediction array for 1:2 / 1:4 pruning and generates the corresponding sparse index. Then the unpruned low-bit Q and index are stored in the prediction PE for subsequent Q fixed matrix multiplication.

[0057] Step 2: The low-bit K blocks flow into the prediction array and perform matrix multiplication with Q in PE to generate the corresponding low-bit S matrix. After the S matrix is ​​generated, it immediately flows into the ordered N:M selector and partial sum accumulator.

[0058] Step 3: After each block of S prediction is completed in step 2, the ordered N:M selector immediately performs N:M pruning on a group of S in each row, generating a software block for each S. At the same time, the accumulator also calculates the partial sum of the group of S and stores the pruning index and the partial sum in the N index cache. When a round of prediction in step 2 is completed, a round of step 3 is also completed at the same time.

[0059] Step 4: When steps 2 and 3 are completed, X:N pruning begins. Each group of N:M indexes and partial sums flow into the X:N generator, which determines the specific value of X for the group based on the pre-set threshold and sequence length. For groups where X is less than N, the redundant indexes are directly discarded to further save space. Since N:M has been sorted, the subsequent indexes can be truncated directly.

[0060] Step 5: For the index generated in step 4, the generator will decide whether the current group needs to be split or skipped based on the specific size of X, and generate the corresponding row ID for it and store it in the X index cache. Specifically, the group with X greater than N / 2 (X=N) will be split into two groups with the same row number, and the group with less than N / 2 (X=0) will be skipped.

[0061] Step 6: After the final X:M index is generated, the block scheduler will schedule according to the index information in the X index cache. Specifically, the scheduler will prioritize the index blocks that reach the maximum round size, regardless of whether they reach the hardware height, followed by blocks with a height sufficient for the size of the hardware block.

[0062] Step 7: After block scheduling, the index will be mapped to PEA for SDDMM and SpMM. In the SDDMM stage, Q flows into all PEs in the same row through broadcasting, and K flows into the register group through pulsation. Each PE is then selected by index, and the result S of Q*K is stored in the PE for subsequent calculations. When Q*K is completed, S is exponentially operated inside the PE and accumulated through the adder tree for the final softmax calculation. Then the SpMM stage is performed, V and K flow into the PE in the same way, and are multiplied with the exponential result of S inside the PE, and the results are accumulated through the adder tree. The partial sum of the final input is generated and stored in the partial sum cache, which is used to accumulate with the partial sum generated in subsequent rounds to generate the final output.

[0063] According to the X:M dynamic attention sparse device of an embodiment of the present invention, the regularity of the attention score is improved through the fine-grained structured pruning of X:M, and the score blocks are reorganized into hardware blocks that can perfectly match the size of the processing element array (PEA) in combination with the scheduling algorithm to perform SDDMM and SpMM operations, ultimately achieving extremely high PE utilization.

[0064] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0065] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. An X:M dynamic attention sparse method, characterized in that: include: Predict the attention score matrix and perform X:M structured pruning based on the prediction results to obtain the pruning results; Based on the pruning result, the minimum row number and the number of index rows in the cache are updated and the scheduling threshold is adjusted, and the qualified index block numbers are stored in the priority queue to map the corresponding indexes to the PEA processing unit array for execution.

2. The method according to claim 1, characterized in that Predict the attention score matrix and perform X:M structured pruning based on the prediction results to obtain the pruning results, including: Perform fine-grained structured pruning on the original Q matrix to generate a sparse low-bit Q matrix and related index information; Use a 4-bit multiplication array to multiply the sparse low-bit Q matrix with the K matrix to generate a low-precision attention score matrix; The generated low-precision score matrix and the corresponding index information are used to perform N:M pruning and X:N pruning to obtain a sparse attention score matrix, and the final index block is generated based on the pruned attention score matrix; wherein each index block contains the sparse score and corresponding index information of the segment.

3. The method according to claim 2, characterized in that Based on the pruning result, the minimum row number and the number of index rows in the cache are updated and the scheduling threshold is adjusted, and the qualified index block number is stored in the priority queue to map the corresponding index to the PEA processing unit array for execution, including: Update the minimum row number and number of index rows of each block index stored in the cache where the X:M index is located according to the index block generated in the current round, and update the scheduling threshold according to the current round number; The index block number whose minimum index number is less than the scheduling threshold is stored in the first priority queue, and the index block number whose index row number is not less than the row number threshold is stored in the second priority queue; According to the index number in the priority queue, the corresponding index is mapped to the corresponding PEA processing unit array.

4. An X:M dynamic attention sparse device, characterized in that: The device integrates an ASIC chip, and the modules on the ASIC chip include a prediction array, an N:M selector, an X:M generator, a block scheduler, a PEA, a QKV cache, an output cache, an N index cache, and an X index cache; wherein, The prediction array performs 1:2 / 1:4 sparsification on the Q matrix and generates predicted attention scores using the Q fixed systolic array; The N:M selector and the X:N generator generate an X:M sparse index according to the predicted attention score and store it in the N index cache and the X index cache; The block scheduler reads sparse indexes from the N index cache and the X index cache according to the scheduling algorithm, determines the execution order of each round of sparse indexes, and maps the sparse indexes to the PEA processing unit array; The PEA reads the data of the Q, K, and V matrices from the QKV buffer, and performs SDDMM and SpMM calculations to store the calculation results in the output buffer.

5. The device according to claim 4, characterized in that Also used for: Input the predicted attention score of each row into the corresponding N:M selector and accumulator; each selector consists of N comparators, which are used to retain the maximum value of the inflow and its corresponding index. The position index is generated by the counter at the head of the selector; each accumulator contains two registers, which store the M part and the corresponding row sum respectively; After inputting M elements, the index in the selector and the partial sum in the accumulator are stored in an N index buffer; wherein the N index buffer has a width of H groups and a depth of G, where each group includes a 24-bit partial sum and N 5 / 6-bit intra-group indices; When a round of attention score prediction is completed, the N:M index generated in this round and the corresponding M partial sum and row sum are input into the X:N generator, and the X value of the current N:M index is determined according to the threshold; The indexes with larger X are split and the indexes with X equal to 0 are skipped to generate the final X:M index format.

6. The device according to claim 4, characterized in that The hardware of the block scheduler consists of a mandatory hardware block detector, a full hardware block detector, a counter, two first-in-first-out queues and a multiplexer.

7. The device according to claim 6, characterized in that The mandatory hardware block detector compares the corresponding block count with the maximum number of blocks allowed to generate a write enable signal WE0 of FIFO0; The full hardware block detector compares the corresponding group count with the block high to generate a write enable signal WE1 for issuing FIFO1; If WE0 is valid, FIFO ID is pushed into FIFO0, otherwise, if WE1 is valid, FIFO ID is pushed into FIFO1; If both are invalid, no FIFO ID is pushed into any queue; If FIFO0 becomes non-empty, the FIFO ID is read from FIFO0 first, and the corresponding hardware block is read from the corresponding X-FIFO; if FIFO0 is empty and FIFO1 is non-empty, the FIFO ID is read from FIFO1.

8. The device according to claim 4, characterized in that The PEA is composed of H*(N / 2) reconfigurable PEs, H addition trees and H rows of registers; the K / V group is input into the PEA in a pulsating manner through the register group.

Images