A sparse mask sharing mechanism based transformer hardware accelerator and a method for sparse neural network operation

Abstract

This invention discloses a Transformer hardware accelerator and a sparse neural network operation method based on a sparse mask sharing mechanism, belonging to the technical field of computation, inference, or counting. The accelerator includes a sparse weight storage module, a shared mask library storage unit, a decoding and addressing unit, a global control module, and a computation processing module. By dividing the Transformer weight matrix into sub-blocks and constructing a finite number of shared bitmap mask libraries during the pre-compilation stage, the sparse weight structure is decoupled into reusable shared masks and corresponding indices. During inference, the hardware reads the bitmap mask from the shared mask library as a gating signal based on the mask index, and directly extracts the effective elements aligned with non-zero weights from the input activations for multiplication and addition operations. This invention reduces hardware storage and control overhead and improves the inference efficiency and energy efficiency of the Transformer model by sharing the sparse mask structure among multiple weight sub-blocks.

Description

Technical Field

[0001] This invention belongs to the field of neural network accelerator design, specifically relating to a Transformer hardware accelerator based on a sparse mask sharing mechanism and a method for sparse neural network operations. Background Technology

[0002] With the development of deep learning technology, models represented by the Transformer architecture have been widely used in natural language processing and computer vision. The core computational power of these models is mainly consumed in QKV generation, multi-head attention mechanisms, and large-scale matrix multiplication and accumulation operations in feedforward networks. This places high demands on the computational throughput, on-chip storage capacity, and storage bandwidth of hardware accelerators. Meanwhile, facing the explosive growth in the size of model parameters and the accompanying storage and energy consumption bottlenecks, model compression techniques—especially sparse pruning—have become a key path to solving this contradiction.

[0003] While sparsity theoretically reduces computational cost significantly, existing sparsity solutions still have significant shortcomings in practical deployments on existing regular hardware architectures. To describe the positional distribution of non-zero elements in sparse weights, existing sparsity acceleration solutions typically require the introduction of a large amount of auxiliary data (such as index arrays in COO or CSR formats, bitmap masks, etc.) to record non-zero positions, indices, or structural information—that is, sparse metadata. As the model size and the number of weight blocks increase, the sparse metadata used to describe the sparse structure occupies a large proportion of the overall storage, which not only significantly compresses the storage space of effective weights but also increases the proportion of invalid payload (non-computational data) in memory access bandwidth, reducing the utilization efficiency of hardware storage and bandwidth resources.

[0004] Furthermore, existing methods generally treat the sparse structure of each weighted sub-block as an independent entity for compression encoding, ignoring the inherent correlation and representative differences in sparse distribution characteristics among different weighted sub-blocks in the Transformer model. Due to the lack of comprehensive analysis of the importance and distribution characteristics of the sparse structure, even if some weighted sub-blocks share strong commonalities in their sparse structure distribution or have low importance, their corresponding sparse structure description information still needs to be stored separately. This introduces significant redundancy at the storage level and increases the complexity of hardware in managing, scheduling, and loading sparse metadata. This redundancy not only increases the overall storage overhead but also further burdens the hardware in managing and scheduling sparse metadata.

[0005] During the inference execution phase, the high diversity and irregularity of sparse metadata further constrain the data flow efficiency of the hardware. The hardware needs to frequently load and parse the sparse metadata corresponding to different weighted sub-blocks to generate input filtering and computation enable signals, thereby increasing the proportion of non-computational data in memory access bandwidth. At the same time, due to the inconsistent complexity of decoding and control logic corresponding to different sparse structures, the sparse decoding latency fluctuates with the sparse pattern, disrupting the original fixed-cycle and pipeline structure in the regular computation array, easily introducing pipeline stalls or array idling, thus limiting the hardware parallelism and overall energy efficiency.

[0006] In summary, existing Transformer sparse acceleration solutions exhibit significant redundancy and efficiency bottlenecks in the storage, access, and scheduling of sparse metadata, making it difficult to achieve efficient sparse computation execution while maintaining the hardware's rule-based computation characteristics. Therefore, a new hardware-software co-processing technology is urgently needed to reduce the additional storage and control overhead caused by sparse metadata while maintaining the stability of the data flow in the computing array, thereby improving hardware resource utilization efficiency and overall system energy efficiency. Summary of the Invention

[0007] This invention addresses the problems existing in the prior art by providing a Transformer hardware accelerator based on a sparse mask sharing mechanism and a method for sparse neural network operations. This method can effectively maintain the stable transmission of data flow within the systolic array while maintaining sparse computing power, thereby improving array parallelism and overall execution efficiency.

[0008] To solve the above technical problems, the present invention provides the following technical solution: a Transformer hardware accelerator based on a sparse mask sharing mechanism, comprising: a sparse weight storage module, a shared mask library storage unit, a computation processing module, an output buffer module, a nonlinear processing module, and a global control module;

[0009] A sparse weight storage module is configured to store sparse weight data packets; each sparse weight data packet corresponds to a weight sub-block and includes a mask index field and a data field; the mask index field stores the mapping address of the current weight sub-block in the shared mask set; the data field stores the non-zero weight sequence retained after the sub-block is sparsified;

[0010] A shared mask library storage unit is configured to store a shared set containing several predefined bitmap masks, which are used to characterize the sparse structure of sub-blocks of the weight matrix;

[0011] The decoding and addressing unit is connected to the sparse weight storage module and the shared mask library storage unit, respectively. It is configured to parse the sparse weight data packet, output the non-zero value sequence, and read the corresponding shared bitmap mask from the shared set using the mask index. The decoding and addressing unit realizes table lookup addressing of the shared bitmap mask through a fixed-width mask index, so that the control logic and timing of the decoding process are independent of the specific sparse structure.

[0012] The computation processing module includes an array of processing units consisting of multiple processing units. The array of processing units is configured to receive input activation data, a non-zero weight sequence in a weighted sub-block, and a corresponding shared bitmap mask. The shared bitmap mask is configured to apply a uniform sparse structure constraint to the multiple weighted sub-blocks mapped to the shared bitmap mask, so that the multiple weighted sub-blocks form a non-zero weight sequence format with consistent length after sparsification. When processing different weighted sub-blocks, the processing unit maintains that the number of multiply-accumulate operation units and the computation timing do not change with the difference in sparse structure, thereby completing the computation within a fixed computation cycle.

[0013] An output buffer module, connected to the computation processing module, is used to receive the result matrix output by the computation processing module and concatenate them according to the operation window number;

[0014] A nonlinear processing module, connected to the output buffer module, is used to perform normalization or nonlinear activation processing on the spliced ​​output matrix.

[0015] The global control module coordinates the timing of each module and dynamically updates the addressing through streaming parsing mask indexes. It drives the hardware array to directly reuse the shared mask logic when processing sub-blocks with different weights, achieving seamless switching of sparse mode and reuse of hardware resources.

[0016] Furthermore, in the aforementioned computational processing module, the input activation data is mapped to the row dimension of the processing unit array, and the non-zero weight sequence and shared bitmap mask are mapped to the column dimension of the processing unit array. Each processing unit is cascaded via a data transmission path and performs aligned input activation and non-zero weight multiplication and addition operations under the control of the shared bitmap mask, so that the computational array maintains its pipeline structure, data timing, and execution path unchanged when switching weight sub-blocks.

[0017] Furthermore, the number of bitmap masks stored in the aforementioned shared mask library storage unit is less than the total number of weighted sub-blocks, enabling multiple weighted sub-blocks to be mapped to the same shared bitmap mask through the same mask index, thus realizing the sharing and reuse of sparse structures at the hardware level.

[0018] Furthermore, the aforementioned decoding and addressing unit parses the mask index field of multiple sparse weight data packets corresponding to different weight sub-blocks stored continuously in the input data stream according to the data stream order, and uses the mask index as a lookup index to read the shared bitmap mask from the shared mask library storage unit and output it to the calculation processing module, so that multiple weight sub-blocks can reuse the same sparse control logic during the calculation process.

[0019] Furthermore, in the aforementioned computation processing module, the number of non-zero weight positions indicated in each shared bitmap mask matches the number of parallel multiply-accumulate units in the processing unit, so that the weight sub-blocks mapped to the same shared bitmap mask can make full use of the parallel multiply-accumulate units during the computation process.

[0020] Furthermore, the aforementioned processing unit integrates a sparse sensing circuit: The sparse sensing circuit is configured to receive the shared bitmap mask and the input activation tensor output by the decoding and addressing unit, and use the shared bitmap mask as a hardware gating signal to perform bit-by-bit filtering on the input activation tensor, directly extracting the valid input elements aligned with the non-zero positions of the current weights, thereby realizing the sparse mapping of different input data streams using a single shared mask.

[0021] Furthermore, the aforementioned processing unit also includes a multiply-accumulate circuit cascaded with the sparse sensing circuit: The multiply-accumulate circuit is configured to receive the valid input elements and corresponding non-zero weight sequences output by the sparse sensing circuit, and under the control of a shared bitmap mask, perform multiply-accumulate operations on multiple valid elements to generate a partial sum, and iteratively update the partial sum with the historical values ​​in the internal accumulator to complete the matrix multiplication operation of the current weight sub-block; the iterative update process follows the sliding of the input data stream and is repeatedly executed for multiple sub-blocks along the matrix dimension direction until the traversal is completed, thereby obtaining the final output matrix elements.

[0022] The present invention also provides a method for implementing sparse neural network operations on a Transformer hardware accelerator based on a sparse mask sharing mechanism, the method comprising a pre-compilation stage and a hardware inference stage;

[0023] Pre-compilation stage: Offline build of a shared sparse pattern library for hardware parallel structure, and retraining to adapt model weights to a fixed sparse structure to generate a packaged sparse weight data packet.

[0024] Hardware inference stage: Load shared sparse mask and data packets, filter input activations in fixed sparse mode, perform regular multiplication and addition operations, and complete result recombination and output.

[0025] Furthermore, the aforementioned pre-compilation stage includes the following sub-steps:

[0026] Step P1: Divide the pre-trained Transformer model weight matrix into sub-blocks, and perform sparsification processing on each weight sub-block according to the parallel computing capability of the target hardware to generate a non-zero weight sequence and corresponding sub-block sparse mask that matches the hardware computing structure; Based on the distribution characteristics of the sub-block sparse mask, select several representative sparse structures and determine them as shared bitmap masks to build a shared mask library.

[0027] Step P2: Map each weight sub-block to the shared bitmap mask with the highest matching degree in the shared mask library, and under the premise of fixing the constraints of the shared bitmap mask, perform retraining or fine-tuning on the Transformer model to update the weight values; encapsulate the retrained non-zero weight sequence with the corresponding shared bitmap mask index to generate a sparse weight data package for loading in the hardware inference stage.

[0028] Furthermore, the aforementioned hardware inference phase includes:

[0029] Step S1: Load the shared mask library and sparse weight data package into the hardware storage module, and read the corresponding shared bitmap mask from the shared mask library according to the mask index, and inject the non-zero weight sequence and the shared bitmap mask into the computing array.

[0030] Step S2: Under the control of the shared bitmap mask, the input activation data is filtered and multiplied and added with the corresponding non-zero weight sequence to generate the partial sum corresponding to the sub-block of the output matrix;

[0031] Step S3: Iteratively accumulate the partial sum and write it into the output buffer, and complete the splicing of the output matrix according to the preset matrix block order. After passing through the nonlinear processing module, the inference result is output.

[0032] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0033] First, this invention proposes a Transformer hardware accelerator based on a sparse mask sharing mechanism. By introducing this mechanism, the sparse structure corresponding to multiple weighted sub-blocks is mapped to a finite number of shared bitmap masks. Only the corresponding mask index is stored for each weighted sub-block, thus replacing the traditional "sub-block-level independent sparse structure description" storage mode with an "index + shared mask" approach. This significantly reduces the size of the metadata used to describe the sparse structure, increases the proportion of on-chip storage space allocated to effective weight data, and effectively alleviates the problem of excessive storage resource consumption by sparse metadata in existing sparse acceleration solutions.

[0034] Secondly, this invention proposes a Transformer hardware accelerator based on a sparse mask sharing mechanism. Through hardware and software co-design, the computational units in the inference execution phase only need to access the corresponding shared mask based on the mask index to directly generate the enable control signals for input activation filtering and multiply-accumulate operations, without the need for parsing and dynamic scheduling of element-wise indexing or irregular sparse structures. By transforming sparse control into a unified and predictable mask control flow, sparse computation is equivalent to regular computation on the hardware side, thereby maintaining a fixed execution rhythm of the computation array, reducing control path complexity, avoiding pipeline stalls, and improving array parallelism and computational resource utilization.

[0035] Third, this invention proposes a Transformer hardware accelerator based on a sparse mask sharing mechanism. The shared mask is reused multiple times on-chip via a systolic array, reducing repeated loading and memory accesses of mask data and lowering the proportion of non-computational data in memory access bandwidth. Based on this, this invention improves computational throughput while effectively increasing the utilization efficiency of on-chip and off-chip storage bandwidth, reducing overall data migration energy consumption, and thus improving the overall energy efficiency of the system. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the architecture of the present invention.

[0037] Figure 2 This is a flowchart of the solution described in this invention. In the figure, (a) is a flowchart of the coding stage and (b) is a flowchart of the reasoning stage.

[0038] Figure 3 This is a schematic diagram illustrating the construction of the shared mask library in this invention.

[0039] Figure 4 This is a schematic diagram illustrating the matching process of the weighted submatrix to the shared library in this invention.

[0040] Figure 5 This is the weighted data packet structure of the present invention.

[0041] Figure 6 It is the sparse sensing unit of the processing unit of the present invention.

[0042] Figure 7 This is a schematic diagram of the data flow for matrix operations according to the present invention. Detailed Implementation

[0043] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0044] In this invention, various aspects of the invention are described with reference to the accompanying drawings, in which numerous illustrative embodiments are shown. Embodiments of the invention are not limited to those depicted in the drawings. It should be understood that the invention is implemented through any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.

[0045] like Figure 1 As shown, this embodiment provides a Transformer hardware accelerator based on a sparse mask sharing mechanism, including: a sparse weight storage module, a shared mask library storage unit, a decoding and addressing unit, a computation processing module, an output buffer module, a nonlinear processing module, and a global control module.

[0046] A sparse weight storage module is configured to store sparse weight data packets; each sparse weight data packet corresponds to a weight sub-block and includes a mask index field and a data field; the mask index field stores the mapping address of the current weight sub-block in the shared mask set; the data field stores the non-zero weight sequence retained after the sub-block is sparsified;

[0047] The shared mask library storage unit is configured to store a shared set containing several predefined bitmap masks, which are used to represent the sparse structure of the weight matrix sub-blocks. The number of bitmap masks stored in the shared mask library storage unit is less than the total number of weight sub-blocks, so that multiple weight sub-blocks can be mapped to the same shared bitmap mask through the same mask index, thereby realizing the sharing and reuse of sparse structures at the hardware level.

[0048] The decoding and addressing unit is connected to the sparse weight storage module and the shared mask library storage unit, respectively. It is configured to parse the sparse weight data packet, output the non-zero value sequence, and read the corresponding shared bitmap mask from the shared set using the mask index. The decoding and addressing unit does not parse the non-zero element position distribution inside the weight sub-block, nor does it generate element-wise sparse index information. The decoding and addressing unit realizes table lookup addressing of the shared bitmap mask through a fixed-width mask index, so that the control logic and timing of the decoding process are independent of the specific sparse structure.

[0049] Meanwhile, the decoding and addressing unit parses the mask index field of multiple sparse weight data packets corresponding to different weight sub-blocks stored continuously in the input data stream according to the data stream order, and uses the mask index as the lookup index to read the shared bitmap mask from the shared mask library storage unit and output it to the calculation processing module, so that multiple weight sub-blocks can reuse the same sparse control logic in the calculation process.

[0050] The computational processing module includes an array of processing units composed of multiple processing units. The processing unit array is configured to receive input activation data, non-zero weight sequences in weighted sub-blocks, and corresponding shared bitmap masks. The input activation data is stored in the computational input sub-blocks of the input buffer module. The shared bitmap mask is configured to apply a uniform sparse structure constraint to the multiple weighted sub-blocks mapped to it, ensuring that the multiple weighted sub-blocks form a non-zero weight sequence format of consistent length after sparsification. When processing different weighted sub-blocks, the processing units maintain the number of multiply-accumulate operation units and the computation timing without changing due to differences in sparse structure, thus completing the computation within a fixed computation cycle. The input activation data is mapped to the row dimension of the processing unit array, and the non-zero weight sequences and shared bitmap masks are mapped to the column dimension of the processing unit array. Each processing unit is cascaded via a data transmission path and performs aligned multiply-accumulate operations on the input activation and non-zero weights under the control of the shared bitmap mask, ensuring that the computational array maintains its pipeline structure, data cycle time, and execution path when switching weighted sub-blocks.

[0051] Furthermore, the number of non-zero weight positions indicated in each shared bitmap mask matches the number of parallel multiply-accumulate units in the processing unit, so that weight sub-blocks mapped to the same shared bitmap mask can make full use of the parallel multiply-accumulate units during the calculation process.

[0052] An output buffer module, connected to the computation processing module, is used to receive the output result matrix of the computation processing module and concatenate them according to the operation window number;

[0053] A nonlinear processing module, connected to the output buffer module, is used to perform normalization or nonlinear activation processing on the spliced ​​output matrix.

[0054] In a preferred embodiment, the processing unit integrates a sparse sensing circuit: The sparse sensing circuit is configured to receive a shared bitmap mask and an input activation tensor output by the decoding and addressing unit, and use the shared bitmap mask as a hardware gating signal to perform bit-by-bit filtering on the input activation tensor, directly extracting the valid input elements aligned with the current non-zero weight position, thereby realizing the sparse mapping of different input data streams using a single shared mask.

[0055] The processing unit also includes a multiply-accumulate circuit cascaded with the sparse sensing circuit: The multiply-accumulate circuit is configured to receive the valid input elements and corresponding non-zero weight sequences output by the sparse sensing circuit, and under the control of a shared bitmap mask, perform multiply-accumulate operations on multiple valid elements to generate a partial sum, and iteratively update the partial sum with the historical values ​​in the internal accumulator to complete the matrix multiplication operation of the current weight sub-block; the iterative update process follows the sliding of the input data stream and is repeatedly executed for multiple sub-blocks along the matrix dimension direction until the traversal is completed, thereby obtaining the final output matrix elements.

[0056] Figure 2 The implementation process of this invention in the pre-compilation and inference stages is illustrated. Figure (a) shows the encoding stage flow, and (b) shows the inference stage flow. Specifically, the accelerator implementation process is as follows:

[0057] Step 1: After constructing and pre-training the neural network based on the Transformer model architecture, the pre-trained weight matrix is ​​divided into sub-blocks. In this embodiment, the weight matrix size is 768×768, and the sub-block size is set to 8×8, thus dividing the weight matrix into 9216 weight sub-blocks. Subsequently, sparsification is performed on each weight sub-block, retaining only a predetermined number or proportion of weight elements, while setting the remaining weight elements to zero, to form sparse weight sub-blocks and their corresponding sparse masks. The predetermined number or proportion is set according to the parallel processing capability of the target hardware computing array, the number of multiply-accumulate operation units, and on-chip storage and bandwidth constraints, so that the number of non-zero weights after sparsification matches the scale of the hardware processing units.

[0058] After completing sub-block-level sparsity reduction, importance evaluation is performed on the sub-block sparse mask corresponding to each weighted sub-block. For example... Figure 3 As shown, each sub-block sparse mask is multiplied one by one with each weighted sub-block in the weight matrix, and the calculation results are statistically processed to obtain an index characterizing the suitability and representativeness of the sub-block sparse mask in the overall weight matrix. Sub-block sparse masks are screened based on the importance index, and several sub-block sparse masks with high importance are selected to form a candidate mask set. On this basis, the sparse structure distribution characteristics in the candidate mask set are statistically analyzed, and several masks that are representative in terms of positional distribution, non-zero element arrangement, and constraint characteristics are extracted. These masks are then identified as shared bitmap masks, and a shared mask library is constructed. This shared mask library is used to provide unified sparse control information for multiple weighted sub-blocks in the subsequent hardware inference stage.

[0059] Through the above processing, this invention no longer records the specific distribution of non-zero elements within each sub-block. Instead, it constructs a shared mask library containing only 512 sparse structures, thereby converging and mapping the sparse topology of up to 9216 sub-blocks to 512 predefined shared bitmap masks. Calculations show that, under the aforementioned 50% sparsity setting, this invention only needs to store a 9-bit index for each weighted sub-block and 512 shared masks, requiring only approximately 14KB to describe the sparse structure of the weight matrix. If the traditional independent bitmap format is used, describing the sparse structure of the same matrix requires approximately 72KB of storage space; this invention achieves compression improvement of more than 5 times. Compared to the common CSR format, this invention saves more than 25 times the metadata storage space. This index-based mapping mechanism not only achieves centralized management of sparse structure description information, significantly reducing metadata storage overhead, but also utilizes the finiteness and reusability of masks to provide a highly unified and regular control benchmark for hardware accelerators during the inference phase.

[0060] Step 2: For each weighted sub-block to be processed, perform projection matching calculations on it one by one with multiple shared bitmap masks in the shared mask library. Specifically, perform dot-matrix multiplication and summation operations on the weight value matrix in the weighted sub-block and the shared bitmap mask. This operation is equivalent to using the shared bitmap mask as a gate to extract the weight elements at corresponding positions in the weighted sub-block and calculate their absolute sum. The result is the matching degree of the shared bitmap mask relative to the current weighted sub-block. Based on the matching degree comparison results, select the shared bitmap mask with the highest matching degree (i.e., the one that can retain the most important weight information in the sub-block) as the target mask corresponding to the weighted sub-block, and record the index of the target mask in the shared mask library. The process is as follows: Figure 4 As shown, under the premise of a fixed shared mask library and the mapping relationship between each weight sub-block and the target mask remaining unchanged, the Transformer model is retrained and fine-tuned. During the retraining process, the target mask is treated as a non-trainable hard constraint: in forward propagation, only the weight contribution at the corresponding position of the target mask is calculated; in backpropagation, gradient updates at non-mask positions are forcibly blocked. Through this process, the model parameters are fine-tuned under a fixed sparse topology, thereby compensating for the potential accuracy loss caused by the mask sharing constraint and achieving optimal fit between the weight numerical distribution and the selected shared mask.

[0061] After retraining, the non-zero weight sequence corresponding to each weight sub-block is encapsulated with its target mask index to generate a sparse weight data packet. Each sparse weight data packet corresponds to a weight sub-block and includes a mask index field and a data field. The mask index field stores the mapping address of the current weight sub-block in the shared mask library, but does not store the complete sparse structure information of the weight sub-block. The data field stores the non-zero weight sequence retained after sparsification of the weight sub-block.

[0062] Finally, the shared mask library and the generated sparse weight data package are loaded into the storage module of the hardware accelerator for use by the computation processing module in the inference stage, thereby completing the sparse computation of the corresponding weight sub-blocks based on the shared mask.

[0063] Step 3: The decoding and addressing unit sequentially reads the sparse weight data packets from the sparse weight storage module and parses the sparse weight data packets, such as... Figure 5 As shown, the mask index and non-zero weight sequence are obtained by separation. The decoding and addressing unit is internally configured with hardware addressing logic. After receiving the mask index, it is converted into a physical storage address in the shared mask library storage unit, thereby selecting and reading the corresponding shared bitmap mask.

[0064] After reading the shared bitmap mask, the decoding and addressing unit performs timing alignment between the non-zero weight sequence and the mask bitstream corresponding to the shared bitmap mask, and injects the aligned non-zero weight sequence and mask bitstream into the vertical input port of the computation processing module. Simultaneously, it injects the input activation data required for the current computation cycle into the horizontal input port of the computation processing module. This data loading provides aligned data input for subsequent processing unit arrays to perform efficient sparse multiply-accumulate operations under unified shared mask gating control.

[0065] Step 4: During the calculation process, the processing unit receives the shared bitmap mask bitstream corresponding to the current clock cycle and its corresponding non-zero weight sequence through the vertical port, and receives input activation data through the horizontal port. The shared bitmap mask is obtained by the decoding and addressing unit from the shared mask library storage unit according to the mask index. Although different weight sub-blocks load their respective shared masks sequentially during inference, the shared masks all originate from a predefined finite mask set, and their bit width, encoding format, and control semantics remain consistent.

[0066] The processing unit internally incorporates a sparse sensing circuit. This circuit uses the mask bits in the shared bitmap mask as a unified hardware gating signal to perform bitwise logic filtering on the horizontally input activation data. It extracts valid input activation elements from the input activation vector that are aligned with valid positions in the shared bitmap mask. Each valid input activation element corresponds one-to-one with a non-zero weight sequence received vertically and is loaded in parallel into multiple multiply-accumulate units within the processing unit.

[0067] After data loading is complete, the processing unit drives each multiply-accumulate unit to perform multiplication operations on the paired input activation elements and non-zero weights within the current computation cycle, and sends each multiplication result to the adder tree circuit for hierarchical aggregation, generating the partial sum corresponding to the current output matrix position within a predetermined computation time sequence. For example... Figure 6 The diagram shows the process of multiplying four selected active elements with non-zero weights. Since multiple weighted sub-blocks mapped to the same shared bitmap mask follow uniform constraints in the sparse structure, the processing unit does not need to dynamically adjust the number of multiply-accumulate units or reconstruct the control logic when switching calculations between different weighted sub-blocks. This eliminates the load imbalance problem in traditional sparse computing, resulting in a highly regular and deterministic timing delay throughout the entire computation process.

[0068] Step 5: The processing unit integrates a high-precision accumulator. Within each computation clock cycle, the accumulator reads the currently stored historical accumulated value, adds it to the current multiply-accumulate result generated in Step 4, and writes the updated result back to the accumulator register. This accumulation process continues as the input activation data pulsates through the processing unit array, repeatedly performing the "multiply-accumulate-update" operation on multiple weighted sub-blocks arriving sequentially along the matrix dimension, until all sub-blocks required for the current computation task have been traversed, thus completing the final calculation of the corresponding output matrix elements. For example... Figure 7 As shown, the computing array architecture employs a "row-based pulsation, column-based broadcasting" strategy: weights and the sparse mask of the current sub-block are broadcast in the column direction to achieve parallel processing. Furthermore, while the multiply-accumulate and accumulate operations are performed, the processing unit transmits the input activation data along the horizontal direction of the array through cascaded data paths established with adjacent processing units. Specifically, since multiple weight sub-blocks can be mapped to the same shared bitmap mask, this shared bitmap mask is continuously loaded and used multiple times within the processing unit array, allowing sparse control signals to be reused within the array. This significantly reduces the memory access bandwidth of the control signals while maintaining a unified control logic across the entire array.

[0069] In this way, the processing unit array uses a shared bitmap mask as a unified basis for sparsity control during inference, driving input activation filtering, multiplication and addition calculations, and data transmission in the array. This reconstructs the originally irregular sparse computation process into a regular, predictable, and pipeline bubble-free data flow execution mode at the hardware level.

[0070] Step 6: After the processing unit array completes the multiplication, addition, and accumulation operations of the corresponding weight sub-blocks, the output results of each processing unit are written to the intermediate layer cache. Since the weight matrix is ​​divided into regular sub-blocks during the pre-compilation stage, and the sparse weight data packets are arranged in the storage module according to a predefined matrix block order (which can be row-major or column-major), the loading and calculation process of the sub-blocks in the systolic array during the inference stage also strictly follows this order.

[0071] Under these conditions, the sub-matrix blocks output by the processing unit array correspond one-to-one with their logical positions in the output matrix in time sequence. When the accumulation calculation of a certain sub-matrix block is completed and output to the intermediate layer buffer, its corresponding output position is implicitly determined by the data flow sequence, without the need to add additional coordinate information, window number, or address index to the sub-matrix block. The intermediate layer buffer is equipped with a write pointer, which increments by a fixed step size according to a pre-set sub-matrix block size after each receipt of sub-matrix block data, thereby continuously writing the sequentially arriving sub-matrix block data to the corresponding positions in the output tensor memory, realizing the physical concatenation of the output matrix.

[0072] In this way, the output matrix concatenation process relies entirely on the data flow sequence and fixed-step addressing logic, avoiding dependence on complex address remapping, circuit-level coordinate calculations, or dynamic scheduling logic. The concatenated output data is stored in an intermediate layer cache and then sent to the nonlinear processing module. This module performs normalization or nonlinear activation operations on the output matrix to obtain the final inference output result.

[0073] This invention discloses a Transformer hardware accelerator based on a sparse mask sharing mechanism and its corresponding data flow execution method. By analyzing the sparse structure of weight sub-blocks during the pre-compilation stage and mapping the sparse structure of a large number of weight sub-blocks to a limited number of shared bitmap masks, the sparse control information can be recovered only through mask indexing during the hardware inference stage, thereby reducing the metadata storage requirements for describing the sparse structure. At the hardware execution level, this invention uses the shared bitmap mask as a unified basis for sparse control, driving input activation filtering, multiply-accumulate calculation, and data transmission within the processing unit array, enabling the sparse computation process to be executed in hardware in a regular and predictable data flow format. Combined with a mask-driven systolic array architecture, continuous transmission and multiplexing of weights, masks, and input activation data within the array are achieved, avoiding the complex decoding and dynamic scheduling logic introduced by structural differences in traditional sparse computation.

[0074] Figure 2 The implementation process of this invention in the pre-compilation and inference stages is demonstrated, and the specific process of its accelerator implementation is as follows:

[0075] Step 1: After constructing and pre-training the neural network based on the Transformer model architecture, the pre-trained weight matrix is ​​divided into sub-blocks. In this embodiment, the weight matrix size is 768×768, and the sub-block size is set to 8×8, thus dividing the weight matrix into 9216 weight sub-blocks. Subsequently, sparsification is performed on each weight sub-block, retaining only a predetermined number or proportion of weight elements, while setting the remaining weight elements to zero, to form sparse weight sub-blocks and their corresponding sparse masks. The predetermined number or proportion is set according to the parallel processing capability of the target hardware computing array, the number of multiply-accumulate operation units, and on-chip storage and bandwidth constraints, so that the number of non-zero weights after sparsification matches the scale of the hardware processing units.

[0076] After completing sub-block-level sparsity reduction, importance evaluation is performed on the sub-block sparse mask corresponding to each weighted sub-block. For example... Figure 3 As shown, each sub-block sparse mask is multiplied one by one with each weighted sub-block in the weight matrix, and the calculation results are statistically processed to obtain an index characterizing the suitability and representativeness of the sub-block sparse mask in the overall weight matrix. Sub-block sparse masks are screened based on the importance index, and several sub-block sparse masks with high importance are selected to form a candidate mask set. On this basis, the sparse structure distribution characteristics in the candidate mask set are statistically analyzed, and several masks that are representative in terms of positional distribution, non-zero element arrangement, and constraint characteristics are extracted. These masks are then identified as shared bitmap masks, and a shared mask library is constructed. This shared mask library is used to provide unified sparse control information for multiple weighted sub-blocks in the subsequent hardware inference stage.

[0077] Through the above processing, this invention no longer records the specific distribution of non-zero elements within each sub-block. Instead, it constructs a shared mask library containing only 512 sparse structures, thereby converging and mapping the sparse topology of up to 9216 sub-blocks to 512 predefined shared bitmap masks. Calculations show that, under the aforementioned 50% sparsity setting, this invention only needs to store a 9-bit index for each weighted sub-block and 512 shared masks, requiring only approximately 14KB to describe the sparse structure of the weight matrix. If the traditional independent bitmap format is used, describing the sparse structure of the same matrix requires approximately 72KB of storage space; this invention achieves compression improvement of more than 5 times. Compared to the common CSR format, this invention saves more than 25 times the metadata storage space. This index-based mapping mechanism not only achieves centralized management of sparse structure description information, significantly reducing metadata storage overhead, but also utilizes the finiteness and reusability of masks to provide a highly unified and regular control benchmark for hardware accelerators during the inference phase.

[0078] Step 2: For each weighted sub-block to be processed, perform projection matching calculations on it one by one with multiple shared bitmap masks in the shared mask library. Specifically, perform dot-matrix multiplication and summation operations on the weight value matrix in the weighted sub-block and the shared bitmap mask. This operation is equivalent to using the shared bitmap mask as a gate to extract the weight elements at corresponding positions in the weighted sub-block and calculate their absolute sum. The result is the matching degree of the shared bitmap mask relative to the current weighted sub-block. Based on the matching degree comparison results, select the shared bitmap mask with the highest matching degree (i.e., the one that can retain the most important weight information in the sub-block) as the target mask corresponding to the weighted sub-block, and record the index of the target mask in the shared mask library. The process is as follows: Figure 4 As shown, under the premise of a fixed shared mask library and the mapping relationship between each weight sub-block and the target mask remaining unchanged, the Transformer model is retrained and fine-tuned. During the retraining process, the target mask is treated as a non-trainable hard constraint: in forward propagation, only the weight contribution at the corresponding position of the target mask is calculated; in backpropagation, gradient updates at non-mask positions are forcibly blocked. Through this process, the model parameters are fine-tuned under a fixed sparse topology, thereby compensating for the potential accuracy loss caused by the mask sharing constraint and achieving optimal fit between the weight numerical distribution and the selected shared mask.

[0079] After retraining, the non-zero weight sequence corresponding to each weight sub-block is encapsulated with its target mask index to generate a sparse weight data packet. Each sparse weight data packet corresponds to a weight sub-block and includes a mask index field and a data field. The mask index field stores the mapping address of the current weight sub-block in the shared mask library, but does not store the complete sparse structure information of the weight sub-block. The data field stores the non-zero weight sequence retained after sparsification of the weight sub-block.

[0080] Finally, the shared mask library and the generated sparse weight data package are loaded into the storage module of the hardware accelerator for use by the computation processing module in the inference stage, thereby completing the sparse computation of the corresponding weight sub-blocks based on the shared mask.

[0081] Step 3: The decoding and addressing unit sequentially reads the sparse weight data packets from the sparse weight storage module and parses the sparse weight data packets, such as... Figure 5 As shown, the mask index and non-zero weight sequence are obtained by separation. The decoding and addressing unit is internally configured with hardware addressing logic. After receiving the mask index, it is converted into a physical storage address in the shared mask library storage unit, thereby selecting and reading the corresponding shared bitmap mask.

[0082] After reading the shared bitmap mask, the decoding and addressing unit performs timing alignment between the non-zero weight sequence and the mask bitstream corresponding to the shared bitmap mask, and injects the aligned non-zero weight sequence and mask bitstream into the vertical input port of the computation processing module. Simultaneously, it injects the input activation data required for the current computation cycle into the horizontal input port of the computation processing module. This data loading provides aligned data input for subsequent processing unit arrays to perform efficient sparse multiply-accumulate operations under unified shared mask gating control.

[0083] Step 4: During the calculation process, the processing unit receives the shared bitmap mask bitstream corresponding to the current clock cycle and its corresponding non-zero weight sequence through the vertical port, and receives input activation data through the horizontal port. The shared bitmap mask is obtained by the decoding and addressing unit from the shared mask library storage unit according to the mask index. Although different weight sub-blocks load their respective shared masks sequentially during inference, the shared masks all originate from a predefined finite mask set, and their bit width, encoding format, and control semantics remain consistent.

[0084] The processing unit internally incorporates a sparse sensing circuit. This circuit uses the mask bits in the shared bitmap mask as a unified hardware gating signal to perform bitwise logic filtering on the horizontally input activation data. It extracts valid input activation elements from the input activation vector that are aligned with valid positions in the shared bitmap mask. Each valid input activation element corresponds one-to-one with a non-zero weight sequence received vertically and is loaded in parallel into multiple multiply-accumulate units within the processing unit.

[0085] After data loading is complete, the processing unit drives each multiply-accumulate unit to perform multiplication operations on the paired input activation elements and non-zero weights within the current computation cycle, and sends each multiplication result to the adder tree circuit for hierarchical aggregation, generating the partial sum corresponding to the current output matrix position within a predetermined computation time sequence. For example... Figure 6 The diagram shows the process of multiplying four selected active elements with non-zero weights. Since multiple weighted sub-blocks mapped to the same shared bitmap mask follow uniform constraints in the sparse structure, the processing unit does not need to dynamically adjust the number of multiply-accumulate units or reconstruct the control logic when switching calculations between different weighted sub-blocks. This eliminates the load imbalance problem in traditional sparse computing, resulting in a highly regular and deterministic timing delay throughout the entire computation process.

[0086] Step 5: The processing unit integrates a high-precision accumulator. Within each computation clock cycle, the accumulator reads the currently stored historical accumulated value, adds it to the current multiply-accumulate result generated in Step 4, and writes the updated result back to the accumulator register. This accumulation process continues as the input activation data pulsates through the processing unit array, repeatedly performing the "multiply-accumulate-update" operation on multiple weighted sub-blocks arriving sequentially along the matrix dimension, until all sub-blocks required for the current computation task have been traversed, thus completing the final calculation of the corresponding output matrix elements. For example... Figure 7 As shown, the computing array architecture employs a "row-based pulsation, column-based broadcasting" strategy: weights and the sparse mask of the current sub-block are broadcast in the column direction to achieve parallel processing. Furthermore, while the multiply-accumulate and accumulate operations are performed, the processing unit transmits the input activation data along the horizontal direction of the array through cascaded data paths established with adjacent processing units. Specifically, since multiple weight sub-blocks can be mapped to the same shared bitmap mask, this shared bitmap mask is continuously loaded and used multiple times within the processing unit array, allowing sparse control signals to be reused within the array. This significantly reduces the memory access bandwidth of the control signals while maintaining a unified control logic across the entire array.

[0087] In this way, the processing unit array uses a shared bitmap mask as a unified basis for sparsity control during inference, driving input activation filtering, multiplication and addition calculations, and data transmission in the array. This reconstructs the originally irregular sparse computation process into a regular, predictable, and pipeline bubble-free data flow execution mode at the hardware level.

[0088] Step 6: After the processing unit array completes the multiplication, addition, and accumulation operations of the corresponding weight sub-blocks, the output results of each processing unit are written to the intermediate layer cache. Since the weight matrix is ​​divided into regular sub-blocks during the pre-compilation stage, and the sparse weight data packets are arranged in the storage module according to a predefined matrix block order (which can be row-major or column-major), the loading and calculation process of the sub-blocks in the systolic array during the inference stage also strictly follows this order.

[0089] Under these conditions, the sub-matrix blocks output by the processing unit array correspond one-to-one with their logical positions in the output matrix in time sequence. When the accumulation calculation of a certain sub-matrix block is completed and output to the intermediate layer buffer, its corresponding output position is implicitly determined by the data flow sequence, without the need to add additional coordinate information, window number, or address index to the sub-matrix block. The intermediate layer buffer is equipped with a write pointer, which increments by a fixed step size according to a pre-set sub-matrix block size after each receipt of sub-matrix block data, thereby continuously writing the sequentially arriving sub-matrix block data to the corresponding positions in the output tensor memory, realizing the physical concatenation of the output matrix.

[0090] In this way, the output matrix concatenation process relies entirely on the data flow sequence and fixed-step addressing logic, avoiding dependence on complex address remapping, circuit-level coordinate calculations, or dynamic scheduling logic. The concatenated output data is stored in an intermediate layer cache and then sent to the nonlinear processing module. This module performs normalization or nonlinear activation operations on the output matrix to obtain the final inference output result.

[0091] This invention discloses a Transformer hardware accelerator based on a sparse mask sharing mechanism and its corresponding data flow execution method. By analyzing the sparse structure of weight sub-blocks during the pre-compilation stage and mapping the sparse structure of a large number of weight sub-blocks to a limited number of shared bitmap masks, the sparse control information can be recovered only through mask indexing during the hardware inference stage, thereby reducing the metadata storage requirements for describing the sparse structure. At the hardware execution level, this invention uses the shared bitmap mask as a unified basis for sparse control, driving input activation filtering, multiply-accumulate calculation, and data transmission within the processing unit array, enabling the sparse computation process to be executed in hardware in a regular and predictable data flow format. Combined with a mask-driven systolic array architecture, continuous transmission and multiplexing of weights, masks, and input activation data within the array are achieved, avoiding the complex decoding and dynamic scheduling logic introduced by structural differences in traditional sparse computation.

[0092] While the present invention has been described above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A Transformer hardware accelerator based on a sparse mask sharing mechanism, characterized in that, include: The system includes a sparse weight storage module, a shared mask library storage unit, a computation processing module, an output buffer module, a nonlinear processing module, and a global control module. The sparse weight storage module is configured to store sparse weight data packets. Each sparse weighted data packet corresponds to a weighted sub-block and includes a mask index field and a data field; the mask index field stores the mapping address of the current weighted sub-block in the shared mask set; the data field stores the non-zero weight sequence retained after sparsification of the sub-block; A shared mask library storage unit is configured to store a shared set containing several predefined bitmap masks, which are used to characterize the sparse structure of sub-blocks of the weight matrix; The decoding and addressing unit is connected to the sparse weight storage module and the shared mask library storage unit, respectively, and is configured to parse the sparse weight data packet, output the non-zero value sequence, and read the corresponding shared bitmap mask from the shared set using the mask index; The decoding and addressing unit implements table lookup addressing of the shared bitmap mask through a fixed-width mask index, making the control logic and timing of the decoding process independent of the specific sparse structure; The computational processing module includes a processing unit array consisting of multiple processing units; the processing unit array is configured to receive input activation data, a non-zero weight sequence in a weighted sub-block, and a corresponding shared bitmap mask; The shared bitmap mask is configured to apply a uniform sparse structure constraint to multiple weighted sub-blocks mapped to the shared bitmap mask, so that the multiple weighted sub-blocks form a non-zero weight sequence format with consistent length after sparsification. When the processing unit processes different weighted sub-blocks, it keeps the number of multiply-accumulate operation units and the calculation timing unchanged with the difference in sparse structure, so as to complete the operation within a fixed calculation cycle. An output buffer module, connected to the computation processing module, is used to receive the result matrix output by the computation processing module and concatenate them according to the operation window number; A nonlinear processing module, connected to the output buffer module, is used to perform normalization or nonlinear activation processing on the spliced ​​output matrix. The global control module coordinates the timing of each module and dynamically updates the addressing through streaming parsing mask indexes. It drives the hardware array to directly reuse the shared mask logic when processing sub-blocks with different weights, achieving seamless switching of sparse mode and reuse of hardware resources.

2. The sparse mask sharing mechanism based Transformer hardware accelerator according to claim 1, wherein, In the computational processing module, the input activation data is mapped to the row dimension of the processing unit array, and the non-zero weight sequence and shared bitmap mask are mapped to the column dimension of the processing unit array. Each processing unit is cascaded through the data transmission path and performs aligned input activation and non-zero weight multiplication and addition operations under the control of the shared bitmap mask, so that the computational array maintains the pipeline structure, data tick and execution path unchanged when switching weight sub-blocks.

3. The sparse mask sharing mechanism based Transformer hardware accelerator according to claim 1, wherein, The number of bitmap masks stored in the shared mask library storage unit is less than the total number of weighted sub-blocks, enabling multiple weighted sub-blocks to be mapped to the same shared bitmap mask through the same mask index, thus realizing the sharing and reuse of sparse structures at the hardware level.

4. The sparse mask sharing mechanism based Transformer hardware accelerator according to claim 1, wherein, The decoding and addressing unit parses the mask index field of multiple sparse weight data packets corresponding to different weight sub-blocks stored continuously in the input data stream according to the data stream order, and uses the mask index as the lookup index to read the shared bitmap mask from the shared mask library storage unit and output it to the calculation processing module, so that multiple weight sub-blocks can reuse the same sparse control logic during the calculation process.

5. The sparse mask sharing mechanism based Transformer hardware accelerator according to claim 1, wherein, In the computation processing module, the number of non-zero weight positions indicated in each shared bitmap mask matches the number of parallel multiply-accumulate units in the processing unit, so that the weight sub-blocks mapped to the same shared bitmap mask can make full use of the parallel multiply-accumulate units during the computation process.

6. The sparse mask sharing mechanism based Transformer hardware accelerator according to claim 1, wherein, The processing unit integrates a sparse sensing circuit: The sparse sensing circuit is configured to receive a shared bitmap mask and an input activation tensor output by the decoding and addressing unit, and use the shared bitmap mask as a hardware gating signal to perform bit-by-bit filtering on the input activation tensor, directly extracting the valid input elements aligned with the non-zero positions of the current weights, thereby realizing the sparse mapping of different input data streams using a single shared mask.

7. The sparse mask sharing mechanism based Transformer hardware accelerator according to claim 1, wherein, The processing unit also includes a multiply-accumulate circuit cascaded with the sparse sensing circuit: The multiply-accumulate circuit is configured to receive the valid input elements and corresponding non-zero weight sequences output by the sparse sensing circuit, and under the control of a shared bitmap mask, perform multiply-accumulate operations on multiple valid elements to generate a partial sum, and iteratively update the partial sum with the historical values ​​in the internal accumulator to complete the matrix multiplication operation of the current weight sub-block; the iterative update process follows the sliding of the input data stream and is repeatedly executed for multiple sub-blocks along the matrix dimension direction until the traversal is completed, thereby obtaining the final output matrix elements.

8. A method for implementing a sparse neural network operation by a Transformer hardware accelerator based on the sparse mask sharing mechanism according to any one of claims 1-7, characterized in that, The method includes a pre-compilation phase and a hardware inference phase; Pre-compilation stage: Offline build of a shared sparse pattern library for hardware parallel structure, and retraining to adapt model weights to a fixed sparse structure to generate encapsulated sparse weight data package; Hardware inference stage: Load shared sparse mask and data packets, filter input activations in fixed sparse mode, perform regular multiplication and addition operations, and complete result recombination and output.

9. The method for implementing sparse neural network operation of the Transformer hardware accelerator based on the sparse mask sharing mechanism according to claim 8, characterized in that, The pre-compilation stage includes the following sub-steps: Step P1: Divide the pre-trained Transformer model weight matrix into sub-blocks, and perform sparsification processing on each weight sub-block according to the parallel computing capability of the target hardware to generate a non-zero weight sequence and corresponding sub-block sparse mask that matches the hardware computing structure. Based on the distribution characteristics of the sub-block sparse mask, several representative sparse structures are selected and determined as shared bitmap masks to construct a shared mask library. Step P2: Map each weight sub-block to the shared bitmap mask with the highest matching degree in the shared mask library, and under the premise of fixing the constraints of the shared bitmap mask, perform retraining or fine-tuning on the Transformer model to update the weight values; encapsulate the retrained non-zero weight sequence with the corresponding shared bitmap mask index to generate a sparse weight data package for loading during the hardware inference stage.

10. The method for implementing sparse neural network operation of the Transformer hardware accelerator based on the sparse mask sharing mechanism according to claim 9, characterized in that, The hardware inference phase includes: Step S1: Load the shared mask library and sparse weight data packet into the hardware storage module, and read the corresponding shared bitmap mask from the shared mask library according to the mask index, and inject the non-zero weight sequence and the shared bitmap mask into the computing array; Step S2: Under the control of the shared bitmap mask, the input activation data is filtered and multiplied and added with the corresponding non-zero weight sequence to generate the partial sum corresponding to the sub-block of the output matrix; Step S3: Iteratively accumulate the partial sum and write it into the output buffer, and complete the splicing of the output matrix according to the preset matrix block order. After passing through the nonlinear processing module, the inference result is output.

Images