An adaptive data block mapping method and device for sparse matrix multiplication kernel

Abstract

The application discloses a kind of self-adapting data block mapping methods and devices for sparse matrix multiplication kernel, the method includes: partitioning for sparse matrix, according to the number of matrix sparsity and multiplier determines an initial matrix block scheme;Dynamic adjustment matrix block shape;Lightweight non-zero element index analysis is introduced, and the performance monitoring of random continuous matrix block in running time is combined, dynamic adjustment matrix block shape is determined optimal block configuration;Mapping based on matrix block.The device is used to execute the above method.The application has the advantages of simple principle, easy to implement, wide application range, good compatibility, can improve sparse matrix multiplication kernel calculation efficiency and hardware resource utilization, etc..

Description

Technical Field

[0001] This invention mainly relates to the field of multiplication accelerator technology, specifically an adaptive data block mapping method and apparatus for sparse matrix multiplication kernels. Background Technology

[0002] Sparsity is becoming increasingly important in contemporary scientific and machine learning applications. Sparse matrix multiplication (SpMSpM) is a key computational kernel in many algorithms, such as object detection and natural language processing, and a central concern in the field of custom accelerators. However, in many real-world scenarios, the structure of sparse matrices is highly irregular, especially with the advent of large-scale graph analysis and various model compression techniques in recent years. For example, many CNNs use the ReLU activation function to convert negative values ​​to 0, and pruning algorithms also introduce significant sparsity, in some cases removing more than 60% of convolutional layer weights and more than 90% of fully connected layer weights. Real-world graph datasets often exhibit extremely high sparsity and huge dimensionality, sometimes reaching 10^13 dimensions, with the number of non-zero values ​​less than 0.0001%. These real-world workloads exhibit different sparsity patterns, leading to highly irregular memory accesses, which poses a significant challenge to efficiently processing sparse matrix multiplication kernels.

[0003] There are currently three basic sparse matrix multiplication data streams: inner product, outer product, and row product. These three data streams make different trade-offs between input reuse and output reuse, so none of them are suitable for all types of workloads.

[0004] Many sparse matrix multiplication accelerators have been proposed in the existing technology, which are usually optimized for input or output data reuse based on a fixed data flow. These accelerators often perform well only in specific sparse ranges or specific sparse patterns because their fixed data flow strategy is difficult to adapt to the different memory access characteristics of different workloads.

[0005] Currently, some practitioners have proposed dynamic data mapping methods to optimize sparse matrix multiplication data streams. These methods pre-analyze the characteristics of the workload to determine the optimal data stream in advance, i.e., selecting the most suitable one from three basic data streams. These methods offer significant performance gains compared to a single fixed data stream strategy. However, these methods still have the following problems: First, it cannot capture the sparse pattern changes within the matrix in a fine-grained manner.

[0006] Pre-analysis methods can determine the optimal data flow for a given workload in a coarse-grained manner. However, the sparsity patterns within a matrix in the real world are often variable, and previous coarse-grained methods cannot further explore the reusability of different types of data within the matrix.

[0007] Second, offline analysis requires a large dataset.

[0008] Offline analysis often requires building datasets using a large number of real-world test cases. These datasets include various characteristics of the workloads, hardware resources, and results from the three basic data streams. Generally, the wider the range of workloads covered and the larger the data volume, the more reliable the offline analysis method will be. Therefore, building a good dataset is very time-consuming and labor-intensive.

[0009] In summary, there is an urgent need for a data partitioning and mapping method that can handle highly irregular sparse patterns with fine granularity without requiring a large dataset. Summary of the Invention

[0010] The technical problem to be solved by this invention is: in view of the technical problems existing in the prior art, this invention provides an adaptive data block mapping method and device for sparse matrix multiplication kernel that is simple in principle, easy to implement, widely applicable, and highly compatible.

[0011] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: An adaptive data block mapping method for sparse matrix multiplication kernels includes: To partition a sparse matrix, an initial matrix partitioning scheme is determined based on the matrix sparsity and the number of multiply-accumulators. Dynamically adjust the matrix block shape; introduce lightweight non-zero element index analysis, and combine it with runtime performance monitoring of random continuous matrix blocks to dynamically adjust the matrix block shape to determine the optimal block configuration; Mapping is performed based on matrix blocks. As a further improvement of the present invention, the partitioning of the sparse matrix includes: By analyzing the row pointer array representing row offsets in the sparse matrix CSR format, the matrix is ​​divided into many partitions according to the row length; Set a row stability threshold. When the length difference between adjacent rows exceeds this threshold, a new partition will be created.

[0012] As a further improvement of the present invention, all partitions are divided into two types: partitions with more than 128 rows are called large partitions, and partitions with fewer than 128 rows are called small partitions.

[0013] As a further improvement of the present invention: the pre-divided partition is further divided into matrix blocks; the overall sparsity of the matrix and the number of multiply-accumulators are analyzed to determine an initial matrix block shape, i.e., height and width; the set of scalar vector multiplication tasks corresponding to the divided matrix blocks have sufficient hardware resources for processing.

[0014] As a further improvement of the present invention: for small partitions, an initial matrix block shape is adopted; for large partitions, dynamic adjustment is performed during runtime.

[0015] As a further improvement of the present invention: the dynamic adjustment includes monitoring the performance of random continuous matrix blocks and adjusting the shape of the matrix blocks, the shape including the height and width of the matrix blocks; if a performance degradation is detected, the process returns to repartitioning; if a sustained performance improvement is observed when increasing or decreasing a certain shape parameter, the parameter will continue to be increased or decreased until the performance degrades.

[0016] As a further improvement of the present invention: for matrix block shape adjustment, the performance monitoring includes the reusability of data within each matrix block and the efficiency of the merger.

[0017] As a further improvement of the present invention, the feedback processing of performance monitoring includes: if the merger spends a lot of time in an idle state, that is, the merger frequently waits for the output of the multiplication stage, then the width of the matrix block will be increased; when increasing the height of the matrix block brings higher reusability benefits, this parameter will be continuously increased until the hardware resources are fully loaded; after the optimal matrix block shape is determined, the subsequent elements in the partition will continue to use this partitioning scheme until the next monitoring detects a performance degradation.

[0018] As a further improvement of the present invention: for all partitions, the default matrix block mapping order is to slide by column; initially, the matrix block will only slide once in the column direction, and in each subsequent iteration, the number of times the matrix block slides in the column direction will increase by one.

[0019] As a further improvement of the present invention: when frequent cache access failures are detected, the system immediately switches to row-by-row sliding to merge a large number of output rows belonging to the current iteration in the cache or off-chip storage; in the next iteration, the system will revert to the sliding mode of the previous iteration.

[0020] The present invention further provides an adaptive data block mapping apparatus for sparse matrix multiplication kernels to implement any of the above methods, comprising a processor and a storage unit, wherein the storage unit includes a computer program for implementing the adaptive data block mapping method for sparse matrix multiplication kernels.

[0021] Compared with the prior art, the advantages of the present invention are as follows: 1. The adaptive data block mapping method and apparatus for sparse matrix multiplication kernel of the present invention is simple in principle, easy to implement, and can be dynamically adjusted at runtime to adapt to the sparse mode inside the matrix, ultimately improving the computational efficiency of sparse matrix multiplication kernel and hardware resource utilization.

[0022] 2. The present invention provides an adaptive data block mapping method and apparatus for sparse matrix multiplication kernels, which implements a sparsity-aware matrix block compression and mapping scheme for sparse matrix multiplication kernels based on row product data streams. First, the present invention determines an initial matrix block scheme based on matrix sparsity and the number of multiply-accumulators. Second, by introducing lightweight non-zero element index analysis and combining it with runtime performance monitoring of random continuous matrix blocks, the present invention dynamically adjusts the shape of the matrix blocks to determine the optimal block configuration, thereby maximizing performance gains. Finally, the method of the present invention also employs a greedy strategy-based matrix block mapping stream optimization, which further explores the potential for data reuse by continuously monitoring the cache status and tentatively increasing the number of matrix blocks mapped in the column direction. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the topological structure principle of the present invention.

[0024] Figure 2 This is a schematic diagram of the modular hardware architecture of the present invention.

[0025] Figure 3 This is a flowchart illustrating the principle of matrix block shape adjustment in a specific application example of the present invention.

[0026] Figure 4 This is a schematic diagram illustrating the principle of matrix block shape adjustment in a specific application example of the present invention. Detailed Implementation

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] like Figure 1 and Figure 2 As shown, this invention discloses an adaptive data block mapping method for sparse matrix multiplication kernels. It is designed for sparse matrix multiplication kernels based on row product data streams and implements a sparsity-aware matrix block compression and mapping scheme. For sparse matrix multiplication kernels based on row product data streams, the matrix multiplication operation (A*B) is decomposed into multiple scalar vector multiplication operations. This invention adaptively groups the scalars in matrix A to optimize the data mapping stream. The process of this invention includes: To partition a sparse matrix, an initial matrix partitioning scheme is determined based on the matrix sparsity and the number of multiply-accumulators. Dynamically adjust the shape of matrix blocks; by introducing lightweight non-zero element index analysis and combining it with runtime performance monitoring of random continuous matrix blocks, the shape of matrix blocks is dynamically adjusted to determine the optimal block configuration, thereby maximizing performance gains. Mapping is performed based on matrix blocks; a greedy strategy-based matrix block mapping flow optimization is adopted, which further explores the potential for data reuse by continuously monitoring the cache status and tentatively increasing the number of matrix blocks mapped in the column direction.

[0029] In a specific application example, the sparse matrix is ​​partitioned before adaptive data block mapping. Adjacent rows of a sparse matrix workload typically have a similar distribution of non-zero elements, and their row lengths are relatively stable; based on this characteristic, the process of this invention includes: First, by analyzing the row pointer array representing row offsets in the sparse matrix CSR format, the matrix is ​​divided into many partitions according to the row length.

[0030] Secondly, a row stability threshold is set. When the length difference between adjacent rows exceeds this threshold, a new partition will be created.

[0031] In specific application examples, this invention divides all partitions into two types: partitions with more than 128 rows are called large partitions, and partitions with fewer than 128 rows are called small partitions.

[0032] Next, the partitioned area needs to be further divided into matrix blocks. Specifically, the overall sparsity of the matrix and the number of multiply-accumulators are analyzed to determine the shape of an initial matrix block, namely its height and width. The set of scalar vector multiplication tasks corresponding to the partitioned matrix blocks should have sufficient hardware resources to process, but there should be no excessive waste of hardware resources.

[0033] As a preferred embodiment, in this example, the initial matrix block shape is used by default for small partitions; for large partitions, dynamic adjustment is performed during runtime.

[0034] Furthermore, the dynamic adjustment specifically includes monitoring the performance of random continuous matrix blocks and adjusting the shape (height and width) of the matrix blocks. If a performance degradation is detected, the process returns to the previous step. If a sustained performance improvement is observed when increasing or decreasing a certain shape parameter (height or width), the parameter will continue to be increased or decreased until the performance degrades.

[0035] As a preferred embodiment, the performance monitoring for matrix block shape adjustment in this example mainly includes the reusability of data within each matrix block and the efficiency of the merger (adder tree). If the merger spends a lot of time in an idle state, that is, the merger frequently waits for the output of the multiplication stage, then the width of the matrix block will be increased. When increasing the height of the matrix block brings higher reusability benefits, this parameter will be continuously increased until the hardware resources are fully loaded. After the optimal matrix block shape is determined, subsequent elements in the partition will continue to use this partitioning scheme until the next monitoring detects a performance degradation.

[0036] In a specific implementation, for all partitions, the default matrix block mapping order is column-wise sliding. Initially, the matrix block will only slide once in the column direction. In each subsequent iteration, the number of times the matrix block slides in the column direction will increase by one. When frequent cache access failures are detected, it will immediately switch to row-wise sliding to merge a large number of output rows belonging to this iteration in the cache or off-chip storage. In the next iteration, it will revert to the sliding mode of the previous iteration.

[0037] In a specific application example, the detailed process of this invention includes: Sparse Matrix Partitioning: First, the sparse matrix multiplication task to be executed is sent to an adaptive compressor. The row offset indices of the sparse matrix are analyzed, and the matrix is ​​divided into two partitions: large and small. Specifically, a row stability threshold of 10 can be set. When the length difference between adjacent rows is less than this threshold, they are included in the same partition; when the length difference between adjacent rows exceeds this threshold, a new partition is created. Partitions with more than 128 rows are called large partitions, and partitions with fewer than 128 rows are called small partitions. These partitions mostly have a similar distribution of non-zero elements.

[0038] Initial shape of matrix blocks determined: The partitions already divided in the previous step are further divided into matrix blocks; specifically, an initial matrix block size is estimated based on the overall sparsity of the matrix. The non-zero elements in a matrix block of this size correspond to a set of scalar vector multiplication tasks, and these scalar vector multiplication tasks have sufficient hardware resources to process them without excessive waste of hardware resources.

[0039] Matrix block shape adjustment: such as Figure 3 As shown, within a large partition, for a given initial matrix block shape, the accelerator's status needs to be monitored at runtime. This status is a comprehensive indicator, including merger utilization and data reusability, which are also the most critical parameters affecting performance. When low merger utilization is detected, the width of the matrix block is gradually increased until performance no longer improves. When poor data reusability is detected, i.e., the computation task corresponding to a matrix block results in very little reuse of B matrix rows, and the computation unit is still not fully loaded, the height of the matrix block is gradually increased until full load. Once a suitable matrix block shape is found, subsequent matrix blocks will use this configuration until the next random monitoring detects a performance degradation or all matrix blocks within the entire large partition have completed computation.

[0040] Adjusting the order of matrix block mapping: such as Figure 4As shown, for all partitions, the default matrix block mapping order is column-wise sliding. Row-wise sliding can merge a large number of partial output rows as early as possible, resulting in a single row of the output matrix, but its reuse efficiency is low. Column-wise sliding can achieve higher reusability, but it is necessary to consider whether the on-chip cache can tolerate a large number of partial output rows. Initially, the matrix block will only slide once in the column direction. In each subsequent iteration, the number of times the matrix block slides in the column direction will increase by one. When frequent cache access failures are detected, it will immediately switch to row-wise sliding because the cache cannot tolerate a large number of partial output results. It is necessary to immediately merge a large number of partial output rows belonging to this iteration in the cache or off-chip storage to release the cache space occupied by them. At the same time, the matrix block mapping order that brings the best cache utilization has been found, and the mapping mode of the previous iteration will be restored in the next iteration.

[0041] An adaptive data block mapping apparatus for sparse matrix multiplication kernels for implementing any of the above methods includes a processor and a storage unit, wherein the storage unit includes a computer program for implementing the adaptive data block mapping method for sparse matrix multiplication kernels.

[0042] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. An adaptive data block mapping method for sparse matrix multiplication kernels, characterized in that, include: To partition a sparse matrix, an initial matrix partitioning scheme is determined based on the matrix sparsity and the number of multiply-accumulators. Dynamically adjust the shape of the matrix blocks; Lightweight non-zero element index analysis is introduced, and combined with runtime performance monitoring of random continuous matrix blocks, the matrix block shape is dynamically adjusted to determine the optimal block configuration; Mapping based on matrix blocks; The matrix block performance monitoring includes performance detection feedback processing, which includes: if the merger spends a lot of time in an idle state, that is, the merger frequently waits for the output of the multiplication stage, then the width of the matrix block will be increased; when increasing the height of the matrix block brings higher reusability benefits, this parameter will be continuously increased until the hardware resources are fully loaded; once the optimal matrix block shape is determined, subsequent elements in the partition will continue to use this partitioning scheme until the next monitoring detects a performance degradation. For all partitions, the default matrix block mapping order is column-wise sliding. Initially, the matrix block will only slide once in the column direction. In each subsequent iteration, the number of times the matrix block slides in the column direction will increase by one. When frequent cache access failures are detected, the system will immediately switch to row-wise sliding to merge a large number of output rows belonging to this iteration from the cache or off-chip storage. In the next iteration, the sliding mode of the previous iteration will be restored. The number of slides in the column direction increases sequentially to tentatively approach the upper limit of the number of partial output rows that the on-chip cache can tolerate; the switch to row-by-row sliding is used to release the cache space occupied by the partial output rows.

2. The adaptive data block mapping method for sparse matrix multiplication kernels according to claim 1, characterized in that, Partitioning a sparse matrix includes: By analyzing the row pointer array representing row offsets in the sparse matrix CSR format, the matrix is ​​divided into many partitions according to the row length; Set a row stability threshold. When the length difference between adjacent rows exceeds this threshold, a new partition will be created.

3. The adaptive data block mapping method for sparse matrix multiplication kernels according to claim 2, characterized in that, All partitions are divided into two types: partitions with more than 128 rows are called large partitions, and partitions with fewer than 128 rows are called small partitions.

4. The adaptive data block mapping method for sparse matrix multiplication kernels according to claim 2, characterized in that, The pre-defined partitions are further divided into matrix blocks; the overall sparsity of the matrix and the number of multiply-accumulators are analyzed to determine an initial matrix block shape, i.e., height and width; the set of scalar vector multiplication tasks corresponding to the partitioned matrix blocks have sufficient hardware resources for processing.

5. The adaptive data block mapping method for sparse matrix multiplication kernels according to claim 3, characterized in that, For small partitions, the initial matrix block shape is used; for large partitions, dynamic adjustments are made during runtime.

6. The adaptive data block mapping method for sparse matrix multiplication kernels according to claim 5, characterized in that, The dynamic adjustment includes monitoring the performance of random, continuous matrix blocks and adjusting the shape of the matrix blocks, including their height and width; if a performance degradation is detected, the process returns to repartitioning. If a sustained performance improvement is observed when increasing or decreasing a certain shape parameter, then that parameter will continue to be increased or decreased until the performance degrades.

7. The adaptive data block mapping method for sparse matrix multiplication kernels according to claim 6, characterized in that, For matrix block shape adjustment, performance monitoring includes the reusability of data within each matrix block and the efficiency of the merger.

8. An adaptive data block mapping apparatus for implementing the method of any one of claims 1-7, comprising a processor and a storage unit, characterized in that, The storage unit includes a computer program for implementing the adaptive data block mapping method for the sparse matrix multiplication kernel described above.

Images