Apparatus, method and program product for accelerating unstructured sparse matrix multiplication computation

Abstract

This application discloses an apparatus, method, and program product for accelerating unstructured sparse matrix multiplication computation. The apparatus includes a preprocessing unit, a partial sum generation unit, and a merging unit. The preprocessing unit is configured to convert the unstructured sparse weight matrix to be multiplied into multiple column groups by compressing non-zero elements within the matrix rows while retaining their original column indices, and aggregating non-zero elements from different rows along the column direction. The partial sum generation unit is configured to, for each column group, perform a scalar-vector multiplication operation on each non-zero element within the group with the corresponding row data of the dense matrix to obtain the product result, and combine all product results of each column group as a partial sum data block. The merging unit is configured to accumulate all partial sum data blocks to obtain a merged result. This application significantly improves the execution efficiency of unstructured sparse SpMM in large language models.

Images