Acceleration methods, devices, electronic devices, and media based on irregular sparse models

Abstract

This disclosure presents embodiments of an acceleration method, apparatus, electronic device, and medium based on a random sparse model. One specific implementation of the method includes: inputting a sparse model into a target compiler; in a layer intermediate representation, splitting the sparse weight matrix of the sparse model into sparse sub-blocks to obtain a split sparse weight matrix; generating attribute information for each sparse sub-block in the split sparse weight matrix based on a sparse operator performance model; in a block-layer intermediate representation, performing fusion processing on each sparse sub-block according to its attribute information; in an operator-layer intermediate representation, mapping each sparse block to a computational unit according to its corresponding attribute information to obtain a mapping result; and generating a processing task corresponding to the sparse model based on the mapping result, and executing the processing task. This implementation achieves end-to-end optimization through multi-layer optimization, breaking through the benefit boundary of traditional random sparse acceleration designs.

Description

Technical Field

[0001] Embodiments of this disclosure relate to computer technology, including acceleration methods, apparatus, electronic devices, and media based on random sparse models. Background Technology

[0002] With the widespread application of deep neural networks (DNNs) in edge computing and large language models (LLMs), accelerating the processing efficiency of irregular sparse models has become a key challenge. Currently, sparse model acceleration is mainly achieved through two types of methods: static optimization of sparse formats and compiler-guided sparsity-aware optimization. Static optimization of sparse formats typically refers to using a fixed sparse coding format (such as CSR, COO) and a single computational paradigm (such as the pure sparse computational kernel of cuSPARSE), improving regularity through data rearrangement or block partitioning strategies. Compiler-guided sparsity-aware optimization typically refers to using traditional frameworks (such as TVM-Sparse, SparseTIR) to achieve multi-format hybrid coding and operator fusion, but still relies on static sparse block partitioning strategies.

[0003] However, the above methods often have the following technical problems when dealing with irregular sparse models (such as the sparse weight matrix of Llama-2): Sparse acceleration schemes (such as Sputnik and ASpT) do not model the performance of sparse sub-blocks in regions with different sparsity rates, resulting in the inability to accurately identify sub-blocks suitable for sparse or dense computation in irregular distribution scenarios, causing a waste of computing resources; Traditional sparse coding (such as CSR and COO) uses a uniform format for irregular distribution matrices, resulting in low memory efficiency of sparse sub-blocks due to redundant storage; Existing frameworks (such as cuSPARSE) map the entire sparse matrix to a single type of computation unit, causing dense sub-blocks to slow down the overall performance due to invalid zero-value computation when using sparse computation units, or sparse sub-blocks to waste resources due to empty computation when using dense units; Traditional batch processing schemes (such as SparseTIR) do not consider cross-batch load correlation, and when processing large-scale data, computation units often experience computational bottlenecks due to the continuous allocation of high-density sub-blocks, resulting in a decrease in inference throughput.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the present disclosure concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure propose acceleration methods, apparatuses, electronic devices, and media based on irregular sparse models to explain one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure propose an acceleration method based on an irregular sparse model. The method includes: inputting a sparse model into a target compiler, wherein the target compiler includes a layer intermediate representation, a block intermediate representation, and an operator intermediate representation; in the layer intermediate representation, splitting the sparse weight matrix of the sparse model into sparse sub-blocks to obtain a split sparse weight matrix, wherein the split sparse weight matrix includes each sparse sub-block; generating attribute information of each sparse sub-block in the split sparse weight matrix based on a sparse operator performance model; and in the block intermediate representation... In the representation, based on the attribute information of each sparse sub-block, the sparse sub-blocks are fused to obtain a fused sparse weight matrix. Each sparse block in the fused sparse weight matrix includes a target fused sparse block and an unfused sparse sub-block, and each target fused sparse block has corresponding attribute information. In the intermediate representation of the operator layer, based on the attribute information corresponding to each sparse block, each sparse block is mapped to a computational unit to obtain a mapping result. Based on the mapping result, a processing task corresponding to the sparse model is generated, and the processing task is executed.

[0008] Secondly, some embodiments of this disclosure propose an acceleration device based on an irregular sparse model. The device includes: an input unit configured to input a sparse model into a target compiler, wherein the target compiler includes a layer intermediate representation, a block intermediate representation, and an operator intermediate representation; a splitting unit configured to split the sparse weight matrix of the sparse model into sparse sub-blocks in the layer intermediate representation to obtain a split sparse weight matrix, wherein the split sparse weight matrix includes each sparse sub-block; a first generation unit configured to generate attribute information of each sparse sub-block in the split sparse weight matrix based on a sparse operator performance model; and a fusion unit configured to... In the aforementioned block layer intermediate representation, based on the attribute information of each sparse sub-block, the aforementioned sparse sub-blocks are fused to obtain a fused sparse weight matrix. Each sparse block in the fused sparse weight matrix includes each target fused sparse block and each unfused sparse sub-block, and each target fused sparse block corresponds to attribute information. A mapping unit is configured to map each sparse block to a computation unit in the aforementioned operator layer intermediate representation based on the attribute information corresponding to each sparse block, obtaining a mapping result. A second generation unit is configured to generate a processing task corresponding to the aforementioned sparse model based on the mapping result, and to execute the aforementioned processing task.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0011] Fifthly, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0012] The above embodiments of this disclosure have the following beneficial effects: Through the acceleration method based on an irregular sparse model in some embodiments of this disclosure, dynamic resource scheduling and end-to-end collaborative optimization are achieved, significantly improving the computational efficiency of the sparse model. Specifically, traditional sparse acceleration methods often have the following problems: They fail to model the performance of sparse sub-blocks in regions with different sparsity rates, resulting in the inability to accurately identify sub-blocks suitable for sparse or dense computation in irregular distribution scenarios, leading to wasted computational resources; they use a uniform format for the irregular distribution matrix, resulting in low memory efficiency due to redundant storage of sparse sub-blocks; they map the entire sparse matrix to a single type of computational unit, causing different types of sub-blocks in the sparse matrix to fail to be mapped to the corresponding computational units, resulting in low performance improvement and wasted resources; and they do not consider cross-batch load correlation, causing a decrease in end-to-end inference throughput. Based on this, the acceleration method based on an irregular sparse model in some embodiments of this disclosure first inputs the sparse model into a target compiler, wherein the target compiler includes layer intermediate representation, block layer intermediate representation, and operator layer intermediate representation. Thus, the sparse model can be used as a data source. Next, in the intermediate representation of the above layer, the sparse weight matrix of the sparse model is split into sparse sub-blocks to obtain a split sparse weight matrix, which includes each sparse sub-block. This allows for the partitioning of the sparse weight matrix in the sparse model for subsequent processing. Then, based on the sparse operator performance model, the attribute information of each sparse sub-block in the split sparse weight matrix is ​​generated. Thus, by modeling the performance of sparse sub-blocks in different sparsity regions, a sparse operator performance model can be obtained, facilitating the prediction of the attribute information of each sparse sub-block and reducing computational resource waste. Then, in the intermediate representation of the block layer, based on the attribute information of each sparse sub-block, the sparse sub-blocks are fused to obtain a fused sparse weight matrix. In this fused sparse weight matrix, each sparse block includes each target fused sparse block and each unfused sparse sub-block, and each target fused sparse block corresponds to attribute information. Therefore, the aforementioned sparse sub-blocks can be fused and their format encoded to reduce redundant storage of the fused sparse weight matrix, thereby improving memory efficiency. Then, in the intermediate representation of the operator layer, based on the attribute information corresponding to each sparse block, a mapping is performed between each sparse block and a computational unit to obtain the mapping result. This allows the sparse blocks to be mapped to different computational units according to their type, improving overall performance and allocating resources more efficiently. Finally, based on the mapping result, a processing task corresponding to the sparse model is generated and executed. Thus, the model can be optimized and tasks processed using the mapping result and the sparse operator performance model, resulting in an acceleration effect.Furthermore, when processing computational tasks across batches, high-load and low-load sparse sub-blocks in adjacent sparse weight matrices can be cross-mapped to different computational units. This improves the load correlation between different batches and increases end-to-end inference throughput. It adapts to different sparsity patterns and hardware environments, demonstrating significant advantages in large-model inference scenarios. It provides an efficient sparse acceleration solution for edge computing and large language model deployment, significantly improving the applicability of models in mobile and cloud inference scenarios, and breaking through the benefit boundaries of traditional irregular sparse acceleration designs. Attached Figure Description

[0013] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0014] Figure 1 This is a flowchart of some embodiments of the acceleration method based on the irregular sparse model according to the present disclosure;

[0015] Figure 2 These are schematic diagrams of some embodiments of the acceleration method apparatus based on the irregular sparse model according to the present disclosure;

[0016] Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure.

[0017] Figure 4 This is a bar chart comparing the performance of some embodiments of this disclosure in multiple dimensions. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0020] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0021] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0022] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0023] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] Figure 1 A flow 100 of some embodiments of an acceleration method based on a random sparse model according to the present disclosure is shown. This acceleration method based on a random sparse model includes the following steps:

[0025] Step 101: Input the sparse model into the target compiler.

[0026] In some embodiments, the execution entity (e.g., a computing device) of the acceleration method based on a random sparse model can input the sparse model into a target compiler. This target compiler includes layer intermediate representations, block intermediate representations, and operator intermediate representations. The target compiler can be a compilation tool supporting multi-level intermediate representations, specifically designed for processing acceleration tasks based on random sparse models. The weight matrix of the sparse model can be split, merged, and mapped using the layer intermediate representations, block intermediate representations, and operator intermediate representations to ultimately generate and execute efficient processing tasks. The layer intermediate representations are used to describe the overall structure of the model for high-level optimization. The block intermediate representations are used to describe the local structure of the model for mid-level optimization. The operator intermediate representations are used to describe the specific computations of the model for low-level optimization. The sparse model can be a random sparse model (such as a DNN). A random sparse model is typically a mathematical model obtained through unstructured sparsity. Unstructured sparsity usually refers to obtaining a random matrix by pruning arbitrary elements in the model's matrix. The pruning operation described above typically involves setting any element in the matrix to 0, thereby reducing the model's complexity and computational cost while preserving its performance as much as possible. In practice, the execution entity can input the irregular sparse DNN model into the target compiler according to the model's file format (such as ONNX, TensorFlow SavedModel, or PyTorch) for subsequent processing.

[0027] Step 102: In the intermediate representation of the layer, the sparse weight matrix of the sparse model is split into sparse sub-blocks to obtain the split sparse weight matrix.

[0028] In some embodiments, the execution entity can decompose the sparse weight matrix of the sparse model into sparse sub-blocks in the intermediate representation of the layer, obtaining a decomposed sparse weight matrix. The decomposed sparse weight matrix includes various sparse sub-blocks. The weight matrix of the sparse model is typically a matrix form used to describe model parameters. Each sparse sub-block is typically a local matrix partitioned from the sparse weight matrix, and each local matrix can serve as a sparse sub-block of the sparse weight matrix. In practice, the execution entity can decompose the sparse weight matrix into sparse sub-blocks according to a preset sub-block dimension (e.g., 2×2 or 4×4) and record the position of each sparse sub-block in the sparse weight matrix. Since the position of each sparse sub-block in the sparse weight matrix does not change, the sparse weight matrix decomposed into sparse sub-blocks can be determined as the decomposed sparse weight matrix.

[0029] Step 103: Based on the sparse operator performance model, generate the attribute information of each sparse sub-block in the split sparse weight matrix.

[0030] In some embodiments, the execution entity can generate attribute information of each sparse sub-block in the split sparse weight matrix based on the sparse operator performance model. The sparse operator performance model can be a structured table including feature information of each candidate matrix. The feature information of each candidate matrix typically includes the shape, threshold, and fitting execution time of each candidate matrix. Each candidate matrix can be a matrix of various pre-set shapes (e.g., 256×256, 128×128, 64×64, etc.). The thresholds of each candidate matrix typically include a dense threshold and a sparse threshold. These thresholds can be used to determine the type of each matrix. When the sparsity of a matrix is ​​greater than the sparse threshold, the matrix is ​​classified as sparse; when the sparsity is between the sparse and dense thresholds, the matrix is ​​classified as wobbly; when the sparsity is less than the dense threshold, the matrix is ​​classified as dense. These types include dense, wobbly, and sparse. The above-mentioned fitting execution time can be the time spent performing matrix multiplication on each predicted candidate matrix at various sparsity rates. The matrix multiplication includes both intensive and sparse computation. Intensive computation refers to performing operations on all elements one by one in matrix multiplication. Sparse computation refers to performing operations only on non-zero elements in matrix multiplication. The fitting execution time for each candidate matrix can include both the intensive and sparse computation times for each candidate matrix. The intensive computation time for each candidate matrix can be the actual execution time for intensive computation on each candidate matrix. The sparse computation time for each candidate matrix can be the time spent performing sparse computation on each predicted candidate matrix at different sparsity rates. The sparse computation time for each candidate matrix is ​​usually represented by the fitting parameters corresponding to each candidate matrix. The fitting parameters represent the relationship between the sparsity rate of each candidate matrix and the sparse computation time. The attribute information can include the sparsity rate, shape, type, coordinates, and prediction execution time of each sparse sub-block. The sparsity rate usually refers to the proportion of zero elements in the matrix. The aforementioned prediction execution time typically refers to the matrix multiplication time for each predicted sparse block. This prediction execution time includes both intensive computation prediction execution time and sparse computation prediction execution time. The intensive computation prediction execution time can be the actual execution time for each sparse block to perform intensive computation on its corresponding candidate matrix within the aforementioned sparse operator performance model. The sparse computation prediction execution time can be the fitting sparse computation time for each sparse block on its corresponding candidate matrix within the aforementioned sparse operator performance model.

[0031] In some optional implementations of certain embodiments, the aforementioned execution entity can generate attribute information for each sparse sub-block based on the sparse operator performance model through the following steps:

[0032] Step 1: Obtain the actual execution time data for matrix multiplication of each candidate matrix at various sparsity rates. This actual execution time data refers to the actual time spent performing matrix multiplication on each candidate matrix at various sparsity rates. This data includes the actual execution time of sparse computation and the actual execution time of dense computation. In practice, the execution entity can implement sparse computation using Sputnik and dense computation using cuBlas. Then, the actual execution time data for each candidate matrix at various sparsity rates can be obtained. For example, the execution entity can generate 6 sets of candidate matrices with different shapes and take an average sparsity rate of 23 within the sparsity rate range [0.55, 0.99]. Then, the 6 sets of candidate matrices with different shapes can be subjected to sparse and dense computations with a preset feature matrix (generally a dense matrix) at the 23 sparsity rates. Finally, the execution time of the 6 sets of candidate matrices with different shapes performing the two matrix operations at the 23 sparsity rates is obtained as the actual execution time data for each candidate matrix.

[0033] Step 2: Based on the actual execution time data mentioned above, generate a sparse operator performance model.

[0034] Step 3: Based on the sparse operator performance model, generate the attribute information of each sparse sub-block. In practice, the execution entity can identify the dimension of each sparse sub-block (e.g., 256×256 or 1024×1024) and use the dimension as the shape of each sparse sub-block. Next, the proportion of zero elements in each sparse sub-block can be identified and used as the sparsity rate of each sparse sub-block. In the sparse operator performance model, candidate matrices corresponding to each sparse sub-block are retrieved according to their shapes. Then, based on the feature information of the corresponding candidate matrices, the attribute information of each sparse sub-block is obtained. For example, for a sparse sub-block with a shape of (256×256), the execution entity can retrieve candidate matrices with a shape of 256×256 in the sparse operator performance model. Then, based on the sparsity rate of the sparse sub-block (e.g., 0.9) and the thresholds of the 256×256 candidate matrix (dense threshold of 0.675, sparse threshold of 0.835), the type of the sparse sub-block can be determined as sparse. Next, the position of the sparse sub-block in the aforementioned split sparse weight matrix can be used as the coordinates of the sparse sub-block. Then, the fitting dense computation time corresponding to the 256×256 candidate matrix can be used as the dense computation prediction execution time of the sparse sub-block. Finally, the sparsity rate of the sparse sub-block and the fitting parameters corresponding to the 256×256 candidate matrix can be solved using a quadratic polynomial to obtain the sparse computation prediction execution time of the sparse sub-block. Finally, the dense computation prediction execution time and the sparse computation prediction execution time of the sparse sub-block can be integrated into the prediction execution time of the sparse sub-block. Finally, the sparsity, shape, type, coordinates, and prediction execution time of the sparse sub-block can be integrated into its attribute information. Similarly, attribute information for each sparse sub-block in the resulting sparse weight matrix can be generated.

[0035] In some optional implementations of certain embodiments, the aforementioned execution entity can generate a sparse operator performance model based on the aforementioned actual execution time data through the following steps:

[0036] The first step is to generate the fitting execution time for each candidate matrix based on the actual execution time data mentioned above. In practice, the execution entity can use the polynomial curve fitting algorithm to perform polynomial calculations on the actual execution time of sparse computation for each candidate matrix and the corresponding sparsity rate of each candidate matrix to obtain the fitting sparse computation time for each candidate matrix. The actual execution time of dense computation for each candidate matrix can be used as the fitting dense computation time for each candidate matrix. The fitting sparse computation time and the fitting dense computation time of each candidate matrix are then integrated to obtain the fitting execution time for each candidate matrix. For example, for a candidate matrix of any shape, the execution entity uses the 23 sparsity rates as independent variables and the actual execution time of sparse computation for the candidate matrix corresponding to the 23 sparsity rates as the dependent variable. A polynomial is constructed using the independent and dependent variables. Then, the least squares method is used to solve for the fitting parameters of the polynomial, i.e., the fitting sparse computation time of the candidate matrix. Then, the actual execution time of the dense computation of the candidate matrix is ​​used as the predicted execution time of the sparse computation. Similarly, the fitting execution time of each of the above candidate matrices can be obtained.

[0037] The second step involves generating thresholds for each candidate matrix based on the fitting execution time of each candidate matrix. In practice, the execution entity can perform quadratic polynomial calculations on the actual execution time of the dense computations for each candidate matrix and the fitting parameters to obtain the dense threshold for each candidate matrix. Then, using a performance advantage coefficient (e.g., 0.8), the actual execution time of the dense computations for each candidate matrix can be combined with the performance advantage coefficient and the fitting parameters of each candidate matrix to obtain the sparse threshold for each candidate matrix. Finally, the dense thresholds and sparse thresholds of each candidate matrix are integrated into a single threshold for each candidate matrix.

[0038] The third step is to generate a sparse operator performance model based on the threshold values ​​and fitting execution times of each candidate matrix. In practice, the execution entity can integrate the shapes, threshold values, and fitting execution times of each candidate matrix into a single Table model, which serves as the sparse operator performance model.

[0039] Step 104: In the intermediate representation of the block layer, the sparse sub-blocks are fused according to their attribute information to obtain the fused sparse weight matrix.

[0040] In some embodiments, the execution entity may perform a fusion process on the sparse sub-blocks in the intermediate representation of the block layer according to the attribute information of each sparse sub-block, to obtain a fused sparse weight matrix. Each sparse block in the fused sparse weight matrix includes a target fused sparse block and unfused sparse sub-blocks, and each target fused sparse block corresponds to attribute information. The fusion process typically refers to merging the sparse sub-blocks into sparse blocks according to preset fusion conditions, resulting in fused sparse blocks. The fusion process may include homogeneous block fusion and non-homogeneous block fusion. The preset fusion conditions may include homogeneous block fusion conditions and non-homogeneous block fusion conditions. The homogeneous block fusion condition typically requires that the type of the sparse block after homogeneous block fusion is the same as the type of its two corresponding sparse sub-blocks. The non-homogeneous block fusion condition typically requires that the type of the sparse block after non-homogeneous block fusion is the same as the type of the sparse sub-block with the higher sparsity rate among its two corresponding sparse sub-blocks. The aforementioned fused sparse blocks include sparse blocks fused from homogeneous blocks and sparse blocks fused from non-homogeneous blocks. The aforementioned fused sparse blocks from homogeneous blocks can include sparse blocks fused from target homogeneous blocks and sparse blocks fused from homogeneous blocks that do not meet the homogeneous block fusion conditions. The aforementioned fused sparse blocks from non-homogeneous blocks can include sparse blocks fused from target non-homogeneous blocks and sparse blocks fused from non-homogeneous blocks that do not meet the non-homogeneous block fusion conditions. The aforementioned target fused sparse blocks can include sparse blocks fused from target homogeneous blocks and sparse blocks fused from target non-homogeneous blocks. The aforementioned target fused sparse blocks from homogeneous blocks refer to sparse blocks fused from homogeneous blocks that meet the aforementioned homogeneous block fusion conditions. The aforementioned target fused sparse blocks from non-homogeneous blocks refer to sparse blocks fused from non-homogeneous blocks that meet the aforementioned non-homogeneous block fusion conditions. The aforementioned unfused sparse sub-blocks generally refer to the sparse sub-blocks corresponding to the fused sparse blocks that do not meet the aforementioned preset fusion conditions. In practice, the aforementioned execution entity can fuse the sparse sub-blocks based on their attribute information to obtain fused sparse blocks. The fused sparse blocks that satisfy the aforementioned preset fusion conditions are then identified as target fused sparse blocks. The target fused sparse blocks and the corresponding sparse sub-blocks of the fused sparse blocks that do not satisfy the aforementioned preset fusion conditions are then integrated into a fused sparse weight matrix.

[0041] In some optional implementations of certain embodiments, the execution entity may perform the following steps in the intermediate representation of the block layer to fuse the sparse sub-blocks according to the attribute information of each sparse sub-block, thereby obtaining a fused sparse weight matrix:

[0042] Step one: Based on the attribute information of each sparse sub-block, perform homogeneous block fusion on adjacent sparse sub-blocks of the same type within each sparse sub-block to obtain the fused sparse block. Homogeneous block fusion typically refers to integrating two adjacent sparse sub-blocks of the same type from the split sparse weight matrix into a single matrix, i.e., the fused sparse block. The attribute information of the fused sparse block is also determined by the attribute information of the two adjacent sparse sub-blocks corresponding to it. In practice, the executing entity can merge two adjacent sparse sub-blocks of the same type from the split sparse weight matrix to obtain the fused sparse block.

[0043] Step 2: For each homogeneous block merged into a sparse block, perform the following steps:

[0044] Sub-step one: Based on the attribute information of the adjacent sparse sub-blocks corresponding to the sparse block after the homogeneous blocks are merged, the attribute information of the sparse block after the homogeneous blocks are merged is generated. This attribute information includes the sparsity rate, shape, type, coordinates, and predicted execution time of the sparse block after the homogeneous blocks are merged. In practice, the execution entity can obtain the sparsity rate of the sparse block after the homogeneous blocks are merged by the proportion of zero elements in the sparse block. Furthermore, based on the shape and coordinates of the two sparse sub-blocks corresponding to the sparse block after the homogeneous blocks are merged, the shape and coordinates of the sparse block after the homogeneous blocks are merged. When the sparsity rate and shape of the sparse block after the homogeneous blocks are known, the type and predicted execution time of the sparse block after the homogeneous blocks are merged can be obtained through the sparse operator performance model. The sparsity rate, shape, type, coordinates, and predicted execution time of the sparse block after the homogeneous blocks are merged are then integrated into the attribute information of the sparse block after the homogeneous blocks are merged. For example, the aforementioned execution entity can perform homogeneous block merging of any two adjacent sparse sub-blocks of the same type. The attribute information of the two adjacent sparse sub-blocks of the same type includes [Sparseness: 0.84, Block Type: Sparse, Coordinates (256, 0), Block Shape (256×256)] and [Sparseness: 0.84, Block Type: Sparse, Coordinates (256, 256), Block Shape (256×256)]. The partial attribute information of the sparse block after homogeneous block merging of the two adjacent sparse sub-blocks of the same type can be obtained, including [Sparseness: 0.84, Coordinates (256, 0), Block Shape (256×512)]. Based on the partial attribute information of the sparse block after homogeneous block fusion, and combined with the sparse operator performance model, the type and predicted execution time of the obtained sparse block after homogeneous block fusion can be further obtained. For specific implementation details, refer to step 103, which involves generating attribute information for each sparse sub-block based on the sparse operator performance model; these details will not be elaborated here. The partial attribute information, along with the type and predicted execution time of the obtained sparse block after homogeneous block fusion, are integrated to form the attribute information of the sparse block after homogeneous block fusion.

[0045] Sub-step two: In response to the fact that the attribute information of the sparse block after the above homogeneous block fusion satisfies the homogeneous block fusion condition, the sparse block after the above homogeneous block fusion is determined as the target sparse block after homogeneous block fusion. In practice, when the type of the sparse block after the above homogeneous block fusion is the same as the type of the two corresponding adjacent sparse sub-blocks of the same type, the sparse block after the above homogeneous block fusion can be determined as the target sparse block after homogeneous block fusion.

[0046] Step 3: Merge dissimilar adjacent sparse sub-blocks within each of the aforementioned sparse sub-blocks using non-homogeneous block fusion to obtain the merged sparse blocks. This non-homogeneous block fusion typically refers to integrating two dissimilar adjacent sparse sub-blocks from the split sparse weight matrix into a single matrix, i.e., the merged sparse block. The attribute information of each merged sparse block is determined by the attribute information of its corresponding two adjacent sparse sub-blocks. In practice, the executing entity can merge any two dissimilar adjacent sparse sub-blocks from the split sparse weight matrix to obtain the merged sparse blocks.

[0047] Step 4: For each sparse block obtained after merging non-homogeneous blocks, perform the following steps:

[0048] Sub-step one: Based on the attribute information of the adjacent sparse sub-blocks corresponding to the sparse block after the fusion of the non-homogeneous blocks, generate the attribute information of the sparse block after the fusion of the non-homogeneous blocks. The attribute information of the sparse block after the fusion of the non-homogeneous blocks includes the sparsity, shape, type, coordinates, and prediction execution time of the sparse block after the fusion of the non-homogeneous blocks. In practice, the execution entity can obtain the sparsity of the sparse block after the fusion of the non-homogeneous blocks, and based on the shape and coordinates of the two sparse sub-blocks corresponding to the sparse block after the fusion of the non-homogeneous blocks, obtain the shape and coordinates of the sparse block after the fusion of the non-homogeneous blocks. The specific implementation method can refer to the steps described above for generating some attribute information of the sparse block after the fusion of homogeneous blocks, and will not be repeated here. Given the sparsity and shape of the sparse block after merging the non-homogeneous blocks, the type and prediction execution time of the sparse block can be obtained through the aforementioned sparse operator performance model. The specific implementation method can refer to step 103 above, which describes generating the attribute information of each sparse sub-block based on the sparse operator performance model; it will not be repeated here. Finally, the sparsity, shape, type, coordinates, and prediction execution time of the sparse block after merging the non-homogeneous blocks can be integrated into the attribute information of the sparse block after merging the non-homogeneous blocks.

[0049] Sub-step two: In response to the fact that the attribute information of the sparse block after the above-mentioned non-homogeneous block fusion satisfies the non-homogeneous block fusion condition, the sparse block after the above-mentioned non-homogeneous block fusion is determined as the target sparse block after non-homogeneous block fusion. In practice, when the type of the sparse block after the above-mentioned non-homogeneous block fusion is the same as the type of the sparse sub-block with the higher sparsity rate among the two dissimilar types of sparse sub-blocks corresponding to the above-mentioned non-homogeneous block fusion, the above-mentioned execution entity can determine the sparse block after the above-mentioned non-homogeneous block fusion as the target sparse block after non-homogeneous block fusion. For example, when a sparse sub-block of type 'wobbly' and a sparse sub-block of type 'sparse' are fused to obtain a sparse block of type 'sparse', the sparse block after the above-mentioned non-homogeneous block fusion is determined as the target sparse block after non-homogeneous block fusion.

[0050] Step 5: Based on the sparse blocks resulting from the fusion of the homogeneous blocks and the sparse blocks resulting from the fusion of the non-homogeneous blocks, generate a fused sparse weight matrix. In practice, the executing entity can integrate the target fused sparse blocks and the sparse sub-blocks corresponding to the fused sparse blocks that do not meet the preset fusion conditions into a fused sparse weight matrix according to their corresponding attribute information.

[0051] Optionally, the aforementioned implementing entity may also perform the following steps:

[0052] The first step is to obtain the attribute information of each sparse block in the fused sparse weight matrix. Each sparse block in the fused sparse weight matrix includes the target fused sparse blocks and the unfused sparse sub-blocks. The attribute information of each sparse block includes its sparsity, shape, type, coordinates, and prediction execution time. In practice, the execution entity can use the sparse operator performance model to extract the attribute information of each sparse block in the fused sparse weight matrix based on its shape and sparsity. For specific implementation details, refer to step 103 above, which describes generating the attribute information of each sparse sub-block based on the sparse operator performance model; these details will not be elaborated here.

[0053] The second step involves format encoding each matrix in the fused sparse weight matrix based on the attribute information of each sparse block. This format encoding typically refers to switching each matrix in the fused sparse weight matrix to an encoding format that occupies less storage. In practice, the executing entity can encode sparse blocks of type sparse using row or column compression methods (such as CSR or CSC) or block compression methods (such as Block-CSR or ELLPACK). Sparse blocks of type dense can be encoded using native array methods or quantization compression (such as INT8 quantization). Sparse blocks of type wobbly can be encoded using hybrid encoding, generating both sparse and dense format copies simultaneously, facilitating adaptation to various computational units.

[0054] Step 105: In the intermediate representation of the operator layer, each sparse block is mapped to a computational unit based on the attribute information corresponding to each sparse block, and the mapping result is obtained.

[0055] In some embodiments, the execution entity can map the sparse blocks to computation units in the intermediate representation of the operator layer based on the attribute information corresponding to each sparse block, obtaining a mapping result. The computation units are typically pre-defined functional modules for performing computational tasks. These computation units include dense computation units and sparse computation units. The mapping result represents the mapping relationship information between each sparse block and the computation unit. The dense computation unit is typically a computation module for processing dense matrix computations. The sparse computation unit is typically a computation module for processing sparse matrix computations. The mapping relationship information includes mapping sparse blocks of sparse type to sparse computation units, dense sparse blocks of dense type to dense computation units, and oscillating sparse blocks to the computation unit with the smaller load between the two computation units. The load of the computation unit can be represented by the predicted execution time of each sparse block on the computation unit. In practice, the execution entity can adjust the mapping relationship between each sparse block and the computation unit to optimize the total load of the computation unit.

[0056] In some optional implementations of certain embodiments, the execution entity can map each sparse block to a computational unit in the intermediate representation of the operator layer according to the attribute information corresponding to each sparse block, thereby obtaining the mapping result:

[0057] Step 1: For each of the sparse blocks mentioned above, perform the following steps:

[0058] In sub-step one, in response to the sparse block's attribute information representation type being sparse, the sparse block is mapped to a sparse computation unit. This sparse computation unit is typically a computation module that supports direct computation in compressed formats (such as CSR or Block-CSR) or zero-value skipping. In practice, the execution entity can map the sparse block with a sparse attribute information type to the sparse computation unit (such as NVIDIA A100's Sparse Tensor Core, Intel DLBoost sparse instruction set, or Google Sparsity Core).

[0059] Sub-step two: In response to the fact that the attribute information representation type of the sparse block is dense, the sparse block is mapped to a dense computing unit. This dense computing unit is typically a computing module that maximizes rule-based computation throughput through continuous memory access and vectorized computation (such as SIMD). In practice, the execution entity can map the sparse block with dense attribute information to the dense computing unit (such as a GPU's CUDA Core, a CPU's AVX-512 unit, or a TPU's MXU).

[0060] In sub-step three, in response to the sparse block's attribute information representation type being "oscillating," the sparse block is mapped to a computing unit that meets a preset load condition. This preset load condition is typically the computing unit with the lower load between the sparse and dense computing units. The computing unit load can be calculated as the sum of the predicted execution times of the sparse blocks mapped to that computing unit. In practice, when mapping sparse blocks with an attribute information type of "oscillating," the execution entity dynamically identifies the current sparse and dense computing unit loads. Then, the sparse block with an attribute information type of "oscillating" can be mapped to one of the sparse or dense computing units with the lower load.

[0061] Step two: In response to the simultaneous mapping of the sparse weight matrix fused between adjacent batches to the computing units, the mapping order of each sparse block in the fused sparse weight matrix between adjacent batches and the computing units is rearranged to obtain inter-matrix adjustment information. This inter-matrix adjustment information can be the mapping order of each sparse block and computing unit in the merged matrix. In practice, since adjacent batches use the fused sparse weight matrix during task generation, and each batch sequentially maps each sparse block in the fused sparse weight matrix to the computing units, the execution entity can merge the fused sparse weight matrices from the two batches and adjust the mapping order of each sparse block in each fused sparse weight matrix to the computing units. This ensures that the load mapped to the computing units is relatively small during the sequential mapping of each sparse sub-block in the two batches, thus obtaining the inter-matrix adjustment information. For example, let's name two identical fused sparse weight matrices A and B. Both matrices contain four sparse blocks. In matrix A, the first two sparse blocks (sparse block 1 and sparse block 2) are mapped to dense computation units, with sparse block 1 having a higher computational load and sparse block 2 a lower load. The last two sparse blocks (sparse block 3 and sparse block 4) are mapped to sparse computation units, with sparse block 3 having a higher computational load and sparse block 4 a lower load. Similarly, in matrix B, the first two sparse blocks (sparse block 5 and sparse block 6) are mapped to dense computation units, with sparse block 5 having a higher computational load and sparse block 6 a lower load. The last two sparse blocks (sparse block 7 and sparse block 8) are mapped to sparse computation units, with sparse block 7 having a higher computational load and sparse block 8 a lower load. After fusion, because sparse blocks 1 and 5 will be processed simultaneously, the computational unit load will become excessively high. The order of sparse blocks in matrix B can be adjusted so that sparse block 1 and sparse block 6 are processed together on dense computing units, sparse block 2 and sparse block 5 are processed on dense computing units, sparse block 3 and sparse block 8 are processed on sparse computing units, and sparse block 4 and sparse block 7 are processed on sparse computing units, ensuring that the load mapped to computing units each time differs little.

[0062] Step three involves determining the mapping result by combining the mapping relationship information, the adjustment information between the matrices, and the attribute information of each sparse block. This mapping result is a set of sparse computing resource scheduling strategies and execution process optimization schemes. The sparse computing resource scheduling strategy can reduce utilization differences by dynamically mapping computing units. The execution process optimization scheme can reduce excessive load on technical units by rearranging sparse blocks across matrices. In practice, the execution entity can use the sparse computing resource scheduling strategy and the execution process optimization scheme as the mapping result.

[0063] Step 106: Based on the mapping results, generate the corresponding sparse model processing task and execute the processing task.

[0064] In some embodiments, the execution entity may generate a processing task corresponding to the sparse model based on the mapping result, and execute the processing task.

[0065] In some optional implementations of certain embodiments, the execution entity may generate a processing task corresponding to the sparse model based on the mapping result, and execute the processing task, through the following steps:

[0066] The first step is to insert a block-level intermediate representation of the compiler stack between the layers and operator layers of the compiler stack. This block-level intermediate representation carries information and format conversion. The information carried by this representation includes the execution time information of the sparse blocks, the attribute information corresponding to each sparse block, and the mapping results. In practice, the execution entity first divides the input sparse matrix into 128×128 blocks. It records the sparsity of each block (e.g., 92%, meaning 92% of the elements in the block are zero), the predicted execution time (e.g., 1.2 milliseconds using a sparse core, 2.1 milliseconds using a dense core), and the hardware mapping strategy (e.g., allocating the sparse block to a dedicated sparse computing unit of the GPU). Using the newly added BlockLevelPass module, the attributes of these blocks (e.g., shape, distribution of non-zero elements) are extracted from the layers, and the processing time for different computation modes is predicted using the sparse operator performance model. Next, a format decision is made based on the sparsity, such as converting sparse blocks with high sparsity (e.g., >90%) from COO format (which stores rows, columns, and values ​​of non-zero elements via triples) to CSR format (which optimizes storage by compressing row pointers and column indexes).

[0067] The second step involves adjusting the shape of each merged sparse block in the block-level intermediate representation of the compiler stack based on hardware characteristic information, resulting in shaped merged sparse blocks. This hardware characteristic information typically refers to the physical constraints and performance characteristics of the target hardware. It may include thread bundles, memory alignment, and shared memory capacity. Merging sparse blocks usually involves merging multiple adjacent small sparse blocks into a larger block. Shape adjustment typically refers to modifying the block dimensions or padding with zero elements based on hardware constraints. Shape adjustment may include column alignment and row alignment. In practice, in the block-level intermediate representation of the TVM compiler, the merging sparse blocks (e.g., merging four adjacent 64×64 blocks into a 128×128 block) are shaped based on hardware characteristic information (e.g., the thread bundle size of an NVIDIA A100 GPU is 32, and the memory alignment requirement is 128 bytes). If the actual non-zero data area of ​​any merged block is 128 rows × 93 columns, zero elements are padded to 128×128 to align the block size with the hardware memory access granularity. Simultaneously, the index array in CSR format (Compressed Sparse Row Format) is updated. This can be done by expanding the `col_idx` array to include virtual indices for the padded columns (e.g., marking the padded columns as -1) and adjusting the offsets of the `row_ptr` array to reflect the padded row boundaries. The final output is the shaped, merged sparse blocks. Shape adjustments can also align the number of block columns to hardware access granularity (e.g., 128 columns), ensuring that each thread accesses adjacent elements and that all threads within the same thread bundle access contiguous and aligned memory addresses.

[0068] The third step involves generating dynamic scheduling information for each fused sparse block after shape adjustment within the operator layer of the compiler stack. This information is based on the sparse block execution time information and the corresponding attribute information of each sparse block. The dynamic scheduling information includes a sparse block priority queue, a computation unit status monitoring task, and a real-time load feedback task. The sparse block priority queue is typically a task queue sorted by optimization objective, determining the block computation order and hardware allocation priority. The sorting rule for the sparse block priority queue can prioritize high-sparse-rate, low-time-consumption sparse blocks. The computation unit status monitoring task is typically a background task that periodically collects hardware resource usage data, providing real-time input for dynamic scheduling. Monitoring metrics for the computation unit status monitoring task can include streaming multiprocessor utilization, memory bandwidth (memory access throughput), and GPU temperature. The streaming multiprocessor is typically the core computation unit of the GPU, containing multiple CUDA cores, shared memory, and a scheduler. The real-time load feedback task is typically an algorithm module that dynamically adjusts task allocation based on monitoring data. When the aforementioned real-time load feedback task detects that the utilization of a streaming multiprocessor is >90% for three consecutive cycles, it migrates two sparse blocks (such as Block_3 and Block_5) in its queue to an idle streaming multiprocessor (e.g., idle streaming multiprocessor utilization is 45%). When the actual execution time of the aforementioned sparse block exceeds the predicted value by 20% (e.g., predicted 1.2ms, actual 1.5ms), the aforementioned real-time load feedback task lowers the priority of its subsequent blocks of the same type by one level. In the operator layer of the TVM compiler, the dynamic scheduler for the aforementioned block-level intermediate representation (containing a 128×128 block shape, CSR format, 92% sparsity, and a predicted execution time of 1.2ms / 2.1ms for dense cores) constructs a sparse block priority queue, starts a computational unit status monitoring task, and executes a real-time load feedback task to obtain dynamic scheduling information.

[0069] The fourth step involves generating sparse and dense computation kernels based on the aforementioned dynamic scheduling information. The sparse kernel can be a kernel function optimized for sparse formats (such as CSR). The dense kernel can be a kernel function optimized for dense data, utilizing vectorized instructions (such as SIMD) to batch process contiguous memory data. In practice, the execution entity performs index parsing, multiply-accumulate operations, and hardware optimization on the sparse blocks (row_ptr, col_idx, data) and dense input matrix in CSR format to obtain the sparse kernel. The hardware optimization typically refers to using the GPU's Tensor Cores (which support structured sparse computation) and shared memory to cache frequently accessed data. The shared memory typically refers to an on-chip cache shared within the sparse blocks of GPU threads, used to reduce global memory access latency (e.g., caching frequently accessed portions of the input matrix). Next, vectorized computation and memory optimization are performed on the dense format blocks (contiguous memory arrays) and the input matrix to obtain the dense kernel. The aforementioned vectorized computation typically refers to using CUDA Core's WMMA (Warp Matrix Multiply Accumulate) instruction to batch process 128×128 matrix multiplications. The aforementioned memory optimization typically refers to reducing global memory access by caching input matrix blocks using shared memory.

[0070] The fifth step involves interleaving the indexing processing unit of the sparse computing core with the vectorized computing unit of the dense computing core through an inter-core communication interface, forming a collaborative computing core. This inter-core communication interface typically refers to the mechanism for exchanging data between different computing cores in a GPU, including shared memory, global memory atomic operations, and semaphores. The indexing processing unit usually refers to the module in the sparse core that parses sparse formats (such as row_ptr and col_idx in CSR), used to locate non-zero elements. The vectorized computing unit is typically the module in the dense core that uses SIMD instructions (such as WMMA in Tensor Cores) to batch process data. The instruction-level interleaving typically refers to coordinating the timing of different computing units through synchronization instructions (such as __syncthreads()), allowing them to execute alternately in a pipeline. The collaborative computing core is typically a hybrid kernel function integrating sparse index parsing and dense computing. The atomic operations typically refer to uninterruptible read / write operations (such as atomicAdd used to accumulate semaphores). The semaphores typically refer to synchronization flag variables (such as updating semaphores after the sparse core completes data writing). In practice, the aforementioned execution entity can, on an NVIDIA A100 GPU, use CUDA's inter-core communication interface (such as shared memory and atomic operations) to interleave the index processing unit (parsing row_ptr and col_idx in CSR format) of the sparse computing core with the vectorized computing unit (using WMMA instructions of the Tensor Core) of the dense computing core at the instruction level, forming a unified collaborative computing core.

[0071] Step 6: Based on the aforementioned collaborative computing core, generate the execution time of the corresponding processing task for that collaborative computing core. In practice, the execution entity can obtain the execution time of the collaborative computing core by combining the performance model of the sparse operator and hardware test calibration (using nvprof to collect GPU instruction cycle count and memory transaction count), along with task parameters (95% sparsity, block size 128×128) and hardware configuration (such as 108 SMs of A100, 1555GB / s memory bandwidth). For example, first, the theoretical time consumption of each stage is added together (such as 0.3ms parsing + 0.15ms loading + 0.2ms calculation = 0.65ms), then it is corrected according to the parallelism (such as the GPU processing 4 blocks simultaneously, the equivalent single block time is 0.65ms / 4≈0.16ms), and finally the calibrated execution time is output (such as measured 0.17ms ± 5% error).

[0072] Step 7: Generate visualization information based on the execution time mentioned above. In practice, the execution entity can organize the execution time of the processing tasks corresponding to the collaborative computing kernels, collect the speedup ratio of each model to this method under different sparsity rates, and use visualization tools (such as Matplotlib or Seaborn) to generate visualization information (such as charts or curves).

[0073] Step 8: Display the above visualization information. In practice, the execution entity can use Python libraries such as Matplotlib or Seaborn to generate static charts and color-code the visualization information. Then, display the static charts.

[0074] Steps one through eight above constitute an inventive point of this disclosure, solving the technical problem that "existing solutions (such as TVM-Sparse) only support static sparse format compilation, lacking runtime sparsity awareness and computational paradigm switching capabilities, resulting in the inability to achieve end-to-end optimization from high-level model sparsification to low-level hardware instructions." This leads to the inability of existing technologies to meet the requirements for efficient sparse matrix computation as follows: existing sparse matrix computation schemes are deficient in dynamic sparsity awareness, flexible computational paradigm switching, and end-to-end optimization implementation, making it difficult to effectively adapt to data distributions with different sparsity rates. Furthermore, they have deficiencies in hardware resource utilization and load balancing, resulting in low computational efficiency and failing to fully meet the requirements for efficient sparse matrix computation. Solving these factors can improve sparse matrix computation efficiency, optimize hardware resource utilization, and achieve end-to-end performance optimization. To achieve this effect, this disclosure employs a sparse matrix computation method based on runtime sparsity awareness and computational paradigm switching. This method involves inserting block-level intermediate representations to carry the execution time and attribute information of sparse blocks; adjusting the shape of the fused sparse blocks according to hardware characteristics; generating dynamic scheduling information by combining the sparse block execution time and attribute information; generating sparse and dense computational cores based on the dynamic scheduling information; achieving instruction-level interleaving of sparse and dense computational cores through inter-core communication interfaces to form a collaborative computational core; generating the execution time of the tasks corresponding to the collaborative computational cores; and generating and displaying visualization information. This enables end-to-end optimization from high-level model sparsification to low-level hardware instructions. It effectively detects runtime sparsity, flexibly switches computational paradigms, optimizes each stage of sparse matrix computation, improves computational efficiency and hardware resource utilization, thereby meeting the requirements for efficient sparse matrix computation.

[0075] The above embodiments of this disclosure have the following beneficial effects: Through the acceleration method based on an irregular sparse model in some embodiments of this disclosure, dynamic resource scheduling and end-to-end collaborative optimization are achieved, significantly improving the computational efficiency of the sparse model. Specifically, traditional sparse acceleration methods often have the following problems: They fail to model the performance of sparse sub-blocks in regions with different sparsity rates, resulting in the inability to accurately identify sub-blocks suitable for sparse or dense computation in irregular distribution scenarios, leading to wasted computational resources; they use a uniform format for the irregular distribution matrix, resulting in low memory efficiency due to redundant storage of sparse sub-blocks; they map the entire sparse matrix to a single type of computational unit, causing different types of sub-blocks in the sparse matrix to fail to be mapped to the corresponding computational units, resulting in low performance improvement and wasted resources; and they do not consider cross-batch load correlation, causing a decrease in end-to-end inference throughput. Based on this, the acceleration method based on an irregular sparse model in some embodiments of this disclosure first inputs the sparse model into a target compiler, wherein the target compiler includes layer intermediate representation, block layer intermediate representation, and operator layer intermediate representation. Thus, the sparse model can be used as a data source. Next, in the intermediate representation of the above layer, the sparse weight matrix of the sparse model is split into sparse sub-blocks to obtain a split sparse weight matrix, which includes each sparse sub-block. This allows for the partitioning of the sparse weight matrix in the sparse model for subsequent processing. Then, based on the sparse operator performance model, the attribute information of each sparse sub-block in the split sparse weight matrix is ​​generated. Thus, by modeling the performance of sparse sub-blocks in different sparsity regions, a sparse operator performance model can be obtained, facilitating the prediction of the attribute information of each sparse sub-block and reducing computational resource waste. Then, in the intermediate representation of the block layer, based on the attribute information of each sparse sub-block, the sparse sub-blocks are fused to obtain a fused sparse weight matrix. In this fused sparse weight matrix, each sparse block includes each target fused sparse block and each unfused sparse sub-block, and each target fused sparse block corresponds to attribute information. Therefore, the aforementioned sparse sub-blocks can be fused and their format encoded to reduce redundant storage of the fused sparse weight matrix, thereby improving memory efficiency. Then, in the intermediate representation of the operator layer, based on the attribute information corresponding to each sparse block, a mapping is performed between each sparse block and a computational unit to obtain the mapping result. This allows the sparse blocks to be mapped to different computational units according to their type, improving overall performance and allocating resources more efficiently. Finally, based on the mapping result, a processing task corresponding to the sparse model is generated and executed. Thus, the model can be optimized and tasks processed using the mapping result and the sparse operator performance model, resulting in an acceleration effect.Furthermore, when processing computational tasks across batches, high-load and low-load sparse sub-blocks in adjacent sparse weight matrices can be cross-mapped to different computational units. This improves the load correlation between different batches and increases end-to-end inference throughput. It adapts to different sparsity patterns and hardware environments, demonstrating significant advantages in large-model inference scenarios. It provides an efficient sparse acceleration solution for edge computing and large language model deployment, significantly improving the applicability of models in mobile and cloud inference scenarios, and breaking through the benefit boundaries of traditional irregular sparse acceleration designs.

[0076] Further reference Figure 4 This serves as a multi-dimensional performance comparison bar chart according to one embodiment of the present disclosure.

[0077] like Figure 4As shown, this chart presents a performance comparison analysis of different sparse acceleration algorithms on large-scale language model inference tasks. The bar charts compare the speedup effects of seven acceleration algorithms (cuBLAS, SparseTIR, and the applicant's proposed method "Ours") on five mainstream models (Bert-base, Bert-large, GPT-2, Llama-2, and Bart-large) at sparsity rates ranging from 50% to 90%, using a multi-dimensional arrangement. At the top of the charts, each sub-chart title clearly indicates the corresponding sparsity rate (e.g., "90% Sparsity"). The five horizontally arranged sub-charts visually present the speedup differences between different models at the same sparsity rate. The X-axis of each sub-chart lists the specific model names such as BERT and GPT-2, while the Y-axis uniformly uses "Speedup Ratio" to quantify the speedup effect. The baseline is the unaccelerated original performance (speedup ratio of 1.0x). Different colored bars distinguish algorithm types, with the red bars specifically highlighting the researcher's proposed "Ours" method, creating a visual focal point for comparison. Looking at the specific data distribution, in the scenario of 90% high sparsity (leftmost subplot), the "Ours" method achieves a peak speedup of 3.75x on the Llama-2 model, with its red bars significantly outperforming the adjacent SparseTIR (blue, speedup approximately 2.8x) and cuBLAS (gray, speedup approximately 1.5x). As the sparsity decreases to 50% (rightmost subplot), the speedup of all algorithms shows a downward trend. For example, the speedup of "Ours" on Llama-2 drops to 2.3x, but it still maintains a performance advantage over other algorithms. Particularly noteworthy is that the traditional algorithm cuBLAS (gray bars) consistently achieves a speedup below 2.0x at all sparsity levels, and exhibits negative optimization with a speedup less than 1.0 on large models such as Bert-large. The charts reveal the impact of model architecture on speedup performance through vertical comparison: Llama-2, based on the Transformer-XL architecture (the rightmost model in each subgraph), exhibits the highest speedup across all sparsity rates, while the speedup gains of the basic BERT model (Bert-base) are relatively limited. This difference may stem from the synergistic effect of large model parameter count and sparse computation optimization. Furthermore, the legend box in the upper right corner of the taskbar clearly indicates the color coding of the seven algorithms, helping to quickly identify the performance distribution characteristics of different algorithm series.

[0078] Overall, this set of comparative graphs systematically verifies the effectiveness of sparse acceleration algorithms in LLM inference optimization. Through hierarchical visualization, it reveals the patterns of algorithm performance variation with model size and sparsity, providing data support for the joint optimization of algorithm selection and sparsity strategies in practical deployments. For example, in 50% sparse scenarios requiring high precision, the "SparseTIR+Ours" combination can be prioritized, while in 90% sparse scenarios with high compression rates, the "Ours" method alone can achieve optimal acceleration.

[0079] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an acceleration device based on a random sparse model. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0080] like Figure 2 As shown, an acceleration device 200 based on an irregular sparse model in some embodiments includes: an input unit 201, a splitting unit 202, a first generation unit 203, a fusion unit 204, a mapping unit 205, and a second generation unit 206. The input unit 201 is configured to input the sparse model into a target compiler, wherein the target compiler includes a layer intermediate representation, a block layer intermediate representation, and an operator layer intermediate representation; the splitting unit 202 is configured to split the sparse weight matrix of the sparse model into sparse sub-blocks in the layer intermediate representation to obtain a split sparse weight matrix, wherein the split sparse weight matrix includes each sparse sub-block; the first generation unit 203 is configured to generate attribute information of each sparse sub-block in the split sparse weight matrix based on a sparse operator performance model; the fusion unit 204 is configured to, in the block layer intermediate representation, based on... Based on the attribute information of each sparse sub-block, the sparse sub-blocks are fused to obtain a fused sparse weight matrix. Each sparse block in the fused sparse weight matrix includes a target fused sparse block and an unfused sparse sub-block. Each target fused sparse block has corresponding attribute information. The mapping unit 205 is configured to map each sparse block to a computation unit in the intermediate representation of the operator layer according to the attribute information corresponding to each sparse block, and obtain a mapping result. The second generation unit 206 is configured to generate a processing task corresponding to the sparse model based on the mapping result, and execute the processing task.

[0081] It is understandable that the units described in the device 200 are related to the reference. Figure 1The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.

[0082] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0083] like Figure 3 As shown, the electronic device 300 may include a processing unit 301 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0084] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0085] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0086] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0087] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0088] The aforementioned computer-readable medium may be included within the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: input a sparse model into a target compiler, wherein the target compiler includes a layer intermediate representation, a block intermediate representation, and an operator intermediate representation; in the layer intermediate representation, decompose the sparse weight matrix of the sparse model into sparse sub-blocks to obtain a decomposed sparse weight matrix, wherein the decomposed sparse weight matrix includes each sparse sub-block; and based on a sparse operator performance model, generate attribute information for each sparse sub-block in the decomposed sparse weight matrix; In the aforementioned block-layer intermediate representation, based on the attribute information of each sparse sub-block, the sparse sub-blocks are fused to obtain a fused sparse weight matrix. Each sparse block in the fused sparse weight matrix includes target fused sparse blocks and unfused sparse sub-blocks, and each target fused sparse block corresponds to attribute information. In the aforementioned operator-layer intermediate representation, based on the attribute information corresponding to each sparse block, each sparse block is mapped to a computational unit to obtain a mapping result. Based on the mapping result, a processing task corresponding to the aforementioned sparse model is generated, and the processing task is executed.

[0089] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0091] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an input unit, a splitting unit, a first generation unit, a fusion unit, a mapping unit, and a second generation unit. The names of these units do not necessarily limit the specific unit; for example, an input unit may also be described as "a unit that inputs a sparse model into a target compiler."

[0092] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0093] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the above-described acceleration methods based on an irregular sparse model.

[0094] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. An acceleration method based on an irregular sparse model, comprising: The sparse model is input into the target compiler, which includes layer intermediate representation, block layer intermediate representation and operator layer intermediate representation; In the intermediate representation of the layer, the sparse weight matrix of the sparse model is split into sparse sub-blocks to obtain the split sparse weight matrix, wherein the split sparse weight matrix includes each sparse sub-block. Based on the sparse operator performance model, the attribute information of each sparse sub-block in the split sparse weight matrix is ​​generated; Based on the attribute information of each sparse sub-block, adjacent sparse sub-blocks of the same type in each sparse sub-block are fused to obtain a sparse block after the fusion of each homogeneous block. For each homogeneous block merged into a sparse block, perform the following steps: Based on the attribute information of the adjacent sparse sub-blocks corresponding to the sparse block after the homogeneous block fusion, the attribute information of the sparse block after the homogeneous block fusion is generated. In response to the fact that the attribute information of the sparse block after the homogeneous block fusion satisfies the homogeneous block fusion condition, the sparse block after the homogeneous block fusion is determined as the target sparse block after the homogeneous block fusion. The adjacent sparse sub-blocks of different types in each sparse sub-block are fused into non-homogeneous blocks to obtain a sparse block after the fusion of each non-homogeneous block. For each sparse block obtained after merging non-homogeneous blocks, perform the following steps: Based on the attribute information of the adjacent sparse sub-blocks corresponding to the sparse block after the fusion of the non-homogeneous blocks, the attribute information of the sparse block after the fusion of the non-homogeneous blocks is generated. In response to the fact that the attribute information of the sparse block after the fusion of the non-homogeneous blocks satisfies the non-homogeneous block fusion condition, the sparse block after the fusion of the non-homogeneous blocks is determined as the target sparse block after the fusion of the non-homogeneous blocks. Based on the sparse blocks resulting from the fusion of the homogeneous blocks and the sparse blocks resulting from the fusion of the non-homogeneous blocks, a fused sparse weight matrix is ​​generated. In the intermediate representation of the operator layer, each sparse block is mapped to a computation unit based on the attribute information corresponding to each sparse block to obtain the mapping result; Based on the mapping result, a processing task corresponding to the sparse model is generated, and the processing task is executed.

2. The method according to claim 1, wherein, The attribute information of each sparse sub-block in the split sparse weight matrix generated based on the sparse operator performance model includes: Obtain the actual execution time data for matrix multiplication of each candidate matrix under various sparsity rates; Based on the actual execution time data, a sparse operator performance model is generated; Based on the sparse operator performance model, the attribute information of each sparse sub-block is generated.

3. The method according to claim 2, wherein, The generation of a sparse operator performance model based on the actual execution time data includes: Based on the actual execution time data, the fitted execution time of each candidate matrix is ​​generated; Based on the fitting execution time of each candidate matrix, a threshold for each candidate matrix is ​​generated; A sparse operator performance model is generated based on the threshold of each candidate matrix and the fitting execution time of each candidate matrix.

4. The method according to claim 1, wherein, The method further includes: Obtain the attribute information of each sparse block in the fused sparse weight matrix, wherein each sparse block in the fused sparse weight matrix includes each target fused sparse block and each unfused sparse sub-block. Based on the attribute information of each sparse block in the fused sparse weight matrix, the format encoding of each sparse block in the fused sparse weight matrix is ​​performed.

5. The method according to claim 1, wherein, In the intermediate representation of the operator layer, based on the attribute information corresponding to each sparse block, each sparse block is mapped to a computational unit to obtain a mapping result, including: For each of the sparse blocks, perform the following steps: In response to the sparse type of the attribute information representation of the sparse block, the sparse block is mapped to a sparse computing unit; In response to the fact that the attribute information representation type of the sparse block is dense, the sparse block is mapped to a dense computing unit; In response to the sparse block's attribute information representation type being a swing type, the sparse block is mapped to a computing unit that meets a preset load condition. In response to the simultaneous mapping of the sparse weight matrix after fusion between adjacent batches on the computing unit, the mapping order of each sparse block and the computing unit in the sparse weight matrix after fusion between adjacent batches is rearranged to obtain matrix adjustment information. The mapping relationship information, the adjustment information between the matrices, and the attribute information of each sparse block are determined as the mapping result.

6. An acceleration device based on an irregular sparse model, comprising: An input unit is configured to input a sparse model into a target compiler, wherein the target compiler includes layer intermediate representations, block intermediate representations, and operator intermediate representations; A splitting unit is configured to split the sparse weight matrix of the sparse model into sparse sub-blocks in the intermediate representation of the layer to obtain a split sparse weight matrix, wherein the split sparse weight matrix includes each sparse sub-block. The first generation unit is configured to generate attribute information of each sparse sub-block in the split sparse weight matrix based on the sparse operator performance model. The fusion unit is configured to perform homogeneous block fusion on adjacent sparse sub-blocks of the same type in each sparse sub-block according to the attribute information of each sparse sub-block, to obtain fused sparse blocks; for each fused sparse block obtained, the following steps are performed: generating attribute information of the fused sparse block according to the attribute information of the adjacent sparse sub-blocks corresponding to the fused sparse block; in response to the attribute information of the fused sparse block satisfying the homogeneous block fusion condition, determining the fused sparse block as the target fused sparse block; and merging dissimilar types in each sparse sub-block. The adjacent sparse sub-blocks are fused into non-homogeneous blocks to obtain sparse blocks after fusion. For each sparse block after fusion, the following steps are performed: Attribute information of the sparse block after fusion is generated based on the attribute information of the adjacent sparse sub-blocks corresponding to the sparse block after fusion; In response to the attribute information of the sparse block after fusion satisfying the non-homogeneous block fusion condition, the sparse block after fusion is determined as the target sparse block after fusion; A fused sparse weight matrix is ​​generated based on the sparse blocks after fusion of homogeneous blocks and the sparse blocks after fusion of non-homogeneous blocks. The mapping unit is configured to map each sparse block to a computation unit in the intermediate representation of the operator layer according to the attribute information corresponding to each sparse block, so as to obtain the mapping result; The second generation unit is configured to generate a processing task corresponding to the sparse model based on the mapping result, and to execute the processing task.

7. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 5.

8. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images