CPU double-path acceleration method for large-scale LLM sparse matrix multiplication

Abstract

The application discloses a CPU double-path acceleration method for large-scale LLM sparse matrix multiplication, and relates to the technical field of high-performance computing. The method first constructs a fast matrix multiplication algorithm lookup table, generates an optimized algorithm through random walk and performs Hensel promotion to compensate for numerical errors; then performs two-stage division and zero block identification on the weight matrix, classifies it into complete zero blocks, near-zero blocks and dense blocks; then aggregates non-zero elements for the near-zero blocks and merges dense blocks into large blocks using dynamic programming; uses multi-core CPU to concurrently perform sparse path and dense path calculation, respectively calls sparse kernels and matching fast algorithms, and finally fuses the results based on position mapping using vector instructions. The application solves the problem of low utilization of unstructured sparse matrices through double-path shunting strategy and software and hardware co-optimization, significantly improves the calculation throughput and hardware resource utilization while ensuring the model accuracy.

Description

Technical Field

[0001] This invention relates to the field of high-performance computing technology, specifically to a CPU dual-path acceleration method for large-scale LLM sparse matrix multiplication. Background Technology

[0002] With the rapid development of natural language processing technology, large-scale language models (LLMs) have demonstrated outstanding capabilities in various artificial intelligence applications. However, their massive parameter scale brings enormous computational pressure and resource consumption to inference deployment. To reduce model deployment costs and improve inference efficiency, model compression techniques, especially weight pruning, have become a mainstream approach, aiming to introduce sparsity by removing redundant parameters from the model. Although graphics processing units (GPUs) dominate the model training phase, central processing units (CPUs) remain indispensable computing platforms in edge computing and cost-sensitive inference scenarios due to their versatility, accessibility, and advantages in large-capacity memory management.

[0003] In existing technologies, accelerating the inference process of pruned models on CPUs typically relies on general-purpose linear algebra libraries or specialized sparse matrix multiplication (SpMM) libraries. These existing solutions usually employ compressed storage formats to handle sparse matrices or attempt to reduce the theoretical number of computations using general-purpose fast matrix multiplication algorithms. In practical applications, the system calls matrix operation tasks during inference to standard libraries such as MKL and OpenBLAS, hoping to improve throughput by skipping zero-element calculations or utilizing instruction set parallelism to meet the computational needs of pruned model weights.

[0004] However, existing technologies still have significant shortcomings when handling large-scale LLM sparse matrix multiplication. First, traditional CPU sparse computing libraries are mainly optimized for extremely high sparsity (typically greater than 95%) or specific structured sparse patterns. Modern LLMs, to maintain model accuracy, typically employ unstructured pruning with sparsity at a moderate level (e.g., 50%-80%). This unstructured weight matrix with moderate sparsity leads to severe instruction pipeline stalls and load imbalances in existing libraries, making it difficult to translate sparsity into actual performance gains. Second, while general-purpose fast matrix multiplication algorithms reduce computational complexity, they often introduce non-negligible numerical approximation errors. These accumulated errors significantly degrade model accuracy in deep networks, causing inference results with perplexity exceeding usable limits. Furthermore, existing general-purpose computing solutions often lack deep understanding of the underlying hardware topology. When facing large-scale matrix operations, they fail to effectively address the high latency issues of cross-memory domain access and cannot fully utilize the CPU's multi-level cache and vectorized instructions for irregular sparse data distributions, resulting in actual hardware resource utilization far below theoretical peaks. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a CPU dual-path acceleration method for large-scale LLM sparse matrix multiplication, which solves the problems of low computing resource utilization, high model inference latency, and loss of accuracy caused by the mismatch between the unstructured sparse weight matrix and the dense computing units of the CPU in large-scale language models in CPU inference scenarios.

[0006] To achieve the above objectives, this invention provides the following technical solution: a CPU dual-path acceleration method for large-scale LLM sparse matrix multiplication, comprising the following steps: constructing a fast matrix multiplication algorithm lookup table; defining the search space by enumerating scenario parameters for sparse matrix multiplication; iteratively generating an optimized fast matrix multiplication algorithm using a random walk strategy; performing Hensel boosting on the algorithm to compensate for numerical errors; and storing the algorithm and parameters in the lookup table; performing two-level matrix partitioning and zero-block identification; performing a first-level partitioning of the unstructured pruned weight matrix; identifying the sub-matrices as completely zero blocks, near-zero blocks, or dense blocks based on the element value distribution characteristics of the sub-matrices; preparing dual-path data; extracting non-zero elements from near-zero blocks and aggregating them into a highly sparse matrix; performing a second-level partitioning for dense blocks to form large blocks; concurrently executing dual-path computation using a multi-core CPU; wherein the sparse path calls the sparse matrix multiplication kernel for the highly sparse matrix, and the dense path queries the lookup table for large blocks and calls the matching algorithm to perform block multiplication; and finally establishing a position mapping relationship, concatenating the computation results of the sparse path and the dense path into the final matrix.

[0007] Preferably, the construction of the fast matrix multiplication algorithm lookup table specifically includes: setting the matrix block granularity to align with the basic computing unit of the CPU, and updating the current solution with minimizing the number of block multiplications as the evaluation index; the Hensel boosting operation ensures, through a numerical precision compensation mechanism, that the perplexity of the model corresponding to the algorithm calculation result does not exceed a preset threshold.

[0008] Preferably, the execution matrix two-level partitioning and zero block identification specifically includes: setting the block size to an integer multiple of the CPU basic computing unit, uniformly cutting the weight matrix along the row and column directions; if the absolute value of all elements in the submatrix is ​​less than or equal to the first threshold, it is marked as a completely zero block; if the proportion of non-zero elements with an absolute value greater than the first threshold is lower than the second threshold, it is marked as a near-zero block; and the rest are marked as dense blocks.

[0009] Preferably, the dual-path data preparation specifically includes: traversing near-zero blocks to extract non-zero element values ​​and their coordinate information based on block index and intra-block offset, and aggregating them into a highly sparse matrix according to the coordinate format; for dense blocks, based on the algorithm estimated by the lookup table, using a dynamic programming method to merge them into large blocks with the goal of minimizing the total number of inter-block multiplications, and limiting the storage size of each large block to no more than the capacity of a single CPU memory domain.

[0010] Preferably, the dense path calculation specifically includes: for each large block after secondary partitioning, matching algorithm parameters containing block granularity and block multiplication order in a lookup table, splitting the large block into sub-blocks and performing multiplication and addition operations according to the block multiplication order; wherein completely zero blocks do not perform multiplication operations and directly output a zero matrix, and the calculation results of each block are aggregated according to the original position information.

[0011] Preferably, the result fusion specifically includes: mapping sparse path results to near-zero block regions and dense path results to dense block and completely zero block regions based on the block index information of the first-level partition; and using vector transmission instructions to splice data according to the position mapping relationship to generate a final matrix multiplication result with consistent dimensions.

[0012] Preferably, the method further includes a multi-core CPU adaptation optimization strategy: by using NUMA-aware thread binding, the working thread is bound to a physical core in the same memory domain, and the computation data is stored locally; in dense path computation, hardware-oriented prefetch instructions are called according to the block multiplication order of the algorithm to prefetch subsequent submatrices into the L2 cache; in the block multiplication and aggregation process, fusion multiply-accumulate instructions are called and data is processed using vector alignment.

[0013] This invention provides a CPU dual-path acceleration method for large-scale LLM sparse matrix multiplication. It offers the following advantages: 1. This invention precisely decomposes the unstructured pruned weight matrix into completely zero blocks, near-zero blocks, and dense blocks. The sparse path uses a dedicated kernel to aggregate and process discrete non-zero elements, while the dense path uses dynamic programming to merge dense blocks into large blocks that fit the CPU cache and calls a fast matrix multiplication algorithm. This divide-and-conquer strategy effectively eliminates invalid computations on zero elements, avoids instruction pipeline stalls caused by irregular data distribution, and fully leverages the advantages of dense computation in continuous memory access. It achieves a significant improvement in the throughput of unstructured sparse matrix multiplication operations on multi-core CPUs.

[0014] 2. By using a random walk strategy to iteratively generate a fast matrix multiplication algorithm that adapts to the scene parameters within the search space, and introducing a numerical precision compensation mechanism based on Hensel enhancement to correct potential numerical errors in the algorithm, this combination of algorithm adaptation and precision compensation not only reduces computational complexity by decreasing the total number of multiplication operations, but also effectively curbs the cumulative errors caused by the approximate calculations of the fast algorithm. This ensures that the perplexity of the accelerated matrix multiplication results remains within the usable threshold range when substituted into the model for inference, achieving a balance between optimizing computational efficiency and maintaining the accuracy of model inference.

[0015] 3. By implementing NUMA-aware thread binding to achieve localized data storage, and by calling hardware-directed prefetch instructions in the order of the algorithm to load submatrices into the L2 cache during intensive path calculations, and by using the AVX-512 instruction set to perform vectorized fusion multiplication and addition operations in the low-level operation, this hardware and software co-optimization mechanism significantly reduces cross-memory domain access latency, masks the memory overhead of data loading, maximizes the instruction-level parallelism and cache hit rate of multi-core processors, and effectively reduces end-to-end latency of large-scale matrix operations. Attached Figure Description

[0016] Figure 1 This is a flowchart of the workflow of AlphaSparseLLM of the present invention; Figure 2 Performance evaluation diagram of the present invention at different sparsity levels Figure 3 This diagram illustrates the effective performance of the LLM sparse matrix workload under different token lengths according to the present invention. Figure 4 This is a performance evaluation graph of the LLM matrix of the present invention under different sparsity ratios. Figure 5 The performance graph of the LLM matrix of this invention under different sparsity ratios is shown. Figure 6 This is a schematic diagram illustrating the end-to-end inference throughput and latency of the six 70% pruning models of this invention. Detailed Implementation

[0017] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] The system of this invention is used to accelerate sparse matrix multiplication operations on multi-core CPUs for large-scale language models (LLMs) that have undergone unstructured pruning. Through hardware-aware automated workflows and CPU architecture-specific optimizations, it achieves a balance between computational efficiency and model accuracy. The system architecture is built around four core components: fast matrix multiplication algorithm adaptation, two-level matrix partitioning, dual-path parallel computation, and multi-core CPU optimization. These components work together to complete the entire process from algorithm generation to parallel computation. The complete workflow is as follows: See attached document Figure 1-6 , Figure 1 This is a flowchart of the AlphaSparseLLM workflow according to an embodiment of the present invention. The Fast Matrix Multiplication (FMM) algorithm search and lookup table construction is the first core stage of the entire workflow. By deriving and storing optimized FMM algorithms for different scenario parameters, efficient algorithmic support is provided for subsequent matrix multiplication calculations. The specific implementation process is as follows: S201: The scenario parameter enumeration clarifies the core scenario parameters for sparse matrix multiplication in LLM inference, covering three key types of information: matrix multiplication dimension, sparsity level, and sparsity pattern. The matrix multiplication dimension adapts to the matrix dimensions of each layer in models such as LLaMA-2 7B / 13B / 70B and Qwen-2 7B / 57B-A14B / 72B, specifically represented by different triples (a×b×c), where a represents the number of rows in the weight matrix, b represents the number of columns in the weight matrix and is equal to the number of rows in the input matrix, and c represents the number of columns in the input matrix. The sparsity level includes 50%, 60%, 70%, 80%, 90%, and 98%, covering the common sparsity range after LLM pruning. The sparsity pattern is divided into two categories: unstructured sparsity and structured sparsity, with unstructured sparsity being the mainstream sparsity pattern for maintaining accuracy in LLM.

[0019] S202: Random walk-based FMM algorithm search: For each set of enumerated scene parameters, perform FMM algorithm search to obtain a suitable optimization algorithm.

[0020] S2021: Define the search space, which is composed of a combination of matrix partitioning method and block multiplication order. The partitioning method supports partitioning along the matrix rows and columns, and the partitioning granularity is aligned with the basic computing units supported by the CPU to ensure that the algorithm is compatible with the hardware computing characteristics. The block multiplication order includes multiple methods such as row priority and column priority, which are adapted to the CPU cache access characteristics to reduce the probability of cache miss.

[0021] S2022: Initialize the search starting point, using traditional matrix multiplication as the initial solution. The block granularity of traditional matrix multiplication is set to the basic computing unit of the CPU, and the block multiplication order is set to row priority.

[0022] S2023: Iteratively generate the optimal solution. Traverse the search space using a random walk strategy. In each iteration, make minor adjustments to the block granularity and block multiplication order of the current solution to generate multiple candidate solutions. For each candidate solution, calculate the number of block multiplications corresponding to the candidate solution using only minimizing the number of block multiplications as the evaluation metric. Select the candidate solution with the fewest number of block multiplications as the current solution for the next iteration. The number of iterations is preset to 100, or until the number of block multiplications no longer decreases after 10 consecutive iterations. The current solution at this point is the Mabc algorithm of the FMM algorithm adapted to the parameters of this scenario.

[0023] S203: The Hensel boosting operation performs Hensel boosting on the searched FMM algorithm Mabc to compensate for numerical errors that may be introduced by reducing computational complexity, ensuring that the algorithm meets the accuracy requirements of LLM inference. The boosting process is achieved through a numerical accuracy compensation mechanism, so that the perplexity of the model corresponding to the optimized FMM algorithm calculation result is within an acceptable range. Referring to the perplexity levels under different pruning methods in Tables 1 and 2, those skilled in the art can implement the specific numerical calculation logic of Hensel boosting based on existing numerical accuracy compensation techniques. Its core principle is a well-known technology in this field and will not be elaborated here.

[0024] In this step, the accuracy compensation effect of Hensel enhancement must be directly linked to the perplexity requirement of LLM inference. The calculation result of the compensated FMM algorithm must ensure that the perplexity of the model does not exceed the acceptable threshold for the corresponding sparsity in Tables 1 and 2. For example, for the LLaMA2-7B model with 70% unstructured sparsity, the perplexity calculated by the FMM algorithm after Hensel enhancement must be ≤30.29; for the Qwen2-7B model with 80% sparsity, the perplexity must be ≤319.94, ensuring that the algorithm reduces computational complexity without compromising model fidelity.

[0025] Table 1:

[0026] Table 2:

[0027] S204: Lookup table construction and storage: S2041: Design a lookup table structure with scene parameter combinations as index keys and corresponding optimized FMM algorithms and related parameters as storage values. The index key is composed of matrix multiplication dimension triples (a×b×c), sparsity level, and sparsity mode identifier to ensure that each scene parameter combination corresponds to a unique index key. The storage value contains the derived FMM algorithm code snippet, block granularity parameters, and block multiplication order parameters to provide complete algorithm information for subsequent calls.

[0028] S2042: Fill and store the lookup table. Store the optimized FMM algorithm and parameters corresponding to all scene parameter combinations into the lookup table one by one. The storage medium is a high-speed memory database, which supports fast query operations. This ensures that when performing matrix multiplication calculations, the corresponding FMM algorithm can be quickly matched according to the current scene parameters, thus meeting the efficiency requirements of real-time inference.

[0029] The two-level partitioning and zero-block extraction mechanism is the core link connecting algorithm adaptation and dual-path execution. By performing hierarchical partitioning on the pruned LLM weight matrix, zero blocks are accurately identified and non-zero elements are extracted, laying the foundation for subsequent dual-path parallel computation. The specific implementation process is as follows: S301: First-level partitioning (small block partitioning): S3011: Determine the block size, which is aligned with the basic computing units supported by the CPU. The core purpose is to maximize the extraction of all-zero blocks. The block size is set to an integer multiple of the CPU's basic computing units to adapt to the vector processing capabilities of the AVX-512 instruction set and ensure that the sub-blocks can be efficiently adapted to subsequent hardware optimization operations.

[0030] S3012: Perform a partitioning operation to uniformly divide the unstructured pruned LLM weight matrix. Cut the matrix along the row and column directions according to the preset block size to form several regular small sub-matrices. If the matrix dimension cannot be divided by the block size during the partitioning process, zeros are added to the end of the matrix. The zero-added parts are subsequently marked as zero blocks and do not participate in the actual calculation.

[0031] S3013: Sub-block storage, all sub-blocks are stored in row-major order in the CPU's L3 cache or local memory domain, adapting to the memory distribution characteristics of NUMA architecture and reducing cross-node transmission overhead during subsequent data access.

[0032] S302: Zero Block Recognition: S3021: Set the judgment criteria. A completely zero block is defined as a sub-block where the absolute value of all elements is less than or equal to 10. -6 A near-zero block is defined as a sub-block containing non-zero elements (with an absolute value greater than 10). -6 The proportion of ( ) is less than 5%.

[0033] S3022: Block-by-block traversal judgment. Traverse all the small blocks after the first-level division in storage order, check the element values ​​of each sub-block one by one, and mark them as complete zero blocks (Z0), near zero blocks (Z1) and dense blocks (D) according to the above criteria. Among them, dense blocks refer to sub-blocks that do not belong to either complete zero blocks or near zero blocks.

[0034] S303: Extraction and aggregation of non-zero elements: S3031: Extract non-zero elements. For all sub-blocks marked as near-zero blocks, traverse each element within the block and record the coordinate information and value of the non-zero elements. The coordinate information adopts the encoding method of "block index + intra-block offset". The block index represents the sequence number of the sub-block after the first-level division, and the intra-block offset represents the row and column position of the non-zero element in the sub-block.

[0035] S3032: Aggregate into a sparse matrix. Aggregate all extracted non-zero elements according to the COO (Coordinate Format) to form a highly sparse temporary matrix. The COO format contains three core arrays: a row index array that stores the row coordinates of the non-zero elements in the original matrix, a column index array that stores the column coordinates of the non-zero elements in the original matrix, and a value array that stores the specific values ​​of the corresponding non-zero elements, providing data input for sparse path calculation.

[0036] S3033: Complete Zero Block Marking and Flow. For all sub-blocks marked as complete zero blocks (Z0), no elements need to be extracted. The block index and position information are directly recorded. They participate in the partitioning together with dense blocks in the secondary partitioning (S304) and finally enter the dense path (S403) processing. In the dense path calculation, the zero matrix is ​​directly output without performing multiplication operations to minimize invalid calculations.

[0037] S304: Second-level partitioning (large block partitioning): S3041: Clearly define the partitioning constraints. The partitioning must meet three core constraints: First, load balancing constraint, ensuring that the computational difference among all large blocks does not exceed 10%; second, memory constraint, the storage size of a single large block does not exceed the capacity of a single CPU memory domain; and third, algorithm adaptation constraint, the dimension of the large block must match the algorithm dimension stored in the sparse FMM algorithm lookup table.

[0038] S3042: Computational complexity estimation. For each sub-block marked as a dense block, query the sparse FMM algorithm lookup table, match the corresponding FMM algorithm according to the sub-block dimension and sparsity level, and estimate the multiplication computation of each dense block based on the computational complexity model of the algorithm.

[0039] S3043: Dynamic programming partitioning, guided by a fast matrix multiplication lookup table, determines the optimal partitioning scheme through dynamic programming. The dynamic programming state is defined as "the minimum total number of multiplications required to divide the first n dense blocks into k large blocks". During the state transition process, the estimated computational cost of each dense block is combined to optimize the partitioning result to minimize the total number of multiplications between blocks. Finally, all dense blocks are divided into m large blocks (m≤ number of CPU physical cores), and each large block is assigned to a CPU core for independent processing, promoting inter-core parallelism.

[0040] The dual-path parallel execution strategy is the core computational component adapted to the characteristics of unstructured sparse matrices. By splitting the matrix multiplication task into sparse and dense paths, and processing them with targeted algorithms respectively, concurrent execution between paths is achieved, maximizing computational efficiency. The specific implementation process is as follows: S401: Path task splitting is based on the results of two-level partitioning and zero-block extraction. It clarifies the processing objects of the two paths. The sparse path is responsible for processing the highly sparse matrix after aggregation, which is composed of non-zero elements in near-zero blocks and has significant unstructured sparsity characteristics. The dense path is responsible for processing the large blocks after the second-level partitioning (including dense blocks and completely zero blocks). The non-zero elements of these blocks are relatively concentrated and are suitable for processing with the optimized fast matrix multiplication algorithm. The task partitioning boundaries of the two paths are clear, ensuring no duplicate calculations or task omissions.

[0041] S402: Sparse path calculation: S4021: Input data matching. Based on the column index information of the highly sparse matrix, extract the corresponding column vectors from the original input matrix to form an input sparse sub-block that matches the dimension of the highly sparse matrix. The dimension matching rule is that the number of rows of the input sparse sub-block is equal to the number of columns of the highly sparse matrix, and the number of columns of the input sparse sub-block is consistent with the number of columns of the original input matrix, ensuring the dimension validity of matrix multiplication operations.

[0042] S4022: Call the sparse matrix multiplication kernel and select the mature CPU sparse matrix multiplication (SpMM) kernel to perform the calculation. Those skilled in the art can choose the mkl_sparse_d_mm function of Intel MKL, the corresponding kernel in the LIBXSMM library, etc. The kernel call process is a well-known technology in the field and will not be described in detail here. During the calculation, the kernel automatically adapts to the unstructured sparse mode and efficiently processes irregularly distributed non-zero elements through optimized memory access strategies and computation scheduling.

[0043] S4023: Output sparse path results. After calculation, the matrix multiplication results corresponding to the sparse path are obtained. The dimension of the result matrix is ​​the number of rows of the highly sparse matrix multiplied by the number of columns of the input sparse sub-block. The results are stored in the CPU cache or local memory domain, waiting for subsequent fusion.

[0044] S403: Dense Path Calculation S4031: Matching FMM algorithms. For each large block after secondary partitioning, extract the dimensional information (number of rows, number of columns, and dimensions corresponding to the input matrix) and sparsity level of the large block. Query the sparse FMM algorithm lookup table to match the corresponding optimized FMM algorithm and parameters (block granularity, block multiplication order, etc.) to ensure that the algorithm is adapted to the characteristics of the large block.

[0045] S4032: Block-based computation execution. For each large block, block-based and multiplication operations are performed according to the matched FMM algorithm parameters. Completely zero blocks directly output the zero matrix of the corresponding dimension without performing multiplication operations. Dense blocks are decomposed into smaller sub-blocks according to the block-based rules of the FMM algorithm, and multiplication and addition operations are performed according to the preset block multiplication order. The computational complexity is reduced by using algorithm optimization.

[0046] S4033: Intermediate result aggregation. The calculation results (including the zero matrix) of each large block are aggregated according to the original second-level partitioning location information to form the complete calculation results of dense paths. The dimension of the result matrix is ​​consistent with the dimension of the sparse path result matrix to prepare for subsequent fusion. The aggregated results are stored in the CPU cache or local memory domain.

[0047] S404: Dual-path concurrent scheduling leverages the parallel processing capabilities of multi-core CPUs to achieve concurrent execution of sparse and dense paths. Task scheduling is performed through Intel's Threading Building Blocks (TBB), which encapsulates the computational tasks of sparse paths and the computational tasks of each block of dense paths into independent parallel tasks and allocates them to different CPU cores. During scheduling, the allocation of CPU core resources is dynamically adjusted according to the computational workload of the tasks to ensure that the execution progress of the two paths is roughly synchronized, avoiding efficiency losses caused by single-path waiting.

[0048] S405: Results Fusion: S4051: Location mapping establishment. Based on the block index information of the first-level partition, establish the location mapping relationship between each sub-block in the sparse path result and the dense path result in the final result matrix. The mapping rule is: the sub-block in the sparse path result corresponds to the region where the near-zero block is located in the original matrix, and the sub-block in the dense path result corresponds to the region where the dense block and the completely zero block are located in the original matrix. There is no positional overlap between the two.

[0049] S4052: Matrix concatenation. According to the positional mapping relationship, the result matrices of the two paths are concatenated into a complete final matrix multiplication result. The concatenation operation is implemented through the CPU's memory copy instruction. Vector transmission instructions in the AVX-512 instruction set are used first to improve data concatenation efficiency. The dimension of the concatenated result matrix is ​​consistent with the number of rows of the original weight matrix and the number of columns of the original input matrix, which meets the input requirements of the subsequent decoding process of LLM inference.

[0050] S4053: Result Validation. The validity of the computation is verified using a dual metric: computational efficiency and model accuracy. Computational efficiency is verified using the effective GFLOPs metric, which is calculated as: Effective GFLOPs = (2PQR) / (2πQR) PR) / time×10 -9Where P is the number of rows in the original weight matrix, Q is the number of columns in the original weight matrix, R is the number of columns in the original input matrix, and time is the total execution time of the dual-path parallel execution (in seconds). The verification results must ensure that they match the theoretical computational complexity. The model accuracy is verified by the perplexity index. The final matrix multiplication result is substituted into the LLM inference decoding process to calculate the perplexity of the model on the WikiText2 dataset. It must meet the corresponding sparsity and perplexity threshold requirements of the model in Tables 1 and 2 (e.g., the perplexity of the OPT-125M model with 50% unstructured sparseness is ≤33.17, and the perplexity of the LLaMA2-13B model with 70% OWL pruning is ≤17.90), ensuring that the numerical accuracy of the results meets the requirements of LLM inference.

[0051] The multi-core CPU adaptation and optimization scheme is designed around memory hierarchy and parallelism. It maximizes hardware utilization through four key optimization techniques, providing underlying support for dual-path parallel execution. The specific implementation process is as follows: S501: NUMA-aware thread binding optimization: S5011: Hardware topology detection. Calls the CPU hardware topology interface to obtain the number of CPU memory domains, the CPU core number corresponding to each memory domain, and the local memory capacity. Taking the Intel Xeon Platinum 8358P CPU as an example, the dual-socket server contains two memory domains, each corresponding to 32 physical cores (64 physical cores in total) and 64GB of local memory. The CPU core numbers in each memory domain are continuously allocated.

[0052] S5012: Thread-core binding configuration allocates worker threads in dual-path execution according to the rule of "one thread bound to one physical core". The binding process is implemented through the CPU's thread affinity interface to ensure that the thread is only bound to the core in the memory domain where it was created, and does not allocate the binding relationship between the thread and the core across memory domains.

[0053] S5013: Data localization storage. Large blocks of data after secondary partitioning are stored in the local memory of the corresponding memory domain according to the memory domain to which the allocated CPU core belongs. Intermediate results of sparse and dense paths are also preferentially stored in the local memory domain of the corresponding thread-bound core, reducing cross-node memory access during sparse data processing.

[0054] S502: Submatrix prefetch optimization: S5021: Prefetch timing is determined based on the block multiplication order of the FMM algorithm in the dense path. The preceding data requirements of each block multiplication step are analyzed. The prefetch operation is triggered before the current block multiplication reaches the data dependency node. Specifically, it is determined by the CPU instruction counter. When the number of executed instructions of the current block multiplication reaches 50% of the total number of instructions, the prefetch operation is started.

[0055] S5022: Prefetch parameter configuration, calls the CPU's hardware-directed prefetch instruction (such as the builtin_prefetch function in x86 architecture), sets the prefetch address to the memory starting address of the submatrix required for subsequent block multiplication, specifies the prefetch cache level as L2 cache, and sets the prefetch operation type to read operation.

[0056] S5023: Prefetch intensity control, dynamically adjust the prefetch method according to the size of the submatrix: if the storage size of the submatrix is ​​≤ 1 / 2 of the L2 cache capacity, the entire submatrix is ​​prefetched at once; if the storage size of the submatrix is ​​> 1 / 2 of the L2 cache capacity, it is prefetched in multiple times, with each prefetching data amount being 1 / 2 of the L2 cache capacity, until the required submatrix is ​​fully loaded into the cache.

[0057] S503: TBB Parallel Execution Optimization: S5031: Task encapsulation, which encapsulates the computational tasks in dual-path execution into independent TBB tasks. Each task contains core information such as input data address, output data address, and algorithm parameters. Among them, the priority of large-block computational tasks in dense paths is set higher than that of sparse path tasks, ensuring that dense paths with greater computational load get CPU resources first.

[0058] S5032: Dynamic scheduling configuration, enabling TBB's dynamic task scheduler. The scheduler monitors the task queue length of each CPU core in real time. When the task queue length of a certain CPU core is less than 50% of the task queue length of other CPU cores, the task stealing mechanism is triggered, and the CPU core obtains the task to be executed from the core with the longer task queue.

[0059] S5033: Task synchronization mechanism, which uses a counting semaphore of TBB to realize task synchronization. The initial value of the semaphore is set to the total number of tasks to be executed. After each task is completed, the semaphore count is decremented by 1. When the semaphore count is 0, it indicates that all parallel tasks have been completed, triggering the dual-path result fusion step.

[0060] S504: AVX-512 Vectorization Optimization: S5041: Vector alignment processing performs storage address alignment checks on sub-block data in the FMM algorithm to ensure that the memory starting address of the sub-block is an integer multiple of 64 bytes (adapting to the vector length of the AVX-512 instruction set). If the address is not aligned, the data storage location is adjusted by the CPU's memory copy instruction to achieve address alignment.

[0061] S5042: The fused multiply-add instruction is used in the block multiplication and intermediate aggregation process of the FMM algorithm. The fused multiply-add instruction mm512_fmadd_ps of AVX-512 is called. This instruction completes the "multiplication + addition" operation in one operation. Specifically, it implements c=a×b+c, where a, b, and c are all AVX-512 vector registers, and each vector register stores 8 single-precision floating-point numbers.

[0062] S5043: Dense sub-block acceleration processing. For dense sub-blocks in dense paths, the AVX-512 optimized dgemm function is used to perform multiplication operations. This function uses vector-aligned block processing to divide dense sub-blocks into several vector blocks according to the AVX-512 vector length, and improves arithmetic throughput by batch execution of vector instructions.

[0063] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A CPU dual-path acceleration method for large-scale LLM sparse matrix multiplication, characterized in that, Includes the following steps: S1. Construct a lookup table for the Fast Matrix Multiplication (FMM) algorithm: The scene parameters for sparse matrix multiplication are defined by enumerating scene parameters. Based on the scene parameters, a search space including the matrix partitioning method and the block multiplication order is defined. In the search space, the optimized FMM algorithm is generated iteratively using a random walk strategy. The FMM algorithm is then boosted using Hensel to compensate for numerical errors. The boosted FMM algorithm and its parameters are stored in the lookup table. S2. Perform two-level partitioning and zero-block identification of the matrix: Obtain the unstructured pruned LLM weight matrix and the input matrix to be processed, perform a first-level partitioning of the LLM weight matrix to form small block sub-matrices, and identify the small block matrix as a complete zero block, near-zero block, or dense block based on the element value distribution characteristics by traversing the element values ​​contained in the small block matrix. S3. Dual-path data preparation: Extract non-zero elements from the identified near-zero blocks and aggregate them into a highly sparse matrix; perform secondary partitioning on the identified dense blocks to form large blocks. S4. Dual-path parallel computation: Utilizes a multi-core CPU to concurrently execute sparse path computation and dense path computation; wherein, sparse path computation calls the sparse matrix multiplication kernel for the highly sparse matrix, and dense path computation calls the matching FMM algorithm to perform block multiplication operations on the lookup table constructed in the large block query step S1. S5. Result Fusion: Establish the positional mapping relationship between the sparse path calculation results and the dense path calculation results, and concatenate the result matrices of the two paths into the final matrix multiplication result.

2. The CPU dual-path acceleration method for large-scale LLM sparse matrix multiplication according to claim 1, characterized in that, The construction of the Fast Matrix Multiplication (FMM) lookup table in step S1 specifically includes: determining the scene parameters for enumeration, wherein the scene parameters cover the matrix multiplication dimension, sparsity level, and sparsity mode; The search starting point is initialized using traditional matrix multiplication as the initial solution, and the search dimension of the search space is set, wherein the granularity of the matrix partitioning method is aligned with the basic computing unit of the CPU. The search space is traversed by a random walk strategy. In each iteration, the block granularity and block multiplication order of the current solution are adjusted to generate candidate solutions. The current solution is updated with minimizing the number of block multiplications as the evaluation index until the preset iteration termination condition is met to obtain the FMM algorithm adapted to the scene parameters. The obtained FMM algorithm is subjected to Hensel boosting operation, and the numerical precision compensation mechanism is used to ensure that the perplexity of the model corresponding to the algorithm calculation result does not exceed the preset threshold. Construct a lookup table with the combination of the matrix multiplication dimension, sparsity level, and sparsity mode as the index key, and the FMM algorithm code snippet, block granularity parameter, and block multiplication order parameter as the storage value.

3. The CPU dual-path acceleration method for large-scale LLM sparse matrix multiplication according to claim 1, characterized in that, The specific steps of performing the two-level matrix partitioning and zero-block identification in step S2 include: First-level partitioning: The block size is set to an integer multiple of the CPU basic computing unit, and the block is uniformly divided along the row and column directions of the LLM weight matrix according to the block size to form several small sub-matrices after first-level partitioning; Zero block identification: Traverse the small block sub-matrix after the first-level division, and detect the absolute value of all elements in the small block sub-matrix. If the absolute value of all elements is less than or equal to the first threshold, it is marked as a completely zero block; if the proportion of non-zero elements with an absolute value greater than the first threshold is lower than the second threshold, it is marked as a near-zero block; if it is neither a completely zero block nor a near-zero block, it is marked as a dense block.

4. The CPU dual-path acceleration method for large-scale LLM sparse matrix multiplication according to claim 3, characterized in that, The dual-path data preparation in step S3 specifically includes: Sparse path data aggregation: Traverse the small block submatrix marked as near-zero blocks, extract the values ​​of non-zero elements within the block and their coordinate information based on the block index and the offset within the block, and aggregate them into a highly sparse matrix in COO format; Dense path dynamic programming partitioning: For the small block submatrix marked as dense blocks, the corresponding FMM algorithm and estimated computational cost are obtained by querying the lookup table based on the dimension and sparsity level of the dense blocks. With the goal of minimizing the total number of inter-block multiplications, the dynamic programming method is used to merge multiple dense blocks into several large blocks, and the storage size of each large block does not exceed the capacity of a single CPU memory domain.

5. The CPU dual-path acceleration method for large-scale LLM sparse matrix multiplication according to claim 4, characterized in that, The sparse path calculation in step S4 specifically includes: Based on the column index information of the highly sparse matrix, the corresponding column vectors are extracted from the input matrix to form an input sparse sub-block, and the number of rows of the input sparse sub-block is equal to the number of columns of the highly sparse matrix; The CPU sparse matrix multiplication (SpMM) kernel is invoked to perform multiplication operations on the highly sparse matrix and the input sparse sub-block to obtain the sparse path result matrix.

6. The CPU dual-path acceleration method for large-scale LLM sparse matrix multiplication according to claim 4, characterized in that, The dense path calculation in step S4 specifically includes: For each large block after the secondary partitioning, the dimensionality information and sparsity level of the large block are extracted, and the corresponding FMM algorithm parameters are matched in the lookup table. The parameters include the block granularity and the block multiplication order. The large block is broken down into sub-blocks according to the matching block granularity, and multiplication and addition operations are performed in the order of block multiplication; wherein, for completely zero blocks, the zero matrix is ​​directly output and no multiplication operation is performed. The calculation results of each large block are aggregated according to the original two-level partitioning location information to obtain a dense path result matrix.

7. The CPU dual-path acceleration method for large-scale LLM sparse matrix multiplication according to claim 1, characterized in that, The result fusion in step S5 specifically includes: Based on the block index information of the first-level partition, the sub-blocks in the sparse path calculation result are mapped to the region where the near-zero block is located in the original weight matrix, and the sub-blocks in the dense path calculation result are mapped to the region where the dense block and the completely zero block are located in the original weight matrix. Using vector transfer instructions in the AVX-512 instruction set, the result matrices of the two paths are concatenated according to the established position mapping relationship to generate a final matrix multiplication result with the same dimension as the number of rows in the original weight matrix and the number of columns in the original input matrix.

8. The CPU dual-path acceleration method for large-scale LLM sparse matrix multiplication according to claim 1, characterized in that, The execution of steps S4 and S5 also includes a multi-core CPU adaptation optimization strategy, which specifically includes: NUMA-aware thread binding: Detects CPU hardware topology, allocates dual-path execution worker threads according to the rule of binding one thread to one physical core, and restricts the worker thread to only the core in the memory domain where the worker thread was created, and stores the relevant computational data in the local memory of the corresponding memory domain; Submatrix prefetching: In dense path computation, according to the block multiplication order of the FMM algorithm, when the number of instructions executed in the current block multiplication reaches a preset proportion, the hardware-directed prefetching instruction is called to prefetch the subsequent required submatrix into the L2 cache. AVX-512 Vectorization: During the block multiplication and intermediate aggregation process of the FMM algorithm, the AVX-512 fusion multiply-accumulate instruction is called to perform the operation, and dense sub-blocks are processed using vector alignment.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the CPU dual-path acceleration method for large-scale LLM sparse matrix multiplication as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the CPU dual-path acceleration method for large-scale LLM sparse matrix multiplication as described in any one of claims 1 to 8.

Images