Large language model reasoning-oriented sparse reasoning method, system, equipment and product

Abstract

The invention discloses a sparse reasoning method, system, equipment and product for large language model reasoning, and relates to the technical field of artificial intelligence. The method comprises the following steps: firstly, acquiring a weight matrix of a large language model, and then carrying out pruning sparsification processing on the weight matrix to obtain a sparse matrix; the storage format of the sparse matrix is converted into a bitmap coding storage format which is suitable for tensor calculation core perception and adopts a multi-level block structure to respectively correspond to different calculation granularities in a GPU / NPU hardware architecture so as to obtain a sparse model, and then the sparse model is deployed to respond to a reasoning request to perform reasoning service; sparse matrix multiplication is completed through a sparse matrix multiplication kernel based on on-chip storage bitmap coding so as to perform reasoning, so that through an innovative sparse matrix storage format and calculation optimization, the storage efficiency and the calculation performance in the reasoning process are remarkably improved, and especially in a low-sparseness scene, the reasoning efficiency is greatly improved. And the performance blank of the existing sparse reasoning framework in the field is filled.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a sparse reasoning method, system, device, and product for reasoning with large language models. Background Technology

[0002] In recent years, large language models (LLMs) have achieved great success in the field of artificial intelligence. However, their massive scale (often with billions of parameters) has led to high memory requirements and huge computational overhead in the reasoning process.

[0003] Currently, model compression techniques (such as pruning and quantization) are the main directions for solving the above problems. Among them, weight pruning (i.e., sparsification) refers to reducing computation and memory requirements by removing unimportant connections in the weight matrix and introducing sparsity.

[0004] In the current sparse inference framework, there are several main methods: (1) Structured Pruning, which removes components of the entire neural network (such as channels or layers), but requires expensive post-training optimization; (2) Semi-Structured Pruning, such as N:M pruning (where N and M represent positive integers), achieves a balance between flexibility and efficiency, but only supports specific sparse patterns; (3) Unstructured Pruning, which removes arbitrary weights, has the highest flexibility and excellent accuracy, but is limited by the support of AI (Artificial Intelligence) acceleration hardware for sparse matrix operations.

[0005] While current unstructured sparse inference frameworks (such as Flash-LLM, SparTA, and Sputnik) perform well at high sparsity (>70%), they face the following drawbacks at low sparsity (30%-50%): (1) High indexing overhead, i.e., sparse matrix storage requires additional storage of index information for non-zero elements, which significantly increases memory overhead at low sparsity and may even negate the advantages brought by pruning; (2) Low computational efficiency, i.e., existing sparse matrix multiplication kernels cannot fully utilize the computational power of AI accelerators such as GPUs (Graphics Processing Units) and NPUs (Neural Processing Units) at low sparsity, especially the tensor computation core architecture; (3) Limited actual acceleration effect, i.e., Flash-LLM is difficult to surpass the performance of dense computation libraries (such as cuBLAS) at low sparsity (below 50%).

[0006] In summary, existing sparse inference techniques for large language model inference have the following drawbacks: (1) High index storage and access overhead, i.e., traditional sparse matrix storage formats (such as CSR and Tiled-CSL) have excessively high index overhead at low sparsity, and the compression ratio is insufficient to reflect the storage advantages of sparse matrices; (2) Low hardware utilization, i.e., existing sparse inference frameworks fail to fully utilize the computing power of tensor computing cores of AI accelerators such as GPUs and NPUs, resulting in the actual performance of sparse computing not meeting theoretical expectations; (3) Inadequate pipeline design, i.e., data transmission and computation cannot be fully overlapped in existing sparse inference frameworks, resulting in insufficient utilization of GPU / NPU resources. Summary of the Invention

[0007] The purpose of this invention is to provide a sparse reasoning method, system, computer device, computer-readable storage product, and computer program product for large language model reasoning, in order to solve the problems of high index storage and access overhead and low hardware utilization in existing sparse reasoning schemes for large language model reasoning.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a sparse reasoning method for reasoning with large language models is provided, including: Collect and obtain the weight matrix of the large language model; The weight matrix is ​​pruned and sparsified to obtain a sparse matrix; The storage format of the sparse matrix is ​​converted into a bitmap encoding storage format suitable for tensor computing core perception to obtain a sparse model. The bitmap encoding storage format suitable for tensor computing core perception adopts a multi-level block structure to correspond to different computing granularities in the GPU / NPU hardware architecture. Deploy the sparse model to provide inference services in response to inference requests; In the inference service, sparse matrix multiplication is performed for inference by a sparse matrix multiplication kernel based on on-chip storage bitmap encoding.

[0009] Based on the above-mentioned invention, a high-performance sparse inference scheme for large-scale language model inference is provided. First, the weight matrix of the large language model is acquired. Then, the weight matrix is ​​pruned and sparsified to obtain a sparse matrix. The storage format of the sparse matrix is ​​converted into a bitmap encoding storage format suitable for tensor computing core perception, employing a multi-level block structure to correspond to different computational granularities in GPU / NPU hardware architectures, thus obtaining a sparse model. Next, the sparse model is deployed to respond to inference requests and provide inference services. Finally, in the inference service, sparse matrix multiplication is performed using a sparse matrix multiplication kernel based on on-chip storage bitmap encoding for inference. This innovative sparse matrix storage format and computational optimization significantly improve storage efficiency and computational performance during inference, especially in low-sparseness (30%-70%) scenarios, filling the performance gap in existing sparse inference frameworks in this area, facilitating practical application and promotion.

[0010] In one possible design, the multi-level block structure includes bitmap-level block units, tensor computation core-level block units, and group-level block units; The bitmap-level block unit is used to correspond to the basic computation block of the tensor computation core and has a size of 8×8, so that the sparse state of the 8×8 block in the sparse matrix can be represented by a 64-bit bitmap. Each bit in the 64 bits represents whether a matrix element is non-zero, and the value "1" indicates that the matrix element is non-zero, and the value "0" indicates that the matrix element is zero. The tensor computation core-level block unit is used to correspond to the hardware computation instructions of the tensor computation core and is composed of 2×2 bitmap-level block units with a size of 16×16. The non-zero value storage order of each tensor computation core-level block unit is matched with the register distribution of the GPU / NPU in order to optimize the data loading path of the tensor computation core. The group-level block unit is used for the allocation of thread blocks to ensure that each thread block can process the sub-blocks of the sparse matrix. The group-level block unit is composed of multiple tensor computation core-level block units.

[0011] In one possible design, the bitmap-encoded storage format suitable for tensor computation core awareness uses the following three data forms to store the information of the sparse matrix: a group offset index table for recording the starting address of each group-level block unit, a bitmap mask segment for storing the bitmap information of all the bitmap-level block units, and a data segment for storing the non-zero values ​​of the sparse matrix. The bitmap mask segment is represented using 64-bit integers, each bitmap-level block unit occupies 8 bytes, and the non-zero values ​​in the data segment are arranged in a nested order of the group-level block units, the tensor computation core-level block units, and the bitmap-level block units, and are stored using floating-point numbers or integers.

[0012] In one possible design, sparse matrix multiplication is performed using a sparse matrix multiplication kernel based on on-chip storage bitmap encoding, including: The group-level block unit is loaded from global memory to on-chip storage using data loading instructions; The sparse matrix bitmap information in the bitmap-level block unit is decoded in the on-chip memory. Then, the index of the non-zero value in the bitmap-level block unit is dynamically calculated based on the decoding result, and the indexed non-zero value is loaded into the register. The sub-blocks of the dense matrix corresponding to the group-level block unit are loaded into the on-chip storage, and the loading results are allocated to the registers by instructions in order to prepare for computation using the tensor computation core; During the calculation process using the tensor calculation core, calculation instructions are used to complete the multiplication operation between the group-level block unit and the sub-block of the dense matrix; The result of the multiplication operation is stored back in the global memory.

[0013] In one possible design, the index of the non-zero value in the bitmap-level block unit is dynamically calculated based on the decoding result, and the indexed non-zero value is loaded into a register, including: Use the bit counting instructions of the GPU / NPU to calculate the number of non-zero values ​​in the current bitmap based on the decoding result; The non-zero index that each thread needs to load is dynamically calculated using mask bit counting technology; Each thread extracts the corresponding non-zero value based on the binary representation of the bitmap in the decoding result and loads it into the register of the corresponding tensor calculation core.

[0014] In one possible design, the sparse matrix multiplication kernel employs a double-buffering mechanism and a fine-grained asynchronous pipeline. The double-buffering mechanism means that the loading and computation of the current group-level block unit are performed in parallel with the loading of the next group-level block unit. The fine-grained asynchronous pipeline means that the bitmap decoding step is executed in parallel with the step of computation using the tensor computation core.

[0015] Secondly, a sparse reasoning system for large language model reasoning is provided, including a weight matrix acquisition module, a pruning sparse processing module, a matrix format conversion module, a reasoning request response module, and a matrix multiplication completion module that are connected in sequence. The weight matrix acquisition module is used to acquire the weight matrix of the large language model; The pruning and sparsification processing module is used to prune and sparsify the weight matrix to obtain a sparse matrix. The matrix format conversion module is used to convert the storage format of the sparse matrix into a bitmap encoding storage format suitable for tensor computing core perception to obtain a sparse model. The bitmap encoding storage format suitable for tensor computing core perception adopts a multi-level block structure to correspond to different computing granularities in the GPU / NPU hardware architecture. The inference request response module is used to deploy the sparse model to respond to inference requests and provide inference services. The matrix multiplication completion module is used in the inference service to perform sparse matrix multiplication for inference by using a sparse matrix multiplication kernel based on on-chip storage bitmap encoding.

[0016] Thirdly, the present invention provides a computer device comprising a memory, a processor, and a transceiver sequentially and communicatively connected, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the sparse inference method as described in the first aspect or any possible design in the first aspect.

[0017] Fourthly, the present invention provides a computer-readable storage product storing instructions that, when executed on a computer, perform the sparse inference method as described in the first aspect or any possible design in the first aspect.

[0018] Fifthly, the present invention provides a computer program product, including a computer program or instructions, which, when executed by a computer, implement the sparse inference method as described in the first aspect or any possible design in the first aspect.

[0019] The beneficial effects of the above scheme are: (1) This invention creatively provides a high-performance sparse inference scheme for large language model inference. First, the weight matrix of the large language model is collected. Then, the weight matrix is ​​pruned and sparsified to obtain a sparse matrix. The storage format of the sparse matrix is ​​converted into a bitmap encoding storage format suitable for tensor computing core perception and adopting a multi-level block structure to correspond to different computing granularities in GPU / NPU hardware architecture to obtain a sparse model. Then, the sparse model is deployed to respond to inference requests and perform inference services. Finally, in the inference service, sparse matrix multiplication is performed by a sparse matrix multiplication kernel based on on-chip storage bitmap encoding to perform inference. In this way, through innovative sparse matrix storage format and computation optimization method, the storage efficiency and computation performance in the inference process are significantly improved. Especially in low sparsity (30%-70%) scenarios, it fills the performance gap of existing sparse inference frameworks in this field, which is convenient for practical application and promotion. (2) Improves storage efficiency: The bitmap encoding storage format, which is suitable for the core perception of tensor computation, significantly reduces the index storage overhead of sparse matrices. In the sparsity range of 30%-70%, the compression ratio is always higher than that of existing formats (such as CSR), which can significantly reduce the memory requirements in the LLM inference process of large language models. (3) Improved computational performance: The performance of the sparse matrix multiplication kernel based on on-chip storage bitmap encoding is significantly better than existing methods (such as cuSPARSE and Flash-LLM), and it can significantly accelerate end-to-end LLM inference (1.68 times speedup) at 50% sparsity. (4) Improved hardware utilization: Through on-chip storage bitmap encoding and asynchronous pipeline design, the tensor computing core of the AI ​​accelerator is fully utilized, thereby improving the utilization of hardware resources; (5) Wide applicability: It supports sparse matrix multiplication with low sparsity, which solves the problem of insufficient performance of existing methods under low sparsity. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the sparse reasoning method for reasoning with large language models provided in an embodiment of this application.

[0022] Figure 2 This is a schematic diagram of a multi-level block structure of a bitmap-encoded storage format suitable for core perception in tensor computation, provided in an embodiment of this application.

[0023] Figure 3 This is a schematic diagram of the structure of a sparse reasoning system for reasoning with large language models, provided in an embodiment of this application.

[0024] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these embodiments without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0026] It should be understood that although the terms "first" and "second", etc., may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, the first object may be referred to as the second object, and similarly, the second object may be referred to as the first object, without departing from the scope of the exemplary embodiments of the invention.

[0027] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, or A and B exist simultaneously. Another example is A, B and / or C, which can mean that any one of A, B, and C or any combination thereof exists. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone or A and B exist simultaneously. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0028] Example like Figures 1-2 As shown, the sparse inference method for large language model inference provided in the first aspect of this embodiment can be executed, but is not limited to, by computer devices with certain computing resources, such as cloud servers and personal computers (PCs, which are multi-purpose computers of a size, price, and performance suitable for personal use; desktops, laptops, mini-laptops, tablets, and ultrabooks are all considered personal computers). Figure 1As shown, the sparse inference method may include, but is not limited to, the following steps S1 to S5.

[0029] S1. Collect and obtain the weight matrix of the large language model.

[0030] In step S1, the specific method for obtaining the weight matrix of the large language model is the conventional technique used in existing sparse inference schemes, which will not be elaborated here.

[0031] S2. The weight matrix is ​​pruned and sparsified to obtain a sparse matrix.

[0032] In step S2, the pruning sparsification process specifically refers to: employing an unstructured pruning strategy to sparsify the weight matrix, thereby obtaining a sparse matrix with unstructured sparse distribution characteristics (the innovation of this invention lies in the efficient storage and computation of such sparse matrices; its design is decoupled from the specific pruning algorithm at the upper level, and it is applicable as long as the algorithm can generate an unstructured sparse matrix that meets the conditions); the specific implementation may include, but is not limited to, the following configurable steps: (1) Pruning algorithm selection: This embodiment supports and is compatible with a variety of existing unstructured pruning algorithms, such as but not limited to SparseGPT, Wanda (Weight and Activation) or Magnitude Pruning based on weight size; during implementation, one can be selected for pruning according to different requirements for model accuracy and compression rate. (2) Core parameter settings, namely, to balance inference efficiency and model accuracy, it is recommended to set the target sparsity of pruning between 30% and 70% (this parameter range has been experimentally verified to significantly reduce the amount of computation while maintaining the core performance of the large language model, and the sparsity can be achieved through the algorithm's built-in global threshold or layer-by-layer adaptive strategy). (3) Generation and definition of sparse matrix, that is, the weight matrix after processing by the selected pruning algorithm. Its characteristic is that the zero value of the weight is distributed in an unstructured random manner (this matrix is ​​the sparse matrix defined in the subsequent steps of this embodiment, which is the basis for efficient encoding, storage and calculation). (4) The intermediate representation of the sparse matrix is ​​the connection between pruning and subsequent special encoding. The sparse matrix obtained by pruning can be temporarily represented in a conventional compressed sparse format (such as COO format) so that it can be converted into the bitmap encoding or other efficient compression format used in this embodiment.

[0033] S3. Convert the storage format of the sparse matrix into a bitmap encoding storage format suitable for tensor computing core perception to obtain a sparse model, wherein the bitmap encoding storage format suitable for tensor computing core perception adopts a multi-level block structure to correspond to different computing granularities in the GPU / NPU hardware architecture.

[0034] In step S3, the bitmap-encoded storage format suitable for tensor computation core perception is a sparse matrix storage format optimized for the tensor computation core architecture of AI accelerators such as GPUs / NPUs in this embodiment. It uses bitmap encoding to represent the positions of non-zero elements in the sparse matrix and achieves efficient storage and decoding of the sparse matrix at different computational granularities through a multi-level block structure design (i.e., including bitmap-level block units, tensor computation core-level block units, and group-level block units), thereby reducing index storage overhead. The sparse model is a new large language model conventionally obtained based on the new sparse matrix storage format (i.e., the bitmap-encoded storage format) (because the sparse matrix is ​​the sparsified result of the weight matrix of the large language model). Specifically, as... Figure 2 As shown, the multi-level block structure includes, but is not limited to, bitmap-level block units, tensor computation core-level block units, and group-level block units.

[0035] The bitmap-level block unit corresponds to the basic computation block of the tensor computation core and has a size of 8×8 ( Figure 2 The 4×4 unit size shown is merely an example for demonstration purposes; an 8×8 unit is used as the standard. This allows for the use of a 64-bit bitmap to represent the sparse state of an 8×8 block in the sparse matrix. Each bit in the 64 bits represents whether a matrix element is non-zero, with "1" indicating a non-zero value and "0" indicating a zero value. The bitmap-level block unit is the smallest computational unit in the multi-level block structure. For example, if the binary representation of the bitmap-level block unit is 10100011, it indicates that the 1st, 3rd, 7th, and 8th positions in the matrix are non-zero elements.

[0036] The tensor computation core-level block unit is used to correspond to the hardware computation instructions of the tensor computation core and is composed of 2×2 bitmap-level block units with a size of 16×16. The non-zero value storage order of each tensor computation core-level block unit is matched with the register distribution of the GPU / NPU in order to optimize the data loading path of the tensor computation core.

[0037] The group-level block unit is used for thread block allocation to ensure that each thread block can process sub-blocks of the sparse matrix. The group-level block unit consists of multiple tensor computation core-level block units (e.g., Figure 2 The sparse matrix is ​​composed of two computational core-level block units (1 and 2). The size of each block unit can be a typical 64×64 or 128x128 to adapt to the on-chip storage allocation strategy of the GPU / NPU. Through the aforementioned allocation of thread blocks, it can be ensured that each thread block can efficiently process the sub-blocks of the sparse matrix.

[0038] In step S3, specifically, the bitmap encoding storage format suitable for tensor computation core awareness uses the following three data formats to store the information of the sparse matrix: a group offset index table for recording the starting address of each group-level block unit, a bitmap mask segment for storing the bitmap information of all the bitmap-level block units, and a data segment for storing the non-zero values ​​of the sparse matrix. The bitmap mask segment is represented using 64-bit integers, each bitmap-level block unit occupies 8 bytes, and the non-zero values ​​in the data segment are arranged in a nested order of the group-level block units, the tensor computation core-level block units, and the bitmap-level block units, and are stored using floating-point numbers or integers. The group offset index table is used to facilitate quick location of the group-level block units; the precision of the floating-point numbers or integers can be, but is not limited to, FP32, FP16, BF16, FP8, Int8, and Int4, etc.

[0039] S4. Deploy the sparse model to provide inference services in response to inference requests.

[0040] In step S4, the inference request comes from the user, and the sparse model can be deployed in a conventional manner (specifically, the parameters of the transformed sparse model are loaded into GPU memory or NPU memory) in response to the inference request.

[0041] S5. In the inference service, sparse matrix multiplication is performed by a sparse matrix multiplication kernel based on on-chip storage bitmap encoding for inference.

[0042] In step S5, the result of the sparse matrix multiplication is used as the input to the next layer of the large language model, thus enabling seamless integration with existing inference frameworks (such as FasterTransformer and DeepSpeed), and making it suitable for large language model inference tasks. The sparse matrix multiplication kernel based on on-chip storage bitmap encoding is used to achieve fast decompression and computation of sparse matrices in high-speed on-chip storage. Specifically, the sparse matrix multiplication is performed using the sparse matrix multiplication kernel based on on-chip storage bitmap encoding, including but not limited to the following steps S51 to S55.

[0043] S51. Load the group-level block unit from global memory to on-chip storage using a data load instruction.

[0044] In step S51, register transfer can be avoided, thereby improving memory bandwidth utilization.

[0045] S52. Decode the sparse matrix bitmap information in the bitmap-level block unit in the on-chip memory, then dynamically calculate the index of the non-zero value in the bitmap-level block unit based on the decoding result, and load the indexed non-zero value into the register.

[0046] In step S52, the above step S52 is the step of implementing on-chip storage bitmap encoding; in detail, the index of the non-zero value in the bitmap-level block unit is dynamically calculated based on the decoding result, and the indexed non-zero value is loaded into the register, including but not limited to the following steps S521 to S523.

[0047] S521. Use the bit counting instruction of the GPU / NPU to calculate the number of non-zero values ​​in the current bitmap based on the decoding result.

[0048] In step S521, the bit counting instruction is an existing instruction.

[0049] S522. Dynamically calculate the non-zero index that each thread needs to load using mask bit counting technology.

[0050] In step S522, the mask bit counting technique is existing technology. By using step S522, storing and displaying an index table can be avoided.

[0051] S523. Each thread extracts the corresponding non-zero value based on the binary representation of the bitmap in the decoding result and loads it into the register of the corresponding tensor calculation core.

[0052] S53. Load the sub-blocks of the dense matrix corresponding to the group-level block unit into the on-chip storage, and allocate the loading results into the registers via instructions, so as to prepare for computation using the tensor computation core.

[0053] In step S53, due to matrix multiplication in the inference service... ,in, This represents the weight matrix of the large language model (i.e., the matrix that has been pruned and sparsified). The input tensor (Activation) is dense; therefore, the dense matrix is ​​not the weight matrix of step S1, but rather the real-time input data (or activation value matrix) during inference. This allows for the calculation based on the matrix multiplication. middle and The multiplication correspondence can be used to determine the sub-block corresponding to the group-level block unit in the dense matrix, and the sub-block can be loaded into the on-chip memory.

[0054] S54. During the calculation process using the tensor calculation core, calculation instructions are used to complete the multiplication operation between the group-level block unit and the sub-block of the dense matrix.

[0055] S55. Store the result of the multiplication operation back into the global memory.

[0056] In step S5, to improve the hardware utilization of the kernel, the sparse matrix multiplication kernel preferably employs a double-buffering mechanism and fine-grained asynchronous pipeline. The double-buffering mechanism means that the loading and computation of the current group-level block unit are performed in parallel with the loading of the next group-level block unit. The fine-grained asynchronous pipeline means that the bitmap decoding step and the computation using the tensor computation core are executed in parallel. Through the aforementioned double-buffering mechanism, memory access latency can be hidden. The aforementioned bitmap decoding step is step S52, and the aforementioned step of using the tensor computing core for computation is step S54. Through the fine-grained asynchronous pipeline, the overlap between GPU / NPU computation and data transmission can be maximized. That is, by separating and parallelizing the loading, decoding, and computation tasks of the sparse weight matrix (i.e., the sparse matrix obtained in step S2) and the dense input matrix (i.e., the dense matrix in step S53), the complete overlap between data transmission and computation can be achieved, improving hardware utilization. Furthermore, through the optimized data transmission path and asynchronous pipeline mechanism, the computational power of the registers of GPU / NPU and other devices can be fully utilized.

[0057] This embodiment also uses the sparse reasoning method for large language model reasoning described in steps S1 to S5 above. Through experiments, it was found that it has the following performance improvement and advantages: (1) In terms of storage efficiency: the bitmap encoding storage format applicable to tensor computation core perception still maintains a high compression ratio under low sparsity, which is more than 1.5 times higher than the CSR format; (2) In terms of computation performance: the sparse matrix multiplication kernel based on on-chip storage bitmap encoding achieves a performance acceleration of 1.68× under 50% sparsity, which is better than cuBLAS in 94% of the test scenarios; (3) In terms of hardware utilization: through on-chip storage bitmap encoding and asynchronous pipeline, the utilization of tensor computation core is improved by more than 20%.

[0058] Therefore, based on the sparse inference method described in steps S1 to S5 above, a high-performance sparse inference scheme for large language model inference is provided. First, the weight matrix of the large language model is acquired. Then, the weight matrix is ​​pruned and sparsified to obtain a sparse matrix. The storage format of the sparse matrix is ​​converted into a bitmap encoding storage format suitable for tensor computation core perception, employing a multi-level block structure to correspond to different computational granularities in GPU / NPU hardware architectures, thus obtaining a sparse model. Next, the sparse model is deployed to respond to inference requests and provide inference services. Finally, in the inference service, sparse matrix multiplication is performed using a sparse matrix multiplication kernel based on on-chip storage bitmap encoding for inference. This innovative sparse matrix storage format and computational optimization significantly improve storage efficiency and computational performance during inference, especially in low sparsity (30%-70%) scenarios, filling the performance gap in existing sparse inference frameworks in this area, facilitating practical application and promotion.

[0059] like Figure 3 As shown, the second aspect of this embodiment provides a virtual system for implementing the sparse inference method described in the first aspect, including a weight matrix acquisition module, a pruning sparse processing module, a matrix format conversion module, an inference request response module, and a matrix multiplication completion module that are sequentially connected in communication. The weight matrix acquisition module is used to acquire the weight matrix of the large language model; The pruning and sparsification processing module is used to prune and sparsify the weight matrix to obtain a sparse matrix. The matrix format conversion module is used to convert the storage format of the sparse matrix into a bitmap encoding storage format suitable for tensor computing core perception to obtain a sparse model. The bitmap encoding storage format suitable for tensor computing core perception adopts a multi-level block structure to correspond to different computing granularities in the GPU / NPU hardware architecture. The inference request response module is used to deploy the sparse model to respond to inference requests and provide inference services. The matrix multiplication completion module is used in the inference service to perform sparse matrix multiplication for inference by using a sparse matrix multiplication kernel based on on-chip storage bitmap encoding.

[0060] The working process, working details and technical effects of the aforementioned system provided in the second aspect of this embodiment can be found in the sparse inference method described in the first aspect, and will not be repeated here.

[0061] like Figure 4As shown, the third aspect of this embodiment provides a computer device for executing the sparse inference method as described in the first aspect, including a memory, a processor, and a transceiver connected in sequence. The memory stores a computer program, the transceiver sends and receives messages, and the processor reads the computer program and executes the sparse inference method as described in the first aspect. Specifically, the memory may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; the processor may include, but is not limited to, a microprocessor of the STM32F105 series. Furthermore, the computer device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0062] The working process, working details and technical effects of the aforementioned computer device provided in the third aspect of this embodiment can be found in the sparse inference method described in the first aspect, and will not be repeated here.

[0063] This fourth aspect of the embodiment provides a computer-readable storage product that stores instructions comprising the sparse inference method as described in the first aspect. Specifically, the computer-readable storage product stores instructions that, when executed on a computer, perform the sparse inference method as described in the first aspect. The computer-readable storage product refers to a data storage medium, which may include, but is not limited to, computer-readable storage media such as floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0064] The working process, working details and technical effects of the aforementioned computer-readable storage product provided in the fourth aspect of this embodiment can be found in the sparse inference method described in the first aspect, and will not be repeated here.

[0065] This fifth aspect of the embodiment provides a computer program product, including a computer program or instructions, which, when executed by a computer, implement the sparse inference method as described in the first aspect. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0066] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A sparse reasoning method for reasoning in large language models, characterized in that, include: Collect and obtain the weight matrix of the large language model; The weight matrix is ​​pruned and sparsified to obtain a sparse matrix; The storage format of the sparse matrix is ​​converted into a bitmap encoding storage format suitable for tensor computing core perception to obtain a sparse model. The bitmap encoding storage format suitable for tensor computing core perception adopts a multi-level block structure to correspond to different computing granularities in the GPU / NPU hardware architecture. Deploy the sparse model to provide inference services in response to inference requests; In the inference service, sparse matrix multiplication is performed for inference by a sparse matrix multiplication kernel based on on-chip storage bitmap encoding.

2. The sparse inference method according to claim 1, characterized in that, The multi-level block structure includes bitmap-level block units, tensor computation core-level block units, and group-level block units; The bitmap-level block unit is used to correspond to the basic computation block of the tensor computation core and has a size of 8×8, so that the sparse state of the 8×8 block in the sparse matrix can be represented by a 64-bit bitmap. Each bit in the 64 bits represents whether a matrix element is non-zero, and the value "1" indicates that the matrix element is non-zero, and the value "0" indicates that the matrix element is zero. The tensor computation core-level block unit is used to correspond to the hardware computation instructions of the tensor computation core and is composed of 2×2 bitmap-level block units with a size of 16×16. The non-zero value storage order of each tensor computation core-level block unit is matched with the register distribution of the GPU / NPU in order to optimize the data loading path of the tensor computation core. The group-level block unit is used for the allocation of thread blocks to ensure that each thread block can process the sub-blocks of the sparse matrix. The group-level block unit is composed of multiple tensor computation core-level block units.

3. The sparse inference method according to claim 2, characterized in that, The bitmap encoding storage format applicable to tensor computation core perception uses the following three data forms to store the information of the sparse matrix: a group offset index table for recording the starting address of each group-level block unit, a bitmap mask segment for storing the bitmap information of all the bitmap-level block units, and a data segment for storing the non-zero values ​​of the sparse matrix. The bitmap mask segment is represented by a 64-bit integer, each bitmap-level block unit occupies 8 bytes, and the non-zero values ​​in the data segment are arranged in the nested order of the group-level block units, the tensor computation core-level block units, and the bitmap-level block units, and are stored using floating-point numbers or integers.

4. The sparse inference method according to claim 3, characterized in that, Sparse matrix multiplication is performed using a sparse matrix multiplication kernel based on on-chip storage bitmap encoding, including: The group-level block unit is loaded from global memory to on-chip storage using data loading instructions; The sparse matrix bitmap information in the bitmap-level block unit is decoded in the on-chip memory. Then, the index of the non-zero value in the bitmap-level block unit is dynamically calculated based on the decoding result, and the indexed non-zero value is loaded into the register. The sub-blocks of the dense matrix corresponding to the group-level block unit are loaded into the on-chip storage, and the loading results are allocated to the registers by instructions in order to prepare for computation using the tensor computation core; During the calculation process using the tensor calculation core, calculation instructions are used to complete the multiplication operation between the group-level block unit and the sub-block of the dense matrix; The result of the multiplication operation is stored back in the global memory.

5. The sparse inference method according to claim 4, characterized in that, Based on the decoding result, the index of the non-zero value in the bitmap-level block unit is dynamically calculated, and the indexed non-zero value is loaded into the register, including: Use the bit counting instructions of the GPU / NPU to calculate the number of non-zero values ​​in the current bitmap based on the decoding result; The non-zero index that each thread needs to load is dynamically calculated using mask bit counting technology; Each thread extracts the corresponding non-zero value based on the binary representation of the bitmap in the decoding result and loads it into the register of the corresponding tensor calculation core.

6. The sparse inference method according to claim 2, characterized in that, The sparse matrix multiplication kernel employs a double-buffering mechanism and a fine-grained asynchronous pipeline. The double-buffering mechanism means that the loading and computation of the current group-level block unit are performed in parallel with the loading of the next group-level block unit. The fine-grained asynchronous pipeline means that the bitmap decoding step and the computation using the tensor computation kernel are performed in parallel.

7. A sparse reasoning system for reasoning on large language models, characterized in that, It includes a weight matrix acquisition module, a pruning and sparsity processing module, a matrix format conversion module, an inference request and response module, and a matrix multiplication completion module, which are connected in sequence. The weight matrix acquisition module is used to acquire the weight matrix of the large language model; The pruning and sparsification processing module is used to prune and sparsify the weight matrix to obtain a sparse matrix. The matrix format conversion module is used to convert the storage format of the sparse matrix into a bitmap encoding storage format suitable for tensor computing core perception to obtain a sparse model. The bitmap encoding storage format suitable for tensor computing core perception adopts a multi-level block structure to correspond to different computing granularities in the GPU / NPU hardware architecture. The inference request response module is used to deploy the sparse model to respond to inference requests and provide inference services. The matrix multiplication completion module is used in the inference service to perform sparse matrix multiplication for inference by using a sparse matrix multiplication kernel based on on-chip storage bitmap encoding.

8. A computer device, characterized in that, The device includes a memory, a processor, and a transceiver that are sequentially and communicatively connected, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the sparse inference method as described in any one of claims 1 to 6.

9. A computer-readable storage product, characterized in that... The computer-readable storage product stores instructions that, when executed on a computer, perform the sparse inference method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or the instructions are executed by the computer, they implement the sparse inference method as described in any one of claims 1 to 6.

Images