Storage method and apparatus for weight matrix
By merging high and low bit weight matrices and generating a mask identifier list, the problem of excessive GPU memory usage in large language models is solved, achieving efficient utilization of GPU memory and flexible deployment of the model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- MOXIN ARTIFICIAL INTELLIGENCE TECH (SHENZHEN) CO LTD
- Filing Date
- 2026-03-13
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies fail to effectively utilize the storage characteristics of high and low bit weight matrices, resulting in excessive GPU memory usage, which becomes a bottleneck for deploying large language models on hardware with conventional GPU memory configurations.
By merging high-bit and low-bit weight matrices and utilizing the complementarity of their effective data elements to generate a mask identifier list, the merged weight matrix and the mask identifier list are stored, reducing GPU memory usage.
It significantly reduces GPU memory usage, improves the deployment flexibility of large language models, and ensures the accuracy and efficiency of data access.
Smart Images
Figure CN121834110B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of large language model technology, and more specifically, to methods and apparatus for storing weight matrices. Background Technology
[0002] In artificial intelligence fields such as large language models and computer vision, the number of model parameters continues to grow, ranging from billions to trillions, leading to an exponential increase in the demand for video memory capacity in computing hardware, especially accelerator cards such as graphics processing units (GPUs). To achieve efficient deployment and inference of large language models with limited computing resources, model compression and acceleration techniques have become crucial. Among these, the high-low bit sparse scheme under the generalized sparse strategy has become one of the core optimization methods for large language model inference due to its ability to significantly reduce computational complexity. However, the computational advantage of the high-low bit sparse scheme under the generalized sparse strategy is limited in practice by the storage efficiency of the high-low bit weight matrix. Existing storage technologies have not fully considered the storage characteristics of the high-low bit weight matrix after generalized sparse processing, resulting in a still large video memory footprint, which has become a major bottleneck for deploying large language models on hardware with conventional video memory configurations.
[0003] Therefore, a new method and apparatus for storing weight matrices is needed. Summary of the Invention
[0004] The embodiments of this application provide a method and apparatus for storing weight matrices, which can significantly reduce GPU memory usage while ensuring data access accuracy and efficiency, and improve the deployment flexibility of large language models.
[0005] According to a first aspect of this application, a method for storing a weight matrix is provided, wherein the method includes: obtaining a high-bit weight matrix and a low-bit weight matrix after generalized sparse processing from a large language model, wherein the positions of elements in the high-bit weight matrix and the low-bit weight matrix correspond one-to-one, and the positions of valid data elements in the high-bit weight matrix are complementary to the positions of valid data elements in the low-bit weight matrix; generating a mask identifier list based on the positions of valid data elements in the high-bit weight matrix; merging the high-bit weight matrix and the low-bit weight matrix into a merged weight matrix based on the complementarity; and storing the merged weight matrix and the mask identifier list.
[0006] According to an embodiment of the first aspect of this application, merging the high-bit weight matrix and the low-bit weight matrix into a merged weight matrix based on the complementarity comprises: storing valid data elements in the high-bit weight matrix at corresponding positions in the merged weight matrix based on the positions of valid data elements in the high-bit weight matrix, and storing valid data elements in the low-bit weight matrix at corresponding positions in the merged weight matrix based on the positions of valid data elements in the low-bit weight matrix.
[0007] According to an embodiment of the first aspect of this application, generating a mask identifier list based on the positions of valid data elements in the high bit weight matrix includes: dividing the high bit weight matrix into multiple data blocks according to a preset size and in sequence; for each data block, generating a mask identifier sublist for the data block based on the positions of valid data elements in the data block; and concatenating the mask identifier sublists of the multiple data blocks in sequence to form the mask identifier list.
[0008] According to an embodiment of the first aspect of this application, generating a sublist of mask identifiers for the data block based on the positions of valid data elements in the data block includes: using the position identifier of each valid data element in the data block as a corresponding element in the sublist of mask identifiers.
[0009] According to an embodiment of the first aspect of this application, the elements in the mask identifier sublist are of type INT8.
[0010] According to an embodiment of the first aspect of this application, the preset size is based on the hardware storage access granularity of the computing unit, and the number of valid data elements in the data block is based on the preset size and a sparsity factor.
[0011] According to an embodiment of the first aspect of this application, the preset size is 64 elements and the sparsity factor is 8.
[0012] According to an embodiment of a first aspect of this application, the method further includes: reading the high-bit weight matrix and the low-bit weight matrix from the merged weight matrix according to the mask identifier list in response to a read request.
[0013] According to an embodiment of a first aspect of this application, reading the high-bit weight matrix and the low-bit weight matrix from the merged weight matrix in response to a read request, based on the mask identifier list, comprises: obtaining one or more mask identifier sublists based on the mask identifier list; reading a plurality of data blocks from the merged weight matrix according to the one or more mask identifier sublists in a preset size and in sequence, wherein an element in each mask identifier sublist corresponds to a position identifier of a valid data element in the corresponding data block; and concatenating the plurality of data blocks in sequence to form the high-bit weight matrix.
[0014] According to an embodiment of a first aspect of this application, reading the high-bit weight matrix and the low-bit weight matrix from the merged weight matrix in response to a read request, based on the mask identifier list, comprises: obtaining one or more mask identifier sublists based on the mask identifier list; reading a plurality of data blocks from the merged weight matrix according to the one or more mask identifier sublists in a preset size and in sequence, wherein an element in each mask identifier sublist corresponds to a position identifier of a zero-value element in the corresponding data block; and concatenating the plurality of data blocks in sequence to form the low-bit weight matrix.
[0015] According to a second aspect of this application, a storage device for a weight matrix is provided, comprising: a processor, and a memory storing instructions that, when executed by the processor, cause the processor to perform the method of the first aspect of this application.
[0016] According to a third aspect of this application, a computer-readable storage medium is provided having instructions stored thereon that, when executed by a computer, cause the computer to perform the method of the first aspect of this application.
[0017] The storage method and apparatus for weight matrices according to embodiments of this application utilize the complementarity of the positions of the effective data elements of the high-bit weight matrix and the low-bit weight matrix to merge and store the high-bit weight matrix and the low-bit weight matrix, while storing the position identifiers of the effective data elements of the high-bit weight matrix as a mask identifier list. This can significantly reduce the GPU memory usage while ensuring data access accuracy and efficiency, and can also improve the deployment flexibility of large language models. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the drawings without creative effort.
[0019] Figure 1 This is a flowchart of a method for storing a weight matrix according to an embodiment of this application;
[0020] Figure 2 This is a schematic diagram of the hardware structure of a storage device for a weight matrix according to an embodiment of this application. Detailed Implementation
[0021] The features and exemplary embodiments of various aspects of this application will now be described in detail. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain this application and are not configured to limit this application. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples of this application.
[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.
[0023] The features and exemplary embodiments of various aspects of this application will now be described in detail. Furthermore, the features, structures, or characteristics described below may be combined in any suitable manner in one or more embodiments.
[0024] Large language models have become a focal point of research in recent years. They are artificial intelligence models based on deep learning technology that solve various natural language processing tasks through dialogue. The number of parameters in large language models continues to grow, ranging from billions to trillions, leading to an exponential increase in the demand for computing hardware (especially GPUs and other accelerator cards) memory. To achieve efficient deployment and inference of large language models with limited computing resources, model compression and acceleration techniques have become crucial. High-low bit sparsity schemes under generalized sparsity strategies have become one of the core optimization methods for large language model inference due to their ability to significantly reduce computational complexity.
[0025] The core idea of the high-low bit sparsity scheme is to partition and differentiate the trained full-precision (e.g., 32-bit floating-point) weight matrix according to its importance. Specifically, the weight matrix is split into a high-bit (i.e., high-precision) part and a low-bit (i.e., low-precision) part. The high-bit part retains the core weights that are crucial to the model's accuracy with higher precision (e.g., 8 bits), while the low-bit part stores relatively less important weights with lower precision (e.g., 4 bits). Simultaneously, the weights in both parts undergo structured or unstructured sparsity processing, i.e., removing weights with values close to zero and minimal contribution to the output, setting them to zero. Large language models processed in this way, when running on hardware equipped with high / low precision hybrid computing units (e.g., tensor cores supporting INT8 and INT4 operations), can achieve significant inference acceleration by skipping zero-value calculations and utilizing the high throughput of the low-precision computing units.
[0026] However, the computational advantage of the high-low bit sparse scheme under the generalized sparse strategy is limited in practice by the storage efficiency of the high-low bit weight matrix. Existing storage technologies fail to fully consider the storage characteristics of the high-low bit weight matrix after generalized sparse processing, resulting in a still large GPU memory footprint, which becomes a major bottleneck for deploying large language models on hardware with conventional GPU memory configurations. For example, the inventors found that traditional technical solutions mainly have the following two types of defects after research.
[0027] The first type is the independent storage scheme for high and low bit weight matrices. This scheme treats the sparsed high and low bit weight matrices as independent entities and stores them completely in GPU memory. Since both matrices contain a large number of zero-value elements (for example, in an 8x sparse scenario, approximately 87.5% of the elements in the high bit weight matrix are zero, and approximately 12.5% of the elements in the low bit weight matrix are zero, with the non-zero elements in different positions), these zero values, while skipped during computation, occupy the same amount of GPU memory as the effective data during storage. For large language models with trillions of parameters, this independent storage method means that nearly half of the GPU memory is wasted on zero values that contribute nothing to computation, resulting in extremely low GPU memory utilization, often exceeding the capacity limit of a single commercial GPU, forcing the adoption of complex and inefficient multi-GPU splitting strategies.
[0028] The second category: Simple merged storage with mask redundancy. Some improved schemes attempt to merge the high-bit weight matrix and the low-bit weight matrix in physical storage. However, this type of scheme fails to effectively utilize the "natural complementarity of zero-value positions" between the high-bit weight matrix and the low-bit weight matrix (i.e., the position of zero in the high-bit weight matrix often corresponds to the position of valid data in the low-bit weight matrix, and vice versa). Therefore, a complete Boolean mask matrix still needs to be maintained for the merged weight matrix to identify whether the data at the position of each storage element comes from the high-bit part, the low-bit part, or is zero. Each element in the mask matrix occupies 1 byte (i.e., a Boolean value occupies 1 byte). For a weight matrix containing N elements, its mask matrix requires an additional storage overhead of N bytes. For example, for a data block of 64 elements, the mask overhead is 64 bytes. When the number of parameters in a large language model is huge, the memory overhead introduced by the mask matrix itself becomes very considerable, failing to fundamentally solve the problem of high memory consumption.
[0029] Therefore, embodiments of this application provide a method and apparatus for storing weight matrices, which can significantly reduce GPU memory usage while ensuring data access accuracy and efficiency, and improve the deployment flexibility of large language models.
[0030] Figure 1 This is a flowchart of a method for storing a weight matrix according to an embodiment of this application. Figure 1 As shown, the method for storing a weight matrix according to an embodiment of this application includes the following steps S110 to S150:
[0031] S110: Obtain the high-bit weight matrix and low-bit weight matrix after generalized sparse processing from the large language model. The positions of the elements in the high-bit weight matrix and the low-bit weight matrix correspond one-to-one, and the positions of the effective data elements in the high-bit weight matrix and the positions of the effective data elements in the low-bit weight matrix are complementary.
[0032] S120: Generate a list of mask identifiers based on the positions of valid data elements in the high-bit weight matrix;
[0033] S130: Merge the high-bit weight matrix and the low-bit weight matrix into a merged weight matrix based on complementarity;
[0034] S140: Stores the merged weight matrix and the list of mask identifiers.
[0035] In one embodiment, the large language model can be any large language model, such as deepseek, llama, qwen, GPT, etc., and this paper does not impose any restrictions on it.
[0036] In one embodiment, a high-bit weight matrix and a low-bit weight matrix, after generalized sparse processing, can be obtained from a large language model. For example, the generalized sparse algorithm can be any sparse algorithm, including but not limited to Top-K sparse algorithms, attention-based sparse algorithms, etc., which are not limited in this paper.
[0037] In one embodiment, the elements in the high-bit weight matrix and the low-bit weight matrix are in a one-to-one correspondence, and the positions of the effective data elements in the high-bit weight matrix are complementary to those in the low-bit weight matrix. For example, after generalized sparsity processing, the high-bit weight matrix retains the core weights that are crucial to the accuracy of large language models with higher precision (e.g., 8 bits); the low-bit weight matrix stores relatively minor weights with lower precision (e.g., 4 bits). Therefore, the high-bit weight matrix and the low-bit weight matrix have the characteristic of "naturally complementary zero-value positions (or effective data positions)" (i.e., the zero position in the high-bit weight matrix corresponds to the effective data position in the low-bit weight matrix, and vice versa). For example, in an 8x sparsity scenario, approximately 87.5% of the elements in each matrix of the high-bit weight matrix are zero, and approximately 12.5% of the elements in the low-bit weight matrix are zero, and the non-zero elements in the two matrices are in different positions.
[0038] In one embodiment, a high-bit weight matrix and a low-bit weight matrix can be merged into a merged weight matrix based on complementarity. For example, complementarity is the property that "zero-value positions (or valid data positions) are naturally complementary".
[0039] In one embodiment, merging a high-bit weight matrix and a low-bit weight matrix into a merged weight matrix based on complementarity includes: storing valid data elements in the high-bit weight matrix at corresponding positions in the merged weight matrix based on their positions, and storing valid data elements in the low-bit weight matrix at corresponding positions in the merged weight matrix based on their positions.
[0040] For example, for each element position, if the element in the high-bit weight matrix is a non-zero valid data element, then that high-bit valid data element is stored in the merged weight matrix; if the element in the low-bit weight matrix is a non-zero valid data element, then that low-bit valid data element is stored in the merged weight matrix. The merged weight matrix includes both high-bit and low-bit valid data elements, eliminating the memory usage of overlapping zero values in the original two matrices. Compared to independent storage, the memory usage is reduced by approximately 50%.
[0041] In one embodiment, a list of mask identifiers can be generated based on the positions of valid data elements in the high-bit weight matrix. For example, the list of mask identifiers can be used to reconstruct the high-bit weight matrix and the low-bit weight matrix from the merged weight matrix.
[0042] In one embodiment, generating a mask identifier list based on the positions of valid data elements in a high-bit weight matrix includes: dividing the high-bit weight matrix into multiple data blocks of a preset size and in order; generating a sub-list of mask identifiers for each data block based on the positions of valid data elements in the data block; and concatenating the sub-lists of mask identifiers of multiple data blocks in order to form a mask identifier list.
[0043] In one embodiment, the preset size is based on the hardware storage access granularity of the computing unit, and the number of valid data elements in the data block is based on the preset size and the sparsity factor. For example, in one embodiment, the preset size is 64 elements (i.e., a data block has 64 elements), the sparsity factor is 8, and the number of valid data elements in the data block is 8. Furthermore, other preset sizes and sparsity factors are possible, and this application does not limit them.
[0044] In one embodiment, generating a sublist of mask identifiers for a data block based on the positions of valid data elements within the data block includes using the position identifier of each valid data element in the data block as a corresponding element in the sublist of mask identifiers. For example, in a data block with 64 elements and a sparsity factor of 8, the valid data elements (e.g., high-bit valid data elements) in the data block may be located at positions 3, 15, 22, 30, 38, 45, 51, and 59, then the sublist of mask identifiers could be [3, 15, 22, 30, 38, 45, 51, 59]. That is, each element in the sublist of mask identifiers can be a position identifier of a high-bit valid data element. As mentioned above, the positions of valid data elements in the high-bit weight matrix are complementary to the positions of valid data elements in the low-bit weight matrix. Therefore, when a sublist of mask identifiers for the high-bit weight matrix is generated, this sublist of mask identifiers can also be subsequently used to identify the low-bit weight matrix, for example, based on the aforementioned complementarity.
[0045] In one embodiment, the elements in the mask identifier sublist are of type INT8. For example, the mask identifier sublist for each data block can be stored as INT8 type in the video memory of the high-efficiency computing unit. Because each INT8 type occupies 1 byte of space, the mask identifier sublist for each data block only requires 8 × 1 byte = 8 bytes of storage space; while in the traditional scheme, each data block needs to correspond to a Boolean mask matrix (where each Boolean element occupies 1 byte of space), and each mask matrix occupies 64 × 1 byte = 64 bytes of storage space. By using the mask identifier sublist, the storage overhead of the mask identifier sublist for each data block is reduced from 64 bytes to 8 bytes, resulting in a video memory saving of 87.5%. In addition, the mask identifier sublists of all data blocks are stored sequentially to form a mask identifier list, which facilitates fast subsequent retrieval.
[0046] In one embodiment, the method further includes: reading the high-bit weight matrix and the low-bit weight matrix from the merged weight matrix according to a list of mask identifiers in response to a read request. For example, the complete high-bit weight matrix and the low-bit weight matrix can be read from the merged weight matrix according to the list of mask identifiers.
[0047] In one embodiment, reading a high-bit weight matrix and a low-bit weight matrix from a merged weight matrix in response to a read request, based on a list of mask identifiers, includes: obtaining one or more sublists of mask identifiers based on the list of mask identifiers; reading multiple data blocks from the merged weight matrix according to a preset size and in sequence based on the one or more sublists of mask identifiers, wherein an element in each sublist of mask identifiers corresponds to a position identifier of a valid data element in the corresponding data block; and concatenating the multiple data blocks in sequence to form a high-bit weight matrix.
[0048] For example, when reading multiple data blocks from a merged weight matrix based on one or more sublists of mask identifiers, since the elements in the sublists of mask identifiers are position identifiers of the high-bit valid data elements in the original high-bit weight matrix, the multiple data blocks are data blocks of the high-bit weight matrix if each element in the sublist of mask identifiers corresponds to the position identifier of a valid data element in the corresponding data block. Accordingly, the multiple data blocks can be concatenated sequentially to form the high-bit weight matrix.
[0049] In one embodiment, reading a high-bit weight matrix and a low-bit weight matrix from a merged weight matrix in response to a read request, based on a list of mask identifiers, includes: obtaining one or more sublists of mask identifiers based on the list of mask identifiers; reading multiple data blocks from the merged weight matrix according to a preset size and in sequence based on the one or more sublists of mask identifiers, wherein an element in each sublist of mask identifiers corresponds to a position identifier of a zero-value element in the corresponding data block; and concatenating the multiple data blocks in sequence to form a low-bit weight matrix.
[0050] For example, when reading multiple data blocks from a merged weight matrix based on one or more sublists of mask identifiers, since the elements in the sublists of mask identifiers are position identifiers of the high-bit valid data elements in the original high-bit weight matrix, and each element in the sublist of mask identifiers corresponds to the position identifier of the zero-value element in the corresponding data block, these multiple data blocks are data blocks of the low-bit weight matrix. Accordingly, the multiple data blocks can be concatenated sequentially to form the low-bit weight matrix.
[0051] Furthermore, the restored high-bit weight matrix and low-bit weight matrix can be output to subsequent calculation modules to complete the data reading process. Because the storage format of the mask identifier list (e.g., INT8) is adapted to the parallel access characteristics of the GPU, the latency of the entire reading process is comparable to that of traditional independent storage schemes, and the access efficiency is not significantly reduced due to the saving of GPU memory.
[0052] The storage method for weight matrices according to embodiments of this application utilizes the complementarity of the positions of the effective data elements of the high-bit weight matrix and the low-bit weight matrix to merge and store the high-bit weight matrix and the low-bit weight matrix. At the same time, the position identifiers of the effective data elements of the high-bit weight matrix are stored as a mask identifier list. This method can significantly reduce the GPU memory usage while ensuring data access accuracy and efficiency, and can also improve the deployment flexibility of large language models.
[0053] This application also provides a storage device for a weight matrix, including: a processor and a memory storing instructions, which, when executed by the processor, cause the processor to perform the above-described storage method for the weight matrix.
[0054] Figure 2 This is a schematic diagram of the hardware structure of a storage device for a weight matrix according to an embodiment of this application. Figure 2 The storage device shown for the weight matrix may include a processor 21 and a memory 22 storing computer program instructions.
[0055] Specifically, the processor 21 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application. Furthermore, the processor 21 may also include an accelerator such as a graphics processing unit (GPU) or a tensor processor (TPU).
[0056] Memory 22 may include a large-capacity memory for data or instructions. Where appropriate, memory 22 may include removable or non-removable (or fixed) media. In a particular embodiment, memory 22 is a non-volatile solid-state memory. Memory 22 may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the storage method described above for the weight matrix.
[0057] The processor 21 implements the storage method for the weight matrix in the above embodiments by reading and executing computer program instructions stored in the memory 22.
[0058] In one example, the storage device for the weight matrix may further include a communication interface 23 and a bus 24. Wherein, as Figure 2 As shown, the processor 21, memory 22, and communication interface 23 are connected through bus 24 and complete communication with each other.
[0059] Bus 24 includes hardware, software, or both, that couples components of the storage device used for the weighting matrix together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 24 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.
[0060] This application also provides a computer-readable storage medium having instructions stored thereon, which, when executed by a computer, cause the computer to perform the above-described storage method for the weight matrix.
[0061] Examples of computer-readable storage media include non-transitory computer-readable storage media such as portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, etc.
[0062] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0063] The above description is merely a specific embodiment of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A method for storing a weight matrix, characterized in that, The method includes: Obtain a high-bit weight matrix and a low-bit weight matrix after generalized sparse processing from a large language model. The positions of the elements in the high-bit weight matrix and the low-bit weight matrix correspond one-to-one, and the positions of the effective data elements in the high-bit weight matrix and the positions of the effective data elements in the low-bit weight matrix are complementary. A mask identifier list is generated based on the position of the valid data elements in the high-bit weight matrix; Based on the complementarity, the high-bit weight matrix and the low-bit weight matrix are merged into a merged weight matrix; and Store the merged weight matrix and the mask identifier list. The generation of the mask identifier list based on the positions of valid data elements in the high-bit weight matrix includes: The high-bit weight matrix is divided into multiple data blocks according to a preset size and in order. For each data block, a sublist of mask identifiers for the data block is generated based on the positions of the valid data elements in the data block; The mask identifier sublists of the multiple data blocks are concatenated in sequence to form the mask identifier list. The process of generating a sublist of mask identifiers for the data block based on the positions of valid data elements within the data block includes: The position identifier of each valid data element in the data block is used as a corresponding element in the mask identifier sublist.
2. The method according to claim 1, characterized in that, Merging the high-bit weight matrix and the low-bit weight matrix into a merged weight matrix based on the complementarity includes: Based on the position of the valid data element in the high-bit weight matrix, the valid data element in the high-bit weight matrix is stored in the corresponding position in the merged weight matrix, and Based on the position of the valid data element in the low-bit weight matrix, the valid data element in the low-bit weight matrix is stored in the corresponding position in the merged weight matrix.
3. The method according to claim 1, characterized in that, The elements in the mask identifier sublist are of type INT8.
4. The method according to claim 1, characterized in that, The preset size is based on the hardware storage access granularity of the computing unit, and the number of valid data elements in the data block is based on the preset size and the sparsity factor.
5. The method according to claim 4, characterized in that, The preset size is 64 elements, and the sparsity factor is 8.
6. The method according to claim 1, characterized in that, The method further includes: In response to a read request, the high-bit weight matrix and the low-bit weight matrix are read from the merged weight matrix according to the mask identifier list.
7. The method according to claim 6, characterized in that, Reading the high-bit weight matrix and the low-bit weight matrix from the merged weight matrix according to the mask identifier list in response to a read request includes: Obtain one or more sub-lists of mask identifiers based on the mask identifier list; Multiple data blocks are read from the merged weight matrix according to a preset size and in sequence based on one or more sublists of mask identifiers, wherein each element in the sublist of mask identifiers corresponds to the position identifier of a valid data element in the corresponding data block; and The multiple data blocks are concatenated in sequence to form the high-bit weight matrix.
8. The method according to claim 6, characterized in that, Reading the high-bit weight matrix and the low-bit weight matrix from the merged weight matrix according to the mask identifier list in response to a read request includes: Obtain one or more sub-lists of mask identifiers based on the mask identifier list; Multiple data blocks are read from the merged weight matrix according to a preset size and in sequence based on one or more sublists of mask identifiers, wherein each element in the sublist of mask identifiers corresponds to the position identifier of the zero-value element in the corresponding data block; and The multiple data blocks are concatenated in sequence to form the low-bit weight matrix.
9. A storage device for a weight matrix, characterized in that, The device includes: processor, and A memory storing instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-8.
10. A computer-readable storage medium storing instructions, characterized in that, When executed by a computer, the instructions cause the computer to perform the method according to any one of claims 1-8.