Storage method and device for weight matrix

Abstract

The invention relates to a storage method and device for a weight matrix. The invention provides a weight matrix storage method, which comprises the following steps that: a high-bit weight matrix and a low-bit weight matrix which are subjected to generalized sparse processing are obtained from a large language model, the positions of elements in the high-bit weight matrix and the low-bit weight matrix are in one-to-one correspondence, and the positions of elements in the high-bit weight matrix and the positions of elements in the low-bit weight matrix are in one-to-one correspondence; the positions of the valid data elements in the high bit weight matrix and the positions of the valid data elements in the low bit weight matrix have complementarity; generating a mask identifier list according to the positions of the valid data elements in the high bit weight matrix; combining the high-bit weight matrix and the low-bit weight matrix into a combined weight matrix according to complementarity; and storing the merge weight matrix and the mask identifier list.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of large language model, and more particularly, to a storage method and device for weight matrix. BACKGROUND

[0002] In the field of artificial intelligence such as large language model and computer vision, the number of model parameters continues to grow, for example, from tens of billions to hundreds of billions, which presents an exponential increase in the demand for video memory capacity of computing hardware, especially accelerator cards such as GPUs. In order to achieve efficient deployment and inference of large language models under limited computing resources, model compression and acceleration technology becomes crucial. Among them, the high-low bit sparsity scheme under the generalized sparse strategy has become one of the core optimization means for large language model inference because it can significantly reduce the computational complexity. However, the computational advantage of the high-low bit sparsity scheme under the generalized sparse strategy is limited in practice by the storage efficiency of the high-low bit weight matrix. Existing storage technologies fail to fully consider the storage characteristics of the high-low bit weight matrix processed by the generalized sparse strategy, resulting in a still large video memory occupation, which becomes the main bottleneck for the deployment of large language models on regular video memory configuration hardware.

[0003] Therefore, there is a need for a new storage method and device for weight matrix. SUMMARY

[0004] Embodiments of the present application provide a storage method and device for weight matrix, which can greatly reduce the video memory occupation of GPU while ensuring data access accuracy and efficiency, and improve the deployment flexibility of large language model.

[0005] According to a first aspect of the present application, a storage method for weight matrix is provided, wherein the method comprises: obtaining a high-bit weight matrix and a low-bit weight matrix processed by a generalized sparse strategy from a large language model, wherein the elements in the high-bit weight matrix and the low-bit weight matrix correspond one-to-one, and the positions of valid data elements in the high-bit weight matrix and the low-bit weight matrix have complementarity; generating a mask identifier list according to the positions of valid data elements in the high-bit weight matrix; merging the high-bit weight matrix and the low-bit weight matrix into a merged weight matrix according to the complementarity; and storing the merged weight matrix and the mask identifier list.

[0006] According to an embodiment of the first aspect of the present application, wherein merging the high-bit weight matrix and the low-bit weight matrix into a merged weight matrix according to the complementarity comprises storing valid data elements in the high-bit weight matrix in respective positions in the merged weight matrix according to positions of the valid data elements in the high-bit weight matrix, and storing valid data elements in the low-bit weight matrix in respective positions in the merged weight matrix according to positions of the valid data elements in the low-bit weight matrix.

[0007] According to an embodiment of the first aspect of the present application, wherein generating the list of mask identifiers according to positions of valid data elements in the high-bit weight matrix comprises dividing the high-bit weight matrix into a plurality of data blocks in a preset size and in order; for each data block, generating a sub-list of mask identifiers of the data block according to positions of valid data elements in the data block; and concatenating the sub-lists of mask identifiers of the plurality of data blocks in order to form the list of mask identifiers.

[0008] According to an embodiment of the first aspect of the present application, wherein generating the sub-list of mask identifiers of the data block according to positions of valid data elements in the data block comprises identifying positions of each valid data element in the data block as a respective element in the sub-list of mask identifiers.

[0009] According to an embodiment of the first aspect of the present application, wherein elements in the sub-list of mask identifiers are of INT8 type.

[0010] According to an embodiment of the first aspect of the present application, wherein the preset size is based on hardware memory access granularity of a computing unit, and a number of valid data elements in the data block is based on the preset size and a sparsity multiple.

[0011] According to an embodiment of the first aspect of the present application, wherein the preset size is 64 elements, and the sparsity multiple is 8 times.

[0012] According to an embodiment of the first aspect of the present application, the method further comprises reading the high-bit weight matrix and the low-bit weight matrix from the merged weight matrix according to the list of mask identifiers in response to a read request.

[0013] According to an embodiment of the first aspect of the present application, wherein reading the high-bit weight matrix and the low-bit weight matrix from the merged weight matrix according to the mask identifier list in response to a read request comprises: obtaining one or more mask identifier sub-lists according to the mask identifier list; reading a plurality of data blocks from the merged weight matrix according to the one or more mask identifier sub-lists in a preset size and in order, wherein an element in each mask identifier sub-list corresponds to a position identifier of a valid data element in a corresponding data block; and splicing the plurality of data blocks in order to form the high-bit weight matrix.

[0014] According to an embodiment of the first aspect of the present application, wherein reading the high-bit weight matrix and the low-bit weight matrix from the merged weight matrix according to the mask identifier list in response to a read request comprises: obtaining one or more mask identifier sub-lists according to the mask identifier list; reading a plurality of data blocks from the merged weight matrix according to the one or more mask identifier sub-lists in a preset size and in order, wherein an element in each mask identifier sub-list corresponds to a position identifier of a zero-value element in a corresponding data block; and splicing the plurality of data blocks in order to form the low-bit weight matrix.

[0015] According to a second aspect of the present application, a storage device for weight matrix is provided, comprising: a processor, and a memory storing instructions which, when executed by the processor, cause the processor to perform the method of the first aspect of the present application.

[0016] According to a third aspect of the present application, a computer-readable storage medium is provided, storing instructions thereon which, when executed by a computer, cause the computer to perform the method of the first aspect of the present application.

[0017] According to the embodiments of the present application, the storage method and device for weight matrix can greatly reduce the GPU memory occupation while ensuring the data access accuracy and efficiency by utilizing the complementarity of the positions of the valid data elements of the high-bit weight matrix and the low-bit weight matrix, and storing the high-bit weight matrix and the low-bit weight matrix in a merged manner, and storing the position identifiers of the valid data elements of the high-bit weight matrix as a mask identifier list, and can improve the deployment flexibility of large language models. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced as follows. Obviously, the drawings described below only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the drawings.

[0019] Figure 1 is a flow chart of a storage method for a weight matrix according to an embodiment of the present application; Figure 2 is a hardware structure schematic diagram of a storage device for a weight matrix according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The features and exemplary embodiments of various aspects of the present application will be described in detail below, in order to make the purposes, technical solutions and advantages of the present application more clear and apparent, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present application, and are not configured to limit the present application. The present application can be implemented without some of these specific details by those skilled in the art. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0021] It should be noted that in this paper, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the elements defined by the statement "include" do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0022] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In addition, the features, structures or characteristics described below can be combined in any suitable manner in one or more embodiments.

[0023] Large language models are the focus of researchers in recent years. They are a kind of artificial intelligence model based on deep learning technology, which solves various natural language processing tasks through dialogue. The number of model parameters of large language models continues to grow, for example, from tens of billions to hundreds of billions, which has exponentially increased the demand for video memory capacity of computing hardware, especially accelerator cards such as graphics processing units. In order to achieve efficient deployment and inference of large language models under limited computing resources, model compression and acceleration technology becomes crucial. High-low bit sparse schemes under the generalized sparse strategy have become one of the core optimization methods for large language model inference because they can significantly reduce computational complexity.

[0024] The core idea of the high-low bit sparse scheme is to divide and differentially process the trained full-precision (for example, 32-bit floating-point number) weight matrix according to importance. Specifically, the weight matrix is split into a high-bit (i.e., high-precision) part and a low-bit (i.e., low-precision) part. The high-bit part retains the core weights that are crucial to model accuracy at a higher precision (for example, 8 bits), and the low-bit part stores the relatively secondary weights at a lower precision (for example, 4 bits). At the same time, the weights of the two parts are respectively structured or unstructured sparse processing, that is, the weights with values close to zero and little contribution to the output are removed and set to zero. The large language model processed in this way can achieve significant inference acceleration when running on hardware equipped with high / low precision mixed computing units (for example, tensor cores supporting INT8 and INT4 operations) by skipping zero value calculation and utilizing the high throughput characteristics of low precision computing units.

[0025] However, the computational advantage of the high-low bit sparse scheme under the generalized sparse strategy is limited in practice by the storage efficiency of the high-low bit weight matrix. Existing storage technologies fail to fully consider the storage characteristics of the high-low bit weight matrix processed by the generalized sparse processing, resulting in a still large video memory occupation, which becomes the main bottleneck for the deployment of large language models on regular video memory configuration hardware. For example, the inventors found in their research that the traditional technical solutions mainly have the following two types of defects.

[0026] The first type is the independent storage scheme for high and low bit weight matrices. This scheme treats the sparsed high and low bit weight matrices as independent entities and stores them completely in GPU memory. Since both matrices contain a large number of zero-value elements (for example, in an 8x sparse scenario, approximately 87.5% of the elements in the high bit weight matrix are zero, and approximately 12.5% ​​of the elements in the low bit weight matrix are zero, with the non-zero elements in different positions), these zero values, while skipped during computation, occupy the same amount of GPU memory as the effective data during storage. For large language models with trillions of parameters, this independent storage method means that nearly half of the GPU memory is wasted on zero values ​​that contribute nothing to computation, resulting in extremely low GPU memory utilization, often exceeding the capacity limit of a single commercial GPU, forcing the adoption of complex and inefficient multi-GPU splitting strategies.

[0027] The second category: Simple merged storage with mask redundancy. Some improved schemes attempt to merge the high-bit weight matrix and the low-bit weight matrix in physical storage. However, this type of scheme fails to effectively utilize the "natural complementarity of zero-value positions" between the high-bit weight matrix and the low-bit weight matrix (i.e., the position of zero in the high-bit weight matrix often corresponds to the position of valid data in the low-bit weight matrix, and vice versa). Therefore, a complete Boolean mask matrix still needs to be maintained for the merged weight matrix to identify whether the data at the position of each storage element comes from the high-bit part, the low-bit part, or is zero. Each element in the mask matrix occupies 1 byte (i.e., a Boolean value occupies 1 byte). For a weight matrix containing N elements, its mask matrix requires an additional storage overhead of N bytes. For example, for a data block of 64 elements, the mask overhead is 64 bytes. When the number of parameters in a large language model is huge, the memory overhead introduced by the mask matrix itself becomes very considerable, failing to fundamentally solve the problem of high memory consumption.

[0028] Therefore, embodiments of this application provide a method and apparatus for storing weight matrices, which can significantly reduce GPU memory usage while ensuring data access accuracy and efficiency, and improve the deployment flexibility of large language models.

[0029] Figure 1 This is a flowchart of a method for storing a weight matrix according to an embodiment of this application. Figure 1 As shown, the method for storing a weight matrix according to an embodiment of this application includes the following steps S110 to S150: S110: Obtain the high-bit weight matrix and low-bit weight matrix after generalized sparse processing from the large language model. The positions of the elements in the high-bit weight matrix and the low-bit weight matrix correspond one-to-one, and the positions of the effective data elements in the high-bit weight matrix and the positions of the effective data elements in the low-bit weight matrix are complementary. S120: Generate a mask identifier list based on the positions of valid data elements in the high-bit weight matrix; S130: Merge the high-bit weight matrix and the low-bit weight matrix into a merged weight matrix based on complementarity; S140: Stores the merged weight matrix and the list of mask identifiers.

[0030] In one embodiment, the large language model can be any large language model, such as deepseek, llama, qwen, GPT, etc., and this paper does not impose any restrictions on it.

[0031] In one embodiment, a high-bit weight matrix and a low-bit weight matrix, after generalized sparse processing, can be obtained from a large language model. For example, the generalized sparse algorithm can be any sparse algorithm, including but not limited to Top-K sparse algorithms, attention-based sparse algorithms, etc., which are not limited in this paper.

[0032] In one embodiment, the elements in the high-bit weight matrix and the low-bit weight matrix are in a one-to-one correspondence, and the positions of the effective data elements in the high-bit weight matrix are complementary to those in the low-bit weight matrix. For example, after generalized sparsity processing, the high-bit weight matrix retains the core weights that are crucial to the accuracy of large language models with higher precision (e.g., 8 bits); the low-bit weight matrix stores relatively minor weights with lower precision (e.g., 4 bits). Therefore, the high-bit weight matrix and the low-bit weight matrix have the characteristic of "naturally complementary zero-value positions (or effective data positions)" (i.e., the zero position in the high-bit weight matrix corresponds to the effective data position in the low-bit weight matrix, and vice versa). For example, in an 8x sparsity scenario, approximately 87.5% of the elements in each matrix of the high-bit weight matrix are zero, and approximately 12.5% ​​of the elements in the low-bit weight matrix are zero, and the non-zero elements in the two matrices are in different positions.

[0033] In one embodiment, a high-bit weight matrix and a low-bit weight matrix can be merged into a merged weight matrix based on complementarity. For example, complementarity is the property that "zero-value positions (or valid data positions) are naturally complementary".

[0034] In one embodiment, merging a high-bit weight matrix and a low-bit weight matrix into a merged weight matrix based on complementarity includes: storing valid data elements in the high-bit weight matrix at corresponding positions in the merged weight matrix based on their positions, and storing valid data elements in the low-bit weight matrix at corresponding positions in the merged weight matrix based on their positions.

[0035] For example, for each element position, if the element in the high-bit weight matrix is ​​a non-zero valid data element, then that high-bit valid data element is stored in the merged weight matrix; if the element in the low-bit weight matrix is ​​a non-zero valid data element, then that low-bit valid data element is stored in the merged weight matrix. The merged weight matrix includes both high-bit and low-bit valid data elements, eliminating the memory usage of overlapping zero values ​​in the original two matrices. Compared to independent storage, the memory usage is reduced by approximately 50%.

[0036] In one embodiment, a list of mask identifiers can be generated based on the positions of valid data elements in the high-bit weight matrix. For example, the list of mask identifiers can be used to reconstruct the high-bit weight matrix and the low-bit weight matrix from the merged weight matrix.

[0037] In one embodiment, generating a mask identifier list based on the positions of valid data elements in a high-bit weight matrix includes: dividing the high-bit weight matrix into multiple data blocks of a preset size and in order; generating a sub-list of mask identifiers for each data block based on the positions of valid data elements in the data block; and concatenating the sub-lists of mask identifiers of multiple data blocks in order to form a mask identifier list.

[0038] In one embodiment, the preset size is based on the hardware storage access granularity of the computing unit, and the number of valid data elements in the data block is based on the preset size and the sparsity factor. For example, in one embodiment, the preset size is 64 elements (i.e., a data block has 64 elements), the sparsity factor is 8, and the number of valid data elements in the data block is 8. Furthermore, other preset sizes and sparsity factors are possible, and this application does not limit them.

[0039] In one embodiment, generating a sublist of mask identifiers for a data block based on the positions of valid data elements within the data block includes using the position identifier of each valid data element in the data block as a corresponding element in the sublist of mask identifiers. For example, in a data block with 64 elements and a sparsity factor of 8, the valid data elements (e.g., high-bit valid data elements) in the data block may be located at positions 3, 15, 22, 30, 38, 45, 51, and 59, then the sublist of mask identifiers could be [3, 15, 22, 30, 38, 45, 51, 59]. That is, each element in the sublist of mask identifiers can be a position identifier of a high-bit valid data element. As mentioned above, the positions of valid data elements in the high-bit weight matrix are complementary to the positions of valid data elements in the low-bit weight matrix. Therefore, when a sublist of mask identifiers for the high-bit weight matrix is ​​generated, this sublist of mask identifiers can also be subsequently used to identify the low-bit weight matrix, for example, based on the aforementioned complementarity.

[0040] In one embodiment, the elements in the mask identifier sublist are of type INT8. For example, the mask identifier sublist for each data block can be stored as INT8 type in the video memory of the high-efficiency computing unit. Because each INT8 type occupies 1 byte of space, the mask identifier sublist for each data block only requires 8 × 1 byte = 8 bytes of storage space; while in the traditional scheme, each data block needs to correspond to a Boolean mask matrix (where each Boolean element occupies 1 byte of space), and each mask matrix occupies 64 × 1 byte = 64 bytes of storage space. By using the mask identifier sublist, the storage overhead of the mask identifier sublist for each data block is reduced from 64 bytes to 8 bytes, resulting in a video memory saving of 87.5%. In addition, the mask identifier sublists of all data blocks are stored sequentially to form a mask identifier list, which facilitates fast subsequent retrieval.

[0041] In one embodiment, the method further includes: reading the high-bit weight matrix and the low-bit weight matrix from the merged weight matrix according to a list of mask identifiers in response to a read request. For example, the complete high-bit weight matrix and the low-bit weight matrix can be read from the merged weight matrix according to the list of mask identifiers.

[0042] In one embodiment, reading a high-bit weight matrix and a low-bit weight matrix from a merged weight matrix in response to a read request, based on a list of mask identifiers, includes: obtaining one or more sublists of mask identifiers based on the list of mask identifiers; reading multiple data blocks from the merged weight matrix according to a preset size and in sequence based on the one or more sublists of mask identifiers, wherein an element in each sublist of mask identifiers corresponds to a position identifier of a valid data element in the corresponding data block; and concatenating the multiple data blocks in sequence to form a high-bit weight matrix.

[0043] For example, when reading multiple data blocks from a merged weight matrix based on one or more sublists of mask identifiers, since the elements in the sublists of mask identifiers are position identifiers of the high-bit valid data elements in the original high-bit weight matrix, the multiple data blocks are data blocks of the high-bit weight matrix if each element in the sublist of mask identifiers corresponds to the position identifier of a valid data element in the corresponding data block. Accordingly, the multiple data blocks can be concatenated sequentially to form the high-bit weight matrix.

[0044] In one embodiment, reading a high-bit weight matrix and a low-bit weight matrix from a merged weight matrix in response to a read request, based on a list of mask identifiers, includes: obtaining one or more sublists of mask identifiers based on the list of mask identifiers; reading multiple data blocks from the merged weight matrix according to a preset size and in sequence based on the one or more sublists of mask identifiers, wherein an element in each sublist of mask identifiers corresponds to a position identifier of a zero-value element in the corresponding data block; and concatenating the multiple data blocks in sequence to form a low-bit weight matrix.

[0045] For example, when reading multiple data blocks from a merged weight matrix based on one or more sublists of mask identifiers, since the elements in the sublists of mask identifiers are position identifiers of the high-bit valid data elements in the original high-bit weight matrix, and each element in the sublist of mask identifiers corresponds to the position identifier of the zero-value element in the corresponding data block, these multiple data blocks are data blocks of the low-bit weight matrix. Accordingly, the multiple data blocks can be concatenated sequentially to form the low-bit weight matrix.

[0046] Furthermore, the restored high-bit weight matrix and low-bit weight matrix can be output to subsequent calculation modules to complete the data reading process. Because the storage format of the mask identifier list (e.g., INT8) is adapted to the parallel access characteristics of the GPU, the latency of the entire reading process is comparable to that of traditional independent storage schemes, and the access efficiency is not significantly reduced due to the saving of GPU memory.

[0047] The storage method for weight matrices according to embodiments of this application utilizes the complementarity of the positions of the effective data elements of the high-bit weight matrix and the low-bit weight matrix to merge and store the high-bit weight matrix and the low-bit weight matrix. At the same time, the position identifiers of the effective data elements of the high-bit weight matrix are stored as a mask identifier list. This method can significantly reduce the GPU memory usage while ensuring data access accuracy and efficiency, and can also improve the deployment flexibility of large language models.

[0048] This application also provides a storage device for a weight matrix, including: a processor and a memory storing instructions, which, when executed by the processor, cause the processor to perform the above-described storage method for the weight matrix.

[0049] Figure 2 This is a schematic diagram of the hardware structure of a storage device for a weight matrix according to an embodiment of this application. Figure 2 The storage device shown for the weight matrix may include a processor 21 and a memory 22 storing computer program instructions.

[0050] Specifically, the processor 21 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application. Furthermore, the processor 21 may also include an accelerator such as a graphics processing unit (GPU) or a tensor processor (TPU).

[0051] Memory 22 may include a large-capacity memory for data or instructions. Where appropriate, memory 22 may include removable or non-removable (or fixed) media. In a particular embodiment, memory 22 is a non-volatile solid-state memory. Memory 22 may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the storage method described above for the weight matrix.

[0052] The processor 21 implements the storage method for the weight matrix in the above embodiments by reading and executing computer program instructions stored in the memory 22.

[0053] In one example, the storage device for the weight matrix may further include a communication interface 23 and a bus 24. Wherein, asFigure 2 As shown, the processor 21, memory 22, and communication interface 23 are connected through bus 24 and complete communication with each other.

[0054] Bus 24 includes hardware, software, or both, that couples components of the storage device used for the weighting matrix together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 24 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0055] This application also provides a computer-readable storage medium having instructions stored thereon, which, when executed by a computer, cause the computer to perform the above-described storage method for the weight matrix.

[0056] Examples of computer-readable storage media include non-transitory computer-readable storage media such as portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, etc.

[0057] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0058] The above description is merely a specific embodiment of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for storing a weight matrix, characterized in that, The method includes: Obtain a high-bit weight matrix and a low-bit weight matrix after generalized sparse processing from a large language model. The positions of the elements in the high-bit weight matrix and the low-bit weight matrix correspond one-to-one, and the positions of the effective data elements in the high-bit weight matrix and the positions of the effective data elements in the low-bit weight matrix are complementary. A mask identifier list is generated based on the position of the valid data elements in the high-bit weight matrix; Based on the complementarity, the high-bit weight matrix and the low-bit weight matrix are merged into a merged weight matrix; and Store the merged weight matrix and the mask identifier list.

2. The method according to claim 1, characterized in that, Merging the high-bit weight matrix and the low-bit weight matrix into a merged weight matrix based on the complementarity includes: Based on the position of the valid data element in the high-bit weight matrix, the valid data element in the high-bit weight matrix is ​​stored in the corresponding position in the merged weight matrix, and Based on the position of the valid data element in the low-bit weight matrix, the valid data element in the low-bit weight matrix is ​​stored in the corresponding position in the merged weight matrix.

3. The method according to claim 1, characterized in that, The mask identifier list is generated based on the positions of the valid data elements in the high-bit weight matrix, including: The high-bit weight matrix is ​​divided into multiple data blocks according to a preset size and in a specific order; For each data block, a sublist of mask identifiers for the data block is generated based on the positions of the valid data elements in the data block; The mask identifier sublists of the multiple data blocks are concatenated in sequence to form the mask identifier list.

4. The method according to claim 3, characterized in that, Generating a sublist of mask identifiers for a data block based on the positions of valid data elements within the data block includes: The position identifier of each valid data element in the data block is used as a corresponding element in the mask identifier sublist.

5. The method according to claim 4, characterized in that, The elements in the mask identifier sublist are of type INT8.

6. The method according to claim 3, characterized in that, The preset size is based on the hardware storage access granularity of the computing unit, and the number of valid data elements in the data block is based on the preset size and the sparsity factor.

7. The method according to claim 6, characterized in that, The preset size is 64 elements, and the sparsity factor is 8.

8. The method according to claim 1, characterized in that, The method further includes: In response to a read request, the high-bit weight matrix and the low-bit weight matrix are read from the merged weight matrix according to the mask identifier list.

9. The method according to claim 8, characterized in that, Reading the high-bit weight matrix and the low-bit weight matrix from the merged weight matrix in response to a read request, based on the mask identifier list, includes: Obtain one or more sub-lists of mask identifiers based on the mask identifier list; Multiple data blocks are read from the merged weight matrix according to a preset size and in sequence based on one or more sublists of mask identifiers, wherein each element in the sublist of mask identifiers corresponds to the position identifier of a valid data element in the corresponding data block; and The multiple data blocks are concatenated in sequence to form the high-bit weight matrix.

10. The method according to claim 8, characterized in that, Reading the high-bit weight matrix and the low-bit weight matrix from the merged weight matrix in response to a read request, based on the mask identifier list, includes: Obtain one or more sub-lists of mask identifiers based on the mask identifier list; Multiple data blocks are read from the merged weight matrix according to a preset size and in sequence based on one or more sublists of mask identifiers, wherein each element in the sublist of mask identifiers corresponds to the position identifier of the zero-value element in the corresponding data block; and The multiple data blocks are concatenated in sequence to form the low-bit weight matrix.

11. A storage device for a weight matrix, characterized in that, The device includes: processor, and A memory storing instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-10.

12. A computer-readable storage medium storing instructions, characterized in that, When executed by a computer, the instructions cause the computer to perform the method according to any one of claims 1-10.

Images