Model weight storage and calculation processing method, device, storage medium, and program

Abstract

Embodiments of the present disclosure provide a model weight storage and calculation processing method, a device, a storage medium, and a program. The method comprises: acquiring a quantized weight matrix, a quantized parameter matrix, and a plurality of storage matrices, used for storing a non-quantized weight matrix, of a target network layer, the plurality of storage matrices including a first storage matrix used for storing non-zero element values in the non-quantized weight matrix in columns, a second storage matrix used for storing line numbers of the non-zero element values in the non-quantized weight matrix, a third storage matrix used for storing the number of non-zero element values in each column of the non-quantized weight matrix, and a fourth storage matrix used for storing a corresponding starting index position of each column of the non-quantized weight matrix in the first storage matrix; and determining a first calculation result and a second calculation result of a same position in an output matrix on the basis of the plurality of storage matrixes, the first calculation result being the product of an input matrix and a dequantized weight matrix, and the second calculation result being the product of the input matrix and the non-quantized weight matrix. The solution can realize accelerated calculation of matrix multiplication.

Description

[0001] This disclosure claims priority to Chinese patent application number 202410102457.7, filed with the China Patent Office on January 23, 2024, entitled "Model Weight Storage, Calculation Processing Method and Device, Storage Medium and Program," the entire contents of which are incorporated herein by reference. Technical Field This disclosure relates to the field of artificial intelligence technology, and more particularly to a model weight storage, calculation processing method and device, storage medium and program. Background Art With the continuous development of artificial intelligence technology, neural network models with larger weight values ​​are emerging, such as large language models. A neural network model includes several network layers (such as convolutional layers and fully connected layers), and different network layers correspond to different weight matrices. After training a neural network model, the weight values ​​in the weight matrices of each network layer are generally in a floating-point data format, such as FP16. Storing the weight matrix in this data format consumes more storage space. To save storage space, the weight matrices of each network layer in a neural network model typically require quantization. For example, this is done using the near-lossless LLM (Large Language Model) weight compression (SpQR) quantization method. For any network layer in a neural network model, the quantization results of the weight matrix of that layer include a quantized weight matrix, a quantization parameter matrix, and a non-quantized weight matrix. The quantization parameter matrix includes a zero-value matrix and a quantization scale matrix. The quantized weight matrix is ​​composed of the quantized results of the weight values ​​in the weight matrix that require quantization. Quantization is implemented using the quantization parameter matrix. The non-quantized weight matrix is ​​composed of the weight values ​​in the weight matrix that do not require quantization. The proportion of weight values ​​that do not require quantization is very small (typically less than 1%), so the non-quantized weight matrix is ​​a sparse matrix. The data formats used for the element values ​​in the quantization weight matrix, quantization parameter matrix, and non-quantization weight matrix are different. For example, the element values ​​in the quantization weight matrix are in INT4 format, the element values ​​in the non-quantization weight matrix are in FP16 format, the zero values ​​(zero) given in the zero value matrix are in INT4 format, and the quantization scales (scale) given in the quantization scale matrix are in FP16 format.For the aforementioned non-sparse quantization weight matrix and quantization parameter matrix, the element values ​​in the corresponding matrices can be stored row by row in storage space. However, since the non-quantization weight matrix is ​​a sparse matrix, meaning only a small number of elements are non-zero, storing the elements in each row of the non-quantization weight matrix continuously, as is done for the quantization weight matrix and quantization parameter matrix, would consume a significant amount of storage space. Therefore, to save storage space, the non-quantization weight matrix is ​​typically stored in a coordinating format (COO). In the COO storage structure, the non-zero elements in the non-quantization weight matrix are stored using a triplet (row, col, data), where row and col are the row and column numbers of the non-zero element, respectively, and data is the value of the non-zero element. Although C is used. This storage structure can reduce the storage space required for storing non-quantized weight matrices, but it hinders the acceleration of matrix multiplication calculations during the inference phase of a neural network model. This matrix multiplication is the multiplication between the input matrix of any layer in the neural network model and the weight matrix of that layer. SUMMARY OF THE INVENTION Embodiments of the present disclosure provide a method and device for storing and calculating model weights, as well as a storage medium and program, to improve the computational efficiency of matrix multiplication.In a first aspect, embodiments of the present disclosure provide a model weight calculation processing method, the method comprising: in response to input of an input matrix of a target network layer in a neural network model obtained based on an inference task, obtaining a quantized weight matrix and a quantized parameter matrix corresponding to the target network layer; obtaining multiple storage matrices for storing a non-quantized weight matrix corresponding to the target network layer, the multiple storage matrices including a first storage matrix for storing non-zero element values ​​in the non-quantized weight matrix by column, a second storage matrix for storing row numbers corresponding to non-zero element values ​​in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values ​​in each column of the non-quantized weight matrix, and a fourth storage matrix for storing starting index positions corresponding to each column of the non-quantized weight matrix in the first storage matrix; wherein the non-quantized weight matrix is ​​a sparse matrix, and a quantization result of the weight matrix of the target network layer includes the quantized weight matrix, the quantization parameter matrix, and the non-quantized weight matrix; determining, based on the multiple storage matrices, a first calculation result and a second calculation result corresponding to the same position in an output matrix of the target network layer, so as to obtain a target calculation result for the same position based on the cumulative sum of the first calculation result and the second calculation result. The first calculation result corresponds to the product of the input matrix and an inverse quantization weight matrix, the second calculation result corresponds to the product of the input matrix and the unquantized weight matrix, and the inverse quantization weight matrix is ​​obtained by inverse quantizing the quantization weight matrix based on the quantization parameter matrix.In a second aspect, an embodiment of the present disclosure provides a model weight calculation processing device, the device comprising: an acquisition module, configured to, in response to input of an input matrix of a target network layer in a neural network model obtained based on an inference task, acquire a quantized weight matrix and a quantized parameter matrix corresponding to the target network layer, and acquire multiple storage matrices for storing a non-quantized weight matrix corresponding to the target network layer, the multiple storage matrices comprising a first storage matrix for storing non-zero element values ​​in the non-quantized weight matrix by column, a second storage matrix for storing row numbers corresponding to non-zero element values ​​in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values ​​in each column of the non-quantized weight matrix, and a fourth storage matrix for storing starting index positions corresponding to each column of the non-quantized weight matrix in the first storage matrix; wherein the non-quantized weight matrix is ​​a sparse matrix, and a quantization result of the weight matrix of the target network layer comprises the quantized weight matrix, the quantization parameter matrix, and the non-quantized weight matrix; and an inference module, configured to determine, based on the multiple storage matrices, a first calculation result and a second calculation result corresponding to the same position in an output matrix of the target network layer. The target calculation result for the same position is obtained based on the cumulative sum of the first calculation result and the second calculation result, wherein the first calculation result corresponds to the product of the input matrix and the inverse quantization weight matrix, the second calculation result corresponds to the product of the input matrix and the unquantized weight matrix, and the inverse quantization weight matrix is ​​obtained by inverse quantizing the quantization weight matrix based on the quantization parameter matrix. In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the processor executes the executable code, the processor is enabled to at least implement the model weight calculation method described in the first aspect. In a fourth aspect, an embodiment of the present disclosure provides a non-transitory machine-readable storage medium, the non-transitory machine-readable storage medium storing executable code, and when the processor of the electronic device executes the executable code, the processor is enabled to at least implement the model weight calculation method described in the first aspect. Furthermore, an embodiment of the present disclosure provides a computer program product, comprising a computer program, and when the processor of the electronic device executes the computer program product, the processor is enabled to at least implement the model weight calculation method described in the first aspect.In a fifth aspect, an embodiment of the present disclosure provides a model weight storage method, comprising: obtaining a quantization result of a weight matrix of a target network layer in a neural network model, the quantization result including a quantization weight matrix, a quantization parameter matrix, and a non-quantization weight matrix, wherein the non-quantization weight matrix is ​​a sparse matrix; generating multiple storage matrices for storing the non-quantization weight matrix, the multiple storage matrices including a first storage matrix for storing each non-zero element value in the non-quantization weight matrix by column, a second storage matrix for storing the row number corresponding to each non-zero element value in the non-quantization weight matrix in the non-quantization weight matrix, a third storage matrix for storing the number of non-zero element values ​​in each column of the non-quantization weight matrix, and a fourth storage matrix for storing the starting index position corresponding to each column in the non-quantization weight matrix in the first storage matrix; and storing the multiple storage matrices, the quantization weight matrix, and the quantization parameter matrix. In a sixth aspect, an embodiment of the present disclosure provides a model weight storage device, comprising: an acquisition module, configured to acquire a quantization result of a weight matrix of a target network layer in a neural network model, wherein the quantization result includes a quantization weight matrix, a quantization parameter matrix, and a non-quantization weight matrix, and the non-quantization weight matrix is ​​a sparse matrix; a generation module, configured to generate multiple storage matrices for storing the non-quantization weight matrix, wherein the multiple storage matrices include a first storage matrix for storing each non-zero element value in the non-quantization weight matrix by column, a second storage matrix for storing the row number corresponding to each non-zero element value in the non-quantization weight matrix in the non-quantization weight matrix, a third storage matrix for storing the number of non-zero element values ​​in each column of the non-quantization weight matrix, and a fourth storage matrix for storing the starting index position corresponding to each column in the non-quantization weight matrix in the first storage matrix; and a storage module, configured to store the multiple storage matrices, the quantization weight matrix, and the quantization parameter matrix. In a seventh aspect, embodiments of the present disclosure provide an electronic device comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the processor executes the executable code, the processor is enabled to implement at least the model weight storage method described in the fifth aspect. In an eighth aspect, embodiments of the present disclosure provide a non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores executable code, and when the processor of the electronic device executes the executable code, the processor is enabled to implement at least the model weight storage method described in the fifth aspect.In a ninth aspect, embodiments of the present disclosure provide a computer program product comprising a computer program that, when executed by a processor of an electronic device, enables the processor to at least implement the model weight storage method described in the fifth aspect. The model weight calculation and processing scheme provided by embodiments of the present disclosure first quantizes the weight matrices of each network layer of a neural network model to save storage space. For any target network layer, the quantization results of the target network layer's weight matrix include a quantized weight matrix, a quantization parameter matrix, and a non-quantized weight matrix. The non-quantized weight matrix is ​​a sparse matrix. To save storage space while ensuring the speed of matrix multiplication between the target network layer's input matrix and the weight matrix, a new storage structure for storing the non-quantized weight matrix is ​​provided. Specifically, multiple storage matrices are generated for storing the unquantized weight matrix. These include a first storage matrix for storing the values ​​of each non-zero element in the unquantized weight matrix by column, a second storage matrix for storing the row number corresponding to each non-zero element in the unquantized weight matrix, a third storage matrix for storing the number of non-zero elements in each column of the unquantized weight matrix, and a fourth storage matrix for storing the starting index position corresponding to each column in the first storage matrix. Thus, these four storage matrices enable continuous storage of each non-zero element in the unquantized weight matrix and the position coordinates (row and column numbers) of each non-zero element in the unquantized weight matrix across different storage matrices. These four storage matrices can be stored continuously in the storage space, just as the quantized weight matrix is ​​normally stored. When calculating the product of the input matrix and the weight matrix of the target network layer, the first and second calculation results corresponding to the same position in the output matrix (the product of the multiplication) are determined based on the multiple storage matrices. The target calculation result for the same position is then obtained by summing the first and second calculation results. The first calculation result corresponds to the product of the input matrix and the inverse quantized weight matrix, where the inverse quantized weight matrix is ​​obtained by inverse quantizing the quantized weight matrix based on the quantization parameter matrix. The second calculation result corresponds to the product of the input matrix and the unquantized weight matrix. During the calculation of the second calculation result, accelerated multiplication of the input matrix and the unquantized weight matrix can be implemented based on the four storage matrices, thereby improving the efficiency of the multiplication of the input matrix and the weight matrix of the target network layer.To more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below represent some embodiments of the present disclosure. Persons skilled in the art can also derive other drawings based on these drawings without inventive effort. Figure 1 is a schematic diagram illustrating the principle of a conventional matrix multiplication calculation based on weight matrix quantization results, provided in an embodiment of the present disclosure; Figure 2 is a flow chart of a model weight storage method, provided in an embodiment of the present disclosure; Figure 3 is a schematic diagram of a new storage structure for storing unquantized weight matrices, provided in an embodiment of the present disclosure; Figure 4 is a schematic diagram of a storage method for weight matrix quantization results, provided in an embodiment of the present disclosure; Figure 5 is a flow chart of a model weight calculation processing method, provided in an embodiment of the present disclosure; Figure 6 is a schematic diagram of the structure of a model weight calculation processing device, provided in an embodiment of the present disclosure; Figure 7 is a schematic diagram of the structure of a model weight storage device, provided in an embodiment of the present disclosure; and Figure 8 is a schematic diagram of the structure of an electronic device, provided in an embodiment of the present disclosure. DETAILED DESCRIPTION To further clarify the objectives, technical solutions, and advantages of the embodiments of the present disclosure, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. It should be noted that the described embodiments represent only a portion of the embodiments of the present disclosure, and are not exhaustive. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present disclosure without inventive effort are within the scope of protection of the present disclosure. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display) involved in the embodiments of the present disclosure are all authorized by the user or fully authorized by all parties. The collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or reject. The following detailed description of some embodiments of the present disclosure is provided in conjunction with the accompanying drawings. The following embodiments and features may be combined unless there is a conflict between the embodiments. Furthermore, the sequence of steps in the following method embodiments is provided for illustrative purposes only and is not intended to be a strict limitation. Large language models are rapidly developing and are widely used in many application scenarios. However, the number of model parameters often reaches over 10 billion, which makes it impossible to perform model inference on conventional consumer-grade graphics cards.Therefore, compressing and storing model parameters through quantization has become a mainstream approach, with the key model parameters being the weight matrices of multiple network layers within the model. SpQR is a relatively effective quantization method, but because it retains weight values ​​within 1% (as outliers, or outliers) without quantization, this quantization approach can hinder fast inference. It is understandable that during the inference phase, the input matrix of each network layer is generated layer by layer based on the current inference task. The matrix multiplication result of each network layer's input matrix and the weight matrix of that network layer serves as the output matrix of that network layer, which is then input into the next network layer. The speed of matrix multiplication between the input matrix and the corresponding weight matrix directly affects the efficiency of the inference phase, so accelerating the matrix multiplication speed of the input matrix and the weight matrix is ​​essential. The following diagram illustrates the results of quantizing a weight matrix using the SpQR quantization method, as well as the calculation method used during inference. As shown in Figure 1, assume that the input matrix of a network layer of a neural network model (such as a large language model) is X, and the weight matrix is ​​W. The matrix multiplication of these two matrixes ultimately yields the output matrix out. Assume that the dimension of X is x_row*x_column, and the data format of each element value is fp16 format; assume that the dimension of the weight matrix W is w_row*w_column, and the data format of each element value (i.e., weight value) is fp16 format. It is understandable that to reduce the storage space occupied by the weight matrix, before the inference phase, that is, after the neural network model is trained, the weight matrix of each network layer in the neural network model can be quantized, and the quantized results stored in the video memory space. Therefore, when the product of the input matrix X and the weight matrix W is desired, the input matrix X and the weight matrix W are not directly multiplied, because the weight matrix W is not actually stored, but the quantized result of the weight matrix W is stored. As shown in FIG1 , it is assumed that after SpQR quantization processing is performed on the weight matrix W, the quantization weight matrix W′, the quantization parameter matrix, and the non-quantization weight matrix O shown in the figure are obtained. The quantization parameter matrix includes a zero value matrix Z and a quantization scale matrix S. The quantization scale matrix can also be called a quantization coefficient matrix.The quantized weight matrix W' contains the quantized results of each weight value in the weight matrix W that requires quantization. Before quantization, the data format of these weight values ​​is fp16. After quantization, the data format of the quantized results of these weight values ​​is irrt4 (not limited to this format and can also be other integer formats with a length less than fp16). The unquantized weight matrix O contains each weight value in the weight matrix W that does not require quantization (these values ​​are also called outliers values), so these weight values ​​maintain the fp16 data format. As shown in Figure 1, the original dimensions of the quantized weight matrix W' and the unquantized weight matrix O are: w_row*w_column. However, in practice, the positions corresponding to weight values ​​that do not require quantization in the quantized weight matrix W' can be set to a certain set value, while the positions corresponding to weight values ​​that require quantization in the unquantized weight matrix O can be set to zero. Therefore, the unquantized weight matrix O is actually a sparse matrix with only a small number of non-zero elements. The rectangular bars in the figure represent the multiple non-zero elements in different columns. The determination of which weight values ​​in the weight matrix W require quantization, or which weight values ​​do not, can be determined using existing methods and will not be elaborated upon here. In the process of generating the aforementioned quantized weight matrix W', a quantization parameter matrix, namely, a zero value matrix Z and a quantization scale matrix S, is first calculated to perform quantization based on the quantization parameter matrix. The zero value matrix Z specifies the zero value corresponding to each element in the quantized weight matrix W' during quantization, i.e., the zero value corresponding to each weight value requiring quantization. Similarly, the quantization scale matrix S specifies the quantization scale corresponding to each element in the quantized weight matrix W' during quantization, i.e., the quantization scale corresponding to each weight value requiring quantization. In practical applications, to simplify computational complexity, it is not necessary for each weight value that needs to be quantized to have its own zero value and quantization scale. Instead, the weight values ​​in the weight matrix W can be grouped, and each group shares the same zero value and quantization scale. For example, grouping can be done by row: for example, every 128 rows in the same column form a group.Therefore, a group length can be pre-set: groups i ze. In the above example, its value is 128. Based on this, it can be understood that to reduce the storage space occupied by the quantization parameter matrix, as shown in Figure 1, the dimensions of the zero value matrix Z and the quantization scale matrix S can be: w row / groups i ze, and w_co I umn columns. That is, each group stores only one zero value and quantization scale. During inference calculation, the original dimensions can be restored, that is, w_row*w_co I umn. o For example, assuming w_row = 5120, w_column = 15360, and groups i ze = 128, then w row / groups i ze = 40. Consequently, column i in the zero-value matrix Z contains 40 zero values. The first zero value corresponds to the first 128 elements in column i of the quantization weight matrix VT, the second zero value corresponds to the 128 elements in the second group of column i in the quantization weight matrix W', and so on. During inference, the dimensions are restored to w_row = 5120, w_column = 15360. This means that the first zero value is replicated 128 times to form 128 rows of elements in the first group of column i, and so on. The same applies to the quantization scale matrix S. As shown in Figure 1, the data format of each zero value in the zero value matrix Z is consistent with the data format of each element value in the quantization weight matrix W', which is in nt4 format. The data format of each quantization scale in the quantization scale matrix S is consistent with the original data format of each weight value in the weight matrix W, which is in fp16 format. According to the principle of matrix multiplication, the dimensions of the output matrix are: out_row = x_row, out_co_umn = w_co_umn. Based on the quantization results of the weight matrix W, the calculation process of the output matrix can be expressed as the following formula: <x_column k = 0. The calculation formula for the element value in the i-th row and j-th column of the output matrix out is shown above. Specifically, it is the sum of the first and second calculation results. The first calculation result reflects the multiplication of the quantized weight matrix W' after dequantization by the input matrix X, and the second calculation result reflects the multiplication of the input matrix X by the unquantized weight matrix 0. The dequantization process can be simply expressed as: (W' - Z) * ​​So. For the above-mentioned quantized weight matrix W', zero value matrix Z, and quantization scale matrix S, these matrices are all non-sparse matrices. When storing, the element values ​​in the corresponding matrices can be stored row by row in the storage space. Taking the quantization weight matrix W' as an example, a continuous storage space can be allocated in the video memory. The values ​​of the first row of elements in the quantization weight matrix are sequentially stored in this storage space. Then, the values ​​of the second row of elements are sequentially stored in this storage space, and so on. Based on this, when searching for the value of the element in the i-th row and k-th column of the quantization weight matrix W', the value of the element can be found in this storage space based on the position index of "row number * total number of columns + column number". In other words, the value of the element at the storage location "i*w_count+k" in this storage space is the value of the element in the i-th row and k-th column of the quantization weight matrix W'. The same applies to the zero-value matrix Z and the quantization scale matrix S, and will not be further described. However, since the non-quantization weight matrix O is a sparse matrix, meaning that only a small number of elements are non-zero, if the elements of each row of the non-quantization weight matrix O are stored continuously according to the above-mentioned continuous storage method for non-sparse matrices such as the quantization weight matrix W', a large amount of storage space will be occupied. Therefore, to save storage space, the coordinating format (COO) storage structure is typically used for storing unquantized weight matrices. In the COO storage structure, the non-zero element values ​​in the unquantized weight matrix are stored using a triplet (row, col, data), where row and col represent the row and column numbers of the non-zero element, respectively, and data represents the non-zero element value. While using the COO storage structure can reduce the storage space required for storing unquantized weight matrices, it is not conducive to accelerating the multiplication of the input matrix and the weight matrix during the inference phase of the neural network model. Because the video memory no longer stores all the element values ​​in each row of the unquantized weight matrix, the aforementioned "row number * total number of columns + column number" traversal method can no longer be used to quickly find the element value required to be read from the unquantized weight matrix during the current calculation.Solutions that use acceleration methods such as CUDA (Compute Unified Device Architecture) to accelerate matrix multiplication are all based on the premise that the element value can be found by traversing the storage space based on the element's subscript (i.e., row and column number) using the above traversal method. When storing the non-quantized weight matrix 0 using the COO storage structure, only the subscript indices and values ​​of the non-zero element values ​​are stored. Therefore, when performing matrix multiplication, for example, X is not a problem. ik * 0 kj When calculating [k, j], after finding X[k, j] using the above-described continuous storage method, it is necessary to first query the coo storage structure for the presence of the index item (k, j). If so, the element value corresponding to this index item is further queried. If this index item is not present, it indicates that the calculation is unnecessary, and the search for X[k, j] is useless, which is not conducive to achieving fast calculations. Based on this, the disclosed embodiments provide a new storage structure for storing the unquantized weight matrix [k]. This new storage structure not only reduces the storage space occupied when storing the unquantized weight matrix [k], but also accelerates the multiplication calculation of the input matrix and the weight matrix. The following embodiments illustrate this new storage structure and the matrix multiplication calculation process based on this new storage structure. The methods provided in the following embodiments can be executed by a graphics processing unit (GPU) included in an electronic device. The electronic device can be a terminal device such as a PC, a laptop, a smartphone, or a server such as a cloud server. FIG2 is a flow chart of a method for storing model weights provided by an embodiment of the present disclosure. As shown in FIG2 , the method may include the following steps:

[0002] 201. Obtain a quantization result of a weight matrix of a target network layer in a neural network model, where the quantization result includes a quantized weight matrix, a quantized parameter matrix, and a non-quantized weight matrix, and the non-quantized weight matrix is ​​a sparse matrix.

[0003] 202. Generate multiple storage matrices for storing unquantized weight matrices, the multiple storage matrices including a first storage matrix for storing non-zero element values ​​in the unquantized weight matrix by column, a second storage matrix for storing row numbers corresponding to non-zero element values ​​in the unquantized weight matrix, a third storage matrix for storing the number of non-zero element values ​​in each column of the unquantized weight matrix, and a fourth storage matrix for storing starting index positions corresponding to each column in the unquantized weight matrix in the first storage matrix.

[0004] 203. Storing multiple storage matrices, quantization weight matrices, and quantization parameter matrices. In this embodiment, assuming that the weight matrix of the target network layer is the weight matrix W shown in FIG1 , its quantization results include the quantization weight matrix W', the non-quantization weight matrix O, and the quantization parameter matrix composed of the zero-value matrix Z and the quantization scale matrix S. The non-sparse matrices, quantization weight matrix W', the zero-value matrix Z, and the quantization scale matrix S, are stored using the "continuous storage" method described above. Simply put, three storage spaces are allocated in the video memory for storing these three matrices. The storage addresses of each storage space are continuous, and the elements in each matrix are stored row by row in the corresponding storage space. For the non-quantization weight matrix O, which is a sparse matrix, a new storage structure composed of the four storage matrices described above is used for storage. In practice, these four storage matrices can be four row vectors. It should be noted that because the non-quantization weight matrix O is a sparse matrix, only the non-zero element values ​​are actually involved in the calculation. Specifically, the first storage matrix (which can be expressed as outliers va I) is used to store the non-zero element values ​​in the unquantized weight matrix W by column. These non-zero element values ​​are weight values ​​in the weight matrix W that do not require quantization. In the above example, they are floating-point numbers in the fp16 format. Therefore, the first storage matrix is ​​used to store all non-zero element values ​​(i.e., outliers values) contained in the unquantized weight matrix 0. The storage order is to first traverse column by column and, for each column, store the values ​​row by row in the order of the first row (row1), the second row (row2), etc. The second storage matrix (which can be expressed as outliers row) is used to store the row numbers corresponding to the non-zero element values ​​in the unquantized weight matrix 0. Therefore, it can be seen that the dimension of the second storage matrix is ​​equal to the dimension of the first storage matrix. Assuming that there are M non-zero element values ​​in the unquantized weight matrix 0, the dimension of the first storage matrix can be: 1*M, and the dimension of the second storage matrix is ​​also: 1*M, which is used to record the row number corresponding to each non-zero element value in the first storage matrix in the unquantized weight matrix 0. In the second storage matrix, each row number can be represented using, for example, a uni t32 data format.The third storage matrix (which can be expressed as out I i ers co I umn num) is used to store the number of non-zero element values ​​(as described above, this refers to non-zero element values) in each column of the unquantized weight matrix 0. Since the number of columns of the unquantized weight matrix 0 is the same as the number of columns w_co I umu of the weight matrix W, the dimension of the third storage matrix is: 1 * w_co I umu. The fourth storage matrix (which can be expressed as out I i ers co I umn start) is used to store the starting index position corresponding to each column of the unquantized weight matrix 0 in the first storage matrix. In practice, the starting index position corresponding to the nth column is determined based on the starting index position corresponding to the n-1th column and the number of non-zero element values ​​contained in the n-1th column. The starting index position of the starting column (i.e., the first column with column number 0: column 0) is 0. The position index range of the first storage matrix is ​​[0, M-1], where M is the number of non-zero element values ​​stored in the first storage matrix. After obtaining the four storage matrices corresponding to the unquantized weight matrix 0, since these four storage matrices are non-sparse matrices, they can be stored using a "contiguous storage" method, similar to non-sparse matrices such as the quantized weight matrix W'. Specifically, four storage spaces are allocated in the video memory to store these four storage matrices, each with a consecutive address. The element values ​​contained in a storage matrix are then sequentially stored in the corresponding storage space. To facilitate understanding, the storage contents of the four storage matrices are illustrated below with reference to FIG3 . In Figure 3 , it is assumed that the unquantized weight matrix 0 is a 5*5 matrix, so that its row and column numbers both range from [0, 4]. For ease of description, the position coordinates of the element values ​​in the matrix are expressed as: (rowi, columnj). As shown in Figure 3 , it is assumed that the element value at position (2, 1) is 1.0, the element value at position (3, 4) is 3.0, the element value at position (4, 4) is 2.0, and the element values ​​at all other positions are 0. Based on the above assumptions, the following four storage matrices are generated: The first storage matrix outputs_value = [1.0, 3.0, 2.0] , that is, in the order of column traversal, the non-zero element values ​​in the unquantized weight matrix 0 are stored row by row in each column into the first storage matrix.The second storage matrix outLiers_row=[2, 3, 4], i.e., sequentially stores the row number corresponding to each non-zero element value in the first storage matrix in the unquantized weight matrix 0, where 2 is the row number corresponding to the element value 1.0, 3 is the row number corresponding to the element value 3.0, and 4 is the row number corresponding to the element value 2.0. The third storage matrix outLierscoIumn_num=[0, 1, 0, 0, 2], i.e., stores the number of non-zero element values ​​contained in each column of the unquantized weight matrix 0. The first column (i.e., column 0) contains no non-zero element values, so the corresponding number is 0. The second column (i.e., column 2) contains one non-zero element value (1.0), so the corresponding number is 1. Similarly, the third and fourth columns do not contain any non-zero element values, so the corresponding number is 0. The fifth column (i.e., column 4) contains two non-zero element values ​​(3.0 and 2.0), so the corresponding number is 2. The fourth storage matrix out I i ers co I umn_start = [0, 0, 1, 1, 1], which stores the starting index positions corresponding to each column in the unquantized weight matrix 0 in the first storage matrix. Assuming that the first storage matrix stores M element values, its position index range is [0, M-1]. Based on this, it can be understood that regardless of whether the first column (column 0) in the unquantized weight matrix 0 contains non-zero elements, its corresponding starting index position in the first storage matrix is ​​0. Since the first column (column 0) of the unquantized weight matrix 0 in the above example has no nonzero elements, the starting index of the second column (column 1) in the first storage matrix is ​​0 + 0 = 0, where the first added 0 refers to the starting index of the previous column, and the second added 0 refers to the number of nonzero elements in the previous column. Similarly, the starting index of the third column (column 2) in the first storage matrix is ​​0 + 1 = 1, which is determined by the sum of the starting index of the previous column, 0, and the number of nonzero elements in the previous column, 1.The corresponding starting position index of the fourth column (column 3) in the first storage matrix is: 1 + 0 = 1. The corresponding starting position index of the fifth column (column 4) in the first storage matrix is: 1 + 0 = 1. The above describes a new storage structure for storing the unquantized weight matrix 0, provided in an embodiment of the present disclosure. Based on this storage structure, accelerated calculation of the multiplication of the input matrix and the weight matrix can be achieved. The specific calculation process will be described in subsequent embodiments. As described above, the unquantized weight matrix 0 can be stored using the above-described new storage structure, while the quantized weight matrix W' and the quantization parameter matrix obtained after the weight matrix W is quantized can be stored using the method described above. In an optional embodiment, to further save storage space occupied by the quantization weight matrix W' and the quantization parameter matrix, the quantization weight matrix W' and the quantization parameter matrix may be stored in the following manner: obtaining an initial quantization weight matrix and an initial quantization parameter matrix of the weight matrix of the target network layer, wherein the initial quantization parameter matrix includes an initial zero value matrix, and the initial quantization weight matrix and the initial zero value matrix have the same dimension and element values ​​have the same first data format; grouping the initial quantization weight matrix by row to obtain a grouped quantization weight matrix, wherein element values ​​within a group corresponding to the same column in the initial quantization weight matrix are concatenated into element values ​​in the second data format for storage in the grouped quantization weight matrix; grouping the initial zero value matrix by column to obtain a grouped zero value matrix, wherein element values ​​within a group corresponding to the same row in the initial zero value matrix are concatenated into element values ​​in the second data format for storage in the grouped zero value matrix, and the initial quantization weight matrix and the initial zero value matrix are grouped according to the same group length; and storing the grouped quantization weight matrix and the grouped zero value matrix. For ease of understanding, the above storage method is illustrated with reference to FIG4 . Assume that the initial quantization weight matrix is ​​W1, the initial zero value matrix is ​​Z1, and that their dimensions are w_row*w_column, and their element values ​​are all in int4 format (the first data format). In an optional embodiment, as described above, the initial zero value matrix Z1 can be grouped row-wise according to the set groups i ze. For each row group corresponding to the same column, the zero values ​​are identical, and only one is stored. For details, refer to the relevant schematic diagram in FIG1 .It should be noted that the storage method corresponding to the embodiment shown in FIG4 is applicable both to situations where the initial zero-value matrix Z1 is grouped and stored based on groups i ze, and to situations where the initial zero-value matrix Z1 is not grouped and stored based on groups i ze. FIG4 illustrates the situation where the grouped storage is performed based on groups i ze. In FIG4 , the initial quantization weight matrix W1 and the initial zero-value matrix Z1 are grouped using the same group length of 8. The initial quantization weight matrix W1 is grouped by row to obtain a grouped quantization weight matrix W2. The initial zero-value matrix Z1 is grouped by column to obtain a grouped zero-value matrix Z2. As shown in FIG4 , after this grouping process, the dimension of the grouped quantization weight matrix W2 becomes: w_row / 8*w_column, and the dimension of the grouped quantization weight matrix W2 becomes: w_row / groups i ze*w_column / 8. The element values ​​within a group corresponding to the same column in the initial quantization weight matrix W1 are concatenated into element values ​​in the second data format and stored in the grouped quantization weight matrix W2. For example, the eight element values ​​in the first group (first 8 rows) corresponding to the first column in the initial quantization weight matrix W1 in the int4 format can be combined into a single value in the int32 format (second data format) and stored in the first row and first column of the grouped quantization weight matrix W2. The element values ​​within a group corresponding to the same row in the initial zero value matrix Z1 are concatenated into element values ​​in the second data format and stored in the grouped zero value matrix Z2. For example, the eight element values ​​in the first group (first 8 columns) corresponding to the first row in the initial zero value matrix Z1 in the int4 format can be combined into a single value in the int32 format and stored in the first row and first column of the grouped zero value matrix Z2. The eight element values ​​in the int4 format are combined into an int32 format value. Simply put, these eight element values ​​are sequentially represented as 4-bit binary strings, resulting in a 32-bit binary string. This binary string is then format-converted to obtain the corresponding decimal value. This storage method can further reduce the storage space occupied by the quantized result of the weight matrix.It should be noted that during the inference phase, when multiplication calculations are required between the input matrix and the weight matrix of the target network layer, the grouped quantized weight matrix W2 and the grouped zero-value matrix Z2 need to be loaded from the storage space and restored to the initial quantized weight matrix W1 and the initial zero-value matrix Z1. Furthermore, for example, when the quantization scale matrix S is stored in groups i ze as shown in FIG1 , it also needs to be restored to the initial matrix with dimensions w_row*w_column. The multiplication calculation process between the input matrix and the weight matrix of the target network layer is described below. FIG5 is a flowchart of a model weight calculation method provided in an embodiment of the present disclosure. As shown in FIG5 , the method may include the following steps:

[0005] 501. In response to an input matrix of a target network layer in a neural network model obtained based on an inference task, obtain a quantization weight matrix and a quantization parameter matrix corresponding to the target network layer.

[0006] 502. Obtain multiple storage matrices for storing an unquantized weight matrix corresponding to a target network layer, the multiple storage matrices including a first storage matrix for storing non-zero element values ​​in the unquantized weight matrix by column, a second storage matrix for storing row numbers corresponding to non-zero element values ​​in the unquantized weight matrix, a third storage matrix for storing the number of non-zero element values ​​in each column of the unquantized weight matrix, and a fourth storage matrix for storing starting index positions corresponding to each column in the unquantized weight matrix in the first storage matrix.

[0007] 503. Determine, based on multiple stored matrices, a first calculation result and a second calculation result corresponding to the same position in the output matrix of the target network layer, and obtain a target calculation result at the same position based on the cumulative sum of the first calculation result and the second calculation result. The first calculation result corresponds to the product of the input matrix and the dequantized weight matrix, and the second calculation result corresponds to the product of the input matrix and the unquantized weight matrix. The dequantized weight matrix is ​​obtained by dequantizing the quantized weight matrix based on the quantization parameter matrix. In this embodiment, the multiplication process of the input matrix X of the target network layer and its weight matrix W is used as an example for description. The weight matrix W of the target network layer has been quantized. The quantized weight matrix results include the quantized weight matrix W', the quantization parameter matrix (including the zero-value matrix Z and the quantization scale matrix S mentioned above), and the unquantized weight matrix O. The quantized weight matrix W', the zero-value matrix Z, and the quantization scale matrix S are stored using the continuous storage method described above and have been stored in the corresponding storage space of the video memory. The unquantized weight matrix O, as a sparse matrix, is stored using the new storage structure described in the previous embodiment, and the corresponding multiple storage matrices have been stored in the corresponding storage space of the video memory. The output matrix, which is the calculation result, is denoted as out. Based on the input of the inference task, each network layer in the neural network model will sequentially receive its own input matrix. For example, if the inference task is to predict the word that should be filled in a certain position in an input sentence, the input matrix of a network layer in the neural network model can be the matrix obtained by encoding the input sentence. The output matrix obtained by multiplying this input matrix with the weight matrix of this network layer can serve as the input matrix of the next network layer. Of course, this output matrix can also be processed (such as downsampling) and then used as the input matrix of the next network layer. For example, when the input matrix X of the target network layer is obtained and the multiplication calculation of this input matrix X with the weight matrix W of the target network layer is performed, the multiple computational objects actually calculated with the input matrix X must first be read from the video memory: the quantized weight matrix W', the quantization parameter matrix (zero value matrix Z) and the quantization scale matrix S, as well as multiple storage matrices for storing the non-quantized weight matrix 0. These matrices are then loaded into a pre-allocated target cache space in the video memory. In practical applications, this target cache space can be determined based on the dimensions of the weight matrix of the largest network layer in the neural network model.In reality, the dimensions of the weight matrices corresponding to different network layers may vary, and the target cache space is shared by all network layers. That is, each time a network layer is calculated, the various matrices mentioned above need to be loaded into the target cache space for calculation. Therefore, the target cache space must be at least larger than the dimensions of the largest weight matrix. Furthermore, intermediate and final calculation results (such as the first, second, and target calculation results mentioned above) generated during the calculation process can be temporarily stored in the target cache space. As described above, the calculation formula for the output matrix out is as follows: k <x_column。 k = 0. According to the calculation formula, the output matrix ut has an element value in the i-th row and j-th column, determined by the sum of the first calculation result (the first item added) and the second calculation result (the second item added). Therefore, during the calculation process, the first and second calculation results can be calculated separately and then added to OU. Furthermore, as shown in the above calculation formula, the calculation of the first and second calculation results actually involves the separate calculation of element values ​​at several positions in different matrices, which are ultimately added together. Therefore, parallel multithreading methods such as CUDA can be used to accelerate the calculation. Specifically, for the calculation process of the first calculation result, the calculation positions corresponding to the multiple parallel threads can be first determined. The calculation position corresponding to the target thread includes the first position identifier i, the second position identifier j, and the third position identifier k. The target thread is any of the multiple threads. The meanings of these position identifiers are described in the above calculation formula. The target thread then searches the input matrix X, the quantization weight matrix, and the quantization parameter matrix to obtain the first, second, and third element values ​​based on the first position identifier i, the second position identifier j, and the third position identifier k. The quantization parameter matrix includes the zero-value matrix Z and the quantization scale matrix S, so the third element value includes the third element values ​​found in the zero-value matrix Z and the quantization scale matrix S, respectively. The first element value corresponds to the position in the input matrix X with the first position identifier i as the row number and the third position identifier k as the column number, which is X / . The second element value corresponds to the position in the quantization weight matrix W' with the third position identifier k as the row number and the second position identifier j as the column number, which is W'. kj aThe third element value corresponds to the position of the quantization parameter matrix with the third position identifier k as the row number and the second position identifier j as the column number, which are Zkj and Skj. Subsequently, the target position of the output matrix is ​​updated based on the first element value xj, the second element value vj, and the third element values ​​Zcj and Scj. The first calculation result of the first calculation result is obtained. Specifically, these element values ​​are calculated accordingly according to the above calculation formula to obtain a calculation sub-item that is accumulated in the first calculation result corresponding to the current first position identifier i, second position identifier j, and third position identifier k. Following a similar calculation process based on the above calculation formula, while keeping the first position identifier i and the second position identifier j unchanged, different values ​​of the third position identifier k are traversed to obtain multiple calculation sub-items, which are accumulated into the first calculation result to ultimately obtain the first calculation result. As can be seen from the above formula, the value range of k is determined by the number of columns x_column in the input matrix X. The above describes the process by which a target thread corresponding to the first position identifier i, the second position identifier j, and the third position identifier k calculates and obtains a calculation sub-item that is accumulated in the first calculation result. It is understandable that for the first calculation result corresponding to x% •, the actual number of threads required is x_column, where each thread corresponds to the same first position identifier i and second position identifier j, but each thread corresponds to a different third position identifier k. The following, still using the target thread as an example, describes the process by which the target thread calculates or obtains the corresponding second calculation result, and then describes the thread setting method and the method for determining the position identifier corresponding to the thread. Regarding the calculation process for the second calculation result, the target thread no longer simply reads the corresponding element value from the corresponding matrix according to the position identifier, as when calculating the first calculation result. Instead, the target thread must first determine the fourth element value in the unquantized weight matrix 0 based on the second position identifier j, the third position identifier k, and multiple storage matrices. The fourth element value corresponds to the element value at the target sort position in the column of the unquantized weight matrix 0 with the second position identifier j as the column number. The target sort position is the value corresponding to the third position identifier k. In addition, the target thread determines a fifth element value in the input matrix X. The fifth element value corresponds to a position in the input matrix X having the first position identifier i as a row number and the fourth position identifier as a column number. The fourth position identifier corresponds to the row number of the fourth element value in the unquantized weight matrix 0. Thereafter, the target thread updates the second calculation result of the target position OUttj in the output matrix according to the product of the fourth element value and the fifth element value.Specifically, the process of determining the fourth element value and the fifth element value includes: querying the third storage matrix out I i ers_co I umn_num for the number of non-zero element values ​​contained in the target with the second position identifier j as the column number of the non-quantized weight matrix 0 out I i ers_co I umn_num [j]. o If the number of non-zero element values ​​is greater than the value corresponding to the third position identifier k, then the starting index position corresponding to the target column in the first storage matrix out I i ers_va I is queried in the fourth storage matrix out I i ers_co I umn_start: out I i ers_co I umn_start [ j], that is, if out I i ers_co I umn_num [j] >k, then the starting index position corresponding to the j-th column in the first storage matrix out I i ers_va I is found out I i ers_co I umn_start [ j] o If out li ers_co I umn_num [ j] Wk, then skip, that is, no longer calculate. Query the fourth element value corresponding to the target index position in the first storage matrix out li ers_val l, the target index position is the sum of the values ​​corresponding to the starting index position and the third position marker: out li ers_co I umn_start [ j] +k o So the value of the fourth element is: out I i ers_va I [out I i ers_co I umn_start [ j] + k] o The actual row number corresponding to the fourth element value in the non-quantized weight matrix 0 is: out I i ers_row [out I i ers_co I umn_start [j] + k], that is, the actual row number corresponding to the fourth element value in the non-quantized weight matrix 0 is found in the second storage matrix out I i ers_row. In other words, the fourth element value is the non-zero element value -.0 in the jth row and jth column of the non-quantized weight matrix 0. rowja According to the principle of matrix multiplication, the fifth element value in the input matrix X to be multiplied should be the element value of the i-th row and the i-th column in the input matrix X: iOrow Based on this, the fourth position is marked as Or. *. Then, X iOrow * 0o rowjThe product of is used as a calculation sub-item in the second calculation result of OU% • and is superimposed on 014% •. From the above calculation process, it can be seen that in the embodiment of the present disclosure, the calculation method for the input matrix X and the non-quantized weight matrix 0 is essentially to traverse the non-quantized weight matrix according to the input matrix X. 0 does not store the values ​​of each row element in a continuous storage manner, so this traversal cannot be implemented. In the embodiment of the present disclosure, the calculation idea of ​​traversing the input matrix X according to the position of the non-zero element value in the non-quantized weight matrix 0 is adopted. The non-zero element values ​​contained in the non-quantized weight matrix 0 and the row number and column number corresponding to each non-zero element value in the non-quantized weight matrix 0 are stored using the storage structure of the above-mentioned multiple storage matrices. When performing calculations, the non-zero element values ​​required for calculation by the current thread can be queried according to the multiple storage matrices according to the above-mentioned traversal method, and the element values ​​that need to be searched in the input matrix are determined accordingly, thereby realizing accelerated calculation based on multi-threading. In this way, for the non-zero element values ​​of the jth column in the non-quantized weight matrix 0, it is not necessary to correspond to the calculation dimension of the quantized weight matrix in this matrix multiplication, thereby realizing the memory access of the complex non-zero element values ​​of 0(1), thereby improving the reasoning speed. The following introduces the setting of multi-threading and the method for determining the position identifier corresponding to the thread. Specifically, based on the number of rows (out_row) and columns (out_column) of the output matrix out, and the number of columns (x_column) of the input matrix X, the number of threads is determined as: out_row * out_column * x_column. This yields the total number of threads required. In practical applications, the upper limit of thread numbers supported by GPUs is limited. For example, some GPUs limit thread numbers to 1024. If the total number of threads exceeds this upper limit, the concepts of thread blocks and thread grids are necessary. In large-scale neural network models (such as large language models), the dimensions of the weight matrix are often very large, exceeding these limits. Therefore, the thread block dimensions can be set first, and then the thread grid dimensions can be determined based on the set thread block dimensions. The thread block dimensions determine the number of threads contained in each thread block, while the thread grid dimensions determine the number of thread blocks contained in the thread grid. Among them, the dimension of the thread block can be three-dimensional: b I lockd i m. x, b I lockd i m. y, b I lockd i m. z, and the dimension of the thread grid can be three-dimensional: griddim.x, griddim.y, griddim.z.Based on the above three dimensions, the number of threads that can be contained in a thread block is: b I lockd i m. x*b I lockd i m. y* b I lockd im. zo Similarly, the number of thread blocks that can be contained in a thread grid is: gr i dd i m. x*gr i dd i m. y* gr i dd i m. z. o In practical applications, you can first set the values ​​of blockdim.x, blockdim.y, and blockdim.z, for example, to (32, 1, 1). Based on this setting and the total number of threads, you can determine the number of thread blocks required and thus determine the values ​​of griddim.x, griddim.y, and griddim.z. Based on these thread settings, the process of determining the computational positions corresponding to multiple threads includes: determining the identifiers corresponding to the multiple threads based on the dimensions of the thread grid, the dimensions of the thread blocks, the relative positions of each thread block in the thread grid, and the relative positions of each thread within its corresponding thread block; and then determining the computational positions corresponding to the multiple threads based on the identifiers. The following describes this calculation process. Assuming the total number of threads is N, then N= gr i dd i m. x*gr i dd i m. y* gr i dd i m. z* b I ockd i m. x*b I ockd i m. y* b I ockd i m. z o In the following calculation process, block l dx.x, block l dx.y, and b I lock I dx.z represent the relative position of the current thread block in the thread grid to which it belongs, and thread I dx.x, thread I dx.y, and thread I dx.z represent the relative position of the current thread in the thread block to which it belongs. Determining each thread's identity involves arranging all threads into a sequence, numbered 0, 1, 2, ..., N, and finding the current thread's sequence number within it. This sequence number serves as the thread's identity. First, determine which thread block the current thread is in the thread grid: block id b I lock I d ​​= b I lock I dx.x + b I lock I dx.y*grid i m.x + b I lock I dx.z*grid i m.x*grid i m.y oAfter that, find out which thread in the thread block the current thread is in: thread I d thread I d=thread I dx. x+thread I dx. y*b I ockd i m. x+thread I dx. z*b I ockd i m. x* b I ockd i m. y o Then, calculate how many threads there are in a thread block: M

[0008] M=b I ockd i m. x*b I ockd i m. y*b I ockd i m. z o Finally, the serial number of the current thread is obtained: i dx i dx=thread I d+M*b I ock I d oThis yields a thread identifier, which is equivalent to expanding all threads into a one-dimensional space and the corresponding position of each thread in that one-dimensional space. The aforementioned (i, j, k) represents a three-dimensional space, and according to the predefined mapping formula, the three position identifiers corresponding to each thread in that three-dimensional space can be determined: (i, j, k). The calculation of the thread identifier and the corresponding three-dimensional position coordinates can be implemented with reference to existing related technologies and will not be elaborated upon here. The following describes in detail the model weight calculation processing and model weight storage device of one or more embodiments of the present disclosure. Those skilled in the art will appreciate that these devices can be constructed using commercially available hardware components configured according to the steps taught in this solution. Figure 6 is a schematic diagram of the structure of a model weight calculation processing device provided in an embodiment of the present disclosure. As shown in Figure 6, the device includes: an acquisition module 11 and an inference module 12. An acquisition module 11 is configured to, in response to an input matrix of a target network layer in a neural network model obtained based on an inference task, acquire a quantization weight matrix and a quantization parameter matrix corresponding to the target network layer, and acquire multiple storage matrices for storing a non-quantization weight matrix corresponding to the target network layer, wherein the multiple storage matrices include a first storage matrix for storing non-zero element values ​​in the non-quantization weight matrix by column, a second storage matrix for storing row numbers corresponding to non-zero element values ​​in the non-quantization weight matrix, a third storage matrix for storing the number of non-zero element values ​​in each column of the non-quantization weight matrix, and a fourth storage matrix for storing starting index positions corresponding to each column of the non-quantization weight matrix in the first storage matrix; wherein the non-quantization weight matrix is ​​a sparse matrix, and a quantization result of the weight matrix of the target network layer includes the quantization weight matrix, the quantization parameter matrix, and the non-quantization weight matrix. The inference module 12 is configured to determine, based on the multiple stored matrices, a first calculation result and a second calculation result corresponding to the same position in the output matrix of the target network layer, and obtain a target calculation result for the same position based on the cumulative sum of the first calculation result and the second calculation result. The first calculation result corresponds to the product of the input matrix and a dequantized weight matrix, and the second calculation result corresponds to the product of the input matrix and the unquantized weight matrix. The dequantized weight matrix is ​​obtained by dequantizing the quantized weight matrix based on the quantization parameter matrix. The apparatus shown in FIG6 can execute the steps in the aforementioned embodiment. For detailed execution procedures and technical effects, refer to the description of the aforementioned embodiment and are not further elaborated here.FIG7 is a schematic diagram of the structure of a model weight storage device provided by an embodiment of the present disclosure. As shown in FIG7 , the device includes: an acquisition module 21, a generation module 22, and a storage module 23. The acquisition module 21 is configured to acquire the quantization results of the weight matrix of a target network layer in a neural network model. The quantization results include a quantized weight matrix, a quantization parameter matrix, and a non-quantized weight matrix, where the non-quantized weight matrix is ​​a sparse matrix. The generation module 22 is configured to generate multiple storage matrices for storing the non-quantized weight matrix. The multiple storage matrices include a first storage matrix for storing the non-zero element values ​​in the non-quantized weight matrix by column, a second storage matrix for storing the row number corresponding to each non-zero element value in the non-quantized weight matrix in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values ​​in each column of the non-quantized weight matrix, and a fourth storage matrix for storing the starting index position corresponding to each column in the non-quantized weight matrix in the first storage matrix. The storage module 23 is configured to store the multiple storage matrices, the quantized weight matrix, and the quantization parameter matrix. The apparatus shown in FIG7 can execute the relevant steps in the aforementioned embodiments. The detailed execution process and technical effects are described in the aforementioned embodiments and are not further elaborated here. In one possible design, the apparatus structures shown in FIG6 and FIG7 can be implemented as an electronic device. As shown in FIG8 , the electronic device may include: a processor 31, a memory 32, and a communication interface 33. Memory 32 stores executable code. When executed by processor 31, processor 31 can at least implement the model weight calculation method or model weight storage method provided in the aforementioned embodiments. Furthermore, embodiments of the present disclosure provide a non-transitory machine-readable storage medium storing executable code. When executed by a processor of an electronic device, the processor can at least implement the model weight calculation method or model weight storage method provided in the aforementioned embodiments. The apparatus embodiments described above are merely illustrative. The network elements described as separate components may or may not be physically separate. Some or all of these modules can be selected to achieve the objectives of the present embodiments according to actual needs. Those skilled in the art can understand and implement the present invention without inventive effort. Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general-purpose hardware platform, or can also be implemented by combining hardware and software.Based on this understanding, the essence of the above-mentioned technical solutions, or the portion that contributes to the prior art, can be embodied in the form of a computer product. The present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM (Compact Disk Read-Only Memory), optical storage, etc.) containing computer-usable program code. The embodiments of the present disclosure provide a computer program product that includes a computer program. When executed by a processor of an electronic device, the computer program product enables the processor to at least implement the model weight calculation and processing method or the model weight storage method provided in the aforementioned embodiments. Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present disclosure and are not intended to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may be modified, or some of the technical features thereof may be replaced by equivalents. Such modifications or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Claims

Claims 1. A method for calculating and processing model weights, wherein, Including: In response to the input of the input matrix of the target network layer in the neural network model obtained based on the inference task, obtaining the quantization weight matrix and quantization parameter matrix corresponding to the target network layer; Obtaining a plurality of storage matrices for storing the non-quantized weight matrix corresponding to the target network layer, where the plurality of storage matrices include a first storage matrix for storing each non-zero element value in the non-quantized weight matrix column by column, a second storage matrix for storing the row numbers corresponding to each non-zero element value in the non-quantized weight matrix in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values in each column of the non-quantized weight matrix, and a fourth storage matrix for storing the starting index position corresponding to each column of the non-quantized weight matrix in the first storage matrix; wherein, the non-quantized weight matrix is a sparse matrix, and the quantization result of the weight matrix of the target network layer includes the quantization weight matrix, the quantization parameter matrix, and the non-quantized weight matrix; determining the first calculation result and the second calculation result corresponding to the same position in the output matrix of the target network layer according to the plurality of storage matrices, so as to obtain the target calculation result of the same position according to the sum of the first calculation result and the second calculation result, wherein, the first calculation result corresponds to the product of the input matrix and the dequantized weight matrix, the second calculation result corresponds to the product of the input matrix and the non-quantized weight matrix, and the dequantized weight matrix is obtained by dequantizing the quantization weight matrix based on the quantization parameter matrix.

2. The method according to claim 1, wherein The step of obtaining the output matrix of the target network layer includes: determining the calculation positions corresponding to each of the plurality of parallel threads, where the calculation position corresponding to the target thread includes a first position identifier, a second position identifier, and a third position identifier, and the target thread is any one of the plurality of threads; querying the first element value, the second element value, and the third element value in the input matrix, the quantization weight matrix, and the quantization parameter matrix according to the first position identifier, the second position identifier, and the third position identifier, wherein, the first element value corresponds to the position of the input matrix with the first position identifier as the row number and the third position identifier as the column number, the second element value corresponds to the position of the quantization weight matrix with the third position identifier as the row number and the second position identifier as the column number, and the third element value corresponds to the position of the quantization parameter matrix with the third position identifier as the row number and the second position identifier as the column number; updating the first calculation result of the target position of the output matrix according to the first element value, the second element value, and the third element value, and the target position corresponds to the position of the output matrix with the first position identifier as the row number and the second position identifier as the column number.

3. The method according to claim 2, wherein The steps for obtaining the output matrix of the target network layer include: determining a fourth element value in the unquantized weight matrix according to the second position identifier, the third position identifier, and the multiple storage matrices, where the fourth element value corresponds to the element value at the target sorting position in a column of the unquantized weight matrix with the second position identifier as the column number, and the target sorting position is the value corresponding to the third position identifier; determining a fifth element value in the input matrix, where the fifth element value corresponds to the position in the input matrix with the first position identifier as the row number and the fourth position identifier as the column number, and the fourth position identifier corresponds to the row number of the fourth element value in the unquantized weight matrix; updating the second calculation result at the target position of the output matrix according to the product of the fourth element value and the fifth element value.

4. The method according to claim 3, wherein The step of determining the fourth element value in the unquantized weight matrix according to the second position identifier, the third position identifier, and the multiple storage matrices includes: querying the number of non-zero element values included in the target column of the unquantized weight matrix with the second position identifier as the column number in the third storage matrix; if the number of non-zero element values is greater than the value corresponding to the third position identifier, querying the starting index position corresponding to the target column in the first storage matrix in the fourth storage matrix; querying the fourth element value corresponding to the target index position in the first storage matrix, where the target index position is the sum of the starting index position and the value corresponding to the third position identifier; the step of determining the fifth element value in the input matrix includes: querying the row number corresponding to the target index position in the second storage matrix, and determining the row number corresponding to the target index position as the fourth position identifier; determining the fifth element value at the position in the input matrix with the first position identifier as the row number and the fourth position identifier as the column number.

5. The method according to claim 2, wherein The method further includes: determining the number of the multiple threads according to the number of rows and columns of the output matrix and the number of columns of the input matrix; determining the dimension of the thread grid to be set according to the set dimension of the thread block, where the dimension of the thread block determines the number of threads included in each thread block, and the dimension of the thread grid determines the number of thread blocks included in the thread grid; the step of determining the respective calculation positions of the multiple parallel threads includes: determining the respective identifiers of the multiple threads according to the dimension of the thread grid, the dimension of the thread block, the relative positions of the thread blocks in the thread grid, and the relative positions of the threads in their respective thread blocks; determining the respective calculation positions of the multiple threads according to the respective identifiers of the multiple threads.

6. The method according to claim 1, wherein The method further includes: in response to the input of the input matrix, loading the input matrix, the quantization weight matrix, the quantization parameter matrix, and the multiple storage matrices into a target cache space to calculate a first calculation result, the second calculation result, and the target calculation result in the target cache space.

7. The method according to any one of claims 1 to 6, wherein The method further includes: obtaining an initial quantization weight matrix and an initial quantization parameter matrix of the weight matrix of the target network layer, where the initial quantization parameter matrix includes an initial zero matrix, the initial quantization weight matrix and the initial zero matrix have the same dimension and the element values have the same first data format; grouping the initial quantization weight matrix by rows to obtain a grouped quantization weight matrix, where the element values in the same column of the initial quantization weight matrix are concatenated into an element value in a second data format for storage in the grouped quantization weight matrix; grouping the initial zero matrix by columns to obtain a grouped zero matrix, where the element values in the same row of the initial zero matrix are concatenated into the element value in the second data format for storage in the grouped zero matrix, and the initial quantization weight matrix and the initial zero matrix are grouped according to the same grouping length; storing the grouped quantization weight matrix and the grouped zero matrix.

8. The method according to claim 7, wherein The obtaining the quantization weight matrix and the quantization parameter matrix corresponding to the target network layer includes: loading the grouped quantization weight matrix and the grouped zero matrix; restoring the grouped quantization weight matrix and the grouped zero matrix to the initial quantization weight matrix and the initial zero matrix.

9. A method for storing model weights, wherein, including obtaining a quantization result of the weight matrix of the target network layer in a neural network model, where the quantization result includes a quantization weight matrix, a quantization parameter matrix, and a non-quantized weight matrix, and the non-quantized weight matrix is a sparse matrix; generating multiple storage matrices for storing the non-quantized weight matrix, where the multiple storage matrices include a first storage matrix for storing each non-zero element value in the non-quantized weight matrix by column, a second storage matrix for storing the row numbers corresponding to each non-zero element value in the non-quantized weight matrix in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values in each column of the non-quantized weight matrix, and a fourth storage matrix for storing the starting index position corresponding to each column of the non-quantized weight matrix in the first storage matrix; storing the multiple storage matrices, the quantization weight matrix, and the quantization parameter matrix.

10. An electronic device, wherein, including: a memory, a processor, and a communication interface; where the memory stores executable code, and when the executable code is executed by the processor, the processor executes the model weight calculation processing method according to any one of claims 1 to 8.

11. A non-transitory machine-readable storage medium, wherein, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to execute the model weight calculation processing method according to any one of claims 1 to 8.

12. A computer program product, wherein, Comprising: A computer program that, when executed by a processor of an electronic device, causes the processor to execute the model weight calculation processing method according to any one of claims 1 to 8.

Images