Model weight storage method and device, model weight calculation processing method and device, storage medium and program

Abstract

The embodiment of the invention provides a model weight storage and calculation processing method and device, a storage medium and a program, and the method comprises the steps: obtaining a quantization weight matrix, a quantization parameter matrix and a plurality of storage matrixes for storing non-quantization weight matrixes of a target network layer, comprising a first storage matrix for storing each non-zero element value in the non-quantized weight matrix according to columns, a second storage matrix for storing the line number of each non-zero element value in the non-quantized weight matrix, and a third storage matrix for storing the number of the non-zero element values in each column in the non-quantized weight matrix, and the fourth storage matrix is used for storing the corresponding initial index position of each column in the non-quantization weight matrix in the first storage matrix. A first calculation result and a second calculation result at the same position in the output matrix are determined according to the multiple storage matrixes, the first calculation result is the product of the input matrix and the inverse quantization weight matrix, and the second calculation result is the product of the input matrix and the non-quantization weight matrix. Through the scheme, accelerated calculation of matrix multiplication can be realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular, to a method and device for storing and calculating model weights, a storage medium, and a program. Background Art

[0002] With the continuous development of artificial intelligence technology, neural network models with larger weight value scales emerge in an endless stream, such as large language models. A neural network model includes several network layers (such as convolutional layers, fully connected layers, etc.), and different network layers correspond to different weight matrices. The weight values in the weight matrices of each network layer obtained after the neural network model is trained generally adopt a certain floating-point data format, such as the fp16 format. Storing the weight matrix in this data format will occupy a larger storage space.

[0003] To save storage space, quantization processing is usually required for the weight matrices of each network layer of the neural network model. For example, a sparse quantization representation for near-lossless LLM weight compression (SpQR) quantization method is used for quantization processing. Taking any network layer in a certain neural network model as an example, the quantization result of the weight matrix of this network layer includes a quantized weight matrix, a quantization parameter matrix, and a non-quantized weight matrix. The quantization parameter matrix includes a zero matrix and a quantization scale matrix. Among them, the quantized weight matrix is composed of the quantization results of the weight values that need to be quantized in the weight matrix, and the quantization process is implemented using the above quantization parameter matrix, while the non-quantized weight matrix is composed of the weight values that do not need to be quantized in the weight matrix. The proportion of weight values that do not need to be quantized is very small (usually less than 1%), so the non-quantized weight matrix is a sparse matrix. The data formats used for the element values in the quantized weight matrix, the quantization parameter matrix, and the non-quantized weight matrix are different. For example, the element values in the quantized weight matrix are in int4 format, the element values in the non-quantized weight matrix are in fp16 format, the zeros given in the zero matrix are in int4 format, and the quantization scales given in the quantization scale matrix are in fp16 format.

[0004] For the above-mentioned quantized weight matrix and quantized parameter matrix of the non-sparse matrix, the element values in the corresponding matrix can be stored row by row in the storage space in the order of rows. Since the non-quantized weight matrix is a sparse matrix, that is, only a small number of element values are non-zero among them. If the rows of elements in the non-quantized weight matrix are continuously stored in the continuous storage manner of the quantized weight matrix and the quantized parameter matrix, a lot of storage space will be occupied. Therefore, in order to save storage space, for the non-quantized weight matrix, a coo (coordinate format) storage structure is usually adopted for storage. In the coo storage structure, a triple (row, col, data) is used to store the non-zero elements in the non-quantized weight matrix, where row and col are the row number and column number of the non-zero element respectively, and data is the value of the non-zero element.

[0005] Although adopting the coo storage structure can reduce the storage space occupied when storing the non-quantized weight matrix, it is not conducive to accelerating the calculation of matrix multiplication in the inference stage of the neural network model. The matrix multiplication refers to the multiplication calculation between the input matrix of any network layer in the neural network model and the weight matrix of this network layer. Summary of the Invention

[0006] Embodiments of the present invention provide a method, device, storage medium and program for storing and calculating model weights to improve the calculation efficiency of matrix multiplication.

[0007] In a first aspect, embodiments of the present invention provide a method for calculating and processing model weights, the method includes:

[0008] In response to the input of the input matrix of the target network layer in the neural network model obtained based on the inference task, obtain the corresponding quantized weight matrix and quantized parameter matrix of the target network layer;

[0009] Obtain a plurality of storage matrices for storing the non-quantized weight matrix corresponding to the target network layer. The plurality of storage matrices include a first storage matrix for storing the non-zero element values in each column of the non-quantized weight matrix, a second storage matrix for storing the row numbers corresponding to the non-zero element values in the non-quantized weight matrix in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values in each column of the non-quantized weight matrix, and a fourth storage matrix for storing the starting index positions corresponding to each column of the non-quantized weight matrix in the first storage matrix; wherein, the non-quantized weight matrix is a sparse matrix, and the quantization result of the weight matrix of the target network layer includes the quantized weight matrix, the quantized parameter matrix and the non-quantized weight matrix;

[0010] Determine a first calculation result and a second calculation result corresponding to the same position in the output matrix of the target network layer according to the multiple storage matrices, so as to obtain a target calculation result at the same position according to the sum of the first calculation result and the second calculation result, where the first calculation result corresponds to the product of the input matrix and the dequantized weight matrix, the second calculation result corresponds to the product of the input matrix and the non-quantized weight matrix, and the dequantized weight matrix is obtained by dequantizing the quantized weight matrix based on the quantization parameter matrix.

[0011] In a second aspect, an embodiment of the present invention provides a model weight calculation and processing device, and the device includes:

[0012] An acquisition module, configured to, in response to an input of an input matrix of a target network layer in a neural network model obtained based on an inference task, acquire the quantized weight matrix and the quantization parameter matrix corresponding to the target network layer, and acquire multiple storage matrices for storing the non-quantized weight matrix corresponding to the target network layer. The multiple storage matrices include a first storage matrix for storing non-zero element values in each column of the non-quantized weight matrix by column, a second storage matrix for storing the row numbers corresponding to the non-zero element values in the non-quantized weight matrix in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values in each column of the non-quantized weight matrix, and a fourth storage matrix for storing the starting index positions corresponding to each column of the non-quantized weight matrix in the first storage matrix; wherein the non-quantized weight matrix is a sparse matrix, and the quantization result of the weight matrix of the target network layer includes the quantized weight matrix, the quantization parameter matrix, and the non-quantized weight matrix;

[0013] An inference module, configured to determine a first calculation result and a second calculation result corresponding to the same position in the output matrix of the target network layer according to the multiple storage matrices, so as to obtain a target calculation result at the same position according to the sum of the first calculation result and the second calculation result, where the first calculation result corresponds to the product of the input matrix and the dequantized weight matrix, the second calculation result corresponds to the product of the input matrix and the non-quantized weight matrix, and the dequantized weight matrix is obtained by dequantizing the quantized weight matrix based on the quantization parameter matrix.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a communication interface; wherein, an executable code is stored on the memory, and when the executable code is executed by the processor, the processor can at least implement the model weight calculation and processing method as described in the first aspect.

[0015] Fourthly, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the model weight calculation processing method as described in the first aspect.

[0016] In addition, an embodiment of the present invention provides a computer program product, which contains a computer program. When the computer program is executed by a processor of an electronic device, the processor can at least implement the model weight calculation processing method as described in the first aspect.

[0017] Fifthly, an embodiment of the present invention provides a model weight storage method, including:

[0018] Obtaining a quantization result of a weight matrix of a target network layer in a neural network model, where the quantization result includes a quantized weight matrix, a quantization parameter matrix, and a non-quantized weight matrix, and the non-quantized weight matrix is a sparse matrix;

[0019] Generating a plurality of storage matrices for storing the non-quantized weight matrix, where the plurality of storage matrices include a first storage matrix for storing non-zero element values in each column of the non-quantized weight matrix, a second storage matrix for storing row numbers corresponding to the non-zero element values in the non-quantized weight matrix in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values in each column of the non-quantized weight matrix, and a fourth storage matrix for storing the starting index position corresponding to each column of the non-quantized weight matrix in the first storage matrix;

[0020] Storing the plurality of storage matrices, the quantized weight matrix, and the quantization parameter matrix.

[0021] Sixthly, an embodiment of the present invention provides a model weight storage device, including:

[0022] An obtaining module, configured to obtain a quantization result of a weight matrix of a target network layer in a neural network model, where the quantization result includes a quantized weight matrix, a quantization parameter matrix, and a non-quantized weight matrix, and the non-quantized weight matrix is a sparse matrix;

[0023] A generation module, configured to generate a plurality of storage matrices for storing the non-quantized weight matrix, where the plurality of storage matrices include a first storage matrix for storing each non-zero element value in the non-quantized weight matrix column by column, a second storage matrix for storing the row numbers corresponding to each non-zero element value in the non-quantized weight matrix in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values in each column of the non-quantized weight matrix, and a fourth storage matrix for storing the starting index positions corresponding to each column of the non-quantized weight matrix in the first storage matrix;

[0024] A storage module, configured to store the plurality of storage matrices, the quantized weight matrix, and the quantized parameter matrix.

[0025] In a seventh aspect, an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a communication interface; wherein, an executable code is stored on the memory, and when the executable code is executed by the processor, the processor can at least implement the model weight storage method as described in the fifth aspect.

[0026] In an eighth aspect, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which an executable code is stored, and when the executable code is executed by a processor of an electronic device, the processor can at least implement the model weight storage method as described in the fifth aspect.

[0027] In a ninth aspect, an embodiment of the present invention provides a computer program product, which includes a computer program, and when the computer program is executed by a processor of an electronic device, the processor can at least implement the model weight storage method as described in the fifth aspect.

[0028] The model weight calculation and processing solution provided by the embodiments of the present invention first performs quantization processing on the weight matrices of each network layer of the neural network model to save storage space. For any target network layer, the quantization result of the weight matrix of the target network layer includes a quantized weight matrix, a quantization parameter matrix, and a non-quantized weight matrix. The non-quantized weight matrix is a sparse matrix. To save storage space and ensure the matrix multiplication calculation speed between the input matrix of the target network layer and this weight matrix, a new storage structure for storing the non-quantized weight matrix is provided. Specifically, multiple storage matrices for storing the non-quantized weight matrix are generated, including a first storage matrix for storing each non-zero element value in the non-quantized weight matrix column by column, a second storage matrix for storing the row numbers corresponding to each non-zero element value in the non-quantized weight matrix in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values in each column of the non-quantized weight matrix, and a fourth storage matrix for storing the starting index positions corresponding to each column in the first storage matrix in the non-quantized weight matrix. It can be seen that through the above 4 storage matrices, the continuous storage of each non-zero element value and the position coordinates (row number, column number) of each non-zero element value in the non-quantized weight matrix in different storage matrices is realized, and these 4 storage matrices can be stored continuously in the storage space normally like the quantized weight matrix.

[0029] When it is necessary to calculate the product of the input matrix of the target network layer and the weight matrix, the first calculation result and the second calculation result corresponding to the same position in the output matrix as the product result are determined according to the above multiple storage matrices, so as to obtain the target calculation result at the same position based on the sum of the first calculation result and the second calculation result. Among them, the first calculation result corresponds to the product of the input matrix and the dequantized weight matrix, and the dequantized weight matrix is obtained by performing dequantization processing on the quantized weight matrix based on the quantization parameter matrix. The second calculation result corresponds to the product of the input matrix and the non-quantized weight matrix. Among them, in the calculation process of the second calculation result, the accelerated multiplication calculation of the input matrix and the non-quantized weight matrix can be realized based on the above 4 storage matrices, thereby improving the multiplication calculation efficiency of the input matrix and the weight matrix of the target network layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0031] Figure 1 It is a schematic diagram of the principle of a traditional matrix multiplication calculation based on the quantization result of the weight matrix provided by the embodiments of the present invention;

[0032] Figure 2 Flow chart of a model weight storage method provided by an embodiment of the present invention;

[0033] Figure 3 Schematic diagram of a new storage structure for storing a non-quantized weight matrix provided by an embodiment of the present invention;

[0034] Figure 4 Schematic diagram of a storage method for quantization results of a weight matrix provided by an embodiment of the present invention;

[0035] Figure 5 Flow chart of a model weight calculation and processing method provided by an embodiment of the present invention;

[0036] Figure 6 Schematic diagram of the structure of a model weight calculation and processing device provided by an embodiment of the present invention;

[0037] Figure 7 Schematic diagram of the structure of a model weight storage device provided by an embodiment of the present invention;

[0038] Figure 8 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0039] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0040] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties. And the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances are provided for the user to select authorization or rejection.

[0041] The following will describe in detail some embodiments of the present invention with reference to the accompanying drawings. Without conflict between the embodiments, the embodiments and the features in the embodiments can be combined with each other. In addition, the step time sequences in the following method embodiments are only examples and are not strictly limited.

[0042] With the booming development of large language models, they are widely used in many application scenarios. However, the number of parameters in the model often reaches more than tens of billions, making it impossible for a single conventional consumer-grade graphics card to perform model inference. Therefore, compressing and storing model parameters through quantization methods has become a mainstream approach, and the main model parameters are the weight matrices of multiple network layers in the model. SpQR is a relatively good quantization method, but since this method retains less than 1% of the weight values (as outliers) without quantization, this quantization method will result in slow inference speed.

[0043] It can be understood that during the inference stage, the input matrix of each network layer is generated layer by layer based on the current inference task. The matrix multiplication result of the input matrix of each network layer and the weight matrix of that network layer serves as the output matrix of that network layer, and then is input into the next network layer. The matrix multiplication calculation speed between the input matrix and the corresponding weight matrix directly affects the efficiency of the inference stage. Therefore, it is necessary to accelerate the matrix multiplication calculation speed between the input matrix and the weight matrix.

[0044] First, the following Figure 1 shows the results of quantifying a weight matrix using the SpQR quantization method and the calculation method during inference.

[0045] As Figure 1 shown, assume that the input matrix of a certain network layer of a neural network model (such as a large language model) is X, the weight matrix is W, and the output matrix finally obtained by multiplying the two matrices is out. Among them, assume that the dimension of X is: x_row * x_column, and the data format of each element value is in fp16 format; assume that the dimension of the weight matrix W is: w_row * w_column, and the data format of each element value (i.e., the weight value) is in fp16 format.

[0046] It can be understood that in order to reduce the storage space occupied by the weight matrix, before the inference stage, that is, for example, after the neural network model is trained, the weight matrices of each network layer in the neural network model can be quantized respectively, and the quantization results are stored in the video memory space. Therefore, when wanting to obtain the product of the input matrix X and the weight matrix W, it is not directly to multiply the input matrix X and the weight matrix W, because the weight matrix W is not actually stored, but the quantization result of the weight matrix W is stored.

[0047] As Figure 1 shown, assume that after performing SpQR quantization processing on the weight matrix W, the quantized weight matrix W’, the quantization parameter matrix, and the non-quantized weight matrix O as shown in the figure are obtained. Among them, the quantization parameter matrix includes the zero matrix Z and the quantization scale matrix S. The quantization scale matrix can also be called the quantization coefficient matrix.

[0048] Among them, the quantized weight matrix W' contains the quantization results of each weight value in the weight matrix W that needs to be quantized. Before quantization, the data format of these weight values is fp16. After quantization, the data format of the quantization results of these weight values is int4 (not limited to this, it can also be other integer formats with a length less than fp16). The non-quantized weight matrix O contains each weight value in the weight matrix W that does not need to be quantized (these values are also called outliers, i.e., outlier values), so that these weight values maintain the fp16 data format. As Figure 1 shown, the original dimensions of the quantized weight matrix W' and the non-quantized weight matrix O are: w_row * w_column. However, in fact, the positions corresponding to the weight values that do not need to be quantized in the quantized weight matrix W' can be set to a certain set value, and the positions corresponding to the weight values that need to be quantized in the non-quantized weight matrix O can be set to zero. Thus, it can be seen that the non-quantized weight matrix O is actually a sparse matrix with only a small number of non-zero element values. The rectangular bars shown in the figure are the multiple non-zero element values existing in different columns.

[0049] And which positions of the weight values in the weight matrix W need to be quantized, that is, which positions of the weight values do not need to be quantized, can be determined by existing related methods, which will not be elaborated here.

[0050] In the process of generating the above-mentioned quantized weight matrix W', a quantization parameter matrix will be calculated first, that is, the zero matrix Z and the quantization scale matrix S, so as to complete the quantization process based on the quantization parameter matrix. Among them, the zero matrix Z gives the zero value (zero) corresponding to each element value in the quantized weight matrix W' during the quantization process, that is, the zero value corresponding to each weight value that needs to be quantized; similarly, the quantization scale matrix S gives the quantization scale (scale) corresponding to each element value in the quantized weight matrix W' during the quantization process, that is, the quantization scale corresponding to each weight value that needs to be quantized.

[0051] In practical applications, to simplify the computational complexity, it is not necessary for each weight value that needs to be quantized to independently correspond to its own zero value and quantization scale. Instead, the weight values in the weight matrix W can be grouped, and each group shares the same zero value and quantization scale. For example, grouping by row: for example, every 128 rows in the same column are grouped into one group. So a group length: groupsize can be preset in advance. In the above example, its value is 128.

[0052] Based on this, it can be understood that in order to reduce the storage space occupied by the quantization parameter matrix, as Figure 1As shown, the dimensions of the actually stored zero matrix Z and quantization scale matrix S can be: the number of rows is w_row / groupsize, and the number of columns is w_column, that is, only one zero value and quantization scale are stored in each group. During inference calculation, the original dimensions can be restored, that is, w_row*w_column.

[0053] For example, assume w_row = 5120, w_column = 15360, and groupsize = 128. Then w_row / groupsize = 40. Thus, there are 40 zero values in column i of the zero matrix Z. The first zero value corresponds to the first 128 elements in column i of the quantized weight matrix W'. The second zero value corresponds to the 128 elements within the second group in column i of the quantized weight matrix W', and so on. During the inference stage, the dimensions of w_row = 5120 and w_column = 15360 are restored, that is, the above-mentioned first zero value is copied 128 times to form 128 rows of elements within the first group in column i, and so on. The quantization scale matrix S is the same.

[0054] As Figure 1 shown, the data format of each zero value in the zero matrix Z is the same as the data format of each element value in the quantized weight matrix W', which is int4 format. And the data format of each quantization scale in the quantization scale matrix S is the same as the original data format of each weight value in the weight matrix W, which is fp16 format.

[0055] According to the matrix multiplication principle, the dimensions of the output matrix are: out_row = x_row, out_column = w_column.

[0056] Based on the quantization result of the above weight matrix W, the calculation process of the output matrix can be expressed as the following formula:

[0057]

[0058] The calculation formula for the element value in the i-th row and j-th column of the output matrix out is as shown above. Specifically, it is the sum of the first calculation result and the second calculation result. Among them, the first calculation result reflects the multiplication of the quantized weight matrix W' after dequantization and the input matrix X, and the second calculation result reflects the multiplication of the input matrix X and the non-quantized weight matrix O. Among them, the dequantization process can be simply expressed as: (W' - Z)*S.

[0059] For the above quantization weight matrix W’, zero matrix Z, and quantization scale matrix S, these matrices are all non-sparse matrices. When storing them, the element values in the corresponding matrices can be stored row by row in the storage space in the order of rows. Taking the quantization weight matrix W’ as an example, a continuous storage space can be allocated in the video memory. The element values of the first row in the quantization weight matrix W’ are sequentially stored in this storage space. Then, the element values of the second row are sequentially stored in this storage space, and so on. Based on this, when wanting to query the element value at the i-th row and k-th column in the quantization weight matrix W’, the element value can be found in this storage space based on the position index of "row number * total column number + column number", that is, reading the element value at the storage position of "i * w_columnu + k" in this storage space is the element value at the i-th row and k-th column in the quantization weight matrix W’. The same applies to the zero matrix Z and the quantization scale matrix S, which will not be elaborated.

[0060] Since the non-quantization weight matrix O is a sparse matrix, that is, only a small number of element values in it are non-zero. If the non-zero elements in each row of the non-quantization weight matrix O are continuously stored in the same way as the above-mentioned continuous storage of non-sparse matrices such as the quantization weight matrix W’, it will occupy a lot of storage space. Therefore, in order to save storage space, for non-quantization weight matrices, a coo (coordinate format) storage structure is usually used for storage. In the coo storage structure, a triple (row, col, data) is used to store the non-zero element values in the non-quantization weight matrix, where row and col are the row number and column number of the non-zero element value respectively, and data is the non-zero element value.

[0061] Although using the coo storage structure can reduce the storage space occupied when storing the non-quantization weight matrix, it is not conducive to accelerating the multiplication calculation of the input matrix and the weight matrix during the inference stage of the neural network model. Because what is stored in the video memory is no longer all the element values in each row of the non-quantization weight matrix O, so it is impossible to quickly find the element value that needs to be read from the non-quantization weight matrix O based on the above traversal method of "row number * total column number + column number". In the solutions for accelerating the matrix multiplication using acceleration methods such as CUDA (Compute Unified Device Architecture), they are all based on the premise that the element value can be traversed and found in its storage space based on the subscript (i.e., row and column numbers) of the element value using the above traversal method. ij When calculating out, the element value that needs to be read from the non-quantization weight matrix O cannot be found.

[0062] When storing the non-quantization weight matrix O based on the coo storage structure, since what is stored are only the subscript indexes and values of non-zero element values, then when performing matrix multiplication calculations, such as performing X ik *Okj When calculating, find X according to the above continuous storage method ik After that, it is necessary to first query whether there is an index item (k, j) in the coo storage structure. If there is, further query the element value corresponding to this index item. If there is no such index item, it means that this calculation is not required, so the search for X ik The operation is useless, which is not conducive to achieving fast calculation.

[0063] Based on this, the embodiment of the present invention provides a new storage structure for storing the non-quantized weight matrix O. Based on this new storage structure, not only can the storage space occupied when storing the non-quantized weight matrix O be reduced, but also the accelerated multiplication calculation of the input matrix and the weight matrix can be realized.

[0064] The following embodiments are used to illustrate the new storage structure and the matrix multiplication calculation process based on this new storage structure. The methods provided in the following embodiments can be executed by a graphics processing unit (GPU) included in an electronic device. The electronic device can be a terminal device such as a PC, a laptop, or a smartphone, or a server such as a cloud server.

[0065] Figure 2 It is a flowchart of a model weight storage method provided by an embodiment of the present invention, as Figure 2 shown, the method may include the following steps:

[0066] 201. Obtain the quantization result of the weight matrix of the target network layer in the neural network model. The quantization result includes a quantized weight matrix, a quantization parameter matrix, and a non-quantized weight matrix. The non-quantized weight matrix is a sparse matrix.

[0067] 202. Generate multiple storage matrices for storing the non-quantized weight matrix. The multiple storage matrices include a first storage matrix for storing each non-zero element value in the non-quantized weight matrix column by column, a second storage matrix for storing the row numbers corresponding to each non-zero element value in the non-quantized weight matrix in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values in each column of the non-quantized weight matrix, and a fourth storage matrix for storing the starting index position corresponding to each column in the first storage matrix.

[0068] 203. Store multiple storage matrices, the quantized weight matrix, and the quantization parameter matrix.

[0069] In this embodiment, assume that the weight matrix of the target network layer is Figure 1For the weight matrix W shown in the figure, its quantization result includes a quantized weight matrix W', a non-quantized weight matrix O, and a quantization parameter matrix composed of a zero matrix Z and a quantization scale matrix S. Among them, the non-sparse matrices such as the quantized weight matrix W', the zero matrix Z, and the quantization scale matrix S are stored in the "continuous storage" method introduced above. Simply put, three storage spaces for storing these three matrices are respectively allocated in the video memory, and the storage addresses of each storage space are continuous. The elements in each matrix are stored row by row into the corresponding storage space.

[0070] For the non-quantized weight matrix O, which is a sparse matrix, it is stored using a new storage structure composed of the above four storage matrices. In fact, these four storage matrices can be four row vectors. First, it should be noted that since the non-quantized weight matrix O is a sparse matrix, only the non-zero element values actually participate in the calculation.

[0071] Specifically, the first storage matrix (which can be denoted as outliers_val) is used to store the non-zero element values in each column of the non-quantized weight matrix. These non-zero element values are the weight values in the weight matrix W that do not need to be quantized. In the above example, they are floating-point numbers in the fp16 format. Therefore, the first storage matrix is used to store all the non-zero element values (i.e., outliers values) contained in the non-quantized weight matrix O. The storage order is to traverse column by column, and for each column, store them in the row-by-row order of the first row (row1), the second row (row2),...

[0072] The second storage matrix (which can be denoted as outliers_row) is used to store the row numbers corresponding to the non-zero element values in the non-quantized weight matrix O in the non-quantized weight matrix O. It can be seen that the dimension of the second storage matrix is equal to that of the first storage matrix. Assuming that there are M non-zero element values in the non-quantized weight matrix O, then the dimension of the first storage matrix can be: 1*M, and the dimension of the second storage matrix is also: 1*M, which is used to record the actual row numbers in the non-quantized weight matrix O corresponding to each non-zero element value in the first storage matrix. In the second storage matrix, each row number can be represented in a data format such as unit32.

[0073] The third storage matrix (which can be denoted as outliers_column_num) is used to store the number of non-zero element values (as mentioned above, referring to non-zero element values) in each column of the non-quantized weight matrix O. Since the number of columns of the non-quantized weight matrix O is the same as the number of columns w_columu of the weight matrix W, the dimension of the third storage matrix is: 1*w_columu.

[0074] The fourth storage matrix (which can be denoted as outliers_column_start) is used to store the starting index positions corresponding to each column in the non-quantized weight matrix O in the first storage matrix. In fact, the starting index position corresponding to the n-th column is determined according to the starting index position corresponding to the (n - 1)-th column and the number of non-zero element values included in the (n - 1)-th column, and the starting index position of the starting column (i.e., the first column with column number 0: column 0) is 0. Among them, the position index range of the first storage matrix is [0, M - 1], where M is the number of non-zero element values stored in the first storage matrix.

[0075] After obtaining the above four storage matrices corresponding to the non-quantized weight matrix O, since these four storage matrices are non-sparse matrices, they can be stored in the same way as non-sparse matrices such as the quantized weight matrix W', that is, by using the "continuous storage" method. That is, four storage spaces are allocated in the video memory respectively for storing these four storage matrices, and each storage space has consecutive addresses. The element values included in a storage matrix can be sequentially stored into the corresponding storage space.

[0076] For ease of understanding, the following is combined with Figure 3 to exemplify and illustrate the stored contents of the above four storage matrices.

[0077] In Figure 3 , it is assumed that the non-quantized weight matrix O is a 5 * 5 matrix. Thus, the value ranges of its row number and column number are both [0, 4]. For ease of description, the position coordinates of the element values in this matrix are represented in the form of: (rowi, columnj). Thus, as shown in Figure 3 , it is assumed that the element value at the position (2, 1) is 1.0, the element value at the position (3, 4) is 3.0, the element value at the position (4, 4) is 2.0, and the element values at other positions are all 0. Based on the above assumptions, the following four storage matrices are generated:

[0078] The first storage matrix outliers_val = [1.0, 3.0, 2.0], that is, in the traversal order of columns, the non-zero element values in the non-quantized weight matrix O are stored row by row in each column into the first storage matrix.

[0079] The second storage matrix outliers_row = [2, 3, 4], that is, the row numbers corresponding to each non-zero element value in the first storage matrix in the non-quantized weight matrix O are stored in sequence. Among them, 2 is the row number corresponding to the element value 1.0, 3 is the row number corresponding to the element value 3.0, and 4 is the row number corresponding to the element value 2.0.

[0080] The third storage matrix outliers_column_num = [0, 1, 0, 0, 2], which stores the number of non-zero element values in each column of the non-quantized weight matrix O. Among them, the first column (i.e., column 0) does not contain non-zero element values, so the corresponding number is 0. The second column (i.e., column 2) contains 1 non-zero element value (which is 1.0), so the corresponding number is 1. Similarly, the third and fourth columns do not contain non-zero element values, so the corresponding numbers are 0. The fifth column (i.e., column 4) contains 2 non-zero element values (which are 3.0 and 2.0), so the corresponding number is 2.

[0081] The fourth storage matrix outliers_column_start = [0, 0, 1, 1, 1], which stores the corresponding starting index positions of each column of the non-quantized weight matrix O in the first storage matrix. Among them, assuming that there are M element values stored in the first storage matrix, then its position index range is [0, M - 1]. Based on this, it can be understood that regardless of whether there are non-zero element values in the first column (column 0) of the non-quantized weight matrix O, its corresponding starting index position in the first storage matrix is 0. And since there are no non-zero element values in the first column (column 0) of the non-quantized weight matrix O in the above example, the starting position index of the second column (column 1) in the first storage matrix is 0 + 0 = 0, where the first 0 added refers to the starting position index of the previous column, and the second 0 added refers to the number of non-zero element values in the previous column. Similarly, the starting position index of the third column (column 2) in the first storage matrix is: 0 + 1 = 1, that is, it is determined by the sum result of the starting position index 0 of the previous column and the number of non-zero element values 1 in the previous column. The starting position index of the fourth column (column 3) in the first storage matrix is: 1 + 0 = 1. The starting position index of the fifth column (column 4) in the first storage matrix is: 1 + 0 = 1.

[0082] The above introduces a new storage structure provided by the embodiments of the present invention for storing the non-quantized weight matrix O. Based on this storage structure, the accelerated calculation of the multiplication of the input matrix and the weight matrix can be realized. The specific calculation process will be introduced in the subsequent embodiments.

[0083] As described above, the non-quantized weight matrix O can be stored using the above new storage structure, and the quantized weight matrix W' obtained after quantization of the weight matrix W and the quantization parameter matrix can be stored based on the methods introduced above. In an optional embodiment, to further save the storage space occupied by the quantized weight matrix W' and the quantization parameter matrix, the quantized weight matrix W' and the quantization parameter matrix can also be stored in the following manner:

[0084] Obtain the initial quantization weight matrix and the initial quantization parameter matrix of the target network layer. The initial quantization parameter matrix includes an initial zero matrix. The initial quantization weight matrix and the initial zero matrix have the same dimension and the element values have the same first data format;

[0085] Group the initial quantization weight matrix by rows to obtain the grouped quantization weight matrix. Among them, the element values corresponding to the same column in the initial quantization weight matrix are concatenated into element values in the second data format and stored in the grouped quantization weight matrix;

[0086] Group the initial zero matrix by columns to obtain the grouped zero matrix. Among them, the element values corresponding to the same row in the initial zero matrix are concatenated into element values in the second data format and stored in the grouped zero matrix. The initial quantization weight matrix and the initial zero matrix are grouped according to the same grouping length;

[0087] Store the grouped quantization weight matrix and the grouped zero matrix.

[0088] For ease of understanding, in combination with Figure 4 to exemplify the above storage method.

[0089] Assume that the initial quantization weight matrix is W1 and the initial zero matrix is Z1, and assume that their dimensions are both: w_row * w_column, and their element values are both in int4 format (the first data format).

[0090] In an optional embodiment, as described above, the initial zero matrix Z1 can be grouped by rows according to the set groupsize. The zero values in each row group corresponding to the same column are the same, and only one needs to be stored. Specifically, refer to Figure 1 for the relevant illustration in

[0091] It should be noted that Figure 4 the storage method corresponding to the illustrated embodiment is applicable to both the case of grouping and storing the initial zero matrix Z1 based on groupsize and the case of not grouping and storing the initial zero matrix Z1 based on groupsize. Figure 4 The illustration in

[0092] In Figure 4 it is set that the same grouping length adopted by the initial quantization weight matrix W1 and the initial zero matrix Z1 = 8. The initial quantization weight matrix W1 is grouped by rows to obtain the grouped quantization weight matrix W2, and the initial zero matrix Z1 is grouped by columns to obtain the grouped zero matrix Z2. As shown in Figure 4As shown in [figure], after this grouping process, the dimension of the quantized weight matrix W2 after grouping becomes: w_row / 8 * w_column, and the dimension of the quantized weight matrix W2 after grouping becomes: w_row / groupsize * w_column / 8.

[0093] Among them, the values of the elements within the same column in the initial quantized weight matrix W1 are concatenated into the element values of the second data format to be stored in the quantized weight matrix W2 after grouping. Taking the 8 element values of int4 format in the first group (the first 8 rows) corresponding to the first column in the initial quantized weight matrix W1 as an example, these 8 element values of int4 format can be combined into a numerical value of int32 format (the second data format) and stored in the position of the first row and first column in the quantized weight matrix W2 after grouping.

[0094] Among them, the values of the elements within the same row in the initial zero matrix Z1 are concatenated into the element values of the second data format to be stored in the zero matrix Z2 after grouping. Taking the 8 element values of int4 format in the first group (the first 8 columns) corresponding to the first row in the initial zero matrix Z1 as an example, these 8 element values of int4 format can be combined into a numerical value of int32 format and stored in the position of the first row and first column in the zero matrix Z2 after grouping.

[0095] Among them, 8 element values of int4 format are combined into a numerical value of int32 format. Simply put, these 8 element values are represented as binary strings with a length of 4 bits in sequence, and finally a binary string with a length of 32 bits is obtained. This binary string is subjected to format conversion to obtain the corresponding decimal value.

[0096] Through the above storage method, the occupation of storage space by the quantization result of the weight matrix can be further reduced.

[0097] It should only be noted that in the inference stage, when the multiplication calculation of the input matrix of the target network layer and the weight matrix needs to be performed, it is necessary to load the quantized weight matrix W2 after grouping and the zero matrix Z2 after grouping from the storage space, and restore the quantized weight matrix W2 after grouping and the zero matrix Z2 after grouping to the initial quantized weight matrix W1 and the initial zero matrix Z1. In addition, for example, for the quantization scale matrix S, when the storage processing of grouping such as Figure 1 shown in [figure] with groupsize grouping is performed, it also needs to be restored to the initial matrix with a dimension of w_row * w_column.

[0098] The following introduces the multiplication calculation process of the input matrix of the target network layer and the weight matrix.

[0099] Figure 5The following is a flowchart of a method for calculating and processing model weights provided by an embodiment of the present invention. As Figure 5 shown, the method may include the following steps:

[0100] 501. In response to the input of the input matrix of the target network layer in the neural network model obtained based on the inference task, obtain the quantization weight matrix and the quantization parameter matrix corresponding to the target network layer.

[0101] 502. Obtain multiple storage matrices for storing the non-quantized weight matrix corresponding to the target network layer. The multiple storage matrices include a first storage matrix for storing each non-zero element value in the non-quantized weight matrix column by column, a second storage matrix for storing the row numbers corresponding to each non-zero element value in the non-quantized weight matrix in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values in each column of the non-quantized weight matrix, and a fourth storage matrix for storing the starting index position corresponding to each column in the first storage matrix in the non-quantized weight matrix.

[0102] 503. Determine the first calculation result and the second calculation result corresponding to the same position in the output matrix of the target network layer according to the multiple storage matrices, so as to obtain the target calculation result at the same position according to the sum of the first calculation result and the second calculation result. Among them, the first calculation result corresponds to the product of the input matrix and the dequantized weight matrix, and the second calculation result corresponds to the product of the input matrix and the non-quantized weight matrix. The dequantized weight matrix is obtained by dequantizing the quantization weight matrix based on the quantization parameter matrix.

[0103] In this embodiment, the multiplication calculation process of the input matrix X of the target network layer and its weight matrix W is taken as an example for illustration. Among them, the weight matrix W of the target network layer has been quantized. The quantization result of the weight matrix includes a quantization weight matrix W', a quantization parameter matrix (including the zero value matrix Z and the quantization scale matrix S in the above text), and a non-quantized weight matrix O. Among them, the quantization weight matrix W', the zero value matrix Z, and the quantization scale matrix S are stored in the continuous storage mode in the above text and have been stored in the corresponding storage space of the video memory. The non-quantized weight matrix O as a sparse matrix is stored in the new storage structure introduced in the foregoing embodiment, and the corresponding multiple storage matrices have been stored in the corresponding storage space of the video memory. The output matrix as the calculation result is denoted as out.

[0104] Based on the input of the inference task, each network layer in the neural network model will successively obtain its own input matrix. For example, if the inference task is to predict what word should be filled in a certain position in the input sentence, at this time, the input matrix of a certain network layer in the neural network model can be a matrix obtained by encoding this input sentence, and the output matrix obtained by multiplying this input matrix with the weight matrix of this network layer can be used as the input matrix of the next network layer. Of course, other processing (such as downsampling, etc.) can also be performed on this output matrix before using it as the input matrix of the next network layer.

[0105] Taking the target network layer as an example, when it is necessary to perform the multiplication calculation of the input matrix X and the weight matrix W of the target network layer, it is necessary to first read from the video memory multiple calculation objects that are actually used to calculate with the input matrix X: the quantized weight matrix W’, the quantization parameter matrix: the zero-value matrix Z and the quantization scale matrix S, as well as multiple storage matrices for storing the non-quantized weight matrix O, and load these matrices into the target cache space pre-allocated in the video memory.

[0106] In practical applications, the target cache space can be determined according to the dimension of the weight matrix of the network layer with the largest scale in the neural network model. Because in fact, the dimensions of the weight matrices corresponding to different network layers may be different, and the target cache space is shared by each network layer, that is, each time a certain network layer is calculated, the above-mentioned multiple matrices related to the calculation need to be loaded into this target cache space for calculation. Therefore, the target cache space should be at least larger than the dimension of the largest weight matrix. In addition, the intermediate results and the final calculation results (such as the first calculation result, the second calculation result, and the target calculation result in the above text) generated during the calculation process can all be temporarily stored in this target cache space.

[0107] As described above, the calculation formula for the output matrix out is as follows:

[0108]

[0109] According to this calculation formula, it can be seen that the element value in the i-th row and j-th column of the output matrix out is determined by the sum result of the first calculation result (the first item to be added) and the second calculation result (the second item to be added) that are added together. Therefore, during the calculation process, the first calculation result and the second calculation result can be calculated separately, and the two calculation results can be respectively superimposed on out ij That's it. Moreover, from the above calculation formula, it can be seen that during the calculation process of the first calculation result and the second calculation result, the element values at several positions in different matrices are actually calculated separately and finally added together. Therefore, methods such as parallel multi-threading using CUDA can be used to accelerate the calculation.

[0110] For the calculation process of the first calculation result, specifically, the calculation positions corresponding to multiple parallel threads can be determined first. Among them, the calculation position corresponding to the target thread includes the first position identifier i, the second position identifier j, and the third position identifier k, and the target thread is any one of the multiple threads. The meanings of these position identifiers are shown in the above calculation formula. After that, the target thread queries the first element value, the second element value, and the third element value from the input matrix X, the quantized weight matrix W', and the quantization parameter matrix according to the first position identifier i, the second position identifier j, and the third position identifier k. Among them, the quantization parameter matrix includes the zero matrix Z and the quantization scale matrix S, so the third element value includes the third element values respectively queried from the zero matrix Z and the quantization scale matrix S.

[0111] Among them, the first element value corresponds to the position of the input matrix X with the first position identifier i as the row number and the third position identifier k as the column number, which is X ik . The second element value corresponds to the position of the quantized weight matrix W' with the third position identifier k as the row number and the second position identifier j as the column number, which is W′ kj . The third element value corresponds to the position of the quantization parameter matrix with the third position identifier k as the row number and the second position identifier j as the column number, which is Z kj and S kj .

[0112] After that, according to the first element value X ik , the second element value W′ kj , and the third element value Z kj and S kj update the first calculation result of the target position out ij of the output matrix. Specifically, it is to perform corresponding calculations on these element values according to the above calculation formula to obtain a calculated sub-item to be accumulated in the first calculation result corresponding to the current first position identifier i, second position identifier j, and third position identifier k. According to a similar calculation process, based on the above calculation formula, fixing the first position identifier i and the second position identifier j unchanged and traversing different values of the third position identifier k, multiple calculated sub-items can be obtained and accumulated into the first calculation result respectively, and finally the first calculation result is obtained. It can be seen from the above formula that the value range of k is determined by the number of columns x_column of the input matrix X.

[0113] The process of calculating a calculated sub-item to be accumulated in the first calculation result by the target thread corresponding to the first position identifier i, the second position identifier j, and the third position identifier k is described above. It can be understood that for out ijFor the corresponding first calculation result, the actual number of threads required is x_column, and each of the threads corresponds to the same first position identifier i and second position identifier j, but the corresponding third position identifier k is different.

[0114] First, still taking the above target thread as an example, introduce the process of calculating the target thread out ij the process of the corresponding second calculation result, and then introduce the method of setting the thread and the method of determining the position identifier corresponding to the thread.

[0115] For the calculation process of the second calculation result, the target thread no longer directly reads the corresponding element value in the corresponding matrix according to the position identifier as in the calculation of the first calculation result, but first needs to determine the fourth element value in the non-quantized weight matrix O according to the second position identifier j, the third position identifier k, and multiple storage matrices. The fourth element value corresponds to the element value at the target sorting position in a column of the non-quantized weight matrix O with the second position identifier j as the column number, and the target sorting position is the value corresponding to the third position identifier k. In addition, the target thread determines the fifth element value in the input matrix X. The fifth element value corresponds to the position in the input matrix X with the first position identifier i as the row number and the fourth position identifier as the column number, and the fourth position identifier corresponds to the row number of the fourth element value in the non-quantized weight matrix O. Then, the target thread updates the target position out of the output matrix according to the product of the fourth element value and the fifth element value ij of the second calculation result.

[0116] Among them, specifically, the determination process of the fourth element value and the fifth element value includes:

[0117] Query the number of non-zero element values column outl iers_column_num[j] included in the target with the second position identifier j as the column number of the non-quantized weight matrix O in the third storage matrix outl iers_column_num. If the number of non-zero element values is greater than the value corresponding to the third position identifier k, query the starting index position corresponding to the target column in the first storage matrix outl iers_val in the fourth storage matrix outl iers_column_start: outl iers_column_start[j], that is, if outl iers_column_num[j]>k, find the starting index position outl iers_column_start[j] of the jth column in the first storage matrix outl iers_val. And if outl iers_column_num[j]≤k, skip it, that is, no longer perform calculations.

[0118] The fourth element value corresponding to the target index position is queried in the first storage matrix outl iers_val, and the target index position is the sum of the values corresponding to the starting index position and the third position identifier: outl iers_column_start[j]+k. Therefore, the fourth element value is: outl iers_val[outl iers_column_start[j]+k]. The actual row number o corresponding to the fourth element value in the non-quantized weight matrix O is row is: outliers_row[outliers_column_start[j]+k], that is, find the real row number corresponding to the fourth element value in the unquantized weight matrix O in the second storage matrix outliers_row. In other words, the fourth element value is the oth element in the unquantized weight matrix O. row The non-zero element value of row j, column j: O orowj According to the principle of matrix multiplication, the fifth element value in the input matrix X to be multiplied should be the value of the oth element in the i-th row of the input matrix X. row Column element values: Based on this, the fourth position mark is o row .

[0119] Afterwards, The product of is out ij A calculation sub-item of the second calculation result is superimposed on out ij Just go on.

[0120] It can be seen from the above calculation process that in the embodiment of the present invention, the calculation of the input matrix X and the non-quantized weight matrix O no longer uses "X ik *O kj "Calculation method: This calculation method is essentially to traverse the non-quantized weight matrix O according to the input matrix X, but since the non-quantized weight matrix O does not store the values of each row of elements continuously in a continuous storage manner, this traversal cannot be achieved. In the embodiment of the present invention, the calculation idea of traversing the input matrix X according to the position of the non-zero element values in the non-quantized weight matrix O is adopted. The non-zero element values contained in the non-quantized weight matrix O and the row number and column number corresponding to each non-zero element value in the non-quantized weight matrix O are stored using the storage structure of the above-mentioned multiple storage matrices. When performing calculations, the non-zero element values required to be calculated by the current thread can be queried according to the multiple storage matrices according to the above-mentioned traversal method, and the element values to be searched in the input matrix are determined accordingly, thereby realizing accelerated calculation based on multi-threading. In this way, for the non-zero element values of the jth column in the non-quantized weight matrix O, it is not necessary to correspond to the calculation dimension of the quantized weight matrix in this matrix multiplication, thereby realizing the memory access of O(1) complex non-zero element values, thereby improving the reasoning speed.

[0121] The following introduces the settings of multiple threads and the method for determining the position identifiers corresponding to the threads.

[0122] Specifically, according to the number of rows out_row and columns out_column of the output matrix out, and the number of columns x_column of the input matrix X, determine the number of multiple threads: out_row * out_column * x_column. In this way, the total number of threads required is obtained. In practical applications, the upper limit of the thread identifiers supported on the GPU is limited. For example, on some GPUs, the maximum thread identifier is up to 1024. If the total number of the above threads exceeds the limit of this identifier upper limit, the concepts of thread blocks (block) and thread grids (grid) need to be introduced. In large-scale neural network models (such as large language models), the dimension of the weight matrix is often very large, which will exceed the above limit.

[0123] Therefore, the dimension of the thread block can be set first, and then according to the set dimension of the thread block, determine the dimension of the thread grid to be set. Among them, the dimension of the thread block determines the number of threads included in each thread block, and the dimension of the thread grid determines the number of thread blocks included in the thread grid.

[0124] Among them, the dimension of the thread block can be three-dimensional: blockdim.x, blockdim.y, blockdim.z, and the dimension of the thread grid can be three-dimensional: griddim.x, griddim.y, griddim.z.

[0125] Based on the above three dimensions, the number of threads that can be included in the thread block is: blockdim.x * blockdim.y * blockdim.z. Similarly, the number of thread blocks that can be included in the thread grid is: griddim.x * griddim.y * griddim.z.

[0126] In practical applications, the values of blockdim.x, blockdim.y, and blockdim.z can be set first, such as (32, 1, 1). Based on this setting result and the total number of threads, it is known how many thread blocks are required, and then the values of griddim.x, griddim.y, and griddim.z can be determined.

[0127] Based on the above settings of the threads, the process of determining the respective calculation positions of multiple threads includes: according to the dimension of the thread grid, the dimension of the thread block, the relative positions of each thread block in the thread grid, and the relative positions of each thread in the thread block to which it belongs, determine the respective identifiers of multiple threads, and then determine the respective calculation positions of multiple threads according to the respective identifiers of multiple threads.

[0128] The above calculation process is introduced below. Assume the total number of threads is N, then N = gridDim.x * gridDim.y * gridDim.z * blockDim.x * blockDim.y * blockDim.z. In the following calculation process, blockIdx.x, blockIdx.y, and blockIdx.z respectively represent the relative positions of the current thread block in the thread grid to which it belongs, and threadIdx.x, threadIdx.y, and threadIdx.z respectively represent the relative positions of the current thread in the thread block to which it belongs.

[0129] The process of determining the identifiers of each thread is to arrange all the threads in a sequence with serial numbers 0, 1, 2,..., N, and find the serial number of the current thread in it. This serial number is the identifier of the thread.

[0130] First, find out which thread block the current thread is in the thread grid: blockId

[0131] blockId = blockIdx.x + blockIdx.y * gridDim.x + blockIdx.z * gridDim.x * gridDim.y.

[0132] After that, find out which thread the current thread is in the thread block: threadId

[0133] threadId = threadIdx.x + threadIdx.y * blockDim.x + threadIdx.z * blockDim.x *

[0134] blockDim.y.

[0135] After that, calculate how many threads there are in a thread block: M

[0136] M = blockDim.x * blockDim.y * blockDim.z.

[0137] Finally, obtain the serial number of the current thread: idx

[0138] idx = threadId + M * blockId.

[0139] In this way, the identifier of a thread is obtained. This identifier is equivalent to expanding all threads into a one-dimensional space, and the corresponding position of each thread in this one-dimensional space. And the above (i, j, k) is a three-dimensional space. According to the set mapping formula, the three position identifiers corresponding to each thread in this three-dimensional space can be determined: (i, j, k).

[0140] The calculation of the above thread identifier and the calculation process of the three-dimensional position coordinates corresponding to the thread can be implemented with reference to the existing related technologies, and will not be elaborated here too much.

[0141] The following will describe in detail the model weight calculation processing and the model weight storage device of one or more embodiments of the present invention. Those skilled in the art can understand that these devices can all be configured by using commercially available hardware components through the steps taught by this solution.

[0142] Figure 6 It is a schematic structural diagram of a model weight calculation processing device provided by an embodiment of the present invention, as Figure 6 shown. The device includes: an acquisition module 11 and an inference module 12.

[0143] The acquisition module 11 is configured to, in response to the input of the input matrix of the target network layer in the neural network model obtained based on the inference task, acquire the quantization weight matrix and the quantization parameter matrix corresponding to the target network layer, and acquire a plurality of storage matrices for storing the non-quantized weight matrix corresponding to the target network layer. The plurality of storage matrices include a first storage matrix for storing each non-zero element value in the non-quantized weight matrix column by column, a second storage matrix for storing the row numbers corresponding to each non-zero element value in the non-quantized weight matrix in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values in each column of the non-quantized weight matrix, and a fourth storage matrix for storing the starting index position corresponding to each column of the non-quantized weight matrix in the first storage matrix; wherein, the non-quantized weight matrix is a sparse matrix, and the quantization result of the weight matrix of the target network layer includes the quantization weight matrix, the quantization parameter matrix, and the non-quantized weight matrix.

[0144] The inference module 12 is configured to determine a first calculation result and a second calculation result corresponding to the same position in the output matrix of the target network layer according to the plurality of storage matrices, so as to obtain the target calculation result of the same position according to the sum of the first calculation result and the second calculation result. Wherein, the first calculation result corresponds to the product of the input matrix and the dequantization weight matrix, the second calculation result corresponds to the product of the input matrix and the non-quantized weight matrix, and the dequantization weight matrix is obtained by dequantizing the quantization weight matrix based on the quantization parameter matrix.

[0145] Figure 6 The device shown can perform the steps in the foregoing embodiments. For the detailed implementation process and technical effects, refer to the descriptions in the foregoing embodiments and will not be elaborated herein.

[0146] Figure 7 The following is a schematic structural diagram of a model weight storage device provided by an embodiment of the present invention. As Figure 7 shown, the device includes: an acquisition module 21, a generation module 22, and a storage module 23.

[0147] The acquisition module 21 is configured to acquire the quantization result of the weight matrix of the target network layer in the neural network model. The quantization result includes a quantized weight matrix, a quantization parameter matrix, and a non-quantized weight matrix, and the non-quantized weight matrix is a sparse matrix.

[0148] The generation module 22 is configured to generate a plurality of storage matrices for storing the non-quantized weight matrix. The plurality of storage matrices include a first storage matrix for storing the non-zero element values of each column in the non-quantized weight matrix, a second storage matrix for storing the row numbers corresponding to the non-zero element values of each column in the non-quantized weight matrix in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values of each column in the non-quantized weight matrix, and a fourth storage matrix for storing the starting index positions corresponding to each column in the first storage matrix in the non-quantized weight matrix.

[0149] The storage module 23 is configured to store the plurality of storage matrices, the quantized weight matrix, and the quantization parameter matrix.

[0150] Figure 7 The device shown can perform the relevant steps in the foregoing embodiments. For the detailed implementation process and technical effects, refer to the descriptions in the foregoing embodiments and will not be elaborated herein.

[0151] In a possible design, the structure of the foregoing Figure 6 、 Figure 7 shown device can be implemented as an electronic device. As Figure 8 shown, the electronic device may include: a processor 31, a memory 32, and a communication interface 33. Among them, executable code is stored on the memory 32. When the executable code is executed by the processor 31, the processor 31 can at least implement the model weight calculation processing method or the model weight storage method provided in the foregoing embodiments.

[0152] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium. Executable code is stored on the non-transitory machine-readable storage medium. When the executable code is executed by the processor of the electronic device, the processor can at least implement the model weight calculation processing method or the model weight storage method provided in the foregoing embodiments.

[0153] The device embodiments described above are merely illustrative. The network elements described as separate components may or may not be physically separated. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative work.

[0154] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of adding a necessary general hardware platform. Of course, it can also be implemented by a combination of hardware and software. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a computer product. The present invention can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for calculating and processing model weights, characterized in that, Including: In response to an input of an input matrix of a target network layer in a neural network model obtained based on an inference task, obtaining a quantization weight matrix and a quantization parameter matrix corresponding to the target network layer; Obtaining a plurality of storage matrices for storing a non-quantized weight matrix corresponding to the target network layer, where the plurality of storage matrices include a first storage matrix for storing non-zero element values in each column of the non-quantized weight matrix, a second storage matrix for storing row numbers corresponding to the non-zero element values in the non-quantized weight matrix in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values in each column of the non-quantized weight matrix, and a fourth storage matrix for storing starting index positions corresponding to each column of the non-quantized weight matrix in the first storage matrix; wherein, the non-quantized weight matrix is a sparse matrix, and the quantization result of the weight matrix of the target network layer includes the quantization weight matrix, the quantization parameter matrix, and the non-quantized weight matrix; Determining a first calculation result and a second calculation result corresponding to the same position in the output matrix of the target network layer according to the plurality of storage matrices, so as to obtain a target calculation result at the same position according to the sum of the first calculation result and the second calculation result, where the first calculation result corresponds to the product of the input matrix and an anti-quantized weight matrix, the second calculation result corresponds to the product of the input matrix and the non-quantized weight matrix, and the anti-quantized weight matrix is obtained by performing an anti-quantization process on the quantization weight matrix based on the quantization parameter matrix.

2. The method according to claim 1, wherein The step of obtaining the output matrix of the target network layer includes: Determining calculation positions corresponding to a plurality of parallel threads respectively, where a calculation position corresponding to a target thread includes a first position identifier, a second position identifier, and a third position identifier, and the target thread is any one of the plurality of threads; Querying a first element value, a second element value, and a third element value from the input matrix, the quantization weight matrix, and the quantization parameter matrix according to the first position identifier, the second position identifier, and the third position identifier, where the first element value corresponds to a position of the input matrix with the first position identifier as the row number and the third position identifier as the column number, the second element value corresponds to a position of the quantization weight matrix with the third position identifier as the row number and the second position identifier as the column number, and the third element value corresponds to a position of the quantization parameter matrix with the third position identifier as the row number and the second position identifier as the column number; Updating a first calculation result at a target position of the output matrix according to the first element value, the second element value, and the third element value, where the target position corresponds to a position of the output matrix with the first position identifier as the row number and the second position identifier as the column number.

3. The method according to claim 2, wherein The step of obtaining the output matrix of the target network layer includes: Determine a fourth element value in the unquantized weight matrix according to the second position identifier, the third position identifier, and the multiple storage matrices, where the fourth element value corresponds to an element value at a target sorting position in a column of the unquantized weight matrix with the second position identifier as the column number, and the target sorting position is the value corresponding to the third position identifier; Determine a fifth element value in the input matrix, where the fifth element value corresponds to a position in the input matrix with the first position identifier as the row number and the fourth position identifier as the column number, and the fourth position identifier corresponds to the row number of the fourth element value in the unquantized weight matrix; Update a second calculation result at the target position of the output matrix according to the product of the fourth element value and the fifth element value.

4. The method according to claim 3, characterized in that The determining the fourth element value in the unquantized weight matrix according to the second position identifier, the third position identifier, and the multiple storage matrices includes: Query the number of non-zero element values included in a target column of the unquantized weight matrix with the second position identifier as the column number in the third storage matrix; If the number of non-zero element values is greater than the value corresponding to the third position identifier, query a starting index position corresponding to the target column in the first storage matrix in the fourth storage matrix; Query a fourth element value corresponding to a target index position in the first storage matrix, where the target index position is the sum of the starting index position and the value corresponding to the third position identifier; The determining the fifth element value in the input matrix includes: Query a row number corresponding to the target index position in the second storage matrix, and determine the row number corresponding to the target index position as the fourth position identifier; Determine a fifth element value at a position in the input matrix with the first position identifier as the row number and the fourth position identifier as the column number.

5. The method according to claim 2, wherein The method further includes: Determine the number of the multiple threads according to the number of rows and columns of the output matrix and the number of columns of the input matrix; Determine a dimension of a thread grid to be set according to a set dimension of a thread block, where the dimension of the thread block determines the number of threads included in each thread block, and the dimension of the thread grid determines the number of thread blocks included in the thread grid; The determining calculation positions corresponding to the multiple parallel threads includes: Determine identifiers corresponding to the multiple threads according to the dimension of the thread grid, the dimension of the thread block, the relative positions of the thread blocks in the thread grid, and the relative positions of the threads in their respective thread blocks; Determine calculation positions corresponding to the multiple threads according to the identifiers corresponding to the multiple threads.

6. The method according to claim 1, characterized in that The method further includes: In response to an input of the input matrix, load the input matrix, the quantized weight matrix, the quantization parameter matrix, and the multiple storage matrices into a target cache space to calculate a first calculation result, the second calculation result, and the target calculation result in the target cache space.

7. The method according to claim 1, wherein The method further includes: Obtain the initial quantization weight matrix and the initial quantization parameter matrix of the weight matrix of the target network layer. The initial quantization parameter matrix includes an initial zero matrix. The initial quantization weight matrix and the initial zero matrix have the same dimension and the element values have the same first data format; Group the initial quantization weight matrix by rows to obtain the grouped quantization weight matrix. Among them, the element values corresponding to the same column in the initial quantization weight matrix are concatenated into element values of the second data format to be stored in the grouped quantization weight matrix; Group the initial zero matrix by columns to obtain the grouped zero matrix. Among them, the element values corresponding to the same row in the initial zero matrix are concatenated into the element values of the second data format to be stored in the grouped zero matrix. The initial quantization weight matrix and the initial zero matrix are grouped according to the same grouping length; Store the grouped quantization weight matrix and the grouped zero matrix.

8. The method according to claim 7, wherein The obtaining of the quantization weight matrix and the quantization parameter matrix corresponding to the target network layer includes: Load the grouped quantization weight matrix and the grouped zero matrix; Restore the grouped quantization weight matrix and the grouped zero matrix to the initial quantization weight matrix and the initial zero matrix.

9. A method for storing model weights, characterized in that, Include Obtain the quantization result of the weight matrix of the target network layer in the neural network model. The quantization result includes a quantization weight matrix, a quantization parameter matrix, and a non-quantized weight matrix. The non-quantized weight matrix is a sparse matrix; Generate multiple storage matrices for storing the non-quantized weight matrix. The multiple storage matrices include a first storage matrix for storing each non-zero element value in the non-quantized weight matrix by column, a second storage matrix for storing the row numbers corresponding to each non-zero element value in the non-quantized weight matrix in the non-quantized weight matrix, a third storage matrix for storing the number of non-zero element values in each column of the non-quantized weight matrix, and a fourth storage matrix for storing the starting index position corresponding to each column of the non-quantized weight matrix in the first storage matrix; Store the multiple storage matrices, the quantization weight matrix, and the quantization parameter matrix.

10. An electronic device, characterized in that, Include: A memory, a processor, and a communication interface; wherein, executable code is stored on the memory. When the executable code is executed by the processor, the processor executes the model weight calculation processing method according to any one of claims 1 to 8.

11. A non-transitory machine-readable storage medium, characterized in that, Executable code is stored on the non-transitory machine-readable storage medium. When the executable code is executed by the processor of the electronic device, the processor executes the model weight calculation processing method according to any one of claims 1 to 8.

12. A computer program product, characterized in that, Include: A computer program. When the computer program is executed by the processor of the electronic device, the processor executes the model weight calculation processing method according to any one of claims 1 to 8.

Images