Quantization method, reasoning method and equipment of industry large model and storage medium

Abstract

The embodiment of the invention provides a quantification method and reasoning method of an industry large model, equipment and a storage medium. When an original weight matrix of a network layer in an industry large model is quantified, values of matrix elements on diagonals of an original Hessian matrix are obtained based on an input data matrix of the network layer and can represent activation values, and the positions of columns in the original weight matrix and the original Hessian matrix of the network layer are reordered according to an activation value descending mode. Carrying out matrix partitioning processing on the reordered first weight matrix and the first Hessian matrix respectively, and quantizing the first sub-weight matrix based on the quantization parameter of each sub-weight matrix in the first weight matrix in sequence by taking the partitioned matrix as granularity; the method comprises the steps of quantizing a first weight matrix, updating other first weight sub-matrixes which are not quantized based on the quantized first weight sub-matrix, and finally determining a quantization result of an original weight matrix of a network layer based on a quantization result of each first weight sub-matrix in the first weight matrix, so that the size of an industry large model is compressed.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to a quantization method, an inference method, a device, and a storage medium for an industry large model. Background Art

[0002] An industry large model refers to a large language model (LLM) applied to a specific industry field. The industry large model is pre-trained or instruction fine-tuned (IFT) with a large amount of data in the industry field, and its effect in a specific industry field is better than that of a general large model. Therefore, an industry large model is usually deployed to serve various specific tasks downstream in the industry field. Due to the huge number of parameters of the industry large model, the inference of a high-precision industry large model with tens of billions of parameters may require multiple high-performance GPUs (Graphics Processing Unit), and the high requirements for deployment resources limit the availability of the industry large model.

[0003] Model quantization technology can effectively compress the size of the model and improve the model inference efficiency. Quantizing the industry large model can compress the size of the industry large model, reduce the deployment cost of the industry large model, accelerate model inference, and ensure the availability of the industry large model. Therefore, it is very necessary to study a model quantization method with better quantization effect. Summary of the Invention

[0004] Multiple aspects of this application provide a quantization method, an inference method, a device, and a storage medium for an industry large model, so as to improve the model quantization effect and further accelerate model inference.

[0005] An embodiment of the present application provides a quantization method for an industry large model. The industry large model includes multiple network layers connected in sequence. The method includes: for the network layers that need to be quantized among the multiple network layers, determining the original Hessian matrix according to the input data matrix of the network layer; respectively reordering the positions of the columns in the original weight matrix and the original Hessian matrix of the network layer according to the descending order of the values of the matrix elements on the diagonal of the original Hessian matrix, to obtain a first weight matrix and a first Hessian matrix, and performing matrix block processing on the first weight matrix and the first Hessian matrix column by column, to obtain multiple first sub-weight matrices and multiple sub-Hessian matrices, where the multiple first sub-weight matrices and the multiple sub-Hessian matrices correspond one by one; sequentially quantizing the first sub-weight matrix or the updated first sub-weight matrix based on the quantization parameters of the first sub-weight matrix in the order of the positions of the multiple first sub-weight matrices in the first weight matrix, to obtain a second sub-weight matrix corresponding to the first sub-weight matrix, and determining a first quantization error between the first sub-weight matrix and its corresponding second sub-weight matrix based on the sub-Hessian matrix corresponding to the first sub-weight matrix, and updating the other first sub-weight matrices that have not been quantized according to the first quantization error; determining the quantized weight matrix of the network layer according to the second sub-weight matrices corresponding to the multiple first sub-weight matrices respectively.

[0006] An embodiment of the present application also provides an inference method for an industry large model. The industry large model has been quantized by the quantization method of the industry large model. The method includes: obtaining the industry data to be processed; inputting the industry data to be processed into the industry large model, so that each network layer in the industry large model performs inference, and obtaining the inference result output by the last network layer in the industry large model; where, when the network layer in the industry large model performs inference, loading the quantized weight matrix of the network layer; using the quantization parameters of each column in the quantized weight matrix to perform an inverse quantization operation on the quantized weight matrix, to obtain a second weight matrix; multiplying the input data matrix of the network layer and the second weight matrix, to obtain the output result matrix of the network layer, and the input data matrix of the network layer is the industry data to be processed or the output result matrix of the previous network layer of the network layer.

[0007] An embodiment of the present application also provides a method for inferring an industry large model, which is applied to a cloud server. The cloud server deploys the industry large model, and the industry large model is quantized by a quantization method for the industry large model. The method includes: receiving industry data to be processed sent by a terminal device; inputting the industry data to be processed into the industry large model so that each network layer in the industry large model performs inference, and obtaining an inference result output by the last network layer in the industry large model; returning the inference result to the terminal device; wherein, when performing inference, the network layer in the industry large model loads the quantized weight matrix of the network layer; uses the quantization parameters of each column in the quantized weight matrix to perform an inverse quantization operation on the quantized weight matrix to obtain a second weight matrix; multiplies the input data matrix of the network layer and the second weight matrix to obtain an output result matrix of the network layer, and the input data matrix of the network layer is the industry data to be processed or the output result matrix of the previous network layer of the network layer.

[0008] An embodiment of the present application also provides a computer device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to execute the steps in the quantization method or the inference method of the industry large model.

[0009] An embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to be able to implement the steps in the quantization method or the inference method of the industry large model.

[0010] In an embodiment of the present application, when quantizing the original weight matrix of the network layer in the industry large model, the value of the matrix element on the diagonal of the original Hessian matrix obtained based on the input data matrix of the network layer can represent the activation value. The positions of each column in the original weight matrix and the original Hessian matrix of the network layer are re-sorted in descending order of the activation value, and the re-sorted first weight matrix and the first Hessian matrix are respectively subjected to matrix block processing. Taking the block matrix as the granularity, each first sub-weight matrix in the first weight matrix is quantized based on the quantization parameter of each first sub-weight matrix in the first weight matrix, and the other first sub-weight matrices that have not been quantized are updated based on the already quantized first sub-weight matrix. Finally, the quantization result of the original weight matrix of the network layer is determined based on the quantization results of each first sub-weight matrix in the first weight matrix. Thus, a new model quantization method is provided. This method can preferentially quantize important weight coefficients, reduce the quantization error between the original weight matrix and its quantized weight matrix, has a good quantization effect, greatly compresses the model size of the industry large model, reduces the deployment cost of the industry large model, accelerates model inference, and ensures the usability of the industry large model. Description of the Drawings

[0011] The accompanying drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0012] Figure 1 It is a flowchart of a quantization method for an industry large model provided by an embodiment of the present application;

[0013] Figure 2 It is a schematic diagram of exemplary matrix reordering;

[0014] Figure 3 It is a flowchart of an inference method for an industry large model provided by an embodiment of the present application;

[0015] Figure 4 It is a flowchart of another inference method for an industry large model provided by an embodiment of the present application;

[0016] Figure 5 It is an exemplary application scenario diagram provided by an embodiment of the present application;

[0017] Figure 6 It is a schematic structural diagram of a quantization device for an industry large model provided by an embodiment of the present application;

[0018] Figure 7 It is a schematic structural diagram of an inference device for an industry large model provided by an embodiment of the present application;

[0019] Figure 8 It is a schematic structural diagram of a computer device provided by an embodiment of the present application. Detailed Embodiments

[0020] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0021] In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the access relationship of associated objects and indicates that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. In the written description of the present application, the character " / " generally represents an "or" relationship between the associated objects before and after. In addition, in the embodiments of the present application, "first", "second", "third", etc. are only used to distinguish the content of different objects and have no other special meanings.

[0022] An industry large model refers to a large language model (LLM) applied to a specific industry field. The industry large model is pre-trained or instruction fine-tuned (IFT) with large-scale industry field data and has better effects in specific industry fields than general large models. Therefore, industry large models are usually deployed to serve various specific tasks downstream in the industry field. Due to the huge number of parameters of the industry large model, the inference of a high-precision industry large model with tens of billions of parameters may require multiple high-performance GPUs (Graphics Processing Unit), and the high requirements for deployment resources limit the availability of the industry large model.

[0023] Model quantization technology can effectively compress the size of the model and improve the model inference efficiency. Quantizing the industry large model can compress the size of the industry large model, reduce the deployment cost of the industry large model, accelerate model inference, and ensure the availability of the industry large model. Therefore, it is very necessary to study model quantization methods with better quantization effects.

[0024] To this end, the embodiments of the present application provide a quantization method, an inference method, a device, and a storage medium for an industry large model. When quantizing the original weight matrix of the network layer in the industry large model, the values of the matrix elements on the diagonal of the original Hessian matrix obtained based on the input data matrix of the network layer can represent the activation values. The positions of each column in the original weight matrix and the original Hessian matrix of the network layer are re-sorted in descending order of the activation values, and the re-sorted first weight matrix and the first Hessian matrix are respectively subjected to matrix block processing. Taking the block matrix as the granularity, the first sub-weight matrix is quantized based on the quantization parameter of each sub-weight matrix in the first weight matrix in turn, and the other first sub-weight matrices that have not been quantized are updated based on the already quantized first sub-weight matrix. Finally, the quantization result of the original weight matrix of the network layer is determined based on the quantization results of each first sub-weight matrix in the first weight matrix. Thereby, a new model quantization method is provided. This method can preferentially quantize important weight coefficients, reduce the quantization error between the original weight matrix and its quantized weight matrix, has a good quantization effect, greatly compresses the model size of the industry large model, reduces the deployment cost of the industry large model, accelerates model inference, and ensures the usability of the industry large model.

[0025] The following will describe in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.

[0026] Figure 1 It is a flowchart of a quantization method for an industry large model provided by an embodiment of the present application. Refer to Figure 1 and the method may include the following steps:

[0027] 101. For the network layers that need to be quantized among multiple network layers, determine the original Hessian matrix according to the input data matrix of the network layer.

[0028] 102. According to the descending order of the values of the matrix elements on the diagonal of the original Hessian matrix, re-sort the positions of each column in the original weight matrix and the original Hessian matrix of the network layer respectively to obtain the first weight matrix and the first Hessian matrix, and perform matrix block processing on the first weight matrix and the first Hessian matrix respectively by column to obtain a plurality of first sub-weight matrices and a plurality of sub-Hessian matrices, where the plurality of first sub-weight matrices and the plurality of sub-Hessian matrices correspond one by one.

[0029] 103. In the order of the positions of the plurality of first sub-weight matrices in the first weight matrix, sequentially perform quantization processing on the first sub-weight matrix or the updated first sub-weight matrix based on the quantization parameter of the first sub-weight matrix to obtain the second sub-weight matrix corresponding to the first sub-weight matrix, determine the first quantization error between the first sub-weight matrix and its corresponding second sub-weight matrix based on the sub-Hessian matrix corresponding to the first sub-weight matrix, and update the other first sub-weight matrices that have not been quantized according to the first quantization error.

[0030] 104. Determine the quantization weight matrix of the network layer according to the second sub-weight matrices corresponding to the respective first sub-weight matrices.

[0031] In this embodiment, the industry large model refers to a large-scale language model applied in a specific industry field, usually having a large number of parameters, requiring a large amount of computing resources, and being able to solve complex problems. The industry field to which the industry large model is applied is not limited. Optionally, the industry large model is used for intelligent question answering in the target industry, and the target industry includes, for example, but is not limited to: government affairs industry, financial industry, power industry, and medical industry. For example, the government affairs large model can perform intelligent question answering, provide accurate and timely answers to government affairs questions such as provident fund, housing, and policies asked by users, and at the same time can help relevant personnel automate the processing and intelligent judgment of reported events, improving the efficiency and accuracy of event handling. For example, the financial large model can perform intelligent question answering, provide accurate and timely answers to financial questions such as financial research reports, stocks, funds, banks, and insurance asked by users. For example, the power large model can perform intelligent question answering, provide accurate and timely answers to power questions such as equipment inspection and power dispatching asked by users. For example, the medical large model can perform intelligent question answering, provide accurate and timely answers to medical questions such as registration, consultation, and ward rounds asked by users.

[0032] In this embodiment, the industry large model includes a plurality of network layers connected in sequence. Among them, the network structure of the network layer is not limited, and the network layer includes, for example, but is not limited to: Deep Neural Networks (DNNs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM), or Transformer network: The Transformer network is a neural network architecture with a self-attention mechanism.

[0033] In this embodiment, all or part of the network layers in the industry large model can be quantized. Here, for the network layers that need to be quantized, the original weight matrix of the network layer is quantized to obtain the quantized weight matrix of the network layer. That is to say, relatively speaking, the original weight matrix is the weight matrix that has not been quantized, and the quantized weight matrix is the weight matrix obtained by quantizing the original weight matrix. The quantization operation can reduce the storage space required by the weight matrix, thereby compressing the model size of the industry large model. For example, after the quantization operation, a 16-bit floating-point number (float 16) is quantized into a 4-bit integer (int 4), and the storage size of the data is reduced to 1 / 4 of the original. Therefore, the quantization operation can compress the size of the industry large model. It can be understood that the more network layers that are quantized, the more the model size of the industry large model can be greatly compressed. In addition, compared with the original weight matrix, the bit width of the quantized weight matrix is reduced, which can save the time of data communication and improve the inference speed of the industry large model.

[0034] In this embodiment, first, a calibration data set required for quantizing the industry large model is prepared. The calibration data set includes multiple calibration data. The calibration data is derived from the industry data of the target industry to which the industry large model is applied. The calibration data includes, for example, but is not limited to: government affairs questions such as provident fund, housing, and policies asked by users, financial questions such as financial research reports, stocks, funds, banks, and insurance, power questions such as equipment inspection and power dispatching, or medical questions such as registration, consultation, and ward rounds. Considering that the selection of calibration data has a certain impact on the final quantization effect of the industry large model, the data distribution of the calibration data is preferably as close as possible to the data distribution of the training data used to train the industry large model. Thus, further optionally, some training data can be randomly selected from the training data used by the industry large model as calibration data. For example, 128 pieces of training data are randomly selected as calibration data. The randomly selected calibration data has good randomness, which can ensure that the quantized industry large model has good generalization performance. Of course, some training data can also be regularly selected from the training data used by the industry large model as calibration data. For example, some training data in a fixed time period or a fixed location is selected as calibration data, and there is no limitation on this.

[0035] In this embodiment, the network layers that need to be quantized in the industry large model are quantized using the calibration data set. Since the calibration data set includes multiple calibration data, one calibration data is sequentially selected from the calibration data set, and the selected calibration data is input into the industry large model, and steps 101 to 104 are iteratively executed until all the calibration data has been selected.

[0036] In practical applications, industry large models are usually composed of dozens of network layers with the same structure stacked layer by layer. In this embodiment, instead of quantifying the original weight matrices of all network layers in the industry large model at once, the original weight matrices of the network layers in the industry large model are quantified layer by layer.

[0037] For the network layers that need to be quantified among multiple network layers, first, the original Hessian matrix is determined according to the input data matrix of the network layer. Among them, the input data matrix of the network layer is the calibration data in the calibration dataset required for quantifying the industry large model or the output result matrix of the previous network layer of the network layer. It can be understood that if the current network layer to be quantified is the first network layer among multiple network layers, the input data matrix of this network layer is the calibration data in the calibration dataset required for quantifying the industry large model, that is, the calibration data in matrix form; if the current network layer to be quantified is not the first network layer among multiple network layers, the input data matrix of this network layer is the output result matrix of the previous network layer of this network layer, and the output result matrix is also the output result in matrix form.

[0038] In this embodiment, the original Hessian matrix of the network layer can be understood as the Hessian Matrix determined based on the input data matrix of the network layer. Optionally, determining the original Hessian matrix according to the input data matrix of the network layer includes: multiplying the input data matrix and its transposed matrix to obtain a multiplication result; determining the original Hessian matrix according to the multiplication result, the identity matrix, and a specified coefficient, where the specified coefficient is used to avoid numerical problems generated when calculating the inverse matrix of the Hessian matrix. The implementation method of determining the original Hessian matrix according to the multiplication result, the identity matrix, and the specified coefficient is not limited. For example, adding the multiplication result and the identity matrix to obtain a new matrix, and multiplying the new matrix and the specified coefficient to obtain the original Hessian matrix. Another example is to add the identity matrix and the specified coefficient to obtain a new matrix, subtract the multiplication result from the new matrix to obtain the original Hessian matrix. Another example is to add the identity matrix and the specified coefficient to obtain a new matrix, and add the multiplication result and the new matrix to obtain the original Hessian matrix. For example, the original Hessian matrix is determined according to formula (1):

[0039]

[0040] Among them, H represents the original Hessian matrix; X represents the input data matrix, X Trepresents the transpose matrix of the input data matrix, I represents the identity matrix; λ represents a specified coefficient set as needed, and λ avoids numerical problems generated when calculating the inverse matrix of the Hessian matrix. Through a large number of experimental data, setting λ to 0.01 has a relatively good effect. Assume that the input data matrix X is a matrix of dimension L×M, then the original Hessian matrix H is a matrix of dimension M×M, and L and M are positive integers.

[0041] In this embodiment, according to the descending order of the values of the matrix elements on the diagonal of the original Hessian matrix, the positions of each column in the original weight matrix and the original Hessian matrix of the network layer are reordered respectively to obtain the first weight matrix and the first Hessian matrix.

[0042] Assume that the original weight matrix is denoted as W, and the original weight matrix W is a matrix of dimension N×M, where N is a positive integer, N is the number of rows of the original weight matrix, and M is the number of columns of the original weight matrix; the first weight matrix is denoted as W′, and the first weight matrix W′ is also a matrix of dimension N×M; assume that the first Hessian matrix is denoted as H′, and the first Hessian matrix H′ is also a matrix of dimension M×M.

[0043] Specifically, the value of the matrix element on the diagonal of the original Hessian matrix represents the influence degree of the feature of the corresponding channel on the model effect. The larger the value, the more important the feature of the corresponding channel. The value of the matrix element in the input data matrix can be characterized as an activation value. As can be seen from formula (1), the value of the matrix element on the diagonal of the original Hessian matrix is positively correlated with the activation value. Therefore, arranging in the descending order of the values of the matrix elements on the diagonal of the original Hessian matrix is equivalent to arranging in the descending order of the activation values. According to the sorting result, the positions of each column in the original weight matrix and the original Hessian matrix of the network layer are adjusted, and the features of the important channels are sorted to the front of the matrix to obtain the first weight matrix W′ and the first Hessian matrix H′.

[0044] In this embodiment, when reordering the positions of each column in the original Hessian matrix according to the descending order of the values of the matrix elements on the diagonal of the original Hessian matrix to obtain the first Hessian matrix, the column with a larger value of the matrix element on the diagonal of the original Hessian matrix is closer to the front in the first Hessian matrix H′. See Figure 2 For the example of, the values of the matrix elements on the diagonal of the original Hessian matrix of dimension M×M are different, and the larger the value of the matrix element on the diagonal, Figure 2The darker the color of the matrix element in it. When adjusting the positions of the columns in the original Hessian matrix of M×M dimension according to the values of the matrix elements on the diagonal of the original Hessian matrix of M×M dimension, the column with a larger value of the matrix element on the diagonal of the original Hessian matrix is placed more forward in the first Hessian matrix H′ of M×M dimension. Taking M as 6 as an example, sorting the columns in the original Hessian matrix in descending order of the values of the matrix elements on the diagonal of the original Hessian matrix, the sorting result is the 6th column of the original Hessian matrix, the 3rd column of the original Hessian matrix, the 1st column of the original Hessian matrix, the 4th column of the original Hessian matrix, the 2nd column of the original Hessian matrix, and the 5th column of the original Hessian matrix. In this way, the 6th column of the original Hessian matrix is adjusted to the 1st column in the first Hessian matrix of M×M dimension; the 3rd column of the original Hessian matrix is adjusted to the 2nd column in the first Hessian matrix of M×M dimension; the 1st column of the original Hessian matrix is adjusted to the 3rd column in the first Hessian matrix of M×M dimension; the 4th column of the original Hessian matrix is adjusted to the 4th column in the first Hessian matrix of M×M dimension; the 2nd column of the original Hessian matrix is adjusted to the 5th column in the first Hessian matrix of M×M dimension; the 5th column of the original Hessian matrix is adjusted to the 6th column in the first Hessian matrix of M×M dimension.

[0045] In this embodiment, according to the descending order of the values of the matrix elements on the diagonal of the original Hessian matrix, when reordering the positions of the columns in the original weight matrix of the network layer to obtain the first weight matrix, the larger the value of the matrix element on the diagonal of the original Hessian matrix, the more forward the column with the same column number as the column where the matrix element on the diagonal of the original Hessian matrix is located in the original weight matrix is in the first weight matrix W′. Continuing Figure 2 the example, taking M as 6 as an example, sorting the columns in the original Hessian matrix in descending order of the values of the matrix elements on the diagonal of the original Hessian matrix, the sorting result is the 6th column of the original Hessian matrix, the 3rd column of the original Hessian matrix, the 1st column of the original Hessian matrix, the 4th column of the original Hessian matrix, the 2nd column of the original Hessian matrix, and the 5th column of the original Hessian matrix. Correspondingly, the sorting result of the original weight matrix W is the 6th column of the original weight matrix, the 3rd column of the original weight matrix, the 1st column of the original weight matrix, the 4th column of the original weight matrix, the 2nd column of the original weight matrix, and the 5th column of the original weight matrix. In this way, the 6th column of the original weight matrix is adjusted to the 1st column in the first weight matrix of N×M dimension; the 3rd column of the original weight matrix is adjusted to the 2nd column in the first weight matrix of N×M dimension; the 1st column of the original weight matrix is adjusted to the 3rd column in the first weight matrix of N×M dimension; the 4th column of the original Hessian matrix is adjusted to the 4th column in the first weight matrix of N×M dimension; the 2nd column of the original Hessian matrix is adjusted to the 5th column in the first weight matrix of N×M dimension; the 5th column of the original Hessian matrix is adjusted to the 6th column in the first weight matrix of N×M dimension.

[0046] In this embodiment, the mapping relationship between the column numbers of the original Hessian matrix and the first Hessian matrix can also be determined according to the column numbers of the same column in the original Hessian matrix and the first Hessian matrix. Continuing to refer to Figure 2 , there is a mapping relationship between the first column in the original Hessian matrix and the third column in the first Hessian matrix, that is, the first column in the original Hessian matrix and the third column in the first Hessian matrix are the same column, but their column numbers in their respective matrices are different. And so on, there is a mapping relationship between the second column in the original Hessian matrix and the fifth column in the first Hessian matrix; there is a mapping relationship between the third column in the original Hessian matrix and the second column in the first Hessian matrix; there is a mapping relationship between the fourth column in the original Hessian matrix and the fourth column in the first Hessian matrix; there is a mapping relationship between the fifth column in the original Hessian matrix and the sixth column in the first Hessian matrix; there is a mapping relationship between the seventh column in the original Hessian matrix and the first column in the first Hessian matrix.

[0047] In this embodiment, the mapping relationship between the column numbers of the original weight matrix and the first weight matrix can also be determined according to the column numbers of the same column in the original weight matrix and the first weight matrix. Continuing to refer to Figure 2 , there is a mapping relationship between the first column in the original weight matrix and the third column in the first weight matrix, that is, the first column in the original weight matrix and the third column in the first weight matrix are the same column, but their column numbers in their respective matrices are different. And so on, there is a mapping relationship between the second column in the original weight matrix and the fifth column in the first weight matrix; there is a mapping relationship between the third column in the original weight matrix and the second column in the first weight matrix; there is a mapping relationship between the fourth column in the original weight matrix and the fourth column in the first weight matrix; there is a mapping relationship between the fifth column in the original weight matrix and the sixth column in the first weight matrix; there is a mapping relationship between the seventh column in the original weight matrix and the first column in the first weight matrix.

[0048] In this embodiment, after obtaining the first weight matrix and the first Hessian matrix, the first weight matrix and the first Hessian matrix are respectively subjected to matrix block processing column by column to obtain a plurality of first sub-weight matrices and a plurality of sub-Hessian matrices, where the plurality of first sub-weight matrices and the plurality of sub-Hessian matrices correspond one by one.

[0049] Specifically, the first weight matrix and the first Hessian matrix are respectively subjected to matrix block processing according to the same matrix block strategy. In this way, the plurality of first sub-weight matrices obtained by subjecting the first weight matrix to matrix block processing correspond one by one to the plurality of sub-Hessian matrices obtained by subjecting the first Hessian matrix to matrix block processing. Taking Figure 2For example, the first 3 columns in the first weight matrix form a first sub - weight matrix, and the last 3 columns in the first weight matrix form another first sub - weight matrix; the first 3 columns in the first Hessian matrix form a sub - Hessian matrix, and the last 3 columns in the first Hessian matrix form another sub - Hessian matrix. The first sub - weight matrix corresponding to the first 3 columns and the sub - Hessian matrix correspond to each other, and the first sub - weight matrix corresponding to the last 3 columns and the sub - Hessian matrix correspond to each other.

[0050] In this embodiment, it is necessary to perform quantization processing on each of the multiple first sub - weight matrices, and there is no limitation on the quantization processing algorithm. Further optionally, in order to improve the quantization effect, quantization processing can be performed according to formula (2):

[0051]

[0052] where X is a 16 - or 32 - bit floating - point number to be quantized, S and Z respectively represent quantization parameters, b represents the number of bits to be quantized, generally taking 4 or 8. For example, when a 16 - bit floating - point number is quantized into a 4 - bit integer, its storage size becomes 1 / 4 of the original; Q(X) represents the quantization result of X. The clamp() function returns a value between 0 and 2 b - 1.

[0053] The de - quantization operation is an operation that converts a low - order integer back into a high - order floating - point number according to the quantization parameters. The de - quantization operation is mainly used in the inference stage. For example, the de - quantization operation corresponding to the quantization operation represented by formula (2) is:

[0054]

[0055] where represents the de - quantization result obtained by performing the de - quantization operation on Q(X) according to formula (3).

[0056] In formulas (2) and (3), S=(Tmax - Tmin) / (Qmax - Qmin), where Tmax represents the maximum value in the floating - point numbers to be quantized, and Tmin represents the minimum value in the floating - point numbers to be quantized. Z = Qmax - Tmax / S, where Qmax represents the maximum value of Q(X), and Qmin represents the minimum value of Q(X).

[0057] In this embodiment, according to the order of the positions of multiple first sub-weight matrices in the first weight matrix, the first sub-weight matrix or the updated first sub-weight matrix is quantized based on the quantization parameters of the first sub-weight matrix in sequence, to obtain a second sub-weight matrix corresponding to the first sub-weight matrix, and the first quantization error between the first sub-weight matrix and its corresponding second sub-weight matrix is determined based on the sub-Hessian matrix corresponding to the first sub-weight matrix, and other first sub-weight matrices that have not been quantized are updated according to the first quantization error.

[0058] In this embodiment, the smaller the column number of the columns included in the first sub-weight matrix, the more forward the position of the first sub-weight matrix in the first weight matrix, and the first sub-weight matrix is quantized with higher priority. In addition, quantization is performed in units of block matrices, and the same quantization parameters are used for the same first sub-weight matrix. In this way, in the subsequent model inference stage, the quantization parameters used for the same first sub-weight matrix can be loaded in units of block matrices, reducing the number of memory accesses and accelerating the model inference.

[0059] It can be understood that for multiple first sub-weight matrices, each first sub-weight matrix is quantized in sequence. After updating other first sub-weight matrices that have not been quantized using the quantization error of the quantized first sub-weight matrix, the next first sub-weight matrix that has not been quantized is quantized until all the multiple first sub-weight matrices are quantized. It can be understood that if the first sub-weight matrix to be quantized currently is the first first sub-weight matrix, the first sub-weight matrix is quantized. If the first sub-weight matrix to be quantized currently is other first sub-weight matrices after the first first sub-weight matrix, since other first sub-weight matrices that have not been quantized are updated using the quantization error of the quantized first sub-weight matrix, when quantizing other first sub-weight matrices that have not been quantized, the updated other first sub-weight matrices are quantized.

[0060] In practical applications, in the order of the positions of multiple first sub-weight matrices in the first weight matrix, one first sub-weight matrix can be successively used as the current first sub-weight matrix to be quantized among the multiple first sub-weight matrices. Based on the quantization parameters of the current first sub-weight matrix, the current first sub-weight matrix or the updated first sub-weight matrix is quantized to obtain the second sub-weight matrix corresponding to the current first sub-weight matrix. The first quantization error between the current first sub-weight matrix and its corresponding second sub-weight matrix is determined based on the sub-Hessian matrix corresponding to the current first sub-weight matrix. According to the first quantization error, the other first sub-weight matrices that have not been quantized are updated, and the step of successively using one first sub-weight matrix as the current first sub-weight matrix to be quantized among the multiple first sub-weight matrices in the order of the positions of the multiple first sub-weight matrices in the first weight matrix is returned until all the first sub-weight matrices are quantized.

[0061] It should be noted that in this embodiment, the second sub-weight matrix can be understood as the quantization result of the first sub-weight matrix, and the first quantization error refers to the quantization error between the first sub-weight matrix and its corresponding second sub-weight matrix. Exemplarily, the first quantization error δ can be calculated according to formula (4).

[0062]

[0063] Among them, w′ q represents the first sub-weight matrix that needs to be quantized currently, quant() represents the quantization operation, and quant(w′ q ) represents the quantization result obtained by performing the quantization operation on w′ q , that is, the second sub-weight matrix corresponding to the first sub-weight matrix. H′ q represents the inverse matrix of the sub-Hessian matrix corresponding to the first sub-weight matrix.

[0064] After a first sub-weight matrix is quantized, in order to compensate for the quantization loss, the other first sub-weight matrices that have not been quantized will be adjusted and updated. For example, for each matrix element in the other first sub-weight matrices that have not been quantized, various operations such as adding, subtracting, or taking the average of the current value of the matrix element and the first quantization error are performed, and the operation result is used as the final value of the matrix element to update the value of the matrix element in the other first sub-weight matrices that have not been quantized.

[0065] Further optionally, in order to better compensate for the quantization loss, when updating the other first sub-weight matrices that have not been quantized according to the first quantization error, the other first sub-weight matrices can be updated according to the first quantization error and the sub-Hessian matrices corresponding to the other first sub-weight matrices that have not been quantized.

[0066] Specifically, when updating other first sub-weight matrices according to the first quantization error and the sub-Hessian matrices corresponding to other first sub-weight matrices that have not been quantized, the first inverse matrix of the sub-Hessian matrices corresponding to other first sub-weight matrices that have not been quantized can be obtained; the first quantization error and the first inverse matrix are multiplied to obtain a second inverse matrix; various operations such as addition, subtraction, or multiplication are performed on other first sub-weight matrices and the second inverse matrix to obtain the updated other first sub-weight matrices. It can be understood that the first inverse matrix is the inverse matrix of the sub-Hessian matrix corresponding to other first sub-weight matrices that have not been quantized; the second inverse matrix is the multiplication result of multiplying the first quantization error and the first inverse matrix.

[0067] As an example, updating other first sub-weight matrices according to the first quantization error and the sub-Hessian matrices corresponding to other first sub-weight matrices that have not been quantized includes: obtaining the first inverse matrix of the sub-Hessian matrices corresponding to other first sub-weight matrices that have not been quantized; multiplying the first quantization error and the first inverse matrix to obtain a second inverse matrix; subtracting the second inverse matrix from other first sub-weight matrices to obtain the updated other first sub-weight matrices.

[0068] Taking formula (5) as an example below, it is introduced how to update other first sub-weight matrices.

[0069] W′ F =W′ F -δ*(H′ F ) -1 ……(5)

[0070] Among them, W′ F on the left side of the equal sign in formula (5) represents the updated other first sub-weight matrix; W′ F on the right side of the equal sign in formula (5) represents the other first sub-weight matrix before update; (H′ F ) -1 represents the first inverse matrix of the sub-Hessian matrix of other first sub-weight matrices. It can be understood that if other first sub-weight matrices that have not been quantized are updated before quantization, then when quantizing other first sub-weight matrices that have not been quantized, the updated other first sub-weight matrices are used.

[0071] In this embodiment, after completing the quantization operation on the first weight matrix corresponding to the original weight matrix of the network layer that needs to be quantized in the industry large model, that is, after all the first sub-weight matrices of the first weight matrix are completely quantized, the quantization weight matrix of the network layer is determined according to the second sub-weight matrices corresponding to the respective first sub-weight matrices.

[0072] Exemplarily, when determining the quantization weight matrix of a network layer according to the second sub-weight matrices corresponding to multiple first sub-weight matrices, an initial quantization weight matrix of the network layer is generated according to the second sub-weight matrices corresponding to the multiple first sub-weight matrices; and according to the target mapping relationship between the column numbers in the original weight matrix and the column numbers in the first weight matrix, the positions of the columns in the initial quantization weight matrix are reordered to obtain the quantization weight matrix. It can be understood that the columns with the same column numbers in the quantization weight matrix and the original weight matrix correspond to each other.

[0073] In this embodiment, the initial quantization weight matrix includes multiple second sub-weight matrices. The sequence of positions of the multiple second sub-weight matrices in the initial quantization weight matrix is the same as the sequence of positions of the multiple first sub-weight matrices in the first weight matrix. The columns with the same column numbers in the initial quantization weight matrix and the first weight matrix correspond to each other, that is, the columns in the initial quantization weight matrix are the quantization results of the columns with the same column numbers in the first weight matrix. For example, under the first weight matrix of N×M dimension, the initial quantization weight matrix is also a matrix of N×M dimension. The first column in the initial quantization weight matrix is the quantization result of the first column in the first weight matrix of N×M dimension, the second column in the initial quantization weight matrix is the quantization result of the second column in the first weight matrix of N×M dimension, and so on. The columns in the initial quantization weight matrix are the quantization results of the columns with the same column numbers in the first weight matrix.

[0074] In this embodiment, the target mapping relationship refers to the mapping relationship between the column numbers in the original weight matrix and the column numbers in the first weight matrix. The column numbers with a mapping relationship in the original weight matrix and the first weight matrix correspond to the same column. According to the target mapping relationship, the positions of the columns in the initial quantization weight matrix are reordered to obtain the quantization weight matrix. In this way, the columns with the same column numbers in the quantization weight matrix and the original weight matrix can correspond to each other. For example, under the original weight matrix of N×M dimension, the quantization weight matrix is also a matrix of N×M dimension. The first column in the quantization weight matrix is the quantization result of the first column in the original weight matrix of N×M dimension, the second column in the quantization weight matrix is the quantization result of the second column in the original weight matrix of N×M dimension, and so on. The columns in the quantization weight matrix are the quantization results of the columns with the same column numbers in the original weight matrix.

[0075] In this embodiment, since the industry large model includes multiple network layers, after completing the quantization processing of the original weight matrix of a network layer, the input data matrix of this network layer and the quantization weight matrix are multiplied, and the multiplication result is used as the input data matrix of the next network layer to be quantized. The above steps 201 to 204 are repeatedly executed until the quantization processing of all the network layers that need to be quantized in the industry large model is completed.

[0076] The quantization method of the industry large model provided by the embodiments of the present application, when quantizing the original weight matrix of the network layer in the industry large model, the values of the matrix elements on the diagonal of the original Hessian matrix obtained based on the input data matrix of the network layer can represent the activation values. The positions of each column in the original weight matrix and the original Hessian matrix of the network layer are reordered in descending order of the activation values, and the reordered first weight matrix and the first Hessian matrix are respectively subjected to matrix block processing. Taking the block matrix as the granularity, the quantization parameter of each sub-weight matrix in the first weight matrix is used to quantize the first sub-weight matrix in turn, and the unquantized other first sub-weight matrices are updated based on the already quantized first sub-weight matrix. Finally, the quantization result of the original weight matrix of the network layer is determined based on the quantization results of each first sub-weight matrix in the first weight matrix. Thus, a new model quantization method is provided. This method can preferentially quantize important weight coefficients, reduce the quantization error between the original weight matrix and its quantized weight matrix, have a good quantization effect, greatly compress the model size of the industry large model, reduce the deployment cost of the industry large model, accelerate model inference, and ensure the usability of the industry large model.

[0077] In practical applications, through a large number of studies, it is found that there is a small part (accounting for about 0.1%-1%) of the weight coefficients in the original weight matrix of the network layer. If these weight coefficients are not quantized, the impact on the model inference accuracy can be greatly reduced. Here, the weight coefficients with a small proportion but a large impact on the model inference accuracy are called significant weight coefficients. Based on this, the significant weight coefficients in the original weight matrix can also be found, and a sparse weight matrix of the network layer is generated based on the significant weight coefficients in the original weight matrix for improving the model inference accuracy by using the sparse weight matrix during model inference.

[0078] Therefore, on the basis of the above embodiments, after quantizing the first sub-weight matrix or the updated first sub-weight matrix based on the quantization parameter of the first sub-weight matrix in turn according to the order of the positions of the multiple first sub-weight matrices in the first weight matrix to obtain the second sub-weight matrix corresponding to the first sub-weight matrix, the above method may further include: setting the matrix elements at the same positions in the first sub-weight matrix and its corresponding second sub-weight matrix to zero; determining the second quantization error between the zeroed first sub-weight matrix and its corresponding zeroed second sub-weight matrix based on the sub-Hessian matrix corresponding to the first sub-weight matrix; if the difference between the first quantization error and the second quantization error is greater than a preset threshold, determining that the currently zeroed matrix element in the first sub-weight matrix is a significant weight coefficient; and obtaining the sparse weight matrix of the network layer according to the significant weight coefficients in the multiple first sub-weight matrices.

[0079] It should be noted that the matrix elements in the first sub-weight matrix can be traversed in sequence. For the matrix element in the first sub-weight matrix currently being traversed, perform the steps of "setting the matrix elements at the same positions in the first sub-weight matrix and its corresponding second sub-weight matrix to zero; determining the second quantization error between the zeroed first sub-weight matrix and its corresponding zeroed second sub-weight matrix based on the sub-Hessian matrix corresponding to the first sub-weight matrix; if the difference between the first quantization error and the second quantization error is greater than a preset threshold, determining that the matrix element in the first sub-weight matrix that is currently zeroed is a significant weight coefficient" until all the matrix elements in the first sub-weight matrix have been traversed. It can be understood that when determining whether the matrix element in the first sub-weight matrix currently being traversed is a significant weight coefficient, only the matrix element in the first sub-weight matrix that needs to be judged currently is set to zero, while the other matrix elements in the first sub-weight matrix maintain their original values.

[0080] It is also possible not to determine whether the matrix elements in the first sub-weight matrix are significant weight coefficients in order. For any matrix element among all the matrix elements in the first sub-weight matrix, when determining whether the matrix element in the first sub-weight matrix is a significant weight coefficient, only the matrix element in the first sub-weight matrix that needs to be judged currently is set to zero, while the other matrix elements in the first sub-weight matrix maintain their original values. That is, for the matrix element in the first sub-weight matrix that needs to be judged currently, perform the steps of "setting the matrix elements at the same positions in the first sub-weight matrix and its corresponding second sub-weight matrix to zero; determining the second quantization error between the zeroed first sub-weight matrix and its corresponding zeroed second sub-weight matrix based on the sub-Hessian matrix corresponding to the first sub-weight matrix; if the difference between the first quantization error and the second quantization error is greater than a preset threshold, determining that the matrix element in the first sub-weight matrix that is currently zeroed is a significant weight coefficient".

[0081] For example, both the first sub-weight matrix and its corresponding second sub-weight matrix are matrices with a dimension of 7×3 and both include 21 matrix elements. Then, it is necessary to determine whether each matrix element in the first sub-weight matrix is a significant weight coefficient. When determining whether a matrix element in the first sub-weight matrix is a significant weight coefficient, only the matrix element currently to be determined in the first sub-weight matrix is set to zero, while the other matrix elements in the first sub-weight matrix retain their original values. Similarly, for the second sub-weight matrix, the matrix element at the same position as the matrix element currently to be determined in the first sub-weight matrix is set to zero, while the other matrix elements in the second sub-weight matrix retain their original values. For example, if it is necessary to determine whether the matrix element in the first row and the third column of the first sub-weight matrix is a significant weight coefficient, then the matrix elements in the first row and the third column of both the first sub-weight matrix and the second sub-weight matrix are set to zero, and the other matrix elements in the first sub-weight matrix and the second sub-weight matrix retain their original values.

[0082] In this embodiment, the second quantization error is also the quantization error between the first sub-weight matrix after being zeroed and its corresponding second sub-weight matrix after being zeroed. The second quantization error between the first sub-weight matrix after being zeroed and its corresponding second sub-weight matrix after being zeroed can be calculated using formula (4). At this time, w′ q can be understood as the first sub-weight matrix after being zeroed, and quant(w′ q ) is understood as the second sub-weight matrix after being zeroed.

[0083] In this embodiment, the first quantization error is the quantization error between the first sub-weight matrix and its corresponding second sub-weight matrix. If the difference between the first quantization error and the second quantization error is greater than or equal to a preset threshold, it is determined that the matrix element currently zeroed in the first sub-weight matrix is a significant weight coefficient; if the difference between the first quantization error and the second quantization error is less than the preset threshold, it is determined that the matrix element currently zeroed in the first sub-weight matrix is not a significant weight coefficient. Among them, the preset threshold can be flexibly set as needed.

[0084] In this embodiment, after determining the significant weight coefficients in multiple first sub-weight matrices, a sparse weight matrix of the network layer is obtained according to the significant weight coefficients in the multiple first sub-weight matrices.

[0085] Exemplarily, when obtaining a sparse weight matrix of the network layer according to the significant weight coefficients in multiple first sub-weight matrices, an initial sparse weight matrix of the network layer can be obtained according to the significant weight coefficients in the multiple first sub-weight matrices; according to the target mapping relationship, the positions of each column in the initial sparse weight matrix are reordered to obtain the sparse weight matrix of the network layer. It can be understood that the columns with the same column numbers in the sparse weight matrix and the original weight matrix correspond to each other.

[0086] Specifically, after finding the significant weight coefficients in each first sub-weight matrix included in the first weight matrix, set the matrix elements in the first weight matrix other than the significant weight coefficients to zero to obtain the initial sparse weight matrix of the network layer. The initial sparse weight matrix of the network layer is also a matrix with dimensions N×M. According to the target mapping relationship, reorder the positions of each column in the initial sparse weight matrix to obtain the sparse weight matrix, which can make the columns with the same column numbers in the sparse weight matrix and the original weight matrix correspond to each other. For example, referring to Figure 2 , under the original weight matrix with dimensions N×M, the sparse weight matrix is also a matrix with dimensions N×M, and the significant weight coefficients in a certain column of the sparse weight matrix are the significant weight coefficients in the column with the same column number in the original weight matrix.

[0087] After quantifying the industry large model based on the above-mentioned quantization method for the industry large model, the inference process of the industry large model is introduced below.

[0088] Figure 3 It is a flowchart of an inference method for an industry large model provided by an embodiment of the present application. Referring to Figure 3 , the method may include the following steps:

[0089] 301. Obtain the industry data to be processed.

[0090] 302. Input the industry data to be processed into the industry large model so that each network layer in the industry large model performs inference, and obtain the inference result output by the last network layer in the industry large model.

[0091] Among them, when the network layer in the industry large model performs inference, load the quantization weight matrix of the network layer; use the quantization parameters of each column in the quantization weight matrix to perform an inverse quantization operation on the quantization weight matrix to obtain the second weight matrix; multiply the input data matrix of the network layer and the second weight matrix to obtain the output result matrix of the network layer. The input data matrix of the network layer is the industry data to be processed or the output result matrix of the previous network layer of the network layer.

[0092] Specifically, the industry data to be processed can be data in any industry, and there is no restriction on this. For example, the industry data to be processed are government affairs questions such as provident fund, housing, and policies asked by users, financial questions such as financial research reports, stocks, funds, banks, and insurance, power questions such as equipment inspection and power dispatching, or medical questions such as registration, consultation, and ward rounds.

[0093] In this embodiment, the industry large model performs inference layer by layer, and only loads the quantization weight matrix of the currently inferred network layer instead of the quantization weight matrices of all network layers, which greatly reduces the occupancy of video memory space during model inference, saves GPU resources, and accelerates model inference.

[0094] In this embodiment, during the quantization phase, quantization parameters are used to perform quantization operations on weight coefficients of various floating-point types such as float16 or float32 in the original weight matrix of the network layer. Then, during the inference phase, it is necessary to use the quantization parameters to perform dequantization operations on weight coefficients of numerical types such as int4 or int8. For the quantization operations and dequantization operations, reference can be made to the relevant introductions in the foregoing embodiments.

[0095] In this embodiment, by using the quantization parameters of each column in the quantized weight matrix, a dequantization operation is performed on the quantized weight matrix to obtain a second weight matrix. The second weight matrix can be understood as the dequantization result of the quantized weight matrix. In practical applications, if the quantization parameters of the columns in the quantized weight matrix are accessed from memory column by column, the number of memory accesses is relatively large, reducing the model inference speed. Further optionally, the quantization parameters used for the same sub-weight matrix can be loaded in terms of block matrices, reducing the number of memory accesses and accelerating the model inference.

[0096] Based on this, the implementation manner of performing a dequantization operation on the quantized weight matrix by using the quantization parameters of each column in the quantized weight matrix to obtain a second weight matrix is as follows: according to the target mapping relationship, the positions of the columns in the quantized weight matrix are reordered to obtain the initial quantized weight matrix of the network layer, so that the columns with the same column numbers in the initial quantized weight matrix and the first weight matrix correspond to each other; the quantization coefficients of each first sub-weight matrix in the first weight matrix are used to perform a dequantization operation on the corresponding second sub-weight matrix in the initial quantized weight matrix to obtain the initial second weight matrix; according to the target mapping relationship, the positions of the columns in the initial second weight matrix are reordered to obtain the second weight matrix, so that the columns with the same column numbers in the second weight matrix and the original weight matrix correspond to each other.

[0097] Specifically, the target mapping relationship refers to the mapping relationship between the column numbers in the original weight matrix and the column numbers in the first weight matrix. The columns with corresponding column numbers in the original weight matrix and the first weight matrix correspond to the same column, and the columns with the same column numbers in the quantized weight matrix and the original weight matrix correspond to each other. Therefore, by reordering the positions of the columns in the quantized weight matrix based on the target mapping relationship, the initial quantized weight matrix of the network layer can be obtained. The initial quantized weight matrix includes multiple second sub-weight matrices. The order of the multiple second sub-weight matrices in the initial quantized weight matrix is the same as the order of the multiple first sub-weight matrices in the first weight matrix. The columns with the same column numbers in the initial quantized weight matrix and the first weight matrix correspond to each other, that is, the columns in the initial quantized weight matrix are the quantization results of the columns with the same column numbers in the first weight matrix.

[0098] In the quantization stage of the industry large model, when performing quantization processing at the granularity of block matrices, the quantization parameters used by each first sub-weight matrix can be recorded. In this way, in the model inference stage, the quantization parameters used by the same first sub-weight matrix can be loaded at the granularity of block matrices, reducing the number of memory accesses and accelerating the model inference. Therefore, in this embodiment, the quantization coefficients of each first sub-weight matrix in the first weight matrix are loaded, and the corresponding second sub-weight matrix in the initial quantized weight matrix is dequantized using the quantization coefficients of each first sub-weight matrix to obtain the initial second weight matrix. It can be understood that the initial second weight matrix is the dequantization result of the initial quantized weight matrix. After obtaining the initial second weight matrix, the positions of the columns in the initial second weight matrix are reordered according to the target mapping relationship to obtain the second weight matrix, so that the columns with the same column numbers in the second weight matrix and the original weight matrix correspond to each other. It can be understood that the weight coefficients in the second weight matrix are obtained by sequentially performing quantization operations and dequantization operations on the weight coefficients in the original weight matrix. The second weight matrix is essentially very close to the original weight matrix, and the weight coefficients in the second weight matrix are also floating-point numbers.

[0099] In this embodiment, when the network layer in the industry large model performs inference, after obtaining the second weight matrix based on the quantized weight matrix of the network layer, the input data matrix of the network layer and the second weight matrix are multiplied to obtain the output result matrix of the network layer. The input data matrix of the network layer is the industry data to be processed or the output result matrix of the previous network layer of the network layer.

[0100] Further optionally, in the quantization stage of the industry large model, not only the original weight matrix of the network layer is quantized to obtain the quantized weight matrix of the network layer, but also the sparse weight matrix of the network layer is obtained. The significant weight coefficients in a certain column of the sparse weight matrix are the significant weight coefficients in the column with the same column number in the original weight matrix. Based on this, in the inference stage of the industry large model, the implementation method of multiplying the input data matrix of the network layer and the second weight matrix to obtain the output result matrix of the network layer is as follows: The second weight matrix is updated according to the sparse weight matrix of the network layer to obtain the final second weight matrix; the input data matrix of the network layer and the final second weight matrix are multiplied to obtain the output result matrix of the network layer.

[0101] Specifically, when updating the second weight matrix according to the sparse weight matrix of the network layer, obtain the significant weight coefficients in the sparse weight matrix, and update the value of the matrix element in the second weight matrix that has the same position as the significant weight coefficient in the sparse weight matrix to the significant weight coefficient. For example, both the second weight matrix and the sparse weight matrix are matrices with a dimension of 7×3. If the value of the matrix element in the 3rd row and 1st column of the sparse weight matrix is the significant weight coefficient, then update the value of the matrix element in the 3rd row and 1st column of the second weight matrix to the value of the matrix element in the 3rd row and 1st column of the sparse weight matrix (i.e., the significant weight coefficient). If the value of the matrix element in the 4th row and 2nd column of the sparse weight matrix is the significant weight coefficient, then update the value of the matrix element in the 4th row and 2nd column of the second weight matrix to the value of the matrix element in the 4th row and 2nd column of the sparse weight matrix (i.e., the significant weight coefficient).

[0102] In this embodiment, the second weight matrix is updated according to the sparse weight matrix of the network layer to obtain the final second weight matrix. It can be understood that the weight coefficients in the final second weight matrix not only retain the significant weight coefficients in the original weight matrix but also record the weight coefficients obtained by sequentially performing quantization operations and inverse quantization operations on the weight coefficients in the original weight matrix. After obtaining the final second weight matrix, multiply the input data matrix of the network layer by the final second weight matrix to obtain the output result matrix of the network layer. Since the final second weight matrix retains the significant weight coefficients in the original weight matrix, considering these significant weight coefficients can effectively improve the model inference accuracy.

[0103] The inference method of the industry large model provided by the embodiments of the present application uses a quantized industry large model for inference. On the one hand, since the quantized industry large model greatly compresses the model size and reduces the deployment cost of the industry large model, thus, the quantized industry large model accelerates model inference and ensures the usability of the industry large model. In addition, the industry large model performs inference layer by layer, only loading the quantized weight matrix of the currently inferred network layer instead of the quantized weight matrices of all network layers, which greatly reduces the occupancy of video memory space during model inference, saves GPU resources, and accelerates model inference.

[0104] Figure 4 It is a flowchart of another inference method of the industry large model provided by the embodiments of the present application. Applied to a cloud server, the cloud server deploys an industry large model, see Figure 4 , and the method may include the following steps:

[0105] 401. Receive the industry data to be processed sent by the terminal device.

[0106] 402. Input the industry data to be processed into the industry large model for inference by each network layer in the industry large model, and obtain the inference result output by the last network layer in the industry large model.

[0107] 403. Return the inference result to the terminal device.

[0108] Among them, when the network layer in the industry large model performs inference, it loads the quantization weight matrix of the network layer; uses the quantization parameters of each column in the quantization weight matrix to perform an inverse quantization operation on the quantization weight matrix to obtain the second weight matrix; multiplies the input data matrix of the network layer and the second weight matrix to obtain the output result matrix of the network layer. The input data matrix of the network layer is the industry data to be processed or the output result matrix of the previous network layer of the network layer.

[0109] For the quantization process of the industry large model of the cloud server, refer to the relevant introduction in the foregoing embodiments and will not be elaborated here.

[0110] Further optionally, multiplying the input data matrix of the network layer and the second weight matrix to obtain the output result matrix of the network layer includes: updating the second weight matrix according to the sparse weight matrix of the network layer to obtain the final second weight matrix; multiplying the input data matrix of the network layer and the final second weight matrix to obtain the output result matrix of the network layer.

[0111] Further optionally, using the quantization parameters of each column in the quantization weight matrix to perform an inverse quantization operation on the quantization weight matrix to obtain the second weight matrix includes: reordering the positions of each column in the quantization weight matrix according to the target mapping relationship to obtain the initial quantization weight matrix of the network layer, so that the columns with the same column numbers in the initial quantization weight matrix and the first weight matrix correspond to each other; using the quantization coefficients of each first sub-weight matrix in the first weight matrix to perform an inverse quantization operation on the corresponding second sub-weight matrix in the initial quantization weight matrix to obtain the initial second weight matrix; reordering the positions of each column in the initial second weight matrix according to the target mapping relationship to obtain the second weight matrix, so that the columns with the same column numbers in the second weight matrix and the original weight matrix correspond to each other.

[0112] For the process of the cloud server using the industry large model for inference, refer to the relevant introduction in the foregoing embodiments and will not be elaborated here.

[0113] In the embodiments of the present application, the terminal device can interact with the cloud server through a wired network or a wireless network. For example, the wired network can include coaxial cables, twisted pairs, optical fibers, etc., and the wireless network can be a 2G (2 Generation) network, a 3G (3 Generation) network, a 4G (4 Generation) network, a 5G (5 Generation) network, a Wireless Fidelity (WIFI) network, etc. The present application does not limit the specific type or form of the interaction, as long as it can implement the function of the interaction between the terminal device and the cloud server.

[0114] Among them, the terminal device can be hardware or software. When the terminal device is hardware, the terminal device is, for example, a mobile phone, a tablet computer, a desktop computer, a wearable intelligent device, a smart home device, etc. When the terminal device is software, it can be installed in the above-listed hardware devices. At this time, the terminal device is, for example, multiple software modules or a single software module, etc., and the embodiments of the present application do not limit. The cloud server can be hardware or software. When the cloud server is hardware, the cloud server is a single server or a distributed server cluster composed of multiple servers. When the cloud server is software, it can be multiple software modules or a single software module, etc., and the embodiments of the present application do not limit.

[0115] The inference method of the industry large model provided by the embodiments of the present application uses the quantized industry large model for inference. On the one hand, since the quantized industry large model greatly compresses the model size and reduces the deployment cost of the industry large model. Therefore, the quantized industry large model accelerates the model inference and ensures the availability of the industry large model. In addition, the industry large model performs inference layer by layer, only loading the quantized weight matrix of the currently inferred network layer instead of the quantized weight matrices of all network layers, which greatly reduces the occupancy of video memory space during model inference, saves GPU resources, and accelerates model inference.

[0116] Figure 5 This is an exemplary application scenario diagram provided by the embodiments of the present application. Refer to Figure 5 , first, use the training data set to train the model to obtain a government affairs large model for government affairs processing. Then, use the calibration data set to quantize the government affairs large model to obtain a quantized government affairs large model. Finally, deploy the quantized government affairs large model in the cloud server.

[0117] In the actual model inference stage, refer to Figure 5As shown in ①, the user triggers the terminal device to send problem information related to government affairs to the cloud server. For example, problems related to the provident fund withdrawal policy or problems related to vehicle transfer procedures, etc. The cloud server calls the government affairs large model to reply to the problem information related to government affairs and obtains the reply content. See Figure 5 As shown in ②, the cloud server returns the reply content to the terminal device for the user to know. Since the government affairs large model is quantized, the cloud server can efficiently and accurately interact with the user in an intelligent question and answer manner.

[0118] It should be noted that Figure 5 The application scenario shown is only an exemplary application scenario, and the embodiments of the present application do not limit the application scenario. The embodiments of the present application do not Figure 5 limit the devices included in Figure 5 nor the positional relationship between the devices in

[0119] Figure 6 It is a schematic structural diagram of a quantization device for an industry large model provided by an embodiment of the present application. The industry large model includes a plurality of network layers connected in sequence. See Figure 6 This device may include:

[0120] The Hessian matrix determination module 61 is used to determine the original Hessian matrix according to the input data matrix of the network layer for the network layer that needs to be quantized among the plurality of network layers;

[0121] The matrix rearrangement module 62 is used to reorder the positions of each column in the original weight matrix and the original Hessian matrix of the network layer respectively according to the descending order of the values of the matrix elements on the diagonal of the original Hessian matrix, and obtain the first weight matrix and the first Hessian matrix;

[0122] The matrix block module 63 is used to perform matrix block processing on the first weight matrix and the first Hessian matrix respectively by column to obtain a plurality of first sub-weight matrices and a plurality of sub-Hessian matrices, where the plurality of first sub-weight matrices correspond to the plurality of sub-Hessian matrices one by one;

[0123] The quantization module 64 is used to sequentially perform quantization processing on the first sub-weight matrix or the updated first sub-weight matrix based on the quantization parameters of the first sub-weight matrix in the order of the positions of the plurality of first sub-weight matrices in the first weight matrix, obtain the second sub-weight matrix corresponding to the first sub-weight matrix, and determine the first quantization error between the first sub-weight matrix and its corresponding second sub-weight matrix based on the sub-Hessian matrix corresponding to the first sub-weight matrix, and update the other first sub-weight matrices that have not been quantized according to the first quantization error; determine the quantization weight matrix of the network layer according to the second sub-weight matrices corresponding to the plurality of first sub-weight matrices respectively.

[0124] Further optionally, when the quantization module 64 updates other first sub-weight matrices that have not been quantized according to the first quantization error, it is specifically used for: updating other first sub-weight matrices according to the first quantization error and the sub-Hessian matrix corresponding to other first sub-weight matrices that have not been quantized.

[0125] Further optionally, when the quantization module 64 updates other first sub-weight matrices according to the first quantization error and the sub-Hessian matrix corresponding to other first sub-weight matrices that have not been quantized, it is specifically used for: obtaining the first inverse matrix of the sub-Hessian matrix corresponding to other first sub-weight matrices that have not been quantized; multiplying the first quantization error by the first inverse matrix to obtain a second inverse matrix; subtracting the second inverse matrix from other first sub-weight matrices to obtain the updated other first sub-weight matrices.

[0126] Further optionally, when the quantization module 64 determines the quantization weight matrix of the network layer according to the second sub-weight matrices corresponding to multiple first sub-weight matrices, it is specifically used for: generating an initial quantization weight matrix of the network layer according to the second sub-weight matrices corresponding to multiple first sub-weight matrices; reordering the positions of each column in the initial quantization weight matrix according to the target mapping relationship between the column numbers in the original weight matrix and the column numbers in the first weight matrix to obtain the quantization weight matrix.

[0127] Further optionally, after obtaining the second sub-weight matrix corresponding to the first sub-weight matrix, the quantization module 64 is further used for: setting the matrix elements at the same positions in the first sub-weight matrix and its corresponding second sub-weight matrix to zero; determining the second quantization error between the zeroed first sub-weight matrix and its corresponding zeroed second sub-weight matrix based on the sub-Hessian matrix corresponding to the first sub-weight matrix; if the difference between the first quantization error and the second quantization error is greater than a preset threshold, determining that the matrix element currently set to zero in the first sub-weight matrix is a significant weight coefficient; obtaining the sparse weight matrix of the network layer according to the significant weight coefficients in multiple first sub-weight matrices.

[0128] Further optionally, when the quantization module 64 obtains the sparse weight matrix of the network layer according to the significant weight coefficients in multiple first sub-weight matrices, it is specifically used for: obtaining the initial sparse weight matrix of the network layer according to the significant weight coefficients in multiple first sub-weight matrices; reordering the positions of each column in the initial sparse weight matrix according to the target mapping relationship to obtain the sparse weight matrix of the network layer.

[0129] Further optionally, the Hessian matrix determination module 61 is specifically used for: multiplying the input data matrix by its transpose matrix to obtain a multiplication result; determining the original Hessian matrix according to the multiplication result, the identity matrix, and a specified coefficient, where the specified coefficient is used to avoid numerical problems generated when calculating the inverse matrix of the Hessian matrix.

[0130] Further optionally, the industry large model is used for intelligent question answering in the target industry, and the target industry includes at least one of the following: government affairs industry, financial industry, power industry, and medical industry.

[0131] Figure 6 The shown device can execute Figure 1 the method shown in the illustrated embodiment, and its implementation principle and technical effects will not be elaborated further. For the Figure 6 shown device in the above embodiment, the specific manners in which each module and unit perform operations have been described in detail in the embodiment related to the method, and will not be elaborated here.

[0132] Figure 7 It is a schematic structural diagram of an inference device for an industry large model provided by an embodiment of the present application. The industry large model has been quantized by the quantization method of the foregoing industry large model. Refer to Figure 7 and the device may include:

[0133] An acquisition module 71, configured to acquire industry data to be processed;

[0134] An inference module 72, configured to input the industry data to be processed into the industry large model, so that each network layer in the industry large model performs inference, and obtain an inference result output by the last network layer in the industry large model;

[0135] Wherein, when performing inference, the network layer in the industry large model loads the quantized weight matrix of the network layer;

[0136] Using the quantization parameters of each column in the quantized weight matrix, perform an inverse quantization operation on the quantized weight matrix to obtain a second weight matrix;

[0137] Multiply the input data matrix of the network layer and the second weight matrix to obtain an output result matrix of the network layer, and the input data matrix of the network layer is the industry data to be processed or the output result matrix of the previous network layer of the network layer.

[0138] Further optionally, when the inference module 72 multiplies the input data matrix of the network layer and the second weight matrix to obtain the output result matrix of the network layer, it is specifically configured to: update the second weight matrix according to the sparse weight matrix of the network layer to obtain a final second weight matrix; multiply the input data matrix of the network layer and the final second weight matrix to obtain the output result matrix of the network layer.

[0139] Further optionally, when the inference module 72 uses the quantization parameters of each column in the quantized weight matrix to perform an inverse quantization operation on the quantized weight matrix to obtain the second weight matrix, it is specifically used for: reordering the positions of each column in the quantized weight matrix according to the target mapping relationship to obtain the initial quantized weight matrix of the network layer, so that the columns with the same column numbers in the initial quantized weight matrix and the first weight matrix correspond to each other; using the quantization coefficients of each first sub-weight matrix in the first weight matrix to perform an inverse quantization operation on the corresponding second sub-weight matrix in the initial quantized weight matrix to obtain the initial second weight matrix; reordering the positions of each column in the initial second weight matrix according to the target mapping relationship to obtain the second weight matrix, so that the columns with the same column numbers in the second weight matrix and the original weight matrix correspond to each other.

[0140] Figure 7 The device shown can execute Figure 3 the method shown in the embodiment, and its implementation principle and technical effects will not be elaborated. For the Figure 7 device shown in the above embodiment, the specific ways in which each module and unit perform operations have been described in detail in the embodiment related to the method, and will not be elaborated here.

[0141] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 101 to 104 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of steps 103 and 104 can be device B; and so on.

[0142] In addition, in some of the processes described in the above embodiments and the accompanying drawings, there are multiple operations that appear in a specific order, but it should be clearly understood that these operations can be executed not in the order in which they appear in this article or in parallel. The operation numbers such as 401 and 402 are only used to distinguish different operations, and the numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and these operations can be executed in sequence or in parallel. It should be noted that the descriptions such as "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit that "first" and "second" are of different types.

[0143] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0144] Figure 8 The following is a schematic structural diagram of a computer device provided by an embodiment of this application. As Figure 8 shown, the computer device includes: a memory 81 and a processor 82;

[0145] The memory 81 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application program or method for operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.

[0146] The memory 81 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc.

[0147] The processor 82 is coupled to the memory 81 and is used to execute the computer program in the memory 81 for: executing the steps in the quantization method of the industry large model or the inference method of the industry large model.

[0148] Further optionally, as Figure 8 shown, the computer device further includes: other components such as a communication component 83, a display 84, a power supply component 85, an audio component 86, etc. Figure 8 Only some components are schematically shown in Figure 8 and it does not mean that the computer device only includes Figure 8The components within the dashed-line box are optional components, rather than essential components, and can be determined according to the product form of the computer device. The computer device in this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT (Internet of Things) device, or can be a server device such as a conventional server, a cloud server, or a server array. If the computer device in this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, or a smart phone, it may include Figure 8 the components within the dashed-line box; if the computer device in this embodiment is implemented as a server device such as a conventional server, a cloud server, or a server array, it may not include Figure 8 the components within the dashed-line box.

[0149] For the detailed implementation process of the processor to execute each action, reference can be made to the relevant descriptions in the foregoing method embodiment or device embodiment, and details are not described herein again.

[0150] Correspondingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed, it can implement each step executable by the computer device in the foregoing method embodiment.

[0151] Correspondingly, an embodiment of the present application further provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the processor can be caused to implement each step executable by the computer device in the foregoing method embodiment.

[0152] The above communication component is configured to facilitate communication between the device where the communication component is located and other devices in a wired or wireless manner. The device where the communication component is located can access a communication standard-based wireless network, such as a mobile communication network like WiFi (Wireless Fidelity), 2G (2 Generation), 3G (3 Generation), 4G (4 Generation) / LTE (long Term Evolution), 5G (5 Generation), etc., or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, The Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0153] The above display includes a screen, and the screen can include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation.

[0154] The above power supply component provides power to various components of the device where the power supply component is located. The power supply component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the device where the power supply component is located.

[0155] The above audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC). When the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive external audio signals. The received audio signals can be further stored in a memory or transmitted via the communication component. In some embodiments, the audio component further includes a speaker for outputting audio signals.

[0156] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0157] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the specified functions in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0158] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the specified functions in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the specified functions in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0160] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), an input / output interface, a network interface, and a memory.

[0161] Memory may include non-permanent memory in the form of computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0162] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transitory media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

[0163] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or apparatus comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising the element.

[0164] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A quantization method for an industry large model, characterized in that, The industry large model includes multiple network layers connected in sequence, and the method includes: For the network layers that need to be quantized in the multiple network layers, determine the original Hessian matrix according to the input data matrix of the network layer; According to the descending order of the values of the matrix elements on the diagonal of the original Hessian matrix, reorder the positions of the columns in the original weight matrix and the original Hessian matrix of the network layer respectively to obtain a first weight matrix and a first Hessian matrix, and perform matrix block processing on the first weight matrix and the first Hessian matrix by column respectively to obtain multiple first sub-weight matrices and multiple sub-Hessian matrices, where the multiple first sub-weight matrices correspond to the multiple sub-Hessian matrices one by one; In the order of the positions of the multiple first sub-weight matrices in the first weight matrix, sequentially perform quantization processing on the first sub-weight matrix or the updated first sub-weight matrix based on the quantization parameters of the first sub-weight matrix to obtain a second sub-weight matrix corresponding to the first sub-weight matrix, and determine the first quantization error between the first sub-weight matrix and its corresponding second sub-weight matrix based on the sub-Hessian matrix corresponding to the first sub-weight matrix, and update the other first sub-weight matrices that have not been quantized according to the first quantization error; Determine the quantized weight matrix of the network layer according to the second sub-weight matrices corresponding to the multiple first sub-weight matrices respectively.

2. The method according to claim 1, characterized in that, Updating the other first sub-weight matrices that have not been quantized according to the first quantization error includes: Updating the other first sub-weight matrices according to the first quantization error and the sub-Hessian matrices corresponding to the other first sub-weight matrices that have not been quantized.

3. The method according to claim 2, characterized in that, Updating the other first sub-weight matrices according to the first quantization error and the sub-Hessian matrices corresponding to the other first sub-weight matrices that have not been quantized includes: Obtain the first inverse matrix of the sub-Hessian matrix corresponding to the other first sub-weight matrices that have not been quantized; Multiply the first quantization error and the first inverse matrix to obtain a second inverse matrix; Subtract the other first sub-weight matrices from the second inverse matrix to obtain the updated other first sub-weight matrices.

4. The method according to claim 1, characterized in that, Determining the quantized weight matrix of the network layer according to the second sub-weight matrices corresponding to the multiple first sub-weight matrices respectively includes: Generate an initial quantized weight matrix of the network layer according to the second sub-weight matrices corresponding to the multiple first sub-weight matrices; Reorder the positions of the columns in the initial quantized weight matrix according to the target mapping relationship between the column numbers in the original weight matrix and the column numbers in the first weight matrix to obtain the quantized weight matrix.

5. The method according to any one of claims 1 to 4, characterized in that, After obtaining the second sub-weight matrix corresponding to the first sub-weight matrix, it further includes: Set the matrix elements at the same positions in the first sub-weight matrix and its corresponding second sub-weight matrix to zero; Determine the second quantization error between the zeroed first sub-weight matrix and its corresponding zeroed second sub-weight matrix based on the sub-Hessian matrix corresponding to the first sub-weight matrix; If the difference between the first quantization error and the second quantization error is greater than a preset threshold, it is determined that the currently zeroed matrix element in the first sub-weight matrix is a significant weight coefficient; Based on the significant weight coefficients in multiple first sub-weight matrices, obtain the sparse weight matrix of the network layer.

6. The method according to claim 5, characterized in that, Obtaining the sparse weight matrix of the network layer according to the significant weight coefficients in multiple first sub-weight matrices includes: Based on the significant weight coefficients in multiple first sub-weight matrices, obtain the initial sparse weight matrix of the network layer; According to the target mapping relationship, reorder the positions of the columns in the initial sparse weight matrix to obtain the sparse weight matrix of the network layer.

7. The method according to any one of claims 1 to 4, characterized in that, Determining the original Hessian matrix according to the input data matrix of the network layer includes: Multiply the input data matrix and its transpose matrix to obtain a multiplication result; According to the multiplication result, the identity matrix, and a specified coefficient, determine the original Hessian matrix, where the specified coefficient is used to avoid numerical problems generated when calculating the inverse matrix of the Hessian matrix.

8. The method according to any one of claims 1 to 4, characterized in that, The industry large model is used for intelligent question answering in the target industry, and the target industry includes at least one of the following: government affairs industry, financial industry, power industry, and medical industry.

9. An inference method for an industry large model, characterized in that, The industry large model is quantized by the method according to any one of claims 1 to 8, and the method includes: Obtain the industry data to be processed; Input the industry data to be processed into the industry large model, so that each network layer in the industry large model performs inference, and obtain the inference result output by the last network layer in the industry large model; Wherein, when the network layer in the industry large model performs inference, load the quantized weight matrix of the network layer; Use the quantization parameters of each column in the quantized weight matrix to perform an inverse quantization operation on the quantized weight matrix to obtain a second weight matrix; Multiply the input data matrix of the network layer and the second weight matrix to obtain the output result matrix of the network layer, and the input data matrix of the network layer is the industry data to be processed or the output result matrix of the previous network layer of the network layer.

10. The method according to claim 9, wherein Multiplying the input data matrix of the network layer and the second weight matrix to obtain the output result matrix of the network layer includes: Update the second weight matrix according to the sparse weight matrix of the network layer to obtain the final second weight matrix; Multiply the input data matrix of the network layer and the final second weight matrix to obtain the output result matrix of the network layer.

11. The method according to claim 9, wherein Using the quantization parameters of each column in the quantized weight matrix to perform an inverse quantization operation on the quantized weight matrix to obtain a second weight matrix includes: According to the target mapping relationship, reorder the positions of the columns in the quantized weight matrix to obtain the initial quantized weight matrix of the network layer, so that the columns with the same column numbers in the initial quantized weight matrix and the first weight matrix correspond to each other; Use the quantization coefficients of each first sub-weight matrix in the first weight matrix to perform an inverse quantization operation on the corresponding second sub-weight matrix in the initial quantized weight matrix to obtain an initial second weight matrix; Reorder the positions of the columns in the initial second weight matrix according to the target mapping relationship to obtain the second weight matrix, so that the columns with the same column number in the second weight matrix and the original weight matrix correspond to each other.

12. An inference method for an industry large model, wherein Applied to a cloud server that deploys an industry large model, and the industry large model has been quantized by the method according to any one of claims 1 to 8. The method includes: Receiving industry data to be processed sent by a terminal device; Inputting the industry data to be processed into the industry large model, so that each network layer in the industry large model performs inference, and obtaining the inference result output by the last network layer in the industry large model; Returning the inference result to the terminal device; Wherein, when a network layer in the industry large model performs inference, it loads the quantized weight matrix of the network layer; uses the quantization parameters of each column in the quantized weight matrix to perform an inverse quantization operation on the quantized weight matrix to obtain a second weight matrix; multiplies the input data matrix of the network layer and the second weight matrix to obtain the output result matrix of the network layer, and the input data matrix of the network layer is the industry data to be processed or the output result matrix of the previous network layer of the network layer.

13. A computer device, wherein Including: A memory and a processor; The memory is used to store a computer program; The processor is coupled to the memory and is used to execute the computer program to perform the steps in the method according to any one of claims 1-12.

14. A computer-readable storage medium storing a computer program, wherein When the computer program is executed by the processor, it causes the processor to be able to implement the steps in the method according to any one of claims 1-12.

Images