Quantitative model-based reasoning method and model quantification method

Abstract

The invention discloses a reasoning method based on a quantitative model, the quantitative model is obtained based on a trained model, the precision of at least part of parameters of the quantitative model is smaller than the parameter precision of the trained model, and the method comprises the following steps: obtaining input task data; inputting the input task data into the quantitative model to obtain an output result; the quantitative model comprises multiple layers of neuron sub-models, and for a first sub-model and a second sub-model of adjacent layers, the output activation value of the first sub-model is used as the input of the second sub-model; under the condition that a second sub-model of a specific adjacent layer of the quantitative model comprises a fusion weight matrix parameter, a specific rotation matrix is controlled to act on an output activation value of the first sub-model to obtain a modified activation value, and the modified activation value is smaller than the output activation value; the fusion weight matrix parameter is obtained by the specific rotation matrix acting on the second initial weight parameter of the second sub-model of the specific adjacent layer corresponding to the trained model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and involve but are not limited to reasoning methods and model quantization methods based on quantitative models. Background Art

[0002] Artificial intelligence is deeply empowering diverse scenarios and is increasingly being used in learning, work, and daily life. On Android devices, users are increasingly looking forward to the implementation of AI and other technologies, as well as closer interactions with end-devices, to provide more convenient and intelligent operations and experiences. However, current state-of-the-art large language models (LLMs) often contain hundreds of millions or even hundreds of billions of parameters, which consumes significant disk space and on-chip computing power.

[0003] In order to reduce dependence on disk space and chip computing power and successfully implement it on resource-limited end-side devices, it is very necessary to miniaturize and lightweight the model.

[0004] Model quantization is currently one of the most popular approaches for model miniaturization. In machine learning, quantizing large language models is an optimization technique aimed at reducing model memory usage and improving inference speed, while maintaining model accuracy. Model quantization involves converting the weights and activation values ​​of a trained deep neural network from high-precision to low-precision.

[0005] However, the current process of compressing high-bit data (such as 32 bits) to low bits (such as 16 bits) through model quantization technology (hereinafter referred to as quantization) of large language models is usually lossy compression, introducing quantization loss (also known as quantization noise). Moreover, as the number of bits representing the data after compression decreases, the loss will also increase. In the process of quantizing model weight parameters and activation parameters, the presence of outliers will lead to large quantization losses. How to minimize the quantization loss caused by outliers in activation values ​​has become a technical problem that needs to be solved urgently. Summary of the Invention

[0006] In view of this, embodiments of the present application provide an inference method based on a quantization model and a model quantization method.

[0007] The technical solution of the embodiment of the present application is implemented as follows:

[0008] In a first aspect, an embodiment of the present application provides an inference method based on a quantization model, where the quantization model is obtained based on a trained model, and the accuracy of at least some parameters of the quantization model is less than the parameter accuracy of the trained model. The method includes:

[0009] Obtaining input task data, where the input task data includes at least one of text data, voice data, image data, and audio and video data;

[0010] Input the input task data into the quantitative model to obtain the output result;

[0011] The quantization model includes a multi-layer neuron sub-model, and for a first sub-model and a second sub-model in adjacent layers, the output activation value of the first sub-model serves as the input of the second sub-model;

[0012] In which, when the second sub-model of a specific adjacent layer of the quantization model includes fusion weight matrix parameters, the specific rotation matrix is ​​controlled to act on the output activation value of the first sub-model to obtain a modified activation value, the modified activation value is less than the output activation value, and the fusion weight matrix parameters are obtained by the specific rotation matrix acting on the second initial weight parameters of the second sub-model of the specific adjacent layer corresponding to the trained model.

[0013] In a second aspect, an embodiment of the present application provides a model quantization method, including:

[0014] Obtaining task data, where the task data includes at least one of text data, voice data, image data, and audio and video data;

[0015] Input task data into the trained model;

[0016] The trained model includes a multi-layer neuron sub-model, and for a first sub-model and a second sub-model in adjacent layers, the output activation value of the first sub-model serves as the input of the second sub-model;

[0017] For a specific adjacent layer whose output activation value satisfies the outlier activation value condition, applying a specific rotation matrix to the output activation value of the first submodel of the specific adjacent layer to obtain a modified activation value, where the modified activation value is less than the output activation value;

[0018] Applying a specific rotation matrix to a second initial weight parameter of a second sub-model of a specific adjacent layer to obtain a fusion weight matrix parameter of the second sub-model;

[0019] Quantize and store the fusion weight matrix parameters including the second sub-model to obtain a quantized model, wherein the accuracy of at least part of the parameters of the quantized model is less than the accuracy of the parameters of the trained model.

[0020] In a third aspect, an embodiment of the present application provides an inference device based on a quantization model, where the quantization model is obtained based on a trained model, and the accuracy of at least some parameters of the quantization model is less than the parameter accuracy of the trained model. The device includes:

[0021] A first obtaining module is used to obtain input task data, where the input task data includes at least one of text data, voice data, image data, and audio and video data;

[0022] The data processing module is used to input the input task data into the quantitative model to obtain the output result;

[0023] The quantization model includes a multi-layer neuron sub-model, and for a first sub-model and a second sub-model in adjacent layers, the output activation value of the first sub-model serves as the input of the second sub-model;

[0024] In which, when the second sub-model of a specific adjacent layer of the quantization model includes fusion weight matrix parameters, the specific rotation matrix is ​​controlled to act on the output activation value of the first sub-model to obtain a modified activation value, the modified activation value is less than the output activation value, and the fusion weight matrix parameters are obtained by the specific rotation matrix acting on the second initial weight parameters of the second sub-model of the specific adjacent layer corresponding to the trained model.

[0025] In a fourth aspect, an embodiment of the present application provides a model quantization device, comprising:

[0026] A second acquisition module is used to obtain task data, where the task data includes at least one of text data, voice data, image data, and audio and video data;

[0027] A data input module, used to input task data into the trained model;

[0028] The trained model includes a multi-layer neuron sub-model, and for a first sub-model and a second sub-model in adjacent layers, the output activation value of the first sub-model serves as the input of the second sub-model;

[0029] a first action module, configured to apply a specific rotation matrix to the output activation value of the first submodel of the specific adjacent layer whose output activation value satisfies the outlier activation value condition, to obtain a modified activation value, wherein the modified activation value is less than the output activation value;

[0030] A second action module is used to apply a specific rotation matrix to a second initial weight parameter of a second sub-model of a specific adjacent layer to obtain a fusion weight matrix parameter of the second sub-model;

[0031] The first quantization storage module is used to quantize and store the fusion weight matrix parameters including the second sub-model to obtain a quantized model, and the accuracy of at least some parameters of the quantized model is less than the accuracy of the parameters of the trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a quantitative model-based reasoning method provided in an embodiment of the present application;

[0033] Figure 2 A schematic diagram of an implementation flow of a model quantization method provided in an embodiment of the present application;

[0034] Figure 3A A schematic diagram of an implementation flow for obtaining an input activation value of a second sub-model provided in an embodiment of the present application;

[0035] Figure 3B A schematic diagram of an implementation flow of a method for obtaining a quantitative model provided in an embodiment of the present application;

[0036] Figure 4A A schematic diagram of activation value quantization provided in an embodiment of the present application;

[0037] Figure 4B The embodiment of the present application provides a schematic diagram of an implementation flow of orthogonal transformation processing outliers;

[0038] Figure 4C A schematic diagram of a matrix rotation transformation provided in an embodiment of the present application;

[0039] Figure 5A A schematic diagram of the structure of an inference device based on a quantitative model provided in an embodiment of the present application;

[0040] Figure 5B A schematic diagram of the structure of a model quantization device provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] To make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the specific technical solutions of the embodiments of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0043] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0046] The embodiment of the present application provides an inference method based on a quantization model, wherein the quantization model is obtained based on a trained model, and the accuracy of at least some parameters of the quantization model is less than the parameter accuracy of the trained model, such as Figure 1 As shown, the method includes:

[0047] Step S110: obtaining input task data, wherein the input task data includes at least one of text data, voice data, image data, and audio and video data;

[0048] Step S120: inputting the input task data into the quantitative model to obtain an output result;

[0049] The quantization model includes a multi-layer neuron sub-model, and for a first sub-model and a second sub-model in adjacent layers, the output activation value of the first sub-model serves as the input of the second sub-model;

[0050] In which, when the second sub-model of the specific adjacent layer of the quantization model includes fusion weight matrix parameters, the specific rotation matrix is ​​controlled to act on the output activation value of the first sub-model to obtain a modified activation value, and the modified activation value is less than the output activation value. The fusion weight matrix parameters are obtained by the specific rotation matrix acting on the second initial weight parameters of the second sub-model of the specific adjacent layer corresponding to the trained model.

[0051] Here, the quantized model is obtained based on the trained model, that is, the initial model can be trained first to obtain the trained model, and then the trained model can be optimized to obtain the quantized model. Since the scale of the current trained model is often in the hundreds of millions or even hundreds of billions of parameters, it consumes a lot of disk space and on-chip chip computing power. In order to reduce the dependence on disk space and chip computing power, and to successfully implement resource-limited end-side devices, it is very necessary to miniaturize and lightweight the model. The accuracy of at least some parameters of the quantized model is lower than the parameter accuracy of the trained model, which can make at least some weights, activation values, etc. of the trained model converted from high precision to low precision. By reducing numerical precision (such as converting floating-point numbers to 8-bit integers) and optimizing computing efficiency, efficient deployment and real-time inference of models can be achieved in resource-constrained environments (such as edge devices and mobile terminals), while balancing accuracy and performance.

[0052] Multi-layer neural submodels are the core building blocks of deep learning models. When the first and second submodels are adjacent layer submodels, the output activation value of the first submodel serves as the input of the second submodel. For example, for the matrix operation of the activation value and weight obtained by the second submodel in the quantization model, the output y of the second submodel can be obtained as follows (1):

[0053] y=x@w T +b (1);

[0054] Where x is the output activation value of the first sub-model, that is, the input value matrix of the second sub-model (the current sub-module), w is the weight value matrix of the second sub-model; b is the bias value matrix of the second sub-model; and the @ symbol represents matrix multiplication.

[0055] The second sub-model of the specific adjacent layer of the quantized model includes fused weight matrix parameters, wherein the fused weight matrix parameters are obtained by applying the specific rotation matrix to the second initial weight parameters of the second sub-model of the specific adjacent layer corresponding to the trained model. For example, after adding the orthogonal rotation matrix H to the second initial weight parameters of the second sub-model, the output y of the second sub-model can be obtained using the following formula (2):

[0056] y=x@H@H T @w T +b (2);

[0057] Among them, for the backward operation of the second sub-model: according to the matrix associative law y=(x@H)@(H T @w T )+b, where the weight parameter w is known and can be directly multiplied into the model weight matrix (which can be directly updated and saved in the state dictionary file of the quantized model) to obtain the weight matrix w' of the fused rotation matrix, as shown in the following formula (3):

[0058] w'=w@H (3);

[0059] Wherein, H is a specific rotation matrix, and the specific rotation matrix H and the second initial weight parameter w of the second sub-model of the specific adjacent layer corresponding to the trained model obtain a fusion weight matrix parameter w'.

[0060] In some embodiments, the specific rotation matrix may be an orthogonal matrix, which has the following properties:

[0061] H -1 =H T , H@H T =H T @H=H -1 @H=H@H-1 =I (I is the unit matrix).

[0062] In some embodiments, the specific rotation matrix may be a Hadamard matrix. Since it is a Hadamard symmetric matrix, H = H T That is, y=(x@H)@(H@w T )+b.

[0063] In some embodiments, since the output activation values ​​of the first sub-model may contain outlier activation values, during implementation, a specific rotation matrix can be controlled to act on the output activation values ​​of the first sub-model to obtain a modified activation value x@H, where the modified activation value x@H is smaller than the output activation value. In this way, the output activation value can be adjusted to the modified activation value, i.e., the output activation value is reduced, so that the obtained modified activation value eliminates outliers, thereby reducing the quantization error caused by outliers.

[0064] In some embodiments, each matrix parameter of the specific rotation matrix is ​​represented by two symbol elements, and each matrix parameter corresponds to one bit for storage.

[0065] Here, the Hadamard matrix is ​​a type of orthogonal matrix. Due to its special properties, it can simplify matrix operations and may also reduce the effect of outliers. Therefore, by constructing a suitable Hadamard matrix in the sub-model within the model, the output activation values ​​are rotated and mapped in real time, thereby reducing the quantization error caused by outliers.

[0066] Hadamard matrices contain only +1 or -1 elements, making them computationally efficient and convenient. Hadamard matrix rotations require only sign manipulation and addition (multiplying matrix elements by ±1). Divide-and-conquer algorithms and parallel computing can significantly reduce computational complexity and time. Hadamard rotation matrices store only two elements (±1) and can be stored as a single bit (retaining only the sign bit), effectively reducing data storage requirements and facilitating storage.

[0067] In an embodiment of the present application, when the second sub-model of a specific adjacent layer of the quantization model includes a fusion weight matrix parameter, a specific rotation matrix is ​​controlled to act on the output activation value of the first sub-model to obtain a modified activation value, the modified activation value is less than the output activation value, and the fusion weight matrix parameter is obtained by the specific rotation matrix acting on the second initial weight parameter of the second sub-model of the specific adjacent layer corresponding to the trained model. In this way, the outlier output activation value can be adjusted to the modified activation value, that is, the output activation value is reduced, so that the obtained modified activation value eliminates the outlier phenomenon, thereby reducing the quantization error caused by the outlier.

[0068] In some embodiments, controlling the specific rotation matrix to act on the output activation value of the first sub-model can be achieved by the following process:

[0069] In the case where the input of the second sub-model includes the first output activation value of the first sub-model and the second output activation values ​​of other previous layers;

[0070] Here, the other previous layers may refer to the data processing layers located before the second sub-model. The second output activation value output by the previous layer also serves as the input data of the second sub-model, that is, the first output activation value is not the only input of the second sub-model. In addition to the first output activation value, the input of the second sub-model also includes the second output activation value of other previous layers.

[0071] During implementation, the input of the second sub-model includes, in addition to the first output activation value of the first sub-model, the second output activation values ​​of other previous layers, that is, the input of the second sub-model includes the first output activation value and at least one second output activation value.

[0072] The specific rotation matrix is ​​applied to the at least first output activation value and the second output activation value to obtain the input activation value of the second sub-model.

[0073] Here, since the second sub-model includes fused weight matrix parameters, that is, the specific rotation matrix acts on the second initial weight parameters of the second sub-model to obtain fused weight matrix parameters. The specific rotation matrix is ​​simultaneously applied to at least the first output activation value and the second output activation value to adjust the outlier output activation value in the first output activation value and the second output activation value to a modified activation value, that is, to reduce the output activation value so that the obtained modified activation value eliminates the outlier phenomenon, thereby reducing the quantization error caused by the outlier. At the same time, since the specific rotation matrix acts on the second initial weight parameter, at least the first output activation value and the second output activation value simultaneously, the impact of adding the specific rotation matrix on the final output result of the model can be eliminated.

[0074] In an embodiment of the present application, when the input of the second sub-model includes the first output activation value of the first sub-model and the second output activation value of other previous layers, a specific rotation matrix is ​​applied to at least the first output activation value and the second output activation value to obtain the input activation value of the second sub-model. In this way, the outlier output activation values ​​in the first output activation value and the second output activation value are adjusted to modified activation values ​​using the specific rotation matrix, that is, the output activation values ​​are reduced so that the obtained modified activation values ​​eliminate outliers, thereby reducing the quantization error caused by outliers. At the same time, because the specific rotation matrix acts on the second initial weight parameter, at least the first output activation value and the second output activation value at the same time, the impact of adding the specific rotation matrix on the final output result of the model can be eliminated.

[0075] In some embodiments, when the first sub-model and the second sub-model of a specific adjacent layer of the quantization model both include fusion weight matrix parameters, the output activation value of the first sub-model is controlled as the input activation value of the second sub-model, and the fusion matrix weight parameters of the first sub-model are obtained by the specific rotation matrix acting on the first initial weight parameters of the first sub-model; the fusion matrix weight parameters of the second sub-model are obtained by the specific rotation matrix acting on the second initial weight parameters of the second sub-model.

[0076] Here, the first and second submodels of specific adjacent layers of the quantized model each include a fused weight matrix parameter rotation matrix, meaning that the specific rotation matrix can be absorbed by the weights of the previous layer. For example, if the submodules between model layers have a unique connecting layer, the weights can be directly fused into the activation value matrices or bias value matrices of the upper and lower layers. If the output activation value of the first submodel is the only input of the second submodel, the rotation matrix can be directly fused into the weight matrices of the first and second submodels.

[0077] The following formulas (4), (5) and (6):

[0078]

[0079] Among them, x1 is the input of the first sub-model, is the transposed matrix of the weight of the first sub-model, b1 is the bias of the first sub-model, y1 is the output of the first sub-model; x2 is the input of the second sub-model, is the weight of the second sub-model, b2 is the bias of the second sub-model, and y2 is the output of the second sub-model. When the output of the first sub-model is used as the input of the second sub-model, x2 is y1, and y1 can be substituted to obtain formula (6).

[0080] The rotation matrix H is fused into the weights of the upper and lower layers using the following formula (7):

[0081]

[0082] Among them, w1'=H@w1, b1'=b1@H, w'2=w2@H can directly perform fusion operations on known weights or bias weights, that is, w1', b1', w'2 can be used as fusion weight matrix parameters and saved in the new quantization model state dictionary file.

[0083] In an embodiment of the present application, when both the first sub-model and the second sub-model of a specific adjacent layer of a quantization model include fusion weight matrix parameters, the output activation value of the first sub-model is controlled as the input activation value of the second sub-model, and the fusion matrix weight parameters of the first sub-model are obtained by acting a specific rotation matrix on the first initial weight parameters of the first sub-model; the fusion matrix weight parameters of the second sub-model are obtained by acting a specific rotation matrix on the second initial weight parameters of the second sub-model. In this way, the specific rotation matrix can be applied to the first initial weight parameters of the first sub-model and the second initial weight parameters of the second sub-model to reduce the output activation value of the second sub-model, so that the obtained modified activation value eliminates outliers, thereby reducing the quantization error caused by outliers.

[0084] In some embodiments, the weight matrix parameters include outlier identifiers, and a method for obtaining outlier weight values ​​is provided, which can be implemented by the following steps:

[0085] Step S130: determining whether the fusion weight matrix parameters have outlier weight values ​​based on the outlier identifier;

[0086] During implementation, since a specific rotation matrix is ​​applied to the initial weight parameters to obtain fused weight matrix parameters, the fused weight matrix parameters may contain some outlier weight values ​​generated by multiplying the parameters with the specific rotation matrix. If an outlier weight value is determined to exist, an outlier flag may be set for the outlier weight value, and the outlier weight value and the corresponding outlier flag may be stored in an outlier codebook.

[0087] Step S140: Acquire an initial weight value corresponding to the outlier weight value from a pre-stored outlier codebook based on an identifier corresponding to the outlier weight value.

[0088] During the implementation process, if it is determined that the fusion weight matrix parameter is set with an outlier flag, that is, the fusion weight matrix parameter has an outlier weight value, the outlier flag can be used to obtain the outlier weight value from a pre-stored outlier codebook, and in the process of processing the data, the initial weight value corresponding to the outlier weight value can be restored and used.

[0089] In an embodiment of the present application, the weight matrix parameters include an outlier identifier, and the presence of an outlier weight value in the fusion weight matrix parameters is determined based on the outlier identifier; and the initial weight value corresponding to the outlier weight value is obtained from a pre-stored outlier codebook based on the identifier corresponding to the outlier weight value. In this way, such outliers are recorded and stored separately in a dictionary-style codebook. During the quantization-dequantization process, a query operation and decoding operation are performed to restore the initial weight value corresponding to the outlier weight value, thereby reducing the output result error caused by the storage of the outlier weight value.

[0090] The present application embodiment provides a model quantization method, such as Figure 2 As shown, this can be achieved by following the steps below:

[0091] Step S201: Obtain task data, where the task data includes at least one of text data, voice data, image data, and audio and video data;

[0092] Step S202: inputting the task data into the trained model;

[0093] The trained model includes a multi-layer neuron sub-model, and for a first sub-model and a second sub-model in adjacent layers, the output activation value of the first sub-model serves as the input of the second sub-model;

[0094] Here, the quantized model is obtained based on the trained model, that is, the initial model can be trained first to obtain the trained model, and then the trained model can be optimized to obtain the quantized model. The accuracy of at least some parameters of the quantized model is less than the parameter accuracy of the trained model, which can convert at least some weights, activation values, etc. of the trained model from high precision to low precision. By reducing numerical precision (such as converting floating-point numbers to 8-bit integers) and optimizing computing efficiency, efficient deployment and real-time inference of the model can be achieved in resource-constrained environments (such as edge devices and mobile terminals), while balancing accuracy and performance.

[0095] Multi-layer neural submodels are the core building blocks of deep learning models. When the first and second submodels are adjacent layers, the output activation values ​​of the first submodel serve as the input of the second submodel.

[0096] Step S203: For a specific adjacent layer whose output activation value satisfies the outlier activation value condition, a specific rotation matrix is ​​applied to the output activation value of the first submodel of the specific adjacent layer to obtain a modified activation value, wherein the modified activation value is less than the output activation value;

[0097] During the implementation, while constructing a specific rotation matrix, the training data used to train the model can be obtained. The data analysis is as follows:

[0098] Prepare a small batch of test data sets, such as sample data with batch_size = 512 and context_length = 2048;

[0099] Through this small batch data set, the distribution of activation values ​​in the model submodules is tested, including the minimum and maximum values ​​[min, max], standard deviation (std), variance (variance), mean (mean) value, etc.

[0100] Calculate the extreme value (maximum value max{x i}, minimum value min{x i});

[0101] Determine the mean value in the sample, mean{x i}, variance std_variance{x i} to determine whether there are outliers in the model's activation values.

[0102] When it is determined that there is an outlier activation value in the output activation value, a specific adjacent layer corresponding to the outlier activation value can be determined.

[0103] During implementation, a specific rotation matrix can be applied to the output activation value of the first sub-model of a specific adjacent layer to obtain a modified activation value, where the modified activation value is less than the output activation value

[0104] For example, for the matrix operation of the activation value and weight obtained by the second sub-model in the quantization model, the output y of the second sub-model can be obtained by the following formula (1):

[0105] y=x@w T +b (1);

[0106] Where x is the output activation value of the first sub-model, that is, the input value matrix of the second sub-model (the current sub-module), w is the weight value matrix of the second sub-model; b is the bias value matrix of the second sub-model; and the @ symbol represents matrix multiplication.

[0107] The second sub-model of the specific adjacent layer of the quantized model includes fused weight matrix parameters, wherein the fused weight matrix parameters are obtained by applying the specific rotation matrix to the second initial weight parameters of the second sub-model of the specific adjacent layer corresponding to the trained model. For example, after adding the orthogonal rotation matrix H to the second initial weight parameters of the second sub-model, the output y of the second sub-model can be obtained using the following formula (2):

[0108] y=x@H@H T @w T +b (2);

[0109] Among them, for the backward operation of the second sub-model: according to the matrix associative law y=(x@H)@(H T @w T )+b. The specific rotation matrix is ​​controlled to act on the output activation value of the first sub-model to obtain a modified activation value x@H, and the modified activation value x@H is smaller than the output activation value.

[0110] Step S204: Applying a specific rotation matrix to the second initial weight parameters of the second sub-model of the specific adjacent layer to obtain fusion weight matrix parameters of the second sub-model;

[0111] Here, the second initial weight parameter w of the second sub-model is known and can be directly multiplied into the model weight matrix (which can be directly updated and saved in the state dictionary file of the quantized model) to obtain the weight matrix parameter w' of the fusion rotation matrix, as shown in the following formula (3):

[0112] w'=w@H (3);

[0113] Wherein, H is a specific rotation matrix, and the specific rotation matrix H and the second initial weight parameter w of the second sub-model of the specific adjacent layer corresponding to the trained model obtain a fusion weight matrix parameter w'.

[0114] Step S205: quantize and store the fusion weight matrix parameters including the second sub-model to obtain a quantized model, wherein the accuracy of at least some parameters of the quantized model is less than the accuracy of the parameters of the trained model.

[0115] In an embodiment of the present application, when the second sub-model of a specific adjacent layer of the quantization model includes a fusion weight matrix parameter, a specific rotation matrix is ​​controlled to act on the output activation value of the first sub-model to obtain a modified activation value, the modified activation value is less than the output activation value, and the fusion weight matrix parameter is obtained by the specific rotation matrix acting on the second initial weight parameter of the second sub-model of the specific adjacent layer corresponding to the trained model. In this way, the outlier output activation value can be adjusted to the modified activation value, that is, the output activation value is reduced, so that the obtained modified activation value eliminates the outlier phenomenon, thereby reducing the quantization error caused by the outlier.

[0116] In some embodiments, as Figure 3A As shown, the embodiment of the present application also provides a method for obtaining the input activation value of the second sub-model, which can be implemented by the following steps:

[0117] Step S301: Determine that the input of the second sub-model includes the first output activation value of the first sub-model and the second output activation values ​​of other previous layers;

[0118] Here, the other previous layers may refer to the data processing layers located before the second sub-model. The second output activation value output by the previous layer also serves as the input data of the second sub-model, that is, the first output activation value is not the only input of the second sub-model. In addition to the first output activation value, the input of the second sub-model also includes the second output activation value of other previous layers.

[0119] Step S302: Store the specific rotation matrix so that the specific rotation matrix can be applied to the first output activation value and the second output activation value during inference based on the quantization model to obtain the input activation value of the second sub-model.

[0120] Here, the specific rotation matrix can be stored. In the reasoning process based on the quantization model, since the second sub-model includes a fusion weight matrix parameter, that is, the specific rotation matrix acts on the second initial weight parameter of the second sub-model to obtain the fusion weight matrix parameter. The specific rotation matrix is ​​simultaneously applied to at least the first output activation value and the second output activation value to adjust the outlier output activation value in the first output activation value and the second output activation value to a modified activation value, that is, to reduce the output activation value so that the obtained modified activation value eliminates the outlier phenomenon, thereby reducing the quantization error caused by the outlier. At the same time, since the specific rotation matrix acts on the second initial weight parameter, at least the first output activation value and the second output activation value at the same time, the influence of adding the specific rotation matrix on the final output result of the model can be eliminated.

[0121] In an embodiment of the present application, when the input of the second sub-model includes the first output activation value of the first sub-model and the second output activation value of other previous layers; the specific rotation matrix is ​​stored so that in the process of using the quantization model for reasoning, the specific rotation matrix is ​​applied to at least the first output activation value and the second output activation value to obtain the input activation value of the second sub-model. In this way, in the process of using the quantization model for reasoning, the outlier output activation value in the first output activation value and the second output activation value is adjusted to a modified activation value using the specific rotation matrix, that is, the output activation value is reduced so that the obtained modified activation value eliminates the outlier phenomenon, thereby reducing the quantization error caused by the outlier. At the same time, since the specific rotation matrix acts on the second initial weight parameter, at least the first output activation value and the second output activation value at the same time, the influence of adding the specific rotation matrix on the final output result of the model can be eliminated.

[0122] In some embodiments, as Figure 3B As shown, the embodiment of the present application provides a method for obtaining a quantization model, which can be implemented by the following steps:

[0123] Step S311: When the input of the second sub-model includes the only first output activation value of the first sub-model, a specific rotation matrix is ​​applied to the first initial weight parameters of the first sub-model to obtain the fusion matrix weight parameters of the first sub-model, and the specific rotation matrix is ​​applied to the second initial weight parameters of the second sub-model to obtain the fusion matrix weight parameters of the second sub-model;

[0124] Here, the first and second submodels of specific adjacent layers of the quantized model each include a fused weight matrix parameter rotation matrix, meaning that the specific rotation matrix can be absorbed by the weights of the previous layer. For example, if the submodules between model layers have a unique connecting layer, the weights can be directly fused into the activation value matrices or bias value matrices of the upper and lower layers. If the output activation value of the first submodel is the only input of the second submodel, the rotation matrix can be directly fused into the weight matrices of the first and second submodels.

[0125] The following formulas (4), (5) and (6):

[0126]

[0127]

[0128] Among them, x1 is the input of the first sub-model, is the transposed matrix of the weight of the first sub-model, b1 is the bias of the first sub-model, y1 is the output of the first sub-model; x2 is the input of the second sub-model, is the weight of the second sub-model, b2 is the bias of the second sub-model, and y2 is the output of the second sub-model. When the output of the first sub-model is used as the input of the second sub-model, x2 is y1, and y1 can be substituted to obtain formula (6).

[0129] The rotation matrix H is fused into the weights of the upper and lower layers using the following formula (7):

[0130]

[0131] Among them, w1'=H@w1, b1'=b1@H, w'2=w2@H can directly perform fusion operations on known weights or bias weights, that is, w1', b1', w'2 can be used as fusion weight matrix parameters and permanently saved in the new model state dictionary file.

[0132] Step S312: quantize and store the fusion weight matrix parameters of the first sub-model and the fusion weight matrix parameters of the second sub-model to obtain a quantized model.

[0133] In an embodiment of the present application, when the input of the second sub-model includes the first output activation value of the only first sub-model, a specific rotation matrix is ​​applied to the first initial weight parameter of the first sub-model to obtain the fusion matrix weight parameter of the first sub-model, and the specific rotation matrix is ​​applied to the second initial weight parameter of the second sub-model to obtain the fusion matrix weight parameter of the second sub-model; and the fusion weight matrix parameters of the first sub-model and the fusion weight matrix parameters of the second sub-model are quantized and stored to obtain a quantized model. In this way, the specific rotation matrix can be applied to the first initial weight parameter of the first sub-model and the second initial weight parameter of the second sub-model to reduce the output activation value of the second sub-model, so that the obtained modified activation value eliminates outliers, thereby reducing the quantization error caused by outliers.

[0134] In some embodiments, the present application also provides a method for storing outlier weight values, which can be implemented by the following steps:

[0135] Step S206: determining whether the fusion matrix weight parameters of the first sub-model and / or the fusion matrix weight parameters of the second sub-model have outlier weight values;

[0136] During implementation, since a specific rotation matrix is ​​applied to the initial weight parameters to obtain fused weight matrix parameters, the fused weight matrix parameters may contain some outlier weight values ​​generated by multiplication with the specific rotation matrix.

[0137] Step S207: Identify the fusion matrix weight parameters with outlier weight values.

[0138] When it is determined that an outlier weight value exists, an outlier flag may be set for the outlier weight value.

[0139] Step S208: Store the initial weight value and the identifier corresponding to the outlier weight value into an outlier codebook.

[0140] During implementation, an outlier codebook may be set to store initial weight values ​​and identifiers corresponding to outlier weight values. When a quantization model is used to process input task data, the initial weight values ​​corresponding to the outlier weight values ​​may be obtained from the pre-stored outlier codebook based on the identifiers corresponding to the outlier weight values.

[0141] In the embodiment of the present application, it is determined that the fusion matrix weight parameters of the first sub-model and / or the fusion matrix weight parameters of the second sub-model have outlier weight values; the fusion matrix weight parameters having outlier weight values ​​are identified;

[0142] The initial weight value and identifier corresponding to the outlier weight value are stored in the outlier codebook. In this way, such outliers are recorded and stored separately in a dictionary-style codebook. During the quantization-dequantization process, the query operation and decoding operation are performed to restore the initial weight value corresponding to the outlier weight value, thereby reducing the output result error caused by the storage of outlier weight values.

[0143] In some embodiments, the present invention provides a method for obtaining a specific rotation matrix, which can be implemented by the following steps:

[0144] Step S209: obtaining an initial rotation matrix, wherein the initial rotation matrix is ​​an orthogonal rotation matrix;

[0145] Here, the characteristic rotation matrix can be realized using the following general rotation matrix R and orthogonal matrix H:

[0146] Common Rotation Matrix R: A learnable rotation matrix (Rotation matrix, R matrix) is added to the model structure. Common layers in the model are typically composed of ordinary linear neurons, following the matrix multiplication operation in linear algebra. By selecting an appropriate spatial basis transformation, the relative range of the elements in the transformed matrix can be reduced.

[0147] In some embodiments, an orthogonal matrix H that meets the orthogonal property can be generated by the optimizer as the initial rotation matrix. The characteristics of the orthogonal matrix are as follows:

[0148] H -1 =H T , H@H T =H T @H=H -1 @H=H@H -1 =I (I is the identity matrix);

[0149] Ordinary orthogonal matrices can be obtained through orthogonal decomposition strategies (such as triangular decomposition, QR decomposition, and singular value decomposition);

[0150] The Hadamard matrix is ​​a type of orthogonal matrix. Due to its special properties, it can simplify matrix operations and may also reduce the effect of outliers. Therefore, by constructing appropriate Hadamard matrices in the submodules of the model, the activation values ​​are rotated and mapped in real time, thereby reducing the quantization error caused by outliers.

[0151] Step S210: Adjust the parameters of the initial rotation matrix based on minimization of the quantization loss value to obtain the specific rotation matrix.

[0152] During the implementation process, the loss value can be used to guide the optimizer to adjust the matrix parameters, that is, by paying attention to the quantization loss value, a specific rotation matrix that completes the matrix parameter adjustment is obtained when the optimizer determines that the quantization loss value is minimized.

[0153] In this embodiment, an initial rotation matrix is ​​obtained, where the initial rotation matrix is ​​an orthogonal rotation matrix. The parameters of the initial rotation matrix are adjusted based on minimization of quantization loss to obtain a specific rotation matrix. This specific rotation matrix ensures minimal precision loss during inference, effectively reducing the precision loss associated with adding the specific rotation matrix.

[0154] Figure 4B The embodiment of the present application provides a schematic diagram of the implementation process of orthogonal transformation processing abnormal values, such as Figure 4B As shown, this can be achieved by following the steps below:

[0155] Step S410: prepare training data and rotation matrix;

[0156] During implementation, the following general orthogonal rotation matrix R and Hadamard orthogonal rotation matrix H can be used:

[0157] General rotation matrix R:

[0158] In the model structure, add a learnable rotation module (Rotation matrix, R matrix).

[0159] Ordinary layers in the model are usually composed of ordinary linear neurons, following the matrix multiplication operation in linear algebra.

[0160] By selecting an appropriate spatial basis transformation, the relative range of the value range of the elements in the transformed matrix can be reduced. Figure 4C A schematic diagram of a matrix rotation transformation provided in an embodiment of the present application is shown in FIG. Figure 4C As shown, from the left figure, the matrix M1 = [[1, -2], [1, 0]] T The projection range on the x-axis is [-2, 1], and M1@R = M2 is mapped to the linear space shown in the right figure, M2 = [[1, 2], [1, 0]] T The range of the x-axis becomes [0, 2]. As the range is reduced, the precision loss after linear quantization and dequantization is smaller, making it more quantization-friendly.

[0161] will contain unusual outliers, such as Figure 4A As shown in , r_max appears, that is, the matrix (activation matrix, weight matrix) that significantly deviates from other data points is linearly mapped (matrix multiplication operation) according to the established learned rotation matrix.

[0162] Construct an orthogonal matrix H. The properties of an orthogonal matrix are as follows:

[0163] H -1 =H T , H@H T =H T @H=H -1 @H=H@H -1 =I (I is the identity matrix);

[0164] In one embodiment, a suitable Hadamard matrix is ​​constructed in the submodules of the model to perform a real-time rotation mapping operation on the activation values, thereby reducing the quantization error caused by outliers.

[0165] In one embodiment, an orthogonal matrix with a sufficiently large dimension (e.g., 14336×14336) is constructed by means of a Hadamard matrix;

[0166] Since the internal elements of the Hadamard matrix are only +1 or -1, the calculation is convenient and efficient;

[0167] Large-scale Hadamard matrices have self-similarity, which makes it easy to perform recursive calculations using the divide-and-conquer algorithm. The complexity can be reduced from O(n) to O(n) for ordinary matrix operations. 2 ) is reduced to O(n×log(n));

[0168] During the implementation process, while constructing the rotation matrix, the training data for training the model can be obtained. The data analysis is as follows:

[0169] Prepare a small batch of test data sets, such as sample data with batch_size = 512 and context_length = 2048;

[0170] Through this small batch data set, the distribution of activation values ​​in the model submodules is tested, including the minimum and maximum values ​​[min, max], standard deviation (std), variance (variance), mean (mean) value, etc.

[0171] Calculate the extreme value (maximum value max{x i}, minimum value min{x i});

[0172] Determine the mean value in the sample, mean{x i}, variance std_variance{x i} to determine whether there are outliers in the model's activation values.

[0173] Step S420: Rotation matrix learning (ordinary orthogonal matrix & Hadamard matrix);

[0174] During the implementation process, the rotation operation operator in the model sublayer is set in the inference structure of the model (paired operations are performed on the activation values ​​and weight values ​​in the submodule).

[0175] You can choose a normal matrix or an orthogonal matrix as the initial matrix.

[0176] Step S430: quantify the loss assessment;

[0177] During the implementation process, the optimizer generates a rotation matrix that conforms to the orthogonal property, and continuously guides the optimizer to adjust the matrix parameters through the loss value;

[0178] Repeat the above steps until a more quantization-friendly rotation matrix is ​​generated. That is, the operation of the rotation matrix can make the quantization loss value as low as possible, that is, the loss value is less than the preset loss threshold.

[0179] Step S440: activating the fusion rotation matrix;

[0180] If it is determined that both the front and rear sub-models can fuse the rotation matrix, step S450 is executed; if it is determined that only the backward sub-model can fuse the rotation matrix, step S460 is executed.

[0181] Step S450: The weights of the front and rear sub-models are fused with the rotation matrix;

[0182] For the matrix operation of activation and weight, the output of the current submodule (second submodel) can be obtained as shown in formula (1) above: y = x@w T +b, where x is the input value matrix of the current submodule, w is the weight value matrix of the current submodule; b is the bias value matrix of the current submodule; and the output y of the current layer is the input x of the next layer. next-layer .

[0183] After adding the orthogonal rotation matrix H, the output of the current submodule can be obtained as y=x@H@H T @w, for the backward operation of the current submodule: by the matrix associative law y=(x@H)@(H T @w T ), where the weight parameter w is known and can be directly multiplied into the model weight matrix (which can be directly updated and permanently saved in the state dictionary file of the quantized model) to obtain the weight matrix parameter w'=w@H of the fusion rotation matrix.

[0184] For the forward operation involving the previous layer submodule: As mentioned above, the rotation matrix in x@H can be absorbed by the weight of the previous layer, that is, w last-layer@H; At this point, the submodules between model layers have a unique connection layer that connects the upper and lower layers, and the weights can be directly fused into the activation value matrix or bias value matrix of the upper and lower layers. If the activation output (output activation value) of submodule 1 (the first submodel) is the only input of the next submodule 2 (the second submodel), then the above formula (7) can be directly executed to fuse the rotation matrix into the weight matrix.

[0185] Step S460: Only the weights of the backward sub-models are fused, and real-time rotation is activated;

[0186] In some embodiments, the case where a rotation matrix that cannot be forward-fused is stored is as follows:

[0187] For a more complex input source, which cannot reach the submodule of the forward-backward bidirectional fusion method described in step S150, the backward weight fusion is completed using the above formulas (2) and (3), that is, the rotation matrix is ​​applied to the weight parameters of the second submodel.

[0188] For the forward operation, it is necessary to retain the operator currently operating on the activation, and use the saved operator to process the activation value in the subsequent data processing process.

[0189] In some embodiments, when it is determined that an abnormal weight outlier is stored in a sub-model, the following actions may be taken:

[0190] For the new weight value matrix w' after fusion, it is possible to detect whether there are extremely large outliers in it; during the implementation process, statistical methods such as variance, mean and percentile can be used to count the outliers in such weight value matrices;

[0191] Such outliers are recorded and stored separately in a dictionary-style codebook. During the quantization-dequantization process, query operations and decoding operations are performed to restore the outlier weight values.

[0192] Step S470: quantify loss and evaluate performance;

[0193] During the quantization process, by paying attention to the quantization loss value and evaluating the performance of the model, a specific rotation matrix can be obtained when it is determined that the quantization loss value is minimized and the performance meets the usage requirements.

[0194] Step S480: discard the added rotation matrix;

[0195] If it is determined that the quantization loss value cannot meet the requirements or the performance cannot meet the usage requirements, the added rotation matrix is ​​discarded.

[0196] Step S490: Obtain a model with smaller quantization loss.

[0197] During the implementation process, after obtaining a model with smaller quantization loss, the quantization, conversion, and deployment processes can be carried out on the end side.

[0198] In the embodiment of the present application, the advantages of using an orthogonal matrix are as follows:

[0199] Lossless transformation, based on orthogonal decomposition, rotates the model's activation values ​​while directly fusing the model's weight values ​​with the rotation matrix. After performing forward and backward bilateral operations, the computational loss is almost negligible (the error can be below 1e-8, caused by computer calculation accuracy issues).

[0200] Reduce the amount of calculation (space complexity, time complexity). In the rotation operation of the Hadamard matrix, only symbolic operations and addition operations (matrix elements multiplied by ±1) are required. The computational complexity and time consumption can be greatly reduced through divide-and-conquer algorithms and parallel computing.

[0201] For ease of storage, the Hadamard rotation matrix only stores two elements (±1) and can be stored in 1 bit (only the sign bit is retained).

[0202] Based on the foregoing embodiments, the embodiments of the present application provide an inference device and a model quantization device based on a quantitative model. The device includes the modules included, each module includes sub-modules, each sub-module includes a unit, and can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0203] Figure 5A This is a schematic diagram of the composition structure of an inference device based on a quantization model provided in an embodiment of the present application. The quantization model is obtained based on a trained model, and the accuracy of at least some parameters of the quantization model is less than the parameter accuracy of the trained model, such as Figure 5A As shown, the apparatus 500 includes:

[0204] A first obtaining module 501 obtains input task data, wherein the input task data includes at least one of text data, voice data, image data, and audio and video data;

[0205] The data processing module 502 is used to input the input task data into the quantitative model to obtain an output result;

[0206] The quantization model includes a multi-layer neuron sub-model, and for a first sub-model and a second sub-model in adjacent layers, the output activation value of the first sub-model serves as the input of the second sub-model;

[0207] In which, when the second sub-model of the specific adjacent layer of the quantization model includes fusion weight matrix parameters, the specific rotation matrix is ​​controlled to act on the output activation value of the first sub-model to obtain a modified activation value, and the modified activation value is less than the output activation value. The fusion weight matrix parameters are obtained by the specific rotation matrix acting on the second initial weight parameters of the second sub-model of the specific adjacent layer corresponding to the trained model.

[0208] In some embodiments, the controlling specific rotation matrix acts on the output activation value of the first sub-model, including: when the input of the second sub-model includes the first output activation value of the first sub-model and the second output activation value of other previous layers; applying the specific rotation matrix to at least the first output activation value and the second output activation value to obtain the input activation value of the second sub-model.

[0209] In some embodiments, when the first sub-model and the second sub-model of a specific adjacent layer of the quantization model both include fusion weight matrix parameters, the output activation value of the first sub-model is controlled as the input activation value of the second sub-model, and the fusion matrix weight parameters of the first sub-model are obtained by the specific rotation matrix acting on the first initial weight parameters of the first sub-model; the fusion matrix weight parameters of the second sub-model are obtained by the specific rotation matrix acting on the second initial weight parameters of the second sub-model.

[0210] In some embodiments, the weight matrix parameters include an outlier identifier, and the quantization model-based inference device 500 further includes: a first determination module and a first acquisition module, wherein the first determination module is used to determine whether the fusion weight matrix parameters have an outlier weight value based on the outlier identifier; the first acquisition module is used to obtain the initial weight value corresponding to the outlier weight value based on the identifier corresponding to the outlier weight value from a pre-stored outlier codebook.

[0211] In some embodiments, each matrix parameter of the specific rotation matrix is ​​represented by two symbol elements, and each matrix parameter corresponds to one bit for storage.

[0212] Figure 5B A schematic diagram of the structure of the model quantization device provided in the embodiment of the present application is shown in FIG. Figure 5B As shown, the device 510 includes:

[0213] A second obtaining module 511 is configured to obtain task data, wherein the task data includes at least one of text data, voice data, image data, and audio and video data;

[0214] a data input module 512 for inputting the task data into the trained model;

[0215] The trained model includes a multi-layer neuron sub-model, and for a first sub-model and a second sub-model in adjacent layers, the output activation value of the first sub-model serves as the input of the second sub-model;

[0216] A first action module 513 is configured to apply a specific rotation matrix to the output activation value of the first submodel of a specific adjacent layer whose output activation value satisfies the outlier activation value condition, to obtain a modified activation value, wherein the modified activation value is smaller than the output activation value;

[0217] A second action module 514 is configured to apply a specific rotation matrix to the second initial weight parameters of the second sub-model of the specific adjacent layer to obtain fusion weight matrix parameters of the second sub-model;

[0218] The first quantization storage module 515 is used to quantize and store the fusion weight matrix parameters including the second sub-model to obtain a quantized model, and the accuracy of at least some parameters of the quantized model is less than the accuracy of the parameters of the trained model.

[0219] In some embodiments, the model quantization device 510 also includes a second determination module and a storage module, wherein the second determination module is used to determine that the input of the second sub-model includes the first output activation value of the first sub-model and the second output activation values ​​of other previous layers; the storage module is used to store the specific rotation matrix so as to apply the specific rotation matrix to the first output activation value and the second output activation value when reasoning based on the quantization model to obtain the input activation value of the second sub-model.

[0220] In some embodiments, the model quantization device 510 also includes a third action module and a second quantization storage module, wherein the third action module is used to apply a specific rotation matrix to the first initial weight parameters of the first sub-model when the input of the second sub-model includes the unique first output activation value of the first sub-model to obtain the fusion matrix weight parameters of the first sub-model, and apply the specific rotation matrix to the second initial weight parameters of the second sub-model to obtain the fusion matrix weight parameters of the second sub-model; the second quantization storage module is used to quantize and store the fusion weight matrix parameters of the first sub-model and the fusion weight matrix parameters of the second sub-model to obtain a quantized model.

[0221] In some embodiments, the model quantization device 510 also includes a third determination module, an identification module and a storage module, wherein the third determination module is used to determine whether the fusion matrix weight parameters of the first sub-model and / or the fusion matrix weight parameters of the second sub-model have outlier weight values; the identification module is used to identify the fusion matrix weight parameters with outlier weight values; and the storage module is used to store the initial weight value and identification corresponding to the outlier weight value in the outlier codebook.

[0222] In some embodiments, the model quantization device 510 also includes a second acquisition module and an adjustment module, wherein the second acquisition module is used to obtain an initial rotation matrix, wherein the initial rotation matrix is ​​an orthogonal rotation matrix; and the adjustment module is used to adjust the parameters of the initial rotation matrix based on minimizing the quantization loss value to obtain the specific rotation matrix.

[0223] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0224] It should be noted that, in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a mobile phone, tablet computer, laptop computer, desktop computer, etc.) to execute all or part of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0225] Correspondingly, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the quantitative model-based reasoning method and the model quantization method provided in the above embodiments are implemented.

[0226] Correspondingly, an embodiment of the present application provides an electronic device, Figure 6 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Figure 6As shown, the hardware entity of the device 600 includes: a memory 601 and a processor 602, wherein the memory 601 stores a computer program that can be run on the processor 602, and when the processor 602 executes the program, the steps in the quantitative model-based reasoning method and the model quantization method provided in the above embodiment are implemented.

[0227] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or processed by the processor 602 and various modules in the electronic device 600 (for example, image data, audio data, voice communication data and video communication data), which can be implemented through flash memory (FLASH) or random access memory (RAM).

[0228] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0229] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0230] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0231] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0232] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0233] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0234] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0235] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words be embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a mobile phone, tablet computer, laptop computer, desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0236] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0237] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0238] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0239] The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for reasoning based on a quantized model, wherein the quantized model is obtained based on a trained model, and the accuracy of at least some parameters of the quantized model is less than the accuracy of parameters of the trained model, the method comprising: Obtaining input task data, wherein the input task data includes at least one of text data, voice data, image data, and audio and video data; Inputting the input task data into the quantitative model to obtain an output result; The quantization model includes a multi-layer neuron sub-model, and for a first sub-model and a second sub-model in adjacent layers, the output activation value of the first sub-model serves as the input of the second sub-model; In which, when the second sub-model of the specific adjacent layer of the quantization model includes fusion weight matrix parameters, the specific rotation matrix is ​​controlled to act on the output activation value of the first sub-model to obtain a modified activation value, and the modified activation value is less than the output activation value. The fusion weight matrix parameters are obtained by the specific rotation matrix acting on the second initial weight parameters of the second sub-model of the specific adjacent layer corresponding to the trained model.

2. The inference method according to claim 1, wherein controlling the specific rotation matrix to act on the output activation value of the first sub-model comprises: Determine that the input of the second sub-model includes the first output activation value of the first sub-model and the second output activation values ​​of other previous layers; The specific rotation matrix is ​​applied to the at least first output activation value and the second output activation value to obtain the input activation value of the second sub-model.

3. In the inference method as described in claim 1, when the first sub-model and the second sub-model of the specific adjacent layer of the quantization model both include fusion weight matrix parameters, the output activation value of the first sub-model is controlled as the input activation value of the second sub-model, and the fusion matrix weight parameters of the first sub-model are obtained by the specific rotation matrix acting on the first initial weight parameters of the first sub-model; the fusion matrix weight parameters of the second sub-model are obtained by the specific rotation matrix acting on the second initial weight parameters of the second sub-model.

4. The method according to any one of claims 1 to 3, wherein the weight matrix parameters include outlier identifiers, and the method further comprises: Determining, based on the outlier identifier, whether the fusion weight matrix parameter has an outlier weight value; An initial weight value corresponding to the outlier weight value is obtained from a pre-stored outlier codebook based on an identifier corresponding to the outlier weight value. 5 . The method according to claim 1 , wherein each matrix parameter in the specific rotation matrix is ​​represented by two symbol elements, and each matrix parameter is stored corresponding to one bit.

6. A model quantization method, comprising: Obtaining task data, wherein the task data includes at least one of text data, voice data, image data, and audio and video data; inputting the task data into the trained model; The trained model includes a multi-layer neuron sub-model, and for a first sub-model and a second sub-model in adjacent layers, the output activation value of the first sub-model serves as the input of the second sub-model; For a specific adjacent layer whose output activation value satisfies the outlier activation value condition, applying a specific rotation matrix to the output activation value of the first submodel of the specific adjacent layer to obtain a modified activation value, wherein the modified activation value is less than the output activation value; Applying a specific rotation matrix to the second initial weight parameters of the second sub-model of the specific adjacent layer to obtain fusion weight matrix parameters of the second sub-model; Quantize and store the fusion weight matrix parameters including the second sub-model to obtain a quantized model, wherein the accuracy of at least part of the parameters of the quantized model is less than the accuracy of the parameters of the trained model.

7. The method of claim 6, further comprising: Determine that the input of the second sub-model includes the first output activation value of the first sub-model and the second output activation values ​​of other previous layers; The specific rotation matrix is ​​stored so that the specific rotation matrix is ​​applied to the first output activation value and the second output activation value during inference based on the quantization model to obtain the input activation value of the second sub-model.

8. The method of claim 6, further comprising: In a case where the input of the second sub-model includes the only first output activation value of the first sub-model, a specific rotation matrix is ​​applied to the first initial weight parameters of the first sub-model to obtain the fusion matrix weight parameters of the first sub-model, and the specific rotation matrix is ​​applied to the second initial weight parameters of the second sub-model to obtain the fusion matrix weight parameters of the second sub-model; The fusion weight matrix parameters of the first sub-model and the fusion weight matrix parameters of the second sub-model are quantized and stored to obtain a quantized model.

9. The method of claim 8, further comprising: Determining whether an outlier weight value exists in a fusion matrix weight parameter of the first sub-model and / or a fusion matrix weight parameter of the second sub-model; Identify the fusion matrix weight parameters with outlier weight values; The initial weight value and the identifier corresponding to the outlier weight value are stored in an outlier codebook.

10. The method according to any one of claims 6 to 9, further comprising: Obtaining an initial rotation matrix, wherein the initial rotation matrix is ​​an orthogonal rotation matrix; The parameters of the initial rotation matrix are adjusted based on minimization of the quantization loss value to obtain the specific rotation matrix.

Images