Quantization method of neural network model and related device

Abstract

The present disclosure relates to a quantization method of a neural network model and a related device. The method comprises: obtaining an activation tensor output by a current layer in a forward propagation process of the neural network; decomposing the activation tensor into a mean component and a residual component; performing quantization processing on the mean component and the residual component respectively to obtain a quantized mean component and a quantized residual component; obtaining a first weight matrix of the current layer and quantizing the first weight matrix to obtain a quantized first weight matrix; and generating a forward output matrix based on the quantized mean component, the quantized residual component and the quantized first weight matrix. The quantization method of the neural network model can solve the problem that the quantization parameter is guided by an extreme value due to consistency bias in the activation tensor in the quantization process of the neural network model.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the fields of computer technology and artificial intelligence technology, and in particular to a method for quantizing neural network models and related equipment. Background Technology

[0002] With the rapid growth in the scale of deep learning models, especially the widespread application of Transformer-type networks containing multi-layer self-attention modules and residual connections in fields such as natural language processing, vision, and speech, the demands on computing power, GPU memory capacity, memory bandwidth, and energy consumption for model training are constantly increasing. To reduce training costs and improve hardware resource utilization, low-bit numerical representation and low-bit matrix multiplication acceleration have gradually become the mainstream direction for engineering implementation. For example, in GPU (Graphics Processing Unit) tensor cores, AI (Artificial Intelligence) dedicated accelerators, and on-chip systolic arrays, block floating-point or low-bit floating-point formats are often used to represent tensors such as weights, activations, and gradients, and scaling factors are calculated in blocks to improve throughput while maintaining a certain numerical range.

[0003] However, in low-bit training, quantization error is no longer a uniformly distributed "slight perturbation," but can be amplified during training, becoming a major factor leading to training instability. One of the core reasons is that block quantization typically determines the scaling factor based on the maximum absolute value or extreme quantiles within the block. When there are a very small number of elements with extremely large amplitudes in the tensor, the scaling factor is forced to increase to accommodate these extreme values, causing the vast majority of small-to-medium amplitude elements within the block to fall into a very small number of quantization levels, resulting in a decrease in effective representation resolution. This phenomenon of "stretching the dynamic range and compressing the effective values" can lead to gradient estimation bias, imbalance in the effective learning rate allocation, slower loss descent, and even training divergence.

[0004] To suppress extreme values ​​and anisotropic distributions in tensors, related techniques often employ spectral domain methods such as singular value decomposition, orthogonal rotation, whitening, and low-rank suppression, or strategies like activation rescaling, weight absorption, and outlier pruning. While these methods can mitigate the impact of outliers in some scenarios, spectral decomposition and iterative orthogonalization typically involve high computational complexity and memory overhead, and are difficult to integrate seamlessly with the standard quantized matrix multiplication kernel (Quantized GeMM (General Matrix Multiply)) in low-bit training, thus reducing hardware efficiency and increasing engineering complexity. On the other hand, some normalization methods (such as normalizing the feature dimensions of each sample) may also introduce additional numerical computation overhead and cross-device communication synchronization costs in low-bit training, and cannot fundamentally solve the core problem of the scaling factor being dominated by extreme values ​​in block quantization. Unlike conventional centralization or normalization processes, related techniques typically perform feature normalization operations on individual samples. Their core purpose is to improve gradient propagation efficiency or accelerate model convergence, rather than addressing the problem of the scaling factor being dominated by extreme values ​​during low-bit block quantization. Summary of the Invention

[0005] This disclosure aims to at least partially address one of the technical problems in the aforementioned background or related technologies.

[0006] To address this issue, this disclosure provides a quantization method and related equipment for neural network models, which can solve the problem that quantization parameters are guided by extreme values ​​due to the consistency bias in the activation tensor during the quantization process of neural network models.

[0007] In a first aspect, this disclosure provides a quantization method for a neural network model, comprising: obtaining the activation tensor output by the current layer during the forward propagation of the neural network; decomposing the activation tensor into a mean component and a residual component; quantizing the mean component and the residual component respectively to obtain quantized mean components and quantized residual components; obtaining a first weight matrix of the current layer and quantizing the first weight matrix to obtain a quantized first weight matrix; and generating a forward output matrix based on the quantized mean component, the quantized residual component, and the quantized first weight matrix.

[0008] In a second aspect, this disclosure provides a computer device including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the one or more programs include instructions for performing the method described in the first aspect.

[0009] Thirdly, this disclosure provides a non-volatile computer-readable storage medium containing a computer program that, when executed by one or more processors, causes the one or more processors to perform the method described in the first aspect.

[0010] Fourthly, this disclosure provides a computer program product comprising one or more computer programs that, when executed by one or more processors, implement the method described in the first aspect.

[0011] The quantization method for the neural network model in this embodiment decomposes the activation tensor of the current layer into a mean component and a residual component during the forward propagation of the neural network. The mean and residual components are then quantized separately, ensuring that each component has its own independent quantization parameter. This prevents the mean component from expanding the quantization range of the residual component, thus explicitly separating the consistent bias structure present in the activation tensor from the residual component. This prevents the mean component from being mixed into the quantization block of the residual component, thus avoiding the forced dominance of extreme values ​​in the quantization parameters of the residual component. The quantization parameters are no longer affected by the consistent bias structure in the mean component, effectively improving the quantization resolution of the residual component and reducing the impact of quantization errors on the stability of model training. Attached Figure Description

[0012] The following description and accompanying drawings will better aid in understanding these and other features and advantages of the various embodiments disclosed herein, wherein the same reference numerals in the drawings consistently denote the same parts, wherein: Figure 1 This is a flowchart of a quantization method for an exemplary neural network model according to an embodiment of this disclosure.

[0013] Figure 2 This is an exemplary flowchart of determining the forward output matrix according to an embodiment of the present disclosure.

[0014] Figure 3 This is a schematic diagram illustrating the calculation process of an exemplary parameter gradient matrix according to an embodiment of this disclosure.

[0015] Figure 4 This is a schematic diagram illustrating an exemplary calculation process of the activation gradient matrix according to an embodiment of this disclosure.

[0016] Figure 5 This is a schematic diagram of the forward and backward propagation process of an exemplary quantization system based on a neural network model according to an embodiment of this disclosure.

[0017] Figure 6 This is an exemplary forward and backward propagation process diagram based on the average quantization matrix multiplication unit of this disclosure.

[0018] Figure 7 This is a schematic diagram of the hardware structure of an exemplary computer device provided in the embodiments of this disclosure. Detailed Implementation

[0019] The technical solutions of this disclosure will be further described in detail below through embodiments and in conjunction with the accompanying drawings. In this specification, the same or similar reference numerals indicate the same or similar components. The following description of the embodiments of this disclosure with reference to the accompanying drawings is intended to explain the overall inventive concept of this disclosure and should not be construed as a limitation thereof.

[0020] In related technologies, during low-bit training of neural networks, the activation matrix is ​​block-quantized and input into the quantization matrix multiplication kernel. However, since the activation matrix often has a bias component shared across tokens (the basic processing unit when text, sequences, or features are processed by the model), consistency drift and a small number of extreme values ​​occur within the block, resulting in an increased scaling factor.

[0021] In view of at least one of the problems existing in the related technologies, this disclosure provides a method for quantizing a neural network model. This method can be executed by a computer, server, or other general-purpose electronic device with a processor and memory, or it can be implemented through a dedicated hardware module or a combination of hardware and software. Figure 1 This is a flowchart of a quantization method for an exemplary neural network model according to an embodiment of this disclosure. Figure 1 As shown, the method includes the following processing: In step S102, the activation tensor output by the current layer during the forward propagation of the neural network is obtained.

[0022] The forward propagation process of a neural network refers to the process by which the neural network performs forward computation layer by layer from input to output to obtain the prediction result. The activation tensor mentioned above refers to the tensor obtained after the input of the current layer of the neural network undergoes a linear transformation and an activation function. The current layer refers to the layer in the neural network that is currently being computed.

[0023] Optionally, in step S102 above, to facilitate matrix multiplication calculations in subsequent processing, the activation tensor can first be converted into an activation matrix. For example, in one instance, the input activation tensor is represented as... ,in, For batch size, For sequence length, This is the hidden dimension. The input activation tensor can be converted into an activation matrix. ,in, That is, the number of rows in the activation matrix. Equal to the batch size in the activation tensor With sequence length The product of the numbers, the number of columns It remains consistent with the hidden dimension of the activation tensor.

[0024] In step S104, the activation tensor is decomposed into mean components and residual components.

[0025] Using the example above, we can first calculate the mean vector based on the activation matrix obtained from the activation tensor transformation, construct the mean matrix based on the mean vector, and then calculate the difference between the activation matrix and the mean matrix to obtain the residual matrix, which serves as the residual component. For example, the mean vector can be implemented based on GPU reduction operations. The specific calculation method will be explained in detail later and will not be repeated here.

[0026] In step S106, the mean component and the residual component are quantized respectively to obtain the quantized mean component and the quantized residual component.

[0027] Optionally, the mean component and the residual component can be quantized separately using low-bit quantization. When quantizing the mean component and the residual component separately, different scaling factor calculation rules can be used to calculate the scaling factor. The scaling factor is a scaling coefficient used to map floating-point values ​​to fixed-point / integer values, which is used to achieve linear conversion between floating-point and fixed-point values.

[0028] In step S108, the first weight matrix of the current layer is obtained and the first weight matrix is ​​quantized to obtain the quantized first weight matrix; In step S110, a forward output matrix is ​​generated based on the quantized mean component, the quantized residual component, and the quantized first weight matrix.

[0029] The aforementioned forward output matrix is ​​a tensor / matrix output by the current layer during the forward propagation of the neural network.

[0030] Optionally, a low-bit matrix multiplication operation can be performed using the quantized mean component and the quantized first weight matrix to generate an output matrix Y1. A low-bit matrix multiplication operation can be performed using the quantized residual component and the quantized first weight matrix to generate an output matrix Y2. The output matrix Y1 is then broadcast to align its shape with that of the output matrix Y2. Finally, the aligned output matrix Y1 and the output matrix Y2 are added together to obtain the forward output matrix.

[0031] The quantization method for the neural network model in this embodiment decomposes the activation tensor of the current layer into a mean component and a residual component during the forward propagation of the neural network. The mean and residual components are then quantized separately, ensuring that each component has its own independent quantization parameter. This prevents the mean component from expanding the quantization range of the residual component, thus explicitly separating the consistent bias structure present in the activation tensor from the residual component. This prevents the mean component from being mixed into the quantization block of the residual component, thus avoiding the forced dominance of extreme values ​​in the quantization parameters of the residual component. The quantization parameters are no longer affected by the consistent bias structure in the mean component, effectively improving the quantization resolution of the residual component and reducing the impact of quantization errors on the stability of model training.

[0032] Furthermore, the quantization method of the neural network model in this disclosure does not rely on costly spectral decomposition operations and can be implemented through reduction and element-wise operations. It can be well adapted to the parallel execution characteristics of hardware such as GPUs and ASICs (Application-Specific Integrated Circuits), while bringing verifiable improvements to the training stability of the model.

[0033] In one or more embodiments of this disclosure, the activation tensor is decomposed into a mean component and a residual component, which may specifically include: Convert the activation tensor into an activation matrix; The mean of each column of the activation matrix is ​​calculated to obtain a first mean vector. A first mean matrix is ​​constructed based on the first mean vector to obtain the mean components. The elements in each row of the first mean matrix are the same and equal to the first mean vector. The first residual matrix is ​​calculated based on the activation matrix and the first mean matrix to obtain the residual components.

[0034] Continuing with the example above, assume that the activation tensor of the current layer is transformed to obtain the activation matrix. First, calculate the mean vector of the feature dimensions of the activation matrix. Then, for each feature dimension, all... The values ​​of each sample in this dimension are summed and divided by . The mean vector is obtained. The following formula represents: ; in, For the whole Vector. The calculation of the mean vector requires only one reduction operation, exhibiting good parallelism. Then, based on this mean vector... The mean matrix that forms a rank-one structure (a rank-one matrix is ​​a matrix with a rank of 1, which can be represented as the outer product of two vectors, and whose rows and columns are all linearly dependent). This involves multiplying a column vector of all ones with the transpose of the mean vector to generate the mean matrix. In the actual processing flow, It can be generated on demand in subsequent processing steps without being explicitly written to video memory, but rather through a broadcast mechanism. Then, the residual matrix is ​​calculated. The residual matrix reflects the fine-grained variation components after removing the global consistency bias, which better matches the effective distribution range within the quantization block. Subsequently, [the following was done]... and Performing low-bit quantization yields: ,in, Indicates bit width as The quantization operator allows quantization to be block floating-point, fixed-point, or mixed precision, and allows for... and Different scaling factor calculation strategies or different quantization formats are employed. Let the weight matrix be... Can be quantified as The output of the current layer can then be calculated using the reconstruction method, as shown in the following formula:

[0035] in, The shape needs to be broadcast to the same Once the shapes are identical, add the two together.

[0036] The quantization method for the neural network model in this disclosure employs a rank-one mean matrix. Since a rank-one matrix can be represented by only two vectors, eliminating the need to store the complete matrix, it significantly reduces storage resource consumption and overhead. The rank-one structure transforms matrix operations into vector operations, reducing the number of operations and improving computational efficiency. Furthermore, the rank-one matrix exhibits a concentrated numerical distribution and minimal fluctuation; its statistical properties are preserved by the rank-one structure, mitigating numerical distortion caused by quantization and contributing to the stability of model training. The rank-one structure is regular and computationally efficient, making it suitable for parallel reduction, pipelined computation, and low-bit quantization on neural network processors, GPUs, and dedicated acceleration units, thereby improving hardware utilization.

[0037] In one or more embodiments of this disclosure, the calculation of a first residual matrix based on the activation matrix and the first mean matrix to obtain the residual components may specifically include: The first residual matrix is ​​obtained by subtracting the first mean matrix from the activation matrix. Alternatively, the first mean matrix can be multiplied by a preset coefficient to obtain an adjusted mean matrix. The adjusted mean matrix is ​​then subtracted from the activation matrix to obtain the first residual matrix. The preset coefficient is determined based on at least one of the following: the number of training steps, layer depth, quantization error, and outlier statistics of the neural network. For example, in certain training stages or layers of the neural network, obtaining the residual matrix by completely removing the mean matrix may not be optimal. Therefore, in one implementation, preset coefficients can be introduced. This is to achieve a reduction in the partial mean, i.e., the residual matrix. The preset coefficient can be set to a constant, or its value can be adaptively adjusted according to the number of training steps, layer depth, quantization error, and outlier statistics of the neural network, so that the calculated residual matrix can achieve a better balance between stability and expressive power.

[0038] Among them, the training steps of a neural network refer to the number of times the neural network model is updated; the layer depth of a neural network refers to the number of layers contained in the neural network, that is, the number of hidden layers between the input layer and the output layer; quantization error refers to the numerical variation of data before and after quantization; outlier statistics refer to the statistical results obtained after performing distribution statistics and outlier detection on the weight parameters or activation function output values ​​in the neural network, which are used to characterize the number, proportion, magnitude, location and frequency of data points that deviate abnormally from the main distribution in the data.

[0039] The following description, in conjunction with the accompanying drawings and formulas, provides an exemplary illustration of the process for determining the forward output matrix during the forward propagation of the quantization method for the neural network model according to an embodiment of this disclosure. Figure 2 This is a schematic flowchart illustrating an exemplary embodiment of the present disclosure for determining the forward output matrix. Figure 2 As shown, this process can specifically include the following steps: The activation matrix output from the previous layer of the current layer. As input to the current layer, it participates in the forward computation process of the current layer. This represents the product of the batch size and the sequence length. It is the dimension of each input token.

[0040] For the input activation matrix Calculate the column mean to obtain the mean vector. (As an example of the first mean vector mentioned above), it is expressed as follows: .

[0041] Based on mean vector Generate the mean matrix .

[0042] The mean matrix (As an example of the first mean matrix mentioned above) From the input activation matrix Subtracting from the middle, we obtain the residual matrix. .

[0043] For the mean vector Quantify to obtain .

[0044] For the residual matrix Quantify to obtain .

[0045] For the weight matrix of the current layer (As an example of the first weight matrix mentioned above) Quantization is performed to obtain... .

[0046] Use the quantized mean vector and the quantized weight matrix Perform low-bit matrix multiplication to generate the output. (This is an example of the output matrix Y1 mentioned above).

[0047] Using the quantized residual matrix and the quantized weight matrix Perform low-bit matrix multiplication to generate the output. (This is an example of the output matrix Y2 mentioned above).

[0048] Finally, The shape is broadcast to and Add the results together to get the final forward output. .

[0049] In one or more embodiments of this disclosure, the mean component and the residual component are quantized respectively to obtain quantized mean components and quantized residual components, which may specifically include: According to the channel dimension of the mean component, an independent quantization parameter is assigned to each channel of the mean component, and the mean component is quantized channel by channel to obtain the quantized mean component. The quantization parameters include, but are not limited to, scaling factors.

[0050] The residual components are subjected to block quantization to obtain quantized residual components.

[0051] The method of this disclosure employs different scaling factor calculation rules (an example of the aforementioned quantization parameters) during the quantization of the mean component and residual component obtained from activation tensor decomposition, enabling adaptive quantization based on their numerical distribution, statistical characteristics, and structure. For the rank-one structure mean matrix, its numerical distribution is concentrated and its fluctuations are small. Using a matching scaling factor can maintain high representation accuracy even with low-bit quantization, avoiding numerical overflow or precision waste caused by unreasonable scaling. The residual matrix, on the other hand, has a more dispersed numerical distribution and a larger dynamic range. Using an independent scaling factor can capture detailed information more precisely, reducing the impact of quantization noise on the model's expressive power. Quantizing both components separately using different scaling factor calculation rules can effectively reduce overall quantization error without significantly increasing computational and storage costs, mitigating the damage quantization causes to model training stability, convergence characteristics, and inference accuracy, achieving a better balance between hardware implementation efficiency and model accuracy.

[0052] In one or more embodiments of this disclosure, the mean component and the residual component are quantized respectively to obtain quantized mean components and quantized residual components, which may specifically include: Convert the activation tensor into an activation matrix; The activation matrix is ​​divided into multiple groups according to at least one of attention head, convolution channel grouping, and quantization block, and multiple second mean vectors are calculated for each group. Multiple second mean matrices are constructed based on the multiple second mean vectors, wherein each row of the second mean matrix has the same elements and is equal to the second mean vector; The residual matrix is ​​calculated based on the activation matrix and the plurality of second mean matrices to obtain the residual components.

[0053] For example, the latent space dimension of the activation matrix can be divided into several independent substructures, such as multiple substructures including attention heads (multi-head attention mechanisms) or groups of convolutional channels. Based on this, the mean vector can be calculated at a finer granularity, and the corresponding rank-one mean matrix can be constructed (an example of the mean matrix mentioned above, referring to a rank-one mean matrix). For instance, the feature dimension of the activation matrix can be divided into several groups, and the mean vector and rank-one mean matrix can be calculated for each group separately. This allows the bias separation to better fit the internal structure and data distribution of the network, thereby achieving better numerical stability without significantly increasing computational overhead. Especially in block quantization processing, if the quantization block itself is divided into hardware tiles, the mean matrix calculation can be aligned with the hardware tiles, thus dividing the mean calculation operation into hardware tiles as the basic unit, ensuring that the boundary of the mean calculation is consistent with the calculation and memory access boundaries of the hardware tiles. This reduces data dependencies across hardware tiles and improves kernel fusion efficiency. Among them, quantization block division by hardware block refers to the process of aligning and dividing the quantization operation unit (quantization block) according to the smallest parallel unit (tile) of the target hardware in the model quantization and inference deployment stage, so that the quantization granularity is consistent with the hardware underlying computing granularity, thereby improving the computing parallelism and memory access efficiency.

[0054] In one or more embodiments of this disclosure, the mean component is encoded using a first precision quantization format, and the residual component is encoded using a second precision quantization format; wherein the bit width of the first precision quantization format is greater than the bit width of the second precision quantization format.

[0055] For example, the mean component can be encoded using a higher precision format than the residual component. For instance, the residual component can be stored and computed using FP4, while the mean component can be stored and computed using FP8 or BF16. Here, FP stands for Floating-Point, FP4 refers to a floating-point number represented by 4 bits, FP6 by 6 bits, and FP8 by 8 bits. The purpose of this mixed-precision strategy is that the mean component, with its concentrated energy and sensitivity to quantization scale, can reduce error propagation due to bias compression if expressed with higher precision, while the residual component can still maintain the throughput advantage of its low bit count.

[0056] In low-bit training, the gradient matrix may still exhibit consistent bias or extreme values, affecting gradient matrix multiplication calculations and optimizer statistics. Therefore, embodiments of this disclosure perform mean-residual decomposition on the gradient matrix during the backpropagation phase, consistent with the forward propagation process. This makes the obtained input gradient and weight gradient less susceptible to the influence of a few extreme values ​​in a low-bit environment, thereby improving training stability and repeatability. Therefore, in one or more embodiments of this disclosure, the quantization method of the neural network model may further include: During the backpropagation of the neural network, the output gradient matrix of the current layer is obtained; wherein, during the backpropagation of the neural network, the output gradient matrix of the current layer is the gradient matrix corresponding to the input activation matrix of the next layer during forward propagation, which can be calculated by the next layer and passed to the current layer.

[0057] The output gradient matrix is ​​decomposed into the gradient mean vector and the gradient residual matrix; The gradient mean vector and the gradient residual matrix are quantized respectively to obtain the quantized gradient mean vector and the quantized gradient residual matrix. Obtain the input activation matrix of the current layer and decompose the input activation matrix into the input activation mean vector and the input activation residual matrix; wherein, during the backpropagation of the neural network, the input activation matrix required by the current layer can be the input activation values ​​of the layer cached and saved in the forward propagation stage.

[0058] The input activation mean vector and the input activation residual matrix are quantized respectively to obtain the quantized input activation mean vector and the quantized input activation residual matrix. The quantized input activation mean vector and the quantized input activation residual matrix are fused together. The first matrix obtained by fusion is transposed to obtain the transposed matrix. Then, the transposed matrix is ​​multiplied with the second matrix obtained by fusion of the quantized gradient mean vector and the quantized gradient residual matrix to obtain the parameter gradient matrix of the current layer.

[0059] The following section, in conjunction with the accompanying figures and formulas, provides an exemplary description of the calculation process of the parameter gradient matrix during backpropagation.

[0060] Figure 3 This is a schematic diagram illustrating the exemplary calculation process of the parameter gradient matrix of this disclosure, as shown below. Figure 3 As shown, this process may specifically include the following steps: Obtain the output gradient matrix The output gradient represents the loss function with respect to the forward output. The gradient is expressed by the following formula: ; in, This represents the product of the batch size and the sequence length. It is the dimension of each output token.

[0061] For the input activation matrix Calculate the column mean to obtain the input activation mean vector. It can be represented by the following formula: ; Based on input activation mean vector Generate the mean matrix .

[0062] For the output gradient matrix Calculate the row mean to obtain the mean vector. mean vector An example of the gradient mean vector mentioned above is represented by the following equation: ; Based on mean vector Generate the mean matrix .

[0063] The mean matrix From the output gradient matrix Subtracting from the middle, we obtain the residual matrix. Residual matrix Here is an example of the gradient residual matrix mentioned above.

[0064] The mean matrix From the input matrix Subtracting from the middle, we obtain the residual matrix. Residual matrix This is an example of the activation residual matrix mentioned above.

[0065] For activation mean vector Quantify to obtain .

[0066] For the activation residual matrix Quantify to obtain .

[0067] The mean vector of the output gradient Quantify to obtain .

[0068] The residual matrix of the output gradient Quantify to obtain .

[0069] Use the quantized mean vector and Perform low-bit matrix multiplication to generate the output. .

[0070] Using the quantized residual matrix and Perform low-bit matrix multiplication to generate the output. .

[0071] Use the quantized mean vector and Perform low-bit matrix multiplication to generate the output. .

[0072] Using the quantized residual matrix and Perform low-bit matrix multiplication to generate the output. .

[0073] Finally, , , The shape is broadcast to and After summing the results, we obtain the final parameter gradient matrix, as shown in the following formula: In the above formula, Here is an example of the first matrix. Here is an example of the second matrix mentioned above.

[0074] In one or more embodiments of this disclosure, the quantization method for the neural network model may further include: During the backpropagation process of the neural network, the output gradient matrix of the current layer is obtained; The output gradient matrix is ​​decomposed into the gradient mean vector and the gradient residual matrix; The gradient mean vector and gradient residual matrix are quantized respectively to obtain the quantized gradient mean vector and the quantized gradient residual matrix. The second weight matrix of the current layer is quantized to obtain the quantized second weight matrix. Perform matrix multiplication on the quantized gradient mean vector and the transpose of the quantized second weight matrix to obtain the first result. The quantized gradient residual matrix is ​​multiplied by the transpose of the quantized second weight matrix to obtain the second result. After broadcasting the first calculation result to a shape identical to the second calculation result, it is added element-wise to the second calculation result to obtain the activation gradient matrix of the current layer. The following, with reference to the accompanying drawings and formulas, provides an exemplary description of the process for calculating the activation gradient matrix during backpropagation.

[0075] Figure 4 This is a schematic diagram illustrating the calculation process of the activation gradient matrix in an exemplary embodiment of this disclosure, as shown below. Figure 4 As shown, this process may specifically include the following steps: Obtain the output gradient matrix The output gradient matrix represents the loss function with respect to the forward output. The gradient is expressed by the following formula: ; in, This represents the product of the batch size and the sequence length. It is the dimension of each output token.

[0076] For the output gradient matrix Calculate the column mean to obtain the mean vector of the output gradient. (As an example of the gradient mean vector above), it is expressed as follows: ; Output gradient mean vector Generate the mean matrix .

[0077] The mean matrix From the output gradient matrix Subtracting from the middle, we obtain the residual matrix. Residual matrix Here is an example of the gradient residual matrix mentioned above.

[0078] The mean vector of the output gradient Quantify to obtain .

[0079] The residual matrix of the output gradient Quantify to obtain .

[0080] For the weight matrix (As an example of the second weight matrix above) Quantize to obtain . This is an example of the second weight matrix after the above quantization.

[0081] Use the quantized mean vector and Perform low-bit matrix multiplication on the transpose of the quantized weight matrix to generate the output. Output This is an example of the result of the first operation mentioned above.

[0082] Using the quantized residual matrix and Perform low-bit matrix multiplication to generate the output. Output This is an example of the result of the second operation mentioned above.

[0083] Will The shape is broadcast to and The constants are then added together to obtain the final activation gradient matrix, as shown in the following formula: .

[0084] The quantization method of the neural network model disclosed herein performs a symmetric mean-residual decomposition on the output gradient tensor during the backpropagation stage of the neural network and quantizes the mean component and residual component obtained by the decomposition respectively. This allows the calculation of the weight gradient and the input gradient to avoid being dominated by the bias component, thereby further reducing the numerical instability in low-bit training.

[0085] This disclosure also provides a quantization system for a neural network model, used to execute the quantization method for the neural network model of this disclosure. The system includes the following components: Processor unit: Includes high-efficiency Tensor Cores for performing matrix multiplication (MatMul) operations. Tensor Cores support efficient low-bit computation, accelerating the quantization matrix multiplication process.

[0086] The reduction acceleration unit is responsible for performing mean calculation and related subtraction operations. This unit is configured to quickly calculate and store the column mean of the matrix, and then subtract this mean from the activation and gradient matrices to generate the residual matrix. This unit reduces computational complexity and improves efficiency.

[0087] High-bandwidth memory (HBM): Used to store large-scale data, such as input matrices. Weight matrix and gradient matrix Video memory provides fast data access speeds, ensuring efficient data processing and exchange during large-scale training.

[0088] Control logic unit: Configured to coordinate the collaborative work between different operations within the system, particularly the integration of element-wise subtraction and quantization operations. It ensures efficient cooperation between units, from input data processing to output result generation.

[0089] Figure 5This is a schematic diagram of the forward and backward propagation processes of a quantization system based on a neural network model according to an embodiment of this disclosure. The forward propagation process is used to calculate the output of the current layer based on the input, and the process may include the following steps: In step S502, the (i-1)th layer, i.e., the layer above the current layer, outputs Y. i-1 X, as the input of the current layer i .

[0090] In step S504, X i Low-precision matrix multiplication is performed between the weight matrix W and the average quantization matrix multiplication operator to obtain the current layer output Y. i ; Y i On one hand, it serves as the result of the current layer; on the other hand, in step S506, it is passed to the next layer, i.e., the (i+1)th layer, as its input X. i+1 .

[0091] Backpropagation is used to calculate the gradient and update the weights, and consists of two gradient flows: The weight gradient calculation (parameter update) includes the following processing: The input gradient passed to the next layer after the current layer, i.e., the (i+1)th layer. As the current layer's output gradient ; In step S508, With the current layer input X i The weight gradient is obtained by calculating using the average quantization matrix multiplication operator. ; In step S510, the weight gradient It works in conjunction with the optimizer state to update the master weight. The sovereign is resynchronized to the weight W that actually participates in the calculation at the current layer, and the parameter iteration is completed.

[0092] Input gradient calculation (gradient propagation) includes the following processing: In step S512, the current layer gradient output The gradient of the current layer is obtained by multiplying the weight matrix W with the average quantization matrix multiplier. ; In step S514, This gradient is passed to the previous layer, i.e., the (i-1)th layer, as its output gradient. This completes the gradient backpropagation.

[0093] Figure 6 This is a schematic diagram of the forward and backward propagation process based on the average quantization matrix multiplication unit in an embodiment of this disclosure.

[0094] The mean-quantization-matmul unit may include the following components: The Mean module is configured to take an input matrix, such as... Figure 6 The mean of each column of input matrix A and input matrix B shown in the diagram will be calculated and stored. The mean calculation can be performed by an efficient reduction acceleration unit, and the result will serve as the basis for subsequent operations.

[0095] The residual calculation module is configured to generate and feed the residual matrix obtained by subtracting the mean from the mean after the mean calculation into the quantization module. The residual matrix provides useful information remaining after removing the mean from the original input.

[0096] The Quantization module is configured to quantize the input matrix, residual matrix, and weight matrix, using low bit precision to reduce memory consumption and accelerate computation. The quantized data is then passed to the matrix multiplication unit (MatMul) for output calculation.

[0097] The matrix multiplication (MatMul) module is configured to perform matrix multiplication on the quantized residual matrix and weight matrix. This computation can be efficiently implemented using Tensor Cores, accelerating low-bit-precision matrix multiplication operations.

[0098] The Add module is configured such that the calculated output is finally added to the mean matrix to generate the final forward propagation output or gradient output.

[0099] The quantization system for the neural network model in this embodiment of the present disclosure, through optimized hardware components and efficient computing units, can support rapid computation and quantization processing of large-scale matrices, ensuring the efficient operation of the system.

[0100] One or more embodiments of this disclosure also provide a computer device, including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the one or more programs include instructions for a quantization method for a neural network model of one or more embodiments of this disclosure.

[0101] In one or more embodiments, the method of the neural network model of this disclosure can be executed by a computing device. The computing device can be a server including a CPU (Central Processing Unit) and a GPU, or a dedicated training platform including an NPU (Neural Processing Unit) / TPU (Tensor Processing Unit) / ASIC (Application-Specific Integrated Circuit). The computing device may include a processor, memory, and a bus, wherein the processor is used to execute program instructions stored in the memory. The processor may include acceleration units for matrix multiplication, such as tensor cores or on-chip arrays. The memory is used to store auxiliary information such as network parameters, activation tensors, gradient tensors, and quantization scales. The system can be deployed in a single-machine, single-GPU environment or in a multi-machine, multi-GPU distributed training environment, performing necessary synchronization operations through a communication interface.

[0102] In the training process of a neural network model, computer equipment is used to perform forward computation and backpropagation. For network structures containing linear layers, attention layers, and feedforward layers, matrix multiplication is a core operation. The neural network model quantization method of this disclosure improves training stability by performing rank-mean decomposition on the activation matrix and / or gradient matrix before performing quantization matrix multiplication, and by quantizing the mean and residual components obtained from the undecomposed matrix separately. This makes the numerical range of the quantization matrix multiplication more consistent with the effective range of low-bit representation.

[0103] One or more embodiments of this disclosure also provide a non-volatile computer-readable storage medium containing a computer program that, when executed by one or more processors, causes the one or more processors to perform a quantization method for a neural network model according to one or more embodiments of this disclosure.

[0104] One or more embodiments of this disclosure also provide a computer program product, including one or more computer programs that, when executed by one or more processors, implement the quantization method of the neural network model of one or more embodiments of this disclosure.

[0105] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0106] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0107] For ease of description, the above computer devices are described in terms of function, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0108] The computer device described in the above embodiments is used to implement the quantization method of the corresponding neural network model in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0109] This disclosure also provides a computer device for implementing the quantization method of the neural network model described above. Figure 7 A schematic diagram of the hardware structure of an exemplary computer device 700 provided in an embodiment of this disclosure is shown.

[0110] like Figure 7 As shown, the computer device 700 may include: a processor 702, a memory 704, a network interface 706, a peripheral interface 708, and a bus 710. The processor 702, memory 704, network interface 706, and peripheral interface 708 are interconnected within the computer device 700 via the bus 710.

[0111] Processor 702 may be a central processing unit (CPU), image processor, neural network processor (NPU), microcontroller (MCU), programmable logic device, digital signal processor (DSP), application-specific integrated circuit (ASIC), or one or more integrated circuits. Processor 702 can be used to perform functions related to the techniques described in this disclosure. In some embodiments, processor 702 may also include multiple processors integrated as a single logic component. For example, such as... Figure 7 As shown, processor 702 may include multiple processors 702a, 702b and 702c.

[0112] Memory 704 can be configured to store data (e.g., instructions, computer code, etc.). Figure 7 As shown, the data stored in memory 704 may include program instructions (e.g., one or more programs for implementing a quantization method for a neural network model according to embodiments of the present disclosure) and data to be processed (e.g., the memory may store configuration files for other modules, etc.). Processor 702 may also access the program instructions and data stored in memory 704 and execute the program instructions to operate on the data to be processed. Memory 704 may include volatile or non-volatile storage devices. In some embodiments, memory 704 may include random access memory (RAM), read-only memory (ROM), optical disk, magnetic disk, hard disk, solid-state drive (SSD), flash memory, memory stick, etc.

[0113] Network interface 706 can be configured to provide communication with other external devices to computer device 700 via a network. This network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, Near Field Communication (NFC), etc.), a cellular network, the Internet, or a combination thereof. It is understood that the type of network is not limited to the specific examples described above.

[0114] The peripheral interface 708 can be configured to connect the computer device 700 to one or more peripheral devices to enable information input and output. For example, peripheral devices may include input devices such as keyboards, mice, touchpads, touch screens, microphones, and various sensors, as well as output devices such as displays, speakers, vibrators, and indicator lights.

[0115] Bus 710 can be configured to transfer information between various components of computer device 700 (such as processor 702, memory 704, network interface 706, and peripheral interface 708), such as internal buses (e.g., processor-memory bus), external buses (USB port, PCI-E bus), etc.

[0116] It should be noted that although the architecture of the computer device 700 described above only shows the processor 702, memory 704, network interface 706, peripheral interface 708, and bus 710, in specific implementations, the architecture of the computer device 700 may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the architecture of the computer device 700 described above may only include the components necessary for implementing the embodiments of this disclosure, and does not necessarily include all the components shown in the figures.

[0117] Based on the same inventive concept, corresponding to any of the above embodiments, this disclosure also provides a non-volatile computer-readable storage medium containing a computer program, which, when executed by one or more processors, causes the one or more processors to perform the quantization method of the neural network model.

[0118] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0119] The computer program stored in the storage medium of the above embodiments is used to cause the one or more processors to execute the quantization method of the neural network model as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0120] Based on the same inventive concept, corresponding to the quantization method of the neural network model in any of the above embodiments, this disclosure also provides a computer program product, which includes one or more computer programs. In some embodiments, the one or more computer programs are executable by one or more processors to cause the one or more processors to perform the quantization method of the neural network model. Corresponding to the execution entity for each step in each embodiment of the quantization method of the neural network model, the processor executing the corresponding step may belong to the corresponding execution entity.

[0121] The computer program product of the above embodiments is used to cause the processor to execute the quantization method of the neural network model as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0122] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0123] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0124] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0125] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A quantization method for a neural network model, characterized in that, include: Obtain the activation tensor of the current layer output during the forward propagation of the neural network; The activation tensor is decomposed into mean components and residual components; The mean component and the residual component are quantized respectively to obtain the quantized mean component and the quantized residual component. Obtain the first weight matrix of the current layer and quantize the first weight matrix to obtain the quantized first weight matrix; The forward output matrix is ​​generated based on the quantized mean component, the quantized residual component, and the quantized first weight matrix.

2. The method according to claim 1, characterized in that, The activation tensor is decomposed into mean components and residual components, including: Convert the activation tensor into an activation matrix; The mean of each column of the activation matrix is ​​calculated to obtain a first mean vector. A first mean matrix is ​​constructed based on the first mean vector to obtain the mean components. The elements in each row of the first mean matrix are the same and equal to the first mean vector. The first residual matrix is ​​calculated based on the activation matrix and the first mean matrix to obtain the residual components.

3. The method according to claim 2, wherein calculating the first residual matrix based on the activation matrix and the first mean matrix to obtain the residual components includes: The first residual matrix is ​​obtained by subtracting the first mean matrix from the activation matrix. Alternatively, the first mean matrix can be multiplied by preset coefficients to obtain an adjusted mean matrix, and the adjusted mean matrix can be subtracted from the activation matrix to obtain the first residual matrix. The preset coefficients are determined based on at least one of the following: the number of training steps of the neural network, the layer depth, the quantization error, and outlier statistics.

4. The method according to claim 1, characterized in that, The mean component and the residual component are quantized respectively to obtain the quantized mean component and the quantized residual component, including: According to the channel dimension of the mean component, an independent quantization parameter is assigned to each channel of the mean component, and the mean component is quantized channel by channel to obtain the quantized mean component. The residual components are subjected to block quantization to obtain quantized residual components.

5. The method according to claim 1, characterized in that, The mean component and the residual component are quantized respectively to obtain the quantized mean component and the quantized residual component, including: Convert the activation tensor into an activation matrix; The activation matrix is ​​divided into multiple groups according to at least one of attention head, convolution channel grouping, and quantization block, and multiple second mean vectors are calculated for each group. Multiple second mean matrices are constructed based on the multiple second mean vectors, wherein each row of the second mean matrix has the same elements and is equal to the second mean vector; The residual matrix is ​​calculated based on the activation matrix and the plurality of second mean matrices to obtain the residual components.

6. The method according to claim 1, characterized in that, The method further includes: During the backpropagation process of the neural network, the output gradient matrix of the current layer is obtained; The output gradient matrix is ​​decomposed into the gradient mean vector and the gradient residual matrix; The gradient mean vector and the gradient residual matrix are quantized respectively to obtain the quantized gradient mean vector and the quantized gradient residual matrix. Obtain the input activation matrix of the current layer and decompose the input activation matrix into the input activation mean vector and the input activation residual matrix; The input activation mean vector and the input activation residual matrix are quantized respectively to obtain the quantized input activation mean vector and the quantized input activation residual matrix. The quantized input activation mean vector and the quantized input activation residual matrix are fused together. The first matrix obtained by fusion is transposed to obtain the transposed matrix. Then, the transposed matrix is ​​multiplied with the second matrix obtained by fusion of the quantized gradient mean vector and the quantized gradient residual matrix to obtain the parameter gradient matrix of the current layer.

7. The method according to claim 1, characterized in that, The method further includes: During the backpropagation process of the neural network, the output gradient matrix of the current layer is obtained; The output gradient matrix is ​​decomposed into the gradient mean vector and the gradient residual matrix; The gradient mean vector and the gradient residual matrix are quantized respectively to obtain the quantized gradient mean vector and the quantized gradient residual matrix. The second weight matrix of the current layer is quantized to obtain the quantized second weight matrix. Perform matrix multiplication on the quantized gradient mean vector and the transpose of the quantized second weight matrix to obtain the first result. Perform matrix multiplication on the quantized gradient residual matrix and the transpose of the quantized second weight matrix to obtain the second result. After broadcasting the first operation result to a shape identical to the second operation result, it is added element-wise to the second operation result to obtain the activation gradient matrix of the current layer.

8. The method according to any one of claims 1 to 7, characterized in that, The mean component is encoded using a first precision quantization format, and the residual component is encoded using a second precision quantization format; wherein the bit width of the first precision quantization format is greater than the bit width of the second precision quantization format.

9. A computer device comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, the one or more programs comprising instructions for performing the method of any one of claims 1 to 8.

10. A non-volatile computer-readable storage medium comprising a computer program, which, when executed by one or more processors, causes the one or more processors to perform the method of any one of claims 1 to 8.

11. A computer program product comprising one or more computer programs that, when executed by one or more processors, implement the method as described in any one of claims 1 to 8.

Images