Neural network optimization method based on outlier separation activation value mixed quantization

Abstract

A neural network optimization method based on outlier separation and activation value hybrid quantization is proposed. This method uses Z-Score to decompose preprocessed activation values ​​into outliers and normal values. Normal values ​​are then quantized using 4-bit grouping, while outliers are quantized using 8-bit grouping and stored in GPU memory. During backpropagation at this layer, the stored quantized outliers and normal values ​​are restored to achieve optimization. This invention reduces the size of activation values ​​stored in GPU memory after forward propagation, ensuring model training while significantly reducing the total size of activation values ​​stored after forward propagation across all layers. Compared to other techniques, it provides higher throughput and minimizes quantization errors.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of neural networks, and particularly relates to a neural network optimization method based on outlying value separation and mixed quantization of activation values. BACKGROUND

[0002] When processing long sequence data, the existing Transformer architecture pays attention to the importance of different parts in the sequence, so that the model can capture the complex internal dependence of the data without relying on the order of the sequence data. However, the model based on the Transformer structure needs to save millions to billions of parameters through pre-training on a large amount of text data, which occupies part of the display memory in the model training process. In addition, part of the display memory is used to save the intermediate results in the model training process, including the optimizer state, intermediate activation values and the like. When the model parameter quantity is too large or the training batch size is large, the activation values calculated by the forward propagation will occupy a large amount of display memory, so that the display memory is overflowed, and the model cannot be trained. SUMMARY

[0003] The application is proposed in view of the problems that the prior art cannot protect the outlying values and the quantization error is large, and the integer quantization means is single, so that the efficiency cannot meet the industrial demand, and the problems of high requirement for the communication bandwidth of CPU memory and GPU memory, additional calculation time and large compression error. The application provides a neural network optimization method based on outlying value separation and mixed quantization of activation values, which can reduce the size of the activation values saved in the display memory after the forward propagation, and ensure that the model can be trained without causing errors.

[0004] The application is implemented by the following technical scheme:

[0005] The application relates to a neural network optimization method based on outlying value separation and mixed quantization of activation values. After the preprocessed activation values are decomposed into outlying values and normal values in a Z-Score manner, the normal values are quantized by using 4-bit grouping quantization, and the outlying values are quantized by using 8-bit quantization and saved to the display memory. When the reverse propagation calculation reaches the layer, the saved quantized outlying values and normal values are recovered to realize optimization.

[0006] The grouping quantization refers to that each element of the activation value is taken as data stored continuously on the memory, and G elements with continuous addresses on the memory are taken as a group for quantization, and each group shares quantization parameters.

[0007] The recovery processing refers to: according to the scaling factor, the original outlier value and the normal value are recovered respectively, and an empty activation value input[s, b, h] is generated, which has the same shape as the original uncompressed activation value. According to the saved outlier channel mask mask and the normal value mask ~ mask, the newly generated activation value is assigned, the original two-dimensional activation value is recovered, and the original view of the activation value is recovered through the view function.

[0008] Technical effects

[0009] According to the distribution characteristics of the outlier value, the outlier value and the normal value are separated and compressed using different compression bits. According to the distribution characteristics of the sum of the activation values in different channels, the outlier value channel and the normal value channel are separated using the Z-Score threshold value. The separated outlier value is quantized using 8-bit integer quantization, and the normal value is quantized using 4-bit integer quantization. After transposing and memory alignment of the separated outlier value and normal value, fine-grained quantization grouping is used for quantization. Each group shares a scaling factor, further reducing the influence of outlier values on quantization. Compared with the prior art, the present application is not affected by the transmission bandwidth size between the central processor memory and the graphics processor memory. It does not require data transmission between memories, so the training effect will not be affected by the transmission bandwidth. In the multi-graphics processor distributed training scene, it does not cause simultaneous transmission, provides higher training throughput and does not produce additional forward propagation to reduce training bandwidth. For the time-consuming operation of using additional forward propagation in the backward propagation, the recovery activation value time can be reduced, and higher training throughput can be provided. Considering the distribution characteristics of the activation value in the training process of the large language model, the compression error of the existing compression technology can be reduced, and the convergence of the model and the inference accuracy can be ensured. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 The flowchart of the present application is shown in the figure;

[0011] Figure 2 The system schematic diagram of the present application is shown in the figure;

[0012] Figure 3 And Figure 4 The effect schematic diagram of the embodiment is shown in the figure. DETAILED DESCRIPTION

[0013] As Figure 2As shown, the embodiment relates to a neural network optimization system based on outlier separation activation value mixed quantization, which comprises an activation value analysis unit, an activation value processing unit, an activation value quantization unit, a storage unit and an activation value dequantization unit, wherein: the activation value analysis unit calculates the Z-Score value of each channel of the activation value according to the input activation value, marks the channel with a Z-Score greater than a certain threshold as an outlier channel according to a preset threshold, and the marking of all channels forms a channel mask mask; the activation value processing unit slices and transposes the original activation value according to the channel mask mask, and divides the activation value into an outlier activation value tensor and a normal activation value tensor; the activation value quantization unit uses 8-bit quantization for the outlier activation value tensor and 4-bit quantization for the normal activation value tensor, respectively, to obtain the corresponding activation value tensor; the storage unit saves the quantized outlier activation value and normal activation value, the quantization parameters thereof and the channel mask into the video memory; and the activation value dequantization unit inversely processes the outlier activation value tensor and the normal activation value tensor according to the channel mask mask and splices them into an activation value of the original size according to the activation value tensor and the scaling factor thereof.

[0014] As shown, the embodiment relates to a neural network optimization system based on outlier separation activation value mixed quantization, which comprises an activation value analysis unit, an activation value processing unit, an activation value quantization unit, a storage unit and an activation value dequantization unit, wherein: the activation value analysis unit calculates the Z-Score value of each channel of the activation value according to the input activation value, marks the channel with a Z-Score greater than a certain threshold as an outlier channel according to a preset threshold, and the marking of all channels forms a channel mask mask; the activation value processing unit slices and transposes the original activation value according to the channel mask mask, and divides the activation value into an outlier activation value tensor and a normal activation value tensor; the activation value quantization unit uses 8-bit quantization for the outlier activation value tensor and 4-bit quantization for the normal activation value tensor, respectively, to obtain the corresponding activation value tensor; the storage unit saves the quantized outlier activation value and normal activation value, the quantization parameters thereof and the channel mask into the video memory; and the activation value dequantization unit inversely processes the outlier activation value tensor and the normal activation value tensor according to the channel mask mask and splices them into an activation value of the original size according to the activation value tensor and the scaling factor thereof. Figure 1 As shown, the embodiment relates to a neural network optimization system based on outlier separation activation value mixed quantization, which comprises an activation value analysis unit, an activation value processing unit, an activation value quantization unit, a storage unit and an activation value dequantization unit, wherein: the activation value analysis unit calculates the Z-Score value of each channel of the activation value according to the input activation value, marks the channel with a Z-Score greater than a certain threshold as an outlier channel according to a preset threshold, and the marking of all channels forms a channel mask mask; the activation value processing unit slices and transposes the original activation value according to the channel mask mask, and divides the activation value into an outlier activation value tensor and a normal activation value tensor; the activation value quantization unit uses 8-bit quantization for the outlier activation value tensor and 4-bit quantization for the normal activation value tensor, respectively, to obtain the corresponding activation value tensor; the storage unit saves the quantized outlier activation value and normal activation value, the quantization parameters thereof and the channel mask into the video memory; and the activation value dequantization unit inversely processes the outlier activation value tensor and the normal activation value tensor according to the channel mask mask and splices them into an activation value of the original size according to the activation value tensor and the scaling factor thereof.

[0015] Step 1, preprocessing the activation value: in the training process, the activation value input[s,b,h] is changed to [s,b*h] by the view function, and the input activation value input shown in the picture is changed to 2D , wherein: s is the sequence length, b is the batch size, and h is the dimension of the hidden layer.

[0016] Step 2, using the outlier separation module to divide the activation value preprocessed in step 1 into an outlier value part and a normal value part, specifically including:

[0017] 2.1 using the absolute value function to first take the absolute value of the outlier value after changing the shape, using the sum function to sum each column sum hi , using the mean function and the standard deviation function to calculate the mean μ and the standard deviation σ of the absolute value and sum hi .

[0018] 2.2 Calculate the Z-Score value of each column using the Z-Score formula and set the Z-Score threshold τ. Compare the calculated Z-Score of each channel with the set Z-Score threshold τ, and calculate the channel mask for that activation value. When the Z-Score threshold of a certain channel is greater than the set threshold τ, it indicates that the channel is an outlier channel, and the value of its corresponding channel mask is set to True.

[0019] 2.3 Use a channel mask to decompose outliers from the activation values ​​by slicing. Invert the channel mask to obtain ~mask, and use this mask to decompose normal values ​​from the activation values.

[0020] 2.4 For the extracted two-dimensional outliers and normal values, the original column-major storage of the activation values ​​is converted to row-major storage by transposing and memory alignment functions, which facilitates data grouping and compression by the compressor.

[0021] Step 3: For the separated normal value portion, use 4-bit quantization to reduce its memory usage. Use group quantization to divide each row of the activation values ​​into a group, and calculate the scaling factor S for each group. f32 The scaling factor is used to quantize all activation values ​​in the group; for outliers, 8-bit quantization is used to minimize quantization error, and the quantized outliers are then... int8 and normal value int4 and their respective scaling factors S f32 Save the data to video memory. Simultaneously, the channel mask of the activation value also needs to be saved for restoring the original activation value during backpropagation.

[0022] The aforementioned group quantization specifically involves: first calculating the scaling factor for each group of quantizations using a symmetric quantization method. Then, calculate the quantized elements. The scaling factor and quantized data for each group are saved to GPU memory and used during backpropagation. Restore the original activation value, where: |x f16 | represents the element with the largest absolute value in the group quantized by this group, and B is the largest signed positive number that the quantization bits can be, corresponding to 8 for 4-bit quantization and 128 for 8-bit quantization, respectively.

[0023] Step 4, when the back propagation calculation reaches this layer, the quantized saved outlier and normal values and the corresponding scaling factors are taken out from the memory respectively, the original outlier and normal values are recovered according to the scaling factors and the values after quantization, and an empty activation value is generated, which has the same shape as the original uncompressed activation value. According to the saved outlier channel mask mask and normal value mask ~ mask, the newly generated activation value is assigned, the original two-dimensional activation value is recovered, and the original view of the activation value is recovered through the view function.

[0024] Step 5, deployment on the distributed training framework Megatron-LM: by saving the activation value before quantizing and compressing the activation value in the forward propagation function of each sub-layer, and restoring the activation value to the original shape and size before calculating the gradient of each layer in the back propagation function, the neural network optimization is realized.

[0025] As shown in Figure 3 and Figure 4 , through specific actual experiments, on an Ubuntu 20.04.4LTS server, a Tesla V100-SXM2-32GB graphics card cluster is used for comparison test, and the technology is realized based on the open source Megatron-LM distributed training framework (software environment). The technology is used on the activation value saved between the forward propagation function and the back propagation function of different layers to achieve the purpose proposed in this paper. After setting the Z-Score threshold to 1 and setting the quantization group size G to 1024, the method is enabled. By comparing the quantization error and the memory occupancy of using the technology with the original method, it is found that after using the technology on different layers, the average quantization error of each number can be reduced by 4-6 times with no more than 3% additional memory. It shows that the scheme can greatly reduce the quantization error with small additional overhead. At the same time, by comparing Figure 2 , experiments are conducted on different batch sizes of the same GPT size, and the throughput of the method reaches 48.746 and 53.76 respectively, which is higher than that of the transfer technology and the recalculation technology. The experimental results show that the technology can achieve the goal of reducing compression error while maintaining high throughput. Figure 3

[0026] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application, the protection scope of the present application is subject to the claims and is not limited by the above specific embodiments, and each implementation scheme within the scope is subject to the constraints of the present application.​

Claims

1. A neural network optimization method based on outlier separation activation value hybrid quantization, characterized in that, Comprise: Step 1, Preprocessing of activation values: Change the activation values input[s, b, h] to [s, b*h] and input activation values input shown in the picture during the training process by the view function 2D where: s is the sequence length, b is the batch size, h is the dimension of the hidden layer; Step 2, using the outlier separation module to separate the activation value preprocessed in step 1 into outlier part and normal value part; Step 3: For the separated normal value portion, use 4-bit quantization to reduce its memory usage. Use group quantization to divide each row of the activation values ​​into a group, and calculate the scaling factor S for each group. f32 The scaling factor is used to quantize all activation values ​​in the group; for outliers, 8-bit quantization is used to minimize quantization error, and the quantized outliers are then... int8 and normal value int4 and their respective scaling factors S f32 Save it to video memory; at the same time, the channel mask of the activation value also needs to be saved for restoring the original activation value during backpropagation; The said group quantization refers to: taking each element of the activation value as data stored continuously on the memory, taking G elements with continuous addresses on the memory as a group for quantization, and each group sharing quantization parameters; Step 4, when the back propagation calculation reaches the hidden layer, the quantized and saved outlier and normal value and the corresponding scaling factor are taken out from the memory respectively, the original outlier and normal value are recovered according to the scaling factor and the quantized value, an empty activation value is generated, which has the same shape as the original uncompressed activation value, the original two-dimensional activation value is recovered by assigning values to the newly generated activation value according to the saved outlier channel mask mask and normal value mask ~mask, and the original view of the activation value is recovered through the view function; The group quantization specifically comprises: calculating a scaling factor of each group quantization by using a symmetric quantization mode Then, calculating elements after quantization Saving the scaling factor of each group and the data after quantization to the video memory, and using the scaling factor and the data after quantization in back propagation Restoring the original activation value, wherein: The element with the largest absolute value in the group quantized group, B is the maximum signed positive number that the quantization bit can be, which respectively corresponds to 8 of 4-bit quantization and 128 of 8-bit quantization.

2. The neural network optimization method based on outlier separation activation value hybrid quantization according to claim 1, characterized in that, The said recovery processing refers to: according to the scaling factor, the original outlier and normal value are recovered respectively, an empty activation value input[s,b,h] is generated, which has the same shape as the original uncompressed activation value; according to the saved outlier channel mask mask and normal value mask ~mask, the newly generated activation value is assigned values to recover the original two-dimensional activation value, and the original view of the activation value is recovered through the view function.

3. The neural network optimization method based on outlier separation activation value hybrid quantization according to claim 1, characterized in that, The said step 2 specifically comprises: 2.1 Use the absolute value function to take the absolute value of the outliers after changing shape, use the sum function to sum each column sum hi ; Use the mean function and the standard deviation function to calculate the mean μ and the standard deviation σ of the absolute value and sum hi respectively; 2.2 Calculate the Z-Score value of each column using the Z-Score formula and set the Z-Score threshold τ, compare the calculated Z-Score of each channel with the set Z-Score threshold τ, and calculate the channel mask mask of the activation value; when the Z-Score threshold of a certain channel is greater than the set threshold τ, it indicates that the channel is an outlier channel, and the corresponding channel mask mask is set to True; 2.3 Use the channel mask mask to decompose the outlier value in the activation value by slicing, get ~mask by inverting the channel mask mask, and decompose the normal value in the activation value by the mask; 2.4 For the extracted two-dimensional outlier value and normal value, convert the original column-priority stored activation value to row-priority storage through the transpose and memory alignment function, which facilitates data grouping and compression by the compressor.

4. The neural network optimization method based on outlier separation activation value hybrid quantization according to claim 1, characterized in that, Further deployed on the distributed training framework Megatron-LM: by saving the activation value before quantizing and compressing the activation value in the forward propagation function of each sublayer, and restoring the activation value to the original shape and size through the activation value recovery module before calculating the gradient of each layer in the back propagation function, the neural network optimization is realized.

5. A neural network optimization system based on outlier separation activation value hybrid quantization, which implements the method of any one of claims 1-4, characterized in that, Comprise: The application discloses an activation value analysis unit, an activation value processing unit, an activation value quantization unit, a storage unit and an activation value dequantization unit, wherein: the activation value analysis unit calculates Z-Score values of each channel of the input activation value, and marks channels with Z-Score values greater than a specific threshold as outlier channels according to a preset threshold, and the marking of all channels forms a channel mask mask; the activation value processing unit slices and transposes the original activation value according to the channel mask mask, and divides the activation value into an outlier activation value tensor and a normal activation value tensor; the activation value quantization unit quantizes the outlier activation value tensor using 8 bits and quantizes the normal activation value tensor using 4 bits, respectively, to obtain corresponding activation value tensors; the storage unit saves the quantized outlier activation value and normal activation value, the quantization parameters thereof and the channel mask into a display memory; and the activation value dequantization unit inversely processes the outlier activation value tensor and the normal activation value tensor according to the channel mask mask, and splices them into an original-size activation value according to a scaling factor of the activation value tensor.

Images