Neural network optimization method based on outlier separation activation value hybrid quantization

Abstract

A neural network optimization method based on outlier separation activation value hybrid quantization comprises the following steps: decomposing a preprocessed activation value into an outlier and a normal value in a Z-Score mode, performing 4bit grouping quantization on the normal value, performing 8bit quantization on the outlier, and storing the outlier and the normal value in a video memory; and when the back propagation calculation reaches the layer, the stored quantized outliers and normal values are subjected to recovery processing to realize optimization. According to the method, the activation value stored in the video memory after forward propagation can be reduced, the total activation value stored after forward propagation of all layers can be remarkably reduced while the model can be trained, and compared with other technologies, higher throughput rate can be provided, and quantization errors can be reduced as much as possible.

Description

Technical Field

[0001] The present invention relates to a technology in the field of neural networks, in particular to a neural network optimization method based on outlier separation, activation value hybrid quantization. Background Art

[0002] When processing long sequence data, the existing Transformer architecture focuses on the importance of different parts in the sequence, allowing the model to capture complex dependencies within the data without relying on the order of the sequence data. However, the Transformer-based model needs to save millions to billions of parameters by pre-training on a large amount of text data, which will occupy a part of the video memory during the model training process. At the same time, part of the video memory is also needed to save the intermediate results of the model training process, including optimizer status, intermediate activation values, etc. When the number of model parameters is too large or the training batch size is large, the activation values ​​calculated by the forward propagation will occupy a large amount of video memory, causing the video memory overflow and making the model unable to train. Summary of the invention

[0003] In view of the problems that the prior art cannot protect outliers and has large quantization errors, as well as the shortcomings of the single integer quantization method resulting in the inability to meet industrial needs in terms of efficiency, high requirements on the communication bandwidth of CPU memory and GPU memory, additional computing time and large compression errors, the present invention proposes a neural network optimization method based on outlier separation and activation value hybrid quantization, which can reduce the size of activation values ​​stored in the video memory after forward propagation, and ensure that the model can be trained without causing errors.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a neural network optimization method based on outlier separation and activation value hybrid quantization. After preprocessed activation values ​​are decomposed into outliers and normal values ​​in a Z-Score manner, 4-bit grouping quantization is used for normal values ​​and 8-bit quantization is used for outliers and the two values ​​are saved in a video memory. When back propagation calculation reaches this layer, the saved quantized outliers and normal values ​​are restored to achieve optimization.

[0006] The group quantization means that each element of the activation value is taken as data stored continuously in the memory, G elements with continuous addresses in the memory are quantized as a group, and each group shares a quantization parameter.

[0007] The restoration process is as follows: after restoring the original outlier and normal values ​​according to the scaling factor, an empty activation value input[s,b,h] is generated, and its shape is the same as the original uncompressed activation value. According to the saved outlier channel mask mask and normal value mask ~mask, the newly generated activation value is assigned a value to restore the original two-dimensional activation value, and then the original view of the activation value is restored through the view function. Technical Effects

[0008] The present invention separates outliers and normal values ​​according to the distribution characteristics of outliers and compresses them using different compression bits. The outlier channel and the normal value channel are separated by using the Z-Score threshold method according to the distribution characteristics of the sum of activation values ​​on different channels. 8-bit integer quantization is used for the separated outliers, and 4-bit integer quantization is used for the normal values. After the separated outliers and normal values ​​are transposed and aligned with the memory, they are quantized using fine-grained quantization groups. The scaling factor is shared within each group to further reduce the influence of outliers on quantization. Compared with the prior art, the present invention is not affected by the size of the transmission bandwidth between the central processing unit memory and the graphics processing unit memory, and does not require data transmission between the memory and the memory, so the training effect will not be affected by the transmission bandwidth, and in the multi-graphics processor distributed training scenario, it will not cause transmission requisition, while providing higher training throughput and will not generate additional forward propagation to reduce the training bandwidth. For the operation that uses additional forward propagation in the reverse transmission, which is relatively time-consuming, the activation value recovery time can be reduced, and higher training throughput is provided. Considering the activation value distribution characteristics of the large language model during the training process can reduce the compression error of existing compression technologies and ensure the convergence and inference accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flow chart of the present invention;

[0010] Figure 2 It is a schematic diagram of the system of the present invention;

[0011] Figure 3 and Figure 4 It is a schematic diagram of the effect of the embodiment. DETAILED DESCRIPTION

[0012] like Figure 2As shown, this embodiment relates to a neural network optimization system based on outlier separation and activation value hybrid quantization, including: an activation value analysis unit, an activation value processing unit, an activation value quantization unit, a storage unit and an activation value dequantization unit, wherein: the activation value analysis unit calculates the Z-Score value of each activation value channel according to the input activation value, and marks the channel with a Z-Score greater than a specific threshold as an outlier channel according to a preset threshold, and the marks of all channels form a channel mask mask; the activation value processing unit performs a channel mask mask on the original activation value according to the channel mask mask. The values ​​are sliced ​​and transposed to divide the activation values ​​into outlier activation value tensors and normal activation value tensors; the activation value quantization unit quantizes the outlier activation value tensors using 8 bits and the normal activation value tensors using 4 bits to obtain the corresponding activation value tensors; the storage unit saves the quantized outlier activation values ​​and normal activation values ​​and their quantization parameters and channel masks into the video memory; the activation value dequantization unit inversely processes the outlier activation value tensors and normal activation value tensors according to the activation value tensors and their scaling factors and the channel mask mask and splices them into activation values ​​of the original size.

[0013] like Figure 1 As shown, this is a neural network optimization method based on the above system in this embodiment. After decomposing the activation value into outliers and normal values ​​by Z-Score, high-bit compression is used to reduce the quantization loss of outliers, and low-bit compression is used to maximize video memory savings, which specifically includes:

[0014] Step 1: Preprocess the activation value: During the training process, the activation value input[s,b,h] is changed to [s,b*h] and the input activation value input shown in the picture through the view function. 2D , where: s is the sequence length, b is the batch size, and h is the dimension of the hidden layer.

[0015] Step 2: Use the outlier separation module to separate the activation values ​​preprocessed in step 1 into outlier parts and normal value parts, specifically including:

[0016] 2.1 Use the absolute value function to calculate the absolute value of the outliers after the shape change, and use the sum function to sum each column hi Use the mean function and standard deviation function to calculate the absolute value and sum respectively. hi The mean μ and standard deviation σ.

[0017] 2.2 Use the Z-Score formula to calculate the Z-Score value of each column and set the Z-Score threshold τ. Compare the Z-Score calculated for each channel with the set Z-Score threshold τ, and calculate the channel mask mask of the activation value. When the Z-Score threshold of a channel is greater than the set threshold τ, it indicates that the channel is an outlier channel, and the value of its corresponding channel mask mask is set to True.

[0018] 2.3 Use the channel mask mask to decompose the outliers in the activation value by slicing, and obtain ~mask by inverting the channel mask mask, and use the mask to decompose the normal values ​​in the activation value.

[0019] 2.4 For the extracted two-dimensional outliers and normal values, the original column-first storage activation values ​​are converted to row-first storage through transposition and memory alignment functions, which facilitates the compressor to group and compress data.

[0020] Step 3: For the separated normal value part, use 4-bit quantization to reduce its video memory usage, use group quantization to divide each row in the activation value into a group, and calculate the scaling factor S for each group. f32 , all activation values ​​in the group are quantized by the scaling factor; for the outliers, 8-bit quantization is used to minimize the quantization error, and the quantized outliers are int8 and normal int4 and their respective scaling factors S f32 Save to video memory. At the same time, the channel mask of the activation value also needs to be saved to restore the original activation value in back propagation.

[0021] The group quantization is specifically as follows: a symmetric quantization method is used to first calculate the scaling factor of each group of quantization After that, calculate the quantized elements Save the scaling factor and quantized data of each group to the video memory and use them in the back propagation Restore the original activation value, where: |x f16 | is the element with the largest absolute value in the group quantized by this group, and B is the maximum signed positive number that the quantization bit can be, corresponding to 8 for 4-bit quantization and 128 for 8-bit quantization respectively.

[0022] Step 4: When the back propagation calculation reaches this layer, the outliers and normal values ​​saved after quantization and the corresponding scaling factors are taken out from the memory, and the original outliers and normal values ​​are restored according to the scaling factors and the values ​​after quantization, and an empty activation value is generated, which has the same shape as the original uncompressed activation value. According to the saved outlier channel mask mask and normal value mask ~mask, the newly generated activation value is assigned a value, the original two-dimensional activation value is restored, and then the original view of the activation value is restored through the view function.

[0023] Step 5. Deploy on the distributed training framework Megatron-LM: Before the forward propagation function of each sub-layer saves the activation value, the activation value is quantized, compressed and saved. Before the back propagation function calculates the gradient of each layer, the activation value is restored to its original shape and size through the activation value recovery module to achieve neural network optimization.

[0024] like Figure 3 and Figure 4 As shown in the figure, after specific practical experiments, a comparative test was conducted on a TeslaV100-SXM2-32GB graphics card cluster on an Ubuntu 20.04.4LTS server. This technology is based on the open source Megatron-LM distributed training framework (software environment). This technology is used on the activation values ​​saved between the forward propagation function and the backpropagation function of different layers to achieve the purpose proposed in this article. Set the Z-Score threshold to 1, set the quantization group size G to 1024, and then enable this method. Compare the quantization error and video memory occupancy of this technology with the original method. Figure 2 By comparing the results of the experiments, we found that the average quantization error of each number can be reduced by 4-6 times with no more than 3% of additional video memory after using this technology on different layers. This shows that this scheme can significantly reduce the quantization error with a small additional overhead. Figure 3 By comparing the results of the experiments on different batch sizes with the same GPT size, the throughput of this method reached 48.746 and 53.76 respectively, which are higher than the transfer technology and the recalculation technology. The experimental results show that this technology can achieve the goal of reducing compression error while maintaining high throughput.

[0025] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A neural network optimization method based on outlier separation and activation value hybrid quantization, characterized in that: After decomposing the preprocessed activation values ​​into outliers and normal values ​​using the Z-Score method, the normal values ​​are quantized by 4-bit grouping and the outliers are quantized by 8-bit and saved to the video memory. When the back propagation calculation reaches this layer, the saved quantized outliers and normal values ​​are restored to achieve optimization. The group quantization means that each element of the activation value is taken as data stored continuously in the memory, G elements with continuous addresses in the memory are quantized as a group, and each group shares a quantization parameter.

2. The neural network optimization method based on outlier separation and activation value hybrid quantization according to claim 1 is characterized in that: The restoration process refers to: after restoring the original outlier value and normal value respectively according to the scaling factor, an empty activation value input[s,b,h] is generated, and its shape is the same as the original uncompressed activation value shape; according to the saved outlier channel mask mask and normal value mask ~mask, the newly generated activation value is assigned a value to restore the original two-dimensional activation value, and then the original view of the activation value is restored through the view function.

3. The neural network optimization method based on outlier separation and activation value hybrid quantization according to claim 1 or 2, characterized in that: include: Step 1: Preprocess the activation value: During the training process, the activation value input[s,b,h] is changed to [s,b*h] and the input activation value input shown in the picture through the view function. 2D , where: s is the sequence length, b is the batch size, and h is the dimension of the hidden layer; Step 2: Use the outlier separation module to separate the activation values ​​preprocessed in step 1 into outlier parts and normal value parts; Step 3: For the separated normal value part, use 4-bit quantization to reduce its video memory usage, use group quantization to divide each row in the activation value into a group, and calculate the scaling factor S for each group. f32 , all activation values ​​in the group are quantized by the scaling factor; for the outliers, 8-bit quantization is used to minimize the quantization error, and the quantized outliers are int8 and normal int4 and their respective scaling factors S f32 Save to video memory; at the same time, the channel mask of the activation value also needs to be saved to restore the original activation value in back propagation; Step 4. When the back propagation calculation reaches this layer, the outliers and normal values ​​saved after quantization and the corresponding scaling factors are taken out from the memory respectively. After restoring the original outliers and normal values ​​according to the scaling factors and the values ​​after quantization, an empty activation value is generated, and its shape is the same as the original uncompressed activation value; according to the saved outlier channel mask mask and normal value mask ~mask, the newly generated activation value is assigned to restore the original two-dimensional activation value, and then the original view of the activation value is restored through the view function.

4. The neural network optimization method based on outlier separation activation value hybrid quantization according to claim 3 is characterized in that: The step 2 specifically includes: 2.1 Use the absolute value function to calculate the absolute value of the outliers after the shape change, and use the sum function to sum each column hi ; Use the mean function and standard deviation function to calculate the absolute value and sum respectively hi The mean μ and standard deviation σ; 2.2 Use the Z-Score formula to calculate the Z-Score value of each column and set the Z-Score threshold τ. Compare the Z-Score calculated for each channel with the set Z-Score threshold τ, and calculate the channel mask of the activation value. When the Z-Score threshold of a channel is greater than the set threshold τ, it indicates that the channel is an outlier channel, and the value of the corresponding channel mask is set to True; 2.3 Use the channel mask to decompose the outliers in the activation value by slicing, and obtain ~mask by negating the channel mask, and use the mask to decompose the normal values ​​in the activation value; 2.4 For the extracted two-dimensional outliers and normal values, the original column-first storage activation values ​​are converted to row-first storage through transposition and memory alignment functions, which facilitates the compressor to group and compress data.

5. The neural network optimization method based on outlier separation and activation value hybrid quantization according to claim 3 is characterized in that: The group quantization is specifically as follows: a symmetric quantization method is used to first calculate the scaling factor of each group of quantization After that, calculate the quantized elements Save the scaling factor and quantized data of each group to the video memory and use them in the back propagation Restore the original activation value, where: |x f16 | is the element with the largest absolute value in the group quantized by this group, and B is the maximum signed positive number that the quantization bit can be, corresponding to 8 for 4-bit quantization and 128 for 8-bit quantization respectively.

6. The neural network optimization method based on outlier separation and activation value hybrid quantization according to claim 3 is characterized in that: It is further deployed on the distributed training framework Megatron-LM: the activation values ​​are quantized, compressed and saved before being saved in the forward propagation function of each sub-layer. Before the back-propagation function calculates the gradients of each layer, the activation values ​​are restored to their original shape and size through the activation value recovery module to achieve neural network optimization.

7. A neural network optimization system based on outlier separation, activation value hybrid quantization for implementing any of the methods described in claims 1-6, characterized in that: include: An activation value analysis unit, an activation value processing unit, an activation value quantization unit, a storage unit and an activation value dequantization unit, wherein: the activation value analysis unit calculates the Z-Score value of each activation value channel according to the input activation value, and marks the channel whose Z-Score is greater than a specific threshold as an outlier channel according to a preset threshold, and the marks of all channels form a channel mask mask; the activation value processing unit slices and transposes the original activation value according to the channel mask mask, and divides the activation value into an outlier activation value tensor and a normal activation value tensor; the activation value quantization unit quantizes the outlier activation value tensor using 8 bits and the normal activation value tensor using 4 bits to obtain the corresponding activation value tensor; the storage unit saves the quantized outlier activation value and normal activation value and their quantization parameters and channel masks into the video memory; the activation value dequantization unit inversely processes the outlier activation value tensor and the normal activation value tensor according to the activation value tensor and its scaling factor, and then according to the channel mask mask and splices them into activation values ​​of the original size.

Images