Method for reducing video memory occupation during quantization parameter calculation
Through the method of updating the quantization parameters in batches and sliding average, the problem of excessive video memory usage during the model quantization process is solved, and the reduction of video memory usage and the maintenance of the quantization effect is achieved.
Patent Information
- Application Number
- CN202310603548.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-26
- Publication Date
- 2024-11-26
AI Technical Summary
In the process of model quantization, the prior art requires a large number of correction sets to count the min and max of each layer, resulting in excessive memory usage and prone to memory overflow problems, especially when the correction set is large or the resolution is high.
By inputting the correction data set in batches, counting and iterating the quantization parameters, updating min and max using the sliding average method, reducing the intermediate layer results saved in the video memory, retaining only the final min and max values, and reducing memory usage.
It effectively reduces the video memory usage when calculating quantization parameters, avoids video memory overflow, and the calculated min and max basically do not affect the quantization effect.
Smart Images

Figure CN119026643A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of neural network quantization, and in particular relates to a method for reducing video memory occupancy when calculating quantization parameters. Background Art
[0002] In the prior art, model quantization is the process of converting the weights and activation values of a trained deep neural network from high precision to low precision, such as converting a 32-bit floating point number into an 8-bit integer int8.
[0003] The basic formula for model quantization:
[0004] r=S(qz)
[0005]
[0006] The above quantization formula is the formula for quantization and dequantization. r and q represent the floating point number before quantization and the fixed point number after quantization respectively. S and Z are two important quantization parameters: scale (step size) and zero point (zero point). To achieve quantization and dequantization, the two parameters S and Z must be calculated.
[0007]
[0008]
[0009] The q in the formula max ,q min Indicates the value range of the fixed-point number q. It also indicates the range of quantization. For example, for 8-bit asymmetric quantization, the value range after quantization is generally 0 to 255, that is, q max =255,q min =0.
[0010] There are two very important quantitative parameters: r min 、r max , respectively represent the numerical range of the floating point number r. For a conventional neural network, as long as we know the r of each layer of feature map min 、r max , we can find S and Z, and quantize the features of each layer of the neural network. This process of quantizing each layer is referred to as quantizing the feature.
[0011] Because in the process of quantifying Features, it is necessary to count r min 、r max , a large number of calibration sets are needed as statistical r min 、r max The sample of r min 、r maxIn a suitable interval, the quantization is more accurate. When quantizing a feature, the statistical samples of this layer of features are often relatively large. In the actual process, these statistical samples will be uniformly stored in the video memory. When the statistical samples are too large, it will cause insufficient video memory and video memory overflow. The method introduced in this article is to reduce the video memory usage in this process. min 、r max In the process of statistics, a method is used to reduce the memory usage during the statistics process.
[0012] The min and max mentioned below refer to r in the above formula. min 、r max .
[0013] The existing methods to reduce video memory are:
[0014] 1. Low-precision calculation: Neural network calculations use 32-bit float type numbers. Low-precision calculations refer to using fewer bits to represent the values in the neural network, such as float16 or int8.
[0015] 2. Segmented calculation: Divide a model into several parts for calculation. This can reduce the use of video memory, but the cost is that the calculation speed will be slowed down.
[0016] However, the current method of calculating model quantization parameters is generally to infer the calibration set through the model, save the calibration set in the video memory through the output of each layer of the model, and then count the min and max values in the output of each layer of the model to quantize the model. In general, the more calibration sets a model has, the more accurate the calculated min and max values will be, thereby improving the quantization effect of the model. The current problem is that when the calibration set of the model input is large, or the original input resolution of the model is large, more video memory will be occupied when calculating the min and max values, resulting in video memory overflow.
[0017] In addition, commonly used technical terms in the prior art include:
[0018] 1. Model quantization: Model quantization is the process of converting the weights and activation values of a trained deep neural network from high precision to low precision.
[0019] 2. Quantization parameters: Parameters generated when establishing a data mapping relationship between fixed-point and floating-point data.
[0020] 3. Video memory: Graphics card memory. Also called frame buffer, video memory is a component used to store graphics information to be processed. Its function is to store rendering data that has been processed or is about to be extracted by the graphics card chip.
[0021] 4. Video memory usage: The proportion of rendered data that has been processed or is about to be extracted to the total video memory.
[0022] 5.Tensor: Tensor, representing a multi-dimensional matrix.
[0023] 6.min, max: The minimum and maximum floating-point values stored in a tensor. Summary of the invention
[0024] In order to solve the above problems, the purpose of this application is to: This method can obtain the min and max of model quantization under a large number of calibration sets, reduce the memory usage during the calculation process, and reduce the problem of memory overflow. And the calculated min and max basically do not affect the quantization effect.
[0025] Specifically, the present invention provides a method for reducing video memory occupancy when calculating quantization parameters, the method comprising the following steps:
[0026] S1, obtain a precise network model:
[0027] A precise network model is needed. The statistical min and max methods are used for quantization. The maximum value Rmax and the minimum value Rmin of the floating point range are obtained. In addition, the maximum value Qmax and the minimum value Qmin of the quantization range are obtained. Then, a linear mapping is performed through the scale s = (Rmax-Rmin) / (Qmax-Qmin), z = Qmax-Rmax / s: q = r / s+z. The precision loss before and after quantization is tested. In addition, the correction sets of different resolutions are used for statistics during the quantization process, and the memory usage during the statistics process is analyzed.
[0028] S2, input the calibration data set in batches:
[0029] S2.1, input the calibration set into the model in batches, and obtain the output results of the calibration set in each layer of Feature in the model;
[0030] S2.2, count the min and max of each layer of Feature and save them;
[0031] The calibration data set is input into the network model for inference. When the calibration set is inferred to the min and max layers of statistical quantization, the output result of the calibration set at the current layer is obtained using the quantization strategy, and then the min and max statistics are performed on the output result.
[0032] S2.3, after each batch of correction input sets, a min and max statistics work is performed. After the statistics are completed, the middle layer of the current batch of model inference saved in the video memory is deleted, and only the current min and max statistics are retained;
[0033] S2.4, iteratively update the min and max of the model in sequence, that is, after the min and max of the calibration set of the next batch are calculated, the min and max are iterated according to the actual min and max calculated and the min and max saved in the previous batch;
[0034] S3, obtain the final min and max as quantization parameters.
[0035] The step S3 uses a sliding average method to iterate the quantization parameters, that is, to update min and max using a sliding average method:
[0036] Compare the min_n and max_n calculated in the current batch with the min_o and max_o updated in the previous batch. Multiply the maximum and minimum values of the four numbers min_n, min_o, max_n and max_o by one. The coefficient of , plus the other two values of the four values multiplied by l, and the min and max values obtained are saved:
[0037] Here the corresponding relationship between the two coefficients
[0038]
[0039] Among them, min_o, max_o represent the minimum and maximum values of the previous batch, min_n, max_n represent the maximum and minimum values of the current batch, and are coefficients less than 1, and add up to 1, representing the previous retention * 100% and the current retention * 100% respectively.
[0040] Therefore, the advantages of the present application are: the method is simple, and a method for reducing video memory usage when calculating quantization parameters is used. By calculating the quantization parameters of the statistical model in batches, calculating the quantization parameters for each batch and iterating the quantization parameters through sliding average, the video memory usage in calculating the quantization parameters is reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.
[0042] Figure 1 It is a schematic diagram of the operation process of this method.
[0043] Figure 2This is a schematic diagram of the operation flow of step S2 in the original method.
[0044] Figure 3 It is a schematic diagram of the operation flow of step S2 in this method. DETAILED DESCRIPTION
[0045] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.
[0046] like Figure 1 As shown, the present application provides a method for reducing video memory usage during quantization, and the specific implementation steps are as follows:
[0047] Step S1: Obtain a precise network model
[0048] A precise network model is needed, and the statistical min and max methods mentioned above are used for quantization. Minmax is the simplest common quantization method. The maximum value Rmax and the minimum value Rmin of the floating point range are obtained. In addition, the maximum value Qmax and the minimum value Qmin of the quantization range are obtained. Then, a linear mapping is performed through the scale s = (Rmax-Rmin) / (Qmax-Qmin), z = Qmax-Rmax / s: q = r / s+z. The precision loss before and after quantization is tested, and the correction sets of different resolutions are used for statistics during the quantization process. The memory usage during the statistical process has different values for different networks, which is very different. For example, the values of resnet18 can be seen in Table 1.
[0049] Step S2: Input calibration data set in batches
[0050] The calibration data set is input into the model for inference. When the calibration set is inferred to the min and max layers of statistical quantization, the output result of the calibration set in the current layer is obtained, and then the min and max statistics are performed on the output results.
[0051] The existing statistics of quantization parameters min and max are to save the intermediate output of each batch of correction data set in the model. When all the correction data sets are inferred, the output results of each batch of correction data sets in the current layer are spliced into a large Tensor, and then min and max are calculated uniformly. This method makes it more convenient to count the quantization parameters, as only min and max need to be counted once. At the same time, this method also has relatively large disadvantages, because saving the output of each layer of the correction data set in the model requires video memory to store. The correction set is constantly input, and the output results of the current layer that need to be stored will increase. The video memory usage will also increase accordingly. This leads to insufficient video memory of the graphics card when the size of the correction data set is large, or the resolution of the correction set is large, or the model to be quantized is large. For example, Figure 2 As shown in the figure, the original method operation flow is: from the first batch of rectified data set inference intermediate layer batch-0 (512M), ..., to the n+1th batch of rectified data set inference intermediate layer, input, to the 2048M graphics card memory, to determine whether the total amount is greater than 2048M, that is, batch-0+batch-1+...+batch-n>2048M? If yes, the graphics memory overflows; if not, the batches are merged and min and max are calculated uniformly.
[0052] Under the above situation, this method proposes a more flexible way to reduce the memory usage in the process of quantizing parameters of statistical models. This method performs optimization work under the above situation:
[0053] After each batch of correction input sets, the min and max statistics are performed once. After the statistics are completed, the intermediate layer of the current batch of model inference saved in the video memory is deleted, and only the current min and max statistics are retained.
[0054] After the min and max values of the next batch of calibration sets are calculated, the min and max values are iterated based on the calculated min and max values and the actual min and max values saved in the previous batch.
[0055] like Figure 3As shown in the figure, the specific operation flow of this method is as follows: the first batch of corrected data set inference intermediate layer batch-0 (512M) is input; to the 2048M graphics card memory; min and max are calculated; then the second batch of corrected data set inference intermediate layer batch-1 (512M); min and max are calculated again, and the original min and max are updated; ...; until the n+1th batch of corrected data set inference intermediate layer is input; min and max are calculated, and min and max are updated to obtain the final min and max.
[0056] It can be further described as follows:
[0057] S2.1, input the calibration set into the model in batches, and obtain the output results of the calibration set in each layer of Feature in the model;
[0058] S2.2, count the min and max of each layer of feature, because each layer of the network needs to perform this operation and save it in the graphics card, but min and max are just two float values, and the features of each layer in the front account for the majority;
[0059] The calibration data set is input into the network model for inference. When the calibration set is inferred to the min and max layers of statistical quantization, the output result of the calibration set in the current layer is obtained, and then the min and max statistics are performed on the output result. A common quantization strategy: mainly focuses on the maximum and minimum values of the floating-point range, and then linearly maps through the scale S. Other quantization strategies include KLD, ADMM, etc. Since they are all public methods, they are not expanded in detail, and other methods can also be replaced here.
[0060] S2.3, after each batch of correction input sets, a min and max statistics work is performed. After the statistics are completed, the intermediate layers of the current batch of model inference saved in the video memory are deleted. The intermediate layers of the current batch are saved by calculating and saving the output features of each layer in step S2.1. Later, when the min and max are counted, only the min and max currently counted are retained; S2.4, iteratively update the min and max of the model in sequence, that is, after the min and max of the next batch of correction sets are calculated, the min and max are iterated according to the actual situation of the min and max counted and the min and max saved in the previous batch;
[0061] Step S3: Update min and max using sliding average
[0062] Compare the min_n and max_n calculated in the current batch with the min_o and max_o updated in the previous batch. Multiply the maximum and minimum values of the four numbers min_n, min_o, max_n and max_o by one. The coefficient of , plus the other two values of the four values multiplied by l, and the min and max values obtained are saved:
[0063]
[0064]
[0065] Among them, min_o, max_o represent the minimum and maximum values of the previous batch, min_n, max_n represent the maximum and minimum values of the current batch, and are coefficients less than 1, and add up to 1, representing the previous retention * 100% and the current retention * 100% respectively.
[0066] The method in this paper and the original method were used to test the quantized resnet18 network. The specific test results are shown in Table 1:
[0067] Table 1 Comparison of test results in resnet18 network
[0068]
[0069] Note: 1. Test network resnet18 2. Graphics card NVIDIA GeForce GTX 1080Ti 3. Video memory capacity 11G
[0070] In summary, this method reduces the memory usage in calculating quantization parameters.
[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for reducing video memory usage when calculating quantization parameters, characterized in that: The method comprises the following steps: S1, obtain a precise network model: Obtain a precise network model, and use the statistical min and max methods to quantize it, that is, obtain the maximum value Rmax and minimum value Rmin of the floating point range, the maximum value Qmax and minimum value Qmin of the quantization range, and then use the scale s = (Rmax-Rmin) / (Qmax-Qmin), z = Qmax-Rmax / s to perform linear mapping: q = r / s+z, test the precision loss before and after quantization, and use correction sets of different resolutions to perform statistics during the quantization process, and the memory usage during the statistics process; S2, input the calibration data set in batches: S2.1, input the calibration set into the model in batches, and obtain the output results of the calibration set in each layer of Feature in the model; S2.2, count the min and max of each layer of Feature and save them; The calibration data set is input into the network model for inference. When the calibration set is inferred to the min and max layers of statistical quantization, the output result of the calibration set at the current layer is obtained using the quantization strategy, and then the min and max statistics are performed on the output result. S2.3, after each batch of correction input sets, a min and max statistics work is performed. After the statistics are completed, the middle layer of the current batch of model inference saved in the video memory is deleted, and only the current min and max statistics are retained; S2.4, iteratively update the min and max of the model in sequence, that is, after the min and max of the calibration set of the next batch are calculated, the min and max are iterated according to the actual min and max calculated and the min and max saved in the previous batch; S3, obtain the final min and max as quantization parameters.
2. The method for reducing video memory usage when calculating quantization parameters according to claim 1, characterized in that: The step S3 uses a sliding average method to iterate the quantization parameters, that is, to update min and max using a sliding average method: Compare the min_n and max_n calculated in the current batch with the min_o and max_o updated in the previous batch. Multiply the maximum and minimum values of the four numbers min_n, min_o, max_n and max_o by one. The coefficient of , plus the other two values of the four values multiplied by l, and the min and max values obtained are saved: Here the corresponding relationship between the two coefficients Among them, min_o, max_o represent the minimum and maximum values of the previous batch, min_n, max_n represent the maximum and minimum values of the current batch, and are coefficients less than 1, and add up to 1, representing the previous retention * 100% and the current retention * 100% respectively.
Citation Information
Patent Citations
Neural network accelerator model quantification method
CN114139683A
Data calculation method and device, storage medium and equipment
CN114861907A
Video memory recalculation optimization method and system for load awareness of heterogeneous GPU (Graphics Processing Unit) cluster
CN115437795A
Model quantization method and apparatus, device, storage medium, computer program product, and computer program
WO2023029349A1
