A method and device for post-training quantization calculation of a convolutional neural network

Abstract

The application discloses a kind of convolutional neural network post-training quantization calculation method and device, the weight data of a convolution layer is sequentially taken out in the way of each channel, the dynamic segmentation of 2 base number and root 2 base number is carried out to the weight data extracted, the weight data after dynamic segmentation is quantized, then the weight after quantization is offset correction operation, the weight data of next layer is quantized in the same order, until all the weight data in model is quantized, the weight data completed quantization is calculated with input data on the special hardware computing unit designed;The application utilizes the characteristics that neural network model weight data is distributed differently in each output channel, and dynamically finds the optimal segmentation parameter R of each channel in the quantization process by the traversal method, this method can reduce quantization error, so as to bring the improvement of model inference accuracy after quantization.

Description

Technical Field

[0001] This invention relates to the fields of neural networks and artificial intelligence, and in particular to a method and apparatus for post-training quantization calculation of convolutional neural networks. Background Technology

[0002] Quantization plays a crucial role in reducing the computational complexity and memory footprint of convolutional neural networks (CNNs). Quantization compresses the weights and input data of CNN models and can reduce overall hardware overhead (including area and power consumption) during model deployment. Lower quantization precision results in less memory data movement and lower computational consumption. Quantization methods are generally divided into two categories: quantization-aware training and post-training quantization. Quantization-aware training introduces quantization at the beginning of model training or retrains an already trained model. In this way, CNN models can adapt to quantization errors of 2-4 bits or even binary, achieving considerable inference accuracy even with extremely low quantization precision.

[0003] Quantization-aware training relies on the original training dataset, which is time-consuming and labor-intensive. Furthermore, datasets are difficult to obtain in fields such as military and medical applications, and can even lead to privacy violations. Post-training quantization methods can typically compress pre-trained convolutional neural network models to 4-8 bits without requiring retraining on the original dataset. Therefore, post-training quantization methods allow for rapid deployment when the original training dataset is unavailable. Due to their lack of dependence on the original training dataset and the fact that they do not require retraining, post-training quantization methods have been widely researched and applied. While existing post-training quantization methods reduce computational complexity through low-precision quantization, they also tend to decrease model accuracy. Maintaining high model accuracy, on the other hand, increases computational load, which can reduce model speed and efficiency to some extent. Therefore, it is necessary to address the problem of decreased model accuracy caused by low-precision quantization. Summary of the Invention

[0004] The purpose of this invention is to provide a method and apparatus for post-training quantization calculation of convolutional neural networks. This method reduces the quantization bit width while maintaining a certain level of model inference accuracy. It employs channel-wise quantization granularity and implements a dedicated hardware computing unit without multipliers through quantization. During the quantization process, it introduces a piecewise logarithmic quantization scheme with base 2 and base √2 to adapt to the bell-shaped distribution of weight data in the neural network. The optimal piecewise parameter R is dynamically selected during quantization to ensure minimal local quantization error. After quantizing the weight data in each channel, offset correction calculations are performed to further reduce quantization error. This invention reduces the quantization accuracy of the weight data in the model to a 4-bit width while maintaining a certain level of inference accuracy. Furthermore, this invention designs a dedicated computing unit without multipliers to support the aforementioned 4-bit channel-wise dynamic piecewise logarithmic post-training quantization calculation method.

[0005] A post-training quantization calculation method for convolutional neural networks includes the following steps: Step 1, extracting the weight data of a convolutional layer sequentially channel by channel; Step 2, dynamically segmenting the extracted weight data into base 2 and base √2; Step 3, quantizing the dynamically segmented weight data, and then performing an offset correction operation on the quantized weights; Step 4, quantizing the weight data of the next layer in the same order until all weight data in the model has been quantized; Step 5, performing calculations on the quantized weight data and input data on a dedicated hardware computing unit.

[0006] Furthermore, the dynamic segmentation employs a mixed-base logarithmic encoding scheme. For weights with larger absolute values, a finer-grained logarithmic encoding based on the square root of 2 is used, while for weights with smaller absolute values, a logarithmic encoding based on base 2 is used. The codebook using mixed-base logarithmic encoding contains N1 logarithmic encodings based on the square root of 2 and N2 logarithmic encodings based on base 2, where the number of base-2 encodings is defined as R.

[0007] Furthermore, quantization introduces logarithmic quantization with base 2. The formula for multiplying input data and weights in a neural network is as follows:

[0008]

[0009] Where Wq is the result of logarithmic quantization of the weight data with base 2, Xq is the input data X after fixed-point quantization, and ax and aw are scaling parameters.

[0010] The segmented quantization method employs a mixed-base logarithmic encoding scheme, which better represents the bell-shaped weight distribution and thus reduces quantization error. For weights with larger absolute values, a finer-grained encoding based on the square root of 2 is used to improve the upper limit of accuracy; while for weights with smaller absolute values, a base-2 encoding is used to cover weights that are more densely distributed in the near-zero region.

[0011] Furthermore, the logarithmic encoding scheme includes one sign bit and (N-1) encoding value bits. The value of the segmentation parameter R is the number of weights quantized to base 2, R = N², N₁ + N₂ = 2. (N-1) Where N is the number of quantization bits, in this invention N=4, N1 is the number of weight data quantized based on the square root of 2, and N2 is the number of weight data quantized based on 2 pairs. During the quantization process, the quantization region is divided into two parts using a breakpoint p: a quantization region R1 with the base 2 and a quantization region R2 with the base 2. The range of R1 is... The range of R2 is p is a breakpoint, and the definition of breakpoint p is as follows:

[0012]

[0013] Here, max refers to the maximum absolute value among the weighted data.

[0014] The meanings of each part in the equation for breakpoint p are as follows: First, the maximum value of the unquantized weight, max, is used according to the equation... The minimum value of the full-precision weighted data quantified using base-2 pairs can be determined; then, the formula can be used... The maximum value of the full-precision weighted data based on base 2 logarithmic quantification can be determined. The sum of the exponents of the two equations divided by 2 gives the value of the breakpoint p.

[0015] Furthermore, in each quantization region, the quantization method is shown in the following formula:

[0016]

[0017]

[0018] Wherein, sgn(x) represents the sign of the full-precision numerical value. In the R1 region, all weights are quantized with the base 2. In the quantization formula Quant, b is the base 2 for all weights in the R1 region. In the quantization formula Quant, b is 2 for all weights in the R2 region. The formula SLog is the quantization and dequantization process of the weight data.

[0019] Compared with layer-by-layer quantization, channel-by-channel quantization can effectively improve quantization performance. Since the weight distribution of each output channel is different, assigning the most suitable segmentation parameter R to each output channel can bring better quantization performance.

[0020] In layer-level piecewise logarithmic quantization, the piecewise parameter R can be determined by layer-by-layer enumeration combined with accuracy testing. However, for channel-by-channel quantization, the total number of channels is much larger than the number of layers in the network model, making the enumeration method combined with accuracy testing extremely inefficient. Therefore, this invention uses the normalized mean square error (NMSE) before and after channel-by-channel weight data quantization as a metric to dynamically determine the piecewise parameter R that minimizes the local quantization error, thereby improving the inference accuracy of the quantized network model.

[0021] Furthermore, the method for determining the locally optimal segmentation parameter R is as follows: Calculate the normalized mean square error of the weighted data before and after quantization under different segmentation parameters R, and quickly search for the optimal segmentation parameter R for each channel. The calculation formula is: In different convolution output channels, the optimal segmentation parameter R for each channel is quickly searched. The local optimal quantization segmentation parameter R is determined by finding the minimum normalized mean square error before and after weight quantization through enumeration.

[0022] In the formula for calculating R, This represents the normalized mean square error, used to describe the similarity between the weighted data after quantization and before quantization, where w is the original data. This means that after segmenting and quantizing the data, a segmentation parameter R is found using a traversal method to maximize the similarity of the weighted data before and after quantization.

[0023] The segmentation parameter R ranges from 0 to 8. During the quantization process, NMSE is used to measure the quantization error at the tensor level to find the local optimum of the quantization segmentation parameter R.

[0024] Furthermore, the offset correction includes two parameters, μ and ξ, which are solved as follows: The quantized weight data is compensated using two calculated offset correction parameters: μ compensates for the error in the mean of the weight data before and after quantization, and ξ compensates for the error in the variance of the weight data before and after quantization. The specific formulas are as follows. Among them W b The weighted data after offset correction. For the input data after fixed-point quantization, This indicates the convolution calculation of the input data and the corrected weight data.

[0025] Furthermore, logarithmic encoding based on mixed bases is divided into three types: the first type is logarithmic encoding with base 2, the second type is logarithmic encoding with base √2 and even exponent, and the third type is logarithmic encoding with base √2 and odd exponent.

[0026] For the first and second types of quantized weights, the multiplication and addition calculations between the input data and the weight data can be directly converted into shift and addition to accelerate the calculation. For the third type of quantized weights, when multiplying the input data and the weight data, it is necessary to approximate the calculation of the square root of 2. The series expansion of the square root of 2 is shown below:

[0027]

[0028] After approximation, the multiplication and addition operations between the third type of quantization weights and the input data can also be converted into shift and addition operations.

[0029] Furthermore, this invention evaluates the approximate calculation of the square root of 2 during the hardware design of the computing unit. This invention comprehensively evaluates the inference accuracy of the post-trained quantization model presented in the form of the original square root of 2 with binomial, trinomial, and tetranomial approximations, as well as the hardware resource consumption, and ultimately selects the binomial expansion of the square root of 2 as the approximation standard. The theoretical basis is as follows: In Synopsys Design Compiler, using the UMC 40nm standard CMOS process, the power consumption and area of ​​the arithmetic unit circuits designed with the square root of 2 using binomial, trinomial, and tetranomial methods were synthesized. The circuit's operating frequency during synthesis was set to 200MHz, and the operating voltage was 1.1V. Figure 2 As shown, the accuracy of the square root of 2 without approximation, and with binomial, trinomial, and tetranomial approximations in piecewise logarithmic quantization, on several different models, is compared with the resource requirements of the computational unit in hardware implementation. Compared with the binomial approximation, the trinomial and tetranomial approximations only improve the Top-1 accuracy of inference for models such as VGG16, ResNet50, and ResNet18 by less than 0.15%, but increase power consumption by 25.7% and 45.9% respectively, and increase area by 14.0% and 30.9% respectively in hardware implementation. Therefore, the benefits of using the binomial approximation in the square root of 2 approximation are greater.

[0030] The advantages of this invention are as follows: In low-bit-precision quantization scenarios, post-training quantization avoids accessing the training dataset, and does not require significant manpower, equipment, and time costs during model compression. The proposed channel-by-channel dynamic segmentation logarithmic quantization method employs a mixed-base logarithmic encoding scheme, better representing the bell-shaped distribution of weights and thus reducing quantization errors. A multiplier-free computational unit is designed to support computation based on the characteristics of the quantization method. This method ensures the accuracy of post-training quantization inference at low bit-precision, and the designed computational unit guarantees the speed and efficiency of hardware computation of input and weight data. This invention utilizes the characteristic that the weight data of a neural network model has a different distribution in each output channel. During the quantization process, the optimal segmentation parameter R for each channel is dynamically found through a traversal method. This method reduces quantization errors, thereby improving the inference accuracy of the quantized model. Simultaneously, after quantizing the weight data of each channel, an offset correction operation is performed on the quantized weights to further compensate for quantization errors. This step improves the performance of the post-training quantization method. Attached Figure Description

[0031] Figure 1 This is a visualization comparison chart of the linearity and logarithmic domain of the channel-by-channel dynamic segmentation quantization method.

[0032] Figure 2 This is a comparison chart showing the decrease in model accuracy and the required hardware resources resulting from using different approximation methods in the channel-by-channel segmentation quantification approach.

[0033] Figure 3 This is a structural diagram of a dedicated hardware computing unit device proposed based on the channel-by-channel segmentation quantization method.

[0034] Figure 4 This is a schematic diagram showing the accuracy results of different models using different configurations for image classification tasks using logistic quantification methods. Detailed Implementation

[0035] To address the shortcomings of existing technologies, this invention provides a post-training quantization calculation method and apparatus for convolutional neural networks. While improving the accuracy of the model, a dedicated hardware computing unit circuit is designed to ensure the computational efficiency of the quantized model. This invention adopts a post-training quantization method, which improves the performance of quantization without consuming a large amount of manpower, equipment, and time.

[0036] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0037] As one implementation method, the channel-by-channel dynamic segmentation quantization employs a logarithmic encoding scheme based on a mixed base. This method can better represent the bell-shaped weight distribution of the weighted data, thereby reducing quantization errors. Figure 1 As shown, (a) illustrates the layer-by-layer quantization method, with one codebook per layer; (b) illustrates the channel-by-channel quantization method, with one codebook per channel; (c) and (d) are visualizations of the logarithmic quantization results of the weight data in layer 2.1.conv1 of the ResNet18 pre-trained model on ImageNet, respectively. (c) shows the quantized image in the linear domain, and (d) shows the quantized image in the logarithmic domain. The two figures respectively demonstrate logarithmic quantization using only base 2, logarithmic quantization using base 2, and piecewise logarithmic quantization based on a mixed base. The piecewise parameter R = 6:2 indicates that at 4-bit quantization precision, there are 6 absolute value points of the logarithmic quantization using base 2 and 2 absolute value points of the logarithmic quantization using base 2. Figure (e) shows the data distribution characteristics of the weight data in layer 2.1.conv1 of the ResNet18 model across different channels. Figure (f) shows the normalized mean square error (NMSE) of the weight data before and after quantization under different segmentation parameters R for segmented logarithmic quantization on different channels. Figure (g) shows the impact of the selection of the segmentation parameter R on the inference accuracy of the model after quantization. As can be seen from Figures (c) and (d), for weights with larger absolute values, a finer-grained base-2 encoding is used to improve the upper limit of accuracy; while for weights with smaller absolute values, base-2 encoding is used to cover weights that are more densely distributed in the near-zero region. Using a hybrid base quantization method, the quantization codebook will contain N1 base-2 logarithmic codes and N2 base-2 logarithmic codes. The segmentation parameter R is defined as the number of base-2 codes.

[0038] Preferred, Figure 1 This demonstrates the impact of different channel weight data distributions and the selection of different segmentation parameters R on the inference accuracy of the quantized model. For example... Figure 1 As shown, the weight data distribution and the channel-by-channel piecewise pair quantization method are as follows: (a) is layer-by-layer quantization, with one codebook for each layer; (b) is channel-by-channel quantization, with one codebook for each channel; (e) shows the weight data distribution on different channels; and (f) shows the normalized mean square error (NMSE) of the weight data of different channels before and after piecewise pair quantization under different piecewise parameters R.

[0039] Preferably, in this case, the quantization code includes one sign bit and (N-1) code value bits. During the quantization process, the quantization region is divided into two parts using a breakpoint p: a quantization region R1 with base √2 and a quantization region R2 with base 2. The range of R1 is... The range of R2 is The breakpoint p is defined as follows: Where max refers to the maximum absolute value in the data. In each quantization region, the segmented implementation of the quantization method is shown in the following formula:

[0040]

[0041] in, The sign refers to rounding; sgn(x) represents the sign of a full-precision numerical value.

[0042] Preferably, the maximum value of the unquantized weights, max, is first used according to the formula... The minimum value of the full-precision weighted data quantified using base-2 pairs can be determined; then, the formula can be used... The maximum value of the full-precision weighted data quantized using base 2 can be determined. The sum of the exponents of the two equations divided by 2 gives the value of the breakpoint p. For example, if the maximum value of the weighted data in a certain channel is 0.25, N1 is 3, the maximum exponent of the quantization value with base 2 is -4, the exponents of the quantization value with base 2 are -4, -5, and -6, and the exponents of the quantization value with base 2 are -4, -5, -6, -7, and -8, then the calculated result of the breakpoint p is (-6) / 2 + (-4) = -7.

[0043] Preferably, in region R1, all weights are quantized to the base 2, so b in the SLog expression is √2 during quantization. Similarly, in region R2, b is 2 during quantization. The SLog expression is the quantization-dequantization process for the weight data. Taking base-2 quantization as an example, Quant(|x|;2) will yield an integer quantized value after calculation. This quantized value in the SLog expression is used as the exponent of 2 for dequantization. For example, suppose the value of x is -4.2, and it belongs to region R2. Then Quant(4.2;2) represents quantizing the data 4.2 to the base 2 and rounding it, resulting in 2. During dequantization, sgn(x) is -1, and the result after dequantization using the SLog expression is -4.

[0044] Preferably, the zero value in the full-precision numerical value and the minimum value in the segmented pair quantization codebook will be mapped to zero.

[0045] Preferably, the present invention employs a channel-by-channel quantization method, such as... Figure 1As shown in (b) and (e), for the weight data of the same convolutional layer, the weight data distribution characteristics differ in different output channels. Practice has proven that, compared to layer-by-layer quantization, channel-by-channel quantization can effectively improve quantization performance. Furthermore, for piecewise logarithmic quantization, since the weight distribution of each output channel is different, assigning the most suitable piecewise quantization parameter R to each output channel intuitively leads to better quantization performance. In layer-level piecewise logarithmic quantization, the piecewise quantization parameter R can be determined by layer-by-layer enumeration combined with accuracy testing. However, the total number of channels is much larger than the number of layers in the network model, making this method of enumerating and combining accuracy to find the optimal R for each channel impractical for channel-level quantization. Therefore, this invention proposes a strategy for dynamically determining R to quickly determine the local optimum R, thereby reducing quantization error and achieving higher quantization model inference accuracy.

[0046] As one implementation method, piecewise pair quantization can find the optimal piecewise parameter R by traversing the model to achieve the best inference accuracy. While this is an intuitive and effective method, it is very time-consuming for channel-by-channel quantization. To evaluate the impact of the piecewise parameter R on the model's accuracy loss, Figure 1 (g) illustrates the accuracy loss of ResNet18 after channel-by-channel quantization under different segmentation parameters R. The segmentation parameter R ranges from 0:8 to 8:0, where 0:8 means all data are logarithmically quantized to base 2, 8:0 means all data are logarithmically quantized to base 2, and R = 6:2 means there are 6 absolute value points of logarithmic quantization to base 2 and 2 absolute value points of logarithmic quantization to base 2. In addition to the inference accuracy of the quantized model, the normalized mean squared error (NMSE) is usually used to measure the quantization error at the tensor level during the quantization process. Figure 1 As shown in (f), the optimal segmentation parameter R for quantizing weights using the channel-by-channel segmentation quantization method is different in different output channels. The dynamic segmentation strategy proposed in this invention can quickly search for the optimal segmentation parameter R for each channel. Specifically, Although normalized mean squared error (NMSE) cannot strictly predict the inference accuracy loss of a quantized model, there is a positive correlation between them. Dynamic segmentation, by minimizing the NMSE before and after weight quantization, is a technique that can improve the performance of quantization through channel-wise post-training and helps to find the locally optimal quantization segmentation parameters R.

[0047] As one implementation method, after the weight values ​​are quantized, their mean and variance inherently deviate from the original weight data. If W is used... c W represents the original weight of a channel.c q W represents the quantized channel weights. c and W c q There is an inherent bias between them, that is, E(W) c )≠E(W c q ), and ||W c -E(Wc)||2≠||W c q -E(W c q 2. Offset correction is a very effective quantization error compensation technique that can significantly improve the performance of post-training quantization. Offset correction has two important parameters, μ and ξ, which are solved as follows: After obtaining these two parameters, they can be used to compensate for the quantized weighted data. Specifically, It should be noted that an offset correction operation is performed after the weight data of each channel is quantized. Therefore, each channel has an independent offset correction parameter, where w b The weighted data after offset correction. For the input data after fixed-point quantization, This indicates the convolution calculation of the input data and the corrected weight data.

[0048] As one implementation method, to investigate the impact of each of the above techniques on model quantization performance, this invention conducted ablation experiments, employing different modalities of logarithmic quantization methods to quantize the weights of eight pre-trained CNN models, such as... Figure 4 As shown, CNN models such as VGG16 to DenseNet are used for image classification tasks and are pre-trained on the ImageNet dataset. FaceNet and Face.evoLVe models are used for face recognition tasks and are pre-trained on the Labeled Faces in the Wild (LFW) dataset and the VGGFace2 and MS-Celeb-1M datasets. Figure 4 Several aspects can be observed: 1) Compared with the layer-by-layer pairwise quantization method, the channel-by-channel pairwise quantization method has better quantization performance; 2) The piecewise pairwise quantization method (DSLog) using a dynamic segmentation strategy can effectively improve the model's performance in 4... Bit Accuracy under quantization precision; 3) The method of dynamic segmentation of each channel combined with offset correction technology proposed in this invention can push the limit of quantization to 4 bits, and has an accuracy loss of <2.5% in TOP-1 accuracy on all the above CNN model inferences.

[0049] Preferred, such as Figure 4 The results show the accuracy of different logarithmic quantization methods on different network models after weights are trained with 4 bits. PL refers to layer-by-layer quantization, PC refers to channel-by-channel quantization, and Log2 indicates that the weight data is logarithmically quantized to base 2. The weighting is based on Logarithmic quantization of the base. DSLog refers to hybrid base logarithmic quantization using dynamic piecewise segmentation. SLogII refers to a channel-by-channel dynamic piecewise logarithmic quantization method using DSLog combined with offset correction techniques.

[0050] As one implementation method, the weights after channel-by-channel logarithmic quantization can be divided into three types: 1) T1: logarithmic quantization with base 2, 2) T2: logarithmic quantization with base √2 and an even exponent, 3) T3: logarithmic quantization with base √2 and an odd exponent. For the quantized weights of categories T1 and T2, the multiplication and addition calculation of the input data and weights can still be easily converted into shift and addition for accelerated calculation. Only for the quantized weights of category T3, the multiplication calculation cannot be directly converted into shift for acceleration, but the √2 can be approximated. The series expansion of √2 is shown below: Using the square root of 2 approximation, the multiplication-addition operation between the quantized weights of category T3 and the input data can also be converted into shift and addition operations. The following formula illustrates the case when calculating the logarithmic quantized weights with base 2 in relation to the input data:

[0051]

[0052] In the formula, ω represents the full-precision weight data. q The logarithmic result of w with base 2 is shown in the formula, where the multiplication of the input data and the weight data is as follows: when ω q When ω is even, there is no need to approximate the calculation of the square root of 2. q When the number is odd, the square root of 2 needs to be approximated. The example shown is that the square root of 2 is approximated as 1 + 1 / 2.

[0053] Preferably, in the approximation of the square root of 2, the more terms in the approximation polynomial, the closer it gets to the true value of the square root of 2. However, increasing the number of polynomial terms will lead to higher hardware overhead in the computational unit implementation. To evaluate the impact of the square root of 2 approximation, such as... Figure 2As shown, the Top-1 accuracy loss of several approximations of the square root of 2 were tested on networks such as VGG16, ResNet18, and ResNet50, and the hardware resource consumption of using several different approximation methods was also tested. In Synopsys Design Compiler, the power consumption and area of ​​different arithmetic unit circuits were synthesized using a 40nm standard CMOS process, with an operating frequency of 200MHz and an operating voltage of 1.1V. For example... Figure 2 As shown, compared with the binomial approximation, the trinomial and tetranomial approximations only improve the Top-1 accuracy of inference for models such as VGG16, ResNet50, and ResNet18 by less than 0.15%, but increase power consumption by 25.7% and 45.9% respectively, and increase area by 14.0% and 30.9% respectively. Therefore, the computing unit in this invention adopts the binomial square root of 2 approximation scheme in its implementation.

[0054] As one implementation method, such as Figure 2 As shown, the accuracy loss of the approximation of the square root of 2 using binomials, trinomials, and tetranomials in the channel-by-channel segmentation quantization method on models such as VGG16, ResNet18, and ResNet50 is compared with the hardware resources required for different approximation methods.

[0055] As one implementation method, such as Figure 3 As shown, this invention designs a dedicated hardware computing unit for the quantization method of 4-bit channel-by-channel dynamic segmentation. The SLogII AE in the figure is the arithmetic operation unit, internally implemented with fully combinational logic, mainly including modules such as decoder, sign determination, approximation calculation, and shifter. The input ports of the operation unit module are described as follows:

[0056] ■4-bit w_code: Weight encoding after quantization through channel-by-channel dynamic segmentation;

[0057] ■8-bit ifmap: Input image data after fixed-point quantization;

[0058] ■ The 4-bit code_info, in the channel-by-channel dynamic segmentation quantization method, each channel has a set of auxiliary codebooks used for w. _ Decoding the code.

[0059] Preferably, the sign determination module functions as follows: when the sign bit of w_code is 0, it means that w_code is a positive number, and the data sent to the approximation calculation module is the original input data ifmap; when the sign bit of w_code is 1, it means that w_code is a negative number, and the data sent to the approximation calculation module is the original input data inverted and then incremented by one, i.e., ~ifmap+1.

[0060] Preferably, the decoder module functions as follows: based on the absolute value of w_code, the segmentation parameter r, and the 1-bit auxiliary code, it can decode the shift bit number Shift_bits and the approximation flag Approx_flag.

[0061] Preferably, the approximation calculation module functions as follows: Based on the outputs of the decoder module and the symbol determination module, if the approximation calculation flag Approx_flag is 1, it indicates that an approximation calculation is needed for the input data ifmap, and the output signal Shift_input of this module is (1+1 / 2)*input; if the approximation calculation flag Approx_flag is 0, it indicates that an approximation calculation is not needed for the input data ifmap, and the value of Shift_input is input.

[0062] Preferably, the shifter module functions as follows: based on the outputs of the decoder module and the approximate calculation module, it shifts the signal to be shifted, Shift_input, by a number of bits, Shift_bits. The output of this module is then fed into the accumulator.

[0063] The beneficial effects of this invention are as follows: In low-bit-precision quantization scenarios, post-training quantization avoids accessing the training dataset, eliminating the need for significant manpower, equipment, and time costs during model compression. The proposed channel-by-channel dynamic segmentation quantization method employs a mixed-base logarithmic encoding scheme, better representing the bell-shaped distribution of weights and thus reducing quantization errors. A computational unit is designed to support the computation based on the characteristics of the quantization method, ensuring the accuracy of post-training quantization inference. The designed computational unit guarantees the speed and efficiency of hardware computation of input and weight data. This invention utilizes the characteristic that the weight data of a neural network model has a different distribution in each output channel. During quantization, the optimal segmentation parameter R for each channel is dynamically found through a traversal method. This method reduces quantization errors, thereby improving the inference accuracy of the model after quantization at low bit-precision. Simultaneously, after quantizing the weight data of each channel, an offset correction operation is performed on the quantized weights to further compensate for quantization errors. This step further improves the performance of the post-training quantization method.

[0064] All patents and publications mentioned in this specification represent publicly available technology that can be used by this invention. All patents and publications cited herein are also listed in the references as individually referenced. The invention described herein can be practiced in the absence of any one or more elements, or one or more limitations, which are not specifically stated herein. For example, the terms “comprising,” “substantially consisting of,” and “consisting of” in each instance herein can be replaced by the other two terms. The terminology and expressions used herein are descriptive and not limiting, and there is no intention to suggest that these terms and interpretations exclude any equivalent features; however, it is understood that any suitable changes or modifications can be made within the scope of this invention and the claims. It is understood that the embodiments described herein are preferred embodiments and features, and any modifications and variations can be made by those skilled in the art based on the spirit of the description, and such modifications and variations are also considered to fall within the scope of this invention and the limitations of the independent and appended claims.

Claims

1. A method for post-training quantization computation of a convolutional neural network, characterized in that, The steps include the following: Step 1: Extract the weight data of a convolutional layer channel by channel. Step 2: Dynamically segment the extracted weighted data using base 2 and base √2. Step 3: Quantize the dynamically segmented weight data, and then perform offset correction on the quantized weights. Step 4: Quantize the weight data of the next layer in the same order until all weight data in the model has been quantized. Step 5: The quantified weight data is then used to perform calculations with the input data on the designed dedicated hardware computing unit; The dynamic segmentation quantization in step 3 adopts a logarithmic encoding scheme based on mixed bases. For weights with larger absolute values, a finer-grained logarithmic quantization based on the square root of 2 is used, while for weights with smaller absolute values, a logarithmic quantization based on the base of 2 is used. The codebook using mixed base logarithmic quantization contains N1 logarithmic quantization codes based on the square root of 2 and N2 logarithmic quantization codes based on the base of 2. The weighted data quantization introduced here uses a logarithmic quantization with base 2. The formula for multiplying input data and weight values ​​in a neural network is as follows: ； where W q is the weight data, X q is the input data, a x and a w are scaling parameters; and where W q is the weight data, X q is the input data, a x and a w are scaling parameters; and The logarithmic coding scheme contains one sign bit and (N-1) coding value bits, where N is the number of quantization bits, and in addition, one segment parameter R is needed for each channel, where The breakpoint p divides the quantization region into two parts, quantization region R1 with base 2 and quantization region R2 with base 2, during the quantization process, the range of R1 is The range of R2 is The definition of the breakpoint p is as follows: ； Where max refers to the maximum absolute value among the weighted data; Channel-by-channel quantization can effectively improve quantization performance. Since the weight distribution of each output channel is different, assigning the most suitable segmentation parameter R to each output channel can bring better quantization performance. A dedicated hardware computing unit was designed for the quantization method of 4-bit channel-wise dynamic segmentation. The SLogII AE is the arithmetic operation unit, which is implemented with full combinational logic and mainly includes modules such as decoder, sign determination, approximation calculation, and shifter. The input ports of the operation unit module are described as follows: 4-bit w_code: Weight encoding after quantization through channel-by-channel dynamic segmentation; 8-bit ifmap: Input image data after fixed-point quantization; In the channel-by-channel dynamic segmentation quantization method, each channel of the 4-bit code_info has an auxiliary codebook used for decoding w_code. 2.The method of claim 1, wherein: In the aforementioned quantization region, the quantization dequantization process is shown in the following formula: ， Where sgn(x) represents the sign of the full-precision numerical value, all weights in region R1 are quantized with b = √2, all weights in region R2 are quantized with b = 2, and the SLog formula is the process of dequantizing the data. 3.The method of claim 1, wherein: The weighted data quantization process quantizes the weighted data of each channel, and ensures that the local quantization error is minimized by selecting the segmented parameter R that minimizes the normalized mean square error before and after quantization.

4. The method for post-training quantization calculation of a convolutional neural network according to claim 1, characterized in that: The method for determining the locally optimal segmentation parameter R is as follows: quantization error is calculated for different R values, and the formula for calculating the quantization error is: In different convolution output channels, the local optimum segmentation parameter R is determined by finding the minimum quantization error through enumeration.

5. The method of claim 1, wherein: The offset correction operation requires two parameters, µ and ξ, which are calculated using the following formulas: ， Where W is the weight tensor before quantization, The quantized weight tensor uses the µ parameter to compensate for errors in the mean of the weight data before and after quantization, and the ξ parameter to compensate for errors in the variance of the weight data before and after quantization. The specific correction process is as follows. ,in The weighted data after offset correction. For the input data after fixed-point quantization, This indicates the convolution calculation of the input data and the corrected weight data.

6. The method for post-training quantization calculation of a convolutional neural network according to claim 1, characterized in that: The logarithmic encoding based on mixed bases is divided into three types: the first type is logarithmic encoding with base 2, the second type is logarithmic encoding with base √2 and even exponent, and the third type is logarithmic encoding with base √2 and odd exponent.

7. The method for post-training quantization calculation of a convolutional neural network according to claim 1, characterized in that: In the aforementioned hardware computation, for the logarithmic quantization with base 2 and an odd exponent, an approximation operation is required during computation. The square root of 2 is expanded into a series. Considering the trade-off between the inference accuracy after model quantization and the hardware overhead of the computation unit design, the square root of 2 is approximated as a binomial, that is, the square root of 2 is approximated as 1 + 1 / 2 during the computation unit design.

Images