Hardware-friendly integer quantization method for post-processing of deep learning algorithms for object detection

Abstract

The present invention provides a hardware-friendly integer quantization method for post-processing of a target detection deep learning algorithm, and relates to the technical field of target detection deep learning algorithms. The main steps are: using the output feature maps of the classification branch and the regression branch of the detection head as the input feature maps for post-processing; generating quantized multiplication factors MULT and shift factors SHIFT, and completing the quantization of the post-processed input; performing post-processing, and then outputting a prediction frame; completing the quantization of anchor point generation; completing the calculation of the coordinates of the upper left corner and the lower right corner of the prediction frame; and completing the calculation of the prediction frame after inverse quantization. The present invention selects negative integer powers of 2 as quantization scale factors for post-processing inputs, and the related calculations of the prediction frame are converted from 32 floating point numbers into integer operations with fewer bits, which can reduce computing resources and computing bandwidth, improve reasoning speed, and meet real-time requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection deep learning algorithms, and in particular to a hardware-friendly integer quantization method for post-processing of target detection deep learning algorithms. Background Art

[0002] In the field of computer vision, deep learning algorithms for target detection have become mainstream algorithms, and deep learning algorithms for target detection are constantly updated and iterated, such as YOLO from version v1 in 2015 to version v8 in 2023. The target detection algorithm model itself contains a backbone network, a neck network, and a detection head. The current mainstream detection head adopts a decoupled structure, that is, separating the classification branch and the regression branch. The detection head will generate a large number of candidate boxes with different confidence levels for the same target, so post-processing is required after the model output to find the best target prediction box and remove redundant prediction boxes.

[0003] The deep learning algorithm for target detection needs to complete multi-GPU training on the server side. In order to improve the inference speed, the deep learning algorithm for target detection needs to be quantized. The commonly used quantization method is the 8-bit integer quantization method, such as the "post-training quantization" PTQ (PostTrainingQuantization) provided by PyTorch. However, the quantization of the deep learning algorithm for target detection is only for the model itself, that is, only for the backbone network (Backbone), neck network (Neck) and detection head (DetectHead) quantization, and the post-processing of the deep learning algorithm for target detection is not quantified. For example, Chinese patents CN202110491535.3 "A method for implementing lightweight convolutional neural network target detection" and CN202310316563.0 "Classroom small target detection method based on lightweight weighted bidirectional connection multi-detection head" did not quantify the post-processing of the algorithm.

[0004] The post-processing of the deep learning algorithm for target detection includes confidence threshold comparison, confidence classification maximum value and index, anchor point generation, prediction box coordinate conversion, non-maximum suppression, etc. If 32-bit floating point calculation is used, it will occupy too much computing resources and computing bandwidth, resulting in long calculation time and poor real-time performance. Summary of the invention

[0005] Based on the above problems, the purpose of the present invention is to provide a hardware-friendly integer quantization method for post-processing of target detection deep learning algorithms, which can improve the inference speed and meet the real-time requirements when the target detection deep learning algorithms are deployed on edge hardware.

[0006] The technical solution adopted by the present invention to achieve its invention object is a hardware-friendly integer quantization method for post-processing of a target detection deep learning algorithm, comprising the following steps:

[0007] S1. Define a decoupled structure detection head, including a classification branch and a regression branch;

[0008] S2, using the output feature maps of the classification branch and regression branch of the detection head as the input feature maps for post-processing;

[0009] S3, generating a quantized multiplication factor MULT and a shift factor SHIFT, and completing the quantization of the processed input, wherein the multiplication factor MULT and the shift factor SHIFT are determined by the following formula:

[0010]

[0011] Q4=MULT[∑(Q1-Z1)(Q2-Z2)]>>SHIFT

[0012] Where: >> represents a right shift operation, S1, Z1 and Q1 are the quantization scale factor Scale, zero point ZeroPoint and quantized integer of the input feature map IFM of the last convolution layer of the classification branch and the regression branch respectively; S2, Z2 and Q2 are the quantization scale factor Scale, zero point ZeroPoint and quantized integer of the weight Kernel respectively; S4, Z4 and Q4 are the quantization scale factor Scale, zero point ZeroPoint and quantized integer of the post-processing input feature map respectively; and S4=2 -n , n is a positive integer;

[0013] S4, performing post-processing, the post-processing including confidence threshold comparison, finding the maximum confidence classification value and index, anchor point generation, prediction box coordinate conversion, and non-maximum suppression; then outputting the prediction box, the prediction box including coordinates, type and confidence information;

[0014] S5. Complete the quantization of anchor point generation. The integer anchor point coordinates (CX, CY) after quantization are determined by the following formula:

[0015]

[0016] Where: Row and Col are the row and column of the feature map where the anchor point is located respectively;

[0017] S6. Complete the calculation of the upper left corner coordinates (X1, Y1) and the lower right corner coordinates (X2, Y2) of the prediction box, which are determined by the following formula:

[0018]

[0019] Where: L, T, R, B are the distances from the anchor point to the left, top, right, and bottom of the prediction box respectively;

[0020] S7, the calculation of the prediction frame after dequantization is completed, which is determined by the following formula:

[0021] F X =Q X 'S4

[0022] Where: F X The dequantized floating point number representing the predicted box corresponding to the feature map of size (X, X), Q X ′ Represents the predicted box after the feature map size (X, X) is restored and quantized.

[0023] Furthermore, the classification branch outputs three feature maps of different sizes after convolution, and the sizes are 20×20, 40×40, and 80×80 respectively; the regression branch outputs three feature maps of different sizes after convolution, and the sizes are 20×20, 40×40, and 80×80 respectively.

[0024] In a specific implementation, each feature map of the classification branch contains 80 channels, i.e., 80 categories (80 categories using the COCO dataset, and the number of categories may be different when using other datasets); each feature map of the regression branch contains 4 channels, i.e., 4 distances from the anchor point to the left, top, right, and bottom of the prediction box.

[0025] Furthermore, the n=3, the

[0026] In step S5, the quantized integer anchor point coordinates (CX, CY) are:

[0027]

[0028] In the formula: << represents a left shift operation. As can be seen from the above formula, the calculation of the quantized integer anchor point coordinates (CX, CY) is a hardware-friendly shift and addition operation.

[0029] Furthermore, in step S7, the feature map size restoration is specifically as follows: Q 20 , Q 40 , Q 80 Restore to the original image size of 640×640, and obtain the quantized prediction box Q after restoration to 640×640 ′ 20 , Q ′ 40 , Q ′ 80 , the calculation formula for restoring the feature map size is:

[0030]

[0031] Where: Q 20 , Q40 , Q 80 The predicted boxes are quantized for 20×20, 40×40, and 80×80 feature maps respectively.

[0032] Furthermore, in step S7, the dequantized floating point number of the prediction frame is:

[0033]

[0034] Where: << represents left shift operation, F 20 , F 40 , F 80 They are the inverse quantized floating point numbers of the prediction boxes corresponding to the 20×20, 40×40, and 80×80 feature maps. As can be seen from the above formula, the inverse quantized floating point number is also an integer, and the calculation of the inverse quantized prediction box is a hardware-friendly shift operation.

[0035] The beneficial effects of the present invention are:

[0036] The present invention selects a negative integer power of 2 as a quantization scale factor for the post-processing input, the post-processing input is quantized to an 8-bit integer, the anchor point is quantized to a 10-bit integer (the largest coordinate of the anchor point of the 80×80 feature map is (636, 636), which requires 10 bits to represent), and the prediction box size restoration and inverse quantization are fused to a 10-bit integer (the largest coordinate of the prediction box is (640, 640), which requires 10 bits to represent). Then, the relevant calculations of the prediction box are converted from 32 floating point numbers to integer operations with fewer bits, which can reduce computing resources and computing bandwidth.

[0037] Before quantization, the calculation is done with 32-bit floating point numbers. Assume that after quantization, the calculation is done with 10-bit integers. (32-10) / 32≈69%, and the number of calculation bits is reduced by about 69%, which greatly reduces the computing resources and computing bandwidth, and improves real-time performance.

[0038] At the same time, the prediction box related calculations are hardware-friendly shifts and additions, which can reduce computing resources. In addition, the prediction box size restoration and inverse quantization are integrated to reduce the number of calculation steps and the amount of calculation.

[0039] In summary, the hardware-friendly integer quantization method for post-processing the target detection deep learning algorithm of the present invention can improve the reasoning speed and meet the real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the distance from the anchor point to the prediction box according to an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of generating floating point anchor points for a 20×20 feature map according to an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of generating integer anchor points after quantization of a 20×20 feature map according to an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the classification branch and regression branch of the detection head used as post-processing input in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] Example

[0046] Figures 1 to 4 A specific implementation of the hardware-friendly integer quantization method for post-processing of the target detection deep learning algorithm of the present invention is shown, including the following steps:

[0047] The steps include:

[0048] S1. Define a decoupled structure detection head, including a classification branch and a regression branch; the classification branch outputs three feature maps of different sizes after convolution, the sizes are 20×20, 40×40, and 80×80 respectively; the regression branch outputs three feature maps of different sizes after convolution, the sizes are 20×20, 40×40, and 80×80 respectively. Each feature map of the classification branch contains 80 channels, i.e. 80 categories, and each feature map of the regression branch contains 4 channels, i.e. 4 distances from the anchor point to the left, top, right, and bottom of the prediction box.

[0049] S2, using the output feature maps of the classification branch and regression branch of the detection head as the input feature maps for post-processing;

[0050] S3, generating a quantized multiplication factor MULT and a shift factor SHIFT, and completing the quantization of the processed input, wherein the multiplication factor MULT and the shift factor SHIFT are determined by the following formula (6) and formula (7):

[0051]

[0052] Q4=MULT[∑(Q1-Z1)(Q2-Z2)]>>SHIFT (7)

[0053] Where: >> represents the right shift operation, S1, Z1 and Q1 are the quantization scale factor Scale, zero point ZeroPoint and quantized integer of the input feature map IFM of the last convolution layer of the classification branch and the regression branch respectively; S2, Z2 are the quantization scale factor Scale, zero point ZeroPoint and quantized integer of the weight Kernel respectively; S4, Z4 and Q4 are the quantization scale factor Scale, zero point ZeroPoint and quantized integer of the post-processing input feature map respectively; and S4=2 -n , n is a positive integer; in this embodiment, n=3,

[0054] The derivation process of formula (6) and formula (7) is as follows:

[0055] The relationship between the floating point real number F before convolution quantization and the integer Q after convolution quantization is shown in formula (1), where S and Z represent the scaling factor Scale and the zero point ZeroPoint respectively.

[0056] F=S(QZ) (1)

[0057] The output feature map OFM (Output Feature Map) F3 is calculated by formula (2). In the formula, F1 and F2 represent the input feature map IFM (Input Feature Map) and the weight Kernel respectively.

[0058] F3=∑F1F2 (2)

[0059] Combining formula (1) and formula (2), OFM calculation can be expressed by formula (3).

[0060] S3(Q3-Z3)=∑S1(Q1-Z1)S2(Q2-Z2) (3)

[0061] The input of post-processing is the output of the detection head convolution, as shown in formula (4):

[0062] S4(Q4-Z4)=S3(Q3-Z3)=∑S1(Q1-Z1)S2(Q2-Z2) (4)

[0063] Assign values ​​to Z4 and S4 as shown in formula (5).

[0064]

[0065] Finally, the quantized post-processed input Q3 is calculated by formula (6).

[0066] Formula (6) can also be expressed as formula (7).

[0067] S4, performing post-processing, the post-processing including confidence threshold comparison, finding the maximum confidence classification value and index, anchor point generation, prediction box coordinate conversion, and non-maximum suppression; then outputting the prediction box, the prediction box including coordinates, type and confidence information;

[0068] S5. Complete the quantization of anchor point generation. The quantized integer anchor point coordinates (CX, CY) are determined by the following formula (10):

[0069]

[0070] In the formula: << represents a left shift operation, Row and Col are the row and column of the feature map where the anchor point is located respectively; from the above formula, it can be seen that the calculation of the quantized integer anchor point coordinates (CX, CY) is a hardware-friendly shift and addition operation.

[0071] The derivation process of formula (10) is as follows:

[0072] Each feature map (20×20, 40×40, 80×80) of the regression branch contains 4 channels, i.e., 4 distances from the anchor point to the predicted box. Figure 1 Taking a 20×20 feature map as an example, it is a 20×20 grid, where the distances from the anchor point (black triangle) of the 3rd row and 4th column grid point (the box with the shaded pattern) to the prediction box are left (L), top (T), right (R), and bottom (B).

[0073] The location of the prediction box, such as Figure 1 As shown in the figure, it is determined by the upper left corner coordinates (X1, Y1) and the lower right corner coordinates (X2, Y2). (X1, Y1) and (X2, Y2) are determined by the anchor point coordinates (CX, CY) and the left (L), top (T), right (R), and bottom (B), as shown in formula (8).

[0074]

[0075] The floating point anchor points of the 20×20 feature map are generated as follows Figure 2 shown.

[0076] The quantization of the floating point anchor points of the 20×20 feature map is shown in formula (9).

[0077]

[0078] The quantized integer anchor point coordinates (CX, CY) are calculated as shown in formula (10).

[0079] The quantized integer anchor coordinates (CX, CY) of the 20×20 feature map are generated as follows Figure 3 shown.

[0080] S6. Complete the calculation of the upper left corner coordinates (X1, Y1) and the lower right corner coordinates (X2, Y2) of the prediction box, which are determined by the following formula:

[0081]

[0082] Where: L, T, R, B are the distances from the anchor point to the left, top, right, and bottom of the prediction box respectively;

[0083] S7, the final output of the prediction frame needs to be dequantized, that is, the prediction frame quantized after restoring 640×640 needs to be dequantized. This step calculates the dequantized prediction frame, that is, calculates the dequantized floating point number of the prediction frame, which is determined by the following formula:

[0084]

[0085] Where: << represents left shift operation, F 20 , F 40 , F 80 The dequantized floating point numbers of the prediction boxes corresponding to the 20×20, 40×40, and 80×80 feature maps, Q 20 , Q 40 , Q 80 They are the prediction boxes after quantization of 20×20, 40×40, and 80×80 feature maps respectively; as can be seen from the above formula, the inverse quantized floating-point number is also an integer, and the calculation of the inverse quantized prediction box is a hardware-friendly shift operation.

[0086] Among them, Q 20 , Q 40 , Q 80 Restore to the original image size of 640×640, and obtain the quantized prediction box Q after restoration to 640×640 ′ 20 , Q ′ 40 , Q ′ 80 , and its calculation relationship is as follows:

[0087]

[0088] The above embodiments of the present invention are merely examples for illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes and modifications can be made based on the above description. It is impossible to list all the embodiments here. Any obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A hardware-friendly integer quantization method for post-processing of deep learning algorithms for target detection, characterized in that The steps include: S1. Define a decoupled structure detection head, including a classification branch and a regression branch; S2, using the output feature maps of the classification branch and regression branch of the detection head as the input feature maps for post-processing; S3, generating a quantized multiplication factor MULT and a shift factor SHIFT, and completing the quantization of the processed input, wherein the multiplication factor MULT and the shift factor SHIFT are determined by the following formula: Q4=MULT[∑(Q1-Z1)(Q2-Z2)]>>SHIFT Where: >> represents a right shift operation, S1, Z1 and Q1 are the quantization scale factor Scale, zero point ZeroPoint and quantized integer of the input feature map IFM of the last convolution layer of the classification branch and the regression branch respectively; S2, Z2 and Q2 are the quantization scale factor Scale, zero point ZeroPoint and quantized integer of the weight Kernel respectively; S4, Z4 and Q4 are the quantization scale factor Scale, zero point ZeroPoint and quantized integer of the post-processing input feature map respectively; and S4=2 -n , n is a positive integer; S4, performing post-processing, the post-processing including confidence threshold comparison, finding the maximum confidence classification value and index, anchor point generation, prediction box coordinate conversion, and non-maximum suppression; then outputting the prediction box, the prediction box including coordinates, type and confidence information; S5. Complete the quantization of anchor point generation. The integer anchor point coordinates (CX, CY) after quantization are determined by the following formula: Where: Row and Col are the row and column of the feature map where the anchor point is located; S6. Complete the calculation of the upper left corner coordinates (X1, Y1) and the lower right corner coordinates (X2, Y2) of the prediction box, which are determined by the following formula: Where: L, T, R, B are the distances from the anchor point to the left, top, right, and bottom of the prediction box respectively; S7, the calculation of the prediction frame after dequantization is completed, which is determined by the following formula: F X =Q X ′S4 Where: F X The dequantized floating point number representing the predicted box corresponding to the feature map of size (X, X), Q X ′ represents the predicted box after the feature map size (X, X) is restored and quantized.

2. The hardware-friendly integer quantization method for post-processing of a deep learning algorithm for target detection according to claim 1, characterized in that: The classification branch outputs three feature maps of different sizes after convolution, and the sizes are 20×20, 40×40, and 80×80 respectively; the regression branch outputs three feature maps of different sizes after convolution, and the sizes are 20×20, 40×40, and 80×80 respectively.

3. The hardware-friendly integer quantization method for post-processing of a deep learning algorithm for target detection according to claim 2, characterized in that: The n=3, the In step S5, the quantized integer anchor point coordinates (CX, CY) are: In the formula: << represents the left shift operation.

4. The hardware-friendly integer quantization method for post-processing of a deep learning algorithm for target detection according to claim 3, characterized in that: In step S7, the feature map size restoration is specifically as follows: Q 20 , Q 40 , Q 80 Restore to the original image size of 640×640, and obtain the quantized prediction box Q′ after restoration to 640×640 20 , Q′ 40 , Q′ 80 , the calculation formula for restoring the feature map size is: Where: Q 20 , Q 40 , Q 80 The predicted boxes are quantized for 20×20, 40×40, and 80×80 feature maps respectively.

5. The hardware-friendly integer quantization method for post-processing of a deep learning algorithm for target detection according to claim 4, characterized in that: In step S7, the dequantized floating point number of the prediction frame is: Where: << represents left shift operation, F 20 , F 40 , F 80 They are the inverse quantized floating point numbers of the prediction boxes corresponding to the 20×20, 40×40, and 80×80 feature maps.

Images