A method for target detection neural network dequantization layer and post-processing merging

Abstract

The application provides a method for combining a target detection neural network inverse quantization layer and post-processing, two interfaces of dequantize (quantize_data) and generate_box (model_output) called by a chip running a target detection neural network are replaced by a new interface of dequantize_generate_box (quantize_data), the dequantize inverse quantization layer is combined with the post-processing generate_box, and a calculation sequence is adjusted, that is, comparison and screening are performed first, then inverse quantization calculation and coordinate calculation are performed, and finally a final result is obtained. The method solves the problem that when a target detection neural network of a Junzheng T series chip runs, redundant invalid detection boxes are calculated during inverse quantization and post-processing operations, a large amount of computing resources are wasted, calculation resources are saved, the overall running time of the target detection model is reduced, and the efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent video processing technology, and specifically relates to a method for merging the inverse quantization layer and post-processing of a target detection neural network. Background Technology

[0002] In existing technologies, neural networks running on chips such as Beijing Junzheng T-series are quantized networks. The weights, activation values, and results of each layer are all quantized values, which reduces the space required to store parameters and improves inference speed. When the model provides its output, dequantization is required to convert integers into floating-point numbers, reducing precision errors caused by quantization. Therefore, the model has a dequantization layer before outputting the result. When running on the Beijing Junzheng T-series chip, the `model_output = dequantize(quantize_data)` interface is called to calculate the quantized value `quantize_data` into a floating-point number, obtaining the model's output `model_output`.

[0003] After the neural network for object detection running on the Beijing Junzheng T-series chip provides its output, a series of additional processing steps are required to optimize the model's predictions or extract more meaningful information. During operation on the Beijing Junzheng T-series chip, the `result = generate_box(model_output)` interface is called to perform post-processing. This ultimately yields the coordinates of the bounding box containing the target, the target's confidence score, and the target's category. The `generate_box` interface is implemented in four steps, as follows:

[0004] (1) Call the calc_result = calculate_model_output(model_output) interface to parse the output result of the neural network.

[0005] The values ​​stored in the neural network output `model_output` are typically relative to the size of the image. First, the `calculate_model_output` interface needs to be called to convert the relative values ​​of the network output back to the absolute coordinates of the original image. Then, based on the size of the input feature map, the coordinates of the bounding box are calculated to reflect its correct position on the original image.

[0006] The output of the neural network used for object detection, model_output, is a multi-dimensional array, typically with the shape [batch_size, height, width, num_anchors, (5 + num_classes)], where:

[0007] batch_size indicates the batch size.

[0008] height and width represent the height and width of the feature map, respectively.

[0009] num_anchors represents the number of anchor boxes predicted for each pixel.

[0010] The information in (5+num_classes) includes bounding boxes, objectness scores, and class scores:

[0011] Bounding boxes: By parsing the coordinate values ​​in the output array, the position information of each anchor box can be obtained. Typically, these coordinate values ​​need to be decoded to obtain the actual bounding box positions.

[0012] Objectness scores: These represent the confidence level that an anchor box contains the target. During post-processing, a threshold is typically set to filter out anchor boxes with low confidence scores.

[0013] Class scores: These represent the confidence level that a target in the anchor frame belongs to each category. These scores are used in post-processing to determine the category of the target in the anchor frame.

[0014] 5 represents the four coordinate values ​​of each anchor box (the offset of the center point of the bounding box relative to the top left corner of the feature map, x and y, and the width and height of the bounding box, w and h) and an objectness score (the anchor box contains the confidence of the object). num_classes represents the number of classes (stores the confidence scores of the objects in the anchor box belonging to each class).

[0015] (2) Call the nms_result = nms(calc_result) interface to remove detection boxes with high overlap using the non-maximum suppression (NMS) algorithm.

[0016] Non-Maximum Suppression (NMS) is a commonly used post-processing algorithm used to remove highly overlapping bounding boxes, retaining only the boxes with the highest confidence. NMS typically measures the overlap between bounding boxes based on Intersection over Union (IOU). If the IOU between two bounding boxes is greater than a certain threshold (e.g., 0.45), the box with the higher confidence is retained, while the box with the lower confidence is suppressed.

[0017] (3) Call the box_result = filter_box(nms_result) interface to filter out detection boxes with low confidence based on the set objectnessscore threshold (e.g., 0.5). The box_result contains information such as the coordinates of the bounding box where the target is located and the confidence of the target.

[0018] (4) Call the result = class_id(box_result) interface to determine the category of the target in the detection box according to the set class score, and obtain the class_id value of the target. The result contains the information of box_result in step (3) and the class_id value of this step. The result is the final output result of the generate_box interface.

[0019] However, the shortcomings of existing technologies are:

[0020] First, when running a neural network for object detection on the Beijing Junzheng T-series chip, the model_output = dequantize(quantize_data) interface is called in the last layer for dequantization. After the model outputs the results, the result = generate_box(model_output) interface is called for post-processing to obtain the final detection results. Among them, the dequantize interface needs to dequantize all the results predicted by the neural network when performing dequantization. The first step of the generate_box interface, parsing the model output results, also parses all the prediction results. Both of these operations require a lot of computation, and both valid and invalid detection boxes are calculated. The invalid detection boxes are usually more numerous than the valid detection boxes. Therefore, both of these operations waste a lot of computing power and consume computing resources. The second step of generate_box uses NMS to remove detection boxes with high overlap, and the third step filters out detection boxes with low confidence. These are just screening operations and do not require computation. Putting the computation steps first and the screening steps later will lead to invalid computation, increase the amount of computation, waste computing power, and thus increase the overall running time of the object detection model, resulting in reduced efficiency.

[0021] In addition, the technical terms in this field include:

[0022] 1. Neural network quantization refers to the process of approximating continuous weights or activation values ​​in a neural network model as a finite number of discrete values. Quantization converts floating-point real numbers into integers (FP32 -> INT8). Using low-precision integers to represent the original high-precision floating-point data reduces the space required to store parameters and improves inference speed. However, because quantization-related operations introduce errors into the values ​​in the network, it leads to a decrease in model accuracy. Neural network dequantization converts integers back to floating-point real numbers (INT8 -> FP32), restoring the precision errors introduced by quantization and reducing the precision errors in the model's output. Therefore, the quantized model has a dequantization layer before the output to perform dequantization, resulting in a higher-precision floating-point real number output.

[0023] 2. Object detection is an important task in the field of computer vision, aiming to identify and locate objects in images or videos. Object detection neural networks can learn complex patterns and features, thereby achieving higher accuracy and performance. In object detection tasks, they are mainly used for two aspects: first, feature extraction from images to identify objects; and second, object localization to locate them within images.

[0024] 3. Post-processing of neural networks refers to a series of additional processing steps performed on the output of the neural network to optimize the model's predictions or extract more meaningful information. Post-processing is an important step in the object detection process. It is mainly responsible for processing the output data of the neural network, filtering out low-confidence detections, applying non-maximum suppression (NMS) to eliminate overlapping bounding boxes, and finally obtaining the final list of detection results. Summary of the Invention

[0025] To address the aforementioned issues, the purpose of this application is to provide a method for merging the dequantization layer and post-processing of a target detection neural network. A new interface, result = dequantize_generate_box(quantize_data), is designed to replace the existing interfaces model_output = dequantize(quantize_data) and result = generate_box(model_output) used in technologies such as the Beijing Junzheng T-series chip for running target detection neural networks. This method merges the dequantization layer with the post-processing generate_box, and adjusts the calculation order by first performing comparison and filtering, then performing dequantization and coordinate calculations to obtain the final result.

[0026] Specifically, the present invention provides a method for merging the inverse quantization layer and post-processing of a target detection neural network, the method comprising the following steps:

[0027] S1 provides a merging interface result = dequantize_generate_box(quantize_data), which receives the quantized data quantize_data output by the neural network as input;

[0028] S2, the filter_box(quantize_data) interface performs preliminary screening of the quantized data based on a preset objectness score threshold, retaining detection boxes that meet the confidence condition, which is that the confidence score is greater than the preset threshold, and outputs box_result that maintains the quantization state; wherein, box_result contains the coordinates of the quantized bounding box and the confidence score;

[0029] S3 performs non-maximum suppression processing on the filtered detection boxes through the nms(box_result) interface and outputs a further optimized nms_result;

[0030] S4, determine the target class of each detection box through the class_id(nms_result) interface, and output box_class_result containing bounding box information and class ID;

[0031] S5 uses the dequantize(box_class_result) interface to perform dequantization calculation on the bounding box coordinates and confidence scores, while keeping the class ID as an integer and outputting dequantize_result;

[0032] S6 uses the calculate_box(dequantize_result) interface to convert the dequantized relative coordinates into the absolute coordinates of the original image and outputs the final detection result result.

[0033] The filter_box interface in step S2 only performs threshold filtering and does not change the quantization state and relative coordinate values ​​of the input data.

[0034] In step S3, the NMS interface performs non-maximum suppression processing on the quantized data output by the filter_box interface. The output is still quantized data, and the resulting detection boxes are all valid detection boxes. The value obtained in this step is the box_result after further filtering.

[0035] In step S4, the box_class_result = class_id(nms_result) interface is called to determine the category of the target in the detection box based on the set class score, and the class_id value of the target is obtained; class_result contains the value of box_result in step S3 and the class_id value of this step.

[0036] In step S5, the dequantize interface only performs inverse quantization conversion from INT8 to FP32 on the bounding box coordinates and confidence data, while the category ID remains an INT8 integer output.

[0037] In step S6, the calculate_box interface converts the relative coordinates (x, y, w, h) into absolute coordinates in the original image space based on the size parameters of the input feature map.

[0038] Both the filter_box interface and the nms interface perform filtering operations in the quantized data state, avoiding inverse quantization calculations for invalid detection boxes.

[0039] The quantize_data is a multidimensional array output by the neural network, with a dimensional structure of [batch_size, height, width, num_anchors, (5+num_classes)], containing the quantized bounding box coordinates, target confidence, and class scores.

[0040] The objectness score threshold and the IOU threshold of NMS can be dynamically configured to balance detection accuracy and computational efficiency.

[0041] The method is implemented in an embedded AI chip and is used to optimize the post-processing computation efficiency of the target detection neural network.

[0042] Therefore, the advantage of this solution is:

[0043] This invention addresses the problem of redundant and invalid detection boxes being calculated during inverse quantization and post-processing operations in neural networks running target detection, such as those in Beijing Junzheng T-series chips, which wastes a lot of computing power. It saves computing resources, reduces the overall running time of the target detection model, and thus improves efficiency. Attached Figure Description

[0044] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0045] Figure 1This is a schematic diagram of the code implementation of calling the box_result = filter_box(quantize_data) interface in a specific embodiment of this application.

[0046] Figure 2 This is a schematic diagram of the code implementation of calling the nms_result = nms(box_result) interface in a specific embodiment of this application.

[0047] Figure 3 This is a schematic diagram of the code implementation of calling the box_class_result = class_id(nms_result) interface in a specific embodiment of this application.

[0048] Figure 4 This is a schematic diagram of the code implementation of calling the dequantize_result = dequantize(box_class_result) interface in a specific embodiment of this application.

[0049] Figure 5 This is a schematic diagram of the code implementation of calling the `result = calculate_box(dequantize_result)` interface in a specific embodiment of this application.

[0050] Figure 6 This is a schematic diagram of the overall process of the method described in this application. Detailed Implementation

[0051] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0052] This application relates to a method for merging the inverse quantization layer and post-processing of a target detection neural network, namely:

[0053] The interface result = dequantize_generate_box(quantize_data) is used when the dequantization layer and post-processing of the neural network are combined. The input data received by this interface is quantized data, that is, the data originally received by the dequantize interface, quantize_data, is processed by the dequantize_generate_box interface to obtain the final output result result, which is the original output data of the generate_box interface.

[0054] Example as follows:

[0055] The quantized output data of the object detection neural network is stored in a tensor with pointer `data_ptr`. The shape is [1, 13, 13, 3, 22], indicating a batch size of 1. The feature map has a width and height of 13. Each pixel predicts 3 anchor boxes, each with 4 coordinate values, 1 target confidence score, and 17 class values ​​(4 + 1 + 17 = 22). The target confidence threshold `box_threshold` is set to 0.6, the NMS threshold `nms_threshold` to 0.8, and the parameters required for dequantization are `in_scale` (0.5) and `in_offset` (1). `data_ptr`, shape, `box_threshold`, `nms_threshold`, `in_scale`, and `in_offset` are passed to the `dequantize_generate_box` function to obtain the output target anchor box information, including the two true coordinate values ​​of the top-left and bottom-right anchor boxes, the target confidence score, and the class ID. The output dimension is [x1, y1, x2, y2, score, class_id].

[0056] Call the function Result=dequantize_generate_box(data_ptr, [batch_size, height, width, num_anchors, (5+num_classes)], box_threshold, nms_threshold, in_scale, in_offset).

[0057] The specific steps are as follows:

[0058] (1) Call the `box_result = filter_box(quantize_data)` interface to filter out detection boxes with low confidence based on the set threshold `box_threshold` for `objectnessscores`. `box_result` contains information such as the coordinates of the bounding box where the target is located and the target's confidence score. This step only uses the threshold for filtering and judgment, and does not perform calculations, so the obtained `box_result` value is still a quantized value and is a relative value relative to the feature map.

[0059] The function `box_result = filter_box(data_ptr, [batch_size, height, width, num_anchors, (5 + num_classes)], box_threshold)` is called. An example code implementation is shown below. Figure 1 As shown.

[0060] (2) Call the nms_result = nms(box_result) interface to remove the detection boxes with high overlap using the non-maximum suppression (NMS) algorithm. This step is a further screening based on (1), and the obtained detection boxes are all valid detection boxes. The value obtained in this step is the box_result after further screening.

[0061] Call the function nms_result = nms(box_result, box_num, nms_threshold). Example code implementation: Figure 2 As shown.

[0062] (3) Call the box_class_result = class_id(nms_result) interface to determine the category of the target in the detection box based on the set classscore, and obtain the class_id value of the target. class_result contains the value of box_result in (2) and the class_id value of this step.

[0063] The function `box_class_result = class_id(nms_result, nms_num, num_classes)` is called. An example code implementation is shown below. Figure 3 As shown.

[0064] (4) Call the dequantize_result = dequantize(box_class_result) interface to perform dequantization calculation on the value obtained in (3), convert the quantized box_result data into floating-point data, and the class_id value should be output as an integer. This step does not calculate class_id.

[0065] The function `dequantize_result = dequantize(box_class_result, nms_num, in_scale, in_offset)` is called. An example code implementation is shown below. Figure 4 As shown.

[0066] (5) Call the result = calculate_box(dequantize_result) interface to dequantize the relative value after (4), calculate the coordinates of the bounding box according to the size of the input feature map, and convert the relative value output by the network back to the absolute coordinates of the original image. result is the final output result, which includes the coordinates of the bounding box where the target is located, the confidence of the target and the category of the target.

[0067] Here's a code example of calling the function `result = calculate_box(dequantize_result, nms_num, height, width)`, such as... Figure 5 As shown.

[0068] In summary, such as Figure 6 As shown, the method includes the following steps:

[0069] S1 provides a merging interface result = dequantize_generate_box(quantize_data), which receives the quantized data quantize_data output by the neural network as input;

[0070] S2, the filter_box(quantize_data) interface performs preliminary screening of the quantized data based on a preset objectness score threshold, retains the detection boxes that meet the confidence conditions, and outputs box_result that maintains the quantization state; the filter_box interface only performs threshold screening and does not change the quantization state and relative coordinate values ​​of the input data;

[0071] S3, Non-maximum suppression processing is performed on the filtered detection boxes through the nms(box_result) interface, and the further optimized nms_result is output; The nms interface performs non-maximum suppression processing on the quantized data output by the filter_box interface, and the output is still quantized data. The obtained detection boxes are all valid detection boxes. The value obtained in this step is the box_result after further filtering.

[0072] S4, determine the target class of each detection box through the class_id(nms_result) interface, and output box_class_result containing bounding box information and class ID;

[0073] The `box_class_result = class_id(nms_result)` interface is called to determine the class of the target in the detection box based on the set class score, and obtain the class_id value of the target; `class_result` contains the value of `box_result` in step S3 and the class_id value of this step;

[0074] S5 uses the dequantize(box_class_result) interface to perform dequantization calculation on the bounding box coordinates and confidence scores, while keeping the class ID as an integer and outputting dequantize_result;

[0075] The dequantize interface only performs INT8 to FP32 dequantization on the bounding box coordinates and confidence data, while the category ID is kept as an INT8 integer output;

[0076] S6 uses the calculate_box(dequantize_result) interface to convert the dequantized relative coordinates into the absolute coordinates of the original image and outputs the final detection result result.

[0077] The calculate_box interface converts relative coordinates (x, y, w, h) into absolute coordinates in the original image space based on the size parameters of the input feature map.

[0078] Both the filter_box interface and the nms interface perform filtering operations in the quantized data state, avoiding inverse quantization calculations for invalid detection boxes.

[0079] The quantize_data is a multidimensional array output by the neural network, with a dimensional structure of [batch_size, height, width, num_anchors, (5+num_classes)], containing the quantized bounding box coordinates, target confidence, and class scores.

[0080] The objectness score threshold and the IOU threshold of NMS can be dynamically configured to balance detection accuracy and computational efficiency.

[0081] The method is implemented in an embedded AI chip and is used to optimize the post-processing computation efficiency of the target detection neural network.

[0082] Therefore, the present invention can achieve:

[0083] 1. A method for merging the dequantization layer and post-processing of a target detection neural network is used, namely the interface result = dequantize_generate_box(quantize_data), which replaces the two interfaces model_output = dequantize(quantize_data) and result = generate_box(model_output) called by the target detection neural network, such as Beijing Junzheng T-series chips. The final detection result is obtained directly from the quantized data output by the model, without having to convert the model result into a floating-point number output, thus saving memory space.

[0084] 2. The `result = dequantize_generate_box(quantize_data)` interface in this invention internally adjusts the calculation order, first performing comparison and filtering, then performing dequantization and coordinate calculations to obtain the final result. This solves the problem of redundant and invalid detection boxes being calculated and wasting a lot of computing power when performing dequantization and post-processing operations in neural networks for target detection, such as those running on Beijing Junzheng T-series chips. It saves computing resources, reduces the overall running time of the target detection model, and thus improves efficiency.

[0085] This technical solution can be applied to all chips in the Ingenic T series and similar chips.

[0086] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for merging the inverse quantization layer and post-processing of a target detection neural network, characterized in that, The method includes the following steps: S1 provides a merging interface result = dequantize_generate_box(quantize_data), which receives the quantized data quantize_data output by the neural network as input; S2, the filter_box(quantize_data) interface performs preliminary screening of the quantized data based on a preset objectness score threshold, retaining detection boxes that meet the confidence condition, which is that the confidence score is greater than the preset threshold, and outputs box_result that maintains the quantization state; wherein, box_result contains the coordinates of the quantized bounding box and the confidence score; S3 performs non-maximum suppression processing on the filtered detection boxes through the nms(box_result) interface and outputs a further optimized nms_result; S4, determine the target class of each detection box through the class_id(nms_result) interface, and output box_class_result containing bounding box information and class ID; S5 uses the dequantize(box_class_result) interface to perform dequantization calculation on the bounding box coordinates and confidence scores, while keeping the class ID as an integer, and outputs dequantize_result; S6 uses the calculate_box(dequantize_result) interface to convert the dequantized relative coordinates into the absolute coordinates of the original image and outputs the final detection result result.

2. The method for merging the inverse quantization layer and post-processing of a target detection neural network according to claim 1, characterized in that, The filter_box interface in step S2 only performs threshold filtering and does not change the quantization state and relative coordinate values ​​of the input data.

3. The method for merging the inverse quantization layer and post-processing of a target detection neural network according to claim 1, characterized in that, In step S3, the NMS interface performs non-maximum suppression processing on the quantized data output by the filter_box interface. The output is still quantized data, and the resulting detection boxes are all valid detection boxes. The value obtained in this step is the box_result after further filtering.

4. The method for merging the inverse quantization layer and post-processing of a target detection neural network according to claim 1, characterized in that, In step S4, the box_class_result = class_id(nms_result) interface is called to determine the category of the target in the detection box based on the set class score, and the class_id value of the target is obtained; class_result contains the value of box_result in step S3 and the class_id value of this step.

5. The method for merging the inverse quantization layer and post-processing of a target detection neural network according to claim 1, characterized in that, In step S5, the dequantize interface only performs inverse quantization conversion from INT8 to FP32 on the bounding box coordinates and confidence data, while the category ID remains an INT8 integer output.

6. The method for merging the inverse quantization layer and post-processing of a target detection neural network according to claim 1, characterized in that, In step S6, the calculate_box interface converts the relative coordinates (x, y, w, h) into absolute coordinates in the original image space based on the size parameters of the input feature map.

7. The method for merging the inverse quantization layer and post-processing of a target detection neural network according to claim 1, characterized in that, Both the filter_box interface and the nms interface perform filtering operations in the quantized data state, avoiding inverse quantization calculations for invalid detection boxes.

8. The method for merging the inverse quantization layer and post-processing of a target detection neural network according to claim 1, characterized in that, The quantize_data is a multidimensional array output by the neural network, with a dimensional structure of [batch_size, height, width, num_anchors, (5+num_classes)], containing the quantized bounding box coordinates, target confidence, and class scores.

9. The method for merging the inverse quantization layer and post-processing of a target detection neural network according to claim 1, characterized in that, The objectness score threshold and the IOU threshold of NMS can be dynamically configured to balance detection accuracy and computational efficiency.

10. The method for merging the inverse quantization layer and post-processing of a target detection neural network according to claim 1, characterized in that, The method is implemented in an embedded AI chip and is used to optimize the post-processing computation efficiency of the target detection neural network.

Images