Fast non-maximum suppression method for object detection post-processing

Abstract

A fast non-maximum suppression method for object detection post-processing includes obtaining raw data from a deep learning model inference output, the raw data comprising a plurality of bounding boxes, preferentially filtering out at least one of the bounding boxes from further consideration when applying the method, filtering the at least one bounding box based on a predetermined condition, processing a plurality of unfiltered bounding boxes of the bounding boxes using a plurality of sigmoid functions or a plurality of exponential functions to generate a plurality of processed bounding boxes, calculating a plurality of final scores for the processed bounding boxes, and selecting a processed bounding box using the final scores.

Description

Technical Field

[0001] The present invention relates to a fast non-maximum suppression method for object detection post-processing, and more particularly to a fast non-maximum suppression method suitable for object detection that reduces computational complexity and power consumption requirements. Background Art

[0002] Deep learning methods for object detection typically consist of three main steps: preprocessing, deep learning model inference, and post-processing.

[0003] Preprocessing processes the original image to produce a suitable input size for the model. Typically, preprocessing includes image cropping, resizing, padding, and pixel value normalization.

[0004] Deep learning model inference typically involves inputting preprocessed image data and outputting model results. For object detection deep learning models, the output typically includes a bounding box confidence score, a classification score, and the bounding box location (corner coordinates or center coordinates, and bounding box size).

[0005] Post-processing typically involves receiving model results for multiple bounding boxes, each with a bounding box location, a confidence score, and a classification score, and selecting a bounding box based on the score and location.

[0006] Non-Maximum Suppression (NMS) is a common post-processing method for object detection. Single Shot [MultiBox] Detector (SSD), You Only LookOnce (YOLO), and other anchor-based object detection deep learning frameworks all use NMS as a post-processing model.

[0007] NMS is slow and consumes a lot of energy because it requires a large number of exponential and / or sigmoid function calculations, which are cumbersome for edge devices and embedded chips. Summary of the Invention

[0008] An embodiment of the present invention provides a fast non-maximum suppression method for object detection post-processing, comprising obtaining raw data from an inference output of a deep learning model, the raw data comprising a plurality of bounding boxes, prioritizing filtering out at least one of the bounding boxes from further consideration when applying the method, filtering the at least one bounding box based on a predetermined condition, processing a plurality of unfiltered bounding boxes in the bounding boxes using a plurality of sigmoid functions or a plurality of exponential functions to generate a plurality of processed bounding boxes, calculating a plurality of final scores for the processed bounding boxes, and selecting a processed bounding box using the final scores.

[0009] By introducing filters for object detection in the NMS method, which avoids further consideration of unnecessary bounding boxes, the number of subsequent operations can be significantly reduced, thereby saving energy and improving speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 Showing step 5 of the NMS method of the present invention;

[0011] Figure 2 Shows the sigmoid and / or exponential calculations required by YOLO V3 during NMS;

[0012] Figure 3 Shows the sigmoid and / or exponential calculations required by YOLO V5 during NMS;

[0013] Figure 4A The experimental results of the present invention are shown when the input shape of 416 and the score threshold of 0.01 are applied to TinyYOLO V3;

[0014] Figure 4B The experimental results of the present invention using an input shape of 608 and a score threshold of 0.01 for TinyYOLO V3 are shown;

[0015] Figure 5A The experimental results of the present invention are shown when the input shape of 416 and the score threshold of 0.01 are applied to YOLO V3;

[0016] Figure 5B The experimental results of the present invention using an input shape of 608 and a score threshold of 0.01 for YOLO V3 are shown;

[0017] Figure 6A The experimental results of the present invention using an input shape of 416, a FoS value, and a score threshold of 0.01 are shown;

[0018] Figure 6B The experimental results of Example 4 using an input shape of 416, 2 FoS values, and a score threshold of 0.01 are shown.

[0019] Figure 7 The experimental results of the present invention for four embodiments using an input shape of 608, a maximum of 2 FoS values, and a score threshold of 0.01 are shown for YOLO V3. DETAILED DESCRIPTION

[0020] Although the embodiments use the non-maximum suppression (NMS) post-processing method for object detection in the You Only Look Once (YOLO) family to explain the concepts of the present invention, the concepts of the present invention are not limited to YOLO, and NMS post-processing can also achieve excellent results in Single Shot [MultiBox] Detector (SSD) and other anchor-based object detection deep learning frameworks.

[0021] A summary of the NMS post-processing in the related art embodiment includes the following five steps.

[0022] Step 1) Obtain the raw data confidence_score, class_score, box_pos_info from the deep learning model inference output.

[0023] Step 2) Process the original data using a sigmoid function or an exponential function.

[0024] Step 3) Calculate the bounding box final_score = confidence_score * max(class_score).

[0025] Step 4) Filter out bounding boxes whose final_score (confidence_th) is too small.

[0026] Step 5) For the remaining bounding boxes:

[0027] 5a) Select the bounding box with the highest final_score.

[0028] 5b) Filter out bounding boxes that are located close to the currently selected bounding box (IOU>th).

[0029] 5c) If there are more bounding boxes, return to step 5a, otherwise execute step 5a.

[0030] NMS pseudo code

[0031] Implementation: Base Yolo Post Process.nms_boxes()

[0032] Terminology: iou = intersection of union

[0033] argmax is an operation that finds the independent variable that obtains the maximum value from the objective function.

[0034] Input: β={b1,…,b N},S={s1,…,s N},Nt

[0035] β is the list of initial detection boxes

[0036] S contains the corresponding detection scores

[0037] N t is the NMS critical value

[0038] Begin

[0039] D←{}

[0040] while β≠empty do

[0041] m←argmax S

[0042] M←b m

[0043] D←DM;β←β–M

[0044] for b i inβdo

[0045] if iou(M,b i )>=N t then

[0046] β←β–b i ;S←S–s i

[0047] end

[0048] end

[0049] return D,S

[0050] end

[0051] Figure 1 Shows step 5 of the NMS method. Figure 1 , the output of the embodiment of the inference model contains 5 different bounding boxes and their respective scores, which are shown in the upper left corner of the figure. In each operation, the remaining scores are sorted and the bounding box with the highest score is selected for further processing. For the remaining bounding boxes, the intersection over union (IoU) of the bounding boxes is calculated, and the scores of the bounding boxes whose calculated result level is greater than a critical value (in the embodiment, the critical value is set to 0.7) are set to 0. If multiple bounding boxes remain, the remaining bounding boxes are passed to the second round, the remaining bounding boxes are sorted again, the high-scoring bounding boxes are selected and the IoU calculation is completed. The procedure continues until there are no more bounding boxes with non-zero scores, as shown in Figure 1 shown.

[0052] Since the five-step NMS post-processing method of the aforementioned embodiment requires a large number of deep learning model output results, a large number of S-type and / or exponential calculations are required to obtain the required values ​​for selection. Figure 2 (YOLO V3) and Figure 3 (YOLOV5) shows other embodiments. Figure 2 Shows the YOLO V3 post-processing procedure that outputs 3 layers of feature maps. Example with input shape 416 as the output of layer 1: Figure 2 As shown, Outs[0].shape = (13, 13, 3, 85) means 13 width x 13 rows x 3 anchor points x 85 channels (1 confidence + 80 classification scores + 4 bounding box positions). Figure 3 Shows similar YOLO V5 post-processing and has 3 layers of feature maps, e.g., outs, outs[0].shape=(1,3,80,80,85).

[0053] Other YOLO and SSD models are similar, as are all anchor-based models. Figure 2 and Figure 3 As shown, selecting each value of each required bounding box (box_confidence, box_class_probs, etc.) requires calculating an S-type or exponential function, which is computationally intensive and consumes additional energy. Sigmoid or exponential functions are crucial in edge devices and embedded chips.

[0054] To overcome the issues of excessive computation and energy consumption, the following fast NMS method can be used. The basic idea is to add a filter as step 1a between steps 1 and 2 of the above method. This filter significantly reduces the required computation by preemptively removing unnecessary bounding boxes from other processing. The filter can be used with existing YOLO, SSD, and other anchor-based post-processing methods without changing the current preprocessing or deep learning model inference model.

[0055] The summary of fast NMS post-processing contains the following five steps.

[0056] Step 1) Obtain the raw data confidence_score, class_score, box_pos_info from the deep learning model inference output.

[0057] Step 1a) prioritizes filtering out unnecessary bounding boxes to avoid further processing. In this context, further processing means removing the bounding boxes filtered out in step 1a from further consideration and not considering them when applying post-processing steps 2-5.

[0058] Step 2) Process the data in the remaining bounding boxes using an S-shaped function or an exponential function.

[0059] Step 3) Calculate the bounding box final_score = confidence_score * max(class_score).

[0060] Step 4) Filter out the bounding boxes with too small final_score (confidence_th).

[0061] Step 5) For the remaining bounding boxes:

[0062] 5a) Select the bounding box with the highest final_score.

[0063] 5b) Filter out the bounding boxes whose positions are close to the currently selected bounding box (IOU > th).

[0064] 5c) If there are more bounding boxes, return to step 5a, otherwise execute step 5a.

[0065] The term "unnecessary bounding box" in step 1a is defined as the bounding box that will be filtered out from further consideration after filtering step 1a according to certain conditions, and the filtered bounding box will not be considered when performing post-processing steps 2 - 5. In some embodiments, the conditions may include but are not limited to the bounding box having a predetermined amount of overlap with the bounding box to be further processed, and / or the bounding box that can be filtered out in other ways in subsequent steps after further processing. In some embodiments, the bounding box has a class_score less than a critical value, resulting in the bounding box being filtered out in step 4.

[0066] The effectiveness of the filter in step 1a can be regarded as a trade-off between reducing accuracy and reducing computational complexity. Although the following filtering method is used, the benefit of reducing computational complexity far exceeds the reduction in accuracy (if the accuracy is reduced).

[0067] In the first embodiment, step 1a defines the "unnecessary bounding box" as the bounding box inferred and output by the deep learning model and having a class_score less than a predetermined critical value.

[0068] filter if(class score<threshold) Formula 1

[0069] No further processing will be performed on the eligible bounding boxes. The bounding boxes that do not meet the conditions and are not filtered out will continue to be processed normally. Since many bounding boxes meet the conditions, many S-shaped and / or exponential calculations can be saved without subsequent steps, thus speeding up the entire program and reducing power consumption.

[0070] The second embodiment defines "unnecessary bounding boxes" in step 1a as the bounding boxes output according to the inference of the deep learning model, where the class_score of the bounding box is less than the result of applying the inverse sigmoid function to a predetermined confidence threshold (confidence_th).

[0071] filter if(class_score<inverse_sigmoid(confidence th)) Formula 2

[0072] No further processing will be performed on the eligible bounding boxes. The bounding boxes that do not meet the conditions and are not filtered out will continue to be processed normally. Since many bounding boxes meet the conditions, many sigmoid and / or exponential calculations can be saved in the subsequent steps that are not required, thus accelerating the entire program and reducing power consumption.

[0073] The third embodiment defines "unnecessary bounding boxes" in step 1a as the bounding boxes output according to the inference of the deep learning model, where the class_score of the bounding box is less than the result of applying the inverse sigmoid function to the product of a predetermined confidence threshold (confidence_th) and a scaling factor (Factor of Scaling, FoS).

[0074] filter if(class score<inverse_sigmoid(confidence_th*FoS)) Formula 3

[0075] No further processing will be performed on the eligible bounding boxes. The bounding boxes that do not meet the conditions and are not filtered out will continue to be processed normally. Since many bounding boxes meet the conditions, many sigmoid and / or exponential calculations can be saved in the subsequent steps that are not required, thus accelerating the entire program and reducing power consumption.

[0076] In the third embodiment, adding the FoS factor increases the way of balancing performance accuracy and the required amount of computation. When Fos>1, more operations will be filtered out, but the performance accuracy may be reduced.

[0077] All embodiments require comparing a certain value with a threshold, including the threshold of Embodiment 1 and the confidence threshold (confidence_th) of Embodiments 2 and 3. Experiments have derived the following formula to determine the possible threshold.

[0078]

[0079]

[0080] In Formula 4-7, “B” equals the threshold, called “sigmoid_in_th”, “FoS” equals the scaling factor, and “Score th ” is equal to the score cutoff value.

[0081] The application results of the above three embodiments are displayed according to the following settings:

[0082] Method 1: Set sigmoid_in_th to -0.375

[0083] Method 2: Directly use the critical value (B) in Formula 5

[0084] Method 3: Use FoS = 4 and the critical value B from formula 7

[0085] Figures 4A-5B Results using the above methods in various post-processing environments are shown.

[0086] Figure 4A Shows the experimental results of TinyYOLO V3 using the input shape of 416 and a score threshold of 0.01 for each standard model and the three fast non-maximum suppression methods mentioned above. Figure 4A The results from Method 2 show that the standard model achieves an accuracy performance of 0.667, with FoS set to 1 and Sigmoid_in_th equal to -4.59512 (based on Equation 5). However, Method 2 saves over 85% of the operations required for the standard model. While impressive, Method 2 may not be enough to save DSP operations depending on the edge device or embedded chip being used. Figure 4A Similar results are shown for Method 3 using FoS equal to 4 and Sigmoid_in_th equal to -3.1705, which slightly reduces the performance accuracy to 0.666 and saves more operations. The results of Method 1 save more operations (99.42%), but the performance accuracy drops further to 0.654.

[0087] Figure 4B Shows the experimental results of TinyYOLO V3 using the input shape of 608 and the score threshold of 0.01 and the three fast non-maximum suppression methods mentioned above for each standard model. Figure 4A The results of Method 2 show that the standard model with a larger input size achieves an accuracy of 0.582, with FoS set equal to 1 and Sigmoid_in_th equal to -4.59512 (based on Equation 5). Method 2 saves more than 84% of the operations required for the standard model, but it may not be enough for DSP. Figure 4BSimilar results are shown for Method 3 using FoS = 4 and Sigmoid_in_th = -3.1705. The performance accuracy is slightly reduced to 0.582, while also saving more computations. The results of Method 1 can save more computations (99.5%), but the performance accuracy drops to 0.566.

[0088] Figure 5A Shows the experimental results of applying the above three fast non-maximum suppression methods to YOLO V3 for each standard model using an input shape of 416 and a score threshold of 0.01. Figure 5A The matching results of Method 2 show that the standard model achieves an accuracy performance of 0.829, with FoS set equal to 1 and Sigmoid_in_th equal to -4.59512 (based on Formula 7). However, Method 2 saves more than 89% of the computation required by the standard model, but the accuracy performance is slightly reduced. Figure 5A Similar results are shown for Method 3 using FoS = 4 and Sigmoid_in_th = -3.1705. Although the performance accuracy is slightly reduced to 0.825, it also saves more operations (over 96%). The results of Method 1 save more operations (99.469%), but the performance accuracy drops further to 0.808.

[0089] Figure 5B Shows the experimental results of applying the above three fast non-maximum suppression methods to YOLO V3 for each standard model using an input shape of 608 and a score threshold of 0.01. Figure 5B The matching results of Method 2 show that the standard model with a larger input size achieves an accuracy of 0.835, with FoS set equal to 1 and Sigmoid_in_th equal to -4.59512 (based on Formula 7). However, in this embodiment, Method 2 saves more than 87% of the operations required by the standard model. Figure 5B Similar results are shown for Method 3 using FoS = 4 and Sigmoid_in_th = -3.1705. Although the performance accuracy is slightly reduced to 0.835, it also saves more operations (over 95.731%). The results of Method 1 save even more operations (99.477%), but the performance accuracy drops slightly to 0.815.

[0090] Summary of Methods 1-3

[0091] Regarding method 1, a fixed threshold value is not applicable in all cases. The variable values ​​used in methods 2 and 3 may be more reasonable. The method in the embodiment saves many sigmoid operations, but -0.375 is a relatively high filter value, and the accuracy score will drop significantly when score_th is low (for example, 0.01). A larger score_th (for example, 0.15) may be more effective.

[0092] In theory, Method 2 works the same as the conventional post-processing and has the same output bounding boxes. However, Method 2 can save up to 85%-90% of the sigmoid operations.

[0093] When FoS equals 4, Method 3 saves a significant number of sigmoid operations, approximately 94%-96%. Results show that accuracy performance only slightly decreases, or does not decrease at all, when FoS equals 4. Method 3 is a flexible method that saves a significant number of operations and has a sigmoid_in_th threshold that automatically adjusts to different score_th scenarios.

[0094] Fourth Embodiment

[0095] Fourth embodiment

[0096] In the fourth embodiment, in step 1a, an “unnecessary bounding box” is defined as a bounding box having the following results: the product of the result of applying the sigmoid function to confidence_score and the result of applying the sigmoid function to class_score is less than a confidence threshold (confidence_th) output by inference of the deep learning model.

[0097] filter if((sigmoid(confidence_score)*sigmoid(class_score))<(Score_th=0.01))

[0098] Formula 8

[0099] However, in order to avoid using functions such as the sigmoid function when using the filter, the fourth embodiment generates these functions without using calculations but using a lookup table 1. In Table 1, A is the confidence_score of the bounding box, and B is the class_score of the bounding box.

[0100]

[0101]

[0102] Table 1

[0103] For example, suppose a bounding box has a bounding box confidence score (A) of -4 and a class_score (B) of -1. Find the column with the appropriate range A in Table 1, which shows that if B (-1) is greater than -1.3198, the bounding box should be skipped and not passed to subsequent processing steps. On the other hand, if the class_score (B) for the same bounding box is -2 but not greater than -1.3198, the bounding box is not skipped and is passed to subsequent processing steps.

[0104] Similar to the previous embodiments, Example 4 can also be modified to include one or more Factor of Scaling (FoS) components. In this embodiment, the lookup table includes the FoS, as shown in Table 2. In Table 2, A is the confidence score of the bounding box, and B is the class score of the bounding box. This embodiment uses two different FoS components, FoS1 and FoS2.

[0105]

[0106]

[0107] Table 2

[0108] When implementing Table 2, depending on the value of FoS1 or FoS2, the inv_sigmoid() function can be run once and the resulting values ​​stored in the table for later use. Otherwise, similar to Table 1, Table 2 is used to filter out unnecessary bounding boxes from further consideration.

[0109] Theoretically, when FoS1 = FoS2 = 1, there is no performance loss. In practice, FoS1 and FoS2 can be adjusted to balance performance accuracy and computational effort. Typical settings are FoS2 = 1 and FoS1 = 2 to 4.

[0110] Figure 6A Shown are the experimental results for each standard model, using an input shape of 416, a FoS value, and a score threshold of 0.01, and applying the previous fast non-maximum suppression method to YOLO V3. Figure 6A As can be seen, Example 2, with the newly added FoS = 1, fully matches the performance of the standard model while saving over 89% of operations. In the third example, when FoS = 4, the performance impact is minimal, but over 96% of operations are saved. In the first example, with Sigmoid_in_th = -0.375, performance further degrades, but the percentage of saved operations increases to over 99%.

[0111] Figure 6BThe experimental results of the fast non-maximum suppression method of Example 4 are shown using an input shape of 416, 2 FoS values ​​and a score threshold of 0.01 applied to the same YOLO V3. In the results shown, only the results of step 2 (Table 2) are considered. In the first row, FoS1=2 and FoS2=1, the performance is 0.829, which is exactly the same as the standard model (see Figure 6A ), but the percentage of saved operations increases to 99.990%. In contrast, if FoS1=3 and FoS2=1, the amount of saved operations remains high, but the model suffers a slight performance hit, just like when FoS1=4 and FoS1=1. The rightmost row shows the results when FoS1=4 and FoS2=2. Although the accuracy is more significantly affected, 99.992% of the subsequent operations are saved.

[0112] Figure 7 The experimental results of four embodiments of the fast non-maximum suppression method and the application of 608 input shapes, up to 2 FoS values ​​and a score threshold of 0.01 to YOLO V3 are shown for each standard model. It can be seen that the accuracy of the standard (conventional) NMS is 0.835. Example 2 with FoS=1 and Sigmoid_in_th=-4.59512 maintains the accuracy performance while saving more than 87% of the required computation. Example 3 with FoS=4 and Sigmoid_in_th=-3.1705 maintains the accuracy performance while increasing the saved computation to more than 95%. For Example 1, using Sigmoid_in_th=-0.375, the computation savings increased to 99.477%, but the accuracy performance decreased. Using Example 4, some results can still be determined. However, with FoS=4, pre-filtering can save 97.288% of the computation while matching the accuracy performance of the standard model.

[0113] In summary, by introducing an object detection filter between steps 1 and 2 of the NMS method in step 1a, unnecessary bounding boxes are avoided, significantly reducing the number of subsequent operations, saving energy and increasing speed. This filter can be used with existing YOLO, SSD, and other anchor-based post-processing methods without modifying the current pre-processing or deep learning model inference model. This energy savings makes the above embodiment ideally suited for edge devices, embedded devices, and battery-powered devices.

[0114] The above are only preferred embodiments of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should fall within the scope of the present invention.

Claims

1. A fast non-maximum suppression (NMS) method for object detection post-processing, characterized in that: Include: Obtaining raw data from an inference output of a deep learning model, the raw data comprising a plurality of bounding boxes; Prioritizing filtering out at least one of the bounding boxes from further consideration, the at least one bounding box being filtered based on a predetermined condition; Processing a plurality of unfiltered bounding boxes in the bounding box using a plurality of sigmoid functions or a plurality of exponential functions to generate a plurality of processed bounding boxes; calculating a plurality of final scores for the processed bounding boxes; and selecting a processed bounding box using the final score; Prioritizing filtering out the at least one of the bounding boxes from the further consideration includes prioritizing filtering out the at least one of the bounding boxes from the further consideration when the at least one bounding box has a classification score less than a result obtained based on an inference output of the deep learning model, the result obtained by applying an inverse sigmoid function to a predetermined confidence threshold.

2. The method according to claim 1, wherein Prioritizing filtering out the at least one of the bounding boxes from the further consideration includes prioritizing filtering out the at least one of the bounding boxes from the further consideration when the at least one bounding box has a classification score less than a predetermined first threshold, the classification score being obtained based on the deep learning model inference output.

3. The method according to claim 1, wherein Prioritizing filtering out the at least one of the bounding boxes from further consideration includes prioritizing filtering out the at least one of the bounding boxes from further consideration when the at least one bounding box has a classification score less than a result obtained based on the deep learning model inference output, the result obtained by applying an inverse sigmoid function to a product of a predetermined confidence threshold and a predetermined scaling factor.

4. The method according to claim 1, wherein Prioritizing filtering out the at least one of the bounding boxes from the further consideration includes prioritizing filtering out the at least one of the bounding boxes from the further consideration when the at least one bounding box obtained based on the inference output of the deep learning model has a product less than a predetermined confidence threshold, the product being obtained by multiplying a result of applying a sigmoid function to a confidence score and a result of applying a sigmoid function to a classification score.

5. The method according to claim 4, wherein Prioritizing filtering out the at least one of the bounding boxes in the further consideration includes prioritizing filtering out the at least one of the bounding boxes in the further consideration when the at least one bounding box obtained based on the inference output of the deep learning model has a classification score less than a predetermined second threshold.

6. The method according to claim 4, wherein The method further includes determining whether the product of the result of applying the sigmoid function to the confidence score and the result of applying the sigmoid function to the classification score is less than the predetermined confidence threshold by using a lookup table indexed according to the confidence score and the classification score.

7. The method according to claim 6, wherein Further comprising: when the at least one bounding box obtained according to the inference output of the deep learning model has the classification score less than a predetermined third threshold, preferentially filtering out the at least one bounding box from the further consideration.

8. The method according to claim 7, wherein The predetermined third threshold is calculated based on a product of a fractional threshold and a first scaling factor.

9. The method according to claim 7, wherein A classification probability of the at least one bounding box depends on a related bounding box confidence score, and the predetermined third threshold is calculated based on a product of the classification probability and a second scaling factor.

Images