Non-maximum suppression optimization method and device

Abstract

The invention provides a non-maximum suppression optimization method and device, and belongs to the technical field of target detection, and the non-maximum suppression optimization method comprises the steps: uniformly distributing a plurality of groups of original prediction data into at least two threads; calculating coordinate data of a plurality of groups of candidate frames in each thread by adopting an NEON instruction set; performing non-maximum suppression processing on the left upper corner coordinate data, the right lower corner coordinate data and the area data in each thread by using an NEON instruction set; classifying and combining the data subjected to the non-maximum suppression processing in each thread, and then performing the non-maximum suppression processing by using an NEON instruction set; and carrying out inverse quantization processing on the final coordinate value of the candidate box to obtain a final screening result. According to the non-maximum suppression optimization method, on the basis of multi-thread processing, descending sorting is carried out by adopting a rapid sorting method, and fixed-point value calculation is adopted during NEON instruction set calculation, so that low delay and high efficiency of a non-maximum suppression processing process can be realized.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a non-maximum suppression optimization method and device. Background Art

[0002] Non-Maximum Suppression (NMS) is a commonly used technique in deep learning target detection and image processing. It is used to remove redundant bounding boxes generated in target detection. That is, when the model generates multiple overlapping bounding boxes for the same target detection multiple times, NMS reduces redundancy by retaining the highest confidence bounding box of each target while removing other highly overlapping bounding boxes.

[0003] In conventional target detection processes, NMS is usually executed on the CPU. However, with the development of deep learning, the complexity of target detection tasks has increased, and the requirements for real-time performance have become increasingly stringent. In real-time applications, low latency and high efficiency have become two important considerations when performing non-maximum suppression. However, in existing technologies, the execution of calculations based on a single instruction and single data method on the CPU is too time-consuming and cannot meet the application scenarios of real-time feedback; or specialized hardware is required to support highly parallel processing capabilities, which is costly; and the use of SIMD (Single Instruction Multiple Data) instructions based on assembly language has poor portability. Summary of the Invention

[0004] In view of this, the present invention provides a non-maximum suppression optimization method and device, which is used to solve the problems in the current existing technology that the calculation execution based on the CPU single instruction and single data method is too time-consuming and cannot meet the application scenarios of real-time feedback; or special hardware is required to support highly parallel processing capabilities, which is costly; and the use of SIMD instructions based on assembly language has poor portability.

[0005] In order to solve the above technical problems, in a first aspect, the present invention provides a non-maximum suppression optimization method, which comprises:

[0006] Evenly distributing multiple groups of raw prediction data obtained after performing target detection on the image to be detected to at least two threads, screening candidate boxes for the multiple groups of raw prediction data according to a confidence threshold, and saving the screened prediction data results in the at least two threads, wherein each group of the raw prediction data includes coordinate data of the candidate box, category confidence, and category probability, and the confidence threshold is a fixed-point value;

[0007] Using a NEON instruction set, calculating coordinate data of multiple sets of candidate boxes in the prediction data results saved in each of the threads to obtain a first processing result, where the first processing result includes upper left corner coordinate data, lower right corner coordinate data, area data, and confidence data of the candidate boxes;

[0008] In each of the threads, a non-maximum suppression process is performed on the upper left corner coordinate data, the lower right corner coordinate data, and the area data using a NEON instruction set;

[0009] Classify and merge the data after non-maximum suppression processing in each thread, and use the NEON instruction set to perform non-maximum suppression processing on the upper left corner coordinate data, lower right corner coordinate data and area data in the merged data again, and save the index corresponding to the candidate box finally retained;

[0010] The corresponding coordinate value, confidence level and category number are extracted according to the index, and the coordinate value is dequantized to obtain the final screening result.

[0011] Optionally, the step of performing non-maximum suppression processing using the NEON instruction set includes:

[0012] The quick sort algorithm is used to sort the data to be processed with non-maximum value suppression in descending order to obtain the descending sort result;

[0013] Initialize the intersection-over-union ratio threshold to a fixed-point value;

[0014] According to the intersection-over-union ratio threshold, a NEON instruction set is used to perform a cyclic processing calculation of non-maximum value suppression on the descending sorting result.

[0015] Optionally, the step of performing a loop processing calculation of non-maximum suppression on the descending sorting result using a NEON instruction set according to the intersection-over-union ratio threshold includes:

[0016] The candidate box with the highest confidence in the descending sorting results is used as the first target box, and the remaining candidate boxes are used as the second target boxes;

[0017] Calculate the intersection area of each second target frame and the first target frame using the NEON instruction set;

[0018] Calculating an area threshold according to the intersection area, the area data of the first target frame, and the intersection-over-union ratio threshold, wherein the threshold area is quantized as a fixed-point value;

[0019] If the area threshold is smaller than the area of the second target frame, the second target frame is retained; otherwise, the second target frame is deleted.

[0020] Optionally, the step of calculating the area fixed-point value of the second target frame according to the area of the intersection region, the area data of the first target frame, and the intersection-over-union ratio threshold is calculated using the following formula:

[0021] AreaQ=(1+T)×AreaI–T×Area0;

[0022] Wherein, AreaQ is the area threshold, T is the intersection-over-union ratio threshold, AreaI is the area of the intersection region, and Area0 is the area of the first target frame.

[0023] Optionally, when the NEON instruction set is used for calculation processing, the upper left corner coordinate data, the lower right corner coordinate data, the area data and the confidence data are calculated using fixed-point values.

[0024] In a second aspect, the present invention further provides a non-maximum suppression optimization device, comprising:

[0025] an allocation module, configured to evenly distribute multiple sets of raw prediction data obtained after performing target detection on an image to be detected to at least two threads, filter candidate boxes on the multiple sets of raw prediction data according to a confidence threshold, and save the filtered prediction data results in the at least two threads, wherein each set of raw prediction data includes coordinate data of a candidate box, a category confidence, and a category probability, and the confidence threshold is a fixed-point value;

[0026] a first NEON processing module, configured to calculate, using a NEON instruction set, coordinate data of multiple sets of candidate boxes in the prediction data results saved in each of the threads to obtain a first processing result, wherein the first processing result includes coordinate data of the upper left corner, coordinate data of the lower right corner, area data, and confidence data of the candidate boxes;

[0027] a second NEON processing module, configured to perform non-maximum suppression processing on the upper left corner coordinate data, the lower right corner coordinate data, and the area data using a NEON instruction set in each of the threads;

[0028] A third NEON processing module is used to classify and merge the data after non-maximum suppression processing in each thread, and use the NEON instruction set to perform non-maximum suppression processing on the upper left corner coordinate data, lower right corner coordinate data and area data in the merged data again, and save the index corresponding to the candidate box finally retained;

[0029] The inverse quantization processing module is used to extract the corresponding coordinate value, confidence level and category number according to the index, perform inverse quantization processing on the coordinate value, and obtain the final screening result.

[0030] Optionally, the second NEON processing module and the third NEON processing module both include:

[0031] A descending order arrangement unit is used to arrange the data to be processed with non-maximum value suppression in descending order by using a quick sorting algorithm to obtain a descending order arrangement result;

[0032] An initialization unit, used to initialize the intersection-over-union ratio threshold to a fixed-point value;

[0033] A loop processing unit is used to perform a loop processing calculation of non-maximum suppression on the descending sorting result using a NEON instruction set according to the intersection-over-union ratio threshold.

[0034] Optionally, the loop processing unit includes:

[0035] a determination subunit, configured to select the candidate box with the highest confidence in the descending sorting result as the first target box, and the remaining candidate boxes as the second target boxes;

[0036] a first area calculation subunit, configured to calculate the area of an intersection region between each second target frame and the first target frame using a NEON instruction set;

[0037] A second area calculation subunit is configured to calculate an area threshold according to the area of the intersection region, the area data of the first target frame, and the intersection-over-union ratio threshold, wherein the threshold area is quantized as a fixed-point value;

[0038] The comparison subunit is configured to retain the second target frame if the area threshold is smaller than the area of the second target frame, and delete the second target frame otherwise.

[0039] Optionally, the second area calculation subunit uses the following formula for calculation:

[0040] AreaQ=(1+T)×AreaI–T×Area0;

[0041] Wherein, AreaQ is the area threshold, T is the intersection-over-union ratio threshold, AreaI is the area of the intersection region, and Area0 is the area of the first target frame.

[0042] Optionally, when the second NEON processing module and the third NEON processing module use the NEON instruction set to perform calculation processing, the upper left corner coordinate data, the lower right corner coordinate data, the area data and the confidence data are calculated using fixed-point values.

[0043] The beneficial effects of the above technical solution of the present invention are as follows:

[0044] In an embodiment of the present invention, multi-level NMS calculation processing is performed in multiple threads, which can realize hierarchical synchronous operation and reduce the amount of data processed by NMS step by step; and a quick sort method is used to sort the results in descending order, and the divide-and-conquer sorting strategy is more efficient; in addition, when calculating with the NEON instruction set, fixed-point value calculation is used, which has strong compatibility and high portability, and can achieve low latency and high efficiency in the non-maximum suppression processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of a flow chart of a non-maximum suppression optimization method in Embodiment 1 of the present invention;

[0046] Figure 2 1 is another flow chart of a non-maximum suppression optimization method in the first embodiment of the present invention;

[0047] Figure 3 A schematic diagram of the process of performing NMS processing using NEON instructions provided in the first embodiment of the present invention;

[0048] Figure 4 This is a structural diagram of a non-maximum suppression optimization device in Example 2 of the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

[0050] Non-maximum suppression (NMS) is a commonly used technique in deep learning object detection and image processing. It is used to remove redundant bounding boxes generated during object detection. That is, when the model generates multiple overlapping bounding boxes for the same object, NMS reduces redundancy by retaining the highest confidence bounding box for each object while removing other highly overlapping bounding boxes. The specific algorithm flow of non-maximum suppression is as follows:

[0051] (1) Sort the confidence of all prediction boxes in descending order;

[0052] (2) Select the prediction box with the highest confidence, confirm it as the correct prediction, and calculate the intersection over union (IoU) between it and other prediction boxes;

[0053] (3) Remove the prediction boxes with high overlap based on the calculated IoU;

[0054] (4) Repeat the above steps until all prediction boxes are processed.

[0055] In conventional target detection processes, NMS is usually executed on the CPU. However, with the development of deep learning, the complexity of target detection tasks has increased, and the requirements for real-time performance have become increasingly stringent. In real-time applications, low latency and high efficiency have become two important considerations when performing non-maximum suppression.

[0056] However, existing methods based on a single CPU instruction and single data execution consume excessive computation time and are unable to meet real-time feedback requirements. For example, the method described in patent publication number CN117372673A utilizes CPU resources for acceleration, achieving an NMS time of 10.716ms when detecting 126 targets.

[0057] Alternatively, specialized hardware supporting highly parallel processing capabilities is required, such as field-programmable gate arrays (FPGAs) and graphics processing units (GPUs). For example, the method provided in patent publication number CN113688703A requires an FPGA-based device to implement NMS processing. This design utilizes the hardware characteristics of the FPGA to specifically design the NMS, eliminating the overall sorting step required by all NMS algorithms and thus reducing the overall latency of the algorithm. However, this method is costly and has limited application scenarios.

[0058] Alternatively, as in the method provided in patent publication number CN117492840A, SIMD assembly language is used to implement NMS optimization. However, the SIMD instructions based on assembly language have poor portability.

[0059] Therefore, in order to address the shortcomings of existing NMS technical solutions, such as the need to support parallel computing hardware units, long delays, and inflexible portability, the first embodiment of the present invention provides a non-maximum suppression optimization method. Figure 1 , Figure 1 A schematic flow chart of a non-maximum suppression optimization method provided in Example 1 of the present invention includes the following steps:

[0060] Step 11: Evenly distribute the multiple groups of original prediction data obtained after target detection on the image to be detected to at least two threads, and perform candidate box screening on the multiple groups of original prediction data according to the confidence threshold, and save the screened prediction data results in the at least two threads, wherein each group of the original prediction data includes the coordinate data of the candidate box, the category confidence and the category probability, and the confidence threshold is a fixed-point value.

[0061] In this embodiment, multiple sets of raw prediction data are evenly distributed to at least two threads, and the raw prediction data are processed by multiple threads, thereby improving the processing speed. Among them, the multiple sets of raw prediction data are obtained by performing target detection on the image to be detected. Specifically, the target detection model can be used for inference prediction output, or it can be obtained by other target detection methods. This embodiment does not specifically limit it. Optionally, the target detection model can be a target detection model based on YOLOv5, YOLOv6, etc. Each set of raw prediction data can include data such as the coordinates of the candidate box, the category score (or category confidence), and the category probability.

[0062] After being evenly distributed to at least two threads, in each thread, the candidate boxes are screened using the set confidence threshold, and the predicted data results after screening are saved. In other words, candidate boxes with confidence greater than the set confidence threshold are retained. Among them, the confidence threshold adopts a fixed-point value after quantization processing. The so-called fixed-point value refers to a numerical representation method in which the decimal point position is fixed in a computer, also known as a fixed-point number. Quantization is a technology that maps high-precision data (such as floating-point numbers) to low-precision data (such as integers) to reduce data storage requirements or speed up calculations. However, the quantization process will cause information loss, so it is necessary to restore the data to a high-precision representation through dequantization. In this embodiment, fixed-point value calculation is used to achieve the purpose of simple calculation and low resource consumption, thereby improving the calculation speed.

[0063] In this embodiment, the number of allocated threads is at least two. In actual application, the number can be selected according to actual factors such as the number of threads of the processor, and is not specifically limited in this embodiment.

[0064] In the embodiment of the present invention, taking the allocation to two threads as an example, the predicted data results after screening include: the coordinate data of the candidate box is data0_1 and data0_2, the w index data is datai_1 and datai_2, the h index data is dataj_1 and dataj_2, the confidence data is dataConf_1 and dataConf_2, the category probability data is dataCl ss_1 and dataClss_2, the category number is dataIdx_1 and data I dx_2, and the number of screened prediction boxes is dataCnt. Among them, the original prediction data format of the candidate box read by the program from the hardware device is a two-dimensional array, and its horizontal and vertical data indexes are replaced by w and h respectively. The category probability represents the probability that the candidate box is detected as a certain category. For example, if the model has two categories, the candidate box data will contain a probability value indicating that the candidate box can be detected and identified as these two categories respectively.

[0065] Step 12: Use the NEON instruction set to calculate the coordinate data of multiple groups of candidate boxes in the predicted data results saved in each of the threads, obtaining a first processing result, where the first processing result includes the upper left corner coordinate data, lower right corner coordinate data, area data, and confidence data of the candidate boxes.

[0066] The NEON instruction set is an advanced SIMD (Single Instruction Multiple Data) instruction set under the ARM architecture, used for parallel data processing to improve the performance of compute-intensive applications. In this embodiment, in each thread, the NEON instruction set is used to calculate the filtered predicted data results. More specifically, it is to calculate the coordinate data of the candidate boxes in the predicted data results, thereby obtaining the first processing result. Among them, since the above coordinate data data0_1 and data0_2 are the original coordinate data relative to the center of the anchor box inferred by the object detection model and are not real coordinates, further calculations are required to obtain the real coordinate data. Therefore, in the first processing result obtained after processing the predicted data results using the NEON instruction set, the upper left corner coordinate data, lower right corner coordinate data, area data, and confidence data of the candidate boxes are calculated.

[0067] In this step, by separately using the NEON instruction set for processing and calculation in multiple threads, the data processing efficiency can be effectively improved, thereby achieving low latency and high efficiency.

[0068] In this embodiment, exemplarily, an implementation manner of this step is specifically as follows: Set the quantization bit number, and initialize the anchor box setting value anchor, stride, and confidence threshold confThresh as quantization fixed-point values according to the quantization bit number; then, perform a loop body calculation with 8 candidate box data and their confidence data as a group of data. The loop processing is as follows:

[0069] The first layer of loop:

[0070] Initialize int j = 0; Determine whether j < blob_n (blob_n is the number of feature maps output by the object detection model) holds? If it holds, continue with the loop body of this layer; if it does not hold, jump out of the loop of this layer;

[0071] The second layer of loop:

[0072] Initialize int i = 0; Determine whether i < dataCnt (that is, the above-mentioned number of predicted boxes) holds? If it holds, continue with the loop body of this layer, and perform i + 8 after each completion of the loop body of this layer to update the judgment condition information; if it does not hold, jump out of the loop of this layer;

[0073] Load 8 groups of four candidate box data (tx, ty, tw, th) starting from index i of data0, totaling 32 data, into the vector register memory, where tx and ty correspond to the first two data values in the candidate box border data data0_1 or data0_2, respectively, and are the coordinate values relative to the center point of the candidate box. tw and th correspond to the last two data values in the candidate box data data0_1 or data0_2 saved in S1, and are the width and height data of the candidate box.

[0074] Calculate the center coordinates of the candidate box (Cx, Cy);

[0075] Calculate the width and height (W, H) of the candidate box;

[0076] Calculate the quantized fixed-point values corresponding to the upper left corner coordinates (X1, Y1) and the lower right corner coordinates (X2, Y2) of the candidate box, and save the four coordinate result data to the Rectt array;

[0077] Calculate the quantized fixed-point value corresponding to the area of the candidate box and save the area result to the RectAreat array;

[0078] Save the confidence value of the candidate box to the RectScoret array;

[0079] The second layer of loop ends;

[0080] The first layer of loop ends.

[0081] Step 13: In each of the threads, non-maximum suppression processing is performed on the upper left corner coordinate data, the lower right corner coordinate data, and the area data using the NEON instruction set.

[0082] In this embodiment, after the upper left corner coordinate data, lower right corner coordinate data, area data and confidence data of the candidate box are calculated using the NEON instruction set, further, in each thread, non-maximum suppression processing is performed on the candidate box respectively. Specifically, the upper left corner coordinate data, the lower right corner coordinate data and the area data are subjected to non-maximum suppression processing using the NEON instruction set. At this point, non-maximum suppression is completed once in each thread, and the number of candidate boxes has been reduced. In this step, by using the NEON instruction set to perform non-maximum suppression processing calculations in multiple threads, the data processing efficiency can be effectively improved, thereby achieving low latency and high efficiency.

[0083] Step 14: Classify and merge the data after non-maximum suppression processing in each thread, and use the NEON instruction set to perform non-maximum suppression processing on the upper left corner coordinate data, lower right corner coordinate data and area data in the merged data again, and save the index corresponding to the candidate box finally retained.

[0084] In this step, the data after non-maximum suppression processing in each thread is classified and merged. The so-called classification and merging means that the coordinate data is merged into the same array, the area data is merged into the same array, the confidence data is merged into the same data, and so on. Afterwards, the merged data is further processed again using the NEON instruction set for non-maximum suppression. Specifically, non-maximum suppression processing is performed on the upper left corner coordinate data, the lower right corner coordinate data, and the area data in the merged data to obtain the final candidate box retained. At this point, the second round of non-maximum suppression processing is completed. In this embodiment, the index corresponding to the final retained candidate box can be saved to the data OutIdx.

[0085] Step 15: extract the corresponding coordinate value, confidence level, and category number according to the index, perform dequantization processing on the coordinate value, and obtain the final screening result.

[0086] In this embodiment, since the aforementioned NEON instruction set uses quantized fixed-point values for calculation when performing NMS processing, it is necessary to extract the Rect coordinate value, RectScore confidence, and ClassID category number corresponding to the candidate box as output processing based on the index corresponding to the final candidate box, and perform dequantization processing on the coordinate value to obtain the final accurate data, that is, the final screening result. Among them, the category number can be the category number of the candidate box. For example, if there are multiple categories of candidate boxes after screening in step 11, each category can be numbered separately.

[0087] Therefore, the non-maximum suppression optimization method provided by the embodiment of the present invention performs multi-level NMS calculation processing in multiple threads, which can realize hierarchical synchronous operation and reduce the amount of data processed by NMS step by step; and, the quick sorting method is used to sort the results in descending order, and the divide-and-conquer sorting strategy is more efficient; in addition, when calculating with the NEON instruction set, fixed-point value calculation is used, which has strong compatibility and high portability, and can achieve low latency and high efficiency in the non-maximum suppression processing process.

[0088] The following example illustrates the above non-maximum suppression optimization method.

[0089] In one optional specific implementation manner, the step of performing non-maximum suppression processing using the NEON instruction set includes:

[0090] The quick sort algorithm is used to sort the data to be processed with non-maximum value suppression in descending order to obtain the descending sort result;

[0091] Initialize the intersection-over-union ratio threshold to a fixed-point value;

[0092] According to the intersection over union threshold, use the NEON instruction set to perform a loop processing calculation of non-maximum suppression on the descending sorting result.

[0093] In the embodiment of the present invention, the processes of using the NEON instruction set for non-maximum suppression in step 13 and step 14 are the same, and both can be implemented by the above process. Only the data to be subjected to non-maximum suppression is different. For example, in step 13, it is the data in the predicted data results saved in each thread, while in step 14, it is the merged data.

[0094] In this embodiment, the data to be subjected to non-maximum suppression is sorted in descending order by the quicksort algorithm, and the descending index result is saved as an array RectIdx. Compared with the bubble sort algorithm used in the prior art, the quicksort algorithm used in the present invention is more efficient and faster.

[0095] In this embodiment, by initializing the intersection over union threshold and quantifying the intersection over union threshold into a fixed-point value, the efficiency is higher in subsequent calculations. Then, according to the set intersection over union threshold, use the NEON instruction set to perform a loop processing calculation of non-maximum suppression on the descending sorting result, and delete the candidate boxes with an intersection over union lower than the intersection over union threshold.

[0096] In this embodiment, optionally, the specific loop processing process of using the NEON instruction set to perform a loop processing calculation of non-maximum suppression on the descending sorting result is as follows:

[0097] The first layer of loop:

[0098] Initialize sSzie = dataCnt (the number of preselected boxes / candidate boxes), int cnt = 0; determine whether sSzie > 0 holds? If it holds, continue with the loop body of this layer, and after each completion of the loop body, increment cnt by 1 and reassign sSzie to the number of remaining candidate boxes that have not been deleted; if it does not hold, jump out of the loop of this layer;

[0099] Save the index of the first target box (i.e., index RectIdx[0]) to the array OutIdx. The first target box corresponds to the maximum confidence, and temporarily store its corresponding Rect value, and take its corresponding RectArea value as Area0;

[0100] The second layer of loop:

[0101] Initialize int i = 1; determine whether i < len (len is the number of candidate boxes to be processed) holds? If it holds, continue with the loop body of this layer, and after each completion of the loop body of this layer, perform i + 4 to update the judgment condition information; if it does not hold, jump out of the loop of this layer;

[0102] The NEON instruction calculates the area AreaI of the intersection region between the predicted bounding boxes of RectIdx[0] and the second target bounding boxes (i.e., the remaining predicted bounding boxes except the one with the maximum confidence value);

[0103] The formula for judging the intersection over union threshold of NMS: AreaI / (Area0 + Area[i] - AreaI) < T;

[0104] Convert the above formula to: (1 + T)*AreaI - T*Area0 < T*Area[i];

[0105] The NEON instruction calculates the corresponding fixed-point value AeraQ = (1 + T)*AreaI - T*Area0;

[0106] Where, T is the fixed-point value of the intersection over union threshold, and Area0 is the fixed-point value of the area of the largest predicted bounding box;

[0107] Compare AeraQ with the area value RectArea[i] of the predicted bounding box corresponding to the currently processed RectIdx. If AeraQ < RectArea[i], keep this RectIdx; otherwise, consider it as having a high overlap degree and delete this RectIdx.

[0108] Repeat the above process for the remaining RectIdx index data values until all the bounding boxes in the RectIdx list have been processed.

[0109] The second-level loop body ends;

[0110] The first-level loop body ends;

[0111] Return the index value OutIdx of the remaining candidate box data after processing.

[0112] In some embodiments, the step of performing non-maximum suppression loop processing calculation on the descending sorting result according to the intersection over union threshold using the NEON instruction set includes:

[0113] Take the candidate box with the highest confidence in the descending sorting result as the first target box, and the remaining candidate boxes as the second target boxes;

[0114] Use the NEON instruction set to calculate the area of the intersection region between each of the second target boxes and the first target box;

[0115] Calculate the area threshold according to the area of the intersection region, the area data of the first target box, and the intersection over union threshold, where the threshold area is quantized to a fixed-point value;

[0116] If the area threshold is less than the area of the second target box, keep the second target box; otherwise, delete the second target box.

[0117] In this embodiment, the candidate frame with the highest confidence is first taken as the first target frame, and the remaining candidate frames are taken as the second target frames. Then, the NEON instruction set is used to obtain the area of the intersection area of each second target frame and the first target frame. Finally, an area threshold is calculated according to the intersection-over-union ratio threshold judgment formula of the non-maximum suppression processing. If the area of a second target frame is greater than the area threshold, it means that the first target frame and the second target frame are far away from each other, and it can be considered that the detected targets are of two categories. Therefore, the second target frame can be retained as a comparison for entering the next preferred frame. If the area of a second target frame is less than or equal to the area threshold, it means that the first target frame and the second target frame are very close and the overlap is high. It can be considered that the detected targets are of the same class, that is, redundant, so the second target frame should be deleted.

[0118] In some embodiments, the step of calculating the area fixed-point value of the second target frame based on the area of the intersection region, the area data of the first target frame, and the intersection-over-union ratio threshold is calculated using the following formula:

[0119] AreaQ=(1+T)×AreaI–T×Area0;

[0120] Wherein, AreaQ is the area threshold, T is the intersection-over-union ratio threshold, AreaI is the area of the intersection region, and Area0 is the area of the first target frame.

[0121] In this embodiment, the NMS intersection-to-union ratio threshold determination formula is:

[0122] AreaI / (Area0+Area[i]-AreaI) <T;

[0123] Wherein, Area[i] is the area of any second target box;

[0124] Convert the above formula to: (1+T)×AreaI–T×Area0 <T×Area[i];

[0125] Then, if we set the area threshold AreaQ = (1 + T) × AreaI - T × Area0, we have:

[0126] AreaQ<T×Area[i];

[0127] In this embodiment, since T is a fixed threshold, NEON will eliminate it during calculation. Therefore, it is sufficient to directly compare the values of AreaQ and Area[i].

[0128] In some embodiments of the present invention, when the NEON instruction set is used for computational processing, the upper-left corner coordinate data, the lower-right corner coordinate data, the area data, and the confidence data are calculated using fixed-point values. Compared to the prior art that uses floating-point calculations, in this embodiment, the upper-left corner coordinate data, the lower-right corner coordinate data, the area data, the confidence data, the intersection-over-union threshold, and the like used in the calculations all use quantized fixed-point values, which can effectively improve computational processing efficiency.

[0129] Please refer to Figure 2 , Figure 2 Another flow chart of a non-maximum suppression optimization method provided in Example 1 of the present invention. In this embodiment of the present invention, the method includes the following steps:

[0130] Step 21: Evenly distribute the multiple data sets output by the on-device object detection model inference to two threads, filter candidate boxes based on the confidence threshold fixed-point value, and save the filtered prediction data results, including the coordinate data of the bounding box (i.e., candidate box / prediction box) as data0_1 and data0_2, the w index data as datai_1 and datai_2, the h index data as dataj_1 and dataj_2, the confidence data as dataConf_1 and dataConf_2, the category probability data as dataClss_1 and dataClss_2, the category ID as dataIdx_1 and dataIdx_2, and the number of filtered prediction boxes as dataCnt.

[0131] Step 22: Use the single instruction multiple data NEON instruction set to accelerate the calculation of the coordinate data arrays data0_1 and data0_2 saved in the two threads and save the upper left corner coordinates (X1, Y1) and lower right corner coordinates (X2, Y2) of the predicted box as the array Rectt, the predicted box area as the array RectAreat, and the confidence as the array RectScoret.

[0132] Step 23: Use NEON instructions to perform NMS processing on the processing results Rectt and RectAreat data in two threads respectively.

[0133] Step 24: Merge the predicted box data Rectt, RectAreat, RectScoret, and ClassIDt saved after NMS processing in the two threads into the same array Rect, RectArea, RectScore, and ClassID respectively;

[0134] Step 25: Use NEON instructions to perform NMS processing on the merged Rect and RectArea data again, and save the index corresponding to the final retained prediction box to the array OutIdx.

[0135] Step 26: Extract the corresponding Rect coordinate values, RectScore confidence levels, and ClassID category numbers according to the indexes in OutIdx for output processing, and perform inverse quantization on the coordinate values to obtain the final result.

[0136] Please refer to Figure 3 , Figure 3 FIG. Figure 3 shows the flowchart of the NMS processing using NEON instructions provided in the first embodiment of the present invention. In this embodiment, the processes of using NEON instructions for NMS processing in steps 23 and 25 are the same. The following takes the processing process in step 25 as an example for illustration. As

[0137] shown, optionally, the specific implementation is as follows:

[0138] Initialize the intersection over union threshold iouThresh to the fixed-point value T according to the quantization bits.

[0139] Use NEON instructions to perform loop processing and calculation on the data sorted in descending order. The loop processing is specifically as follows:

[0140] First layer loop:

[0141] Initialize sSzie = dataCnt (the number of preselected boxes), int cnt = 0; determine whether sSzie > 0 holds? If it holds, continue with the loop body of this layer, and after each completion of the loop body, increment cnt by 1 and reassign sSzie to the number of remaining candidate boxes that have not been deleted; if it does not hold, jump out of the loop of this layer;

[0142] Save the index of the bounding box with the maximum confidence corresponding to the first target (i.e., RectIdx[0]) to OutIdx, temporarily store its corresponding Rect value, and take its corresponding RectArea value as Area0;

[0143] Second layer loop:

[0144] Initialize int i = 1; determine whether i < len (len is the number of candidate boxes to be processed) holds? If it holds, continue with the loop body of this layer, and after each completion of the loop body of this layer, perform i + 4 to update the judgment condition information; if it does not hold, jump out of the loop of this layer;

[0145] Use NEON instructions to calculate the area AreaI of the intersection region between the prediction box of RectIdx[0] and the second target bounding boxes (i.e., the remaining prediction boxes except the prediction box with the maximum confidence value);

[0146] Intersection over Union threshold judgment formula of NMS: AreaI / (Area0 + Area[i] - AreaI) < T;

[0147] Convert the above formula to: (1 + T) * AreaI - T * Area0 < T * Area[i];

[0148] Use NEON instruction to calculate the corresponding fixed-point value AeraQ = (1 + T) * AreaI - T * Area0;

[0149] Among them, T is the fixed-point value of the Intersection over Union threshold, and Area0 is the fixed-point value of the area of the largest predicted bounding box;

[0150] Compare AeraQ with the area value RectArea[i] of the predicted bounding box corresponding to the currently processed RectIdx. If AeraQ < RectArea[i], then keep this RectIdx; otherwise, it is considered to have a high overlap degree and delete this RectIdx.

[0151] Repeat the above process for the remaining RectIdx index data values until all the bounding boxes in the RectIdx list have been processed.

[0152] The second-layer loop body ends;

[0153] The first-layer loop body ends;

[0154] Return the remaining index value OutIdx after processing.

[0155] The comparison results between the non-maximum suppression optimization method provided in the embodiments of the present invention and the traditional NMS processing method are shown in the following table:

[0156] Number of pre-selected boxes after confidence filtering 93 Traditional NMS processing method 16.3ms Non-maximum suppression optimization method of the present invention 0.6ms

[0157] As can be seen from the above table, when the number of preselected bounding boxes / candidate bounding boxes is 93 after confidence filtering, the traditional NMS processing method takes 16.3 ms to complete the processing process, while the method in the present invention only takes 0.6 ms, and the calculation processing speed is significantly improved.

[0158] In summary, in the embodiments of the present invention, performing multi-level NMS calculation processing in multiple threads can achieve hierarchical synchronous operation and gradually reduce the amount of data processed by NMS; moreover, using the quicksort method to sort the results in descending order, the divide-and-conquer sorting strategy has higher efficiency; in addition, when calculating using the NEON instruction set, fixed-point value calculation is adopted, which has strong compatibility, high portability, and can achieve low latency and high efficiency in the non-maximum suppression processing process.

[0159] Please refer to Figure 4 , Figure 44 is a schematic structural diagram of a non-maximum suppression optimization device provided in the second embodiment of the present invention. The device 40 includes:

[0160] an allocation module 41 for evenly distributing multiple sets of raw prediction data obtained after performing object detection on an image to be detected to at least two threads, screening the multiple sets of raw prediction data for candidate boxes based on a confidence threshold, and storing the screened prediction data results in the at least two threads, wherein each set of raw prediction data includes coordinate data of a candidate box, a category confidence, and a category probability, and the confidence threshold is a fixed-point value;

[0161] a first NEON processing module 42, configured to calculate, using a NEON instruction set, coordinate data of multiple sets of candidate boxes in the prediction data results stored in each of the threads to obtain a first processing result, wherein the first processing result includes coordinate data of the upper left corner, coordinate data of the lower right corner, area data, and confidence data of the candidate boxes;

[0162] A second NEON processing module 43 is configured to perform non-maximum suppression processing on the upper left corner coordinate data, the lower right corner coordinate data, and the area data using a NEON instruction set in each thread;

[0163] a third NEON processing module 44, configured to classify and merge the data after non-maximum suppression processing in each thread, and perform non-maximum suppression processing again on the upper left corner coordinate data, lower right corner coordinate data, and area data in the merged data using the NEON instruction set, and save the index corresponding to the candidate box finally retained;

[0164] The inverse quantization processing module 45 is used to extract the corresponding coordinate value, confidence level and category number according to the index, and perform inverse quantization processing on the coordinate value to obtain the final screening result.

[0165] Optionally, the second NEON processing module and the third NEON processing module both include:

[0166] A descending order arrangement unit is used to arrange the data to be processed with non-maximum value suppression in descending order by using a quick sorting algorithm to obtain a descending order arrangement result;

[0167] An initialization unit, used to initialize the intersection-over-union ratio threshold to a fixed-point value;

[0168] A loop processing unit is used to perform a loop processing calculation of non-maximum suppression on the descending sorting result using a NEON instruction set according to the intersection-over-union ratio threshold.

[0169] Optionally, the loop processing unit includes:

[0170] a determination subunit, configured to select the candidate box with the highest confidence in the descending sorting result as the first target box, and the remaining candidate boxes as the second target boxes;

[0171] a first area calculation subunit, configured to calculate the area of an intersection region between each second target frame and the first target frame using a NEON instruction set;

[0172] A second area calculation subunit is configured to calculate an area threshold according to the area of the intersection region, the area data of the first target frame, and the intersection-over-union ratio threshold, wherein the threshold area is quantized as a fixed-point value;

[0173] The comparison subunit is configured to retain the second target frame if the area threshold is smaller than the area of the second target frame, and delete the second target frame otherwise.

[0174] Optionally, the second area calculation subunit uses the following formula for calculation:

[0175] AreaQ=(1+T)×AreaI–T×Area0;

[0176] Wherein, AreaQ is the area threshold, T is the intersection-over-union ratio threshold, AreaI is the area of the intersection region, and Area0 is the area of the first target frame.

[0177] Optionally, when the second NEON processing module and the third NEON processing module use the NEON instruction set to perform calculation processing, the upper left corner coordinate data, the lower right corner coordinate data, the area data and the confidence data are calculated using fixed-point values.

[0178] In an embodiment of the present invention, multi-level NMS calculation processing is performed in multiple threads, which can achieve hierarchical synchronous operation and reduce the amount of data processed by NMS step by step; and a quick sort method is used to sort the results in descending order, and the divide-and-conquer sorting strategy is more efficient; in addition, when calculating with the NEON instruction set, fixed-point value calculation is used, which has strong compatibility and high portability, and can achieve low latency and high efficiency in the non-maximum suppression processing process.

[0179] The embodiment of the present invention is a product embodiment corresponding to the above-mentioned method embodiment 1, so it will not be described in detail here. Please refer to the above-mentioned embodiment 1 for details.

[0180] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A non-maximum suppression optimization method, characterized in that: The method comprises the following steps: Evenly distributing multiple groups of raw prediction data obtained after performing target detection on the image to be detected to at least two threads, screening candidate boxes for the multiple groups of raw prediction data according to a confidence threshold, and saving the screened prediction data results in the at least two threads, wherein each group of the raw prediction data includes coordinate data of the candidate box, category confidence, and category probability, and the confidence threshold is a fixed-point value; Using a NEON instruction set, calculating coordinate data of multiple sets of candidate boxes in the prediction data results saved in each of the threads to obtain a first processing result, where the first processing result includes upper left corner coordinate data, lower right corner coordinate data, area data, and confidence data of the candidate boxes; In each of the threads, non-maximum suppression processing is performed on the upper left corner coordinate data, the lower right corner coordinate data, and the area data using a NEON instruction set; Classify and merge the data after non-maximum suppression processing in each thread, and use the NEON instruction set to perform non-maximum suppression processing on the upper left corner coordinate data, lower right corner coordinate data and area data in the merged data again, and save the index corresponding to the candidate box finally retained; The corresponding coordinate value, confidence level and category number are extracted according to the index, and the coordinate value is dequantized to obtain the final screening result.

2. The method according to claim 1, characterized in that The step of using the NEON instruction set to perform non-maximum suppression processing includes: The quick sort algorithm is used to sort the data to be processed with non-maximum value suppression in descending order to obtain the descending sort result; Initialize the intersection-over-union ratio threshold to a fixed-point value; According to the intersection-over-union ratio threshold, a NEON instruction set is used to perform a loop processing calculation of non-maximum value suppression on the descending sorting result.

3. The method according to claim 2, characterized in that The step of performing a cyclic processing calculation of non-maximum value suppression on the descending sorting result using the NEON instruction set according to the intersection-over-union ratio threshold comprises: The candidate box with the highest confidence in the descending sorting results is used as the first target box, and the remaining candidate boxes are used as the second target boxes; Calculate the intersection area of each second target frame and the first target frame using the NEON instruction set; Calculating an area threshold according to the intersection area, the area data of the first target frame, and the intersection-over-union ratio threshold, wherein the threshold area is quantized as a fixed-point value; If the area threshold is smaller than the area of the second target frame, the second target frame is retained; otherwise, the second target frame is deleted.

4. The method according to claim 3, characterized in that The step of calculating the area fixed-point value of the second target frame according to the area of the intersection region, the area data of the first target frame, and the intersection-over-union ratio threshold is calculated using the following formula: AreaQ=(1+T)×AreaI–T×Area0; Wherein, AreaQ is the area threshold, T is the intersection-over-union ratio threshold, AreaI is the area of the intersection region, and Area0 is the area of the first target frame.

5. The method according to claim 1, wherein When the NEON instruction set is used for calculation processing, the upper left corner coordinate data, the lower right corner coordinate data, the area data and the confidence data are calculated using fixed-point values.

6. A non-maximum suppression optimization device, characterized in that: The device comprises: an allocation module, configured to evenly distribute multiple sets of raw prediction data obtained after performing target detection on an image to be detected to at least two threads, filter candidate boxes on the multiple sets of raw prediction data according to a confidence threshold, and save the filtered prediction data results in the at least two threads, wherein each set of raw prediction data includes coordinate data of a candidate box, a category confidence, and a category probability, and the confidence threshold is a fixed-point value; a first NEON processing module, configured to calculate, using a NEON instruction set, coordinate data of multiple sets of candidate boxes in the prediction data results saved in each of the threads to obtain a first processing result, wherein the first processing result includes coordinate data of the upper left corner, coordinate data of the lower right corner, area data, and confidence data of the candidate boxes; a second NEON processing module, configured to perform non-maximum suppression processing on the upper left corner coordinate data, the lower right corner coordinate data, and the area data using a NEON instruction set in each of the threads; A third NEON processing module is used to classify and merge the data after non-maximum suppression processing in each thread, and use the NEON instruction set to perform non-maximum suppression processing on the upper left corner coordinate data, lower right corner coordinate data and area data in the merged data again, and save the index corresponding to the candidate box finally retained; The inverse quantization processing module is used to extract the corresponding coordinate value, confidence level and category number according to the index, perform inverse quantization processing on the coordinate value, and obtain the final screening result.

7. The device according to claim 6, characterized in that The second NEON processing module and the third NEON processing module both include: A descending order arrangement unit is used to arrange the data to be processed with non-maximum value suppression in descending order by using a quick sorting algorithm to obtain a descending order arrangement result; An initialization unit, used to initialize the intersection-over-union ratio threshold to a fixed-point value; A loop processing unit is used to perform a loop processing calculation of non-maximum suppression on the descending sorting result using a NEON instruction set according to the intersection-over-union ratio threshold.

8. The device according to claim 7, characterized in that The cycle processing unit includes: a determination subunit, configured to select the candidate box with the highest confidence in the descending sorting result as the first target box, and the remaining candidate boxes as the second target boxes; a first area calculation subunit, configured to calculate the area of an intersection region between each second target frame and the first target frame using a NEON instruction set; A second area calculation subunit is configured to calculate an area threshold according to the area of the intersection region, the area data of the first target frame, and the intersection-over-union ratio threshold, wherein the threshold area is quantized as a fixed-point value; The comparison subunit is configured to retain the second target frame if the area threshold is smaller than the area of the second target frame, and delete the second target frame otherwise.

9. The device according to claim 8, characterized in that The second area calculation subunit uses the following formula for calculation: AreaQ=(1+T)×AreaI–T×Area0; Wherein, AreaQ is the area threshold, T is the intersection-over-union ratio threshold, AreaI is the area of the intersection region, and Area0 is the area of the first target frame.

10. The device according to claim 6, characterized in that When the second NEON processing module and the third NEON processing module perform calculation processing using the NEON instruction set, the upper left corner coordinate data, the lower right corner coordinate data, the area data, and the confidence data are calculated using fixed-point values.

Images