Detection frame screening method, apparatus, computer equipment and storage medium

Abstract

This application discloses a detection box filtering method, apparatus, computer device, and storage medium. The method includes: in the i-th similarity calculation round, selecting an (i+1)-th detection box from the detection box set by any thread in a preset thread group and requesting its writing; after detecting that the i-th detection box has been written, performing similarity calculations between the target detection box corresponding to the thread in the detection box set and the i-th detection box by each thread in the thread group, obtaining a similarity calculation result between the target detection box corresponding to the thread and the i-th detection box; determining the redundancy detection result of the target detection box corresponding to the thread based on the similarity calculation result by each thread in the thread group; and determining the detection box filtering result of the detection box set based on the redundancy detection results corresponding to each thread. Using this application can improve the efficiency of detection box filtering.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a detection box screening method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Digital image object detection schemes can output multiple object detection boxes. These output boxes often contain redundant detection boxes. Therefore, filtering can be performed on these boxes to reduce redundancy.

[0003] However, existing redundancy removal methods select pairs of detection boxes for similarity calculation. During the similarity calculation process, the data of the detection boxes needs to be loaded from the video memory into the memory. The loaded detection boxes are then calculated, and the loading process takes a lot of time. Summary of the Invention

[0004] Therefore, it is necessary to provide a detection box screening method, apparatus, computer equipment, and storage medium to address the above-mentioned technical problems, which can improve the efficiency of detection box screening.

[0005] In a first aspect, embodiments of this application provide a detection box filtering method, including:

[0006] Select the first detection box from the acquired detection box set using any thread in the preset thread group, and request to write the first detection box.

[0007] In the i-th similarity calculation round, when there is a detection box to be written in the detection box set, any thread in the thread group selects the (i+1)-th detection box from the detection box set and requests to write the (i+1)-th detection box; where i is an integer.

[0008] After the i-th detection box is detected to be written, each thread in the thread group performs a similarity calculation between the target detection box corresponding to the thread in the detection box set and the i-th detection box to obtain the similarity calculation result between the target detection box corresponding to the thread and the i-th detection box.

[0009] Based on the similarity calculation results, the redundant detection results of the target detection boxes corresponding to each thread in the thread group are determined.

[0010] If there are already written detection boxes that have not been similarized, increase i by 1;

[0011] When there are no detection boxes to be written in the detection box set, the detection box filtering result of the detection box set is determined based on the redundant detection results corresponding to each thread.

[0012] Secondly, embodiments of this application provide a detection frame screening device, comprising:

[0013] The first box writing module is used to select the first detection box from the acquired detection box set through any thread in the preset thread group and request to write the first detection box;

[0014] The intermediate box writing module is used to select the (i+1)th detection box from the detection box set and request to write the (i+1)th detection box when there is a detection box to be written in the detection box set in the i-th similarity calculation round; where i is an integer.

[0015] The similarity calculation module is used to perform similarity calculation between the target detection box corresponding to the thread in the detection box set and the i-th detection box after the i-th detection box is detected to be written, and to obtain the similarity calculation result between the target detection box corresponding to the thread and the i-th detection box.

[0016] The redundancy detection module is used to determine the redundancy detection result of the target detection box corresponding to each thread based on the similarity calculation results.

[0017] The loop judgment module is used to increment i by 1 if there is a written detection box that has not been similarly calculated;

[0018] The detection box filtering module is used to determine the detection box filtering result of the detection box set based on the redundant detection results of each thread when there are no detection boxes to be written in the detection box set.

[0019] Thirdly, this application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the detection box screening method described above.

[0020] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the detection box screening method described above.

[0021] Fifthly, this application provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps in the detection box screening method described above.

[0022] The aforementioned detection box filtering method, apparatus, computer equipment, computer-readable storage medium, and computer program product, by issuing a new detection box write request in the same similarity calculation round, and performing similarity calculation on the current detection box and the target detection box corresponding to the thread when the current detection box is written, achieves parallel execution of the current detection box similarity calculation operation and the new detection box write operation in the same similarity calculation round. This eliminates the need to wait for the current detection box to complete writing in the current similarity calculation round, thus shortening the waiting time for detection box writing in the current similarity calculation round. This solves the problem of low detection box filtering efficiency caused by the long time required for detection box writing operations in the prior art, and improves the efficiency of detection box filtering by achieving parallel execution of the current detection box similarity calculation operation and the new detection box write operation. Attached Figure Description

[0023] Figure 1 An application environment diagram of a detection box screening method provided in an embodiment of this application;

[0024] Figure 2 A schematic flowchart illustrating a detection box filtering method provided in an embodiment of this application;

[0025] Figure 3 This is an application scenario diagram of a detection box filtering method provided in an embodiment of this application;

[0026] Figure 4 This is an application scenario diagram of an existing similar calculation method;

[0027] Figure 5 This is an application scenario diagram of a detection box filtering method provided in an embodiment of this application;

[0028] Figure 6 A structural block diagram of a detection frame screening device provided in an embodiment of this application;

[0029] Figure 7 An internal structural diagram of a computer device provided in an embodiment of this application;

[0030] Figure 8 An internal structural diagram of another computer device provided in an embodiment of this application;

[0031] Figure 9 This is an internal structural diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0033] The detection box filtering method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a communication network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0034] like Figure 2 As shown, this application embodiment provides a detection box filtering method, which is applied to... Figure 1 The method will be illustrated using terminal 102 or server 104 as examples. It is understood that the computer device may include at least one of a terminal and a server. The method includes the following steps:

[0035] S201. Select the first detection box from the acquired detection box set through any thread in the preset thread group, and request to write the first detection box.

[0036] A thread group includes at least one thread. Any thread in the thread group can be selected as any thread in the group. Each thread in the thread group can be used to perform similarity calculations on the detection boxes. The thread group may include Graphics Processing Unit (GPU) threads and / or Central Processing Unit (CPU) threads. A detection box set includes at least one detection box. Object detection is performed on an image, resulting in multiple detection boxes. A detection box set is generated based on the obtained multiple detection boxes. One thread in the thread group selects the first detection box from the detection box set and requests that the first detection box be written from video memory to memory. Writing the detection box specifically involves writing the relevant data of the detection box into memory. For example, the relevant data includes coordinates, such as the coordinates of the detection box's keypoints. The relevant data may also include the detection box's centrality and score (confidence). The thread requests the memory access unit to write the detection box; the memory access unit can access memory and write the detection box into memory.

[0037] S202. In the i-th similarity calculation round, when there is a detection box to be written in the detection box set, select the (i+1)-th detection box from the detection box set through any thread in the thread group, and request to write the (i+1)-th detection box.

[0038] Where i is an integer.

[0039] Typically, there are many detection boxes, and similarity calculations are required for every two detection boxes. A similarity calculation round refers to the similarity calculation operation between a detection box and every other detection box in the detection box set. In this embodiment, in the i-th similarity calculation round, the (i+1)-th operation of writing a new detection box is performed, while the i-th similarity calculation operation is performed in parallel. i is an integer greater than or equal to 0. In the same similarity calculation round, at most one detection box is written, and only one similarity calculation operation is performed. Different detection boxes are written in different similarity calculation rounds. Specifically, in the first similarity calculation round, the second detection box is selected, and a request is made to write it; in the second similarity calculation round, the third detection box is selected, and a request is made to write it; in the third similarity calculation round, no detection box is written.

[0040] The detection box to be written can refer to a non-redundant detection box that has not been written before. The detection box to be written is actually the detection box that needs to be calculated for similarity in the next similarity calculation round. Selecting the (i+1)th detection box from the detection box set is actually selecting the (i+1)th detection box from the detection boxes to be written. In different similarity calculation rounds, the threads requesting to write detection boxes are independent of each other. Any thread in S201 can be the same as or different from any thread in S202, and the thread selection steps of S201 and S202 are independent. S201 and S202 can randomly select any thread, or select according to a certain rule as needed. There are no specific restrictions on this.

[0041] S203. After the i-th detection box is detected to be written, each thread in the thread group performs a similarity calculation between the target detection box corresponding to the thread in the detection box set and the i-th detection box to obtain the similarity calculation result between the target detection box corresponding to the thread and the i-th detection box.

[0042] The calculation operation for the i-th detection box can only begin after the i-th detection box has been written completely. It should be noted that even if the i-th detection box is written completely, the thread may be performing other operations. The calculation operation for the i-th detection box will only begin after these other operations are completed. Each pair of detection boxes in the detection box set is expected to undergo similarity calculation. This embodiment uses multiple threads, with different threads performing similarity calculations on different detection boxes in parallel. Each detection box in the detection box set is assigned a corresponding thread, one thread per detection box. At least one detection box corresponding to a thread is determined as the target detection box for that thread. In each round of similarity calculation, each thread performs similarity calculations between its corresponding target detection boxes and the i-th detection box. Each thread in the thread group performs similarity calculations between two detection boxes in parallel. The target detection boxes corresponding to a thread have been pre-written into memory. Different threads correspond to different target detection boxes. The number of target detection boxes corresponding to a thread is at least one. The number of target detection boxes corresponding to different threads can be the same or different. After the i-th detection box is written, the thread performs similarity calculations on the i-th detection box and each corresponding target detection box, obtaining the similarity calculation results between each target detection box and the i-th detection box. The similarity calculation result can refer to the similarity value between the target detection box and the i-th detection box, where the similarity value can be represented by the Intersection over Union (IOU). The similarity calculation can specifically refer to the IOU calculation, which calculates the ratio of the intersection area to the union area of ​​the two rectangles corresponding to the two detection boxes.

[0043] As in the previous example, in the first similarity calculation round, the IOU between the first detection box and at least one corresponding target detection box is calculated; in the second similarity calculation round, the IOU between the second detection box and at least one corresponding target detection box is calculated; and in the third similarity calculation round, the IOU between the third detection box and at least one corresponding target detection box is calculated.

[0044] Step S203 is also executed in the i-th round of similarity calculation.

[0045] S204. Based on the similarity calculation results, determine the redundant detection results of the target detection boxes corresponding to each thread in the thread group.

[0046] Redundancy detection results refer to the detection results indicating whether a target detection box is redundant. A redundancy detection result of "redundant" indicates that the detection box is redundant. When a target detection box is determined to be similar to the i-th detection box based on similarity calculation results, the target detection box is determined to be redundant. Specifically, if the Intersection over Union (IOU) between the target detection box and the i-th detection box is greater than or equal to a preset threshold, the target detection box is determined to be redundant. The thread detects the redundancy detection results of all corresponding target detection boxes.

[0047] S205. If there are already written detection boxes that have not been similarized, increase i by 1.

[0048] If there are already written detection boxes that have not yet undergone similarity calculation, it indicates that there are still detection boxes that need to be calculated for similarity, and at least one more round of similarity calculation is required. In this case, i is incremented by 1, and the process proceeds to the next round of similarity calculation. Steps S202-S204 are performed in the same round of similarity calculation, i.e., the i-th round of similarity calculation, and S205 indicates that the process proceeds to the next round of similarity calculation, i.e., the (i+1)-th round of similarity calculation.

[0049] S206. When there are no detection boxes to be written in the detection box set, determine the detection box filtering result of the detection box set according to the redundant detection results corresponding to each thread.

[0050] The absence of detection boxes to be written can mean that detection boxes in the detection box set have been written to memory, and / or that the detection boxes are redundant. Detection boxes that have been identified as redundant do not need to be subjected to subsequent similarity calculations, thus reducing the amount of computational data. The detection box filtering result can refer to the detection results of whether each detection box in the detection box set is a redundant box. Based on the redundant detection results calculated by each thread in each similarity calculation round, redundant boxes are counted, and the detection box filtering result of the detection box set is determined based on the counted redundant boxes.

[0051] For example, in the first round of similarity calculation, target detection box A is a redundant box; in the second round of similarity calculation, no redundant boxes are detected; in the third round of similarity calculation, target detection boxes B and C are redundant boxes. The final result of the detection box filtering is: detection boxes A, B, and C are redundant boxes.

[0052] As can be seen, in this embodiment, by issuing a new detection box write request in the same similarity calculation round, and performing similarity calculation on the current detection box and the target detection box corresponding to the thread when the current detection box is written, the similarity calculation operation of the current detection box and the new detection box write operation are executed in parallel in the same similarity calculation round. Thus, there is no need to wait for the current detection box to be written in the current similarity calculation round, thereby shortening the waiting time for detection box writing in the current similarity calculation round. This solves the problem of low detection box screening efficiency caused by the long time required for detection box writing operations in the prior art, and improves the screening efficiency of detection boxes by executing the similarity calculation operation of the current detection box and the new detection box writing operation in parallel.

[0053] In some embodiments, requesting to write to the first detection box includes:

[0054] Send a write request for the first detection frame to the memory access unit so that the memory access unit stores the first detection frame in the first storage area. The memory access unit is used to write the detection frame to the first storage area or the second storage area.

[0055] The request to write to the (i+1)th detection box includes:

[0056] Send a write request for the (i+1)th detection frame to the memory access unit so that the memory access unit stores the (i+1)th detection frame in the target storage area; the target storage area is a storage area other than the storage area where the ith detection frame is written.

[0057] The memory access unit is used to access memory and write data into it. A write request instructs the memory access unit to write the relevant data for the detection frame. The first and second storage areas are used to store different detection frames. The first and second storage areas are different. There is one first storage area and one second storage area. The first storage area can store one detection frame. The second storage area can store one detection frame. Detection frames can be repeatedly written to both the first and second storage areas; newly written detection frames overwrite previously written detection frames.

[0058] Adjacent detection frames are written to different storage areas. The storage area where the i-th detection frame is written is different from the storage area where the (i+1)-th detection frame is written. The target storage area where the (i+1)-th detection frame is written is different from the storage area where the i-th detection frame is written. The target storage area is either the first storage area or the second storage area. If the previous written storage area is the first storage area, the target storage area is the second storage area; if the previous written storage area is the second storage area, the target storage area is the first storage area.

[0059] In a specific example, the first detection box is written to the first storage area; in the first similarity calculation round, the second detection box is written to the second storage area; in the second similarity calculation round, the third detection box is written to the first storage area; and in the third similarity calculation round, the fourth detection box is written to the second storage area.

[0060] As can be seen, in this embodiment, two storage areas are reused and adjacent detection boxes are written to different storage areas. This reduces the storage resources occupied while accommodating parallel computation and writing operations, thereby improving the efficiency of detection box screening.

[0061] In some embodiments, after determining the redundant detection results of the target detection box corresponding to the thread based on the similarity calculation results, the method further includes:

[0062] When the redundant detection result of the target detection box corresponding to the thread is determined to be redundant, the target detection box corresponding to the thread is marked as redundant.

[0063] If a target detection box is found to be redundant, it can be either marked as redundant or removed. Redundant detection boxes do not require further similarity calculations or redundant detection. Redundancy marking can involve processing the relevant data of the detection box; for example, specific key information can be added, or existing data can be modified. For instance, the confidence score of a detection box can be modified to a fixed negative value or an invalid value. Typically, the confidence score is positive; marking a detection box as redundant by changing its confidence score to a negative value simplifies the redundancy marking process.

[0064] As can be seen, in this embodiment of the application, by marking the detection box as redundant when the detection result of the detection box is redundant, the redundant box can be accurately identified for subsequent detection box screening. In addition, compared with directly removing the redundant box, selecting and marking the redundant box has lower time complexity and faster execution speed, thereby improving the detection box screening efficiency.

[0065] In some embodiments, after determining the redundant detection results of the target detection box corresponding to the thread based on the similarity calculation results, the method further includes:

[0066] Delete the target detection boxes that are redundant in the detection box set.

[0067] Besides marking redundant detection boxes, redundant detection boxes can also be removed directly from the detection box set, or from the target detection box corresponding to the thread. This way, the redundant detection box will not be included in the similarity calculation in subsequent rounds. However, this removal method requires the thread to perform additional operations, which consumes more time in the original detection box similarity calculation process.

[0068] In some embodiments, after the writing of the i-th detection box is detected to be complete, a similarity calculation is performed between the target detection box corresponding to the thread in the detection box set and the i-th detection box, including:

[0069] When it is detected that the i-th detection box has been written and there is no redundant marker in the i-th detection box, the target detection box corresponding to the thread in the detection box set is compared with the i-th detection box for similarity calculation.

[0070] When it is detected that the i-th detection box has been written and there is a redundant marker in the i-th detection box, i is increased by 1.

[0071] If the i-th detection box does not have a redundant marker, it is determined that the i-th detection box is not redundant and similarity calculation is required. However, if the i-th detection box has been written completely and has a redundant marker, similarity calculation is not performed on the i-th detection box. Increasing i by 1 is equivalent to proceeding to the next similarity calculation round.

[0072] As can be seen, in this embodiment of the application, by determining whether there is a redundant marker in the i-th detection box before similarity calculation, and if there is, not performing similarity calculation, directly entering the next round of similarity calculation, the similarity calculation of redundant boxes is avoided, the time of similarity calculation is shortened, and thus the detection box screening efficiency is improved.

[0073] In some embodiments, the detection box filtering result of the detection box set is determined based on the redundancy detection results corresponding to each thread, including:

[0074] Based on the redundancy detection results for each thread, count the first number of detection boxes that are considered redundant.

[0075] Calculate the ratio between the first number and the second number of detection boxes included in the detection box set;

[0076] When the quantity ratio meets the majority redundancy condition, the detection boxes whose redundancy detection results are non-redundant are added to a new set, and the detection boxes in the new set are sorted to obtain the first sorting result, which is used as the detection box filtering result of the detection box set.

[0077] When the quantity ratio meets the condition of a few redundancies, the detection boxes are sorted in the detection box set according to the redundancy detection results of each detection box to obtain a second sorting result; the quantity difference between the first quantity and the second quantity is calculated.

[0078] The sorting result of the detection boxes with the difference in quantity is extracted from the second sorting result, and used as the detection box filtering result of the detection box set.

[0079] The quantity ratio refers to the proportion of redundant boxes in the detection box set. The majority redundancy condition is used to determine if the number of redundant boxes in the detection box set is excessive. Specifically, the majority redundancy condition can refer to whether the quantity ratio is greater than or equal to a preset ratio threshold, where the ratio threshold is determined through experimental statistics; for example, the ratio threshold can be 0.9. The minority redundancy condition determines whether the number of redundant boxes in the detection box set is insufficient. Specifically, the minority redundancy condition can refer to whether the quantity ratio is less than a ratio threshold. The ratio threshold for the majority redundancy condition can be the same as the ratio threshold for the minority redundancy condition. The first sorting result is a new set ordered by confidence level. The first sorting result does not contain redundant boxes and can be used as the detection box filtering result and directly output. The second sorting result is a set of detection boxes ordered by confidence level. The second sorting result contains redundant boxes, and it is necessary to extract the non-redundant detection boxes arranged in the order of the second sorting result as the detection box filtering result. The quantity difference is the number of non-redundant detection boxes in the detection box set.

[0080] The final output of object detection operations is usually a sorted result based on confidence level. Therefore, the result of the detection box filtering is an ordered sequence of detection boxes. In practice, the sorting of redundant boxes is a redundant operation. When there are many redundant boxes, only the non-redundant detection boxes are sorted, which is faster. When there are few redundant boxes, sorting the redundant boxes takes less time, and sorting the entire detection box set is faster. When the number of redundant detection boxes is large, the non-redundant detection boxes are extracted to form a new set, and the detection boxes in the new set are sorted according to confidence level. When the number of redundant detection boxes is small, the detection boxes in the new set are directly sorted according to confidence level.

[0081] In a specific example, the confidence of redundant target detection boxes is modified to a preset negative value. The second sorting result is the sorting result obtained by arranging the boxes in descending order of confidence. The top few detection boxes with the largest difference in number are extracted from the second sorting result and determined as the detection box filtering result of the detection box set.

[0082] As can be seen, in this embodiment of the application, by determining whether the number of redundant boxes in the detection box set is too large or too small, and accordingly determining the method of removing and sorting redundant boxes, redundant screening operations can be reduced to the greatest extent and the screening speed of detection boxes can be accelerated.

[0083] In some embodiments, before selecting the first detection box from the acquired set of detection boxes by any thread in a preset thread group and requesting to write the first detection box, the method further includes:

[0084] Obtain at least one detection bounding box at a level in the same image, and the confidence score of each detection bounding box;

[0085] For each level of detection boxes, the detection boxes corresponding to each level are filtered based on their confidence level to obtain the filtering results for each level.

[0086] The detection boxes retained in the screening results of each level are sorted according to their confidence scores to obtain the detection box set.

[0087] Detection boxes in different images correspond to targets in different scenes, and no filtering is performed between detection boxes from different images. During target detection, anchor boxes of different sizes can be used to obtain detection boxes. All detection boxes obtained based on anchor boxes of a single size are considered to be detection boxes at the same level. In reality, the directly output detection boxes may contain detection boxes with low confidence scores. These low-confidence detection boxes are generally considered unreliable and will not be used for subsequent tasks. In this case, confidence scores can be used to coarsely filter the detection boxes, reducing redundant data and improving filtering efficiency.

[0088] Typically, detection boxes with confidence levels below a preset confidence threshold are removed, while those with confidence levels greater than or equal to the preset threshold are included in the filtering results. Detection can be performed individually at each level or across all levels. The confidence thresholds for different levels can be the same or different. Detection boxes from multiple levels can be filtered in parallel. The filtering results from each level are then merged and sorted according to confidence level, resulting in an ordered sequence as the detection box set.

[0089] Correspondingly, when selecting a detection box from the detection box set, the detection boxes can be selected sequentially according to the order in the detection box set, for example, in descending order of confidence.

[0090] As can be seen, in this embodiment of the application, by performing coarse screening on the detection boxes obtained directly from the same image, the amount of data for similarity calculation can be quickly reduced, the screening speed can be improved, and redundant calculation operations on invalid data can be reduced, thus reducing resource waste.

[0091] In some embodiments, the thread group includes an image processor thread.

[0092] As can be seen, in this embodiment, the image processor thread executing steps S201-S204 is equivalent to executing steps S201-S204 through the GPU. This can make full use of GPU resources, thereby avoiding the situation where the CPU performs some computational operations, which introduces additional data transfer overhead. There is no data copying overhead between the CPU and the GPU, which improves resource utilization and improves the efficiency of detection box screening.

[0093] In a specific example, such as Figure 3 The diagram illustrates an application scenario for the bounding box filtering method, providing a multi-level bounding box NMS post-processing step. Specifically, Figure 3 The following content is shown:

[0094] Object detection is performed on an image to obtain multi-level detection boxes. Each detection box has data such as centrality, confidence, and coordinates, which can identify the detection box.

[0095] 1. Coarse Screening: Coarse screening is performed on each level of the multi-level detection boxes. Coarse screening includes processing the centrality and confidence (e.g., performing a sigmoid operation), restoring the coordinates of the detection boxes in the original image, and filtering each detection box according to the coarse screening confidence threshold thresh0. Coarse screening at each level can be performed concurrently, and the results are summarized and concatenated into the first array in the video memory, with a size of [n]. Theoretically, the time complexity of the coarse screening step is O(1) parallel time. The sigmoid function is S(x) = 1 / (1+e^(-x) / x). -x ).

[0096] 2. Sorting: Sort the elements in the first array (i.e., the detection boxes) by their confidence scores as the key. There are many GPU sorting methods; you can choose a sorting method with a time complexity of O(n) parallel time to sort the elements in the first array and obtain the set of detection boxes.

[0097] 3. Redundancy Marking: Start a GPU thread group with k threads. Allocate two buffers in the shared memory of the thread group, each buffer having storage capacity for at least k+1 detection boxes. Any thread sends a load instruction to the memory access unit. The memory access unit executes the load instruction, loading the information of the detection box corresponding to the first element of the detection box set into the (k+1)th position of the second buffer. Then, each thread loads the information of the (t+k*p)th detection box (0≤p≤n / k, p is an integer) in its own detection box set corresponding to its thread number t (0<t≤k, t is an integer) into the t-th position of the second buffer. Alternatively, each buffer can have storage capacity for at least one detection box, and each thread loads the information of the (t+k*p)th detection box into its own thread-specific storage space (e.g., a register). The aforementioned first storage area can be the second buffer, or the (k+1)th position of the second buffer. The aforementioned second storage area can be the first buffer, or the (k+1)th position of the first buffer.

[0098] All threads are synchronized.

[0099] Starting from the second element in the detection box set, traverse the elements within the detection box set (e.g., using a for loop). When traversing to the s-th element, which corresponds to the i-th similarity calculation round, any thread in the thread group sends a load instruction to the memory access unit. The memory access unit executes the load instruction, loading the information of the detection box corresponding to the s-th element (the (i+1)-th detection box) into the (k+1)-th position of the (1+s&1)-th buffer. Simultaneously, retrieve the s-1-th element (the i-th detection box) and determine whether to perform similarity calculation. If the detection box corresponding to the s-1-th element is redundant (confidence is a fixed negative number), then no similarity calculation is performed, and traversal continues to the next element. Otherwise, calculate the IOU between the detection box corresponding to the thread and the detection box corresponding to the s-1-th element. If the IOU is greater than the preset threshold thresh1, the detection box corresponding to thread number t is redundant, and the confidence of the redundant box is modified to a fixed negative number (illegal value). Then, accumulate the count of redundant boxes m within the thread group, which is used as the number of redundant boxes m in the i-th similarity calculation round.

[0100] After traversing the set of detection boxes, the m values ​​from each similarity calculation round are accumulated into the global count M. The complexity of redundant marking is theoretically O(n) parallel time.

[0101] 4. Redundancy Removal After Sorting: If M < 0.9n (M / n < 0.9), sort the detection boxes in the detection box set using confidence score as the key, and then output the first nM boxes as the final result of the algorithm. If M ≥ 0.9n (M / n ≥ 0.9), concurrently merge the non-redundant detection boxes in the detection box set into a new second array, which is the new set with size nM. Sort the detection boxes in the second array using confidence score as the key, and then output the second array as the final result of object detection.

[0102] like Figure 4 The diagram illustrates an application scenario of NMS provided by existing technology. The left side shows the operations performed by thread t according to the timeline, and the right side shows the operations performed by the memory access unit according to the corresponding timeline. LOAD loads the relevant data of the detection box into memory, CALC calculates the IOU for the data in memory, and LOAD From MEM writes data from video memory to the shared memory of the thread group. Specifically, thread t (or any other thread) requests to write a detection box, the memory access unit writes the detection box into memory, and after the memory access unit completes the writing, thread t performs similarity calculations on the detection box. All threads are synchronized; the square rectangles represent synchronization operations. This process continues, with each round of similarity calculation first writing the detection box to be calculated, and after writing, each thread executes the corresponding similarity calculation steps. Figure 4 It can be seen that the blank area between the request to write to LOAC and the calculation of CALC is the waiting time for thread t. If thread t does not perform the request to write operation, the area between the synchronization operation and the calculation of CALC is the waiting time for thread t. Thread synchronization refers to waiting for the calculation operations of each thread in the thread group to complete in the same similarity calculation round. In reality, different threads calculate different amounts of data; for example, some threads correspond to one target detection box, while others correspond to multiple target detection boxes. This results in different execution times for the similarity calculations of each thread, requiring the completion of similarity calculations by all threads in the thread group before proceeding to the next similarity calculation round.

[0103] like Figure 5The diagram illustrates an application scenario of the detection box filtering method provided in this embodiment. The left side shows the operations performed by thread t according to the timeline, and the right side shows the operations performed by the memory access unit according to the corresponding timeline. Specifically, thread t (or any other thread) requests to write the first detection box to the first storage area, and the memory access unit executes the write operation to write the first detection box to the first storage area. In the first similarity calculation round, thread t (or any other thread) requests to write the second detection box to the second storage area, and the memory access unit executes the write operation to write the second detection box to the second storage area. When the first detection box is written, thread t performs similarity calculations between the corresponding target detection box and the detection box (the first detection box) in the first storage area to obtain the redundancy detection result of the target detection box. Thread synchronization. In the second similarity calculation round, thread t (or any other thread) requests to write the third detection box to the first storage area. The memory access unit executes the write operation to write the third detection box to the first storage area. After the second detection box is written, thread t performs similarity calculations between the corresponding target detection box and the detection box (the second detection box) in the second storage area to obtain the redundancy detection result of the target detection box. Thread synchronization. In the third similarity calculation round, after the third detection box is written, thread t performs similarity calculations between the corresponding target detection box and the detection box (the third detection box) in the first storage area to obtain the redundancy detection result of the target detection box. Thread synchronization. Comparison Figure 4 Execution time of medium threads and Figure 5 The execution time of the intermediate threads clearly shows that the detection box filtering method proposed in this application can improve the overall detection box filtering speed. Furthermore, similar calculation steps for a detection box can only be executed after the detection box is written into memory. Figure 4 and Figure 5 This is represented as follows: the bottom edge of the rectangle of the step where the detection box is written into memory is aligned with the top edge of the rectangle of the step where the detection box is calculated.

[0104] As can be seen, in this embodiment, using GPU threads to perform redundancy removal and IOU calculation in the same stage can utilize GPU computing power and dynamically reduce the number of IOU calculations, thus reducing computational latency. Furthermore, the entire process is implemented on the GPU, eliminating data copying overhead between the CPU and GPU. A software pipeline is introduced within the GPU computation function to fully utilize the parallelism of computation and memory access. When traversing the detection box set, if the corresponding detection box at a certain element is a redundant box, the loop continues to the next element without performing this round of IOU calculation, thereby reducing the computational load. Within the GPU computation function, the computational hardware unit and the memory access unit work in parallel more extensively, improving the overall computational efficiency. The overall time complexity should be O(n) parallel time.

[0105] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0106] Based on the same inventive concept, this application also provides a detection box filtering device. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more detection box filtering device embodiments provided below can be found in the limitations of the detection box filtering method above, and will not be repeated here.

[0107] like Figure 6 As shown, this application embodiment provides a detection box screening device 600, including:

[0108] The first box writing module 601 is used to select the first detection box from the acquired detection box set through any thread in the preset thread group and request to write the first detection box;

[0109] The intermediate box writing module 602 is used to select the (i+1)th detection box from the detection box set and request to write the (i+1)th detection box when there is a detection box to be written in the detection box set in the i-th similarity calculation round; where i is an integer.

[0110] The similarity calculation module 603 is used to perform similarity calculation between the target detection box corresponding to the thread in the detection box set and the i-th detection box after the i-th detection box is detected to be written, through each thread in the thread group, to obtain the similarity calculation result between the target detection box corresponding to the thread and the i-th detection box.

[0111] The redundancy detection module 604 is used to determine the redundancy detection result of the target detection box corresponding to the thread based on the similarity calculation result of each thread in the thread group;

[0112] The loop judgment module 605 is used to increment i by 1 when there is a written detection box that has not been similarly calculated;

[0113] The detection box filtering module 606 is used to determine the detection box filtering result of the detection box set based on the redundant detection results of each thread when there is no detection box to be written in the detection box set.

[0114] In some embodiments, the first frame writing module 601 includes:

[0115] The first write unit is used to send a write request for the first detection frame to the memory access unit so that the memory access unit stores the first detection frame in the first storage area. The memory access unit is used to write the detection frame into the first storage area or the second storage area.

[0116] The intermediate frame writing module 602 includes:

[0117] The second write unit is used to send a write request for the (i+1)th detection frame to the memory access unit, so that the memory access unit stores the (i+1)th detection frame in the target storage area; the target storage area is a storage area other than the storage area where the i-th detection frame is written.

[0118] In some embodiments, the apparatus further includes:

[0119] The redundancy marking module is used to mark the target detection box corresponding to the thread as redundant after determining the redundant detection result of the target detection box corresponding to the thread based on the similarity calculation result.

[0120] In some embodiments, the similarity calculation module 603 includes:

[0121] A redundancy detection unit is used to perform similarity calculation between the target detection box corresponding to the thread in the detection box set and the i-th detection box when the i-th detection box is detected to have been written and there is no redundant mark in the i-th detection box.

[0122] A redundant cyclic unit is used to increment i by 1 when it is detected that the i-th detection box has been written and that there is a redundant mark in the i-th detection box.

[0123] In some embodiments, the detection box filtering module 606 includes:

[0124] The redundancy count unit is used to count the first number of redundant detection boxes based on the redundancy detection results of each thread.

[0125] The redundancy ratio calculation unit is used to calculate the ratio between the first quantity and the second quantity of detection boxes included in the detection box set;

[0126] The non-redundant extraction and sorting unit is used to add detection boxes whose redundancy detection results are non-redundant to a new set when the quantity ratio meets the majority redundancy condition, and sort each detection box in the new set to obtain the first sorting result, which serves as the detection box filtering result of the detection box set.

[0127] The set sorting unit is used to sort the detection boxes in the detection box set according to the redundant detection results of each detection box when the quantity ratio meets the condition of a few redundancies, and obtain a second sorting result.

[0128] Extraction quantity calculation unit, used to calculate the quantity difference between the first quantity and the second quantity;

[0129] The sorting result extraction unit is used to extract the sorting result of the detection boxes with quantity differences in the second sorting result, and use it as the detection box filtering result of the detection box set.

[0130] In some embodiments, the thread group includes: an image processor thread.

[0131] In some embodiments, the apparatus further includes:

[0132] The multi-level bounding box acquisition module is used to select the first detection box from the acquired detection box set and, before requesting to write the first detection box, acquire at least one level of detection boxes detected in the same image, as well as the confidence of each detection box.

[0133] The hierarchical box coarse screening module is used to filter the detection boxes at each level based on the confidence level of the detection boxes at each level, and obtain the screening results at each level.

[0134] The hierarchical box fusion and sorting module is used to sort the remaining detection boxes in the filtering results of each level according to the confidence level of each detection box, so as to obtain a set of detection boxes.

[0135] Each module in the aforementioned detection frame screening device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0136] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to the detection box screening method. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the aforementioned detection box screening method.

[0137] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it performs the steps in the detection box screening method described above. The display unit of the computer device forms a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen; the input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs or touchpads set on the casing of the computer device, or external keyboards, touchpads or mice, etc.

[0138] Those skilled in the art will understand that Figure 7 or Figure 8The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0139] In some embodiments, a computer device is provided, the computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0140] In some embodiments, such as Figure 9 The diagram shows the internal structure of a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above-described method embodiments.

[0141] In some embodiments, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0143] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for screening detection frames, characterized in that, include: Select the first detection box from the acquired detection box set using any thread in the preset thread group, and request to write the first detection box. In the i-th similarity calculation round, when there is a detection box to be written in the detection box set, any thread in the thread group selects the (i+1)-th detection box from the detection box set and requests to write the (i+1)-th detection box; where i is an integer; After the i-th detection box is detected to be written, each thread in the thread group performs a similarity calculation on the target detection box corresponding to the thread in the detection box set and the i-th detection box to obtain the similarity calculation result between the target detection box corresponding to the thread and the i-th detection box. Based on the similarity calculation results, the redundant detection results of the target detection boxes corresponding to each thread in the thread group are determined. If there are already written detection boxes that have not been similarized, increase i by 1; When there is no detection box to be written in the detection box set, the detection box filtering result of the detection box set is determined according to the redundant detection results corresponding to each thread.

2. The method according to claim 1, characterized in that, The request is written to the first detection box, including: Send a write request for the first detection frame to the memory access unit so that the memory access unit stores the first detection frame in the first storage area. The memory access unit is used to write the detection frame to the first storage area or the second storage area. The request to be written into the (i+1)th detection box includes: A write request for the (i+1)th detection frame is sent to the memory access unit so that the memory access unit stores the (i+1)th detection frame in the target storage area; the target storage area is a storage area other than the storage area where the i-th detection frame is written.

3. The method according to claim 1, characterized in that, After determining the redundancy detection result of the target detection box corresponding to the thread based on the similarity calculation result, the method further includes: When it is determined that the redundant detection result of the target detection box corresponding to the thread is redundant, the target detection box corresponding to the thread is marked as redundant.

4. The method according to claim 3, characterized in that, After the i-th detection box is detected as having been written, the similarity calculation is performed between the target detection box corresponding to the thread in the detection box set and the i-th detection box, including: When it is detected that the i-th detection box has been written and there is no redundant marker in the i-th detection box, the target detection box corresponding to the thread in the detection box set is compared with the i-th detection box for similarity calculation. When it is detected that the i-th detection box has been written and there is a redundant marker in the i-th detection box, i is increased by 1.

5. The method according to claim 3, characterized in that, The step of determining the detection box filtering result of the detection box set based on the redundancy detection results corresponding to each thread includes: Based on the redundancy detection results corresponding to each thread, count the first number of detection boxes that are redundant. Calculate the ratio between the first quantity and the second quantity of detection frames included in the detection frame set; When the ratio of the number of detection boxes satisfies the condition of majority redundancy, the detection boxes whose redundancy detection results are non-redundant are added to a new set, and the detection boxes in the new set are sorted to obtain a first sorting result, which is used as the detection box filtering result of the detection box set. When the quantity ratio satisfies the condition of a few redundancies, the detection boxes are sorted in the set of detection boxes according to the redundancy detection results of each detection box to obtain a second sorting result; Calculate the quantity difference between the first quantity and the second quantity; The sorting result of the detection boxes with the quantity difference is extracted from the second sorting result and used as the detection box filtering result of the detection box set.

6. The method according to claim 1, characterized in that, The thread group includes: the image processor thread.

7. The method according to claim 1, characterized in that, Before selecting the first detection box from the acquired detection box set through any thread in the preset thread group and requesting to write the first detection box, the method further includes: Obtain at least one level of detection bounding boxes detected in the same image, and the confidence score of each detection bounding box; For each level of detection boxes, the detection boxes corresponding to the level are filtered based on the confidence level of the detection boxes corresponding to the level, and the filtering results of the level are obtained. The detection boxes retained in the filtering results of each level are sorted according to the confidence level of each detection box to obtain the detection box set.

8. A detection frame screening device, characterized in that, include: The first detection box writing module is used to select the first detection box from the acquired detection box set by any thread in the preset thread group and request to write the first detection box; The intermediate frame writing module is used to select the (i+1)th detection frame from the detection frame set and request to write the (i+1)th detection frame when there is a detection frame to be written in the detection frame set during the i-th similarity calculation round; where i is an integer. The similarity calculation module is used to perform similarity calculation between the target detection box corresponding to the thread in the detection box set and the i-th detection box after the i-th detection box is detected to be written, through each thread in the thread group, to obtain the similarity calculation result between the target detection box corresponding to the thread and the i-th detection box. The redundancy detection module is used to determine the redundancy detection result of the target detection box corresponding to the thread based on the similarity calculation result through each thread in the thread group; The loop judgment module is used to increment i by 1 if there is a written detection box that has not been similarly calculated; The detection box filtering module is used to determine the detection box filtering result of the detection box set based on the redundant detection results corresponding to each thread when there is no detection box to be written in the detection box set.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the detection box screening method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the detection box screening method according to any one of claims 1-7.

Images