Apparatus and method for processing a detection frame

Abstract

A mechanism for performing non-maximum suppression (NMS) on a plurality of detection boxes identifying potential locations of one or more objects within an image is presented. The mechanism presents a tiling system that divides the image into a plurality of tiles. A per-tile suppression process is performed in which at least some of the detection boxes that overlap a particular tile are processed to determine whether to discard any of the detection boxes.

Description

Technical Field

[0001] This invention relates to apparatus and methods for processing detection frames. Background Technology

[0002] Object detection typically involves processing an image and generating bounding boxes, occasionally labeled as "bounding boxes," that outline the potential location of a target object (e.g., a person, face, tree, etc.). Some object detection processes, such as single-shot detection, may result in the generation of multiple bounding boxes for a single object.

[0003] Therefore, a suppression process is needed to reduce the number of detection boxes that identify the same physical object. One widely used method is Non-maximum Suppression (NMS).

[0004] NMS operates by processing each detection box using an IoU (Intersection over Union) process. The IoU process involves sequentially iteratively calculating the IoU value between the currently processed bounding box and each other detection box. If the IoU value between two boxes exceeds a predetermined threshold, the detection box associated with its minimum confidence in identifying the target object is discarded. Once a detection box is discarded, the IoU process for that box does not need to continue / be executed.

[0005] To address potential ambiguity, such as multiple bounding boxes overlapping, some NMS methods sort the bounding boxes by confidence level before performing the IoU process in sorted order. Therefore, the most confident bounding box is compared with all other lower-confidence boxes, then the second most confident (assuming it hasn't been discarded) is compared with all other lower-confidence boxes, and so on.

[0006] The goal has always been to reduce the complexity of the suppression process in order to reduce the number of detection boxes. Summary of the Invention

[0007] This summary is provided to introduce, in a simplified form, a series of concepts further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0008] This disclosure proposes a mechanism for performing non-maximum suppression (NMS) on multiple detection boxes that identify potential locations of one or more objects within an image. The mechanism proposes a tiling system that divides the image into multiple tiles. A tile-by-tile suppression process is performed, wherein at least some detection boxes overlapping with a particular tile are processed to determine whether any detection boxes should be discarded.

[0009] According to one aspect, a computer-implemented method is provided for performing non-maximum suppression (NMS) on multiple detection boxes that identify potential locations of one or more objects within an image.

[0010] The computer-implemented method includes: dividing the image into multiple patches; for each patch, identifying which detection boxes overlap with a portion of the patch if any overlap exists; and performing a patch-by-patch suppression process, the suppression process including, for each patch having detection boxes that overlap with each other by more than a predetermined amount: comparing the confidence values ​​of the detection boxes that overlap with each other by more than a predetermined amount, the confidence values ​​indicating the confidence that the detection boxes identify objects, and for each comparison, marking the detection box associated with the lowest confidence value for discarding.

[0011] The block-by-block suppression process may include, for each block, iteratively performing a thresholding process, the thresholding process comprising: selecting a first detection box that overlaps with a portion of the block, wherein the first detection box has not yet been marked for discard; selecting a second, distinct detection box that overlaps with a portion of the block for comparison with the first detection box, wherein the second detection box has not yet been marked for discard, and wherein the first and second detection boxes have not previously been compared to each other during the thresholding process for the block; determining a relative overlap between the selected first and second detection boxes; and, in response to the determined relative overlap exceeding a predetermined threshold: marking the first detection box for discard if the confidence value of the first detection box is less than the confidence value of the second detection box; and / or marking the second detection box for discard if the confidence value of the second detection box is less than the confidence value of the first detection box, wherein the thresholding process for the block is repeated until all detection boxes of the block have been compared with every other detection box of the block or have been marked for discard.

[0012] The step of determining the relative overlap between the selected first and second detection boxes may include: determining the total area of ​​the image occupied by the union of the first and second detection boxes; determining the total area of ​​the image occupied by the intersection of the first and second detection boxes; and calculating the relative overlap by dividing the total area of ​​the image occupied by the intersection of the first and second detection boxes by the total area of ​​the image occupied by the union of the first and second detection boxes.

[0013] In some implementations, the method further includes a step of sorting the detection boxes in order of confidence, wherein: the step of selecting a first detection box includes selecting the most confident detection box, which was not marked for discard during the thresholding process for the tile and has not previously been compared with all other detection boxes that were not marked for discard; and the step of selecting a second detection box includes selecting a detection box that was not marked for discard and has not previously been compared with the first detection box (during the thresholding process for the tile). The selected second detection box can be, for example, any selection of detection boxes that meet the criteria, or the (next) most confident detection box that meets the criteria, i.e., the most confident detection box other than the already selected first detection box.

[0014] In some implementations, the method includes sorting the detection boxes according to confidence levels, which is performed before the patch-by-patch suppression process.

[0015] The patch-by-patch suppression process may include, for each patch, sorting the detection boxes that overlap with a portion of the patch in order of confidence.

[0016] The size of each tile preferably depends on an estimated ratio between the detection box size and the image resolution. This can be calculated, for example, based on historical and / or example images and their corresponding detection boxes. In other embodiments, this ratio is provided by user input or a database / dataset.

[0017] The size of each tile can depend on the estimated number of objects in the image. The size of each tile can also depend, or otherwise depends on the estimated detection box size and the image resolution.

[0018] The method may include: identifying detection boxes associated with confidence values ​​below a predetermined confidence threshold; and marking detection boxes with confidence values ​​below the predetermined confidence threshold for discarding.

[0019] In some implementations, for each tile, if any detection boxes overlap with a portion of the tile, the step of identifying which detection boxes overlap with that portion of the tile does not identify detection boxes that have already been marked for discard. In other words, when determining which detection boxes overlap with a tile, it is preferable not to consider detection boxes that have already been marked for discard.

[0020] The steps of performing the block-by-block suppression process can be executed using parallel processing techniques.

[0021] Using bounding box data for each tile, if any bounding box overlaps with a portion of the tile, the step of identifying which bounding boxes overlap with a portion of the tile may include, for each tile, generating a list of tiles that identify which bounding boxes overlap with a portion of the tile.

[0022] In some implementations, the method may include creating an output list of detection boxes that identify detection boxes that were never marked for discard during the tile-by-tile suppression process.

[0023] The method can be adapted such that the step of labeling detection boxes associated with the lowest confidence value for discarding includes modifying the flags associated with the detection boxes; and the step of building an output list of detection boxes includes processing the flags to identify detection boxes that were never labeled for discarding during the non-maximum suppression method.

[0024] The method can be adapted such that the step of establishing the output list includes initializing a global list identifying all detection boxes before performing the tile-by-tile suppression process, and the step of marking detection boxes associated with the lowest confidence value for discarding includes modifying the markers associated with the detection boxes in the global list for marking, such that when the tile-by-tile suppression process is complete, the global list facilitates the provision of an output list identifying detection boxes that were never marked for discarding.

[0025] The method may include the step of obtaining detection box data, the detection box data identifying the position of the detection box within the image for each of the plurality of detection boxes and a confidence value indicating the confidence of the object identified by the detection box, wherein for each patch, the step of identifying which detection boxes overlap with a portion of the patch includes processing the detection box data for each patch, identifying which detection boxes overlap with a portion of the patch if any overlap is found.

[0026] Some implementations provide a computer-implemented method for performing non-maximum suppression (NMS) on multiple detection boxes representing potential locations of one or more objects of different categories within an image, the computer-implemented method comprising: classifying the multiple detection boxes into different categories, each category of detection boxes being associated with an object of a different category; and performing any of the NMS methods described herein on each category of detection boxes.

[0027] In some examples, the step of classifying the plurality of detection boxes into different categories is performed by obtaining classification data and processing the classification data to classify the plurality of detection boxes into different categories, wherein the classification data identifies the category of the object at the possible location of the detection box for each of the plurality of detection boxes.

[0028] According to another aspect, a graphics processing system configured to perform any of the methods described herein is provided.

[0029] The graphics processing system may include: a neural network accelerator configured to process an image to generate multiple detection boxes that identify potential locations of one or more objects within the image; and a processing module configured to obtain the multiple detection boxes from the neural network accelerator and perform any of the methods described herein.

[0030] Any graphics processing system described can be embodied in hardware on an integrated circuit.

[0031] A method may be provided for manufacturing a graphics processing system as described herein at or using an integrated circuit manufacturing system. One manufacturing method may include using a layout processing system to process a computer-readable description of the graphics processing system to generate a circuit layout description of an integrated circuit embodying the graphics processing system; and using an integrated circuit generation system to manufacture the graphics processing system based on the circuit layout description.

[0032] An integrated circuit definition dataset may be provided, which, when processed in an integrated circuit manufacturing system, configures the system to manufacture the graphics processing system described herein. A non-transitory computer-readable storage medium may be provided, on which a computer-readable description of the graphics processing system is stored, which, when processed in the integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the graphics processing system.

[0033] An integrated circuit manufacturing system may be provided, comprising: a non-transitory computer-readable storage medium storing a computer-readable description of a graphics processing system thereon; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the graphics processing system; and an integrated circuit generation system configured to manufacture the graphics processing system according to the circuit layout description.

[0034] Computer program code or computer-readable code for performing any of the methods described herein may be provided. A computer-readable storage medium having a computer program / readable code encoded thereon may be provided.

[0035] A (non-transitory) computer-readable storage medium may be provided having computer-readable instructions stored thereon, which, when executed at a computer system, cause the computer system to perform any of the methods described herein.

[0036] As will be apparent to those skilled in the art, the above features can be appropriately combined, and can be combined with any aspect of the examples described herein. Attached Figure Description

[0037] The example will now be described in detail with reference to the accompanying drawings, in which: Figure 1 and Figure 2 An exemplary image conceptually illustrates a method performed according to an implementation scheme; Figure 3 This is a flowchart illustrating the method according to the implementation scheme; Figure 4 This is a flowchart illustrating one aspect of the method; Figure 5 This is a flowchart illustrating another aspect of the method; Figure 6 This illustrates the relationship between tile size and the time spent executing the method according to the implementation plan; Figure 7 and 8 The relationship between various parameters of the method according to the implementation scheme and the recommended intermediate parameter R is shown; Figure 9 A computer system is shown in which a graphics processing system for executing the method is implemented; and Figure 10 This illustrates a manufacturing system for generating integrated circuits that embody a graphics processing system.

[0038] The accompanying drawings illustrate various examples. Those skilled in the art will understand that the element boundaries (e.g., boxes, groups of boxes, or other shapes) shown in the drawings represent one example of a boundary. In some examples, it may be that one element can be designed as multiple elements, or multiple elements can be designed as one element. Where appropriate, common reference numerals are used throughout the drawings to indicate similar features. Detailed Implementation

[0039] The following description is given by way of example to enable those skilled in the art to make and use the invention. The invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be readily apparent to those skilled in the art. Embodiments are now described by way of example only.

[0040] Figure 1 and Figure 2 Used to conceptually understand the non-maximum suppression process according to the implementation plan.

[0041] Figure 1This is a simplified illustration of image 100, which has undergone an object detection process to identify multiple detection boxes 101-117, but before a non-maximum suppression process has been performed on the detection boxes.

[0042] Each detection box 101-117 identifies the potential location of one or more (target) objects (not shown) within image 100. For the illustrated embodiment, each detection box identifies the potential location of (target) objects of the same category (e.g., faces, people, trees, etc.).

[0043] Each bounding box is associated with a confidence value, which indicates the confidence level that the bounding box has accurately detected the location of an image. This information is a standard output of the object detection process. Confidence values ​​are typically in the range of 0 (least confident) to 1 (most confident), but other ranges are possible, such as 0 to 100.

[0044] Information about the detection boxes can be included in the detection box data. The detection box data may indicate the relative position of the detection boxes within the image and a confidence value indicating how confident the detection boxes are in identifying objects. Those skilled in the art will understand that the detection box data is a standard output of the object detection process and does not require detailed description.

[0045] The relative position or localization of a detection box within an image can be provided using coordinates (e.g., indicating the position of the vertices of the detection box) or a combination of coordinates and size information (e.g., coordinates indicating the relative position of the top-left vertex of the detection box in the image and size information indicating the height and width of the detection box). Other methods for representing the relative position or localization of a detection box within an image will be apparent to those skilled in the art.

[0046] Figure 2 This is a simplified illustration of image 100 after a non-maximum suppression process has been performed on the detection boxes. The non-maximum suppression process aims to remove or discard multiple detection boxes, with the goal of deduplicating detection boxes at potential locations of the same instance of an object.

[0047] A typical non-maximum suppression process involves comparing the overlap between different detection boxes and deleting detection boxes whose locations are less likely to be identified when the overlap exceeds a predetermined amount. This process is typically performed iteratively, preferably starting with the most likely detection box and moving to the least likely one, until all detection boxes have been compared with every other detection box or have been discarded.

[0048] This disclosure relates to alternative methods for performing non-maximum suppression. In particular, methods for performing non-maximum suppression use a block-by-block suppression mechanism.

[0049] Figure 1 and Figure 2 Conceptual diagrams are shown for dividing an image into multiple patches 151-156. Each detection box 101-117 overlaps with at least one of these patches.

[0050] For each tile, identify the bounding boxes that overlap with at least a portion of that tile. It will be apparent that bounding boxes can be identified for multiple different tiles.

[0051] Then, a tile-by-tile suppression process is performed. The tile-by-tile process involves processing each tile in sequence: for comparing the overlap between different detection boxes (within the tile) and marking for discarding / deleting, detection boxes that overlap with another detection box by more than a predetermined amount and are less reliable (in terms of identifying the target object) than the overlapping detection box.

[0052] As a result of the tile-by-tile suppression process, each tile can be processed only once. This allows for non-maximum suppression across multiple distinct detection boxes, while processing each tile only once. One advantage of the tile-by-tile suppression process (such as the suppression process described herein) is its ability to achieve two forms of parallelism across the entire NMS. Each tile can be processed in parallel to provide one form of parallelism. Furthermore, when processing any given tile, comparisons between detection boxes within that tile can be performed in parallel to provide a second form of parallelism.

[0053] In other words, parallelization can be achieved by performing comparisons across multiple tiles at once and / or performing multiple comparisons within each tile.

[0054] In some preferred embodiments, during processing of a particular tile, all detection boxes that overlap with at least a portion of the tile are compared with each other (i.e., the overlap between different pairs of detection boxes is compared), unless the detection boxes have previously been marked as to be discarded during (earlier) processing of the particular tile.

[0055] Preferably, comparisons between bounding boxes within a tile are performed in order of confidence level. Therefore, the most confident bounding box can be compared with all other bounding boxes with lower confidence levels in that tile, then the next most confident bounding box is compared with all other bounding boxes with lower confidence levels, and so on. If a bounding box is marked for discard, the comparison process does not need to be performed (or continued) for the bounding boxes marked for deletion.

[0056] Take patch 151 as an example. Three detection boxes 101-103 overlap with at least a portion of this patch. In the illustrated case, the detection boxes are ordered by confidence level as follows: 102; 101; 103. Starting with the most confident, detection box 102 is compared with detection box 101. The overlap between detection boxes 102 and 101 exceeds a predetermined threshold, so the less confident detection box 101 is marked for deletion. Then, detection box 102 is compared with detection box 103. The overlap between detection boxes 102 and 103 also exceeds a predetermined threshold, so the less confident detection box 103 is marked for deletion.

[0057] Therefore, in the example described for tile 151, only a single detection box is left unmarked for deletion.

[0058] After the tile-by-tile suppression process has been performed, detection boxes 102, 106, 108, 112, 115, and 117 that were never marked for deletion / discard (during the tile-by-tile suppression process) can be identified. An output list of these detection boxes can then be generated. Figure 2 The image shows the detection boxes in the output list.

[0059] By identifying only those detection boxes that were never marked for deletion, this avoids retaining detection boxes that would otherwise be discarded in conventional NMS methods in the NMS mechanism proposed here.

[0060] Returning to example tile 151, we can see that the only remaining detection box (which previously overlapped with tile 151) is detection box 102. This is because during the tile-by-tile suppression process, detection boxes 101 and 103 were marked for discarding / removal.

[0061] In another example, block 153, the detection box is not retained. This is because the only detection box 113 overlapping with block 153 is deleted because it overlaps with another more reliable detection box (e.g., detection box 115) by more than a predetermined amount. Therefore, detection box 113 is marked for deletion during the processing of block 156.

[0062] Therefore, more than one tile overlapped with a detection box can be marked for deletion during any tile-specific process. As an example, detection box 113 may not be marked for deletion when processing tile 153, but may be marked for deletion when processing tile 156.

[0063] In other tiles, more than one detection box is retained, such as detection boxes 115 and 117 in tile 156. This is because the detection boxes overlap each other by no more than a predetermined amount (meaning that when the detection boxes are compared, none of them are marked for deletion).

[0064] The overlap between two detection boxes is preferably calculated using the intersection-union method. This involves calculating the area occupied by the intersection of the two detection boxes and dividing it by the area occupied by the union of the two detection boxes. This allows for easy calculation of the relative overlap between the two detection boxes and effectively standardizes the relative overlap.

[0065] This disclosure recognizes that the number of comparisons made between different detection boxes can be reduced (e.g., to identify overlaps between bounding boxes) by limiting comparisons to detection boxes that are local to each other. This disclosure proposes using a tiling method to efficiently identify which detection boxes are close to each other, i.e., grouping the detection boxes.

[0066] Thus, the complexity of the proposed NMS process differs from that of existing NMS processes. In many cases and / or use cases, this can enable a significant reduction in the number of processing steps performed when executing an NMS process.

[0067] The processing described in the tile-by-tile procedure also enables the parallelization of the suppression process. This helps to further improve the speed of executing the NMS process. As mentioned earlier, the use of the tile-by-tile suppression process facilitates two forms of parallelization.

[0068] The step of using detection box data to identify which detection boxes (if any) overlap with a portion of the tile can be performed before the tile-by-tile suppression process.

[0069] In other examples, this step is integrated into the tile-by-tile suppression process. Therefore, the tile-by-tile suppression process may include, for each tile identifier, comparing any detection boxes that overlap with a portion of the tile, comparing detection boxes that overlap with each other by more than a predetermined amount, and for the compared detection boxes, marking the detection box associated with the lowest confidence value for discarding.

[0070] A basic understanding of the fundamental mechanisms has now been described, through reference Figures 3 to 7 To gain a more complete understanding of the process of performing non-maximum suppression, these figures provide flowcharts and graphs illustrating various methods (and parts thereof) according to the implementation scheme.

[0071] Figure 3 This is a flowchart illustrating a computer-implemented method 300 according to an embodiment. Method 300 is designed to perform non-maximum suppression (NMS) on multiple detection boxes that identify potential locations of one or more objects within an image.

[0072] Method 300 may include step 310 of obtaining detection box data. In other instances, the detection box data may have already been obtained before performing method 300. As previously described, the detection box data identifies the relative position of each of the plurality of detection boxes within the image and a confidence value indicating the confidence level of the detection box in identifying an object. The confidence value can be of any scale, but is preferably in the range of 0 to 1.

[0073] Method 300 then proceeds to step 320, which involves dividing the image into multiple tiles. Those skilled in the art will understand that a tile is a different region, block, or portion of the image. The tiles are preferably rectangular. Suitable methods for selecting or controlling the number and / or size of the tiles will be described later.

[0074] Method 300 then moves to step 330, which detection boxes (if any) overlap with a portion of the tile for each tile. In other words, step 330 includes identifying any detection boxes that overlap with a portion of the tile for each tile. This can be performed by processing the detection box data obtained in step 310. Therefore, step 330 identifies which tile(s) a detection box overlaps with for each detection box. Step 330 may include generating an array or list for each tile that identifies which detection boxes overlap with a portion of that tile—i.e., a “tile list.”

[0075] Method 300 then performs a block-by-block suppression process 340.

[0076] Process 340 includes step 341, where, for each patch, confidence values ​​are compared between detection boxes that overlap by more than a predetermined amount. This overlap is determined relative to the entire image, not just to the patches.

[0077] Process 340 also includes step 342, whereby for each comparison, the detection box associated with the lowest confidence value is marked for discarding.

[0078] Steps 341 and 342 can utilize, for example, the tile list generated in step 330 to make it possible to identify which detection boxes are contained in the tiles.

[0079] To mark detection boxes for discarding, some embodiments of steps 341 and 342 may use a global list of flags—a “global flag array” or a “global list”—each entry in the global flag array or global list identifies a detection box for the entire image. This global flag array may be generated before performing the patch-by-patch suppression process 340 (or initialized in the first iteration of that process). The global flag array may be modified appropriately if the detection boxes will be marked for discarding (e.g., a value of 0 indicates that the detection box will be discarded, 1 indicates that the detection box will not be discarded, and vice versa).

[0080] One advantage of using a global flag array is that, for subsequent processing of tiles, detection boxes already marked for deletion can be identified and skipped / bypassed. This can help improve the efficiency of the proposed mechanism.

[0081] In some implementations, a flag array can be generated for each tile, where each element of the flag array corresponds to a corresponding detection box in that tile – the “tile flag array”. If a detection box is to be marked for discarding, the flag array can be appropriately modified, similar to a global flag array.

[0082] One benefit of using a tile marker array is that it avoids race conditions when accessing the tile marker array, i.e., during the processing of a specific tile. This can facilitate less complex parallelization of the tile-by-tile suppression process.

[0083] In some implementations, the functionality of the "tile list" and the "tile marker array" can be combined into a single array or list. For example, each entry in the "tile list" can identify different boxes contained within a tile. When a detection box is marked for discard, the value of the entry in the tile list corresponding to that detection box can be replaced with an invalid pointer / value (or a known "empty" pointer / value) so that the tile list no longer identifies that detection box as part of forming a tile.

[0084] Steps 341 and 342 can be repeated in a tile until all detection boxes have been compared to each other and / or marked for discarding.

[0085] Steps 341 and 342 are repeated for each tile, as illustrated by decision step 343 and next tile selection step 344. A more specific implementation of steps 341 and 342 (i.e., the tile-by-tile suppression process) will be described later.

[0086] After the tile-by-tile suppression process is completed, i.e., after all tiles have been processed, method 300 may execute step 350 of establishing an output list of detection boxes. The output list of detection boxes identifies detection boxes that were never marked for discard during the non-maximum suppression method.

[0087] Method 300 may thereby generate an output list of detection boxes that indicate the detection boxes that have passed the NMS process and are considered to represent the actual locations of target objects within the image.

[0088] In the presence of a global flag array, creating an output list may include processing the global flag array to identify detection boxes that have not yet been marked for discarding.

[0089] In the presence of a tile marker array (or any other array that performs the same function as a tile marker array), creating an output list may include processing the tile marker array to identify which detection boxes are not marked for any tile marker array to discard.

[0090] By performing a block-by-block suppression process, the number of other bounding boxes that a single bounding box must compare against is reduced. In many cases, this can significantly reduce the complexity of the NMS process.

[0091] Method 300 may be adapted to further include step 360, namely, sorting the detection boxes according to confidence level. This may be useful for easily identifying the order of detection boxes within a processed tile (for use during the tile-by-tile suppression process).

[0092] This step 360 may be performed before step 330 (as illustrated), but may otherwise be implemented in the block-by-block suppression process (e.g., performed on the block-by-block process before step 341). Those skilled in the art will understand that step 360 may be performed before, after, or simultaneously with step 320.

[0093] Preferably, step 360 includes sorting the detection boxes using a sorting index. This avoids the need for moving and sequentially storing data structures (a resource-intensive process). The sorting index may include, for example, an array or vector of entries, where each entry corresponds to a specific confidence rank. Each entry in the sorting index identifies a distinct detection box (e.g., using a value or pointer), where the order of the entries in the sorting index indicates the rank of the detection boxes (e.g., the first entry is the most confident, and the last entry is the least confident).

[0094] The sorting index can be a "global sorting index", which identifies the confidence ranking of all detection boxes in the entire image.

[0095] In some implementations, a global sort index and a global flag array are combined.

[0096] Therefore, the global list can include a sorted index. The value corresponding to the entry of the detection box to be marked for discard can be set to a predetermined value (e.g., -1) to indicate that the previously associated detection box has been marked for discard.

[0097] In one implementation, in response to the step of marking the corresponding detection box for discarding (e.g., replacing the value / pointer of the detection box within the sort index with an invalid value / pointer such as -1), the value in the global sort index is modified. This avoids the need to move the detection box data and / or elements in the sort index (during the discarding process). In this way, the global sort index can effectively serve as an output list.

[0098] Of course, after the tile-by-tile suppression process is complete, the global sort index may need to be further processed to build a clean output list, for example by copying the remaining valid values ​​of the global sort index (i.e., values ​​that identify boxes that have not yet been marked for discard) into a new output array, or by simply removing entries in the sort index that have already been marked for discard (e.g., entries that save invalid values).

[0099] In other implementations, multiple sorting indexes can be generated, one for each tile, known as a "tile sorting index".

[0100] The “tile list” and “tile sorting index” can be combined into a single array or list.

[0101] Specifically, a tile sorting index can also be used to maintain a list of detection boxes within each tile. The tile sorting index can be a regular sorting index, where each entry identifies a different detection box within the tile, and the order of the entries indicates the order of the detection boxes' confidence scores. Therefore, the tile sorting index can rank each detection box contained within a tile according to its confidence score.

[0102] In a preferred example, the sorting steps 360 and 330 of the detection boxes are performed simultaneously. Therefore, each detection box can be processed to identify which tile(s) it overlaps with and its position / rank (of confidence value) relative to other detection boxes (e.g., within a tile). In other words, a tile sorting index can be generated for each tile, indicating the order (of confidence value) of the detection boxes within the tile (e.g., by containing only / referencing related detection boxes) and the order of confidence values ​​of the detection boxes within the tile.

[0103] In other examples, step 360, which sorts the detection boxes, is executed after step 330, for example, within a tile-by-tile suppression process. Therefore, a tile sorting index can be generated when processing tiles.

[0104] Furthermore, the block sorting index can be used to mark whether a detection box has been marked for discard for that block. In other words, the functionality of the "block sorting index" and the "block flag array" can be performed by a single array or list. This can be done by setting the value corresponding to the detection box to be discarded in the block sorting index to an invalid / negative number.

[0105] It will also be clear that the tile sorting index can act as both an array of bounding boxes and a label array identifying tiles, while also providing appropriate confidence ranking information. In other words, the tile sorting index can perform the functions of both a tile list and a tile label array.

[0106] Combining "tile lists," "tile sorting indexes," and / or "tile tag arrays" makes memory usage more efficient. However, it is possible for one or more of these lists / indexes / arrays to remain independent of the others, for example, to increase the ease of vulnerability patching or for distributed storage of information.

[0107] Clearly, as mentioned earlier, a global sort index and / or a global flag array can be used. The global sort index and global flag array can be combined in a similar way to the tile sort index and tile flag array, for example, by discarding values ​​corresponding to specific detection boxes (in the global sort index) by setting them to negative / invalid numbers / values. Combining them in this way provides similar memory advantages.

[0108] To further improve the efficiency of the NMS method at the expense of potential accuracy, method 300 may further include a step (not shown) of marking to discard a number of the least reliable detection boxes. Therefore, in an embodiment, only the most reliable (e.g., 100 or 200 most reliable) detection boxes are processed in step 330.

[0109] In step 330, the decision on whether to process the boundaries of the detection boxes can be based on a predetermined number of detection boxes (e.g., only the first 200, i.e., the most reliable detection boxes) or a predetermined percentage of detection boxes (e.g., only the first 40% or the first 50%).

[0110] This process is particularly advantageous if sorting step 360 is performed, as it makes it easy to identify the most reliable detection boxes. Preferably, this process is performed simultaneously with the sorting operation.

[0111] Step 341 may include comparing detection boxes in order of confidence level. In other words, step 341 may include starting with the most confident detection box in the tile and identifying other detection boxes that overlap with the detection box by more than a predetermined amount before proceeding to the next most confident detection box, etc.

[0112] To further improve the efficiency of method 300, an additional step 370 of confidence thresholding can be performed. Step 370 can be performed before or after step 320, but is preferably performed before step 360 (if present) to improve efficiency.

[0113] Step 370 includes marking any detection boxes associated with confidence values ​​below a predetermined threshold or "confidence value threshold" for discarding. Therefore, low-confidence detection boxes can be marked for discarding, thereby reducing the number of detection boxes that need to be processed.

[0114] For a predetermined threshold with confidence values ​​in the range of 0 to 1, an appropriate value is in the range of 0.1 to 0.9, for example, 0.7 or 0.9. A predetermined threshold in the range of 0.6 to 0.7 would be a suitable compromise. Exemplary appropriate values ​​of the predetermined thresholds provided herein can be varied based on the maximum possible range of confidence values ​​(e.g., a threshold referred to as 0.1 for the range of 0-1 could alternatively be a threshold of 10 for the range of 0-100).

[0115] Those skilled in the art will understand that appropriate confidence values ​​can vary depending on the use case and / or implementation details. For example, confidence values ​​can vary depending on the quality of the image (e.g., resolution, focus, distortion, noise, etc.) and / or the accuracy of the object detection process used to generate the detection boxes. A threshold between 0.6 and 0.8 corresponds to fairly good input image quality and resolution (e.g., a well-focused typical mobile phone image, no motion blur, good illumination) combined with current prior art object detection processes.

[0116] Furthermore, the appropriate threshold may depend on whether it is desirable to avoid false positives (e.g., for security analytics applications) and / or false negatives (e.g., for motor vehicle detection). False positives can be reduced by setting the threshold confidence value high (e.g., >0.7). False negatives can be reduced by setting the threshold confidence value low (e.g., <0.3, for example, <0.15).

[0117] You can also use a very low threshold (e.g., 0.1) to remove any unreliable detections (e.g., detections generated by noise), which can result in a fairly short list of detections to be processed further.

[0118] To reduce the processing performed by this method, step 370 is preferably executed as early as possible, for example, immediately after the detection box data is obtained. This helps reduce processing time by discarding irrelevant detection boxes as early as possible.

[0119] The disclosed method 300 can be parallelized. Specifically, processing 340 can be parallelized (e.g., using different threads to process different tiles). This provides specific and concrete advantages of the proposed method compared to conventional or typical non-maximum suppression methods.

[0120] For example, in step 350, parallelization may introduce some complexity in processing the results from all the tiles. Multiple implementations for performing step 350 can be envisioned.

[0121] In the first example, each thread produces a tile output list of unlabeled detection boxes for deletion on a specific single tile. The tile output list can be generated from a tile tag array (or an equivalent tile tag array). These results can be combined into a single output list that identifies the unlabeled detection boxes (e.g., by combining all tile output lists and removing duplicate entries).

[0122] The second example recognizes that only detection boxes need to be removed (never added), thus using a global flag array. When a detection box is to be removed, the flag can be cleared / modified by a thread writing to the flag. If multiple threads attempt to change the same flag simultaneously (i.e., remove the same detection), they will write the same flag value to the same memory location, thus the probability of any race conditions is low (at least at the algorithm level).

[0123] In some implementations, there may be more than one object category identified by detection boxes. That is, different groups of detection boxes can identify the potential locations of different categories of target objects (e.g., one group can identify a person's face, while different groups identify cats in the image).

[0124] In such an implementation, method 300 can be executed individually for each class of the target object. In other words, method 300 can be executed on first detection box data (associated with a single class of the target object) and / or second detection box data (associated with different classes of the target object).

[0125] Obtain classification data for each of the plurality of detection boxes, which identifies the category of the object at a possible location for each detection box; divide the plurality of detection boxes into different categories by processing the classification data, with each category of detection boxes associated with a different class of objects; and perform method 300 on each category of detection boxes.

[0126] Therefore, the detection box data can be effectively divided into multiple categories, and method 300 can be applied to each category.

[0127] When method 300 is executed for different categories, multiple instances of method 300 (one instance for each of the multiple categories) can be executed in parallel, for example, using different threads of the processing system. This further improves the efficiency of the entire NMS process.

[0128] When method 300 is performed for different categories, the method may include the step of segmenting or binning the detection box into different groups / sets of categories, i.e., performing a "category separation" step.

[0129] When performing such segmentation / binning steps, sorting and / or confidence thresholding steps can be performed simultaneously. This further improves the efficiency of the method, which is a simple two-level comparison function. Performing the class separation step and the sorting (by confidence) step simultaneously is functionally equivalent to performing a sorting step where the first priority is the class ID and the second priority is the confidence.

[0130] Figure 4 The block-by-block suppression process according to the implementation scheme is shown. Figure 3 The process (340) involves aspects 400. Specifically, Figure 4 An implementation scheme for steps 341 and 342 (the process performed for a single tile) is shown.

[0131] Those skilled in the art will understand that process 400 is merely one implementation and that process 400 will be readily adaptable to different versions of the disclosed concept.

[0132] Process 400 includes a step 410 of selecting a first detection box. The first detection box overlaps with a portion of a tile and is not marked for discard (within process 400 for that tile). Furthermore, it will be apparent that the selected first detection box will not be a detection box that has already been compared with all other detection boxes (not marked for discard).

[0133] Process 400 also includes a step 420 of selecting a second (different) detection box for comparison with the first detection box. The second detection box also overlaps with a portion of the tile and is not marked for discard (within process 400 for that tile). Furthermore, the second detection box will be a detection box that has not previously been compared with the first detection box (within process 400 for said tile).

[0134] Steps 410 and 420 can be performed sequentially (as shown) or in parallel.

[0135] Preferably, the detection boxes have been previously sorted in order of confidence, for example, based on the associated confidence value of each detection box.

[0136] In such an example, the first detection box can be the detection box with the highest confidence value that also satisfies the aforementioned criteria for the first detection box (i.e., it is not marked for discarding and is not compared with all other (not marked for discarding) detection boxes).

[0137] Similarly, the second detection box can be a detection box with the highest confidence value (next) that also meets the above criteria for the second detection box (i.e., one that was not marked for discard and has not previously been compared with the first detection box). This aspect is not required; for example, pseudo-random selection of detection boxes can be performed, or the least confident detection box can be selected.

[0138] The method then proceeds to step 430, which determines the relative overlap between the selected detection boxes. This may include determining the proportion of detection boxes that overlap with each other relative to the total area of ​​the image occupied by the detection boxes. A more complete example of step 430 will be provided later.

[0139] Then, in step 440, the method determines whether the (relative) overlap is greater than a predetermined amount, that is, greater than a predetermined threshold.

[0140] The relative overlap can range from 0 (no overlap) to 1 (complete overlap). In such an implementation, the predetermined threshold can be a value in the range of 0.4 to 0.9. Preferably, the predetermined threshold is not less than 0.5, such that most of the two detection frames need to overlap. This improves the accuracy of the method.

[0141] Those skilled in the art will understand that the relative overlap can have different possible ranges (e.g., from 0 to 100) and will be able to adapt the predetermined threshold accordingly. Thus, the predetermined threshold can be, for example, in the range of 0.4 to 0.9 times the maximum possible relative overlap value, and for example, not less than 0.5 times the maximum possible relative overlap value.

[0142] In response to the determined overlap exceeding a predetermined amount, in steps 450, 455, and 456, one of the first and second detection boxes is marked for discard. Otherwise, the method returns to step 410.

[0143] Specifically, step 450 includes determining whether the confidence value of the first detection box is less than the confidence value of the second detection box.

[0144] In response to the affirmative determination in step 450, process 400 marks the first detection box for discarding in step 455 because the confidence value of the first detection box is lower than the confidence value of the second detection box. In response to the negative determination in step 450, process 400 marks the second detection box for discarding in step 456 because the confidence value of the second detection box is lower than the confidence value of the first detection box.

[0145] If the confidence values ​​are equal, either the first or second detection box can be discarded. This can be done on a (pseudo)random basis. In another implementation, if the confidence values ​​are equal, one implementation may decide to always select the first / second detection box for discarding. In yet another implementation, if the detection boxes have different sizes, the larger detection box can be selected for discarding (because it can be assumed that the smaller detection box more accurately identifies the location of the target object).

[0146] In other words, steps 450, 455, and 456 include the following processes: in response to the determined relative overlap exceeding a predetermined threshold: if the confidence value of the first detection box is less than the confidence value of the second detection box, then the first detection box is marked for discarding; and / or if the confidence value of the second detection box is less than the confidence value of the first detection box, then the second detection box is marked for discarding.

[0147] If it is already known that the first detection box has a higher / lower confidence value than the second detection box (e.g., if the detection boxes have been sorted in order of confidence and the first and second detection boxes were selected based on the sorting order), steps 450 and 455 can be omitted. Instead, the method can simply move directly to step 456.

[0148] Repeat process 400 until all detection boxes of the tile have been compared with every other detection box of the tile, or have been marked for discard. This is shown in decision step 460.

[0149] Figure 5 An embodiment of step 430 of process 400 is shown.

[0150] In the illustrated embodiment, step 430 includes a sub-step 430A of determining the total image area occupied by the union of the first and second detection boxes (i.e., the combined region).

[0151] Step 430 further includes sub-step 430B, which determines the total area of ​​the image occupied by the intersection of the first and second detection boxes. This sub-step effectively calculates the size of the overlap.

[0152] Substeps 430A and 430B can be executed sequentially (in any order) or in parallel.

[0153] Then, step 430 proceeds to sub-step 430C, where the relative overlap between the first and second detection boxes is calculated by dividing the total image area occupied by the intersection by the total image area occupied by the union of the first and second detection boxes.

[0154] This effectively standardizes the size of the overlap between detection boxes relative to the area occupied by the detection boxes. Figure 5 The process shown can be labeled as an Intersection over Union (IoU) process.

[0155] Return to reference Figure 4In such an implementation, the predetermined threshold can be no less than (or equal to) 0.5, for example, no less than (or equal to) 0.6. This will result in only those detection boxes whose majority of their area overlaps with another detection box being considered for discarding. In other words, the only detection boxes considered for discarding are those whose majority of their area overlaps with another detection box.

[0156] As mentioned earlier, by performing a patch-by-pattern suppression process, the number of other bounding boxes that a single bounding box must compare against is reduced. In many cases, this can significantly reduce the complexity of the NMS process.

[0157] In a regular NMS process that includes a suppression procedure, where each detection box is compared with every other detection box (i.e., no tiling), the suppression procedure uses OO(N) with big O notation. 2 The complexity is given by N, where N is the number of detection boxes in the image / used for the image.

[0158] The proposed block-by-block suppression process has different levels of complexity.

[0159] Let N continue to represent the number of detection boxes in the image, and let T... 2 The number of tiles into which the image is split (i.e., the total T spanning the image). 2 (a few blocks).

[0160] A single iteration of the patch-by-patch suppression process (i.e., performing steps 341 and 342 for a single patch) can be modeled as having a complexity of O(n log n). ) or O(M 2 ), where M equals N divided by T 2 M represents the average number of detection boxes overlapping with the block. Therefore, the complexity for the entire image-by-image block suppression process is O(T). 2 M 2 ), which is equal to O( It can be seen that the patch-by-patch suppression process itself reduces the suppression complexity by T. 2 Multiples (or the number of tiles).

[0161] However, it should be understood that this is merely a model estimation, and there are at least two additional elements that need to be considered. First, as the number of tiles increases (from 1 to many), the probability that each detection box will span multiple tiles increases. Second, we have an additional step 330, where we need to identify which detections overlap with each tile.

[0162] The first element can be explained by assuming an average number of tiles “S” that the detection box may traverse. Specifically, if each detection traverses an average of S tiles, performing S processing steps is equivalent to performing only one N processing step. S-detection. Therefore:

[0163] And the overall complexity is:

[0164] S can be calculated by estimating the ratio (R) between the tile size (e.g., tile height / width) and the average size of the detection rectangle (e.g., average detection box height / width), i.e., tile size (S) ÷ detection size.

[0165] The ratio R can be composed of two components R H and R W Formed, for example, in the form of a 2D vector. R W This represents the ratio of the tile width to the average detection rectangle width (tile width ÷ detection width), and R... H This represents the ratio of the tile height to the average detection rectangle height (tile height ÷ detection height).

[0166] In other implementations, a single R value can be used, where R represents the "average" ratio of height and width. Such an R can be derived using the average of the height and width; that is, R can be the average of the tile's height and width (T). H and T W ) and the height and width of the detection frame (D AVH and D AVW The ratio of ), for example:

[0167] As another example, by calculating R... W and R H By averaging the values, we can obtain a single R.

[0168] Assume a simplified case where the block and the detection box are identical rectangular shapes (i.e., have the same aspect ratio), such as squares. When R = 1 (and, if it exists, R...), W = R H When R = 0.5 (and if present, R...), each detection box is the same size as a tile, and it will most likely intersect exactly with four tiles (unless it aligns exactly with the tile edges). W = R H When the detection box spans 2×2 tiles, it is very likely that it will intersect with 3×3 tiles.

[0169] Typically, it can be assumed that the parameter S follows the following equation, where R consists of two components R0. W and R H form:

[0170] Where R takes a single value, this can be simplified to:

[0171] The appropriate method for calculating and / or selecting R will be described later. It will be understood that the value R responds to the tile size (i.e., height and / or width).

[0172] The second element recognizes that identifying which detection boxes overlap with each tile introduces some complexity. As those skilled in the art will understand, the simple operation used to process detection boxes to identify which tiles overlap with them has a complexity of O(N×S). In particular, it is recognized that tiles divide the image into a regular rectangular grid, and each detection occupies a rectangular segment of the grid. This means that tiles overlapping with a particular detection can be directly computed (rather than searched). Since the average number of overlapping tiles for each detection is S, the process of building the tile list is an O(N×S) operation.

[0173] Therefore, the overall complexity of method 300 (excluding low-complexity steps 310, 320, and 350 (and 370, if executed)) is O(N×S) + O( ) and .

[0174] If sorting step 360 is performed (for example, following...), Figure 3 The complete process shown in the figure has some additional complexity for the sorting step, which has a complexity of O(N×log(N)).

[0175] Therefore, the overall complexity of method 300, including the sorting steps, can be modeled as O(N×S) + O( The sum of O(N×log(N)) + O(N×log(N)). For completeness, note that the complexity of steps 350 and 370 is O(N).

[0176] Those skilled in the art will understand that conventional NMS techniques typically utilize sorting steps, ensuring that including sorting steps does not increase the complexity of the proposed NMS technique beyond that of conventional NMS techniques.

[0177] Table 1 contains the theoretical complexity values ​​for the standard (CON) and the proposed (TILE) versions of NMS under different use cases, including the number of detection boxes N, the approximate number of patches in the image dimension T (estimated to be...). The complexity values ​​for the Tile version of NMS only include the complexity of steps 330 and 340 (the major contributors to the complexity). The value of N depends on the use case. The values ​​T and R are algorithm parameters, where T can be derived from R, the image size (denoted by I), and the average detection rectangle size (denoted by D). AV (Indicates) Export.

[0178] R is used as an "intermediate parameter". For Table 1, R is a single value, for example, rather than different values ​​for height and width.

[0179]

[0180] Table 1

[0181] In the context of this specification, size refers to the dimensions of a component, such as its height and / or width, and not its area. Image size I and detection rectangle size D AV It can depend on the usage.

[0182] The trill symbol “” is used where appropriate in the following description, especially in equations. The term "hat" (sometimes referred to as "hat," "roof," or "house") indicates the (size) dimensions of a component, which can be replaced by appropriate dimensions (e.g., height or width). Therefore, the parameter... (Referring to the image size) can be determined by I w (Image width) or I H (Image height) is used instead. Size substitution is consistent throughout the same equation, such that if the equation includes two parameters annotated with the corresponding wrapper, each substituted parameter must refer to the same size (i.e., height or width, respectively).

[0183] Average detection rectangle D AV The height and / or width (i.e., size components or dimensions) can be derived from the following equation:

[0184] in This represents a single dimension (i.e., height or width) of the average detection rectangle, and can be expressed as the height D of the average detection rectangle where appropriate. AVH Or the average detection rectangle width D AVW To replace. Representing the corresponding dimension of the image (e.g., where Use D AVH replace, Image height I H(Instead). The value R can be a single value (as mentioned before, for both height and width) or a different value depending on whether the height or width is being calculated (e.g., using R). H Or R W replace).

[0185] Parameters in Equation 6 It can be determined by parameters (That is, the size of the tile, which is the height or width of the tile where appropriate) is used instead.

[0186] In Table 1, numerical values ​​were selected such that T×R is constant for each use case. Use case 1 represents a use case with a small number of relatively large detection boxes, such as a holiday image with detection boxes targeting faces. Use case 2 represents a use case where each image contains a moderate number of relatively medium-sized detection boxes, such as a self-driving car detecting pedestrians. Use case 3 represents a use case where there are a large number of relatively small detection boxes, such as an image of a crowd identifying faces.

[0187] Table 1 clearly shows how computation can be reduced for certain use cases when the tile-based NMS method disclosed here is used instead of the conventional method.

[0188] The proposed NMS technique performs particularly well when there are many small objects in the scenario (so that each detection occupies a small number of tiles, i.e., S is small) and the objects are evenly distributed across the image (i.e., the maximum number of detection boxes per tile is small). Suitable use cases can include surveillance camera images from medium or long distances (in crowds or congested areas), where the target object is a human face or a photograph of a swarm of locusts.

[0189] An important consideration in improving and / or optimizing the implementation of the proposed method is to appropriately select the number of tiles and / or the tile size (height and width).

[0190] The block size (height and / or width of each block) can significantly impact the efficiency of the proposed NMS method. An inappropriate choice of block size will negatively affect the efficiency of the NMS method.

[0191] It has been recognized that the efficiency of the proposed method depends at least on the average number of intersecting / overlapping patches per detection box and the average number of detections per patch. Both of these factors are affected by the patch size (relative to the characteristics of the detection boxes in the image).

[0192] Specifically, if the size of R decreases, the detection boxes may overlap with more tiles (thus increasing the repetitive processing of the same detection boxes in different tiles). However, if the value of R increases, there will be more detection boxes in each tile, thus increasing the amount of computation that needs to be performed for each tile.

[0193] Therefore, a trade-off needs to be struck between these two extreme cases, and an appropriate choice of R is needed to compute the appropriate tile size.

[0194] Choosing or determining R is a particularly suitable method for calculating tile size (tile height or width) because it can be calculated independently of a specific image size, for example, based on an estimate of the use case scenario. An example of this is provided below.

[0195] For a given R, the tile size can be easily calculated using the following equation. That is, the size of the tile:

[0196] In some implementations, equation (7) is modified to calculate the tile width T. W and / or tile height T H By using T W or T H Replacement indicator parameters and use D AVW Or D AVH Replacement indicator parameters (Respectively) where appropriate, in a manner similar to that described in reference equation (6). In particular, the tile width T can be calculated using the following equation. W :

[0197] The following equation is used to calculate the block height T. H :

[0198] Where D AVW It is the average detection box width, and D AVH It is the average detection frame height.

[0199] R can remain constant (for equations 8 and 9), or in calculating the tile width T. W and tile height T H Different R values ​​can be used (i.e., R values ​​can be R1 and R2 respectively). W and R H Unless otherwise specified, the following description assumes that R is the value used for tile width T. W and tile height T H The individual values ​​of both.

[0200] To further understand, Figure 6 The relationship between the value R (here a common value shared between the width and height sizes) and the time taken to execute the NMS method of this disclosure to test the detection box data for a specific situation is shown.

[0201] In the scenario illustrated, the detection box data and image data remain constant. Therefore, the number of detection boxes, the average size of the detection boxes, and the image size remain constant. Since the average size of the detection boxes remains constant, an increase in R represents an increase in the block size (height and width).

[0202] Line 610 shows the total time spent performing the NMS method on the test detection box data. Line 620 shows the time spent performing the block-by-block suppression process (e.g., Figure 3 The time spent in process 340. Line 630 represents the time spent identifying which detection boxes overlap with each tile. Line 640 represents the time spent constructing the output list of detection boxes that were not marked for deletion. In the case of performing a sorting step, line 640 may alternatively and / or additionally represent the time spent performing the sorting step (since this is also of low complexity).

[0203] As from Figure 6 As can be seen, the value of R affects the time spent executing the NMS method. Therefore, choosing an appropriate R, and thus the block size T, is crucial. s This can improve the efficiency of the NMS method (for example, choosing R to minimize the time spent executing the NMS method).

[0204] However, the relationship between R and the time spent executing the proposed NMS method may vary depending on different use case scenarios. Therefore, the tile size can vary depending on the use case.

[0205] This disclosure recognizes that the relationship between R and the time spent performing the proposed NMS method varies in response to changes in several parameters, including (but not limited to) the ratio (Rs) between the average detection box size and the image size (e.g., the average detection box size divided by the image size) and the number n of (target) objects in the image.

[0206] Figure 7 Showing the "ideal" R I The relationship between (i.e., the R that minimizes the time required to perform the proposed NMS method for some sample data) and the ratio Rs, which has been derived from testing the sample detection box data. All other variables (except the ratio Rs) remain constant.

[0207] This relationship is shown by line 700.

[0208] It's understandable that the value Rs will always be less than 1 (because the average detection box will not be larger than the image size). Although in Figure 7 The example shown is merely an illustration, but the value of Rs ranges from 0 to 0.6.

[0209] from Figure 7 As those skilled in the art will understand, ideal R I The relationship between the value and the value Rs seems to be derived from: express.

[0210] Figure 8 Showing the "ideal" R I The relationship between the number of target objects (n) in the image and the number of target objects (n). This relationship has also been derived from testing with sample detection box data, during which all other variables (except the number of objects and the total number of detection boxes) remain constant.

[0211] This relationship is shown by line 800.

[0212] from Figure 8 In this context, those skilled in the art will readily understand that the ideal R... I The relationship between the value and the number of (target) objects in the image appears to be derived from: express.

[0213] Although tests have shown that this relationship is low, there may still be a small dependency on the number of repeated tests for each object.

[0214] If one or more of these parameters can be estimated or calculated, then R can be selected / generated, which can be used to calculate the “optimal” or “ideal” tile size.

[0215] The values ​​of at least some of these parameters (e.g., the ratio Rs and the number n) can be predicted based on the use case scenario. For example, if the use case scenario is for a detection box to identify faces within a large crowd, the number n of objects can be estimated in advance (e.g., based on the capacity of the area and / or the known business of the area) and the ratio Rs can be estimated (e.g., based on the estimated distance between the camera providing the image and the object to be detected).

[0216] Other methods for estimating these parameters will be apparent to those skilled in the art. As an example, experiments can be performed based on sample images used in the use case scenario, or the values ​​Rs and / or n can be estimated using previous frames captured by an image capture device (e.g., a camera). Other implementations may employ the operator's experience to select appropriate values ​​for these parameters.

[0217] This allows for the estimation of these values ​​to obtain appropriate values ​​for R (e.g., if generated using Rs and n). These estimates are independent of a specific image resolution and can therefore be generalized for different implementation scenarios (e.g., different cameras monitoring the same scenario). Thus, the value R is particularly convenient to use in the analysis and during the initial setup.

[0218] Therefore, in some preferred embodiments, R (and thus the tile size) depends on 1) the (predicted) ratio between the average detection box size and the image size and / or 2) the (predicted) number of (target) objects in the image.

[0219] As previously mentioned, the tile's (height and width) dimensions, i.e., the tile size, can be calculated using one or more of equations (7) - (9) based on R. The parameters of this equation... The following equation can be used to calculate:

[0220] in This is the image size, i.e., the tile size, which is a known quantity. Parameter D AVW and D AVH A modified version of formula (10) can be used to calculate, where the parameter Each is determined by parameter R SW (representing the ratio between the average detection box width and the image width) and R SH (representing the ratio between the average detection box height and the image height) is used instead.

[0221] Optionally, R (and therefore the tile size) may further or otherwise depend on 3) the (estimated) number of repeated detections per object.

[0222] It has been previously described how the ideal R depends on the ratio Rs (the ratio between the average detection box size and the image size). Therefore, R (and / or the tile size) can depend on an estimate of R. In some implementations, R (and / or the tile size) may depend on... .

[0223] It has been previously described how the ideal R depends in some way on the number of objects n in the image. Therefore, R (and / or tile size) can depend on an estimate of n (the estimated number of objects). In some implementations, R (and / or tile size) can depend on -log(n).

[0224] Ideally, R should have some dependence on the number of duplicate detections per object. Therefore, R (and / or tile size) can depend on the estimated number of duplicate detections per object.

[0225] Those skilled in the art will readily derive suitable equations or functions for calculating R based on these parameters. This may include iteratively performing multiple tests to determine the “ideal” R based on different values ​​of the parameters.

[0226] In one example, the equations / functions for calculating R are derived by combining partial equations for each parameter (derived from tests with each parameter modified).

[0227] However, by fitting all parameters (Rs and n) at once to estimate R, a more optimized equation / function can be obtained for calculating R.

[0228] A suitable equation for calculating an appropriate value of R is as follows:

[0229] To produce appropriate values ​​for R in this equation, the following parameters can be used: P1=43; P2=0; P3=10; P4=23; Q1=0.2; Q2=1; Q3=6; Q4= -25; W1= -23; W2=0.1; W3= -23.5. This yields the following equation:

[0230] When attempting to compute / design / derive equations that map parameters Rs and n to the “ideal” value R, other suitable parameters and equations will be obvious to those skilled in the art.

[0231] Any of the above mechanisms can be used to calculate tile height and / or tile width.

[0232] In one implementation, the two component values ​​of R are R W (or R width) value and R H (or R height) value, which are used to derive the tile width T. W and tile height T H .

[0233] In another instance, a single or combined R is calculated, and then said R is used to calculate the tile height and / or width using Equations 8 and 9.

[0234] For clarity, a complete working example for calculating tile sizes for a specific use case is described below.

[0235] In this example, the image has a resolution of 4096×2160 and represents a personal image of one or more people (e.g., a holiday or social outing). The target object (the location the detection box attempts to identify) is a person's face.

[0236] For typical use cases, the number of people per individual image will vary between 2 and 5, so a "busy" situation would likely have 4-5 people. Based on this understanding, the value of N is estimated to be approximately 4.

[0237] In the first case, from the individual R values ​​(R respectively) w and R H Calculate the block width T W and tile height T H This requires estimating the width ratio R, which is the ratio between the average detection box width and the image width. SW And the height ratio R, which is the ratio between the average detection box height and the image height. SH .

[0238] This can be done by processing an example personal image using object detection techniques, calculating the average detection box width / height, and dividing this / these values ​​by the image width / height. In another example, this is performed by the user and provided via user input.

[0239] According to one estimate, the estimated R SW It is 0.0700, and the estimated R is... SH It is 0.1256.

[0240] We can, for example, apply Equation 12 for the width and height (R respectively) W and R H Calculate the value of R separately. Using the R value described above... SW and R SH Value (and when n = 4), R W = 4.593 and R H = 3.435.

[0241] Finally, we use equations 8 and 9 to calculate the desired tile width and height (i.e., tile size). For a target camera resolution of 4096×2160: D AVW = 4096×R SW → T W =1317 pixels; D AVH = 2160×R SH → T H =932 pixels. This means a grid of 3.11×2.32 tiles. Grid values ​​can be rounded to the nearest integer (e.g., for 3×2 tiles).

[0242] In the second case, the tile width T is calculated from a single combination (or average) R(R). W and tile height T H This requires estimating the ratio Rs between the average detection box size and the image size.

[0243] This can be performed by processing one or more exemplary personal images (for a specific use case) using an object detection process to generate sample detection boxes. The midpoint between the detection box height and width can be calculated for each sample detection box. The average of these midpoints can be used to calculate the combined average Rs value. In other words:

[0244] Where X is the number of bounding boxes, DB indicates a unique bounding box, W(DB) is the width of the bounding box, H(DB) is the height of the bounding box, and I... H It is the image height, and I W It is the image width.

[0245] Based on an estimate from this example, the value Rs = 0.0917.

[0246] By applying Equation 12, the value of R is calculated using this Rs value (and n = 4), resulting in an output of R = 3.87.

[0247] Finally, the calculated R can be used to calculate the tile width T. W and height T H (i.e., tile size). For a target camera resolution of 4096×2160, and applying equations 8 and 9, T W = 1453 pixels and T H =766 pixels. This means a 2.82×2.82 tile grid ((4096 / 1453) × (2160 / 766)). Grid values ​​can be rounded to the nearest integer (e.g., 3×3 tiles).

[0248] It can be seen that there are two methods for determining the tile size (i.e., obtaining individual R values). W and R H Alternatively, obtaining a combined R yields roughly the same result. In this particular example, the difference between the two methods is approximately 10%-20%, which is not significant in practical implementations.

[0249] As an alternative to rounding the grid size up or down (thus dividing the image into tiles of the same size), the tile size can be calculated directly, for example, by applying padding to the image and / or accepting the rightmost and bottommost tiles where appropriate.

[0250] The implementation scheme is useful in any environment where non-maximum suppression (NMS) is performed on the detection boxes of an image, such as those generated at the output of object detection methods.

[0251] When implemented in a graphics processing system, the implementation scheme can be particularly useful, where the graphics processing system performs object recognition of an image.

[0252] The implementation is particularly useful when object detection methods identify a large number of relatively small objects in an image, where the small objects are distributed throughout the image. In these cases, image tiling significantly reduces the complexity of performing NMS.

[0253] The methods described above can be implemented in software, hardware, or a combination of both. Some steps, such as the sorting process, may be considered too complex and / or have limited use to guarantee hardware acceleration, but this should not be excluded from consideration.

[0254] Therefore, the implementation can find specific utility in processing systems such as graphics processing systems or artificial intelligence accelerator systems (e.g., including neural network accelerators).

[0255] Figure 9 A computer system in which such a graphics processing system can be implemented is shown. The computer system includes a CPU 902, a GPU 904, memory 906, and other devices 914, such as a display 916, a speaker 918, and a camera 106. A processing block 910 (for performing any of the methods described herein) may be implemented on the GPU 904. In other examples, the processing block 910 may be implemented on the CPU 902. The components of the computer system may communicate with each other via a communication bus 920. Memory 912 is implemented as part of memory 906 and may be used to store computer code executed by the processing block 910 for performing any of the methods described herein.

[0256] Although Figure 9 An implementation of a graphics processing system is shown, but it will be understood that a similar block diagram can be drawn for an artificial intelligence accelerator system, for example, by supplementing or replacing the CPU 902 or GPU 904 with a neural network accelerator (NNA) 911, or by adding the NNA as an additional unit. In this case, the processing block 910 can be implemented (at least partially) in the NNA.

[0257] In the illustrated embodiment, GPU 904 includes a neural network accelerator 911 adjacent to processing tile 910, which may be labeled as a processing module. The neural network accelerator 911 may be configured to process the image to generate multiple detection boxes identifying the potential locations of one or more objects within the image, wherein processing tile 910 performs any of the methods described herein for performing non-maximum suppression on the generated multiple detection boxes. The NNA may generate, for example, detection box data that identifies the location of each of the multiple detection boxes within the image and a confidence value indicating the confidence that the detection box identifies an object.

[0258] In other implementations, the processing block may be located on another side of the computer system (e.g., as part of CPU 902) and / or the NNA may be located on another side of the entire computer system (e.g., a dedicated separate module).

[0259] Figures 3 to 5 The method shown is illustrated as comprising multiple functional blocks. This is merely illustrative and not intended to define a strict division or order between the different logical elements of these entities. Each functional block can be provided in any suitable manner.

[0260] It should be understood that any intermediate values ​​(e.g., R) described herein need not be physically generated at any point by a processing tile (e.g., a graphics processing system), and may simply represent logical values ​​that conveniently describe the process performed when the method is executed (between its inputs and outputs).

[0261] The methods described herein can be executed in hardware on an integrated circuit. A graphics processing system can be configured to execute any of the methods described herein.

[0262] Generally, any of the functions, methods, techniques, or components described above can be implemented in software, firmware, hardware (e.g., a fixed logic circuit system), or any combination thereof. The terms “module,” “function,” “component,” “element,” “unit,” “block,” and “logic” are used herein to generally denote software, firmware, hardware, or any combination thereof. In the case of a software implementation, a module, function, component, element, unit, block, or logic represents program code that, when executed on a processor, performs a specified task. The algorithms and methods described herein can be executed by one or more processors that execute code that causes the processor to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disk, flash memory, hard disk storage, and other memory devices that can use magnetic, optical, and other techniques to store instructions or other data and are accessible by a machine.

[0263] As used herein, the terms computer program code and computer-readable instructions refer to any kind of executable code for a processor, comprising code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, bytecode, code defining integrated circuits (e.g., hardware description languages ​​or netlists), and code expressed in programming languages ​​such as C, Java, or OpenCL. Executable code can be, for example, any kind of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, or run in a virtual machine or other software environment, causes the processor of a computer system that supports the executable code to perform tasks specified by said code.

[0264] A processor, computer, or computer system can be any kind of device, machine, or special-purpose circuit, or a collection or part thereof, that has the processing capability to execute instructions. A processor can be or includes any kind of general-purpose or special-purpose processor, such as a CPU, GPU, NNA, system-on-a-chip, state machine, media processor, application-specific integrated circuit (ASIC), programmable logic array, field-programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0265] This invention also intends to encompass software, such as hardware description language (HDL) software, that defines the configuration of hardware as described herein, for designing integrated circuits or for configuring programmable chips to perform desired functions. That is, a computer-readable storage medium on which computer-readable program code in the form of an integrated circuit definition dataset is encoded may be provided, which, when processed (i.e., executed) in an integrated circuit manufacturing system, configures the system to manufacture a graphics processing system configured to perform any of the methods described herein. The integrated circuit definition dataset may, for example, be an integrated circuit description.

[0266] Therefore, a method for manufacturing a graphics processing system for performing any of the methods described herein can be provided at an integrated circuit manufacturing system. Furthermore, an integrated circuit definition dataset can be provided, which, when processed in the integrated circuit manufacturing system, enables the execution of the method for manufacturing the graphics processing system.

[0267] Integrated circuit definition datasets can be in the form of computer code, such as netlists, code for configuring programmable chips, or hardware description languages ​​suitable for manufacturing at any level in integrated circuits, including register-transfer level (RTL) code, high-level circuit representations (such as Verilog or VHDL), and low-level circuit representations (such as OASIS (RTM) and GDSII). Higher-level representations (such as RTL) that logically define hardware suitable for manufacturing in integrated circuits can be processed on a computer system configured to generate manufacturing definitions of integrated circuits within the context of a software environment that includes definitions of circuit elements and rules for combining these elements to generate the manufacturing definition of the integrated circuit defined by that representation. As is typically the case where software executes at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to configure the computer system to generate the manufacturing definition of the integrated circuit, executing code that defines the integrated circuit in order to generate the manufacturing definition of the integrated circuit.

[0268] Now refer to Figure 10 This describes an example of processing integrated circuit definition datasets at an integrated circuit manufacturing system in order to configure the system for manufacturing graphics processing systems.

[0269] Figure 10 An example of an integrated circuit (IC) manufacturing system 1002 is shown, configured to manufacture a graphics processing system as described in any of the examples herein. Specifically, the IC manufacturing system 1002 includes a layout processing system 1004 and an integrated circuit generation system 1006. The IC manufacturing system 1002 is configured to receive an IC definition dataset (e.g., defining a graphics processing system as described in any of the examples herein), process the IC definition dataset, and generate an IC (e.g., embodying the graphics processing system as described in any of the examples herein) based on the IC definition dataset. Through the processing of the IC definition dataset, the IC manufacturing system 1002 is configured to manufacture an integrated circuit embodying the graphics processing system as described in any of the examples herein.

[0270] The layout processing system 1004 is configured to receive and process an IC definition dataset to determine a circuit layout. Methods for determining a circuit layout based on an IC definition dataset are known in the art and may involve, for example, synthesizing RTL code to determine the gate-level representation of the circuit to be generated, for example, in relation to logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining the location information of the logic components, the circuit layout can be determined based on the gate-level representation of the circuit. This can be done automatically or with user intervention to optimize the circuit layout. Once the layout processing system 1004 has determined the circuit layout, it can output the circuit layout definition to the IC generation system 1006. The circuit layout definition may be, for example, a circuit layout description.

[0271] As is known in the art, IC generation system 1006 generates ICs according to a circuit layout definition. For example, IC generation system 1006 can implement a semiconductor device manufacturing process for generating ICs, which may involve a multi-step sequence of photolithography and chemical processing steps, during which electronic circuits are gradually formed on a wafer made of semiconductor material. The circuit layout definition may be in the form of a mask, which can be used in the photolithography process to generate ICs according to the circuit definition. Alternatively, the circuit layout definition provided to IC generation system 1006 may be in the form of computer-readable code, which IC generation system 1006 can use to form a suitable mask for generating ICs.

[0272] The various processes performed by the IC manufacturing system 1002 can all be implemented in one location, for example, by one party. Alternatively, the IC manufacturing system 1002 can be a distributed system, allowing some processes to be performed in different locations and by different parties. For example, some of the following stages can be performed in different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition dataset to form a gate-level representation of the circuit to be generated; (ii) generating a circuit layout based on the gate-level representation; (iii) forming a mask based on the circuit layout; and (iv) using the mask to manufacture the integrated circuit.

[0273] In other examples, processing of an integrated circuit definition dataset in an integrated circuit manufacturing system can configure the system to manufacture a graphics processing system without processing the IC definition dataset to determine circuit layout. For example, an integrated circuit definition dataset can define the configuration of a reconfigurable processor, such as an FPGA, and processing of the dataset can configure the IC manufacturing system (e.g., by loading the configuration data into the FPGA) to generate a reconfigurable processor with the defined configuration.

[0274] In some implementations, when processed in an integrated circuit manufacturing system, an integrated circuit manufacturing definition dataset can enable the integrated circuit manufacturing system to generate devices as described herein. For example, using an integrated circuit manufacturing definition dataset, with reference to the above... Figure 10 The described method allows for the configuration of an integrated circuit manufacturing system to produce equipment as described in this article.

[0275] In some examples, an integrated circuit definition dataset may include software running on hardware defined at the dataset, or software running in combination with hardware defined at the dataset. Figure 10 In the example shown, the IC generation system can also be further configured by the integrated circuit definition dataset to load firmware onto the integrated circuit according to the program code defined in the integrated circuit definition dataset during the manufacturing of the integrated circuit, or otherwise provide the integrated circuit with program code to be used with the integrated circuit.

[0276] Compared to known implementations, the implementation of the concepts set forth in this application in devices, apparatuses, modules, and / or systems (and in the methods implemented herein) can lead to performance improvements. Performance improvements may include one or more of increased computational performance, reduced latency, increased throughput, and / or reduced power consumption. During the manufacture of such devices, apparatuses, modules, and systems (e.g., in integrated circuits), trade-offs can be made between performance improvements and physical implementation methods, thereby improving manufacturing methods. For example, a trade-off can be made between performance improvements and layout area to match the performance of known implementations but using less silicon. This can be accomplished, for example, by reusing functional blocks serially or sharing functional blocks among elements of a device, apparatus, module, and / or system. Conversely, the concepts set forth in this application that lead to improvements in the physical implementation of devices, apparatuses, modules, and systems (such as reduced silicon area) can be traded off for performance improvements. This can be accomplished, for example, by manufacturing multiple instances of a module within a predefined area budget.

[0277] The applicant has independently disclosed each individual feature described herein, as well as any combination of two or more such features, to the extent that such features or combinations can be implemented based on the specification as a whole, in accordance with the common knowledge of those skilled in the art, regardless of whether such features or combinations of features solve any problem disclosed herein. In view of the foregoing description, those skilled in the art will understand that various modifications can be made within the scope of this invention.

Claims

1. A computer-implemented method (300) for performing non-maximum suppression (NMS) on multiple detection boxes (101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117) of potential locations of one or more objects within an image (100), the computer-implemented method comprising: Obtain (310) detection box data, the detection box data being for each of the plurality of detection boxes, identifying the position of the detection box within the image and indicating the confidence level of the object identified by the detection box; The image is divided (320) into multiple image blocks (151, 152, 153, 154, 155, 156); For each tile, if a detection box overlaps with a portion of the tile, the detection box data is processed to identify (330) which detection boxes overlap with a portion of the tile; Perform (340) a tile-by-tile suppression process, the tile-by-tile suppression process comprising: for each tile, if the tile includes at least two detection boxes that overlap each other by more than a predetermined amount, then perform the following operations: Compare the confidence values ​​of detection boxes (341) that overlap by more than a predetermined amount, and For each comparison, the detection box associated with the lower confidence value is marked (342) for discarding. Specifically, the steps of comparing confidence values ​​and marking are iteratively repeated until each detection box that partially overlaps with the patch satisfies the following condition: i) The confidence value has been compared with each other detection box that meets the following criteria: a. Not marked as discarded; b. Overlapping with the portion of the map; and c. The overlap with the detection frame exceeds a predetermined amount; or ii) has been marked as discarded.

2. The computer-implemented method (300) of claim 1, wherein the per-block suppression process (340, 400) comprises, for each block, iteratively performing a thresholding process, the thresholding process comprising: Select (410) a first detection box that overlaps with a portion of the tile, wherein the first detection box has not yet been marked for discarding; Select (420) a different second detection box that overlaps with a portion of the patch for comparison with the first detection box, wherein the second detection box has not yet been marked for discarding, and wherein the first detection box and the second detection box have not previously been compared with each other during the thresholding process for the patch; Determine the relative overlap between the selected first and second detection boxes (430); as well as In response to the determined relative overlap exceeding the predetermined amount: If the confidence value of the first detection box is less than the confidence value of the second detection box, then mark (455) the first detection box for discarding; and / or If the confidence value of the second detection box is less than the confidence value of the first detection box, then mark (456) the second detection box for discarding. The thresholding process for the patch is repeated until all detection boxes of the patch have been compared with each other detection box of the patch or have been marked for discard.

3. The computer-implemented method (300) of claim 2, wherein the step (430) of determining the relative overlap between the selected first and second detection boxes comprises: Determine (430A) the total area of ​​the image occupied by the union of the first detection box and the second detection box; Determine (430B) the total area of ​​the image occupied by the intersection of the first detection box and the second detection box; as well as The relative overlap is calculated (430C) by dividing the total area of ​​the image occupied by the intersection of the first detection box and the second detection box by the total area of ​​the image occupied by the union of the first detection box and the second detection box.

4. The computer-implemented method (300) of any one of claims 2 or 3, further comprising the step (360) of sorting the detection boxes according to confidence levels, wherein: The step (410) of selecting the first detection box includes selecting the most reliable detection box that was not marked for discard during the thresholding process of the tile and has not been previously compared with all other detection boxes that were not marked for discard; and The step (420) of selecting the second detection box includes selecting a detection box that has not been previously compared with the first detection box.

5. The computer-implemented method (300) as described in any one of claims 1 to 4, wherein: The computer-implemented method further includes a step (360) of sorting the detection boxes according to confidence levels, the sorting step being performed before the block-by-block suppression process; or The per-block suppression process includes, for each block, sorting the detection boxes that overlap with a portion of the block according to their confidence level.

6. The computer-implemented method (300) as described in any one of claims 1 to 5, wherein the size of each tile depends on: The estimated ratio between the size of the detection box and the resolution of the image; and / or The estimated number of objects in the image; and / or The estimated detection box size and the resolution of the image.

7. The computer-implemented method (300) as described in any one of claims 1 to 6, further comprising: Identify detection boxes associated with confidence values ​​below a predetermined confidence threshold; as well as The marker (370) is associated with a detection box that is below the predetermined confidence threshold for discarding. For each tile, if a detection box overlaps with a portion of the tile, step (330) of identifying which detection boxes overlap with a portion of the tile does not identify detection boxes that have already been marked for discard.

8. The computer-implemented method (300) of any one of claims 1 to 7, wherein the step of performing the block-by-block suppression process (340) is performed using parallel processing techniques.

9. The computer-implemented method (300) of any one of claims 1 to 8, wherein for each tile, if there is a detection box that overlaps with a portion of the tile, the step (330) of identifying which detection boxes overlap with a portion of the tile comprises, for each tile, generating a list of tiles that identify which detection boxes overlap with a portion of the tile if there is a detection box that overlaps with a portion of the tile.

10. The computer-implemented method (300) of any one of claims 1 to 9, further comprising establishing (350) an output list of detection boxes, the output list identifying detection boxes that were never marked for discard during the block-by-block suppression process.

11. The computer-implemented method (300) of claim 10, wherein the output list of the detection boxes established (350) satisfies any one of the following: The step of marking the detection boxes associated with the lower confidence values ​​for discarding includes modifying the flags associated with the detection boxes; and the step of building an output list of detection boxes includes processing the flags to identify detection boxes that were never marked for discarding during the non-maximum suppression method; or The step of establishing the output list includes initializing a global list identifying all detection boxes before performing the tile-by-tile suppression process, and marking the detection boxes associated with the lower confidence values ​​for discarding. The step of marking the detection boxes includes modifying the flags in the global list associated with the detection boxes for marking, such that when the tile-by-tile suppression process is completed, the global list facilitates the provision of the output list identifying detection boxes that were never marked for discarding.

12. A computer-implemented method for performing non-maximum suppression (NMS) on multiple detection boxes identifying potential locations of one or more objects of different categories within an image, the computer-implemented method comprising: The multiple detection boxes are divided into different categories, and each category of detection box is associated with an object of a different category; as well as Perform the method of any one of claims 1 to 11 on each category of the detection box.

13. A computer-readable storage medium storing computer-readable code, which, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 11.

14. A graphics processing system (904) configured to perform the method of any one of claims 1 to 11.

15. The graphics processing system of claim 14, wherein the graphics processing system comprises: A neural network accelerator (911) is configured to process an image to generate multiple detection boxes that identify the potential locations of one or more objects within the image; as well as The processing module (910) is configured to obtain the plurality of detection boxes from the neural network accelerator and perform the method as described in any one of claims 1 to 11.

16. A method of manufacturing the graphics processing system of claim 14 or 15 using an integrated circuit manufacturing system (1002), the method comprising: The layout processing system (1004) processes the computer-readable description of the graphics processing system to generate a circuit layout description of the integrated circuits embodying the graphics processing system. as well as The graphics processing system is manufactured using an integrated circuit manufacturing system (1006) based on the circuit layout description.

17. A non-transitory computer-readable storage medium storing a computer-readable description of a graphics processing system as claimed in claim 14 or 15, wherein when processed in an integrated circuit manufacturing system (1002), the computer-readable description causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the graphics processing system.

18. An integrated circuit manufacturing system, comprising: A non-transitory computer-readable storage medium having stored thereon a computer-readable description of the graphics processing system as described in claim 14 or 15; A layout processing system (1004) is configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the graphics processing system; as well as An integrated circuit manufacturing system (1006) is configured to manufacture the graphics processing system according to the circuit layout description.

Images