Method and apparatus for small object detection in images and videos

Abstract

Methods, apparatus, systems, and articles of manufacture for small object detection in images and videos are disclosed. An example apparatus for small object detection includes a memory, computer readable instructions, and at least one processor to execute the computer readable instructions to: receive at least an input image; identifying a first packet reference box for a first object representation in the input image, the first packet reference box based on feature extraction performed using a feature extractor network; extracting a first coordinate and a second coordinate of the corner position from a thermodynamic diagram, wherein the thermodynamic diagram is used for determining a first grouping reference frame; generating a second grouping reference frame for the first object representation based on the corner position; and updating the first packet reference frame with the second packet reference frame.

Description

Technical Field

[0001] The present disclosure relates generally to computing systems and, more particularly, to methods and apparatus for small object detection in images and videos. Background Art

[0002] Object detection in images and videos is a common computer vision task. Object detection has been widely used in various applications, such as smart transportation, smart retail, robotics, and aerospace. Existing object detection methods include one-stage, two-stage, anchor-based, and anchor-free. In some examples, keypoint-based methods are used for small object and occlusion detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 An example known small object detection is shown and compared with the small object detection using single-stage soft group non-maximum suppression (SG-NMS) disclosed herein.

[0004] Figure 2 Shown for Figure 1 Single-stage soft-group non-maximum suppression (SG-NMS) for small object detection, the SG-NMS comprising an object detector circuit system constructed according to the teachings of the present disclosure.

[0005] Figure 3 yes Figure 2 00140] A block diagram of an example implementation of an object detector circuit system.

[0006] Figure 4 It means that it can be executed to achieve Figure 3 A flow chart of example machine readable instructions for object detector circuitry.

[0007] Figure 5 is a flowchart representing example machine readable instructions that are executed by Figure 2 The object detector circuit system, when executed, causes the object detector circuit system to train a neural network to determine keypoints as part of a convolutional encoder-decoder network for keypoint based detection.

[0008] Figure 6 It means that it can be executed to use Figure 3 A flow chart of example machine readable instructions for grouping corner keypoints using a soft grouping (SG) algorithm and non-maximum suppression (NMS) for an object detector circuit system.

[0009] Figure 7 is a flowchart representing example machine readable instructions that are executed by Figure 2The object detector circuitry, when executed, causes the object detector circuitry to train a neural network to determine a width and / or height of a group reference box (GRB) as part of a convolutional encoder-decoder network for keypoint based detection.

[0010] Figure 8 Example steps representing the algorithms for soft grouping (SG) and non-maximum suppression (NMS) soft object detection disclosed herein are shown.

[0011] Fig. 9 Example graphs illustrating object detection performance using anchor-based and anchor-free methods, including soft grouping (SG) and non-maximum suppression (NMS) soft object detection disclosed herein, are shown.

[0012] Fig.10 is a block diagram of an example processing platform including processor circuitry configured to perform Figures 4 to 7 Example machine readable instructions to implement Figure 2 Object detector circuit system.

[0013] Fig.11 is constructed to execute Figure 5 Instructions to achieve Figure 3 A block diagram of an example processing platform of an example first computing system.

[0014] Fig.12 is constructed to execute Figure 7 Instructions to achieve Figure 3 0026] A block diagram of an example processing platform of an example second computing system.

[0015] Fig.13 yes Fig.10 , Fig.11 , Fig.12 A block diagram of an example implementation of a processor circuit system.

[0016] Fig.14 yes Fig.10 , Fig.11 , Fig.12 A block diagram of another example implementation of a processor circuit system.

[0017] Fig.15 is a block diagram of an example software distribution platform (eg, one or more servers) for distributing software (eg, corresponding to Fig.10 , Fig.11 , Fig.12Example machine-readable instructions of the software (software) are distributed to client devices associated with end-users and / or consumers (e.g., for licensing, sale, and / or use), retailers (e.g., for sale, resale, licensing and / or sub-licensing), and / or original equipment manufacturers (OEMs) (e.g., for inclusion in products to be distributed to, for example, retailers and / or other end-users such as direct purchasing customers).

[0018] The drawings are not drawn to scale. Generally, the same reference numerals will be used throughout the drawings and the attached written description to refer to the same or similar parts. Unless otherwise specifically stated, descriptors such as "first", "second", "third", etc. are used herein without attributing or otherwise indicating any meaning of priority, physical order, arrangement in a list, and / or sorting, but are only used as labels and / or arbitrary names to distinguish elements, so as to facilitate understanding of the disclosed examples. In some examples, the descriptor "first" can be used to refer to an element in the detailed description, while the same element can be referred to in a claim with a different descriptor (such as "second" or "third"). In such an example, it should be understood that such descriptors are only used to clearly identify those elements that may, for example, share the same name in other ways. As used herein, "approximately" and "about" refer to dimensions that may be imprecise due to manufacturing tolerances and / or other real-world defects. As used herein, "substantially real-time" refers to the occurrence of recognition in a near-instantaneous manner that there may be real-world delays for computing time, transmission, etc. Therefore, unless otherwise stated, "substantially real-time" refers to real-time + / - 1 second. As used herein, the phrase "communication" (including variations thereof) encompasses direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but additionally includes selective communication at periodic intervals, scheduled intervals, non-periodic intervals, and / or one-time events. As used herein, "processor circuitry" is defined as including (i) one or more special-purpose electrical circuits that are constructed to perform specific operations and include one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors), and / or (ii) one or more general-purpose semiconductor-based electrical circuits that are programmed with instructions to perform specific operations and include one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Examples of processor circuitry include a programmed microprocessor, a field programmable gate array (FPGA) that can instantiate instructions, a central processor unit (CPU), a graphics processor unit (GPU), a digital signal processor (DSP), an XPU or microcontroller, and an integrated circuit such as an application-specific integrated circuit (ASIC). For example, the XPU may be implemented by a heterogeneous computing system including multiple types of processor circuit systems (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more DSPs, etc. and / or combinations thereof) and an application programming interface (API), which may assign computing tasks to any one or more of the multiple types of processing circuit systems that are most suitable for performing the computing tasks. DETAILED DESCRIPTION

[0019] Disclosed herein are methods and apparatus for small object detection in images and videos. Object detection is used in computer vision tasks to infer information (e.g., three-dimensional information) of a given object identified in an image and / or video. Object detection associated with computer vision tasks can be applicable to many fields, including robotics, autonomous driving, and / or augmented reality. In the examples disclosed herein, "small" object detection refers to detecting objects of small size in an image of interest (e.g., including objects that may be physically large but occupy a small area on the image and / or appear small compared to other objects in the image of interest). For example, small object detection presents challenges associated with a small representation of a given object in an image of interest, assuming that images can have different resolutions so that the visual information used for identification based on small objects can be limited (e.g., small objects can be deformed and / or covered by larger objects in the image of interest).

[0020] Existing object detectors can implement clustering algorithms such as non-maximum suppression (NMS) to perform post-processing on multiple boxes generated for each recognized object in a given image. For example, NMS allows an entity (e.g., a bounding box) to be selected from multiple overlapping entities (e.g., multiple bounding boxes used to represent object detection). For example, an object detector can form a bounding box or window around a given object detected in an image, but there may be multiple bounding boxes generated for a single entity (e.g., thousands of windows of various sizes and shapes). NMS can be used to filter the resulting bounding boxes (e.g., using an intersection over union (IOU) metric) to select the most accurately located bounding box representing a given object or entity of interest. In some examples, NMS relies on selecting the prediction with the maximum confidence while suppressing all other generated predictions (e.g., bounding boxes), thereby taking the maximum value and suppressing non-maximum predictions.

[0021] Successful object detection using known object detectors can also depend on effective feature extraction based on the detection of important image regions. Feature extraction can include keypoint detection, which refers to the simultaneous detection of objects and the localization of their keypoints (e.g., spatial locations or points in an image that define an object and are invariant to rotation, shrinkage, deformation, etc.). In some examples, a particular algorithm (e.g., CornerNet, CenterNet, CentripetalNet, etc.) can detect an object as a pair of keypoints (e.g., the upper left corner and lower right corner of a bounding box, etc.). In some examples, such an algorithm can use a convolutional network to generate a heat map of a particular corner (e.g., the upper left corner, the lower right corner, etc.) of all instances of an object. However, keypoint-based methods face challenges including hard grouping (e.g., such methods depend on robust estimation of pairs of corner points, such that if one of the paired corners is incorrect, the rest of the grouping is inaccurate). In addition, keypoint-based methods can rely on complex pipelines with post-processing calculations (e.g., such methods can use a model that divides the grouping and NMS processes into two separate stages, which requires a 2×O(n 2 (Big O) computational complexity of two separate calculations of distance measurement). For example, CornerNet and its variants (e.g., CenterNet and CentripetalNet) define corner grouping and NMS as two additional stages, with two predefined corner points or one center point as the conditions for grouping. Such techniques may introduce additional complexity and / or require additional computational resources, which may limit computational efficiency for the purpose of object detection.

[0022] The method and apparatus disclosed herein use a simplified pipeline to introduce small object detection in images and / or videos to improve object detection efficiency. In the examples disclosed herein, the grouping and NMS stages are merged into a single stage and / or can share distance metric calculations. Additionally, the method and apparatus disclosed herein allow different numbers of (multiple) corner points to improve object detection accuracy. In the examples disclosed herein, soft grouping non-maximum suppression (SG-NMS) can be used for object detection by merging soft grouping with NMS into a single stage. In this way, instead of using two separate distance metric calculations, distance calculations can be shared between the SG and NMS stages to improve efficiency. In addition, a flexible number of estimated corners (e.g., 1 to 4) can be used, while existing methods of object detection based on key points can depend on a fixed pair of two corners. Using the method and apparatus disclosed herein, the average precision of object detection is significantly improved. For example, known techniques using center-guided methods (e.g., Faster R-CNN, RetinaNet, FCOS, CenterNet, etc.) for bounding box recognition use center points to model object bounding boxes without corner grouping. However, given that the center point of the bounding box may need to be determined by all four boundaries of the instance (e.g., with four degrees of freedom), this known center-guided group-free approach makes it difficult to produce high-quality detection boxes, especially for small objects and occlusions. The methods and devices disclosed herein improve the object detection efficiency of artificial intelligence-related tasks that can be performed using grouping and / or NMS (e.g., face detection, human detection, action detection, etc.). In particular, the methods and devices disclosed herein introduce linear efficiency and accuracy improvements using a general modular algorithm that can linearly reduce the computational cost of conventional grouping and NMS (e.g., increasing accuracy by more than 6.2% at twice the processing speed).

[0023] Figure 1 An example known small object detection 100 is shown and compared with a small object detection using single-stage soft group non-maximum suppression (SG-NMS) 150 disclosed herein. Figure 1In an example of , a group of images 102 may be provided to a backbone 108 for processing as part of a convolutional neural network (CNN). For example, the grouped images 102 may include an original image 104 and a flipped image 106 of the original image 104. For example, the backbone 108 (e.g., a feature extractor network) may be a CNN for purposes of object detection applications involving classification, detection, or segmentation models. In some examples, the performance of object detection may depend on the features extracted by the backbone 108 (e.g., using networks such as ResNet-50, ResNet-101, ResNet-152, etc.). In some examples, the backbone 108 may be used for object classification tasks, where the classifier classifies a single object in an image, outputs a single category per image, and / or provides a probability of matching a particular category. However, for the purposes of object detection, several objects may be identified in a single image and / or coordinates may be provided to identify the location of an object in an image. For example, in Figure 1 , the image 104 includes an object (e.g., a first object 115) that can be identified using object detection techniques. CNN-based object detectors can be classified into two-stage detectors (e.g., a detector that performs region proposal generation on one network and performs object classification for each region proposal on another network) and one-stage detectors (e.g., a detector that performs region proposal and object classification on a single network). For example, object detection can be performed by generating regions of interest (e.g., region proposals), which are a large set of bounding boxes that span the entire image (e.g., as part of object localization). Visual features can be extracted for each of the bounding boxes, where it is determined which objects are present in the region proposal based on the visual features (e.g., object classification). Overlapping boxes can then be combined into a single bounding box using non-maximum suppression (NMS).

[0024] exist Figure 1In the example of , a backbone 108 (e.g., a feature extractor network) is applied in a known small object detection 100, using feature extraction of images 104, 106 to result in identifying corners associated with an object (e.g., a first object 115). For example, the known small object detection 100 may require the use of two fixed pairs of corners for object recognition (e.g., a first fixed corner pair 114, 116 of the first object 115, a second fixed corner pair 120, 122 of the first object 115). For example, the fixed corners of the first object 115 can be identified based on the original image 104 object 115 position and the flipped image 106 object 115 position. The known object detection method includes separate stages, such as grouping 128 and example non-maximum suppression (NMS) 130, which can be performed in the post-feature extraction stage 125 of the object detection process. Separating the grouping 128 and NMS 130 into two different stages reduces network efficiency. Furthermore, since fixed corner pairs are required for object detection purposes, object detection becomes more difficult as the number of available identified corners decreases. In the example of known small object detection 100 , the final object identification 132 includes the corner point(s) 134 , 136 of the first object 115 .

[0025] In the example of SG-NMS based object detection, the grouping 128 and NMS 130 stages of known small object detection 100 are combined into a single stage to improve efficiency, as disclosed herein. For example, only the original image 104 is required as input to the backbone 108 (e.g., feature extraction network). Likewise, a flexible number of corners can be used for object detection (e.g., corners 154, 156, 158, 160 associated with feature extraction of the original image 104). Figure 1 In the example of FIG. 1 , a combined soft group non-maximum suppression (SG-NMS) stage 162 may be used for the grouping 128 and NMS 130 stages, such as in combination with Figure 2 A resulting object detection image 164 (eg, including a bounding box for each object) may be obtained for the first object 115 (eg, wherein the bounding box of the first object 115 is defined using the corner points 166, 168, 170, 182).

[0026] Figure 2 Shown for Figure 1 An overview 200 of a single-stage soft-group non-maximum suppression (SG-NMS) 150 for small object detection is provided, the SG-NMS 150 including an object detector circuitry 201 constructed in accordance with the teachings of the present invention. Figure 2 In the example, Figure 1 The original image 104 is provided to the object detector circuit system 201, which includes a feature extractor network 202 (e.g., Figure 1The main trunk 108). Figure 1 As described, the feature extractor network 202 can be a convolutional neural network (CNN) for extracting features associated with the input image 104. In some examples, the object detection model described herein can include a head 204 (e.g., a pre-trained backbone 108 and a random head 204 representing the top of the network). For example, a classification network can include a backbone and a fully connected layer as the only prediction head. While the backbone 108 can be used to extract a feature map containing high-level summary information from an image (e.g., the original image 104), the head 204 uses the feature map as input to predict a desired result. Figure 2 In the example of , a grouped reference box 206 is generated to be provided as an input to the SG-NMS 150 algorithm. In some examples, the bounding box can be regressed between the corner locations and the corners extracted using the heat map. The grouped reference box 206 provides a specific regression target to allow the SG-NMS 150 algorithm to match a flexible (e.g., soft) number of corners belonging to the same object instance (e.g., the first object 115, etc.). Figure 2 In an example of , the SG-NMS 150 algorithm can be used for several corner keypoints (e.g., all four corner keypoints) to match corners to the same object instance and simplify and / or reduce the computational complexity of post-processing steps of a corner-based object detection pipeline (e.g., using the grouped reference box (GRB) output 208). For example, the use of corner matching can be determined based on a distance metric of corresponding GRBs (e.g., determined using the shared distance measurement 212), such as in combination with Figure 3 In some examples, as part of the NMS algorithm 214, the distance metric may be based on an intersection-over-union (IoU) distance metric. Thus, the IoU distance measurement may be shared between the soft grouping (SG) algorithm 210 and the NMS algorithm 214.

[0027] exist Figure 2 In the example of , the soft grouping (SG) output 250 includes an illustration of corner point generation. The corner points can be generated as a single corner point 252, a diagonal corner point 256, an anti-diagonal corner point 260, a horizontally adjacent corner point 262, and / or a vertically adjacent corner point 264. For example, the upper left corner 254 can be extracted based on the single corner point 252 extraction. The diagonal corner point 256 extraction can result in the identification of the upper left corner 254 and the lower right corner 258. In some examples, a heat map can be used to extract the corners 254, 258, where the dotted lines shown connecting the corners 254, 258 represent regression group reference boxes (GRBs) (e.g., a first GRB 255 determined using the upper left corner 254, a second GRB 259 determined using the upper left corner 254 and the lower right corner 258, etc.). The heat map represents a matrix filled with values ​​from 0.0 to 1.0, where peaks on the heat map indicate the presence of an object. In Figure 2 In the example of , the third GRB 261 can be determined based on the first GRB 255 and the second GRB 259. For example, a single corner point 252 recognition can be performed using a single GRB. Thus, the methods and apparatus disclosed herein even allow object-based recognition using a single keypoint (e.g., based on a GRB generated using a single corner point 252). In some examples, the keypoints associated with the upper left corner 254 and the lower right corner 258 can be determined using a vanilla-based grouping process (e.g., a standard back propagation algorithm), such as in combination with Figure 3 In some examples, object detection may be performed based on any other number of available corner points (e.g., opposite corner points 260 such as corner points 262, 266, horizontally adjacent corner points 262 such as corner points 265, 268, and / or vertically adjacent corner points 264 such as corner points 270, 272). Thus, as part of the soft grouping (SG) output 250, any number of corner points may be used to generate a regression grouping reference box. Figure 2 In the example of , the soft grouping (SG) 210 and NMS 214 stages are merged into a single stage (e.g., SG-NMS 216), and a distance metric calculation (e.g., shared distance measurement 212) can be shared. Therefore, instead of using two separate distance metric calculations, the distance calculation can be shared between the SG and NMS stages to improve object detection efficiency (e.g., Figure 2 In addition, a flexible number of estimated corners (e.g., 1 to 4) can be used (e.g., while existing methods for keypoint-based object detection may depend on a fixed pair of two corners), thereby improving the average precision of object detection, such as by combining Fig. 9 described.

[0028] Figure 3 yes Figure 2 A block diagram of an example implementation of an object detector circuit system. Figure 3 , the object detector circuit system 201 includes an example input receiver circuit system 302, a trunk generator circuit system 304, a grouping reference box generator circuit system 306, a dimension identifier circuit system 308, a regression map generator circuit system 310, a heat map generator circuit system 312, a threshold identifier circuit system 314, an output generator circuit system 316, a tester circuit system 318 and / or a data storage 320.

[0029] The input receiver circuitry 302 receives an object image input (e.g., the original image 104) and / or any other information associated with the object image input (e.g., image size, region of interest in the input object image, etc.). In some examples, the input receiver circuitry 302 may receive the object image input from a single source or multiple sources (e.g., digital images, videos, etc.).

[0030] The backbone generator circuit system 304 uses a feature extractor network to extract feature maps containing high-level summary information from an image (e.g., the original image 104 obtained using the input receiver circuit system 302). For example, the backbone generator circuit system 304 can be a convolutional neural network for the purpose of object detection applications involving classification, detection, or segmentation models. In some examples, the performance of object detection can depend on the features extracted by the backbone generator circuit system 304 (e.g., using the backbone 108) using a network such as ResNet-50, ResNet-101, ResNet-152, etc. In some examples, the backbone generator circuit system 304 can be used for object classification tasks, where the classifier classifies individual objects in the image, outputs a single category per image, and / or provides a probability of matching a specific category. Figure 3 As shown, the backbone generator circuitry 304 is in communication with a first computing system 325 that trains a neural network. As disclosed herein, the backbone generator circuitry 304 implements a neural network model to generate a backbone (eg, backbone 108) for feature extraction.

[0031] Artificial intelligence (AI), including machine learning (ML), deep learning (DL), and / or other artificial machine-driven logic, enables a machine (e.g., a computer, logic circuit, etc.) to process input data using a model to generate output based on patterns and / or associations previously learned by the model via a training process. For example, a model can be trained with data to recognize patterns and / or associations, and follow such patterns and / or associations when processing input data so that other inputs result in outputs consistent with the recognized patterns and / or associations.

[0032] There are many different types of machine learning models and / or machine learning architectures. In the examples disclosed herein, a deep neural network model is used. Typically, a machine learning model / architecture suitable for the example methods disclosed herein will be based on supervised learning. However, other types of machine learning models, such as, for example, semi-supervised learning, may be used in addition or alternatively.

[0033] Typically, implementing an ML / AI system involves two phases, a learning / training phase and an inference phase. In the learning / training phase, a training algorithm is used to train the model to operate according to, for example, patterns and / or associations based on training data. Typically, the model includes internal parameters that guide how input data is converted into output data, such as through a series of nodes and connections within the model to convert input data into output data. Additionally, hyperparameters are used as part of the training process to control how learning is performed (e.g., learning rate, number of layers to be used in the machine learning model, etc.). Hyperparameters are defined as training parameters that are determined before initiating the training process.

[0034] Different types of training can be performed based on the type and / or expected output of the ML / AI model. For example, supervised training uses inputs and corresponding expected (e.g., labeled) outputs to select parameters for the ML / AI model (e.g., by iteratively selecting combinations of parameters) to reduce model error. As used herein, labeling refers to the expected output of the machine learning model (e.g., classification, expected output value, etc.). Alternatively, unsupervised training (e.g., for deep learning, subsets of machine learning, etc.) involves inferring patterns from inputs to select parameters for the ML / AI model (e.g., without the benefit of expected (e.g., labeled) outputs).

[0035] In the examples disclosed herein, training can be performed based on the early stopping principle, where training continues until the model stops improving. In the examples disclosed herein, training can be performed remotely or locally. In some examples, training can be initially performed remotely. Further training (e.g., retraining) can be performed locally based on data generated due to the execution of the model. Training is performed using hyperparameters (e.g., learning rate, the number of layers to be used in the machine learning model, etc.) that control how to perform learning. In the examples disclosed herein, hyperparameters that control the complexity, performance, duration, and / or training process of the model are used. Such hyperparameters are selected by, for example, random search and / or prior knowledge. In some examples, retraining can be performed. This retraining can be performed in response to new input data sets, drift of model performance, and / or updates to model standards and system specifications.

[0036] Training is performed using training data. In examples disclosed herein, the training data is derived from previously generated images that include the identified objects. If supervised training is used, the training data is labeled. In examples disclosed herein, labels are applied to the training data based on, for example, the number of objects in the image data. In some examples, the training data is segmented so that a portion of the data is used for validation purposes. Once training is complete, the model is stored in one or more databases (e.g., Figure 3 Database 320 and / or Figure 3in database 328, 352).

[0037] Once trained, the deployed model can operate in the reasoning phase to process data. In the reasoning phase, the data to be analyzed (e.g., real-time data) is input to the model, and the model executes to create an output. This reasoning phase can be considered as AI "thinking" to generate output based on what it has learned from training (e.g., by executing the model to apply the learned patterns and / or associations to real-time data). In some examples, the input data undergoes preprocessing before being used as input to the machine learning model. In addition, in some examples, the output data can undergo post-processing after being generated by the AI ​​model to convert the output into a useful result (e.g., a display of data, instructions to be executed by the machine, etc.). In some examples, the output of the deployed model can be captured and provided as feedback. By analyzing the feedback, the accuracy of the deployed model can be determined. If the feedback indicates that the accuracy of the deployed model is less than a threshold or other criteria, the training of the updated model can be triggered using feedback and updated training data sets, hyperparameters, etc. to generate an updated deployed model.

[0038] like Figure 3 As shown, the first computing system 325 trains a neural network to generate a backbone model based on an input image (e.g., the original image 104). The example computing system 325 includes a neural network processor 336. In the examples disclosed herein, the neural network processor 336 implements a first neural network. Figure 3 The example first computing system 325 includes a first neural network trainer 334. Figure 3 The example first neural network trainer 334 performs training of the neural network implemented by the first neural network processor 336. Figure 3 The example first computing system 325 includes a first training controller 332. The example training controller 332 instructs a first neural network trainer 334 to perform training of a neural network based on the first training data 330. Figure 3 In the example of , first training data 330 used by first neural network trainer 334 to train the neural network is stored in database 328. Figure 3 The example database 328 of the illustrated example is implemented by any memory, storage device, and / or storage disk for storing data, such as, for example, flash memory, magnetic media, optical media, etc. In addition, the data stored in the example database 328 can be in any data format, such as, for example, binary data, comma-delimited data, tab-delimited data, structured query language (SQL) structures, image data, etc. Although the illustrated example database 328 is shown as a single element, the database 328 and / or any other data storage elements described herein may be implemented by any number and / or type of memory.

[0039] exist Figure 3 In an example of , training data 330 may include image data that includes objects in different locations or positions. Training data 330 may include features extracted based on an input image. A first neural network trainer 334 uses training data 330 to train a neural network implemented by a neural network processor 336. Based on the objects in training data 330, first neural network trainer 334 trains the neural network to recognize and / or extract features associated with the input image. As a result of the neural network training, a backbone model 340 is generated. Backbone model 340 is stored in database 338. Databases 328, 338 may be the same storage device or different storage devices. Backbone generator circuit system 304 executes backbone model 340 to generate a backbone (e.g., backbone 108, 202) associated with original image 104, as combined Figure 2 In Figure 2 In some examples, the backbone generator circuit system 304 is a convolutional neural network (CNN) that includes feature extraction and / or weight calculation during the training process. In some examples, the feature extraction network associated with the backbone generator circuit system 304 includes a convolution and / or pooling layer pair.

[0040] The reference frame generator circuitry 306 groups the reference frame generator circuitry 306 based on the corner points (eg, Figure 2 The group reference box generator circuit system 306 may be used to predict the GRB based on the identified corner points. For example, the upper left corner, the lower right corner, the upper right corner, and the lower left corner may be represented as (tl x ,tl y ),(br x ,br y ), (tr x ,tr y ), and (bl x ,bl y ). GRB may be defined as being equivalent to [(x, y), w, h] (e.g., GRB = [(x, y), w, h]), where w and h represent width and height, and exist as directions (either "+" directions or "-" directions) that may be regressed at corner positions according to Equation 1, Equation 2, Equation 3, and / or Equation 4:

[0041] GRB tl = [(tl x , tl y ), + |br x -tl x |, + |bry -tl y |] Equation 1

[0042] GRB br = [(br x , br y ), - |tl x -br x |, - |tl y -br y |] Equation 2

[0043] GRB tr = [(tr x , tr y ), - |bl x -tr x |, + |bl y -tr y |] Equation 3

[0044] GRB bl = [(bl x , bl y ), + |tr x -bl x |, - |tr y -bl y |] Equation 4

[0045] In some examples, the grouped reference box generator circuit system 306 can be used to generate four two-dimensional (e.g., width and height) regression maps for the grouped reference box (e.g., using the regression map generator circuit system 310). In the examples disclosed herein, the grouped reference box generator circuit system 306 can be based on a reference box model (e.g., reference box model 364) generated using the second computing system 350, as described in more detail below. For example, during training (e.g., performed using the trainer 358), the width and height of each GRB can be trained using a smooth L1-loss. The smooth L1-loss represents a combination of an L1-loss and an L2-loss and can be used for box regression on an object detection system (e.g., using a loss function that is sensitive to outliers). In some examples, the grouped reference box generator circuit system 306 can use the regression map generator circuit system 310 and / or the heat map generator circuit system 312 to decode the GRB at each corner position based on the corner heat map and / or the regression GRB map. In the examples disclosed herein, the GRB is used to design the SG-NMS algorithm (e.g., Figure 2 For example, if corners of different identities (upper left, lower left, upper right, or lower left) belong to the same instance, their GRBs will overlap significantly, as shown in the combined Figure 2 Therefore, if combined with Figure 8 As described, SG-NMS removes any GRB-based b that has a larger overlap with the highest GRB-based score M. k (e.g., using vanilla NMS). In some examples, the overlapping intersection area between two bounding boxes divided by the total area of ​​the two bounding boxes can be used to identify an accuracy score for measuring how closely the two bounding boxes match. Unlike conventional NMS, the SG-NMS algorithm disclosed herein (e.g., in combination with Figure 8 The coordinate values ​​xmin, ymin, xmax and / or ymax in M ​​extracted from the heat map are also retained and are related to b k The estimated coordinate values ​​in the heat map are exchanged for the other coordinate values. The process is repeated recursively on the remaining GRBs. In some examples, the extracted coordinate values ​​in the heat map may be known using prior knowledge: for the upper left corner, (xmin, ymin), for the lower right corner, (xmax, ymax), for the upper left corner, (xmax, ymin), and for the lower right corner, (xmin, ymax). Therefore, grouping and NMS can be completed using a single algorithm instead of separate algorithms.

[0046] like Figure 3 As shown, the second computing system 350 trains a neural network to generate a reference frame model 364 (e.g., using four two-dimensional (e.g., width and height) regression maps for grouped reference frames). The example second computing system 350 includes a neural network processor 360. In the examples disclosed herein, the neural network processor 360 implements a second neural network. Figure 3 The second computing system 350 includes a second neural network trainer 358 . Figure 3 The second neural network trainer 358 performs training of the neural network implemented by the second neural network processor 360. Figure 3 The second computing system 350 includes a second training controller 356. The training controller 356 instructs the second neural network trainer 358 to perform training of the neural network based on the second training data 354. Figure 3 In the example of , second training data 354 used by second neural network trainer 358 to train the neural network is stored in database 352 . Figure 3The illustrated example database 352 is implemented by any memory, storage device, and / or storage disk for storing data, such as, for example, flash memory, magnetic media, optical media, etc. In addition, the data stored in the example database 352 may be in any data format, such as, for example, binary data, comma-delimited data, tab-delimited data, structured query language (SQL) structures, image data, etc. Although the illustrated example database 352 is shown as a single element, the database 352 and / or any other data storage element described herein may be implemented by any number and / or type of memory.

[0047] exist Figure 3 In the example of , training data 354 may include width and / or height data associated with a regression map of the grouped reference box. A second neural network trainer 358 trains a neural network implemented by a neural network processor 360 using training data 354. The second neural network trainer 358 trains the neural network using a smoothed L1-loss to train the width and height of each GRB. As a result of the neural network training, a reference box model 364 is generated. The reference box model 364 is stored in a database 362. Databases 352, 362 may be the same storage device or different storage devices. The grouped reference box generator 306 executes the reference box model 364 to generate a grouped reference box (e.g., Figure 2 Grouping reference box 206).

[0048] Dimension identifier circuitry 308 determines a shared distance metric (e.g., Figure 2 212). For example, the soft grouping (SG) 210 and NMS 214 stages are merged into a single stage (e.g., SG-NMS 216) and can share a distance metric calculation (e.g., shared distance measurement 212), thereby allowing the use of corner matching to be determined based on the distance metric of the corresponding GRB. In some examples, as part of the NMS algorithm 214, the dimension identifier circuit system 308 determines the distance metric based on the intersection-over-union (IoU) distance metric. For example, the IoU evaluation metric can be used to measure the accuracy of an object detector on a particular data set based on a true bounding box (e.g., specifying where the object of interest exists in the image) and a predicted bounding box (e.g., based on a model used to generate the bounding box). IoU can be identified by calculating the ratio of the overlapping area between the bounding boxes (e.g., the predicted bounding box and the true bounding box) to the area of ​​the union of the bounding boxes (e.g., including the area of ​​both the predicted bounding box and the true bounding box). Therefore, the evaluation metric can be used to reward predicted bounding boxes that overlap highly with the true bounding box.

[0049] The regression map generator circuit system 310 generates a regression map for the group reference box (GRB). Therefore, the group reference box generator circuit system 306 can use the regression map generator circuit system 310 to decode the GRB at each corner position based on the corner heat map and / or the regression GRB map. For example, the regression map generator circuit system 310 can be used by placing a fully connected layer with four neurons corresponding to the upper left and lower right (x, y) coordinates. In some examples, an S-shaped activation function can be used so that the output is returned in the range [0,1]. In addition, a loss function can be used to train the model on training data including the bounding box of the input image and the object in the image. Once trained, the bounding box regressor network can receive the input image, then perform a forward pass and predict the output bounding box coordinates of the object.

[0050] Heatmap generator circuitry 312 can be used to generate a heatmap. For example, bounding boxes can be regressed at corner locations (e.g., using regression map generator circuitry 310) and corners extracted using the heatmap. Heatmap generator circuitry 312 generates a heatmap represented by a matrix filled with values ​​from 0.0 to 1.0, where peaks on the map indicate the presence of an object. In some examples, the heatmap can be used to extract Figure 2 , where the dashed line connecting the corners 254, 258 is shown to represent the regressed group reference box (GRB).

[0051] Threshold identifier circuitry 314 determines a threshold associated with a group reference box (GRB). For example, object detector circuitry 201 removes GRB values ​​that overlap with a maximum score determined in conjunction with a given GRB, such as in conjunction with Figure 8 In some examples, the threshold identifier circuitry 314 removes any GRB-based b having a greater overlap with the highest GRB-based score M. k (e.g., using vanilla NMS). For example, the overlapping intersection area between two bounding boxes divided by the total area of ​​the two bounding boxes can be used to identify an accuracy score for measuring how closely two bounding boxes match. For example, the threshold identifier circuitry 314 can be used to detect bounding boxes with high overlap, which may correspond to the same object, so that the bounding boxes can be grouped and reduced to one box.

[0052] Output generator circuitry 316 generates the final output associated with object detector circuitry 201. For example, as combined Figure 1 to Figure 2 As shown, object detection output 164 includes a final bounding box associated with the detected object and / or identified corner points. In some examples, output generator circuitry 316 includes other metrics associated with the object detection process (eg, shared distance measurements, etc.).

[0053] Tester circuitry 318 may be used to perform linear efficiency and / or accuracy measurements. For example, tester circuitry 318 may be used to verify that the computational cost of conventional grouping and NMS is linearly reduced using the methods and apparatus disclosed herein, such as in combination with Fig. 9 In some examples, tester circuitry 318 may be used to evaluate the inference speed of a given model, which may be further used to determine algorithm efficiency.

[0054] The data storage device 320 can be used to store any information associated with the input receiver circuit system 302, the trunk generator circuit system 304, the group reference frame generator circuit system 306, the dimension identifier circuit system 308, the regression plot generator circuit system 301, the heat map generator circuit system 312, the threshold identifier circuit system 314, the output generator circuit system 316 and / or the tester circuit system 318. Figure 3 The example data storage 320 of the illustrated example may be implemented by any memory, storage device, and / or storage disk for storing data, such as flash memory, magnetic media, optical media, etc. Furthermore, the data stored in the example data storage 320 may be in any data format, such as binary data, comma-delimited data, tab-delimited data, structured query language (SQL) structures, image data, etc.

[0055] Although in Figure 3 The implementation is shown in Figure 2 The object detector circuit system 201 is an example of an Figure 3 One or more of the elements, processes and / or devices shown in the drawings may be combined, divided, rearranged, omitted, eliminated and / or implemented in any other manner. Figure 2 The example input receiver circuit system 302, the example trunk generator circuit system 304, the example grouped reference box generator circuit system 306, the example dimension identifier circuit system 308, the example regression map generator circuit system 310, the example heat map generator circuit system 312, the example threshold identifier circuit system 314, the example output generator circuit system 316, the example tester circuit system 318 and / or more generally the example object detector circuit system 201 can be implemented by hardware, software, firmware and / or any combination of hardware, software and / or firmware. Thus, for example, Figure 2Any of the example input receiver circuit system 302, the example trunk generator circuit system 304, the example grouped reference box generator circuit system 306, the example dimension identifier circuit system 308, the example regression map generator circuit system 310, the example heat map generator circuit system 312, the example threshold identifier circuit system 314, the example output generator circuit system 316, the example tester circuit system 318 and / or more generally the example object detector circuit system 201 can be implemented by a processor circuit system, an analog circuit, a digital circuit, a logic circuit, a programmable processor, a programmable microcontroller, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD) and / or a field programmable logic device (FPLD) such as a field programmable gate array (FPGA). When any apparatus or system claim of this patent is read to cover pure software and / or firmware implementations, Figure 2 At least one of the example input receiver circuit system 302, the example trunk generator circuit system 304, the example grouping reference box generator circuit system 306, the example dimension identifier circuit system 308, the example regression map generator circuit system 310, the example heat map generator circuit system 312, the example threshold identifier circuit system 314, the example output generator circuit system 316, the example tester circuit system 318 and / or more generally the example object detector circuit system 201 is expressly defined herein as including a non-transitory computer-readable storage device or storage disk, such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc., including software and / or firmware. In addition, Figure 2 The example object detector circuitry 201 may include, in addition to Figure 3 In addition to or in lieu of those shown Figure 3 One or more elements, processes and / or devices of those shown, and / or may include more than one of any or all of the shown elements, processes and devices.

[0056] Figure 4-Figure 7 The diagram shows a method for implementing Figure 2 Flowchart of an example hardware logic circuitry, machine readable instructions, hardware implemented state machine, and / or any combination thereof for object detector circuitry 201. The machine readable instructions may be a flowchart for use by a processor circuitry (such as the following in conjunction with Fig.10 The processor circuitry 1000 shown in the example processor platform 1000 discussed below and / or in conjunction with Fig.11 and / or Fig.12The program may be embodied in software stored on one or more non-transitory computer-readable storage media (such as a CD, floppy disk, hard disk drive (HDD), DVD, Blu-ray disk, volatile memory (e.g., any type of random access memory (RAM), etc.) or non-volatile memory (e.g., flash memory, HDD, etc.)) associated with the processor circuit system located in one or more hardware devices, but the entire program and / or its portion may alternatively be executed by one or more hardware devices other than the processor circuit system and / or embodied in firmware or dedicated hardware. The machine-readable instructions may be distributed across multiple hardware devices and / or executed by two or more hardware devices (e.g., a server and a client hardware device). For example, the client hardware device may be implemented by an endpoint client hardware device (e.g., a hardware device associated with a user) or an intermediate client hardware device (e.g., a radio access network (RAN) gateway that can facilitate communication between a server and an endpoint client hardware device). Similarly, the non-transitory computer-readable storage medium may include one or more media located in one or more hardware devices. In addition, although reference is made to Figure 4-Figure 7 The flowchart shown describes an example program, but many other methods of implementing the example object detector circuit system 201 may be used alternatively. For example, the execution order of the blocks may be changed, and / or some of the blocks described may be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., processor circuit systems, discrete and / or integrated analog and / or digital circuit systems, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.), which are constructed to perform corresponding operations without executing software or firmware. The processor circuit system may be distributed in different network locations and / or local to one or more hardware devices in a single machine (e.g., single-core processors (e.g., single-core central processor units (CPUs)), multi-core processors (e.g., multi-core CPUs), etc.), multiple processors distributed on multiple servers across server racks, multiple processors distributed across one or more server racks, CPUs and / or FPGAs in the same package (e.g., the same integrated circuit (IC) package or in two or more separate housings, etc.).

[0057] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a segmented format, a compiled format, an executable format, a packaged format, etc. The machine-readable instructions as described herein may be stored as data or data structures (e.g., as portions of instructions, codes, representations of codes, etc.) that may be utilized to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be segmented and stored on one or more storage devices and / or computing devices (e.g., servers) located at the same or different locations of a network or a collection of networks (e.g., in the cloud, in edge devices, etc.). The machine-readable instructions may require one or more of installation, modification, adaptation, update, combination, supplementation, configuration, decryption, decompression, unpacking, distribution, redistribution, compilation, etc., so that they may be directly read, interpreted, and / or executed by a computing device and / or other machine. For example, the machine-readable instructions may be stored in multiple parts that are individually compressed, encrypted, and / or stored on separate computing devices, where the parts, when decrypted, decompressed, and / or combined, form a machine-executable instruction set that implements one or more operations, which together may form a program such as described herein.

[0058] In another example, the machine-readable instructions may be stored in a state where they are readable by the processor circuitry, but require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., in order to execute the machine-readable instructions on a particular computing device or other device. In another example, the machine-readable instructions may need to be configured (e.g., stored settings, data inputs, recorded network addresses, etc.) before the machine-readable instructions and / or corresponding programs can be executed in whole or in part. Therefore, as used herein, a machine-readable medium may include machine-readable instructions and / or programs regardless of the particular format or state of the machine-readable instructions and / or programs when stored or otherwise at rest or in transport.

[0059] The machine-readable instructions described herein may be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, the machine-readable instructions may be represented using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0060] As mentioned above, Figure 4-Figure 7The example operations of can be implemented using executable instructions (e.g., computer and / or machine readable instructions) stored on one or more non-transitory computer and / or machine readable media, such as optical storage devices, magnetic storage devices, HDDs, flash memory, read-only memory (ROM), CDs, DVDs, caches, any type of RAM, registers, and / or any other storage device or storage disk where information is stored for any duration (e.g., extended periods of time, permanently, transiently, temporarily buffered, and / or cached information). As used herein, the terms non-transitory computer readable medium and non-transitory computer readable storage medium are expressly defined to include any type of computer readable storage device and / or storage disk, and to exclude propagating signals and to exclude transmission media.

[0061] "Including" and "comprising" (and all forms and tenses thereof) are used herein as open-ended terms. Thus, whenever a claim employs any form of "include" or "comprise" (e.g., comprises, includes, comprising, including, having, etc.) as a preamble or within any kind of claim recitation, it should be understood that additional elements, terms, etc. may be present without falling outside the scope of the corresponding claim or recitation. As used herein, when the phrase "at least" is used as a transitional term (e.g., a preamble to a claim), it is open-ended, just as the terms "comprising" and "including" are open-ended. When used, for example, in a form such as A, B, and / or C, the term "and / or" refers to any combination or subset of A, B, and C, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, or (7) A and B and C. The phrase "at least one of A and B" as used herein in the context of describing structures, components, items, objects, and / or things is intended to refer to implementations that include any of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, the phrase "at least one of A or B" as used herein in the context of describing structures, components, items, objects, and / or things is intended to refer to implementations that include any of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. The phrase "at least one of A and B" as used herein in the context of describing the execution or performance of a process, instruction, action, activity, and / or step is intended to refer to implementations that include any of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, the phrase "at least one of A or B" as used herein in the context of describing the execution or performance of a process, instruction, action, activity, and / or step is intended to refer to implementations that include any of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0062] As used herein, singular references (e.g., "a", "an", "first", "second", etc.) do not exclude a plurality. As used herein, the term "a" or "an" object refers to one or more of the object. The terms "a" (or "an"), "one or more", and "at least one" are used interchangeably herein. In addition, although listed separately, multiple units, elements, or method actions may be implemented by, for example, the same entity or object. Additionally, although individual features may be included in different examples or claims, these may be combined, and inclusion in different examples or claims does not mean that a combination of features is not feasible and / or not advantageous.

[0063] Although in Figure 3 An example method of implementing the first computing system 325 is shown in FIG. Figure 3 One or more of the elements, processes and / or devices shown in the drawings may be combined, divided, rearranged, omitted, eliminated and / or implemented in any other manner. Figure 3 The example neural network processor 336, the example trainer 334, the example training controller 332, the example databases 328, 338 and / or more generally the example first computing system 325 may be implemented by hardware, software, firmware and / or any combination of hardware, software and / or firmware. Thus, for example, Figure 3 Any of the neural network processor 336, the example trainer 334, the example training controller 332, the example databases 328, 338 and / or more generally the example first computing system 325 can be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), programmable logic devices (PLDs) and / or field programmable logic devices (FPLDs). When any device or system claim of this patent is read to cover pure software and / or firmware implementations, at least one of the neural network processor 336, the example trainer 334, the example training controller 332 and / or the example databases 328, 338 is thereby expressly defined as including a non-transitory computer-readable storage device or storage disk, such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc., including software and / or firmware. In addition, Figure 3 An example of a first computing system 325 may include in addition to Figure 3 In addition to or in lieu of those shown Figure 3One or more elements, processes and / or devices of those shown, and / or may include more than one of any or all of the elements, processes and devices shown. As used herein, the phrase "communication" (including variations thereof) encompasses direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but additionally includes selective communication at periodic intervals, scheduled intervals, non-periodic intervals and / or one-time events.

[0064] Figure 5 The diagram shows a method for implementing Figure 3 Flowchart of example hardware logic, machine readable instructions, hardware implemented state machine and / or any combination thereof of the example first computing system 325. The machine readable instructions may be a flowchart for executing by a computer processor (such as the following in combination Fig.11 The example processor platform 1100 discussed herein may be an executable program or portion of an executable program executed by the processor 1112 shown in the example processor platform 1100. The program may be embodied in software stored on a non-transitory computer-readable storage medium such as a CD-ROM, floppy disk, hard drive, DVD, Blu-ray disk, or memory associated with the processor 1112, but the entire program and / or portions thereof may alternatively be executed by a device other than the processor 1112 and / or embodied in firmware or dedicated hardware.

[0065] Although in Figure 3 An example method of implementing the second computing system 350 is shown in FIG. Figure 3 One or more of the elements, processes and / or devices shown in the drawings may be combined, divided, rearranged, omitted, eliminated and / or implemented in any other manner. Figure 3 The example neural network processor 360, the example trainer 358, the example training controller 356, the example databases 352, 362, and / or more generally the example second computing system 350 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, Figure 3Any of the example neural network processor 360, the example trainer 358, the example training controller 356, the example databases 352, 362, and / or more generally the example second computing system 350 may be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field programmable logic devices (FPLDs). When any apparatus or system claim of this patent is read to cover pure software and / or firmware implementations, at least one of the example neural network processor 360, the example trainer 358, the example training controller 356, and / or the example databases 352, 362 is thereby expressly defined to include a non-transitory computer-readable storage device or storage disk, such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc., including software and / or firmware. In addition, Figure 3 An example second computing system 350 may include in addition to Figure 3 In addition to or in lieu of those shown Figure 3 One or more elements, processes and / or devices of those shown, and / or may include more than one of any or all of the elements, processes and devices shown. As used herein, the phrase "communication" (including variations thereof) encompasses direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but additionally includes selective communication at periodic intervals, scheduled intervals, non-periodic intervals and / or one-time events.

[0066] Figure 7 The diagram shows a method for implementing Figure 3 Flowchart of example hardware logic, machine readable instructions, hardware implemented state machine and / or any combination thereof of the example second computing system 350. The machine readable instructions may be a flowchart of an ... Fig.12 The example processor platform 1200 discussed herein may be an executable program or portion of an executable program executed by the processor 1212 shown in the example processor platform 1200. The program may be embodied in software stored on a non-transitory computer-readable storage medium such as a CD-ROM, floppy disk, hard drive, DVD, Blu-ray disk, or memory associated with the processor 1212, but the entire program and / or portions thereof may alternatively be executed by a device other than the processor 1212 and / or embodied in firmware or dedicated hardware.

[0067] Figure 4 It means that it can be executed to achieve Figure 3 A flow chart of example machine readable instructions 400 of the object detector circuitry 201. Figure 4In the example of Figure 3 The input receiver circuit system 302 receives an image input (e.g., the original image 104) (block 405). When the backbone generator circuit system 304 determines that a machine learning model has been trained on key point data (e.g., Figure 3 When the backbone model 340 of FIG. 1 is generated (block 407), the backbone generator circuit system 304 generates a backbone (e.g., Figure 2 202) to perform feature extraction. For example, backbone generator circuitry 304 applies a convolutional encoder-decoder network to keypoint-based detection (block 410). However, if backbone generator circuitry 304 determines that a machine learning model (e.g., Figure 3 The backbone model 340) has not been trained, then control proceeds to Figure 3 The first computing system 325 is used to train the model to determine key points (e.g., extract features based on the input image) (block 408). Figure 3 The trained backbone model 340 uses the backbone generator circuit system 304 to extract key points, and the object detector circuit system 201 uses a soft grouping (SG) algorithm (e.g., Figure 2 Soft packet 210 and non-maximum suppression (NMS) algorithm (e.g., Figure 2 NMS214), grouping the corner key points (block 415), such as combining Figure 6 For example, object detector circuitry 201 uses regression map generator circuitry 310 and / or heat map generator circuitry 312 to use a heat map to determine corners (e.g., Figure 2 254, 258), where the corners can be connected using a regression group reference box (GRB), which is determined using the regression map generator circuit system 310. In some examples, the dimension identifier circuit system 308 can be used to determine a shared distance measurement (e.g., the shared distance measurement 212). In some examples, as part of the NMS algorithm 214, the distance metric can be based on the intersection over union (IoU) distance metric. Therefore, the IoU distance measurement can be shared between the soft grouping (SG) algorithm 210 and the NMS algorithm 214. In some examples, the threshold identifier circuit system 314 determines a threshold associated with the group reference box (GRB). For example, the object detector circuit system 201 removes GRB values ​​that overlap with the maximum score determined in conjunction with a given GRB, such as in conjunction with Figure 3 The output generator circuitry 316 may be used to output a final image (e.g., object detection output 164) including a bounding box identifying each of the objects within the image (block 425), such as in combination with Figure 2 shown.

[0068] Figure 5 is a flow chart representing example machine readable instructions 408 that are executed by Figure 2 The object detector circuitry, when executed, causes the object detector circuitry 201 to train a neural network to determine keypoints as part of a convolutional encoder-decoder network for keypoint-based detection. Figure 5 In the example of , trainer 334 accesses training data 330 (block 505). Training data 330 may include image data that includes extracted features (e.g., identified key points). Trainer 334 identifies data features represented by training data 330 (e.g., data features from which key points are extracted) (block 510). Training controller 332 instructs trainer 334 to perform training of a neural network using training data 330 to generate backbone model 340 (block 515). In some examples, additional training is performed to refine model 340 (block 520).

[0069] Figure 6 is a flow chart representing example machine readable instructions 415 that may be executed to use Figure 3 The object detector circuitry 201 uses a soft grouping (SG) algorithm and non-maximum suppression (NMS) to group corner keypoints. Figure 6 In the example of FIG. 1 , the group reference box generator circuitry 306 determines a group reference box (GRB) (eg, Figure 2 GRB 206) (block 605). Regression map generator circuitry 310 and / or heat map generator circuitry 312 may be used to generate a regression map and / or heat map of grouped reference boxes (GRBs) (block 610). In some examples, grouped reference box generator circuitry 306 determines whether a machine learning model has been trained on width and / or height data associated with the GRBs (block 612). If training has been performed, grouped reference box generator circuitry 306 may be used to extract minimum and / or maximum coordinates from the heat map (block 620). If training has not been performed, control passes to second computing system 350 to perform training to determine the width and / or height of each GRB (block 615), as described in conjunction with Figure 7 Once the group reference frame generator circuitry 306 extracts the minimum and / or maximum coordinates from the heat map determined using the heat map generator circuitry 312 (block 620), as described in conjunction with Figure 3As described, object detector circuitry 201 performs corner matching using the distance metric of the corresponding GRB (block 625). For example, as part of NMS algorithm 214, dimension identifier circuitry 308 determines shared distance measurement 212 based on an intersection-over-union (IoU) distance metric. Thus, dimension identifier circuitry 308 shares the distance metric between SG 210 and NMS 214 algorithms (block 630).

[0070] Figure 7 is a flow chart representing example machine readable instructions 615, which are executed by Figure 2 The object detector circuitry 201 when executed causes the object detector circuitry 201 to train a neural network to determine a width and / or height of a group reference box (GRB) as part of a convolutional encoder-decoder network for keypoint-based detection. Figure 7 In the example of , trainer 358 accesses training data 354 (block 705). Training data 354 may include identification dimensions of one or more GRBs. Trainer 358 identifies data features represented by training data 354 (block 710). Training controller 356 instructs trainer 358 to perform training of a neural network using training data 354 to generate reference frame model 364 (block 715). In some examples, additional training is performed to refine model 364 (block 720).

[0071] Figure 8 800 are shown representing example steps of an algorithm 800 for soft grouping (SG) and non-maximum suppression (NMS) soft object detection disclosed herein. Figure 8 In the example of FIG. 8 , the SG-NMS algorithm 800 receives an input including a group box in an image for a category (eg, β = {b1, b2, b3, ... b N}), where each box includes an identifier associated with the location of a given corner at which the grouping box is extracted (e.g., using a heatmap-based identifier of top-left corner, bottom-right corner, top-right corner, bottom-left corner). For example, the identifier and maximum / minimum coordinates associated with the corner point can be defined as b = [id, xmin, ymin, xmax, ymax]. In addition, the input to the SG-NMS algorithm 800 includes a set of scores S associated with the bounding boxes, an intersection-over-union (IoU) metric, and a grouping score threshold (τ i and τ g ), as shown in line 805 of the usage example code. Figure 8In the example of , the output of the SG-NMS algorithm 800 is a detection bounding box (L) with a score S. The output shown using example code line 810 includes an iterative loop (e.g., shown using example code lines 810, 815, 820). For example, an iterative loop can be performed when all grouping boxes in an image for a class are not evaluated (e.g., β≠EMPTY). As previously described, the SG-NMS algorithm 800 removes any GRB-based b that has a greater overlap with the highest GRB-based score M. k (e.g., using vanilla NMS). Unlike conventional NMS, the SG-NMS algorithm 800 also retains the coordinate values ​​xmin, ymin, xmax, and / or ymax in M ​​extracted from the heatmap and compares them with b k The estimated coordinate values ​​in the example code lines 810, 815, and 820 are exchanged for other coordinate values, as shown in conjunction with the example code lines 810, 815, and 820. Compared to vanilla NMS, the SG-NMS algorithm 800 relies on a few code line changes without large computational overhead. For example, if not all four corners of an object are incorrectly estimated, SG-NMS as a grouping algorithm will not miss the object. Even three of the four corners can provide challenging samples to evaluate, and when only one corner is effectively predicted, SG-NMS retains one GRB as a detection result to prevent missing the object. If any corners with different position IDs are effectively estimated, the two corresponding coarse GRBs can be turned into finer detection boxes by SG-NMS.

[0072] Fig. 9 An example graph 900 is shown illustrating object detection performance using anchor-based and anchor-free methods, including soft grouping (SG) and non-maximum suppression (NMS) soft object detection disclosed herein. Fig. 9In the example of , the selected evaluation method may include object detectors, such as anchor-based object detectors 910, anchor-free object detectors (dense regression, DR) 912, anchor-free object detectors (keypoint-based, KB) 914, and / or SGCDet 916. Object detectors 910, 912, 914, 916 include a backbone 902, an input resolution 904, a first set of test results 906, and a second set of test results 908. For example, the SG-NMS algorithm can be added to CornerNet by replacing the original grouping and NMS operators to produce SGCDet 916. SGCDet 916 can be compared with state-of-the-art corner-based detectors 910, 912, 914. In order to better measure the performance of the SGCDet 916 model, multiple classic detectors can be compared in other existing types (e.g., anchor-based to anchor-free). SGCDet 916 uses the smallest input resolution and achieves 46.7% average precision (AP) without any tricks in single-scale testing, outperforming all reported anchor-free detectors and even comparable to advanced anchor-based detectors. Meanwhile, SGCDet 916 produces 50.2% AP 75 , indicating that the generated detection boxes are more accurate than using other known object detectors. For example, AP 50 and AP 75 The data corresponds to the IoU threshold between the detected bounding box and the ground-truth bounding box defined by the MS-COCO dataset (e.g., a large-scale object detection, segmentation, and captioning dataset). For small objects, SGCDet 916 also achieves improved accuracy, AP S Reached 27.8%. For example, AP S For small object detection, AP M corresponds to medium object detection, and AP LCorresponding to large object detection. These experimental results demonstrate the effectiveness of the SGCDet 916 algorithm described in this article. In particular, SGCDet 916 achieves a significant improvement of 6.2% in AP compared to the CornerNet baseline. Example inference speed comparison 918 (e.g., measured using frames per second (FPS)) with other example keypoint-based detectors 920, 922, 924, 926 also shows improvements when the SGCDet 916 algorithm is used (e.g., using the Hourglass-104 backbone for all models). For example, the speed of SGCDet 916 with Hourglass-104 (e.g., the disclosed net 928) is 9.1FPS, which is much faster than other known keypoint-based detectors 920, 922, 924, 926 (e.g., three times the speed of ExtremeNet and CenterNet and CentripetalNet, and more than twice the speed of the CornerNet baseline). Additionally, known models 920, 922, 924, 926 treat grouping as an additional stage in post-processing and use flipped image augmentation by default in inference to improve its accuracy (e.g., as combined with Figure 1 The design of these known models 920, 922, 924, 926 thus affects the speed of inference, resulting in low efficiency.

[0073] Fig.10 is constructed to execute and / or instantiate Figure 4 and / or Figure 6 Machine readable instructions and / or operations to implement Figure 2 1 is a block diagram of an example processor platform 1000 of the object detector circuit system 201. The processor platform 1000 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, an iPad, etc.). TM tablet computer), personal digital assistant (PDA), Internet appliance, DVD player, CD player, digital video recorder, Blu-ray player, game console, personal video recorder, set-top box, headsets (e.g., augmented reality (AR) headsets, virtual reality (VR) headsets, etc.) or other wearable device, or any other type of computing device.

[0074] The processor platform 1000 of the illustrated example includes a processor circuit system 1012. The processor circuit system 1012 of the illustrated example is hardware. For example, the processor circuit system 1012 can be implemented by one or more integrated circuits, logic circuits, FPGA microprocessors, CPUs, GPUs, DSPs and / or microcontrollers from any desired series or manufacturers. The processor circuit system 1012 can be implemented by one or more semiconductor-based (e.g., silicon-based) devices. In this example, the processor circuit system 1012 implements the input receiver circuit system 302, the trunk generator circuit system 304, the grouping reference frame generator circuit system 306, the dimension identifier circuit system 308, the regression graph generator circuit system 310, the heat map generator circuit system 312, the threshold identifier circuit system 314, the output generator circuit system 316 and / or the tester circuit system 318.

[0075] The processor circuit system 1012 of the illustrated example includes a local memory 1013 (e.g., cache, registers, etc.). The processor circuit system 1012 of the illustrated example communicates with a main memory including a volatile memory 1014 and a non-volatile memory 1016 via a bus 1018. The volatile memory 1014 may be comprised of a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 1016 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1014, 1016 of the illustrated example is controlled by a memory controller 1017.

[0076] The processor platform 1000 of the illustrated example also includes an interface circuit system 1020. The interface circuit system 1120 may be implemented by hardware according to any type of interface standard, such as an Ethernet interface, a universal serial bus (USB) interface, interface, near field communication (NFC) interface, PCI interface and / or PCIe interface.

[0077] In the example shown, one or more input devices 1022 are connected to the interface circuitry 1020. The input devices 1022 allow a user to enter data and / or commands into the processor circuitry 1012. The input devices 1022 may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, an isopoint device, and / or a voice recognition system.

[0078] One or more output devices 1024 are also connected to the interface circuit system 1020 of the illustrated example. The output device 1024 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT) display, an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit system 1020 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics processor circuit system such as a GPU.

[0079] The interface circuitry 1020 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces to facilitate the exchange of data with external machines (e.g., any type of computing device) over the network 1026. Communications may occur over, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, an optical connection, etc.

[0080] The processor platform 1000 of the illustrated example also includes one or more mass storage devices 1028 for storing software and / or data. Examples of such mass storage devices 1028 include magnetic storage devices, optical storage devices, floppy disk drives, HDDs, CDs, Blu-ray disk drives, redundant array of independent disks (RAID) systems, solid-state storage devices such as flash memory devices, and DVD drives.

[0081] Can be Figure 4 and / or Figure 6 The machine-executable instructions 1032 implemented by the machine-readable instructions may be stored in the mass storage device 1028, in the volatile memory 1014, in the non-volatile memory 1016, and / or on a removable non-transitory computer-readable storage medium (such as a CD or DVD).

[0082] Fig.11 is constructed to execute Figure 5 Instructions to achieve Figure 3 The processor platform 1100 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, an iPad, etc.). TM tablet computer), personal digital assistant (PDA), Internet appliance, or any other type of computing device.

[0083] The processor platform 1100 of the illustrated example includes a processor 1112. The processor 1112 of the illustrated example is hardware. For example, the processor 1112 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired series or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements the example neural network processor 336, the example trainer 334, and the example training controller 332.

[0084] The processor 1112 of the illustrated example includes a local memory 1113 (e.g., a cache). The processor 1112 of the illustrated example communicates with a main memory including a volatile memory 1114 and a non-volatile memory 1116 via a bus 1118. The volatile memory 1114 may be comprised of a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 1116 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1114, 1116 is controlled by a memory controller.

[0085] The processor platform 1100 of the illustrated example also includes an interface circuit 1120. The interface circuit 1120 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), interface, a near field communication (NFC) interface, and / or a PCI Express interface.

[0086] In the example shown, one or more input devices 1122 are connected to the interface circuit 1120. The input devices 1122 allow a user to enter data and / or commands into the processor 1112. The input devices may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a track pad, a trackball, a roller stick (isopoint), and / or a voice recognition system.

[0087] One or more output devices 1124 are also connected to the interface circuit 1120 of the illustrated example. The output device 1124 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit 1120 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0088] The interface circuitry 1120 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces to facilitate the exchange of data with external machines (e.g., any type of computing device) via the network 1126. Communications may occur via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, and the like.

[0089] The processor platform 1100 of the illustrated example also includes one or more mass storage devices 1128 for storing software and / or data. Examples of such mass storage devices 1128 include floppy disk drives, hard disk drives, optical disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0090] Figure 5 The machine-executable instructions 408 may be stored in the mass storage device 1128, in the volatile memory 1114, in the non-volatile memory 1116, and / or on a removable, non-transitory computer-readable storage medium such as a CD or DVD.

[0091] Fig.12 is constructed to execute Figure 7 Instructions to achieve Figure 3 The processor platform 1200 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, an iPad, etc.). TM tablet computer), personal digital assistant (PDA), Internet appliance, or any other type of computing device.

[0092] The processor platform 1200 of the illustrated example includes a processor 1212. The processor 1212 of the illustrated example is hardware. For example, the processor 1212 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired series or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements the example neural network processor 360, the example trainer 358, and the example training controller 356.

[0093] The processor 1212 of the illustrated example includes a local memory 1213 (e.g., a cache). The processor 1212 of the illustrated example communicates with a main memory including a volatile memory 1214 and a non-volatile memory 1216 via a bus 1218. The volatile memory 1214 may be comprised of a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 1216 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1214, 1216 is controlled by a memory controller.

[0094] The processor platform 1200 of the illustrated example also includes an interface circuit 1220. The interface circuit 1220 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), interface, a near field communication (NFC) interface, and / or a PCI Express interface.

[0095] In the example shown, one or more input devices 1222 are connected to the interface circuit 1220. The input devices 1222 allow a user to enter data and / or commands into the processor 1212. The input devices may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, a scroll bar (isopoint), and / or a voice recognition system.

[0096] One or more output devices 1224 are also connected to the interface circuit 1220 of the illustrated example. The output device 1224 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit 1220 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0097] The interface circuitry 1220 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces to facilitate the exchange of data with external machines (e.g., any type of computing device) via the network 1226. Communications may occur via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, and the like.

[0098] The processor platform 1200 of the illustrated example also includes one or more mass storage devices 1228 for storing software and / or data. Examples of such mass storage devices 1228 include floppy disk drives, hard disk drives, optical disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0099] Figure 7 The machine-executable instructions 615 may be stored in the mass storage device 1228, in the volatile memory 1214, in the non-volatile memory 1216, and / or on a removable, non-transitory computer-readable storage medium such as a CD or DVD.

[0100] Fig.13 yes Fig.10 , Fig.11 , Fig.12 1012, 1112, 1312 of the processor circuit system 1012, 1112, 1312 of the example implementation. In this example, Fig.10 , Fig.11 , Fig.12 The processor circuit systems 1012, 1112, 1312 are implemented by the microprocessor 1300. For example, the microprocessor 1300 can implement a multi-core hardware circuit system such as a CPU, DSP, GPU, XPU, etc. Although it can include any number of example cores 1302 (for example, 1 core), the microprocessor 1300 of this example is a multi-core semiconductor device including N cores. The cores 1302 of the microprocessor 1300 can operate independently or can cooperate to execute machine-readable instructions. For example, machine code corresponding to a firmware program, an embedded software program, or a software program can be executed by one of the cores 1302 or can be executed by multiple cores in the cores 1302 at the same or different times. In some examples, the machine code corresponding to the firmware program, the embedded software program, or the software program is split into threads and executed in parallel by two or more cores 1302. The software program may correspond to a thread executed by Figure 4 , Figure 5 , Figure 6 and / or Figure 7 A flowchart may represent a portion or all of machine-readable instructions and / or operations.

[0101] The core 1302 can communicate by an example bus 1304. In some examples, the bus 1304 can implement a communication bus to implement communications associated with one of the cores 1302. For example, the bus 1304 can implement at least one of an inter-integrated circuit (I2C) bus, a serial peripheral interface (SPI) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, the bus 1304 can implement any other type of computing or electrical bus. The core 1302 can obtain data, instructions, and / or signals from one or more external devices through an example interface circuit system 1306. The core 1302 can output data, instructions, and / or signals to one or more external devices through the interface circuit system 1306. Although the core 1302 of this example includes an example local memory 1320 (e.g., a level 1 (L1) cache that can be divided into an L1 data cache and an L1 instruction cache), the microprocessor 1300 also includes an example shared memory 1310 that can be shared by the core (e.g., a level 2 (L2_cache)) to access data and / or instructions at high speed. Data and / or instructions may be transferred by writing to and / or reading from the shared memory 1310. The local memory 1320 of each of the cores 1302 and the shared memory 1310 may be a memory including multiple levels of cache memory and main memory (e.g., Fig.10 Typically, higher-level memories in the hierarchy exhibit lower access times and have smaller storage capacities than lower-level memories. Changes to the various levels of the cache hierarchy are managed (eg, coordinated) by a cache coherence policy.

[0102] Each core 1302 may be referred to as a CPU, DSP, GPU, etc., or any other type of hardware circuit system. Each core 1302 includes a control unit circuit system 1314, an arithmetic and logic (AL) circuit system (sometimes referred to as an ALU) 1316, a plurality of registers 1318, an L1 cache 1320, and an example bus 1322. Other structures may exist. For example, each core 1302 may include a vector unit circuit system, a single instruction multiple data (SIMD) unit circuit system, a load / store unit (LSU) circuit system, a branch / jump unit circuit system, a floating point unit (FPU) circuit system, etc. The control unit circuit system 1314 includes a semiconductor-based circuit that is configured to control (e.g., coordinate) data movement within the corresponding core 1302. The AL circuit system 1316 includes a semiconductor-based circuit that is configured to perform one or more mathematical and / or logical operations on data within the corresponding core 1302. Some examples of the AL circuit system 1316 perform integer-based operations. In other examples, the AL circuit system 1316 also performs floating-point operations. In other examples, AL circuitry 1316 may include a first AL circuitry that performs integer-based operations and a second AL circuitry that performs floating-point operations. In some examples, AL circuitry 1316 may be referred to as an arithmetic logic unit (ALU). Registers 1318 are semiconductor-based structures for storing data and / or instructions, such as the results of one or more operations performed by AL circuitry 1316 of corresponding core 1302. For example, registers 1318 may include vector registers, SIMD registers, general registers, flag registers, segment registers, machine-specific registers, instruction pointer registers, control registers, debug registers, memory management registers, machine check registers, etc. Registers 1318 may be arranged such as Fig.13 13. Alternatively, registers 1318 may be organized in any other arrangement, format, or structure, including being distributed throughout core 1302 to reduce access time. Bus 1322 may implement at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0103] Each core 1302 and / or more generally the microprocessor 1300 may include additional and / or alternative structures to those structures shown and described above. For example, there may be one or more clock circuits, one or more power supplies, one or more power gates, one or more cache home agents (CHA), one or more convergence / common grid stops (CMS), one or more shifters (e.g., barrel shifters) and / or other circuit systems. The microprocessor 1300 is a semiconductor device manufactured to include many transistors, which are interconnected to implement the above structure in one or more integrated circuits (ICs) contained in one or more packages. The processor circuit system may include one or more accelerators and / or collaborate with one or more accelerators. In some examples, the accelerator is implemented by a logic circuit system to perform specific tasks faster and / or more efficiently than a general-purpose processor can achieve. Examples of accelerators include ASICs and FPGAs, such as those discussed herein. A GPU or other programmable device may also be an accelerator. The accelerator may be onboard the processor circuit system, in the same chip package as the processor circuit system, and / or in one or more packages separated from the processor circuit system.

[0104] Fig.14 yes Fig.10 , Fig.11 , Fig.12 1400. The FPGA circuit system 1400 can be used, for example, to perform operations that can be performed by other methods. Fig.13 The example microprocessor 1300 executes the corresponding machine-readable instructions to perform operations. However, once configured, the FPGA circuit system 1400 instantiates the machine-readable instructions in hardware, and therefore, can generally perform operations faster than a general-purpose microprocessor executing corresponding software.

[0105] More specifically, Fig.13 The microprocessor 1300 (the microprocessor 1300 is a general-purpose device that can be programmed to perform the Figure 4 , Figure 5 , Figure 6 and / or Figure 7 Compared to a flowchart that may contain some or all of the machine-readable instructions, but whose interconnections and logic circuitry are fixed once manufactured, Fig.14 The example FPGA circuit system 1400 includes interconnect and logic circuit systems, which can be configured and / or interconnected in different ways after manufacturing to instantiate, for example, Figure 4 , Figure 5 , Figure 6and / or Figure 7 In particular, FPGA 1400 can be thought of as an array of logic gates, interconnects, and switches. The switches can be programmed to change how the logic gates are interconnected by the interconnects, effectively forming one or more dedicated logic circuits (unless and until FPGA circuit system 1400 is reprogrammed). The configured logic circuits enable the logic gates to cooperate in different ways to perform different operations on data received by the input circuit system. These operations can correspond to the operations performed by Figure 4 , Figure 5 , Figure 6 and / or Figure 7 Thus, FPGA circuit system 1400 can be constructed to efficiently Figure 4 , Figure 5 , Figure 6 and / or Figure 7 Some or all of the machine-readable instructions of the flowchart of the FPGA are instantiated into a dedicated logic circuit that performs the operations corresponding to those software instructions in a dedicated manner similar to an ASIC. Figure 4 , Figure 5 , Figure 6 and / or Figure 7 A general-purpose microprocessor can more quickly execute operations corresponding to some or all of the machine-readable instructions.

[0106] exist Fig.14 In the example of FPGA circuit system 1400, FPGA circuit system 1400 is configured to be programmed (and / or reprogrammed one or more times) by an end user via a hardware description language (HDL) such as Verilog. Fig.14 FPGA circuit system 1400 includes example input / output (I / O) circuit system 1402 to obtain data from example configuration circuit system 1404 and / or external hardware (e.g., external hardware circuit system) 1406, and / or to output data to example configuration circuit system 1404 and / or external hardware (e.g., external hardware circuit system) 1406. For example, configuration circuit system 1404 can implement an interface circuit system that can obtain machine-readable instructions to configure FPGA circuit system 1400 or a portion thereof. In some such examples, configuration circuit system 1404 can obtain machine-readable instructions from a user, a machine (e.g., a hardware circuit system (e.g., a programmed or dedicated circuit system) that can implement an artificial intelligence / machine learning (AI / ML) model to generate instructions), etc. In some examples, external hardware 1406 can implement Fig.13FPGA circuit system 1400 also includes an array of example logic gate circuit system 1408, a plurality of example configurable interconnects 1410, and example storage circuit system 1412. Logic gate circuit system 1408 and interconnect 1410 may be configured to instantiate a Figure 4 , Figure 5 , Figure 6 and / or Figure 7 One or more operations of at least some of the machine-readable instructions and / or other desired operations. Fig.14 The logic gate circuit system 1408 shown is manufactured in groups or blocks. Each block includes a semiconductor-based electrical structure that can be configured into a logic circuit. In some examples, the electrical structure includes a logic gate (e.g., an AND gate, an OR gate, a NOR gate, etc.) that provides a basic building block for the logic circuit. Within each of the logic gate circuit systems 1408, there is an electrically controllable switch (e.g., a transistor) to enable the electrical structure and / or logic gate to be configured to form a circuit to perform the desired operation. The logic gate circuit system 1408 may include other electrical structures, such as a lookup table (LUT), a register (e.g., a flip-flop or latch), a multiplexer, etc.

[0107] The interconnect 1410 of the illustrated example is a conductive path, trace, via, etc., which may include an electrically controllable switch (e.g., a transistor) whose state can be changed by programming (e.g., using an HDL instruction language) to activate or deactivate one or more connections between one or more logic gate circuit systems 1408 to program a desired logic circuit.

[0108] The storage circuit system 1412 of the illustrated example is configured to store the results of one or more operations performed by the corresponding logic gates. The storage circuit system 1412 may be implemented by registers, etc. In the illustrated example, the storage circuit system 1412 is distributed between the logic gate circuit system 1408 to facilitate access and increase execution speed.

[0109] Fig.14The example FPGA circuit system 1400 also includes an example special operation circuit system 1414. In this example, the special operation circuit system 1414 includes a special circuit system 1416, which can be called to implement common functions to avoid the need to program these functions in the field. Examples of such special circuit systems 1416 include memory (e.g., DRAM) controller circuit systems, PCIe controller circuit systems, clock circuit systems, transceiver circuit systems, memory, and multiplier-accumulator circuit systems. There may be other types of special circuit systems. In some examples, the FPGA circuit system 1400 may also include an example general programmable circuit system 1418, such as an example CPU 1420 and / or an example DSP 1422. Additionally or alternatively, there are other general programmable circuit systems 1418 that can be programmed to perform other operations, such as GPUs, XPUs, etc.

[0110] although Fig.13 and Fig.14 Shows Fig.10 , Fig.11 and / or Fig.12 These are two example implementations of the processor circuitry 1012, 1112, 1212 of FIG. 1 , but many other approaches are contemplated. For example, as described above, modern FPGA circuitry may include an onboard CPU, such as Fig.14 One or more of the example CPUs 1420. Thus, Fig.10 , Fig.11 and / or Fig.12 The processor circuit systems 1012, 1112, 1212 may additionally be combined Fig.13 An example microprocessor 1300 and Fig.14 In some such hybrid examples, the Figure 4 , Figure 5 , Figure 6 and / or Figure 7 The first portion of the machine-readable instructions represented by the flowchart may be represented by Fig.13 1302 and is executed by one or more of the cores Figure 4 , Figure 5 , Figure 6 and / or Figure 7 The second portion of the machine-readable instructions represented by the flowchart may be represented by Fig.14 FPGA circuit system 1400 executes.

[0111] In some examples, Fig.10 , Fig.11 and / or Fig.12The processor circuit systems 1012, 1112, 1212 may be in one or more packages. For example, Fig.13 The processor circuit system 1300 and / or Fig.14 The FPGA circuit system 1400 can be in one or more packages. In some examples, the XPU can be composed of Fig.10 , Fig.11 and / or Fig.12 The processor circuit system 1012, 1112, 1212 of the embodiment of the present invention can be implemented in one or more packages. For example, the XPU can include a CPU in one package, a DSP in another package, a GPU in another package, and an FPGA in another package.

[0112] exist Fig.15 The diagram shows a method for converting Fig.10 , Fig.11 , Fig.12 1032, 1132, 1232 to a hardware device owned and / or operated by a third party. The example software distribution platform 1505 can be implemented by any computer server, data facility, cloud service, etc. that can store software and send software to other computing devices. The third party can be a customer of the entity that owns and / or operates the software distribution platform 1505. For example, the entity that owns and / or operates the software distribution platform 1505 can be a developer, seller, and / or licensor of the software, such as Fig.10 , Fig.11 and / or Fig.12 The third party may be a consumer, user, retailer, OEM, etc., who purchases and / or licenses the software for use and / or resale and / or sub-license. In the example shown, the software distribution platform 1505 includes one or more servers and one or more storage devices. The storage device stores the machine-readable instructions 1032, 1132, 1232, which may correspond to the above-described Figure 4 , Figure 5 , Figure 6 and / or Figure 7The example machine-readable instructions 400, 408, 415, 615 of the example software distribution platform 1505 are in communication with a network 1510, which may correspond to any one or more of the Internet and / or any of the example networks described above. In some examples, one or more servers respond to a request to send software to a requesting party as part of a commercial transaction. The delivery, sale, and / or payment of a license for the software may be processed by one or more servers of the software distribution platform and / or a third-party payment entity. The server enables a purchaser and / or licensor to download machine-readable instructions 1032, 1132, 1232 from the software distribution platform 1505. For example, the server may correspond to Figure 4 , Figure 5 , Figure 6 and / or Figure 7 The software of the example machine readable instructions 400, 408, 415, 615 can be downloaded to the example processor platform 1000, 1100, 1200, which is used to execute the machine readable instructions 1032, 1132, 1232 to implement the object identifier circuit system 201. In some examples, one or more servers of the software distribution platform 1505 periodically send software (e.g., Fig.10 , Fig.11 , Fig.12 Example machine readable instructions 1032, 1132, 1232) provide, send and / or force updates to ensure that improvements, patches, updates, etc. are distributed and applied to software at end-user devices.

[0113] In light of the foregoing, it should be understood that example systems, methods, devices, and articles of manufacture have been disclosed that use a simplified pipeline to introduce small object detection in images and / or videos to improve object detection efficiency. In the examples disclosed herein, the grouping and NMS stages are merged into a single stage and / or distance metric calculations can be shared. Additionally, the methods and devices disclosed herein allow for different numbers of corner points to improve object detection accuracy. In the examples disclosed herein, soft grouping non-maximum suppression (SG-NMS) can be used for object detection by merging soft grouping with NMS into a single stage. Therefore, instead of using two separate distance metric calculations, distance calculations can be shared between the SG and NMS stages to improve efficiency. In addition, a flexible number of estimated corners (e.g., 1 to 4) can be used, while existing methods of object detection based on key points can depend on a fixed pair of two corners. Using the methods and devices disclosed herein, the average accuracy of object detection is significantly improved.

[0114] Example methods and apparatus for small object detection in images and videos are disclosed herein. Further examples and combinations thereof include the following:

[0115] Example 1 includes a device for object detection, comprising at least one memory, instructions in the device, and a processor circuit system, wherein the processor circuit system is used to execute the instructions to: receive an input image; identify a first grouping reference frame representing a first object in the input image, the first grouping reference frame being based on feature extraction performed using a feature extractor network; extract first and second coordinates of a corner position from a heat map, the heat map being used to determine the first grouping reference frame; generate a second grouping reference frame representing the first object based on the corner position; and when the corner position of the first grouping reference frame exceeds a corner position threshold of the second grouping reference frame, update the first grouping reference frame using the second grouping reference frame.

[0116] Example 2 includes the apparatus of Example 1, wherein the feature extractor network is a convolutional encoder-decoder network for keypoint based detection.

[0117] Example 3 includes the apparatus of Example 1, wherein, when the corner position includes a first corner position and a second corner position, the processor circuit system is used to group the first corner position and the second corner position using a soft grouping (SG) algorithm and a non-maximum suppression (NMS) algorithm.

[0118] Example 4 includes the apparatus of Example 3, wherein the processor circuitry is to determine a distance metric corresponding to the first grouping reference frame and the second grouping reference frame, the distance metric being shared between the SG algorithm and the NMS algorithm.

[0119] Example 5 includes the apparatus of Example 4, wherein the distance metric is an intersection-over-union (IoU) distance metric determined as part of an NMS algorithm.

[0120] Example 6 includes the apparatus of Example 1, wherein the processor circuitry is to train the reference box model to determine a width and a height of the second grouping reference box.

[0121] Example 7 includes the apparatus of Example 6, wherein the processor circuitry is to generate regression maps for the second grouped reference frames, the regression maps being four two-dimensional regression maps identified using smoothed L1 training of the reference frame model.

[0122] Example 8 includes a method for object detection, the method comprising: receiving an input image; identifying a first grouping reference box for a first object representation in the input image, the first grouping reference box based on feature extraction performed using a feature extractor network; extracting first coordinates and second coordinates for a corner position from a heat map, the heat map being used to determine the first grouping reference box; generating a second grouping reference box for the first object representation based on the corner position; and when the corner position of the first grouping reference box exceeds a corner position threshold of the second grouping reference box, updating the first grouping reference box using the second grouping reference box.

[0123] Example 9 includes the method of Example 8, wherein the feature extractor network is a convolutional encoder-decoder network for keypoint based detection.

[0124] Example 10 includes the method of Example 8, wherein, when the corner position includes a first corner position and a second corner position, further comprising grouping the first corner position and the second corner position using a soft grouping (SG) algorithm and a non-maximum suppression (NMS) algorithm.

[0125] Example 11 includes the method of Example 10, further comprising determining a distance metric corresponding to the first grouping reference box and the second grouping reference box, the distance metric being shared between the SG algorithm and the NMS algorithm.

[0126] Example 12 includes the method of Example 11, wherein the distance metric is an intersection-over-union (IoU) distance metric determined as part of an NMS algorithm.

[0127] Example 13 includes the method of Example 8, further comprising training the reference box model to determine a width and a height of the second grouping reference box.

[0128] Example 14 includes the method of Example 13, further comprising generating regression maps for a second grouping reference box, the regression maps being four two-dimensional regression maps identified using smoothed L1 training of the reference box model.

[0129] Example 15 includes a non-transitory computer-readable storage medium comprising instructions that, when executed, cause a processor to at least: receive an input image; identify a first grouping reference box of a first object representation in the input image, the first grouping reference box being based on feature extraction performed using a feature extractor network; extract first and second coordinates of a corner position from a heat map, the heat map being used to determine the first grouping reference box; generate a second grouping reference box of the first object representation based on the corner position; and update the first grouping reference box using the second grouping reference box when the corner position of the first grouping reference box exceeds a corner position threshold of the second grouping reference box.

[0130] Example 16 includes the non-transitory computer readable storage medium of Example 15, wherein the instructions, when executed, cause the processor to group the first corner position and the second corner position using a soft grouping (SG) algorithm and a non-maximum suppression (NMS) algorithm.

[0131] Example 17 includes the non-transitory computer readable storage medium of Example 16, wherein the instructions, when executed, cause the processor to determine a distance metric corresponding to the first grouping reference frame and the second grouping reference frame, the distance metric being shared between the SG algorithm and the NMS algorithm.

[0132] Example 18 includes the non-transitory computer readable storage medium of Example 15, wherein the instructions, when executed, cause the processor to train the reference box model to determine a width and a height of the second grouping reference box.

[0133] Example 19 includes the non-transitory computer readable storage medium of Example 18, wherein the instructions, when executed, cause the processor to generate regression maps for a second grouped reference frame, the regression maps being four two-dimensional regression maps identified using smoothed L1 training of the reference frame model.

[0134] Example 20 includes the non-transitory computer-readable storage medium of Example 15, wherein the feature extractor network is a convolutional encoder-decoder network for keypoint based detection.

[0135] Although certain example systems, methods, apparatus, and articles of manufacture have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all systems, methods, apparatus, and articles of manufacture fairly falling within the scope of the claims of this patent.

[0136] The following claims are hereby incorporated into this Detailed Description of the Invention by reference, with each claim standing on its own as a separate embodiment of the present disclosure.

Claims

1. A device for object detection, comprising: at least one memory; instructions in said device; as well as A processor circuit system is configured to execute the instructions to perform the following operations: receiving an input image; identifying first grouped reference boxes for first object representations in the input image, the first grouped reference boxes based on feature extraction performed using a feature extractor network; extracting a first coordinate and a second coordinate for a corner position from a heat map, wherein the heat map is used to determine the first grouping reference frame; generating a second grouping reference box for the first object representation based on the corner positions; as well as When the corner position of the first grouping reference frame exceeds a corner position threshold of the second grouping reference frame, the first grouping reference frame is updated using the second grouping reference frame.

2. The device according to claim 1, wherein: The feature extractor network is a convolutional encoder-decoder network for keypoint based detection.

3. The device according to claim 1, wherein: When the corner positions include a first corner position and a second corner position, the processor circuit system is configured to group the first corner position and the second corner position using a soft grouping SG (SG) algorithm and a non-maximum suppression NMS (NMS) algorithm.

4. The device according to claim 3, wherein: The processor circuitry is configured to determine a distance metric corresponding to the first grouped reference frame and the second grouped reference frame, the distance metric being shared between the SG algorithm and the NMS algorithm.

5. The device according to claim 4, wherein: The distance metric is the Intersection over Union (IoU) distance metric determined as part of the NMS algorithm.

6. The device according to claim 1, wherein: The processor circuitry is used to train a reference frame model to determine a width and a height of the second grouping reference frame.

7. The device according to claim 6, wherein: The processor circuitry is configured to generate regression maps for the second grouping of reference frames, the regression maps being four two-dimensional regression maps identified using smoothed L1 training of the reference frame model.

8. A method for object detection, the method comprising: receiving an input image; identifying first grouped reference boxes for first object representations in the input image, the first grouped reference boxes based on feature extraction performed using a feature extractor network; extracting a first coordinate and a second coordinate for a corner position from a heat map, wherein the heat map is used to determine the first grouping reference frame; generating a second grouping reference box for the first object representation based on the corner positions; as well as When the corner position of the first grouping reference frame exceeds a corner position threshold of the second grouping reference frame, the first grouping reference frame is updated using the second grouping reference frame.

9. The method according to claim 8, wherein: The feature extractor network is a convolutional encoder-decoder network for keypoint based detection.

10. The method according to claim 8, wherein: When the corner position includes a first corner position and a second corner position, the method further includes grouping the first corner position and the second corner position using a soft grouping SG (SG) algorithm and a non-maximum suppression NMS (NMS) algorithm.

11. The method of claim 10, further comprising determining a distance metric corresponding to the first grouping reference frame and the second grouping reference frame, the distance metric being shared between the SG algorithm and the NMS algorithm.

12. The method according to claim 11, wherein: The distance metric is the Intersection over Union (IoU) distance metric determined as part of the NMS algorithm.

13. The method of claim 8, further comprising training a reference box model to determine a width and a height of the second grouping reference box.

14. The method of claim 13, further comprising generating regression maps for the second grouped reference frames, the regression maps being four two-dimensional regression maps identified using smoothed L1 training of the reference frame model.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed, cause a processor to at least perform the following operations: receiving an input image; identifying first grouped reference boxes for first object representations in the input image, the first grouped reference boxes based on feature extraction performed using a feature extractor network; extracting a first coordinate and a second coordinate for a corner position from a heat map, wherein the heat map is used to determine the first grouping reference frame; generating a second grouping reference box for the first object representation based on the corner positions; as well as When the corner position of the first grouping reference frame exceeds a corner position threshold of the second grouping reference frame, the first grouping reference frame is updated using the second grouping reference frame.

16. The non-transitory computer-readable storage medium of claim 15, wherein: The instructions, when executed, cause the processor to group the first corner position and the second corner position using a soft grouping SG (SG) algorithm and a non-maximum suppression NMS (NMS) algorithm.

17. The non-transitory computer-readable storage medium of claim 16, wherein: The instructions, when executed, cause a processor to determine a distance metric corresponding to the first grouping reference frame and the second grouping reference frame, the distance metric being shared between the SG algorithm and the NMS algorithm.

18. The non-transitory computer-readable storage medium of claim 15, wherein: The instructions, when executed, cause the processor to train a reference box model to determine a width and a height of the second grouping reference box.

19. The non-transitory computer-readable storage medium of claim 18, wherein: The instructions, when executed, cause the processor to generate regression maps for the second grouping of reference frames, the regression maps being four two-dimensional regression maps identified using smoothed L1 training of the reference frame model.

20. The non-transitory computer-readable storage medium of claim 15, wherein: The feature extractor network is a convolutional encoder-decoder network for keypoint based detection.

Images