IMPROVED IMAGE AND VIDEO OBJECT RECOGNITION USING A MULTI-STAGE PARADIGM

Abstract

This disclosure describes systems, methods, and devices relating to object recognition in images. A device can input an image representing an object into a manual labeling learning system; using the system, identify the first coordinates of an upper-left corner of a bounding box representing the object, based on a heatmap indicating the probability that the first coordinates represent the upper-left corner; use the system to identify the second coordinates of a lower-right corner of the bounding box, based on the first coordinates and a first distance regression map indicating the coordinate differences between the second coordinates and ground-truth coordinates, which are input into the machine learning model as training data.Using the system, generate adjustments to the first and second coordinates based on a second distance regression map; and using the system, generate the bounding box with the adjusted first and second coordinates.

Description

TECHNICAL FIELD

[0001] This disclosure relates generally to apparatus, systems, and methods for image processing, and more particularly to improved object detection in images using a multi-stage paradigm. STATE OF THE ART

[0002] Object detection in images and videos is a popular task in computer vision and is widely used in various applications, such as intelligent transportation, smart retail, robotics, and aerospace. There are different ways to classify existing methods, such as anchor-based, anchor-free, center-guided, and corner-guided. However, some object detection engines do not generate high-quality bounding boxes, which limits their application in scenarios where high-quality bounding boxes are required, such as dense vehicle localization in intelligent transportation applications. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1A illustrates an exemplary manual labeling learning system for enhanced object recognition that simulates manual object labeling, according to some embodiments of the present disclosure. Fig. 1B shows an example of the machine learning backbone from Fig. 1A according to some embodiments of the present disclosure. Fig. 2 shows exemplary processes for object detection according to some embodiments of the present disclosure. Fig. 3A is an example image with multiple bounding boxes used for object detection, according to some example embodiments of the present disclosure. Fig. 3B is an example image with multiple bounding boxes used for object detection, according to some example embodiments of the present disclosure. Fig. 4 illustrates an exemplary process for multi-phase manual labeling learning for enhanced object recognition that simulates manual object labeling, according to some exemplary embodiments of the present disclosure. Fig. 5 illustrates a flow diagram of an illustrative process for multi-phase manual label learning for improved object recognition that simulates manual object labeling, according to one or more embodiments of the present disclosure. Fig. 6 illustrates an embodiment of an example system according to one or more example embodiments of the present disclosure. DETAILED DESCRIPTION

[0003] The following description and drawings illustrate specific embodiments sufficiently to enable those skilled in the art to implement them. Other embodiments may incorporate structural, logical, electrical, process, algorithmic, and other changes. Portions and features of some embodiments may be included in, or substituted for, those of other embodiments. Embodiments recited in the claims include all available equivalents of those claims.

[0004] Image and video processing techniques (here referred to as "image processing"), such as computer vision, often rely on bounding boxes as reference points for an object represented by an image. Bounding boxes can be generated based on the x and y coordinates of the upper left and lower right corners of the bounding box (e.g., a corner-guided method) or on the x and y coordinates of the center of a bounding box (e.g., a center-guided method). Modern object detection units are typically center-guided or corner-guided. Center-guided detectors are the most active branch in the object detection community, such as Faster R-CNN (region-to-region convolutional neural networks) and DETR (detection transformer). They define a set of center positions (e.g., points / regions) as positive samples to directly derive the heights and widths of target objects via regression.Corner-guided detectors estimate and extract corner keypoints in heatmaps to decode object frames.

[0005] It is not easy to accurately locate the center point of an object box. This is because the center point of a bounding box must be determined by all four boundaries of the instance (requiring four degrees of freedom). This makes it difficult to generate high-quality detection boxes using this center-guided cluster-free approach, especially toward dense objects, small objects, or occlusions. Compared with a center-guided approach, corner-guided methods are more capable of finding a precise bounding box. However, the current corner-guided approach requires complex post-processing, such as clustering, which results in high computational cost and many false positives.

[0006] The accuracy of a bounding box is important for object detection. For example, incorrectly identifying an object's position due to an inaccurate bounding box can influence decisions that control vehicle operation. In particular, a high-quality bounding box is critical in dense or occluded localization scenarios (e.g., a traffic scenario with multiple cars that may not be visible in their entirety to a vehicle camera).

[0007] Although some modern object detection engines are not good at generating high-quality bounding boxes, humans can accurately label a bounding box. Empirically, humans perform two steps to manually position an object bounding box: 1) clicking the mouse in the upper left corner of an object and then dragging the mouse to the lower right corner; and 2) refining the corner positions to make the bounding box more precise if necessary.

[0008] Object recognition can therefore be improved by using machine learning to simulate a human labeling process using a multi-stage paradigm.

[0009] In one or more embodiments, improved object detection in images using a multi-stage paradigm may be referred to as a manual label learning (MLL) unit. The MLL may be a human-like object detection unit with two stages to simulate the two manual steps described above. Compared to state-of-the-art detection units, the MLL provides a novel paradigm with high bounding box quality. Our method is evaluated on the Microsoft Common Objects in Context (COCO) dataset. The MLL far outperforms state-of-the-art methods (e.g., DETR / CornerNet) with lower computational cost, especially for dense or occluded objects. In particular, compared to FasterRCNN, which represents a baseline for two-stage detection units, the MLL provides a 12.3% improvement in mean average precision mAP@90 on COCO.MLL can be used in object detection applications such as smart transportation, retail, surveillance, and the like.

[0010] In one or more embodiments, MLL may represent a two-stage recognition engine that simulates the two steps in manual labeling. In the first stage, MLL simulates the click and drag operations based on three output feature maps: a heat map for estimating the top-left corner coordinate, an offset map for refining the corner, and a distance map for determining the relative distance from the bottom-right corner to the top-left corner via regression. In the second stage, MLL may extract RoIs (Region of Intersect) from frame proposals generated in the first stage to refine the top-left and bottom-right corners. The first stage outputs three feature maps (e.g., modern recognition engines typically output four or six feature maps in comparison), and the second stage outputs two feature maps.

[0011] In one or more embodiments, the MLL may use a keypoint estimation method to model the click-and-drag operation. The MLL applies a heatmap to predict and accurately determine the upper-left corner. After locating the upper-left position, the next procedure is to "drag" the mouse from the upper-left position to the lower-right position. The MLL may apply distance regression to output the route of the "drag." In particular, the MLL may regress Δx and Δy pairs at each upper-left coordinate that point to the lower-right coordinate. Empirically, it is difficult to label the bounding box sufficiently accurately using a one-time operation. The MLL may therefore include a second step to adjust the bounding box boundary.In the second stage, the MLL can first extract RoI features from proposed bounding boxes generated in the first stage. Afterward, the MLL can use N cascaded blocks to refine the upper-left and lower-right corners.

[0012] In one or more embodiments, there are three inputs in the first stage of the MLL: (A) Upper-left corner heatmap: The MLL predicts a heatmap to estimate keypoints of the upper-left corner. Each pixel value in the output heatmap represents the confidence of being evaluated as an upper-left corner, which is defined as: Gk=exp(−(x−xm)2+(y−ym)2f(object size)2) where (x, y) is a coordinate in the heatmap. (x m , y m ) is the upper left corner of the ground-truth object. The MLL can use focal loss with distance handicap to learn the heatmap.

[0013] In one or more embodiments, the second input in the first stage of the MLL is: (b) Offset map of the upper left corner offset: Due to discretization during downsampling, multiple pixels in large maps are projected onto the same pixel in small maps. Thus, a remapping is added to adjust the upper left position. The MLL may use an offset regression method in which a convolutional neural network predicts heatmaps to represent positions of corners of different image objects and predicts embedding vectors for each corner such that the distance between two embeddings of two corners of the same object is small. The neural network may also predict offsets to slightly adjust the positions of the corners. Using the predicted heatmaps, embeddings, and offsets, the MLL may apply a post-processing algorithm to generate bounding boxes.

[0014] In one or more embodiments, the second input in the first stage of the MLL is: (c) Lower-right corner distance regression map: To model the "drag" mouse procedure during manual labeling, the MLL predicts the relative position for each lower-right corner to the corresponding upper-left corner position. The MLL may apply distance regression to the mapped grid center of the upper-left corner in FPNs (Feature Pyramid Networks). The regression targets may be the coordinate differences (Δx, Δy) between two corners, which are the width and height of the corresponding object: (Δx,Δy)=(xbr−xm,ybr−ym) where (x br , y br ) is the coordinate of a lower right corner and (x m , y m) is a ground-truth position (e.g., a mapped grid center in the feature map). In this part, MLL can use the GIoU (Generalized Intersection over Union) loss as a training objective, with the goal of regularizing objects of different sizes with equal intensity: Lr=1N∑i=1NGIoU(frame(Δx,Δy),frame(Δ^x,Δ^y)) where (Δ̂x, Δ̂y) is a predicted relative distance of the lower right corner and (Δx, Δy) is ground truth (e.g., training frames labeled as ground truth frames), frame(·) represents a frame constructed by the corresponding upper left / lower right corners, and N is the total number of object instances. Thus, by selecting the lower right corner based on the upper left corner instead of selecting the two corners individually, the MLL method simulates a user's manual click-and-drag operations and improves the accuracy of selecting the two corners individually.

[0015] In one or more embodiments, the GIoU loss of equation (3) above may be replaced with another loss function (e.g., IoU loss, etc.). The loss function may have the inputs of an estimated bounding box from the MLL and a ground-truth bounding box, and the output may be a score indicating an overlap between the estimated bounding box and the ground-truth bounding box.

[0016] In one or more embodiments, the use of heatmaps and vector embeddings to predict the top-left and bottom-right corners of a bounding box may utilize the concept of corner pooling. Since a bounding box corner may be external to an object (e.g., not part of the actual object), to determine whether an top-left corner exists at a given location (e.g., x and y coordinates), the MLL may search horizontally from the top-left corner candidate to the right for the top boundary of the object and may search vertically downward for the leftmost boundary of the object. A corner pooling layer of the convolutional neural network may input two feature maps.At each pixel position, the corner pooling layer can max-pool the feature vectors to the right of the first feature map, max-pool the feature vectors directly below from the second feature map, and then add the two pooled results together.

[0017] In one or more embodiments, the purpose of the second stage of the MLL is to refine the upper-left and lower-right corners. The second stage also provides category confidence for each detection box. The MLL may use N cascaded (e.g., N = 3) tiny convolution blocks to gradually improve the quality of the output bounding box. In each block, the MLL may apply both regression and classification, as in equation (4) below: {δ˜x=x*−xΔx;δ˜y=y*−yΔyΔ˜x=log(Δx*Δx);Δ˜y=log(Δx*Δx) where (δ̃x, δ̃y) and (Δ̃x, Δ̃y) are the refinements of the upper left and lower right corners.

[0018] In one or more embodiments, during inference, the MLL may define the detection frame score as the geometric mean of the localization score and class score: rating=s(x,y)⋅s(class)m where s(x,y) is a localization score output in the first stage of the upper left corner heatmap and s(class) m represents a classification confidence output in the second stage.

[0019] In one or more embodiments, the MLL may combine the Lr defined in equation (3) with the Smooth-L1 loss to train the second stage: loss=αLh−tl+βLh−br+γLr where α, β, and γ are balancing weights determined experimentally. In an implementation, the MLL can set them to 1, 1, and 0.001, respectively. L h-tl and L h-brare the smooth L1 losses for the upper left and lower left offsets defined in equation (4): Lh−tl=1N∑i=1NsmoothL1(oi,tl,ol,tl˜) Lh−br=1N∑i=1NsmoothL1(oi,br,ol,br˜) where N is the number of positive samples during training, o i,tl = (δx, δy) is the ground truth offset, ol,tl˜=(δ˜x,δ˜y) is the estimated offset of the upper left corner, o i,br = (Δx, Δy) is the ground truth offset and ol,br˜=(Δ˜x,Δ˜y) is the offset of the lower right corner estimated by stage two.

[0020] The performance of MLL was compared with other detectors using a COCO dataset. As shown in Table 1 below, MLL achieves a mAP of 51.6% under the ResNeXt-101 backbone, which outperforms state-of-the-art methods. Specifically, MLL improves mAP@50 by 4.9% compared to the corner-guided CentripetalNet, which enjoys the status of the current optimal corner grouping algorithm. Furthermore, MLL still achieves a higher mAP@50 compared to the center-guided Deformable DETR. Compared to FasterRCNN, which represents a baseline for two-stage detectors, MLL improves mAP@90 by 9.8% and 12.3% with different backbones. There is reason to believe that the accuracy at higher IoUs (mAP75:95) of MLL far exceeds Deformable DETR, which means that MLL can yield higher quality detection frames by directly finding the top left corner position.Furthermore, compared to the mature two-stage detection unit Cascade R-CNN, MLL increases the mAP@75 by 1.9%, further demonstrating that directly estimating corners is better for setting the boundary of an object than centers. Table 1: MLL compared to other recognition units regarding inference speed when using COCO: Verfahren: FPS AP AP50 AP75 AP90 RetinaNet-R50 12 38,7 58,0 41,5 Faster R-CNN-R50 16 40,2 59,8 43,8 11,3 DETR-R50 7 43,3 63,1 45,9 Sparse R-CNN-R50 14 42,8 61,2 45,7 MLL-R50 23 43,9 60,6 48,3 20,1 Sparse R-CNN-R101 10 44,1 62,1 47,2 DETR-R101 5 44,9 64,7 47,7 MLL-R101 18 47,1 65,0 51,6 23,6 (FPS: Frames pro Sekunde). (AP: Durchschnittspräzedenz - höhere AP bedeutet einen besseren Begrenzungsrahmen).

[0021] The above descriptions are illustrative and not limiting. Numerous other examples, configurations, processes, algorithms, etc., may exist, some of which are described in more detail below. Example embodiments will now be described with reference to the accompanying figures.

[0022] Fig. 1A shows an exemplary multi-phase manual labeling learning system 100 for enhanced object recognition that simulates manual object labeling, according to some embodiments of the present disclosure.

[0023] With reference to Fig. 1A, the multi-stage machine learning system 100 may include an input image 102 (e.g., representing an object such as an eagle, as shown) that is input to an optional ML backbone 103 (e.g., as further described with reference to Fig. 1B). The ML backbone 103 may be a multi-layer network for analyzing image features of the input image 102 and may generate image features 104 as an output. For example, the image features 104 generated by the ML backbone 103 may include detected objects in the input image 102 and their positions in the input image 102, along with other features. Alternatively, the image features 104 may be input based on another analysis, manually or automatically (e.g., using computer vision or other object detection techniques). The image features 104 may be input to the multi-layer manual labeling learning system 100. The input image 102 may be acquired from a camera (e.g., an I / O device 692 of Fig. 6). Stage 1 of the multi-stage manual labeling learning system 100 may include heatmap modules 105 for creating a heatmap for the input image 102 to simulate manual click and drag operations. The heatmap modules 105 may predict and accurately determine an upper left corner of a bounding box used to identify the object in the input image 102 according to equation (1) above (e.g., as in Fig. 2). For example, the value for each point on the heatmap may represent the probability that the upper left corner of the bounding box for the object is located at that position (e.g., a position score). Then, the offset modules 106 of the manual labeling learning system 100 may generate an offset map to refine the upper left corner of the bounding box (e.g., as shown in Fig. 4). The regression modules 108 of the multi-stage manual labeling learning system 100 may generate a regression map to model the dragging procedure in manual labeling, predicting the relative position for a lower right corner of the bounding box of the object shown in the input image 102. The regression map may apply equations (2) and (3) above. The heat map, offset map, and regression map may be input to the bounding box modules 109 to generate estimated bounding boxes by applying the offset map to the estimated upper left corner from the heat map, and then applying the regression map to estimate the lower right corner coordinates of the bounding box.

[0024] Still referring to Fig. 1A, the multi-stage manual labeling learning system 100 may include ROI alignment modules 110 to provide ROI alignment of proposed bounding boxes generated using stage 1. The ROI alignment modules 110 may extract ROIs from proposed boxes generated using stage 1 (e.g., the estimated bounding boxes of the bounding box modules 109). The ROI alignment may generate bounding box features 11 that may be refined by refinement modules 112. Then, the multi-stage manual labeling learning system 100 may receive N cascaded blocks from a convolution head of the refinement modules 112 (e.g.,multiple convolutional heads of a neural network, where each head can represent a convolutional block, where both the regression map and object classification can be applied to the bounding box generated from the upper left corner (generated by the heatmap modules 105) to refine the upper left and lower right corners of any bounding boxes generated by stage 1, respectively. Stage 2 can provide category confidence for each detection box. The N cascaded blocks can be tiny convolutional blocks to gradually improve the quality of the output bounding box. In each block, the multi-stage manual labeling learning system 100 can apply both regression and classification with post-processing modules 114 to generate predicted bounding boxes 116.During inference, the multi-stage manual labeling learning system 100 may calculate the detected bounding box score as a geometric mean of a position score and a class score (e.g., 95% confident that the object represented by the input image 102 is a bird in . Fig. 1A) using equation (5) above, using the upper left and lower right corner coordinates from the heatmap (e.g., x1, y1 representing the x and y coordinates of the upper left corner, and x2, y2 representing the x and y coordinates of the lower right corner). The predicted bounding boxes 116 may be an output score according to equation (5) above.

[0025] Still referring to Fig. 1A, ground-truth frames 150 may be used as training to generate heatmaps, offsets, regression, and ground-truth refinement. The heatmaps, offsets, regression, and ground-truth refinement may be inputs to a loss function 154, which, when applied to the heatmaps, offsets, regression, and ground-truth refinement, may result in a heatmap loss function 156, an offset loss function 158, a regression loss function 160, and a refinement loss function 162. The heatmap loss function 156, the offset loss function 158, the regression loss function 160, and the refinement loss function 162 may be training data inputs to the multi-stage manual labeling learning system 100.

[0026] For example, the heatmap loss function 156 can be input to the heatmap modules 105, the offset loss function 158 can be input to the offset modules 106, and the refinement loss function 162 can be input to the regression modules 108 to train the respective modules. The refinement loss function 162 can also be input to the refinement modules 112 as training data.

[0027] Due to discretization during downsampling, in one or more embodiments, multiple pixels in large maps are projected onto the same pixel in small maps. Thus, a remapping is added to adjust the top left position. The MLL may use an offset regression method in which a convolutional neural network (e.g., represented by the multi-stage manual label learning system 100) predicts heatmaps to represent positions of vertices of different image objects and predicts embedding vectors for each vertex such that the distance between two embeddings of two vertices of the same object is small. The neural network may also predict offsets to slightly adjust the positions of the vertices.

[0028] In one or more embodiments, using the heatmaps and vector embeddings to predict the top-left and bottom-right corners of a bounding box may utilize the concept of corner pooling. Since a bounding box corner may be located outside an object (e.g., not part of the actual object), to determine whether an top-left corner exists at a given location (e.g., x and y coordinates), the MLL may search horizontally from the top-left corner candidate to the right for the top boundary of the object and may search vertically downward for the leftmost boundary of the object. A corner pooling layer (e.g., one of the convolution heads of refinement modules 112) of the convolutional neural network may input two feature maps.At each pixel position, the corner pooling layer can max-pool the feature vectors to the right of the first feature map, max-pool the feature vectors directly below from the second feature map, and then add the two pooled results together.

[0029] In one or more embodiments, the backbone 103 may be a ResNeXt-101 backbone (with, for example, 101 layers), an R50 backbone (with, for example, 50 layers), or another type of backbone capable of extracting features from the pixels of the input image 102. The features may be used to generate the embedding vectors used by the heatmap modules 105 to generate corner candidates for the bounding box, which may be refined by the offset modules 106 and the regression modules 108. As shown above in Table 1, the more layers in the backbone 103, the better the multi-stage machine learning structure 100 generally performs, especially for higher AP n -Metrics. The multi-stage machine learning structure 100 can be trained with bounding boxes labeled as ground truth and with bounding boxes for various labeled objects.

[0030] Fig. 1B shows an example of the machine learning backbone 103 from Fig. 1A according to some embodiments of the present disclosure.

[0031] With reference to Fig. 1B, the input image 102 can be Fig. 1A into the backbone 103, which may include multiple feature layers (e.g., C N and P Nrepresent corresponding feature layers). The fractions (e.g., 1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32, 1 / 64, 1 / 28, etc.) represent the magnification of the input image 102 in the respective layer. For example, at layer C1, the input image 102 may be magnified by 1 / 2, and so on, to generate the output image features 104. As shown, the machine learning backbone 103 is an R50 or ResNeXt 101 backbone, but other backbones may be used to generate the output image features 104. The backbone 103 may be trained using a dataset of images with classified objects (e.g., images with annotations that identify the objects).

[0032] Fig. 2 shows exemplary processes for object detection according to some embodiments of the present disclosure.

[0033] With reference to Fig. 2, a center-guided bounding box generation process 200 and a corner-guided bounding box generation process 210 are shown in combination. As shown, the center-guided process 200 may search for the center of the object represented by the input image 102 of Fig. 1A (e.g., a bird) by defining a set of center positions as positive sample values ​​to directly determine the widths and heights (e.g., W / H) of target objects using regression. The corner-guided process 210 may estimate and extract corner keypoints (e.g., upper left corner 212 and lower right corner 214) in heatmaps to decode object bounding boxes. Accurately generating the output 220 may be difficult because the center of a bounding box determined by the center-guided process 200 may need to be determined by all four boundaries, and the corner-guided process 210 may require complex post-processing.

[0034] Still referring to Fig. 2, a simulated manual labeling process 250 (e.g., MLL using the multi-stage machine learning structure 100 of Fig. 1A) a click 252 on the upper left corner of the object and a drag operation 254 to the lower right corner of the object and then refining 256 the estimated bounding box (e.g., using the multi-stage machine learning structure 100 of Fig. 1A). Due to the grouping of the center-guided process 200 and the corner-guided process 210 from Fig. 2, the simulated manual labeling process 250 can improve bounding box detection because the lower right corner can be determined based on the upper left corner as the starting point for a simulated drag gesture (e.g., instead of using separate heatmaps to estimate the upper left and lower right corners separately). The refinement 256 (e.g., stage 2 of Fig. 1A) may use N cascaded convolution blocks of the multi-stage machine learning framework 100 to adjust the upper left and lower right corners. For each convolution block, refinement 256 may involve applying regression and classification. For example, the regression may involve a distance regression from an upper left or lower right corner to the x,y coordinate differences between the two corners (e.g., according to equation (2) above).

[0035] Fig. 3A is an example image 300 with multiple bounding boxes used for object detection, according to some example embodiments of the present disclosure.

[0036] With reference to Fig. In Figure 3A, the image represents a dog, and several true positive bounding boxes are shown with different metrics. A ground truth box 304 is shown along with a detected bounding box 306 and a detected bounding box 308. For example, the intersection of union (IOU) of the detected bounding box 306 and the ground truth box 304 may be approximately 0.75, as shown, and the IOU of the bounding box 308 and the ground truth box 304 may be approximately 0.9, as shown. Therefore, the detected bounding box 306 and the detected bounding box 308 may be considered true positives at mAP@50 (e.g., an IOU threshold with the ground truth box 304 of 0.5), but only the detected bounding box 308 may be considered a true positive at mAP@90.

[0037] Fig. 3B is an example image 350 having multiple bounding boxes used for object detection, according to some example embodiments of the present disclosure.

[0038] With reference to Fig. 3B, image 350 is an example of a dense and occluded region. Bounding box 352 and bounding box 354 may be ground truth frames of two vehicles, and frame 356 may be a detected frame (e.g., detected for the two vehicles) due to the occlusion. The IOU between frame 356 and each of bounding box 352 and bounding box 354 may be greater than 0.5, but frame 356 is not the true position of any of the vehicles. The occlusion caused by the density in image 350 results in frame 356 being inaccurate as ground truth, and therefore bounding box 352 and bounding box 354 may be inaccurate.

[0039] Referring to Fig. 3A and Fig. 3B is the need for MLL as the multi-level machine learning structure 100 from Fig. 1A to improve the way a bounding box is generated for object detection.

[0040] With reference to Fig. 3B, the multi-stage machine learning structure 100 may be Fig. 1A (and the corresponding manual marking process 250 from Fig. 2 and the process 400 from Fig. 4) improve the accuracy and speed of bounding box detection even for the type of occluded image data used in Fig. 3B. Table 1 above shows the improvement of the MLL technique described herein compared to other detection units.

[0041] Fig. 4 shows an example process 400 for multi-phase machine learning for improved object recognition that simulates manual object labeling, according to some example embodiments of the present disclosure.

[0042] With reference to Fig. 4, the process 400 may comprise the multi-stage machine learning structure 100 Fig. 1A in more detail. The process 400 may determine an offset 402 of an upper left corner 404 (e.g., from the click step of Fig. 1A) using regression. The regression 406 may determine the relative distance (Δx, Δy) of a lower right corner 407 from the upper left corner 404. The process 400 may include an adjustment 408 to refine the upper left corner 404 and the lower right corner 407. The upper left corner 404 may be determined according to equation (1) above.

[0043] The offset 402 may compensate for how multiple pixels on a large map may map to an equal pixel in a smaller map by remapping the pixels to adjust the upper left corner 404. The regression 406 may apply equations (2) and (3) above, and the adjustment 408 may involve applying equation (4) above to refine the upper left corner 404 and the lower right corner 407.

[0044] In one or more embodiments, because a bounding box corner may be located outside of an object (e.g., not part of the actual object), the process 400, to determine whether an upper-left corner exists at a given location (e.g., x and y coordinates), may search horizontally from the upper-left corner candidate to the right for the top boundary of the object and may search vertically downward for the leftmost boundary of the object. A corner pooling layer (e.g., one of the convolution heads of the refinement modules 112 of Fig. 1A) The convolutional neural network can input two feature maps. At each pixel position, the corner pooling layer can perform max-pooling on the feature vectors to the right of the first feature map, max-pooling on the feature vectors directly below it from the second feature map, and then add the two pooled results together.

[0045] Fig. 5 illustrates a flow diagram of an illustrative process 500 for multi-phase machine learning for improved object recognition that simulates manual object labeling, according to one or more embodiments of the present disclosure.

[0046] At block 502, a device (e.g., the object device 619 of Fig. 6) an image (e.g. the input image 102 from Fig. 1A) into a machine learning model (e.g. the multi-stage machine learning structure 100 from Fig. 1A). The image may represent one or more objects that are recognizable using an image recognition technique (e.g., computer vision or another technique) that may rely on using a bounding box to represent an object in the image. The machine learning model may be trained using labeled ground truth coordinates and bounding boxes corresponding to different objects to generate a bounding box for any object represented by the input image.

[0047] At block 504, the device may use the machine learning model to determine first coordinates (e.g., x,y coordinates) of an upper left corner (e.g., coordinates of the upper left corner 404 from Fig. 4) to identify a bounding box for an object in the image. The machine learning model can generate a heatmap for an image in which the heatmap values ​​correspond to probabilities (e.g., a probability for each coordinate position in the heatmap) that the location is the upper left corner of the bounding box for the object.

[0048] At block 506, the device may use the machine learning model to determine second coordinates of a lower right corner of the bounding box (e.g., the coordinates of the lower right corner 407 of Fig. 4) Identify based on the first coordinates. Instead of using the heatmap to predict the coordinates of the lower right corner and the coordinates of the upper left corner separately (e.g., using a corner- and / or center-based technique), the machine learning model can use a distance regression map to model a "pulling" procedure. The machine learning model can predict embedding vectors for each corner, so that the distance between two embeddings of two corners of the same object is small. The neural network can also predict offsets to slightly adjust the positions of the corners. Using the predicted heatmaps, embeddings, and offsets, the MLL can apply a post-processing algorithm to generate bounding boxes.

[0049] At block 508, the device may use the machine learning model to adjust (e.g., refine) the first and second coordinates. The machine learning model may use a second distance regression map to determine the distance between the ground truth coordinates and the first and second coordinates using regression, resulting in a refinement of the first and second coordinates to be used for the bounding box at block 510. For example, the machine learning model may be trained using equation (6) above.

[0050] It is understood that the above descriptions are for illustrative purposes and are not intended to be limiting.

[0051] Fig. 6 illustrates an embodiment of an example system 600 according to one or more example embodiments of the present disclosure.

[0052] In various embodiments, the computing system 600 may include or be implemented as part of an electronic device.

[0053] For example, in some embodiments, computing system 600 may be representative of a computer system that includes one or more components of Fig. 1-5 and / or performs steps of the processes in these figures.

[0054] The embodiments are not limited in this context. More generally, the computing system 600 is configured to implement all of the methods described herein and with respect to Fig. 1-3B and 5, all systems, processes, logic flows, methods, devices, and all functionality described herein and with respect to these figures.

[0055] System 600 may be a multi-core computer system, such as a distributed computing system, a supercomputer, a high-performance computing system, a computing cluster, a mainframe computer, a minicomputer, a client-server system, a personal computer (PC), a workstation, a server, a portable computer, a laptop computer, a tablet computer, a handheld device such as a personal digital assistant (PDA), or other devices for processing, displaying, or transmitting information. Similar embodiments may include, for example, entertainment devices such as a portable music player or video player, a smartphone or other mobile phone, a telephone, a digital video camera, a digital still camera, an external storage device, or the like. Other embodiments implement larger server configurations.In other embodiments, system 600 may include a single processor with one core or more than one processor. Note that the term "processor" refers to a single-core processor or a processor package with multiple processor cores.

[0056] In at least one embodiment, the computing system 600 represents one or more components of Fig. 3A and Fig. 3B. More generally, the computing system 600 is configured to implement all of the logic, systems, processes, logic flows, methods, facilities, and functionality described herein with reference to the above figures.

[0057] As used in this application, the terms "system," "component," and "module" are intended to refer to a computer-associated entity, either hardware, a combination of hardware and software, software, or executing software, examples of which are provided by the example system 600. For example, a component may be a process running on a processor, a processor, a hard disk drive, multiple storage drives (of an optical and / or magnetic storage medium), an object, an executable program, a thread of execution, a program, and / or a computer.

[0058] To illustrate, both an application running on a server and the server may be a component. One or more components may reside within a process and / or thread of execution, and a component may be localized on one computer and / or distributed between two or more computers. Further, components may be communicatively coupled to one another through various types of communication media to coordinate operations. Coordination may involve the exchange of information in one or both directions. For example, the components may communicate information in the form of signals communicated over the communication media. The information may be implemented as signals associated with various signal lines. In such associations, each message is a signal. However, other embodiments may alternatively employ data messages.Such data messages can be sent over various connections. Examples include parallel interfaces, serial interfaces, and bus interfaces.

[0059] As shown in this figure, the system 600 includes a motherboard 605 for mounting platform components. The motherboard 605 is a point-to-point interconnect platform that includes a processor 610, a processor 630 coupled via point-to-point interconnects as an Ultra Path Interconnect (UPI), and an enhanced object detection device 619 (e.g., capable of performing the functions of Fig. 1-5). In other embodiments, system 600 may have a different bus architecture, such as a multi-drop bus. Furthermore, each of processors 610 and 630 may be processor packages with multiple processor cores. As an example, processors 610 and 630 are shown as including one or more processor cores 620 and 640, respectively. While system 600 is an example of a two-socket (2S) platform, other embodiments may include more than two sockets or one socket. For example, some embodiments may include a four-socket (4S) platform or an eight-socket (8S) platform. Each socket is a receptacle for a processor and may have a socket identifier. It is noted that the term platform refers to the motherboard with certain mounted components, such as processors 610 and chipset 660.Some platforms may include additional components, and some platforms may only include sockets for attaching the processors and / or chipset.

[0060] The 610 and 630 processors may be any of various commercially available processors, including, but not limited to, Intel® Celeron®, Core®, Core (2) Duo®, Itanium®, Pentium®, Xeon®, and XScale® processors; AMD® Athlon®, Duron®, and Opteron® processors; ARM® application, embedded, and security processors; IBM® and Motorola® DragonBall® and PowerPC® processors; IBM and Sony® Cell processors, and similar processors. Dual microprocessors, multi-core processors, and other multiprocessor architectures may also be employed as the 610 and 630 processors.

[0061] Processor 610 includes an integrated memory controller (IMC) 614 and point-to-point (PP) interfaces 618 and 652. Similarly, processor 630 includes an IMC 634 and PP interfaces 638 and 654. IMCs 614 and 634 couple processors 610 and 630, respectively, to respective memories, a memory 612 and a memory 632. Memories 612 and 632 may be portions of the main memory (e.g., a dynamic random access memory (DRAM)) for the platform, such as double data rate Type 3 (DDR3) or Type 4 (DDR4) synchronous DRAM (SDRAM). In the present embodiment, memories 612 and 632 are attached locally to the respective processors 610 and 630.

[0062] In addition to processors 610 and 630, system 600 may include an object recognizer 619. Object recognizer 619 may be connected to chipset 660 via PP interfaces 629 and 669. Object recognizer 619 may also be connected to memory 639. In some embodiments, object recognizer 619 may be connected to at least one of processors 610 and 630. In other embodiments, memories 612, 632, and 639 may be coupled to processors 610 and 630 and object recognizer 619 via a bus and a shared memory hub.

[0063] System 600 includes a chipset 660 coupled to processors 610 and 630. Chipset 660 may also be coupled to storage medium 603, for example, via an interface (I / O) 666. I / O 666 may be, for example, a Peripheral Component Interconnect enhanced (PCI-e). Processors 610, 630, and object recognition device 619 may access storage medium 603 via chipset 660.

[0064] The storage medium 603 may comprise any non-transitory computer-readable storage medium or machine-readable storage medium, such as an optical, magnetic, or semiconductor storage medium. In various embodiments, the storage medium 603 may comprise a fabricated article. In some embodiments, the storage medium 603 may store computer-executable instructions, such as the computer-executable instructions 602, to perform one or more of the processes or operations described herein (e.g., process 500 of Fig.5). Storage medium 603 may store computer-executable instructions for any of the equations presented above. Storage medium 603 may further store computer-executable instructions for models and / or networks described herein, such as a neural network or the like. Examples of a computer-readable storage medium or machine-readable storage medium may include any tangible media capable of storing electronic data, including volatile memory or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, and so on. Examples of computer-executable instructions may include any suitable types of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, and the like.It is understood that the embodiments are not limited in this context.

[0065] Processor 610 is coupled to a chipset 660 via PP interfaces 652 and 662, and processor 630 is coupled to a chipset 660 via PP interfaces 654 and 664. Direct media interfaces (DMIs) may couple PP interfaces 652 and 662 and PP interfaces 654 and 664, respectively. The DMI may be a high-speed interconnect enabling, for example, eight gigatransfers per second (GT / s), such as DMI 3.0. In other embodiments, processors 610 and 630 may be interconnected via a bus.

[0066] Chipset 660 may include a control hub, such as a platform control hub (PCH). Chipset 660 may include a system clock for performing clocking functions and may include interfaces for an I / O bus, such as a Universal Serial Bus (USB), Peripheral Component Interconnects (PCIs), Serial Peripheral Interconnects (SPIs), Integrated Interconnects (I2Cs), and the like, to enable connection of peripheral devices on the platform. In other embodiments, chipset 660 may include more than one control hub, such as a chipset with a memory control hub, a graphics control hub, and an input / output (I / O) control hub.

[0067] In the present embodiment, chipset 660 is coupled to a trusted platform module (TPM) 672 and the UEFI, BIOS, Flash component 674 via an interface (I / F) 670. The TPM 672 is a dedicated microcontroller designed to secure hardware by embedding cryptographic keys into devices. The UEFI, BIOS, Flash component 674 can deploy code before booting.

[0068] Chipset 660 further includes I / F 666 for coupling chipset 660 to a high-performance graphics engine, graphics card 665. In other embodiments, system 600 may include a flexible display interface (FDI) between processors 610 and 630 and chipset 660. The FDI connects a graphics processor core in a processor to chipset 660.

[0069] Various I / O devices 692 are coupled to bus 681, along with a bus bridge 680 that couples bus 681 to a second bus 691, and an I / O 668 that connects bus 681 to chipset 660. In one embodiment, second bus 691 may be a low pin count (LPC) bus. Various devices may couple to second bus 691, including, for example, a keyboard 682, a mouse 684, communication devices 686, a storage medium 601, and audio I / O 690 (including, for example, one or more microphones).

[0070] The artificial intelligence (AI) accelerator 667 may be circuitry configured to perform AI-related computations. The AI ​​accelerator 667 may be connected to the storage medium 603 and the chipset 660. The AI ​​accelerator 667 may provide the processing power and energy efficiency needed to enable computing with large data sets. The AI ​​accelerator 667 is a class of specialized hardware accelerators or computer systems designed to accelerate artificial intelligence and machine learning applications, including artificial neural networks and machine vision. The AI ​​accelerator 667 may be applicable to algorithms for robotics, the Internet of Things, other data-intensive and / or sensor-driven tasks.

[0071] Many of the I / O devices 692, the communication devices 686, and the storage medium 601 may be located on the motherboard 605, while the keyboard 682 and the mouse 684 may be additional peripherals. In other embodiments, some or all of the I / O devices 692, the communication devices 686, and the storage medium 601 are additional peripherals and are not located on the motherboard 605.

[0072] Some examples may be described using the phrase "in an example" or "an example," along with their derivatives. These terms mean that a particular feature, structure, or property associated with the example is present in at least one example. The occurrence of the phrase "in an example" in various places in the description does not necessarily always refer to the same example.

[0073] Some examples can be described using the terms "coupled" and "connected," along with their derivatives. These terms are not necessarily intended to be synonymous. For example, descriptions using the terms "connected" and / or "coupled" may indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" can also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0074] Moreover, in the foregoing Detailed Description, various features are grouped into a single example to streamline the disclosure. This method of disclosure should not be construed to reflect an intent that the claimed examples require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in fewer than all of the features of a single disclosed example. Therefore, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate example. In the appended claims, the terms "including" and "in which" are used as the plain English equivalents of the respective terms "comprising" and "wherein."Furthermore, the terms “first,” “second,” “third,” and so on are used only as labels and are not intended to impose any numerical requirements on their objects.

[0075] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as exemplary forms of practicing the claims.

[0076] A data processing system capable of storing and / or executing program code includes at least one processor coupled directly or indirectly via a system bus to storage elements. The storage elements may include local memory used during actual execution of the program code, mass storage, and cache memory that provides temporary storage of at least some program code to reduce the number of times the code must be retrieved from mass storage during execution. The term "code" covers a broad range of software components and constructs, including applications, drivers, processes, routines, procedures, modules, firmware, microcode, and subroutines.Therefore, the term “code” can be used to refer to any collection of instructions that, when executed by a processing system, performs a desired operation or operations.

[0077] Logic circuitry, devices, and interfaces described herein may perform functions implemented in hardware and implemented with code executing on one or more processors. Logic circuitry refers to the hardware, or the hardware and code, that implements one or more logical functions. Circuitry is hardware and may refer to one or more circuits. Each circuit may perform a specific function. A circuit of the circuitry may include discrete electrical components connected to one or more conductors, an integrated circuit, a chip package, a chipset, memory, or the like. Integrated circuits include circuits created on a substrate, such as a silicon wafer, and may include components.Integrated circuits, processor packages, chip packages, and chipsets may contain one or more processors.

[0078] Processors can receive signals, such as instructions and / or data, at their inputs and process the signals to generate one or more outputs. When executing code, the code changes the physical states and characteristics of transistors that comprise a processor pipeline. The physical states of the transistors are translated into logical bits of ones and zeros, which are stored in registers within the processor. The processor can transfer the physical states of the transistors to registers and transfer the physical states of the transistors to another storage medium.

[0079] A processor may include circuitry to perform one or more subfunctions implemented to perform the overall function of the processor. An example of a processor is a state machine or an application-specific integrated circuit (ASIC) that includes at least one input and at least one output. A state machine may manipulate the at least one input to generate the at least one output by performing a predetermined series of serial and / or parallel manipulations or transformations on the at least one input.

[0080] The logic described above may be part of the design for an integrated circuit chip. The chip design is created in a graphical computer programming language and stored on a computer storage medium or data storage medium (such as a disk, tape, physical hard disk drive, or virtual hard disk drive, as in a storage access network). If the designer does not manufacture chips or the photolithographic masks used to fabricate chips, the designer sends the resulting design to such entities, directly or indirectly, by physical means (e.g., by providing a copy of the storage medium that stores the design) or electronically (e.g., via the Internet). The stored design is then converted into the appropriate format (e.g., GDSII) for manufacturing.

[0081] The resulting integrated circuit chips can be distributed by the manufacturer in raw wafer form (that is, as a single wafer containing multiple unpackaged chips), as a bare die, or in packaged form. In the latter case, the chip is mounted in a single-chip package (such as a plastic carrier, with leads attached to a motherboard or other higher-level carrier) or in a multi-chip package (such as a ceramic carrier, having interconnection on one or both surfaces or hidden interconnections). In either case, the chip is then integrated with other chips, discrete circuit elements, and / or other signal processing devices as part of either (a) an intermediate product, such as a processor board, a server platform, or a motherboard, or (b) a final product.

[0082] The word "exemplary" is used herein to mean "serving as an example, example, or illustration." Any embodiments described herein as "exemplary" are not necessarily to be construed as preferred or advantageous over other embodiments. The terms "computing device," "user device," "communication station," "station," "handheld device," "mobile device," "wireless device," and "terminal equipment" (UE), as used herein, refer to a wireless communication device, such as a cellular phone, a smartphone, a tablet, a netbook, a wireless terminal, a laptop computer, a femtocell, a high data rate (HDR) subscriber station, an access point, a printer, a point-of-sale device, an access terminal, or other personal communications system (PCS) device. The device may be either mobile or stationary.

[0083] As used in this document, the term "communicating" is intended to include sending or receiving, or both sending and receiving. This can be particularly useful in claims when describing the organization of data sent by one device and received by another, but only the functionality of one of those devices is required to infringe the claim. Similarly, the bidirectional exchange of data between two devices (both devices sending and receiving during the exchange) can be described as "communicating" when only the functionality of one of those devices is claimed. The term "communicating" as used herein with respect to a wireless communication signal includes sending the wireless communication signal and / or receiving the wireless communication signal.For example, a wireless communication unit capable of communicating a wireless communication signal may include a wireless transmitter to transmit the wireless communication signal to at least one other wireless communication unit and / or a wireless communication receiver to receive the wireless communication signal from at least one other wireless communication unit.

[0084] Unless otherwise indicated, the use of the ordinal adjectives "first," "second," "third," etc., as used herein to describe a common object merely indicates that reference is made to different instances of like objects and is not intended to imply that the objects so described must be in any given order, either temporally, spatially, in rank, or in any other way.

[0085] Some embodiments may be used in connection with various devices and systems, for example, a personal computer (PC), a desktop computer, a mobile computer, a laptop computer, a notebook computer, a tablet computer, a server computer, a handheld computer, a handheld device, a personal digital assistant (PDA) device, a handheld PDA device, an on-board device, an off-board device, a hybrid device, a vehicular device, a non-vehicular device, a mobile or portable device, a consumer device, a non-mobile or non-portable device, a wireless communication station, a wireless communication device, a wireless access point (AP), a wired or wireless router, a wired or wireless modem, a video device, an audio device, an audio-video (A / V) device,a wired or wireless network, a wireless network, a wireless video network (WVAN), a local area network (LAN), a wireless LAN (WLAN), a personal area network (PAN), a wireless PAN (WPAN), and the like.

[0086] Some embodiments may be used in connection with one-way and / or two-way radio communication systems, cellular telephone communication systems, a mobile telephone, a cellular phone, a wireless telephone, a personal communication system (PCS) device, a PDA device incorporating a wireless communication device, a mobile or portable global positioning system (GPS) device, a device incorporating a GPS receiver or transceiver or chip, a device incorporating an RFID element or chip, a multiple input multiple output (MIMO) transceiver or device, a single input multiple output (SIMO) transceiver or device, a multiple input single output (MISO) transceiver or device, a device having one or more internal antennas and / or external antennas,Digital Video Broadcast (DVB) devices or systems, multi-standard radio devices or systems, a wired or wireless handheld device, e.g., a smartphone, a Wireless Application Protocol (WAP) device, or the like.

[0087] Some embodiments may be used in connection with one or more types of wireless communication signals and / or systems following one or more wireless communication protocols, for example, radio frequency (RF), infrared (IR), frequency division multiple access (FDM), orthogonal FDM (OFDM), time division multiple access (TDM), time division multiple access (TDMA), enhanced TDMA (E-TDMA), general packet radio service (GPRS), enhanced GPRS, code division multiple access (CDMA), wideband CDMA (WCDMA), CDMA-2000, single-carrier CDMA, multi-carrier CDMA, multi-carrier modulation (MDM), discrete multi-tone (DMT), Bluetooth®, global positioning system (GPS), WiFi, Wi-Max, ZigBee, ultra-wideband (UWB), GSM (global system for mobile communications), 2G, 2.5G, 3G, 3.5G, 4G, 5G (fifth generation mobile networks), 3GPP, LTE (Long Term Evolution), advanced LTE, EDGE (enhanced data rate for GSM Evolution) or similar protocols.Other embodiments may be used in various other devices, systems, and / or networks.

[0088] The following examples relate to further embodiments.

[0089] Example 1 may be an apparatus for object detection in images, the apparatus comprising processing circuitry coupled to a memory, the processing circuitry configured to: input image features of an image representing an object to a manual labeling learning system; identify, using the manual labeling learning system, first coordinates of an upper left corner of a bounding box representing the object based on a heatmap indicating a probability that the first coordinates represent the upper left corner;Identifying, using the manual labeling learning system, second coordinates of a lower right corner of the bounding box based on the first coordinates and a first distance regression map indicating coordinate differences between the second coordinates and ground truth coordinates input to the manual labeling learning system as training data; Generating, using the manual labeling learning system, adjustments to the first coordinates and the second coordinates based on a second distance regression map; and Generating, using the manual labeling learning system, the adjusted first coordinates and the adjusted second coordinates, the bounding box for use in subsequent object detection for the image.

[0090] Example 2 may include the setup of Example 1 and / or any other example herein, wherein a convolutional neural network having a backbone of at least fifty layers generates the image features.

[0091] Example 3 may include the apparatus of Example 1 and / or any other example herein, wherein identifying the second coordinates comprises determining a loss function of a first bounding box based on the ground truth coordinates and a second bounding box based on predicted distances between the first coordinates and third coordinates estimated for the lower right corner.

[0092] Example 4 may include implementing Example 3 and / or any other example herein, wherein the loss function is an intersection-of-union loss function.

[0093] Example 5 may include the apparatus of Example 4 and / or any other example herein, wherein the loss function is based on a summation of the intersection-of-union of the first bounding box and the second bounding box and an intersection-of-union of the first bounding box and a third bounding box based on predicted distances between the first coordinates and fourth coordinates estimated for the lower right corner.

[0094] Example 6 may include the apparatus of Examples 1-4 and / or any other example herein, wherein identifying the second coordinates comprises: identifying a first embedding vector indicative of first pixel features of the first coordinates; and identifying, based on the first embedding vector, a second embedding vector indicative of second pixel features of the second coordinates.

[0095] Example 7 may include the apparatus of Example 1 and / or any other example herein, wherein the processing circuitry is further configured to generate the heatmap based on an exponential function of a difference between the first coordinates and the ground truth coordinates.

[0096] Example 8 may include implementing Example 7 and / or any other example herein, wherein values ​​of the heatmap correspond to respective coordinates and indicate a probability that the respective coordinates correspond to the upper left corner.

[0097] Example 9 may include a computer-readable storage medium comprising instructions to cause processing circuitry of an apparatus for object recognition in images, upon execution of the instructions by the processing circuitry, to perform the following: input image features of an image representing an object to a manual labeling learning system; identify, using the manual labeling learning system, first coordinates of an upper left corner of a bounding box representing the object based on a heatmap indicating a probability that the first coordinates represent the upper left corner;Identifying, using the manual labeling learning system, second coordinates of a lower right corner of the bounding box based on the first coordinates and a first distance regression map indicating coordinate differences between the second coordinates and ground truth coordinates input to the manual labeling learning system as training data; Generating, using the manual labeling learning system, adjustments to the first coordinates and the second coordinates based on a second distance regression map; and Generating, using the manual labeling learning system, the adjusted first coordinates and the adjusted second coordinates, the bounding box for use in subsequent object detection for the image.

[0098] Example 10 may include the non-transitory computer-readable medium of Example 9 and / or any other example herein, wherein a convolutional neural network having a backbone of at least fifty layers generates the image features.

[0099] Example 11 may include the non-transitory computer-readable medium of Example 9 and / or any other example herein, wherein identifying the second coordinates comprises determining a loss function of a first bounding box based on the ground truth coordinates and a second bounding box based on predicted distances between the first coordinates and third coordinates estimated for the lower right corner.

[0100] Example 12 may include the non-transitory computer-readable medium of Example 11 and / or any other example herein, wherein the loss function is an intersection-of-union loss function.

[0101] Example 13 may include the non-transitory computer-readable medium of Example 12 and / or any other example herein, wherein the loss function is based on a summation of the intersection-of-union of the first bounding box and the second bounding box and an intersection-of-union of the first bounding box and a third bounding box based on predicted distances between the first coordinates and fourth coordinates estimated for the lower right corner.

[0102] Example 14 may include the non-transitory computer-readable medium of Examples 9-13 and / or any other example herein, wherein identifying the second coordinates comprises: identifying a first embedding vector indicative of first pixel features of the first coordinates; and identifying, based on the first embedding vector, a second embedding vector indicative of second pixel features of the second coordinates.

[0103] Example 15 may include the non-transitory computer-readable medium of Example 9 and / or any other example herein, wherein the instructions further cause the processing circuitry to generate the heatmap based on an exponential function of a difference between the first coordinates and the ground truth coordinates.

[0104] Example 16 may include the non-transitory computer-readable medium of Example 15 and / or any other example herein, wherein values ​​of the heatmap correspond to respective coordinates and indicate a probability that the respective coordinates correspond to the upper left corner.

[0105] Example 17 may be a method for object detection in images, the method comprising: inputting, by processing circuitry of a device, image features of an image representing an object to a manual labeling learning system; identifying, using the manual labeling learning system, first coordinates of an upper left corner of a bounding box representing the object based on a heat map indicating a probability that the first coordinates represent the upper left corner; identifying, using the manual labeling learning system, second coordinates of a lower right corner of the bounding box based on the first coordinates and a first distance regression map indicating coordinate differences between the second coordinates and ground truth coordinates input to the manual labeling learning system as training data;Generating, using the manual label learning system, adjustments to the first coordinates and the second coordinates based on a second distance regression map; and generating, using the manual label learning system, the adjusted first coordinates and the adjusted second coordinates, the bounding box for use in subsequent object detection for the image.

[0106] Example 18 may include the method of Example 17 and / or any other example herein, wherein a convolutional neural network having a backbone of at least fifty layers generates the image features, and wherein at least three of the layers are associated with adjusting the first coordinates and the second coordinates.

[0107] Example 19 may include the method of Example 17 and / or any other example herein, wherein identifying the second coordinates comprises determining a loss function of a first bounding box based on the ground truth coordinates and a second bounding box based on predicted distances between the first coordinates and third coordinates estimated for the lower right corner.

[0108] Example 20 may include the method of Example 19 and / or any other example herein, wherein the loss function is an intersection-of-union loss function.

[0109] Example 21 may include the method of Example 20 and / or any other example herein, wherein the loss function is based on a summation of the intersection-of-union of the first bounding box and the second bounding box and an intersection-of-union of the first bounding box and a third bounding box based on predicted distances between the first coordinates and fourth coordinates estimated for the lower right corner.

[0110] Example 22 may include the method of any of Examples 17-21 and / or any other example herein, wherein the loss function is based on a summation of the intersection-of-union of the first bounding box and the second bounding box and an intersection-of-union of the first bounding box and a third bounding box based on predicted distances between the first coordinates and fourth coordinates estimated for the lower right corner.

[0111] Example 23 may include the method of Example 17 and / or any other example herein, further comprising creating the heatmap based on an exponential function of a difference between the first coordinates and the ground truth coordinates.

[0112] Example 24 may include the method of Example 17 and / or any other example herein, wherein values ​​of the heatmap correspond to respective coordinates and indicate a probability that the respective coordinates correspond to the upper left corner.

[0113] Example 25 may include an apparatus comprising means for: inputting image features of an image representing an object to a manual labeling learning system; identifying, using the manual labeling learning system, first coordinates of an upper left corner of a bounding box representing the object based on a heat map indicating a probability that the first coordinates represent the upper left corner; identifying, using the manual labeling learning system, second coordinates of a lower right corner of the bounding box based on the first coordinates and a first distance regression map indicating coordinate differences between the second coordinates and ground truth coordinates input to the manual labeling learning system as training data;Generating, using the manual label learning system, adjustments to the first coordinates and the second coordinates based on a second distance regression map; and generating, using the manual label learning system, the adjusted first coordinates and the adjusted second coordinates, the bounding box for use in subsequent object detection for the image.

[0114] Example 26 may include one or more non-transitory computer-readable media comprising instructions for causing an electronic device, upon execution of the instructions by one or more processors of the electronic device, to perform one or more elements of a method described in any of Examples 1-25 or a method associated therewith, or any other method or process described herein.

[0115] Example 27 may include an apparatus including logic, modules, and / or circuitry to perform one or more elements of a method or associated method described in any of Examples 1-25, or any other method or process described herein.

[0116] Example 28 may include a method, technique, or process as described in or associated with any of Examples 1-25, or portions or sections thereof.

[0117] Example 29 may include an apparatus comprising: one or more processors; and one or more computer-readable media comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the method, techniques, or process described in or associated with any of Examples 1-25 or portions thereof.

[0118] Embodiments according to the disclosure are particularly disclosed in the appended claims, which relate to a method, a storage medium, an apparatus, and a computer program product, wherein any feature mentioned in one claim category, e.g., a method, can also be claimed in another claim category, e.g., a system. The dependencies or backreferences in the appended claims are selected for formal reasons only. However, any subject matter resulting from a targeted reference back to any previous claims (in particular, multiple dependencies) can also be claimed, so that any combination of claims and their features is disclosed and can be claimed independently of the dependencies chosen in the appended claims.The subject matter that can be claimed includes not only the combinations of features as set forth in the appended claims, but also any other combination of features in the claims, wherein any feature recited in the claims can be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or illustrated herein can be claimed in a separate claim and / or in any combination with any embodiment or feature described or illustrated herein, or with any of the features of the appended claims.

[0119] The foregoing description of one or more implementations provides illustration and description, but is not intended to be exhaustive or to limit the scope of the embodiments to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be obtained from practice of various embodiments.

[0120] Certain aspects of the disclosure are described above with reference to block and flow diagrams of systems, methods, devices, and / or computer program products according to various implementations. It is understood that one or more blocks of the block diagrams and flow diagrams, and combinations of blocks in the block diagrams and flow diagrams, respectively, may be implemented by computer-executable program instructions. Likewise, some blocks of the block diagrams and flow diagrams may not necessarily be performed in the order presented or may not necessarily be performed at all.

[0121] These computer-executable program instructions may be loaded onto a special purpose computer or other special purpose machine, processor, or other programmable data processing device to produce a special purpose machine, such that the instructions, executing on the computer, processor, or other programmable data processing device, produce means for implementing one or more functions specified in the flowchart block or blocks.These computer program instructions may also be stored in a computer-readable storage medium or memory that can instruct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement one or more functions specified in the flowchart block or blocks. As an example, certain implementations may provide a computer program product comprising a computer-readable storage medium having computer-readable program code or program instructions implemented therein, the computer-readable program code configured to be executed to implement one or more functions specified in the flowchart block or blocks.The computer program instructions may also be loaded into a computer or other programmable data processing device to cause a series of work elements or steps to be performed on the computer or other programmable device to produce a computer-implemented process, such that the instructions executing on the computer or other programmable device provide elements or steps for implementing the functions specified in the flowchart block or blocks.

[0122] Accordingly, blocks of the block diagrams and flowcharts support combinations of means for performing the specified functions, combinations of elements or steps for performing the specified functions, and program instruction means for performing the specified functions. It is also understood that each block of the block diagrams and flowcharts, and combinations of blocks in the block diagrams and flowcharts, may be implemented by special-purpose hardware-based computer systems that perform the specified functions, elements, or steps, or combinations of special-purpose hardware and computer instructions.

[0123] Conditional language such as, but not limited to, "may," "could," "might," or "may," unless specifically stated otherwise or understood otherwise in the context used, is generally intended to convey that certain implementations might include certain features, elements, and / or operations, while other implementations do not include them. Thus, such conditional language is generally not intended to imply that features, elements, and / or operations are in any way required by one or more implementations, or that one or more implementations necessarily include logic for deciding, with or without user input or prompting, whether those features, elements, and / or operations are included or are to be performed in any particular implementation.

[0124] Many modifications and other implementations of the disclosure set forth herein will become apparent with the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the disclosure is not intended to be limited to the specific implementations disclosed, and that modifications and other implementations are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used only in a generic and descriptive sense and not for purposes of limitation.

Claims

[1] Device for object recognition in images, the device comprising processing circuitry coupled to a memory, which is designed to: Inputting image features of an image representing an object into a manual labeling learning system; Identifying, using the manual labeling learning system, first coordinates of an upper left corner of a bounding box representing the object based on a heatmap indicating a probability that the first coordinates represent the upper left corner; Identifying, using the manual labeling learning system, second coordinates of a lower right corner of the bounding box based on the first coordinates and a first distance regression map indicating coordinate differences between the second coordinates and ground truth coordinates input to the manual labeling learning system as training data; Generating, using the manual label learning system, adjustments to the first coordinates and the second coordinates based on a second distance regression map; and Generating, using the manual label learning system, the adjusted first coordinates and the adjusted second coordinates, the bounding box for use in subsequent object detection for the image. [2] The device according to claim 1, wherein a convolutional neural network having a backbone of at least fifty layers generates the image features. [3] The apparatus of claim 1, wherein identifying the second coordinates comprises determining a loss function of a first bounding box based on the ground truth coordinates and a second bounding box based on predicted distances between the first coordinates and third coordinates estimated for the lower right corner. [4] The device of claim 3, wherein the loss function is an intersection-of-union loss function. [5] The apparatus of claim 4, wherein the loss function is based on a summation of the intersection-of-union of the first bounding box and the second bounding box and an intersection-of-union of the first bounding box and a third bounding box based on predicted distances between the first coordinates and fourth coordinates estimated for the lower right corner. [6] Device according to one of claims 1-4, wherein identifying the second coordinates comprises: Identifying a first embedding vector indicating first pixel features of the first coordinates; and Identifying, based on the first embedding vector, a second embedding vector indicating second pixel features of the second coordinates. [7] The device of claim 1, wherein the processing circuitry is further configured to create the heatmap based on an exponential function of a difference between the first coordinates and the ground truth coordinates. [8] The device according to claim 7, wherein values ​​of the heat map correspond to respective coordinates and indicate a probability that the respective coordinates correspond to the upper left corner. [9] A computer-readable storage medium comprising instructions for causing processing circuitry of an apparatus for object recognition in images to, upon execution of the instructions by the processing circuitry: Inputting image features of an image representing an object into a manual labeling learning system; Identifying, using the manual labeling learning system, first coordinates of an upper left corner of a bounding box representing the object based on a heatmap indicating a probability that the first coordinates represent the upper left corner; Identifying, using the manual labeling learning system, second coordinates of a lower right corner of the bounding box based on the first coordinates and a first distance regression map indicating coordinate differences between the second coordinates and ground truth coordinates input to the manual labeling learning system as training data; Generating, using the manual label learning system, adjustments to the first coordinates and the second coordinates based on a second distance regression map; and Generating, using the manual label learning system, the adjusted first coordinates and the adjusted second coordinates, the bounding box for use in subsequent object detection for the image. [10] The computer-readable medium of claim 9, wherein a convolutional neural network having a backbone of at least fifty layers generates the image features. [11] The computer-readable medium of claim 9, wherein identifying the second coordinates comprises determining a loss function of a first bounding box based on the ground truth coordinates and a second bounding box based on predicted distances between the first coordinates and third coordinates estimated for the lower right corner. [12] The computer-readable medium of claim 11, wherein the loss function is an intersection-of-union loss function. [13] The computer-readable medium of claim 12, wherein the loss function is based on a summation of the intersection-of-union of the first bounding box and the second bounding box and an intersection-of-union of the first bounding box and a third bounding box based on predicted distances between the first coordinates and fourth coordinates estimated for the lower right corner. [14] The computer-readable medium of any of claims 9-13, wherein identifying the second coordinates comprises: Identifying a first embedding vector indicating first pixel features of the first coordinates; and Identifying, based on the first embedding vector, a second embedding vector indicating second pixel features of the second coordinates. [15] The computer-readable medium of claim 9, wherein the instructions further cause the processing circuitry to create the heatmap based on an exponential function of a difference between the first coordinates and the ground truth coordinates. [16] The computer-readable medium of claim 15, wherein values ​​of the heat map correspond to respective coordinates and indicate a probability that the respective coordinates correspond to the upper left corner. [17] A method for object recognition in images, the method comprising: inputting, by processing circuitry of a device, image features of an image representing an object into a manual labeling learning system; Identifying, using the manual labeling learning system, first coordinates of an upper left corner of a bounding box representing the object based on a heatmap indicating a probability that the first coordinates represent the upper left corner; Identifying, using the manual labeling learning system, second coordinates of a lower right corner of the bounding box based on the first coordinates and a first distance regression map indicating coordinate differences between the second coordinates and ground truth coordinates input to the manual labeling learning system as training data; Generating, using the manual label learning system, adjustments to the first coordinates and the second coordinates based on a second distance regression map; and Generating, using the manual label learning system, the adjusted first coordinates and the adjusted second coordinates, the bounding box for use in subsequent object detection for the image. [18] The method of claim 17, wherein a convolutional neural network having a backbone of at least fifty layers generates the image features, and wherein at least three of the layers are associated with adjusting the first coordinates and the second coordinates. [19] The method of claim 17, wherein identifying the second coordinates comprises determining a loss function of a first bounding box based on the ground truth coordinates and a second bounding box based on predicted distances between the first coordinates and third coordinates estimated for the lower right corner. [20] The method of claim 19, wherein the loss function is an intersection-of-union loss function. [21] The method of claim 20, wherein the loss function is based on a summation of the intersection-of-union of the first bounding box and the second bounding box and an intersection-of-union of the first bounding box and a third bounding box based on predicted distances between the first coordinates and fourth coordinates estimated for the lower right corner. [22] The method of any of claims 17-21, wherein identifying the second coordinates comprises: Identifying a first embedding vector indicating first pixel features of the first coordinates; and Identifying, based on the first embedding vector, a second embedding vector indicating second pixel features of the second coordinates. [23] The method of claim 17, further comprising creating the heatmap based on an exponential function of a difference between the first coordinates and the ground truth coordinates. [24] The method of claim 17, wherein values ​​of the heatmap correspond to respective coordinates and indicate a probability that the respective coordinates correspond to the upper left corner. [25] A computer-readable storage medium comprising instructions for performing the method of any one of claims 17-24.

Images