Detecting objects in crowd using geometric context
By introducing geometric context information into the object detection system, the problem of low object detection accuracy in crowded scenarios is solved, the detection ability of partial occlusion objects is improved, and higher accuracy and recall rate are achieved.
Patent Information
- Application Number
- CN202411250545.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-12-17
- Filing Date
- 2019-12-10
- Publication Date
- 2024-11-26
AI Technical Summary
Existing object detection systems have difficulty effectively detecting partially or obstructed objects in crowded scenarios, resulting in a decrease in detection accuracy, especially due to individually trained appearance features and assuming that the entire object is always visible.
By introducing geometric context, estimating the geometric information of the scene using monocular image data, such as camera pose and scene cues, combined with deep neural networks, modeling occlusion and object position consistency, reducing ambiguity, and implicitly in object detectors Use this information in order to improve the accuracy of object detection.
It improves the accuracy and recall rate of object detection, reduces the error rate, especially in crowded scenarios, enhances the detection ability of partially blocked objects, and proves the technical effect in pedestrian detection and other scenarios.
Smart Images

Figure CN119027742A_ABST
Abstract
Description
[0001] Related Applications Citations
[0002] This application is a divisional application of the invention patent application with international application number PCT / US2019 / 065311, international application date December 10, 2019, date of entry into the Chinese national phase June 11, 2021, Chinese national application number 201980082636.6, and invention name “Detecting objects in a crowd using geometric context”. Technical Field
[0003] Embodiments of the present disclosure relate to detecting objects in a crowd using geometric context. Background Art
[0004] Modern object detection systems typically rely on region proposal methods and convolutional neural networks to generate candidate regions and then classify these regions as containing the desired object or not. Using these methods, objects such as people and vehicles can be identified and quantified based on image data. Summary of the invention
[0005] This summary is provided to introduce some concepts in a simplified form, which will be further described in the detailed description below. This summary is neither intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in any part of this disclosure.
[0006] Examples related to classifying objects using geometric context are disclosed. In one example, a computing system is configured to train an object classifier. Monocular image data and ground truth data for a scene are received. Geometric context is determined, including the position of a three-dimensional camera relative to a fixed plane. Regions of interest (RoIs) and a set of potential occluders are identified within the image data. For each potential occluder, the occluded region is projected onto the fixed plane in three dimensions. For each occluded region, a set of occluded RoIs on the fixed plane is generated. Each occluded RoI is projected back to the image data in two dimensions. A classifier is trained by minimizing a loss function generated by inputting information related to the RoI and the occluded RoI into the classifier, and minimizing the position error of each RoI in the set and each occluded RoI on the fixed plane based on the ground truth data. The trained classifier is then output for object detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1 An example scenario is shown in which object detection may be performed.
[0008] Figure 2 An example pipeline for object detection using deep neural networks is schematically shown.
[0009] Figure 3 An example depiction for anchor generation during object detection is shown.
[0010] Figure 4 Example occlusion regions generated for potential occluders in a scene are shown.
[0011] Figure 5 Example bounding boxes generated for foreground occluders and accompanying occluded regions are shown.
[0012] Figure 6 An example of 3D projection loss is schematically shown.
[0013] Figure 7 A flowchart of an example method for training a classifier using geometric context is shown.
[0014] Figure 8 A flow chart illustrating an example method of operating a classifier trained using geometric context is shown.
[0015] Fig. 9 Shown are the precision-recall curves for an example classification model trained on a geometric context.
[0016] Fig.10 is a plot indicating the cross-camera performance of an example classification model trained on a geometric context.
[0017] Fig.11 Plots depicting the mAP gain under occlusion versus camera altitude and radial distance.
[0018] Fig.12 An example computing system is schematically illustrated. DETAILED DESCRIPTION
[0019] Early research in object recognition built on geometric principles inherent to humans, such as occlusion, perspective, and rootedness, using formal geometric models as informative priors to modulate statistical learning. More recently, object detectors are often cast as regressions of deep convolutional image features. Neural networks, such as region-based fully convolutional networks (R-FCN) and region-based convolutional neural networks (e.g., Faster R-CNN), are trained on large datasets, making them robust to most perceptual changes. However, such neural networks may fail to characterize a large number of objects of the same category in crowded scenes.
[0020] For example, such a neural network can perform reasonably well in detecting pedestrians in images generated by wide-angle surveillance or monitoring cameras when the pedestrians are well-spaced and do not occlude each other. However, in crowded areas of the image, pedestrians may be missed completely or exhibit merged detections where multiple pedestrians are detected as a single blob. Therefore, delineating these partially visible instances will help improve the overall detection accuracy and improve downstream tasks such as people tracking and activity recognition.
[0021] One reason for poor performance in crowded scenes is that object detectors are trained solely on statistics of appearance features. For example, person detection may tend to identify objects of a certain size and with certain features such as head, torso, legs, etc. Many off-the-shelf models work well in the ideal case of isolated objects and clean image backgrounds. However, in real-world applications, crowded scenes are characterized by object occlusion or partial occlusion, partial observation, lighting changes, background gradients and patterns, scale, and other issues.
[0022] As an example, Figure 1 An example scene 100 is shown. Individuals 110 and 120 are depicted standing on ground plane 120. Monitoring camera 130, monitoring camera 134, and monitoring camera 138 display imaged scene 100 from various perspectives. Camera 130 is an overhead camera located at a height h1 above ground plane 120. Camera 134 is an eye-level camera located at a height h2 above ground plane 120, where h2
[0023] Two issues with surveillance cameras are scale (e.g., proximity to the camera) and occlusion. In this example, individual 110 is closer to camera 130, camera 134, and camera 138 than individual 112. If individual 112 is significantly farther from the camera, individual 112 may appear smaller than individual 110 in the scene image. For a wide scene, individuals at different distances may have significantly different sizes in the scene image. Thus, filtering out objects based on a simple size threshold may become difficult.
[0024] The far-end individual may be obscured by the near-end individual when an object such as individual 110 is located between the camera and another object such as individual 112. In this example, individual 112 may be partially obscured by individual 110 from the perspective of camera 130 and camera 138, and may be significantly obscured by camera 134. For example, camera 130 may see the head and torso of individual 112, but may not see the legs and feet of individual 112.
[0025] The specific occlusion pattern of nearby objects in the image depends on the camera's viewpoint relative to the ground plane in 3D. Camera 130 is located at height h1 and has image plane 140 located at an acute angle to ground plane 120. Camera 134 is located at height h2 and has image plane 144 located perpendicular to ground plane 120. Camera 138 is located on the ground and has image plane 148 located at an obtuse angle to ground plane 120. The pose of the camera and the scene geometry report the physics of the image view. Information cues, such as the line of sight ("los") angle shown in dashed lines, are different for each camera pose.
[0026] Typically, object detectors form a multi-stage pipeline where the image is first parsed into salient regions and then refined into bounding boxes, all of which are constrained to 2D image (pixel) space. The assumption behind such detectors is that independent and uniformly distributed (iid) sampling from a large set of training data will eventually cover substantially all possible variations in view space, leading to an occlusion-resilient model. These convolutional features do not account for scene regularities, such as the expected size of an object at a certain location, or that most objects are fixed.
[0027] Furthermore, appearance- and image-processing-based object detection methods typically assume that the entire object is always visible. As a result, the recall of such methods is poor and is prone to underestimation due to missed classifications and false negatives. For humans, partial occlusions are compensated through a process of amodal completion, e.g., filling in the missing object information behind the occluded surface based on known or assumed object geometry. Several works have attempted to improve statistical object detectors by incorporating geometric context. In one example of such work, a graphical model places local object detection in the context of the entire 3D scene by modeling the interdependencies between object size, surface orientation, and camera viewpoint. The model can also include additional pose constraints on object configuration within the feedback loop. In another example, a cubic room representation is deployed for indoor scenes, showing that layout estimation improves object detection. These models improve overall detection accuracy, but are typically divided into stages of sequential or iterative components that are coupled with independent object detectors to improve object hypotheses.
[0028] Currently, the most accurate object detectors are mainly single-stage or multi-stage convolutional neural networks (CNNs), which behave as highly parallelized sliding window classifiers. However, it is challenging to implement context inclusion as structured prediction without breaking the end-to-end formulation of the fixed CNN architecture. Some solutions to this challenge include taking the object mask from the current detection as context input during subsequent iterations. To introduce scale context, perspective heatmaps can be generated and fused with reasoning from the coarse network and the fine sub-network. However, these methods may be limited to adapting to scale under perspective distortion and may involve multi-network or multi-iteration pipelines.
[0029] In theory, older models could be trained on scale and occlusion data. However, this would require a very large number of combinatorial possibilities to account for the set of events that could occur in the real world. This would include a large number of possible camera angles, fields of view, etc., applied to different ground plane positioning and sizes of imaging area footprints. Improving accuracy in this way would therefore involve exponentially more training data and training time.
[0030] Thus, examples are disclosed that include candidate-based deep neural networks modulated by the geometric context of a scene. The disclosed examples combine image processing and appearance models with the physics and optics of the real world. Ambiguity can be reduced by using monocular image cues to estimate the geometric context of the scene (e.g., camera pose and scene cues) and implicitly placing this information into the object detector itself. Such geometric context acts as prior knowledge at two levels of learning through occluder-centered candidates to model inter-object occlusion during region candidate generation (RPN), and through a geometric loss that models object position and scale consistency during per-region loss computation, while maintaining the feed-forward topology of the underlying object detector.
[0031] Geometric context can be introduced in deep neural networks through the reprojection loss, which quantifies how close a 3D point estimate is to its true projection in the 2D image space. A fixed plane (e.g. the ground plane) can be established as a common reference frame, allowing position consistency in 3D to be measured. This allows the search space to be narrowed down by using the geometry of the camera. Camera pose and scene cues can therefore be implicitly placed in the object detector and used to make the object detector ready.
[0032] Viewpoint-agnostic region proposals can be enhanced during region proposal generation with sparse, occluder-centric proposals in order to focus on scene-specific hard regions while learning to detect under a different set of occlusion patterns. This mimics amodal completion, which allows humans to naturally perform occlusion reasoning by using scene geometry to focus the classifier on parts of visible objects. While previous approaches have treated amodal completion purely as a learning problem, our approach demonstrates that amodal completion is inherently a geometric phenomenon that relies on underlying scene cues.
[0033] A geometric loss is introduced for bounding box regression, where the scene layout is used as a projection space to enforce the relationship and position consistency of candidates on the ground plane. By exploiting the geometry of the underlying scene to modulate feature activations of this type, discriminative features that improve baseline object detectors can be learned. To this end, the geometric context of the scene (e.g., camera pose relative to the ground plane) is inserted into a deep neural network (DNN) model for object detection, where the variables are back-propagated through all layers of the network. Experiments with state-of-the-art DNN detectors and detection benchmarks are presented in this paper and demonstrate that the disclosed examples have the technical effect of outperforming strong baselines by an average of about 5%, thereby reducing the error rate of object detection and classification.
[0034] The disclosed examples leverage known methods for two-stage object detection, namely Fast RCNN and Region-based Fully Convolutional Networks (R-FCN), to introduce geometric context during feature activation and loss computation. By introducing geometric context into feature learning, occluded portions may be better modeled and recall may be improved. The disclosed examples are described in the context of pedestrian detection, but can be applied to other types of object detection with a fixed reference plane. For example, vehicles, animals, and other objects anchored to the ground plane can be classified by such a model using similar detection parameters as long as the approximate physical size of the object in the real world is known. The fixed plane does not have to be the ground plane. For example, a cave ceiling can provide a fixed plane for bat detection, a ship hull can provide a fixed plane for barnacle detection, and so on.
[0035] Figure 2An example detection pipeline 200 is shown. In the pipeline 200, image data 205 can be fed into the DNN at 210. For example, the DNN can include a residual network (ResNet)-101. The 4th (Res4) ResNet block and the 5th (Res5) ResNet block are depicted, but the image data 205 can be introduced at other layers and blocks of the network. In addition, a geometric context 215 can be received and / or generated from the image data. For example, the camera pose can be included in the ground truth data or other data associated with the image data, and / or a one-time approximate camera pose estimate determined from the image data can be performed as a preprocessing step. The determined geometric context can be applied to objects that are part of a 3D model. For example, if the object is located on the ground, an object that looks like a person but is far away may be identified as a person, while an object suspended in the air will be ignored.
[0036] The geometric context is built on the architecture of a two-stage object detector comprising two modules: a region proposal network 220 (RPN) and a classification stage 225. Bounding boxes are fitted around objects by the RPN 220, and then the objects are classified in the classification stage 225.
[0037] The image data 205 and the geometric context 215 can be used for anchor generation 230. During anchor generation, dense pixel-based anchors are enhanced with 2D projections of sparse, occluder-centered anchors in 3D. Next, the anchors are passed through a region proposal network (RPN) stage 235 where they are filtered and labeled.
[0038] To integrate the geometric context 215, the RPN 235 can be modified to generate regions centered on occluders. Traditionally, RPNs typically look for specific gradients from training on image data (e.g., background / foreground). Two people standing together will be classified together or not at all depending on the training. As further described herein, foreground objects are used to generate occlusion-centered anchors, where adjacent regions are resolved for the presence of objects that are partially occluded by the foreground objects. The occluder-centered anchors and position-sensitive ROIs (245) may be referred to herein as early fusion, referring to the fusion of geometric context with the classification pipeline.
[0039] Next, the multi-task objective loss function 250 is jointly minimized by minimizing the position error of the RoI on the ground plane. This is achieved by reprojecting the RoI and the ground truth back to 3D space, which can be referred to as the late fusion of geometric context and classification pipeline in this article.
[0040] The sampling step may occur before executing the loss function. In some examples, this may include random stratified sampling. A subset of foreground, background, and occluded regions are sampled for the loss function. After the loss function is minimized, the data may be fed back to the RPN 235, which may include a class-agnostic stage in which it is determined whether the RoI contains an object, followed by a stage that includes class-specific bounding box regression.
[0041] The geometric context may be determined at 215 by inferring from image data cues. Such an inference process may be particularly useful in scenarios where a static camera captures a wide-angle view of a crowded scene. In such scenarios, performing a simple one-shot camera pose estimation may provide rich spatial cues about objects in space. In particular, knowing the pose of the camera relative to the ground plane allows depth sorting of objects to be performed proportionally as well as transforming points between the image plane and the ground plane.
[0042] As an example, a geometric algorithm can be used to compute camera extrinsics based on scene cues. The camera projection matrix P = [R, t] can be parameterized based on its view angle and height above the ground plane. Concepts from single view geometry can be used to solve for these parameters. For example, assume a pinhole camera model using known camera intrinsics K, observing the so-called "Manhattan world". For camera angles, some line segments can be annotated to compute the z direction v z Assuming zero roll, the horizon is the corresponding image line v0. The rotation vector is r^3 = K -1 v z / K -1 v z ; which can be matched with the rotation matrix to recover the yaw angle = tan -1 (r^ 31 / r^ 33 ), and pitch angle = cos -1 (r^ 32 ). For the camera height, let y c The height of the camera is expressed in world coordinates. Given a known object image height h i , bottom position v i and world (physical) height y i , use the following formula to calculate the camera height y c ,y c =y i (v i v0) / h i The camera height estimate can be averaged over multiple object instances in the scene to obtain a robust estimate.
[0043] Figure 3An example depiction depicting anchor placement for object detection is shown. Depiction 300 represents anchor placement for conventional object detection. The classifier views the scene in the XY image plane, determines the expected region of interest, and then places a bounding box on the image. In training examples, the region of interest may be determined based at least in part on ground truth data. A high confidence bounding box may be assumed to be associated with an object in the foreground and therefore has a high likelihood of occluding other objects in the scene.
[0044] Depicted diagram 310 represents anchor placement for occluding objects. To determine where to place the anchor, in some examples, once the occluder is identified, an occlusion region is generated, and then multiple RoIs are candidate based on the occlusion region. Then a bounding box is built around the original bounding box for the occluder. As shown in 310, XZ represents the ground plane. Based on the location of potential occluders, concentric circular areas are drawn in XZ. Scene cues can be used to build anchors, and the Y dimension can be known from the bounding box drawn on the XY plane. Therefore, the anchored bounding box can be projected to the XZ ground plane.
[0045] In this example, concentric circles are drawn around the ground plane. Circles can be used to generate ground contact points and anchors are then placed at these locations. In other examples, any other suitable shape can be used. New bounding boxes can be placed in cones behind the occluders relative to the camera. Each new bounding box is anchored to one of the concentric circles at the intersection of the LoS and the circle. In the regression phase of the model, the associated anchors are added to the original anchors.
[0046] Figure 4 An example of occlusion region generation is depicted. In scene 400, two occluders 410 and 412 are shown positioned relative to camera 415 and ground plane 420. Occluder-centered anchors can be generated that are identified by their position relative to foreground occluders 410 and 412 and the layout of the particular scene. In particular, occlusion occurs when multiple objects appear at different depths along the camera's line of sight (LoS).
[0047] The cone-shaped space behind the occluder encloses the occluded region in 3D. For example, occlusion cone 430 is generated behind occluder 410, and occlusion cone 432 is generated behind occluder 412. Objects occupying occlusion cones 430 and 432 may appear partially visible in the 2D image data. For example, depiction 440 shows an example xz projection (bird's eye view) of scene 400, which includes camera 415 and occlusion cones 430 and 432. Each occlusion cone is generated based on a real-world size because the region candidates are projected into the 2D image based on this size. Each occlusion cone is defined by a set of parameters, such as x, z, r, and θ, such as Figure 3As shown, where θ is the angle of the line of sight based on the occluder.
[0048] Anchors centered on the occluder can be generated as dense anchors appearing in the occlusion cone. For example, at 500, Figure 5 An example occluder 510 with a foreground bounding box 512 is shown. Bounding box 512 includes a portion of an occluded individual 515. Bounding boxes are generated based at least on scene geometry, the relative size of the occluder, and the occlusion cone. An example is shown at 520, where multiple bounding boxes 525 for an occlusion cone (not shown) of the occluder 510 are generated. These bounding boxes can be placed within the 3D geometry and then projected back to 2D. As an example, using a known or inferred camera matrix, the 2D bounding box is projected onto the ground plane. This can help to clearly classify the occluded individual 515 in at least one of the bounding boxes 525. The new box covers less of the occluder and covers more of the wings of the occlusion cone. In this way, the indication of the attention mask for the classifier is adjusted, thereby pushing attention to the wing area and away from the occluding object.
[0049] The region proposals in the two-stage object detector simulate a sliding window search over all scales and aspect ratios and are drawn from a fixed set of anchors that are constant across all images. During the RPN stage, anchors are classified into foreground and background classes based on how much they overlap with the ground truth boxes. In the example described in this paper, anchors generated for occluded regions are also fed to the RPN stage. By doing so, RoIs that can be position-sensitively occluded during training can be explicitly prioritized.
[0050] As an example, let (x o ,z o ) is the contact point of the bounding box on the ground. The LoS is the ray connecting the camera center to this point. Let the angle of this Los be θ los,o The occlusion cone associated with this box can then be parameterized as (x o ,z o ,r,θ), where r represents the radial distance from o, and θ = θ los,o θ Δ ,...,θ los,o +θ Δ Indicates the angular separation from Los.
[0051] Next, at each parameterized (x o ,z o ,r,θ), a hypothetical RoI is generated corresponding to the dimensions of the expected person. Specifically, given (r,θ), the 3D positions of the top left and bottom right positions in 3D are (x o -w / 2,h,z o ) and (xo +w / 2,0,z o ), where (h,w) are the expected human height and width (approximately h = 5.6 feet, w = 1.7 feet). These 3D boxes are then reprojected back to the image plane in the form of 2D RoIs. Anchors that overlap these RoIs can be indexed by their corresponding (r,θ) values. Finally, a subset of these anchors can be uniformly sampled from the (r,θ) distribution.
[0052] After the candidate annotation stage, b fg ,b occ ,b bg can be represented as a random subset of foreground, occluded, and background anchors, and the combined loss function at the region classifier stage is expressed as follows:
[0053] Equation 1
[0054]
[0055] where λ is an appropriate scaling function for each candidate set σ, and l1 represents the combination of a softmax loss on the class likelihood and a smooth-L1 loss for bounding box regression between the candidate and the corresponding ground truth box C. Background candidates do not carry a box regression penalty.
[0056] This stage ends with bounding box regression, but classification is not yet complete. After a second regression with a smaller set of boxes, the classification stage can perform better. Introducing a geometric loss term can help ensure that candidate contact points are appropriately close to the corresponding ground truth boxes. This can be done by computing the projection matrix using the established geometric context.
[0057] For example, Figure 6 An example scene 600 is shown, including an occlusion bounding box 610 and an image plane bounding box 620. The occlusion bounding box 610 is generated as a projection of the image plane bounding box 620 along the Z axis into 3D. However, the occlusion bounding box 610 must ultimately be evaluated in the image plane for classification. Thus, using the inverse projection matrix, the occlusion bounding box 610 can be back-projected from 3D to 2D to generate an estimated bounding box in the image plane. In order to efficiently perform this projection, it is desirable to minimize the distance function by back propagation.
[0058] In Equation 1, the multi-task loss This includes a classification loss that penalizes mismatches of object categories, and a regression loss that penalizes the offset of anchor boxes from neighboring ground truth in pixels. In addition, a geometric loss can be incorporated that encodes the position consistency of anchors on the ground.
[0059] In Faster-RCNN, the regression target is parameterized into four dimensions through the convolutional subnetwork bbox_reg as t x =(x g x a ) / w a ,t y =(y g y a ) / h a ,t w =log(w g / w a ),t h =log(h g / h a ),in(.) g as well as(.) a denote the ground truth and anchor respectively, and (x,y) is the box midpoint, and (w,h) is the corresponding width and height. The technique in R-FCN is similar, except that a position-sensitive map is used as an intermediate representation. In both methods, anchor boxes are matched to ground truth boxes based on intersection over union (IoU) overlap. The bbox_reg subnet is expected to learn a small bias to "correct" the anchor positions and align them perfectly. However, good anchor candidates may be farther away from the object than what is learned by correcting the bias. Some bounding boxes may have a large overlap with the chosen ground truth, but be equidistant from nearby ground truth boxes. This leads to coarse localization because multiple ground truth boxes compete for the same set of anchors.
[0060] Given the camera pose with respect to the ground plane, a projection loss function can be introduced to try to address these issues by extending the regression criterion. To this end, two additional dimensions are added to the regression objective, which are w and z w The normalized distance between the predicted anchor and the object in , such as t x w =(tpx g tpx a ) / g x and t w =(tpz g tpz a ) / g z , where tp (.) Represents the contact point of the bounding box projected onto the ground plane in the xz dimension.
[0061] To build the target for regression, the 2D box coordinates (x, y) need to be projected to the position (x w ,y w ,z w). When the accurate recovery from 2D points to 3D is the wrong pose, two reasonable assumptions help make the solution tractable: (a) The pedestrian rests on the ground plane, e.g., y w = 0; and (b) the direction of the ground plane is parallel to the upright person, e.g. Based on these constraints, the position of the RoI on the ground plane (denoted as G(b) yw = 0) is computed algebraically as follows: (a) The camera center is found, given by the empty space of the camera matrix P = null(P); (b) The ray radiated from the camera center to the object is given by: R = inv(P)*X im , where X im The bottom coordinate of the image frame; and (c) the projected ray R and the ground unit vector The intersection points are calculated algebraically to give the corresponding contact points Xg on the ground.
[0062] Given the 3D position of the bounding box, the 2D+3D regression loss can be estimated. Let the foreground set and occlusion anchors be {b fg ,b occ}, and the corresponding ground truth box is C. Then, the bias is penalized by a smooth L1 function. The predicted feature map is expanded to accommodate regression terms along two dimensions, normalized by the width and height of the ground plane, respectively.
[0063] Equation 2
[0064]
[0065] in
[0066] Figure 7 A flow chart of an example method 700 for training a classifier for object classification is depicted. The method 700 can be performed by one or more computing systems and can result in the output of a classifier and / or classification module that can be attached to an object detection and classification system, such as a deep neural network.
[0067] At 705, method 700 includes receiving monocular image data for a scene. The monocular image data may include one or more digital images of the scene taken from a static camera position. At 710, method 700 includes receiving ground truth data for the scene. The ground truth data may include previously attached data describing and parsing the image data. For example, the ground truth data may include multiple objects represented in the image data, may include previously generated regions of interest, etc. The ground truth data may also include additional information about the scene, such as a scene footprint, background information, and / or additional data related to object detection within the monocular image data.
[0068] At 715, method 700 includes determining a geometric context for the scene, the geometric context including a three-dimensional camera position relative to a fixed plane. In some examples, the fixed plane can be a ground plane. In some examples, determining the geometric context for the scene can include extracting the three-dimensional camera position relative to the fixed plane from ground truth data. In some examples, determining the geometric context for the scene can include deriving the three-dimensional camera position relative to the fixed plane from monocular image data, as described herein.
[0069] At 720, method 700 includes determining a set of regions of interest in the monocular data. The set of regions of interest may be determined in any suitable manner, and / or may be included in ground truth data for the scene. Each region of interest may include anchor, size and shape data for the object in question, and / or other data for downstream analysis that may be used to generate the region of interest.
[0070] At 725, method 700 includes identifying a set of one or more potential occluders within the region of interest. For example, the one or more potential occluders may be determined based on a confidence level for the region of interest being greater than a threshold. In other words, if there is a high probability that the region of interest contains a large portion of an object, and that the object is likely to be within the direct line of sight of the camera, it may be identified as a potential occluder for a more distant object in the background.
[0071] At 730, method 700 includes, for each identified potential occluder, projecting an occlusion region in three dimensions to a fixed plane based on the geometric context. Projecting the occlusion region may include generating an occlusion cone in three dimensions based on the geometric context. Figure 3 and Figure 4 As described in
[15] , the occlusion cone can be determined based on the radial distance to the anchor point and the angular separation from the camera and the line of sight between the anchor point and the fixed plane.
[0072] At 735, method 700 includes generating a set of occluded regions of interest for a fixed plane in three dimensions for each occluded region. For example, generating the set of occluded regions of interest may include projecting the regions of interest onto the fixed plane of the occlusion cone based on the geometric context, such as with respect to Figure 5 Additionally, at 740, method 700 includes projecting each occluded region of interest back into two-dimensional image data.
[0073] At 745, method 700 includes training a classifier. Training the classifier includes, at 750, minimizing a loss function generated by inputting information about a set of regions of interest and a set of occluded regions of interest into the classifier. For example, foreground, background, and occluded regions of interest may be sampled to generate each type of RoI set. Minimizing the loss function may then include modeling object position and scale consistency during calculation of a per-region loss for each of the foreground, background, and occluded regions of interest. A technical effect of training a classifier in this manner is a reduced classification error rate.
[0074] At 755, training the classifier includes minimizing the position error of each region of interest and each occluded region of interest for the fixed plane set based on the ground truth data. Minimizing the position error may include reprojecting each of the foreground, background, and occluded regions of interest from the ground truth back to 3D space. In this way, the estimated region of interest may be regressed to the known object location by penalizing the bias of incorrectly placed anchors.
[0075] At 760, method 700 includes outputting a classifier for object detection. The classifier may include one or more candidate-based deep neural networks, convolutional neural networks, or other neural network types. In some examples, method 700 includes outputting a module that can be integrated into multiple two-stage object detection architectures.
[0076] Figure 8 A flow chart of an example method 800 for deploying a trained classifier, such as a classifier trained via method 700, is depicted. Method 800 may be performed by one or more computing systems, cloud computing systems, web applications, or other platforms that execute a two-stage object detection and classification model.
[0077] At 805, method 800 includes receiving monocular image data for a scene. The monocular image data may include one or more digital images of the scene taken from a static camera position. At 810, method 800 includes determining a geometric context for the scene, the geometric context including a three-dimensional camera position relative to a fixed plane. In some examples, determining the geometric context for the scene may include deriving the three-dimensional camera position relative to the fixed plane from the monocular image data, as described herein. The fixed plane may be a ground plane or other suitable plane.
[0078] At 815, method 800 includes generating a set of regions of interest in the monocular data, for example, by generating a set of RoIs in the image data, and then filtering the RoIs via one or more region candidate networks trained on the desired object. For example, each region of interest can be specified by drawing a bounding box on the monocular image data for each region of interest. Each region of interest can be fitted around the expected object by the region candidate network. The size of the region of interest can be based on the size of the desired object. The location of the region of interest can be based on a geometric context, for example, anchored to a fixed plane.
[0079] At 820, method 800 includes identifying a set of one or more potential occluders within the region of interest. For example, the one or more potential occluders may be determined based on a confidence level for the region of interest being greater than a threshold. In other words, if there is a high likelihood that the region of interest contains a large portion of the object, it may be identified as a potential occluder for a more distant object in the background.
[0080] At 825, method 800 includes, for each potential occluder identified, projecting an occlusion region in three dimensions to a fixed plane based on the geometric context. Projecting the occlusion region may include generating an occlusion cone in three dimensions based on the geometric context. Figure 3 and Figure 4 As described in
[15] , the occlusion cone can be determined based on the radial distance to the anchor point and the angular separation from the camera and the line of sight between the anchor point and the fixed plane.
[0081] At 830, method 800 includes drawing a set of occluded regions of interest for each occluded region. For example, generating the set of occluded regions of interest may include placing the regions of interest on a fixed reference plane along the occlusion cone based on the geometric context, such as with respect to Figure 5 The region of interest can then be projected from the 3D space back to the 2D space.
[0082] At 835, method 800 includes classifying each region of interest and each occluded region of interest as a likelihood of being an object of interest. The classification criteria may be based in part on geometric context. For example, an overhead camera may be unlikely to capture a person's feet, so this criterion may be minimized.
[0083] At 840, method 800 includes outputting a classification for each region of interest and each occluded region of interest. In examples where an object is partially included in multiple overlapping occluded regions of interest, the occluded region of interest with the highest likelihood may be indicated as likely to be the object of interest, while adjacent overlapping occluded regions of interest with less likelihood may be designated as not containing the object of interest to avoid double counting of objects.
[0084] The example training algorithms described above are evaluated on two open source pedestrian datasets - ETH WildTrack and MOT 2017 Detection (MOT). The WildTrack dataset contains seven pedestrian sequences from overlapping viewpoints using wide-angle static cameras with variable camera heights. The cameras are jointly calibrated and the corresponding camera intrinsics and extrinsics are available. The MOT dataset is used as an example to test the example algorithms in environmental scenes. The MOT dataset includes seven sequences of various indoor and outdoor pedestrian traffic. The scene calibration is performed using single view geometry (e.g., as described on Figure 1 and 2 described) and the inferred geometric parameters. Both datasets contain captured video frames of crowded dynamic scenes with moving pedestrians (~17 pedestrians per frame), using static cameras mounted at various vantage points (e.g. surveillance view, view below eye level, etc.), and provide full body annotations on all ground truth instances. For evaluation, each video is temporally partitioned such that the first 60% of the frames are used for training and validation, while the remaining 40% of the frames are isolated for testing.
[0085] The example algorithm is evaluated on two well-studied architectures - Region-based Fully Convolutional Network (R-FCN) and Faster-RCNN, where the example algorithm is pipelined on top of these.
[0086] All training experiments share the same protocol. Resnet-101 was used as the backbone. The model was pre-trained on the COCO dataset and fine-tuned by freezing all layers until the 4th resnet block and retraining all layers thereafter. Training was performed using SGD with momentum 0.9, weight decay 5e-4, initial learning rate 5e-5, and batch size 1. Each training routine was allowed to run for 20 epochs. No hyperparameter tuning or data augmentation was performed. An intersection over union (IoU) of 0.7 was used as the threshold for positive detection and mean average precision (mAP) was used as the evaluation metric.
[0087] A budget of 256 RoIs is set at the RPN stage. In the baseline model, the proportion of foreground RoIs is set to 25%, and the remaining 75% is designated as background. In the example algorithm, the occluded-RoI accounts for 12.5% of the total RoI, making the ratio of foreground, occluded, and background RoIs 25:12.5:62.5. The occluded RoIs are sampled from the (r,θ) distribution using uniform sampling. The occluded anchors are densely sampled by varying the distance to the occluder (r) and the direction along the line of sight (θ). Depending on the height of the camera, the radius ranges from 1-10 feet for the overhead camera view and 1-50 feet for the horizontal and below eye level views, with each increment of 1 foot. The angle away from the line of sight ranges from 0 to 60 degrees, with an increment of 5 degrees. To project the anchors back to the image plane, the average human height is assumed to be 5.5 feet for MOT and 5.9 feet for Wildtrack (as provided in the dataset).
[0088] Table 1
[0089]
[0090] Table 2
[0091]
[0092] Using the example algorithm, higher mAP is achieved across both datasets and detection architectures. As shown in Tables 1 and 2 above, the baseline R-FC achieves 75.24% on Wildtrack (Table 1) and 86.20% on MOT (Table 2). Fig. 9The precision-recall curves for the Geometry-RFCN model for Wildtrack (curve 900) and MOT (curve 910) are shown. Adding the example algorithm improves these scores to 93.02% on MOT and 78.16% on Wildtrack, improvements of 8% and 3%, respectively. Fast RCNN has been observed to outperform R-FCN in the art. Adding the example algorithm outperforms the baseline Fast RCNN model by an average of 2%. Tables 1 and 2 show the mAP for each camera view, with the example algorithm consistently outperforming the baseline on all camera views. Interestingly, the mAP gain for the overhead view is higher than for the horizontal (eye level) view.
[0093] As described in this paper, occluded samples can be better detected by the disclosed geometry-initiated method. These results are shown in Table 3. In fact, a gain of 17 percentage points (pp) was observed on Wildtrack and a gain of 9 pp was observed on the MOT dataset. The gain on the occluded part is significantly higher than the gain on the entire dataset. This shows that occluded instances can be better delineated using the example algorithm.
[0094] Table 3
[0095]
[0096] Cross-camera performance is evaluated on the Wildtrack dataset to test whether high-level models trained on a subset of views may also perform better on unobserved views that capture the same underlying scene. Camera views C1-C4 are used for training, and views C5-C7 are tested. As shown in Tables 4 and Fig.10 As shown in the depiction diagram 1000, on the unobserved view, the detection accuracy is improved by 2% over the baseline. The accuracy on the ground plane is also evaluated by measuring the average Euclidean distance (in centimeters) between the predicted position and the ground. Compared to the baseline, a 15.3% reduction in localization error is achieved, as shown in Table 4 and Fig.10 The depiction diagram 1000 is shown.
[0097] Table 4
[0098]
[0099] The mAP gain is then analyzed by camera altitude, spatial distance, and radial angle between occluding objects on the ground plane. The data is shown in Fig.111100 and 1110 depicted in . In Figure 1100 , the camera altitude (x-axis) is shown to have a significant impact on the mAP gain. The higher the camera, the better the disclosed model represents the pattern of occlusion. At eye level, a person may be completely occluded, making it more difficult to model the relationship using geometric constraints. For example, views MOT11, WT4, and WT6, all captured at eye level, show lower mAP gains. This result reflects why most surveillance cameras prefer top views for better scene capture. In Figure 1110 , the gain is plotted with reference to the angle of deviation from the line of sight, where los = 0° is directly behind the occluder. The low-level and overhead camera views show similar patterns - higher gains are achieved with slight angular separations of ~15°. Intuitively, these samples are very challenging without the explicit attention provided by the geometric context. In contrast, at larger radial separations, such as side-by-side configurations, gains can be seen at eye level.
[0100] Table 5 shows the evaluation results of the impact of various design choices and modules in the example algorithm. In the first experiment, occluder-centered candidates were considered during inference alone, as shown in columns 1 and 2. Specifically, two rounds of inference were run using the pre-trained COCO model. In the second round, the top K detections from the first round were used, and occluder-centered candidates were generated, which were centered around these detections. The strategy achieved a gain of 41 points, from 48% in the first round to 68% using enhanced candidates, as shown in column 2. The gain for MOT is smaller, probably because pedestrians appear in various poses that violate occlusion patterns. Next, the example algorithm was evaluated by training with and without online hard negative mining (OHEM), as shown in columns 3 and 4. OHEM retroactively selects a subset of RoIs with the highest loss and backpropagates based on the selected examples. Without OHEM, hard (occluded) ROIs are actively selected. The strategy performs better than the example with OHEM. Finally, the impact of occlusion-centric proposals (early fusion of geometry) is evaluated alone, and the combined impact of occlusion-centric proposals and 3D geometry loss (early + late fusion) is also evaluated, as shown in columns 5 and 6. Each module improves mAP individually, but the best performance is achieved by using both modules together, as shown in column 6.
[0101] Table 5
[0102]
[0103] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.
[0104] Fig.12 A non-limiting embodiment of one or more computing systems 1200 that can perform the above methods and processes is schematically shown. The computing system 1200 is shown in simplified form. The computing system 1200 can take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices.
[0105] The computing system 1200 includes a logic machine 1210 and a storage machine 1220. The computing system 1200 may optionally include a display subsystem 1230, an input subsystem 1240, a communication subsystem 1250, and / or Fig.12 Other components not shown.
[0106] Logic machine 1210 includes one or more physical devices configured to execute instructions. For example, the logic machine may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical structures. Such instructions may implement execution tasks, implement data types, transform the state of one or more components, achieve technical effects, or otherwise achieve a desired result.
[0107] The logic machine may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processors of the logic machine may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The various components of the logic machine may optionally be distributed in two or more separate devices, which may be remotely located and / or configured for coordinated processing. Aspects of the logic machine may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.
[0108] Storage machine 1220 includes one or more physical devices configured to hold logical machine executable instructions to implement the methods and processes described herein. When implementing such methods and processes, the state of storage machine 1220 may be transformed—for example, to hold different data.
[0109] The storage machine 1220 may include removable and / or built-in devices. The storage machine 1220 may include optical storage (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor storage (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic storage (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), etc. The storage machine 1220 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices.
[0110] It should be understood that storage machine 1220 includes one or more physical devices. However, various aspects of the instructions described herein may alternatively be propagated by a communication medium (eg, electromagnetic signals, optical signals, etc.) that is not retained by a physical device for a finite time.
[0111] Aspects of the logic machine 1210 and the storage machine 1220 may be integrated into one or more hardware logic components. Such hardware logic components may include field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs), for example.
[0112] The terms "module," "program," and "engine" may be used to describe an aspect of a computing system 1200 that is implemented to perform a particular function. In some cases, a module, program, or engine may be instantiated by the logic machine 1210 executing instructions held by the storage machine 1220. It should be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" may include a single or a group of executable files, data files, libraries, drivers, scripts, database records, and the like.
[0113] It should be understood that a "service" as used herein is an application program executable across multiple user sessions. A service may be available to one or more system components, programs, and / or other services. In some embodiments, a service may run on one or more server computing devices.
[0114] When included, the display subsystem 1230 can be used to present a visual representation of the data stored by the storage machine 1220. The visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data maintained by the storage machine, thereby changing the state of the storage machine, the state of the display subsystem 1230 can also be converted to visually represent the changes in the underlying data. The display subsystem 1230 can include one or more display devices that virtually utilize any type of technology. Such a display device can be combined with the logic machine 1210 and / or the storage machine 1220 in a shared housing, or such a display device can be a peripheral display device.
[0115] When included, input subsystem 1240 may include or interface with one or more user input devices (e.g., a keyboard, mouse, touch screen, or game controller). In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and conversion and / or processing of input actions may be handled on-board or off-board. Example NUI components may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity.
[0116] When included, the communication subsystem 1250 can be configured to communicatively couple the computing system 1200 with one or more other computing devices. The communication subsystem 1250 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem can be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network. In some embodiments, the communication subsystem can allow the computing system 1200 to send and / or receive messages to and / or from other devices via a network such as the Internet.
[0117] As an example, a computing system for training a classifier to classify an object, the computing system comprising: a logic machine; and a storage machine storing instructions that, when executed, cause the logic machine to: receive monocular image data for a scene; receive ground truth data for the scene; determine a geometric context for the scene, the geometric context comprising a three-dimensional camera position relative to a fixed plane; determine a set of regions of interest within the monocular image data; identify one or more sets of potential occluders based on the set of regions of interest; for each identified potential occluder, project the occluded region onto a fixed plane in three dimensions based on the geometric context; for each occluded region, generate a set of occluded regions of interest on the fixed plane in three dimensions; project each occluded region of interest back to the image data in two dimensions; train the classifier by: minimizing a loss function generated by inputting information about the set of regions of interest and the set of occluded regions of interest into the classifier; and, based on the ground truth data, minimizing the position error of each region of interest in the set and each occluded region of interest on the fixed plane; and outputting a classifier for object detection. In this example or any other example, the fixed plane may additionally or alternatively be a ground plane. In any of the above examples, or any other example, additionally or alternatively, instructions are executed to determine a geometric context for the scene by extracting a three-dimensional camera position relative to the fixed plane from ground truth data. In any of the above examples, or any other example, additionally or alternatively, instructions are executed to determine a geometric context for the scene by obtaining a three-dimensional camera position relative to the fixed plane from monocular image data. In any of the above examples, or any other example, additionally or alternatively, instructions are executed to determine one or more potential occluders based on a confidence level for the region of interest being greater than a threshold. In any of the above examples, or any other example, additionally or alternatively, instructions are executed to project an occluded area by generating an occlusion cone in three dimensions based on the geometric context. In any of the above examples, or any other example, additionally or alternatively, instructions are executed to generate a set of occluded regions of interest by projecting a bounding box to a fixed plane of the occlusion cone based on the geometric context. In any of the above examples, or any other examples, additionally or alternatively, instructions are executed to minimize a loss function by modeling object position and scale consistency during calculation of a per-region loss for each of the foreground, background, and occluded regions of interest. In any of the above examples, or any other examples, additionally or alternatively, minimizing the position error includes reprojecting each of the foreground, background, and occluded regions of interest and the ground truth back to 3D space. In any of the above examples, or any other examples, additionally or alternatively, the classifier comprises one or more candidate-based deep neural networks.
[0118] In another example, a computing system for performing object detection includes: a logic machine; and a storage machine storing instructions, which when executed cause the logic machine to: receive monocular image data for a scene; determine a geometric context of the scene, the geometric context including a position of a three-dimensional camera relative to a fixed plane; generate a set of regions of interest within the monocular image data; identify a set of one or more potential occluders within the set of regions of interest; for each potential occluder identified, based on the geometric context, project the occluded region onto a fixed plane in three dimensions; for each occluded region, draw a set of occluded regions of interest; classify each region of interest and each occluded region of interest as a likelihood of being an object of interest; and output a classification for each region of interest and each occluded region of interest. In such an example system or any other example system, the fixed plane may additionally or alternatively be a ground plane. In any of the above examples, or any other example, additionally or alternatively, the instructions are executed to determine the geometric context for the scene by obtaining the three-dimensional camera position relative to the fixed plane from the monocular image data. In any of the above examples, or any other examples, additionally or alternatively, instructions may be executed to determine a set of one or more potential occluders based on a confidence level for the region of interest being greater than a threshold. In any of the above examples, or any other examples, additionally or alternatively, instructions may be executed to project an occluded region by generating an occlusion cone in three dimensions based on a geometric context. In any of the above examples, or any other examples, additionally or alternatively, instructions are executed to generate a set of occluded regions of interest by projecting the region of interest onto a fixed plane of the occlusion cone based on the geometric context. In any of the above examples, or any other examples, additionally or alternatively, instructions may be executed to classify each region of interest and each occluded region of interest by applying a classification criterion based in part on the geometric context.
[0119] In yet another example, a method for training a classifier to classify an object includes: receiving monocular image data for a scene; receiving ground truth data for the scene; determining a geometric context for the scene, the geometric context including a position of a three-dimensional camera relative to a ground plane; determining a set of regions of interest within the monocular image data; based on the set of regions of interest, identifying a set having one or more potential occluders; for each identified potential occluder, projecting the occluded region onto a ground plane in three dimensions based on the geometric context; for each occluded region, generating a set of occluded regions of interest on the ground plane in three dimensions; projecting each occluded region of interest back to the image data in two dimensions; training the classifier by: minimizing a loss function generated by inputting information about the set of regions of interest and the set of occluded regions of interest into the classifier; and minimizing the position error of each region of interest in the set and each occluded region of interest on the ground plane based on the ground truth data; and outputting the trained classifier for object detection. In any of the above example methods, or any other example method, additionally or alternatively, projecting the occlusion region may include generating an occlusion cone in three dimensions based on the geometric context. In any of the above example methods, additionally or alternatively, minimizing the loss function may include modeling object position and scale consistency during calculation of each per-region loss for the foreground, background, and occluded regions of interest.
[0120] It should be understood that the configuration and / or method described herein is exemplary in nature, and these specific embodiments or examples should not be considered restrictive, because many variations are possible. The specific routines or methods described herein can represent one or more of any number of processing strategies. Therefore, the various actions shown and / or described can be performed in the order shown and / or described, in other orders, in parallel or omitted. Likewise, the order of the above process can be changed.
[0121] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations disclosed herein, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.
Claims
1. A computing system for performing object detection, comprising: Logical Machines; as well as A storage machine stores instructions that, when executed by the logic machine, cause the computing system to: receiving image data for a scene; generating a set of regions of interest within the image data; identifying a set of one or more potential occluders within the set of regions of interest; classifying each region of interest and each occluded region of interest in the plurality of occluded regions of interest for each potential occluder as a likelihood of being an object of interest; as well as Output the classification for each region of interest and each occluded region of interest.
2. The computing system of claim 1, wherein the instructions are executable to: determine a geometric context for the scene, the geometric context comprising a three-dimensional camera position relative to a fixed plane.
3. The computing system of claim 2, wherein the instructions are executable to: determine the geometric context for the scene by deriving the three-dimensional camera position relative to the fixed plane from image data.
4. The computing system of claim 2, wherein the instructions are executable to: identifying potential occluders based on one or more of a likelihood that the region of interest contains a substantial portion of an object and a likelihood that the object intersects a direct line of sight of the three-dimensional camera; and For each potential occluder, an occlusion region is projected by generating an occlusion cone in three dimensions based on the geometric context. 5 . The computing system of claim 4 , wherein instructions are executable to: generate the set of occluded regions of interest by projecting a bounding box onto the fixed plane of the occlusion cone based on the geometric context. 6 . The computing system of claim 2 , wherein the instructions are executable to: determine the set of one or more potential occluders based on a confidence level for the region of interest being greater than a threshold. 7 . The computing system of claim 2 , wherein the instructions are executable to: classify each region of interest and each occluded region of interest by applying a classification criterion based in part on the geometric context.
8. A method for performing object detection, comprising: receiving image data for a scene; generating a set of regions of interest within the image data; identifying a set of one or more potential occluders within the set of regions of interest; classifying each region of interest and each of the plurality of occluded regions of interest for each potential occluder as a likelihood of being an object of interest; as well as Output the classification for each region of interest and each occluded region of interest.
9. The method according to claim 8, further comprising: A geometric context for the scene is determined, the geometric context comprising a three-dimensional camera position relative to a fixed plane.
10. The method according to claim 9, further comprising: The geometric context for the scene is determined by deriving the three-dimensional camera position relative to the fixed plane from image data.
11. The method according to claim 9, further comprising: For each potential occluder, an occlusion region is projected by generating an occlusion cone in three dimensions based on the geometric context.
12. The method according to claim 11, further comprising: The set of occluded regions of interest is generated by projecting a bounding box onto the fixed plane of the occlusion cone based on the geometric context.
13. The method according to claim 9, further comprising: The set of one or more potential occluders is determined based on a confidence level for the region of interest being greater than a threshold.
14. The method according to claim 9, further comprising: Each region of interest and each occluded region of interest is classified by applying a classification criterion based in part on the geometric context.
15. A computer-readable storage device comprising computer-executable instructions that, when executed, cause one or more processing units of a computing device to: receiving image data for a scene; generating a set of regions of interest within the image data; identifying a set of one or more potential occluders within the set of regions of interest; classifying each region of interest and each of the plurality of occluded regions of interest for each potential occluder as a likelihood of being an object of interest; as well as Output the classification for each region of interest and each occluded region of interest.
16. The computer-readable storage device of claim 15, wherein the instructions are further executable to: determine a geometric context for the scene, the geometric context comprising a three-dimensional camera position relative to a fixed plane.
17. The computer-readable storage device of claim 16, wherein the instructions are further executable to: determine the geometric context for the scene by deriving the three-dimensional camera position relative to the fixed plane from image data.
18. The computer readable storage device of claim 16, wherein the instructions are further executable to, for each potential occluder, project an occlusion region by generating an occlusion cone in three dimensions based on the geometric context.
19. The computer readable storage device of claim 18, wherein the instructions are further executable to: generate the set of occluded regions of interest by projecting a bounding box onto the fixed plane of the occlusion cone based on the geometric context.
20. The computer readable storage device of claim 16, wherein the instructions are further executable to: determine the set of one or more potential occluders based on a confidence level for the region of interest being greater than a threshold.
Citation Information
Patent Citations
Apportioning storage units amongst storage sites in a dispersed storage network
US20190065311A1
Context-based priors for object detection in images
CN107851191A
Method and apparatus to measure brand exposure in media streams and to specify regions of interest in associated video frames
GB0820392D0
Occlusion reducing transformations for three-dimensional detail-in-context viewing
US20020122038A1
