Monocular depth management from 3D bounding boxes

Abstract

The method includes capturing two-dimensional (2D) images of an environment adjacent to an ego-vehicle, the environment including at least dynamic and static objects. The method also includes generating a depth map of the environment based on the 2D images via a depth estimation network, wherein the depth estimates for dynamic objects in the depth map are more accurate than the depth estimates for static objects in the depth map. The method further includes generating a three-dimensional (3D) estimate of the environment based on the depth map and identifying positions of the dynamic objects in the 3D estimate. The method additionally includes controlling a behavior of the ego-vehicle based on the identified positions.

Description

[Technical field]

[0001] Certain aspects of the present disclosure relate generally to depth estimation systems. [Background technology]

[0002] Autonomous agents (e.g., vehicles, robots, etc.) rely on depth estimates to perform a variety of tasks. These various tasks may include building a three-dimensional (3D) representation of a surrounding environment or identifying 3D objects. The 3D representation may be used for a variety of tasks, such as localization and / or autonomous navigation. Improving the accuracy of the depth estimate may improve the accuracy of downstream tasks, such as generating the 3D representation or 3D object detection. It is desirable to improve the accuracy of depth estimates obtained from images captured by a sensor of an autonomous agent. Summary of the Invention

[0003] In one aspect of the present disclosure, a method is disclosed. The method includes capturing a two-dimensional (2D) image of an environment adjacent to an ego-vehicle. The environment includes at least dynamic objects and static objects. The method also includes generating a depth map of the environment based on the 2D image via a depth estimation network. An accuracy of the depth estimation for the dynamic objects in the depth map is higher than an accuracy of the depth estimation for the static objects in the depth map. The method further includes generating a three-dimensional (3D) estimation of the environment based on the depth map. The method further includes controlling a behavior of the ego-vehicle based on the identified location.

[0004] In another aspect of the present disclosure, a non-transitory computer readable medium having non-transitory program code recorded thereon is disclosed. The program code, when executed by a processor, includes program code for capturing a 2D image of an environment adjacent to an ego-vehicle, the environment including at least dynamic objects and static objects. The program code also includes program code for generating a depth map of the environment based on the 2D image via a depth estimation network. The accuracy of the depth estimation for dynamic objects in the depth map is higher than the accuracy of the depth estimation for static objects in the depth map. The program code further includes program code for generating a 3D estimation of the environment based on the depth map. The program code further includes program code for controlling a behavior of the ego-vehicle based on the identified location.

[0005] Another aspect of the disclosure is directed to an apparatus. The apparatus includes a memory, one or more processors coupled to the memory, and instructions stored in the memory. The instructions, when executed by the processor, are operable to cause the apparatus to capture a 2D image of an environment adjacent to an ego-vehicle, the environment including at least dynamic objects and static objects. The instructions also cause the apparatus to generate a depth map of the environment based on the 2D image via a depth estimation network. The accuracy of the depth estimation for dynamic objects in the depth map is higher than the accuracy of the depth estimation for static objects in the depth map. The instructions additionally cause the apparatus to generate a 3D estimation of the environment based on the depth map. The instructions further cause the apparatus to control a behavior of the ego-vehicle based on the identified location.

[0006] This has outlined broadly the features and technical advantages of the present disclosure in order that the detailed description that follows may be better understood. Additional features and advantages of the present disclosure are described below. Those skilled in the art should recognize that the present disclosure may readily be utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Those skilled in the art should also recognize that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features believed characteristic of the present disclosure, both as to its organization and method of operation, together with further objects and advantages, will be better understood from the following description taken in conjunction with the accompanying drawings. It is to be expressly understood, however, that each of the figures is provided for the purpose of illustration and description only and is not intended as a definition of the limits of the present disclosure. [Brief description of the drawings]

[0007] The features, nature, and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference characters identify correspondingly throughout the drawings.

[0008] [Figure 1] FIG. 1 illustrates an example of a vehicle in an environment according to an embodiment of the present disclosure. [Figure 2A] FIG. 2A is an example of a single image according to an embodiment of the present disclosure. [Figure 2B] FIG. 2B is an example of a depth map according to an embodiment of the disclosure. [Figure 2C] FIG. 2C is an example of a reconstructed target image according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 illustrates an example of a deep network according to an embodiment of the present disclosure. [Figure 4] FIG. 4 illustrates an example of a posture network according to an embodiment of the present disclosure. [Diagram 5] FIG. 5 illustrates an example training pipeline according to an embodiment of the disclosure. [Figure 6]FIG. 6 illustrates an example hardware implementation for a depth estimation system according to an embodiment of the present disclosure. [Figure 7] FIG. 7 shows a flow diagram for a method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the various concepts. However, it will be apparent to one skilled in the art that these concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0010] An agent, such as an autonomous agent, may perform various tasks based on the depth estimation. For example, the agent may generate a 3D representation of a scene based on images acquired from a sensor. The 3D representation may also be referred to as a 3D model, a 3D scene, or a 3D map. The 3D representation may facilitate various tasks, such as scene understanding, motion planning, and / or obstacle avoidance. For example, the agent may autonomously navigate through an environment based on the 3D representation. Additionally or alternatively, the agent may identify 3D objects based on the depth estimation.

[0011] Artificial neural networks, such as depth networks, can be trained to estimate depth from sensor measurements. Conventional systems for depth training emphasize depth performance as opposed to improving downstream tasks, such as 3D object detection. Depth training refers to a training phase for training a depth network to estimate depth from images. Aspects of the present disclosure are directed to improving depth estimation for downstream tasks by incorporating 3D object detection information as part of the training loss for depth training.

[0012] FIG. 1 illustrates an example of an ego vehicle 100 (e.g., an ego agent) in an environment 150 according to an embodiment of the present disclosure. As illustrated in FIG. 1, the ego vehicle 100 is traveling on a road 110. A first vehicle 104 (e.g., another agent) may be in front of the ego vehicle 100, and a second vehicle 116 may be next to the ego vehicle 100. In this example, the ego vehicle 100 may include a 2D camera 108, such as a 2D RGB camera, and a second sensor 106. The second sensor 106 may be another RGB camera, or another type of sensor, such as radar and / or ultrasonic. Additionally or alternatively, the ego vehicle 100 may include one or more additional sensors. For example, the additional sensor may be a side-facing and / or rear-facing sensor.

[0013] In one configuration, the 2D camera 108 captures a 2D image that includes objects within the field of view 114 of the 2D camera 108. The second sensor 106 may generate one or more output streams. The 2D image captured by the 2D camera includes a 2D image of the first vehicle 104 because the first vehicle 104 is within the field of view 114 of the 2D camera 108.

[0014] The information obtained from the second sensor 106 and the 2D camera 108 may be used to navigate the ego-vehicle 100 along a route when the ego-vehicle 100 is in an autonomous mode. The second sensor 106 and the 2D camera 108 may be powered from power provided by a battery (not shown) of the vehicle 100. The battery may also power the vehicle's motor. The information obtained from the second sensor 106 and the 2D camera 108 may be used to generate a 3D representation of the environment.

[0015] Aspects of the present disclosure improve depth estimation for objects in an environment. The improved depth estimation may improve downstream tasks, such as 3D object detection. A downstream task may refer to a task that is performed based on the depth estimation. In some implementations, ground truth points corresponding to the object may be selected based on 3D bounding box information. A weight for each ground truth point (e.g., pixel) corresponding to the object may be increased. The depth network improves the depth estimation for the object as a result of the increased weight.

[0016] Improved depth estimation may refer to depth estimation with increased accuracy. Objects may include objects that are not prominent in the input image, such as cars and pedestrians. Improving the accuracy of depth estimation for objects that are not prominent in the input image may sacrifice the accuracy of depth estimation for representative objects that are prominent in the input image, such as roads and buildings. Improving the accuracy of depth estimation for objects that are not prominent in the input image may improve object identification, thereby improving models for downstream tasks such as 3D object detection.

[0017] A supervised monocular depth network learns an estimation function by regressing input images to estimated depth outputs. Supervised training refers to learning from labeled ground truth information. For example, a traditional supervised monocular depth network may use ground truth depth (e.g., LiDAR data) to train a neural network as a regression model. In a supervised depth network, a convolutional neural network generates an initial rough prediction, which is then refined using other neural networks to generate more accurate results. As supervised techniques for depth estimation advance, the availability of target depth labels is reduced by the cost of generating labeled data. For example, labeling outdoor scenes is a time-consuming task.

[0018] Some conventional monocular depth solutions replace LiDAR information with vision-based information. That is, instead of using the direct point cloud from LiDAR, the depth network estimates the point cloud from a single image. In such an example, the conventional monocular depth solution uses the estimated point cloud to detect the 3D bounding box. As mentioned above, cameras (e.g., vision-based systems) are ubiquitous in most systems and are less expensive compared to LiDAR sensors. Therefore, camera-based solutions can be applied to a wider range of platforms. However, LiDAR systems still perform better than vision-based systems. Improving the accuracy of the depth estimation network can reduce the gap between LiDAR systems and vision-based systems. Conventional systems can also reduce the gap between LiDAR systems and vision-based systems by including information from a sparse LiDAR sensor during training or testing. Sparse LiDAR information can correct misalignment. These conventional systems reduce the use of LiDAR information.

[0019] The cost of LiDAR sensors may not make them economically feasible. Cameras, such as red-green-blue (RGB) cameras, may provide dense information. In addition, cameras may have higher economic feasibility compared to LiDAR sensors. Aspects of the present disclosure improve monocular depth estimation generated from a depth network trained in a supervised manner. The improved monocular depth estimation bridges the gap between LiDAR and vision-based solutions such that cameras can augment, complement, or replace range sensors. In some implementations, depth training (e.g., training for image-based depth estimation) can be self-supervised by incorporating geometric constraints inherent in the robot or via sparse depth labels from a calibrated LiDAR sensor.

[0020] Bridging the gap between depth estimation from monocular cameras and LiDAR sensors may reduce costs and increase robust solutions, as cameras complement the capabilities of ranging sensors. For example, compared to LiDAR sensors, cameras may perform better in some environments, such as rainy environments. Conversely, compared to cameras, LiDAR sensors may perform better in other environments, such as low light conditions. Thus, monocular depth estimation may improve the ability of agents to perform various tasks.

[0021] Furthermore, agents can generate a larger amount of image data compared to LiDAR data, which can be used as training data for the depth network. Therefore, the use of a monocular sensor can increase the amount of training data, thereby improving self-supervised monocular depth estimation.

[0022] As described above, aspects of the present disclosure improve 3D object detection from monocular images (e.g., pseudo LiDAR point clouds). 3D object detection is a building block for enabling autonomous navigation by autonomous agents. Currently, LiDAR information can be used for 3D object detection. It is desirable to improve 3D object detection by processing monocular point clouds rather than LiDAR information.

[0023] Accurate depth estimation may improve autonomous navigation through an environment. For example, accurate depth estimation may improve collision avoidance with objects such as cars or pedestrians. Aspects of the present disclosure are not limited to autonomous agents. Aspects of the present disclosure also contemplate agents operating in manual or semi-autonomous modes. In manual mode, a human driver manually operates (e.g., controls) the agent. In autonomous mode, an agent control system operates the agent without human intervention. In semi-autonomous mode, a human can operate the agent and the agent control system can override or assist the human. For example. The agent control system can override the human's operation to prevent a collision or to comply with one or more traffic rules.

[0024] In some examples, conventional systems obtain pseudo point clouds generated by a pre-trained depth network. The pre-trained depth network converts the input image into information required for 3D bounding box detection (e.g., pseudo point cloud). The pre-trained depth network includes monocular, stereo, or multi-view networks trained in a semi-supervised or supervised manner. The pre-trained depth network may be a general depth network that is not trained for a specific task. The pre-trained depth network may be referred to as an off-the-shelf network.

[0025] A pre-trained depth network may learn strong priors for the environment (e.g., ground-plane, vertical walls, structures). The learned priors may improve the overall depth results. However, the learned priors do not improve depth estimation for objects relevant to 3D object detection. Objects relevant to 3D object detection are often dynamic objects such as cars, pedestrians, etc. As known to those skilled in the art, dynamic objects are a challenging problem for monocular depth estimation in a semi-supervised setting. For example, the motion of dynamic objects violates the static world assumptions that form the basis for the photometric loss used to train depth networks.

[0026] As mentioned above, conventional off-the-shelf depth networks are trained using different metrics and losses than downstream tasks such as 3D object detection. Therefore, conventional off-the-shelf depth networks may reduce accuracy for related tasks. For example, conventional off-the-shelf depth networks can recover the ground accurately because the ground contains a large portion (e.g., a large number of pixels) in the image. In contrast, the number of pixels representing pedestrians may be smaller than the number of pixels representing ground such as roads. Therefore, pedestrians may be less relevant for depth estimation because the goal of monocular depth networks is to maximize the accuracy of predictions for all pixels.

[0027] Instead of using off-the-shelf depth networks developed only for depth estimation, aspects of the present disclosure train depth networks to improve downstream tasks such as 3D object detection. In some implementations, 3D object detection information can be incorporated as part of the training loss for depth training. 3D object detection information can be identified to train downstream task training. Thus, 3D object detection information is already available to the depth network. Conventional systems do not use such information for depth training.

[0028] In some implementations, if 3D bounding box information is not available at training time, the deep network reverts to depth training without 3D bounding box information. Thus, during the training phase, the network may learn from images with 3D bounding boxes and images without annotated 3D bounding boxes. Such implementations increase the flexibility of depth training to use different sources of information so that available labels are not discarded.

[0029] In some implementations, based on training, the depth network performs depth estimation based on the input images at test time, i.e., even if the depth network was trained using 3D bounding box information and, if available, ground truth depth information, the depth network may only use the input images for depth estimation at test time.

[0030] FIG. 2A illustrates an example of a target image 200 of a scene 202, according to an embodiment of the disclosure. The target image 200 may be captured by a monocular camera. The monocular camera may capture an image in front of an agent (e.g., a vehicle). In one configuration, the monocular camera is integrated with the vehicle. For example, the monocular camera may be defined in a roof structure, windshield, heat dissipation grille, or other part of the vehicle. The vehicle may have one or more cameras and / or other types of sensors. The target image 200 may also be referred to as a current image. The target image 200 captures a 2D representation of the scene.

[0031] 2B illustrates an example of a depth map 220 of a scene 202, according to an embodiment of the present disclosure. The depth map 220 may be estimated from a target image 200 and one or more source images. The source images may be images captured at a previous time step relative to the target image 200. The depth map 220 provides the depth of the scene. The depth may be represented as a color or other feature.

[0032] 2C illustrates an example of a 3D reconstruction 240 of the scene 202, according to an embodiment of the disclosure. The 3D reconstruction may be generated from the depth map 220 along with the pose of the target image 200 and the source image. As illustrated in FIGS. 2A and 2C, the angle of view in the 3D reconstruction 240 is different than the angle of view of the scene 202 in the target image 200. Because the 3D reconstruction 240 is a 3D image of the scene 202, the angle of view may be changed as desired. The 3D reconstruction 240 may be used to control one or more actions of an agent.

[0033] 3 illustrates an example of a depth network 300 according to an embodiment of the present disclosure. As shown in FIG. 3, the depth network 300 includes an encoder 302 and a decoder 304. The depth network 300 generates a pixel-by-pixel depth map, such as the depth map 220 of FIG. 2B, for an input image 320.

[0034] The encoder 302 includes multiple encoder layers 302a-d. Each of the encoder layers 302a-d may be a packing layer for downsampling features during the encoding process. The decoder 304 includes multiple decoder layers 304a-d. In FIG. 3, each of the decoder layers 304a-d may be an unpacking layer for upsampling features during the decoding process. That is, each of the decoder layers 304a-d may unpack a received feature map.

[0035] The skip connections 306 transmit activation functions and gradients between the encoder layers 302a-d and the decoder layers 304a-d. The skip connections facilitate resolving higher resolution details. For example, gradients may be backpropagated directly to the layers via the skip connections 306, thereby improving training. In addition, the skip connections 306 transmit image details (e.g., features) directly from the convolutional layers to the deconvolutional layers, thereby improving image recovery at higher resolutions.

[0036] The decoder layers 304a-d may generate intermediate inverse depth maps 310. Each intermediate inverse depth map 310 may be upsampled before being concatenated to the feature map unpacked by the corresponding skip connection 306 and the corresponding decoder layer 304a-d. The inverse depth map 310 also serves as the output of the depth network on which the loss is computed. In contrast to conventional systems that incrementally super-resolve each inverse depth map 310, aspects of the present disclosure upsample each inverse depth map 310 to the highest resolution using bilinear interpolation. Upsampling to the highest resolution reduces copy-based artifacts and photometric blurring, thus improving depth estimation.

[0037] 4 illustrates an example pose network 400 for ego-motion estimation, according to an embodiment of the present disclosure. In contrast to conventional pose networks, the pose network 400 of FIG. 4 does not use an explainability mask, which in conventional systems removes objects that do not fit the static world assumptions.

[0038] 4, the posture network 400 includes multiple convolutional layers 402, a final convolutional layer 404, and a multi-channel (e.g., 6-channel) average pooling layer 406. The final convolutional layer 404 may be a 1×1 layer. The multi-channel layer 406 may be a 6-channel layer.

[0039] In one configuration, a target image (I t )408 and source image(I s ) 410 is input to the pose network 400. The target image 408 and the source image 410 may be concatenated together such that the concatenated target image 408 and source image 410 are input to the pose network 400. During training, one or more source images 410 may be used during different training epochs. The source images 410 may include images at a previous time step (t-1) and images at a subsequent time step (t+1). The output is a set of six degrees of freedom (DoF) transformations between the target image 408 and the source images 410. The process may be repeated for each source image 410 when more than one source image 410 is considered.

[0040] Different training approaches can be used to train a monocular depth estimation network. Training approaches can include, for example, supervised training, semi-supervised training, and self-supervised training. Supervised training refers to training a network to regress ground truth depth information by applying losses such as L1 loss (e.g., absolute error). In self-supervised training, the depth and pose information are warped to generate a reconstructed image (e.g., a 3D representation of a 2D image). A photometric loss minimizes the difference between the original image and the reconstructed image. Semi-supervised training can be a combination of self-supervised and supervised training.

[0041] As mentioned above, in some implementations, ground truth points corresponding to a first set of objects in an image have a heavier weight than ground truth points corresponding to a second set of objects in the image. The set of objects may be determined based on a desired task. For example, for 3D object detection, the first set of objects may be dynamic objects and / or objects with a lower occurrence rate in the input images. As an example, the first set of objects may include vehicles and / or pedestrians. Additionally, the second set of objects may be static objects and / or objects with a higher occurrence rate in the input images. As an example, the second set of objects may include buildings, roads, and / or sidewalks. In most cases, a sequence of images contains more occurrences of buildings, roads, and / or sidewalks compared to occurrences of humans and / or vehicles.

[0042] Aspects of the present disclosure augment monocular depth estimation training data with 3D bounding boxes, which may be used in addition to depth information during training, i.e., a monocular depth network may be trained with 3D bounding boxes and ground truth depth information.

[0043] In one configuration, the weights of pixels in the 3D bounding box are adjusted. A pseudo point cloud can be generated by increasing the relevance of pixels in the 3D bounding box. Increasing the weights (e.g., relevance) of pixels may reduce the depth metric. However, increasing the weights of pixels still improves 3D object detection.

[0044] 3D object detection can be improved by focusing on parts of an image that are deemed relevant to an assigned task (e.g., 3D object detection). That is, a depth network can be trained for the 3D object detection task. Task-specific training contrasts with conventional systems that use pre-trained general depth networks that are not conceptualized or trained for a specific task.

[0045] Aspects of the present disclosure may improve supervised, self-supervised, and semi-supervised training. For supervised training, different weights may be applied to pixels of an image that contain ground truth information. These pixels may be identified from annotated depth maps. For example, the depth maps may be annotated with bounding boxes. Weights for pixels within the bounding boxes may be adjusted.

[0046] In some aspects, for self-supervised training, the 3D bounding box is backprojected onto the input image to generate a 2D projection. Different weights may be applied to pixels within the 2D reprojected bounding box. FIG. 5 illustrates an example training pipeline 500 for training a depth estimation network 504 according to an aspect of the disclosure. As shown in FIG. 5, the depth estimation network 300 may generate depth estimates 506 from a two-dimensional input image 502 as described in FIG. 3. The training pipeline 500 is not limited to using the depth estimation network 300 as described in FIG. 3, and other types of depth estimation neural networks may be implemented.

[0047] The depth estimation network 504 may be used by the view synthesis module 508 to generate a reconstructed image 510 (e.g., a warped source image). In some implementations, the current image 502 and the source image 504 are input to a pose network as described in FIG. 4 to estimate the pose of a sensor (e.g., a monocular camera). The current image 502 may be an image at time step t, and the source image 502 may be an image at time step t-1. The view synthesis module 508 may generate the reconstructed image 510 based on the estimated depth and the estimated pose. The view synthesis module 508 may also be referred to as a scene reconstruction network. The view synthesis module 508 may be trained based on the difference between the target image 502 and the reconstructed image 510. The network may be trained to minimize losses such as photometric loss 520.

[0048] The photometric loss 520 is calculated based on the difference between the target image 502 and the reconstructed image 510 (e.g., a warped source image that approximates the target image). The photometric loss 520 may be used to update the depth network 300, the view synthesis module 508, the pose network 400, and / or pixel weights.

[0049] Photometric loss 520(L p ) can be determined as follows:

number

[0050] During the testing phase, the training pipeline 500 may generate a reconstructed image 510 as described above. The photometric loss 520 may not be calculated during the testing phase. The reconstructed image 510 may be used for localization and / or other vehicle navigation tasks.

[0051] For example, the view synthesis module 508 may project each point (e.g., pixel) in the current image 502 to a location in the source image 504 based on the estimated depth 506 and the sensor pose. After projecting the points onto the source image 504, bilinear interpolation may be used to warp the points to the warped source image 510. That is, bilinear interpolation obtains the value (e.g., RGB value) of the point in the warped source image 510 based on the source image 504.

[0052] That is, the location (e.g., x, y coordinates) of a point in the warped source image 510 may correspond to the location of the point in the target image 502. But the color of the point in the warped source image 510 may still be based on the color of neighboring pixels in the source image 504. The warped source image 510 may be a 3D reconstruction of a 2D target image.

[0053] In some implementations, the 3D object detection network 518 may estimate the location of the object 514 in the warped source image 510. The location of the object 514 may be annotated with a 3D bounding box 512. For illustrative purposes, the 3D bounding box 512 in FIG. 5 is shown as a 2D bounding box. In one configuration, the 3D bounding box 512 is back-projected onto the current image 502. A 2D bounding box 516 may be generated from the projected 3D bounding box 512. Different weights may be applied to pixels within the 2D bounding box. The weights may be increased, for example, so that pixels within the 2D bounding box contribute more to the depth estimation compared to pixels with lighter weights. In one implementation, pixels with increased weights contribute more to minimize loss during training compared to pixels with decreased weights.

[0054] The processes discussed for supervised and self-supervised training are performed for semi-supervised training. In some implementations, additional weights balance between the supervised and self-supervised training processes. In some implementations, multiple passes may be performed during the training phase, with each pass performed with different parameter values ​​(e.g., weights). Performance (e.g., accuracy) is measured after each pass. The weights may be ordered based on the magnitude of the loss and adjusted (e.g., improved) accordingly.

[0055] Pixelation may be caused by supervised loss. Self-supervised loss mitigates pixelation. Inaccuracies in scale may be caused by self-supervised loss. Supervised loss mitigates inaccuracies in scale. Additional weights may reduce pixelation of the depth map that may be caused by supervised loss. Additional weights may also reduce inaccuracies in scale of the depth map that may be caused by supervised depth error loss.

[0056] In one configuration, when determining pixel weight adjustments, the network determines a total number of valid pixels (NVP), which refers to pixels that have corresponding ground truth depth information or pixels that have valid reprojection values ​​when using photometric loss. For example, for supervised training, the neural network identifies pixels in an image that have depth information in the ground truth depth image.

[0057] In addition, the network determines the number of valid pixels that have a bounding box (NBP). The weight for the valid pixels that have a bounding box is determined based on the following pixel ratio ((NVP-NBP) / NBP). For example, if an image contains 100,000 valid pixels, and 2000 are within the bounding box, the weight for the pixels in the bounding box is 49 (e.g., (100000-2000) / 2000). In this example, the weight outside the bounding box is normalized to 1. In contrast, the weight for the interior of the bounding box is determined as the above ratio (49) (e.g., (NVP-NBP) / NBP). This pixel ratio will vary from image to image.

[0058] However, the pixel ratio still allows learning in regions that do not belong to the bounding box so that the structure and geometrical relationship of the scene are preserved. If the bounding box is not annotated in a particular image, the weight is zero and is not applied to any pixel. Therefore, aspects of the present disclosure may train a depth network from images with and without annotated 3D bounding boxes, thereby improving the robustness of the training data.

[0059] As described above, the ground truth points may be obtained from the 3D bounding box information. The detection of the first set of objects by the neural network (e.g., a depth estimation neural network) may be improved by adjusting the weights. The improved detection may be obtained at the expense of the detection of the second set of objects. That is, reducing the detection accuracy for the second set of objects improves the detection accuracy for the first set of objects. In one aspect, a model for a downstream task of 3D object detection is improved.

[0060] 6 is a diagram illustrating an example of a hardware implementation for a depth estimation system 600 according to an embodiment of the disclosure. The depth estimation system 600 may be a component of a vehicle, a robotic device, or other device. For example, as shown in FIG. 6, the depth estimation system 600 is a component of a vehicle 628. Aspects of the disclosure are not limited to the depth estimation system 600 being a component of a vehicle 628, as it is contemplated that the depth estimation system 600 may be used in other types of agents, such as a bus, a boat, a drone, or a robot.

[0061] The vehicle 628 may be operated in one or more of an autonomous, semi-autonomous, and manual operating mode. Additionally, the vehicle 628 may be an electric vehicle, a hybrid vehicle, a fuel-powered vehicle, or other type of vehicle.

[0062] The depth estimation system 600 may be implemented with a bus architecture, generally represented by bus 660. The bus 660 may include any number of interconnected buses and bridges, depending on the particular application of the depth estimation system 600 and the overall design constraints. The bus 660 links together various circuits, including one or more processors and / or hardware modules, represented by the processor 620, the communication module 622, the position module 618, the sensor module 602, the movement module 626, the navigation module 624, and the computer-readable medium 614. The bus 660 may also link various other circuits, such as timing sources, peripherals, voltage regulators, and power management circuits, which are well known in the art and will not be described further.

[0063] The depth estimation system 600 includes a processor 620, a sensor module 602, a depth estimation module 608, a communication module 622, a position module 618, a movement module 626, a navigation module 624, and a transceiver 616 coupled to a computer readable medium 614. The transceiver 616 is coupled to an antenna 644. The transceiver 616 communicates with various other devices over one or more communications networks, such as an infrastructure network, a V2V network, a V2I network, a V2X network, a V2P network, or other types of networks.

[0064] The depth estimation system 600 includes a processor 620 coupled to a computer readable medium 614. The processor 620 performs processes including executing software stored on the computer readable medium 614 to provide functionality according to the disclosure. The software, when executed by the processor 620, causes the depth estimation system 600 to perform various functions described for a particular device, such as a vehicle 628, or any of the modules 602, 608, 614, 616, 618, 620, 622, 624, 626. The computer readable medium 614 may also be used to store data that is manipulated by the processor 620 when executing the software.

[0065] The sensor module 602 may be used to obtain measurements via different sensors, such as a first sensor 606 and a second sensor 604. The first sensor 606 may be a vision sensor, such as a stereo camera or a red / green / blue (RGB) camera for capturing 2D images. The second sensor 604 may be a ranging sensor, such as a light detection and ranging (LIDAR) sensor or a radio detection and ranging (RADAR) sensor. Of course, aspects of the present disclosure are not limited to the above sensors, as other types of sensors are also contemplated for either of the sensors 606, 604, such as, for example, thermal, sonar, and / or laser.

[0066] Measurements of the first sensor 606 and the second sensor 604 may be processed by one or more of the processor 620, the sensor module 602, the depth estimation module 608, the communication module 622, the position module 618, the movement module 626, and the navigation module 624, in conjunction with the computer readable medium 614, to perform the functions described herein. In one configuration, data captured by the first sensor 606 and the second sensor 604 may be transmitted to an external device via the transceiver 616. The first sensor 606 and the second sensor 604 may be coupled to or in communication with a vehicle 628.

[0067] The location module 618 may be used to determine a location of the vehicle 628. For example, the location module 618 may use a Global Positioning System (GPS) to determine the location of the vehicle 628. The communication module 622 may be used to facilitate communication via the transceiver 616. For example, the communication module 622 may be configured to provide communication capabilities via different wireless protocols, such as WiFi, Long Term Evolution (LTE), 6G, etc. The communication module 622 may also be used to communicate with other components of the vehicle 628 that are not modules of the depth estimation system 600.

[0068] The locomotion module 626 may be used to facilitate movement of the vehicle 628. As an example, the locomotion module 626 may control the movement of wheels. As another example, the locomotion module 626 may be in communication with one or more power sources of the vehicle 628, such as a motor and / or a battery, etc. Of course, aspects of the present disclosure are not limited to providing locomotion via wheels, and contemplate other types of components for providing locomotion, such as propellers, treads, fins, and / or jet engines, etc.

[0069] The depth estimation system 600 also includes a navigation module 624 for planning a path for or controlling the movement of the vehicle 628 via a movement module 626. In one configuration, the navigation module 624 activates a defensive driving mode when the depth estimation module 608 identifies a dangerous agent. The navigation module 624 may override a user input when the user input is expected (e.g., predicted) to cause a collision. The modules may be software modules operating on the processor 620, software modules resident / stored on the computer readable medium 614, one or more hardware modules coupled to the processor 620, or any combination thereof.

[0070] The depth estimation module 608 may be in communication with the sensor module 602, the transceiver 616, the processor 620, the communication module 622, the position module 618, the movement module 626, the navigation module 624, and the computer-readable medium 614. In one configuration, the depth estimation module 608 receives sensor data from the sensor module 602. The sensor module 602 may receive sensor data from the first sensor 606 and the second sensor 604. According to aspects of the present disclosure, the sensor module 602 may filter the data to remove noise, encode the data, decode the data, fuse the data, extract frames, or perform other functions. In an alternative configuration, the depth estimation module 608 may receive sensor data directly from the first sensor 606 and the second sensor 604.

[0071] In one configuration, the depth estimation module 608 may communicate with and / or operate in conjunction with one or more of the sensor module 602, the transceiver 616, the processor 620, the communication module 622, the position module 618, the movement module 626, the navigation module 624, the first sensor 606, the second sensor 604, and the computer-readable medium 614. The depth estimation module 608 may be configured to receive a two-dimensional (2D) image of an environment adjacent to the ego-vehicle. The environment includes dynamic and static objects. The 2D image may be captured by the first sensor 606 or the second sensor 604.

[0072] The depth estimation module 608 may be configured to generate a depth map of the environment based on the 2D image. In one implementation, the accuracy of the depth estimation for dynamic objects in the depth map is higher than the accuracy of the depth estimation for static objects in the depth map. The depth estimation module 608 may work in conjunction with a view synthesis module (not shown in FIG. 6), such as the view synthesis module 508 described with respect to FIG. 5, to generate a three-dimensional (3D) estimation of the environment based on the depth map. In addition, the depth estimation module 608 may work in conjunction with a 3D object detection network (not shown in FIG. 6), such as the 3D object detection network 518 described with respect to FIG. 5, to identify the position of the dynamic object in the 3D estimation. Finally, working in conjunction with at least the position module 618, the movement module 626, and the navigation module 624, the depth estimation module 608 may control the behavior of the ego-vehicle based on the identified position.

[0073] The depth estimation module 608 may implement a depth network, such as the depth network 300 described with reference to FIG. 3, a pose network, such as the pose network 400 described with reference to FIG. 4, a view synthesis module, and / or a 3D object detection network.

[0074] 7 illustrates a flow diagram for a process 700 for identifying objects and controlling a vehicle based on depth estimation according to an aspect of the disclosure. Process 700 may be performed by one or more of a vehicle, such as vehicle 100 described with reference to FIG. 1, a depth network, such as depth network 300 described with reference to FIG. 3, a depth estimation module 608 as described with reference to FIG. 6, a pose network 400 as described with reference to FIG. 4, and / or a training pipeline 500 as described with reference to FIG. 5.

[0075] As shown in Fig. 7, process 700 includes capturing a two-dimensional (2D) image of an environment adjacent to the ego-vehicle. The 2D image may be a target image 200 of a scene 202 as described with reference to Fig. 2. The environment may include dynamic and static objects (block 702). The environment may include one or more dynamic objects, such as vehicles, pedestrians, and / or cyclists. The environment may also include one or more static objects, such as roads, sidewalks, and / or buildings. The 2D image may be captured via a monocular camera integrated into the ego-vehicle.

[0076] The process 700 may also include generating a depth map of the environment based on the 2D image via a depth estimation network (block 704). The depth map may be the depth map 220 of the scene 202 as described with reference to FIG. 2. In one implementation, the accuracy of the depth estimation for dynamic objects in the depth map is higher than the accuracy of the depth estimation for static objects in the depth map. The accuracy may be improved based on training, for example, as described with reference to FIG. 5.

[0077] For example, during training, the weight of each pixel corresponding to the position of the object in the 2D image may be adjusted. A depth estimation network (e.g., a depth network) may be trained based on the ground truth information and the adjusted weights. In addition, during training, the position of the dynamic object may be identified based on the annotated ground truth information. For example, the position may be identified based on a 3D bounding box that identifies the position of the object in the 3D estimation. During training, the 3D bounding box may be converted to a 2D bounding box to identify the position of the object in the 2D image.

[0078] Additionally, in this example, the weights may be adjusted based on a first number of pixels having depth information and a number of pixels corresponding to the location of the object in the 2D image. Alternatively, the weights may be adjusted based on a photometric loss and a supervised depth error loss. The photometric loss may be the photometric loss 520 as described with reference to FIG. 5.

[0079] The process 700 may also include generating a 3D estimate of the environment based on the depth map (block 706). The 3D estimate may be a reconstructed image (e.g., a warped source image), such as the reconstructed image 510 described with reference to FIG. 5 or the 3D reconstruction 240 of the scene 202 as described with reference to FIG. 2. The process 700 also includes identifying positions of dynamic objects in the 3D estimate (block 708). Finally, the process 700 includes controlling the behavior of the ego-vehicle based on the identified positions.

[0080] Based on the teachings, one skilled in the art should recognize that the scope of the present disclosure is intended to include any aspect of the present disclosure, regardless of whether it is implemented independently of any other aspect of the present disclosure or implemented in combination with any other aspect of the present disclosure. For example, an apparatus can be implemented using any number of the described aspects, or a method can be practiced using any number of the described aspects. In addition, the scope of the present disclosure is intended to include such an apparatus using other structures, functions, or such a method practiced using other structures, functions, or such an apparatus using structures and functions in addition to or other than the various aspects of the present disclosure described. It should be understood that any aspect of the present disclosure can be embodied by one or more elements of a claim.

[0081] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0082] Although specific aspects have been described herein, numerous variations or permutations of these aspects are within the scope of the present disclosure. Although some benefits and advantages of the preferred aspects have been mentioned, the scope of the present disclosure is not intended to be limited to the specific benefits, uses, or purposes. Instead, the aspects of the present disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are shown by way of example in the drawings and the following description of the preferred aspects. The detailed description and drawings are not limiting and are merely examples of the present disclosure, the scope of which is defined in the appended claims and their equivalents.

[0083] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" can include calculating, computing, processing, deriving, investigating, examining (e.g., examining a table, database, or other data structure), ascertaining, and the like. Additionally, "determining" can include receiving (e.g., receiving information), evaluating (e.g., evaluating data in a memory), and the like. Further, "determining" can include resolving, selecting, choosing, establishing, and the like.

[0084] As used herein, the phrase "at least one of" a list of items refers to any combination of those items, including the individual items. By way of example, "at least one of a, b, or c" is intended to include "a, b, c, a and b, a and c, b and c, and a, b and c."

[0085] Various example logic blocks, modules, and circuits described in connection with this disclosure may be implemented or executed in a processor that is specially configured to perform the functions discussed in this disclosure. The processor may be a neural network processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device (PDL), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Alternatively, the processing system may include one or more neuromorphic processors for implementing the neuron model and the model of the neural system described herein. The processor may be a microprocessor, a controller, a microcontroller, or a state machine that is specially configured as described herein. The processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or other such special configurations as described herein.

[0086] The steps of a method or algorithm described in connection with the present disclosure may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a storage device or machine-readable medium, including a random access memory (RAM), a read-only memory (ROM), a flash memory, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a CD-ROM, or other optical disk storage device, a magnetic disk storage device, or other magnetic storage device, or any other medium that can be used to execute or store desired program code in the form of instructions or data structures and that can be accessed by a computer. A software module may comprise a single instruction or many instructions, and may be distributed across several storage media over several different code segments in different programs. The storage medium may be coupled to the processor such that the processor can read information from and write information to the storage medium. In the alternative, the storage medium may be integral to the processor.

[0087] The methods disclosed herein comprise one or more steps or actions for achieving the described method. The steps and / or actions of the method may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0088] The described functions may be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, an example hardware configuration may comprise a processing system in an apparatus. The processing system may be implemented in a bus architecture. The bus may include any number of interconnected buses and bridges, depending on the particular application and overall design constraints of the processing system. The bus may link together various circuits including a processor, a machine-readable medium, and a bus interface. The bus interface may be used to connect a network adapter, among other things, to the processing system via the bus. The network adapter may be used to implement signal processing functions. For certain aspects, a user interface (e.g., a keypad, a display, a mouse, a joystick, etc.) may also be connected to the bus. The bus may also link together various other circuits such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and will not be described further.

[0089] The processor may be responsible for managing processes including the execution of software stored on the bus and on the machine-readable medium. Software shall be construed to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0090] In a hardware implementation, the machine-readable medium may be part of a processing system separate from the processor. However, as one skilled in the art will readily appreciate, the machine-readable medium, or any portion thereof, may be external to the processing system. By way of example, the machine-readable medium may include a transmission line, a carrier wave modulated with data, and / or a computer product separate from the device, all of which are accessible by the processor through a bus interface. Alternatively, or in addition, the machine-readable medium, or any portion thereof, may be integrated into the processor, such as in the case of a cache and / or specialized register file. While the various components discussed may be described as having a particular location, such as a local component, they may also be configured in various ways, such as where a component is configured as part of a distributed computing system.

[0091] The machine-readable medium may comprise a number of software modules. The software modules may include a transmitting module and a receiving module. Each software module may reside on a single storage device or may be distributed across multiple storage devices. As an example, a software module may be loaded into RAM from a hard drive upon a triggering event. During execution of the software module, the processor may load some of the instructions into a cache to speed up access. And one or more cache lines may be loaded into a special purpose register file for execution by the processor. When referring to the functionality of a software module below, it will be understood that such functionality is implemented by the processor when executing instructions from that software module. Furthermore, it should be appreciated that aspects of the present disclosure may result in improved functionality of a processor, computer, machine, or other system implementing such aspects.

[0092] If implemented in software, the functions may be stored on or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any storage medium that facilitates transfer of a computer program from one place to another.

[0093] Further, it should be appreciated that modules and / or other suitable means for performing the methods and techniques described herein may be downloaded and / or obtained by a user terminal and / or a base station, where applicable. For example, such a device may be coupled to a server to facilitate the transfer of means for performing the methods described herein. Alternatively, the various methods described herein may be provided via a storage means such that the user terminal and / or base station may obtain the various methods upon coupling or providing the storage means to the device. Moreover, any other suitable technique for providing the methods and techniques described herein to a device may be utilized.

[0094] It is to be understood that the claims are not limited to the precise configuration and components illustrated above. Various modifications, changes and variations may be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. 1. A method comprising: capturing a two-dimensional (2D) image of an environment adjacent to the ego-vehicle, the environment comprising at least dynamic and static objects; generating a depth map of the environment based on the 2D image via a depth estimation network, wherein a precision of depth estimation for the dynamic objects in the depth map is higher than a precision of depth estimation for the static objects in the depth map; generating a three-dimensional (3D) estimate of the environment based on the depth map; and identifying a position of the dynamic object in the 3D estimation; and and controlling a behavior of the ego vehicle based on the identified location.

2. The dynamic object may include a pedestrian, a nearby vehicle, or a cyclist; The method of claim 1 , wherein the static object includes a road, a sidewalk, or a building.

3. adjusting a weight for each pixel corresponding to the location of the object in the 2D image; and Training the depth estimation network based on ground truth information and the adjusted weights. The method of claim 1 , further comprising: training a neural network of the ego vehicle by:

4. The method of claim 3 , further comprising identifying the locations based on annotated ground truth information during training.

5. identifying a position based on a 3D bounding box that identifies a position of the object in a 3D estimate; The method of claim 3 , further comprising converting the 3D bounding box into a 2D bounding box to identify the location of the object in the 2D image.

6. adjusting the weights based on a first number of pixels comprising depth information and a number of pixels corresponding to the position of the object in the 2D image; or The method of claim 3 , further comprising adjusting the weights based on photometric loss and supervised depth error loss.

7. The method of claim 1 , further comprising capturing the 2D image via a monocular camera integrated into the ego-vehicle.

8. A processor; a memory coupled to the processor; and instructions stored in the memory, The instructions, when executed by the processor, cause the device to: capturing a two-dimensional (2D) image of an environment adjacent to the ego-vehicle, the environment comprising at least dynamic objects and static objects; generating a depth map of the environment based on the 2D images via a depth estimation network; generating a three-dimensional (3D) estimate of the environment based on the depth map; identifying a position of the dynamic object in the 3D estimate; and operable to control a behavior of the host vehicle based on the identified location; The apparatus, wherein the accuracy of depth estimation for the dynamic objects in the depth map is greater than the accuracy of depth estimation for the static objects in the depth map.

9. The dynamic object may include a pedestrian, a nearby vehicle, or a cyclist; The apparatus of claim 8 , wherein the static object comprises a road, a sidewalk, or a building.

10. The instructions further include: adjusting a weight for each pixel corresponding to the location of the object in the 2D image; and Train the depth estimation network based on ground truth information and the adjusted weights. The apparatus of claim 8 , further comprising:

11. The apparatus of claim 10 , wherein the instructions further cause the apparatus to train the depth estimation network of the ego-vehicle by identifying the location based on annotated ground truth information during training.

12. The instructions further include: identifying the location based on a 3D bounding box that identifies a location of the object in a 3D estimate; and Transforming the 3D bounding box into a 2D bounding box to identify the location of the object in the 2D image. The apparatus of claim 10 , further comprising:

13. The instructions further include: adjusting the weights based on a first number of pixels comprising depth information and a number of pixels corresponding to the position of the object in the 2D image; or Adjust the weights based on the photometric loss and the supervised depth error loss. The apparatus of claim 10 , further comprising:

14. The apparatus of claim 8 , wherein the instructions further cause the apparatus to capture the 2D image via a monocular camera integrated into the ego vehicle.

15. A non-transitory computer readable medium having program code recorded thereon, the program code being executed by a processor, - program code for capturing a two-dimensional (2D) image of an environment adjacent to the ego-vehicle, the environment comprising at least dynamic objects and static objects; program code for generating a depth map of the environment based on the 2D image via a depth estimation network, wherein a precision of depth estimation for the dynamic objects in the depth map is higher than a precision of depth estimation for the static objects in the depth map; and program code for generating a three-dimensional (3D) estimate of the environment based on the depth map; program code for identifying a position of the dynamic object in the 3D estimation; and program code for controlling a behavior of the ego-vehicle based on the identified location.

16. The dynamic object may include a pedestrian, a nearby vehicle, or a cyclist; The non-transitory computer-readable medium of claim 15 , wherein the static object includes a road, a sidewalk, or a building.

17. The program code comprises: adjusting a weight for each pixel corresponding to the location of the object in the 2D image; and Training the depth estimation network based on ground truth information and the adjusted weights.

16. The non-transitory computer readable medium of claim 15, further comprising program code for training the depth estimation network of the ego vehicle by:

18. 20. The non-transitory computer readable medium of claim 17, further comprising program code for training the depth estimation network of the ego vehicle by identifying the location based on annotated ground truth information during training.

19. The program code comprises: identifying the location based on a 3D bounding box that identifies a location of the object in a 3D estimate; and Transforming the 3D bounding box into a 2D bounding box to identify the location of the object in the 2D image.

20. The non-transitory computer readable medium of claim 17, further comprising program code for training the depth estimation network of the ego vehicle by:

20. The program code comprises: adjusting the weights based on a first number of pixels comprising depth information and a number of pixels corresponding to the position of the object in the 2D image; or Adjust the weights based on the photometric loss and the supervised depth error loss.

20. The non-transitory computer readable medium of claim 17, further comprising program code for training the depth estimation network of the ego vehicle by:

Images