Network Architecture for Monocular Depth Estimation and Object Detection

Abstract

To provide systems and methods for performing depth estimation and object detection using a common network architecture.SOLUTION: In one embodiment, a method includes generating, by using a backbone of a combined network, a feature map at multiple scales from an input image. The method includes decoding, by using a top-down pathway of the combined network, the feature map to provide features at the multiple scales. The method includes generating, by using a head of the combined network, a depth map from the features for a scene depicted in the input image, and bounding boxes identifying objects in the input image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The subject matter described herein relates generally to systems and methods for depth prediction and object detection, and more particularly to a unique network architecture for depth prediction and object detection. [Background technology]

[0002] Various devices that operate autonomously or provide information about their surrounding environment often use sensors to facilitate the perception of obstacles and additional aspects of the surrounding environment. The ability to reconstruct the structure of a scene with high accuracy is important for understanding and navigating that scene. As an example, a robotic device uses information from sensors to develop awareness of its surrounding environment to navigate through the environment and avoid hazards. In particular, the robotic device uses the perceived information to determine the 3D structure of the environment so that the device can distinguish between navigable areas and potential hazards. In various embodiments, this may include perceiving distance and detecting discrete objects.

[0003] In one approach, a robotic device can utilize a monocular camera to capture images of the surrounding environment from which distance and other aspects can be estimated. While this approach can avoid the use of expensive light detection and ranging (LiDAR) sensors, the resulting estimates do not always provide the level of accuracy that may be necessary to navigate the scene. Moreover, this approach may also require a large number of different machine learning models, and training these models can represent a difficult task due to the difficulty of obtaining training data. Summary of the Invention

[0004] In one embodiment, the exemplary system and method relates to an improved approach to depth estimation and object detection from monocular images that leverages combinatorial networks. As noted above, implementing multiple separate networks to perform depth estimation and object detection is resource intensive. Moreover, the process of independently training these networks can also be particularly challenging for object detection networks, which may use labeled ground truth data that is often difficult to obtain.

[0005] Thus, in one arrangement, a combined network is disclosed that integrates both functions within a single network while leveraging the simplicity of training a depth estimation network to improve the training of the network for object detection. For example, the combined network is formed from a common backbone that accepts monocular images as input and generates feature maps at different scales to encode features represented in the images. The feature maps are fed into a top-down pathway that, in one arrangement, functions to decode the feature maps within the feature maps, which may also involve lateral connections with the backbone to further enhance feature generation. Thus, a shared backbone is then used across distinct scales to generate bounding boxes for object detection and, at the highest resolution, to generate depth maps for the input images as well.

[0006] In either case, the backbone and top-down pathways generally form a feature pyramid network shared between the different tasks of depth estimation and object detection. Thus, instead of training the combined network in separate instances for object detection and depth estimation, initial training on depth estimation can be performed to train the backbone and top-down pathways. The combined network is then trained for object detection using training data that may be relatively less scalable. Thus, pre-training on depth estimation initializes the combined network in a way that makes training for object detection a less intensive task than would otherwise be necessary and uses fewer labeled training images, thereby reducing the difficulty of obtaining such data for training. In this way, the combined network improves overall performance for both tasks using a simplified architecture that is easier to train.

[0007] In one embodiment, a depth system is disclosed. The depth system includes one or more processors and a memory communicatively coupled to the one or more processors. The memory stores a network module including instructions that, when executed by the one or more processors, cause the one or more processors to generate feature maps at multiple scales from an input image using a backbone of a combinatorial network. The network module includes instructions to decode the feature maps using a top-down path of the combinatorial network to provide features at multiple scales. The network module includes instructions to generate, using a head of the combinatorial network, a depth map from features for a scene depicted in the input image and a bounding box that identifies an object in the input image.

[0008] In one embodiment, a non-transitory computer-readable medium is disclosed that includes instructions that, when executed by one or more processors, cause the one or more processors to perform various functions. The instructions include instructions for generating feature maps at multiple scales from an input image using a backbone of a combinatorial network. The instructions include instructions for decoding the feature maps to provide features at multiple scales using a top-down path of the combinatorial network. The instructions include instructions for generating a depth map from features for a scene depicted in the input image and bounding boxes that identify objects in the input image using a head of the combinatorial network.

[0009] In one embodiment, a method is disclosed that includes generating feature maps at multiple scales from an input image using a backbone of a combinatorial network, decoding the feature maps using a top-down path of the combinatorial network to provide features at multiple scales, and generating a depth map from the features for a scene depicted in the input image and bounding boxes that identify objects in the input image using a head of the combinatorial network.

[0010] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate various systems, methods, and other embodiments of the present disclosure. It is recognized that the illustrated element boundaries (e.g., boxes, boxes, or other shapes) in the figures represent one embodiment of the boundaries. In some embodiments, one element may be designed as multiple elements, or multiple elements may be designed as one element. In some embodiments, an element shown as an internal component of another element may be implemented as an external component, and vice versa. Additionally, elements may not be drawn to scale. [Brief explanation of the drawings]

[0011] [Figure 1]FIG. 1 illustrates one embodiment of a vehicle in which the systems and methods disclosed herein may be implemented. [Figure 2] FIG. 1 illustrates one embodiment of a depth system coupled to providing depth estimates and bounding boxes for object detection using a combinatorial network. [Figure 3] FIG. 1 illustrates one arrangement of an architecture for a combined network. [Figure 4] FIG. 1 illustrates a flow chart associated with one embodiment of generating a depth map and detecting objects in an image. [Figure 5] FIG. 1 illustrates a flow diagram associated with one embodiment of training a combinatorial network. DETAILED DESCRIPTION OF THE INVENTION

[0012] Systems, methods, and other embodiments are disclosed that are coupled with an improved approach to depth estimation and object detection from monocular images that leverages combinatorial networks. As noted above, implementing multiple separate networks to perform depth estimation and object detection is resource intensive. Moreover, the process of independently training these networks can also be particularly challenging for object detection networks that may use labeled ground truth data, which is often difficult to obtain.

[0013] Thus, in one arrangement, a combined network is disclosed that integrates both functions within a single network while leveraging the simplicity of training a depth estimation network to improve the training of the network for object detection. For example, the combined network is formed from a common backbone that accepts monocular images as input and generates feature maps at different scales to encode features represented in the images. In one arrangement, the feature maps are fed into a top-down pathway that functions to decode the feature maps within the features, which may also involve lateral connections with the backbone to further enhance feature generation. Thus, a shared backbone is then used between the separate scales to generate bounding boxes for object detection and, at the highest resolution, to generate depth maps for the input images as well.

[0014] In either case, the backbone and top-down pathways generally form a feature pyramid network shared between the different tasks of depth estimation and object detection. Thus, instead of training the combined network in separate instances for object detection and depth estimation, initial training on depth estimation can be performed to train the backbone and top-down pathways. The combined network is then trained for object detection using training data that may be relatively less scalable. Thus, pre-training on depth estimation initializes the combined network in a way that makes training for object detection a less intensive task than would otherwise be necessary and uses fewer labeled training images, thereby reducing the difficulty of obtaining such data for training. In this way, the combined network improves overall performance for both tasks using a simplified architecture that is easier to train.

[0015] Referring to FIG. 1 , an example of a vehicle 100 is shown. As used herein, a “vehicle” refers to any form of motorized transportation. In one or more implementations, the vehicle 100 is an automobile. While arrangements are described herein with reference to automobiles, it is understood that embodiments are not limited to automobiles. In some implementations, the vehicle 100 may be any robotic or other device (e.g., a surveillance device) that, for example, observes its surroundings and provides decisions therefrom, thereby benefiting from the functionality discussed herein. In further embodiments, instead of a motorized device, the vehicle 100 may be a statically mounted device, an embedded device, or another device that uses monocular images to derive depth and object information about a scene.

[0016] In either case, vehicle 100 similarly includes various elements. In various embodiments, vehicle 100 may not need to include all of the elements shown in FIG. 1 . Vehicle 100 may include any combination of the various elements shown in FIG. 1 . Furthermore, vehicle 100 may include additional elements to those shown in FIG. 1 . In some arrangements, vehicle 100 may be implemented without one or more of the elements shown in FIG. 1 . While various elements are illustrated as being located within vehicle 100, it is understood that one or more of these elements may also be located external to vehicle 100. Furthermore, the illustrated elements may be physically separated by a large distance and provided as a remote service (e.g., cloud computing services, software as a service (SaaS), etc.).

[0017] Some of the possible elements of vehicle 100 are shown in FIG. 1 and described in conjunction with subsequent figures. However, a description of many of the elements in FIG. 1 is provided following the discussion of FIGS. 2-5 for purposes of brevity of this description. Furthermore, for ease and clarity of illustration, reference numerals have been repeated, where necessary, among the different figures to indicate corresponding or similar elements. Furthermore, the discussion has outlined many specific details to provide a thorough understanding of the embodiments described herein. However, one of ordinary skill in the art will appreciate that the embodiments described herein can be practiced using various combinations of these elements.

[0018] In either case, vehicle 100 includes depth system 170 that functions to generate depth estimates (i.e., depth maps) using a novel network architecture that can utilize multiple information sources. Moreover, while depicted as a standalone component, in one or more embodiments, depth system 170 is integrated with autonomous driving module 160, camera 126, or another component of vehicle 100. Furthermore, as noted above, one or more components of depth system 170 may be cloud-based elements remote from vehicle 100. The noted functions and methods will become more apparent with further discussion of the figures.

[0019] Referring to FIG. 2 , one embodiment of the depth system 170 is further illustrated. The depth system 170 is shown as including the processor 110. Thus, the processor 110 may be part of the depth system 170, or the depth system 170 may access the processor 110 through a data bus or another communication path. In one or more embodiments, the processor 110 is an application-specific integrated circuit (ASIC) configured to implement the functionality associated with the network module 220. Generally, the processor 110 is an electronic processor, such as a microprocessor, capable of performing the various functions described herein. In one embodiment, the depth system 170 includes a memory 210 that stores the network module 220 and / or other modules that may function to assist in generating depth information. The memory 210 may be a random access memory (RAM), a read-only memory (ROM), a hard disk drive, flash memory, or other suitable memory for storing the network module 220. Network module 220 is, for example, computer-readable instructions that, when executed by processor 110, cause processor 110 to perform various functions disclosed herein. In a further arrangement, network module 220 is logic, an integrated circuit, or another device for performing the indicated functions that includes instructions integrated therein.

[0020] Additionally, in one embodiment, depth system 170 includes data store 230. Data store 230, in one embodiment, is an electronic data structure stored in memory 210 or another data store, along with routines that can be executed by processor 110 to analyze the stored data, render the stored data, organize the stored data, etc. Thus, in one embodiment, data store 230 stores data used by network module 220 in performing various functions. For example, as depicted in FIG. 2, data store 230 includes images 240, combinatorial network model 250, depth map 260, and bounding boxes 270, along with other information used and / or generated by network module 220, for example.

[0021] Image 240 is, for example, a monocular image from camera 126 or another imaging device. A monocular image is generally derived from one or more monocular videos comprised of multiple frames, although image 240 could equally well be derived from a still image camera. As described herein, a monocular image is, for example, a color-coded (RGB) image without any additional modalities and is an image from camera 126 or another imaging device that encompasses a field of view (FOV) centered on vehicle 100 of at least a portion of the surrounding environment. That is, in one approach, monocular image 240 is limited to a subregion of the surrounding environment. Thus, image 240 can be a 60-, 90-, or 120-degree FOV in the forward-looking (i.e., driving direction), a rearward / sideward-looking FOV, or other subregion defined by the imaging characteristics of camera 126 (e.g., lens distortion, FOV, etc.). In various embodiments, camera 126 is a pinhole camera, a fisheye camera, a catadioptric camera, or another form of camera that captures images without a specific depth modality.

[0022] An individual monocular image itself includes visual data for the FOV encoded according to an imaging standard (e.g., a codec) associated with camera 126 or another source imaging device. Generally, the characteristics of the source camera (e.g., camera 126) and video standard define the format of the monocular image. Thus, while specific characteristics may vary depending on different implementations, the image generally has a defined resolution (i.e., height and width in pixels) and format. Thus, for example, the monocular image is an RGB visible light image. In further aspects, the monocular image may be an infrared image associated with a corresponding infrared camera, a black and white image, or another suitable format as may be desired. Regardless of which format depth system 170 implements, the image is a monocular image in that there is no explicit additional modality that indicates depth, nor any explicit corresponding image from another camera from which depth is derived (i.e., any stereo camera pair). In contrast to stereo images, which may combine left and right images from separate cameras mounted side-by-side to provide an additional depth channel, monocular images do not contain explicit depth information, such as disparity maps derived from pixel-by-pixel comparison of stereo images. Instead, monocular images provide depth information implicitly in the size and projection relationships of the elements depicted within, from which model 250 derives depth map 260.

[0023] Additionally, in one or more arrangements, the data store 230 further includes training data corresponding to the images 240. While the images 240 and training data are generally discussed in the singular, in various arrangements the data store 230 includes multiple images and corresponding training data to support training of the model 250. Moreover, during actual use, the system 170 may acquire and process the images 240 individually. In either case, the training data associated with the images 240, in one embodiment, includes ground truth depth data, ground truth bounding boxes, class labels, and the like. Of course, in further implementations, one or more aspects of the training data may be omitted, such as when the depth system 170 implements a self-supervised approach to training the model 250 for depth. In the case of self-supervised training for depth, the training data may not include ground truth depth data, but instead include a sequence of images from a video of the same scene to support decisions through the use of structure-from-motion (SfM).

[0024] Model 250 is a combinatorial network that supports inference about depth, object detection, and classification. In one configuration, model 250 is a single-stage fully convolutional network. Furthermore, model 250 may be referred to as a feature pyramid network (FPN), which includes a bottom-up backbone for encoding feature maps from image 240, a top-down path for decoding the feature maps, and a shared head. The top-down path, also referred to as a feature pyramid in some implementations, includes serial connections between distinct layers as well as lateral connections at different scales with the backbone. Thus, the top-down path outputs results at different scales (e.g., bounding boxes, centralities, etc.) except for a depth map 260, which is output at the highest resolution.

[0025] For further illustration of the architecture for model 250, consider FIG. 3. As shown in FIG. 3, model 250 includes a backbone 300, which includes a series of convolutional layers that iteratively reduce the scale of the input image 240. Backbone 300, which may also be referred to as a bottom-up pathway, is a feedforward computation of model 250 that computes a feature hierarchy, including feature maps at multiple different scales. In further embodiments, backbone 300 may include additional components, such as pooling layers. In one approach, top-down pathway 310 uses a series of deconvolutional layers to upsample spatially coarser feature maps to hallucinate higher-resolution features. Top-down pathway 310 enhances features via lateral connections to backbone 300 to provide less sampled instances of original input 240 for improved upsampling, as shown.

[0026] Lateral connections merge feature maps of the same spatial size from the backbone 300 and the top-down path 310, which provides more accurate localization because the feature maps from the backbone 300 are subsampled less times than the feature maps from the previous layers of the top-down path 310. The depth system 170 may use element-like addition to combine the separate maps. The head 320 is shared between the separate output scales of the top-down path 310, but is shown as multiple different heads to emphasize the reasoning at the separate scales. That is, the head 320 is a single structure shared between the separate outputs of the top-down path 310.

[0027] As illustrated, head 320 includes a number of different structures, such as a class component 330, a 2D component 340, and a 3D component 350. Class component 330 generates a C-dimensional vector corresponding to a certain number of classes for which model 250 is trained. Class component 330 may include a multi-class classifier or multiple binary classifiers for generating class decisions for objects.

[0028] The 2D component 340 functions to identify objects in the 2D space of a frame of image 240. In one approach, the 2D component 340 is a convolutional network that generates 2D bounding boxes as identifications from the frame of image 240. The 2D component 340 can function to provide consistency between 2D and 3D object detection.

[0029] Head 320 further includes a depth component 360 that acts as a pseudo-LiDAR 3D detector to predict dense per-pixel depth that is converted into a 3D point cloud using camera intrinsics. 3D component 350 concatenates the point cloud with input image 240 to generate a 6-dimensional tensor containing pixel color values along with 3D coordinates as depth map 260. 3D component 350 also accepts proposed regions from 2D component 340, which are processed by a 3D detection network 370 that generates bounding boxes 270.

[0030] 2, network module 220 generally implements model 250 as a separate component. In various aspects, network module 220 includes instructions for applying model 250, and model 250 may be integrated with network module 220. Generally, network module 220 controls model 250 to process sensor data including images 240, and, when training, uses the associated training data to generate a training loss and update model 250.

[0031] In either case, the depth system 170, in one embodiment, utilizes the model 250 to generate a depth map 260, a bounding box 270, and additional information (e.g., 2D bounding boxes, classification, etc.). Generally, the depth map 260 is a pixel-by-pixel prediction of depth for the image 240, which may be referred to as pseudo-LiDAR depth data or dense depth data. That is, the model 250 provides depth estimates for different aspects depicted in the image 240. Of course, in this approach, the model 250 also integrates information from beyond the depth map 260 to provide object detection in the form of 2D and 3D bounding boxes 270 in addition to other information, such as classification from a single model 250.

[0032] It should be appreciated that, in one embodiment, network module 220 generally includes instructions that function to control processor 110 to perform various actions to control model 250 to generate depth map 260. Network module 220 acquires images 240 in one or more approaches, for example, by controlling camera 126, or acquires images 240 from a data bus, electronic memory, or another available electronic source. Of course, in the instance of training model 250, network module 220 may further acquire training data (e.g., ground truth depths, object bounding boxes, class information, etc.) from electronic memory or another suitable data store.

[0033] The network module 220 initially applies the backbone of the model 250 to the input image 240 to generate feature maps at multiple different scales (i.e., feature layers). Thus, the network module 220 generates multiple feature maps, each with a different scale, that abstract features at different scales (e.g., large features vs. smaller features, such as dogs vs. trucks). The model 250 then feeds the feature maps to the top-down path 310 using two different connections. The first connection is a serial connection from the final layer of the backbone 300, while the second connection is a lateral connection, sometimes referred to as a skip connection, that provides feature maps at different scales. In this way, the network module 220 promotes multiple feature levels into the top-down path 310 to facilitate decoding.

[0034] The network module 220 performs decoding using the top-down path 310 to provide features or more detailed generated feature maps to the shared head 330 of the model 250. The shared head 320 generates a depth map 260 from features for the scene depicted in the input image and a bounding box 270 that identifies objects in the input image 240. As illustrated above, the shared head 320 includes multiple different paths associated with different functions. For example, the head 320 includes multiple sub-heads for 3D object detection, 2D object detection, depth estimation, and classification. The network module 220 controls separate heads 320 to operate on features from the top-down path 310. Thus, the network module 220 can control the heads 320 to iteratively process different depth scales provided as output from the top-down path 310, while further controlling the depth component 360 to operate on the highest-resolution feature map without generating depth for other scales.

[0035] In general, the framework of model 250 shares many aspects among a number of different components, such as backbone 300 and top-down path 310. Certain sections of head 320 all use backbone 300 and top-down path 310, while the components of head 320 include distinct and individual components to facilitate the generation of different outputs. By sharing significant portions of model 250, depth system 170 can train model 250 for relatively less resource-intensive tasks, such as depth estimation, with the goal of initializing model 250 and achieving convergence within model 250 using less ground truth data tied to the specific, distinct task.

[0036] Thus, in one aspect, the network module 220 implements a two-stage training process. In the first stage, the network module 220 trains the model 250 using a loss derived from depth estimates (i.e., depth map 260) over many iterations. The first stage of training can use ground truth depth in a supervised training pipeline, can rely on intrinsic ground truth through a self-supervised monocular approach that uses a series of images from a video taken on a moving platform to facilitate structure-from-motion (SfM) analysis, or can rely on a combined training pipeline (e.g., a semi-supervised one that leverages sparse depth data in combination with the intrinsic depth information of a self-supervised approach).

[0037] In either case, by using the depth component 360 as a source of training for the first stage, the depth system 170 avoids the problems associated with using excessive amounts of ground truth training data for object detection and classification. Because ground truth data for additional tasks is generally more difficult to obtain than depth training data due to the cost and effort associated with generating the data, training in this manner significantly improves the ability to accurately train the model 250 for many different tasks. Thus, the network module 220 can randomly initialize the additional subheads 330 and 340 during the first stage and derive a depth loss according to the training data for the depth head 360. The network module 220 can then adapt the backbone 300 and top-down path 310 to train the model 250 using the depth loss from the first stage.

[0038] The network module 220 can then use a second training stage to refine the 3D detection of additional sub-heads 330, 340 and head 350 for the additional tasks. Thus, in one embodiment, the network module 220 further refines the training of the model 250 using ground truth data for specific tasks in the second stage. However, it should be noted that the model 250 has already been initialized and is therefore expected to converge to accurate estimates for the different tasks using significantly less training data for the different tasks than would otherwise be used. Thus, by sharing large portions of the model 250 between the object detection, classification, and depth tasks, the performance of the model 250 for the object detection and classification tasks is improved by simplifying the training for the depth estimation task.

[0039] Additional aspects of joint learning of depth prediction, object detection, and object classification are discussed in connection with Figure 4. Figure 4 illustrates a flow diagram of a method 400 coupled to generating depth maps, bounding boxes, and classifications using a combinatorial network. Method 400 is discussed in terms of depth system 170 of Figures 1-2. Although method 400 is discussed in conjunction with depth system 170, it should be appreciated that method 400 is not limited to being implemented within depth system 170, but rather is an example of a system in which method 400 may be implemented.

[0040] At 410, the network module 220 acquires an image 240. Generally, the image 240 comprises at least a monocular image from a monocular camera or video of the surrounding environment of the vehicle 100 or another device for which the depth system 170 is to analyze information. Thus, as noted above, the depth system 170 may be integrated with the vehicle 100 or another device, or alternatively, may receive the image 240 via a communications path, such as a wide area network (WAN), and function remotely to analyze the image 240. In either case, the depth system 170 may process a single image 240 to derive multiple different decisions using the same model 250.

[0041] At 420, the network module 220 generates feature maps at multiple scales from the input image using the backbone 300 of the model 250. Thus, in one approach, the network module 220 uses the model 250 to generate feature maps at five different scales. The model 250 uses convolutional layers to process the input image 240 into feature maps that generally indicate the presence of features at different scales. This feature hierarchy provides a representation of the aspects depicted in the image 240 in an encoded form that the model 250 can process to identify objects, depth, etc. The feature maps thus encode the features of the input image 240 and provide a common basis for generating the depth map 260 and bounding boxes 270.

[0042] At 430, the network module 220 uses lateral connections between the backbone 300 and the top-down path 310 to provide multiple scales of feature maps in addition to the output of the previous level from within the top-down path 310. Thus, while one portion of the model 250 functions to iteratively process the image 240 into feature maps and then process the feature maps according to successive iterations at successive scales, the network module 220 also uses lateral connections to provide intermediate feature maps from the backbone 300 to the top-down path 310. The feature maps provided via the lateral connections have not been subsampled as many times, thus facilitating upsampling at higher resolutions.

[0043] At 440, the network module 220 decodes the feature maps to provide features at multiple scales using the top-down path of the model 250. As noted above, the top-down path 310 receives the processed feature maps from the previous layer while also receiving residual feature maps from the backbone via lateral connections. In either case, decoding generally involves applying a deconvolution layer to upsample the input.

[0044] At 450, the network module 220 uses the shared head of the model 250 to generate a depth map 260 from features for the scene depicted in the input image 240 and a bounding box 270 that identifies objects in the input image 240. In a further embodiment, the model 250 also generates a 2D bounding box and classification for the identified object. The network module 220 uses the head between separate layers of a multi-scale, top-down path to generate the bounding box 270 at multiple scales and the depth map 260 at one of multiple scales. The head 320 includes multiple sub-heads that perform 3D object detection, 2D object detection, depth estimation, and classification. Thus, the network module 220 can use the model 250 to output multiple different decisions using a single model 250.

[0045] At 460, network module 220 provides depth map 260 and bounding box 270. In one approach, network module 220 provides depth map 260 to an assistance system of vehicle 100, such as autonomous driving module 160, for navigating (e.g., planning and controlling) vehicle 100 according to depth map 260 and bounding box 270. Thus, network module 220 can use model 250 to assist vehicle 100 in perceiving its surrounding environment to facilitate path planning and other functions to enable vehicle 100 to safely navigate the environment.

[0046] 5 illustrates a flow diagram of a method 500 associated with training a combinatorial network architecture. Method 500 is discussed in terms of depth system 170 of FIGS. 1-2. While method 500 is discussed in conjunction with depth system 170, it should be appreciated that method 500 is not limited to being implemented within depth system 170, but rather is an example of a system in which method 500 may be implemented.

[0047] At 510, the network module 220 obtains training data for training the model 250. In one approach, the training data includes data for two stages of training. Thus, the training data includes information for training the depth head of the model 250. This training data can include a series of consecutive frames from a video of the scene to support self-supervised training. In a further approach, the self-supervised training data can be supplemented with sparse ground truth depth data to support learning at metrically accurate scales while still minimizing the need for ground truth data. In a further approach, depth training can rely on explicit ground truth depth data derived from LiDAR, stereo image pairs, etc.

[0048] Moreover, the training data for the second stage generally includes labeled ground truth data to assist in the additional tasks of object detection and classification. The object classification ground truth data can include both 2D and 3D bounding boxes, while the classification data includes class labels for the number of classes for which model 250 is trained. As outlined above, the amount of training data for the additional tasks is significantly reduced through the process of first training model 250 for depth estimation, as opposed to separately training for all tasks beforehand.

[0049] At 520, the network module 220 trains the model 250 in a first stage on the calculated loss value associated with the depth estimation. As noted, depth training can be supervised or unsupervised depending on the available training data. In either case, the network module 220 trains the model 250 for the depth estimation task across a large number of monocular images in the training data. During this first stage, the network module 220 can randomly initialize the other heads (i.e., classification and 2D bounding box estimation) while deriving the depth loss generally according to the depth map 260 and the associated training data.

[0050] At 530, the network module 220 trains the model 250 in a second stage by using the bounding box and ground truth data to calculate a detection loss. In a further embodiment, the network module 220 also trains the model 250 for classification in addition to 2D and 3D bounding boxes. In either case, the network module 220 trains for all other tasks after the initial depth estimation training with the aim of leveraging depth estimation as a pre-training stage to initialize the shared components of the model 250, thereby accelerating training for other tasks on less robust datasets.

[0051] At 540, the network module 220 provides the model 250. After the network module 220 trains the model 250, the model 250 represents a single network architecture that can provide decisions for many different tasks. As a result, the model 250 simplifies many separate tasks into a single architecture while also simplifying training. Thus, the model 250 becomes more efficient through the sharing of common elements (e.g., the backbone 300 and the top-down pathway 310). In this way, the depth system 170 improves depth, object detection, and classification decisions.

[0052] FIG. 1 is now described in full detail as an exemplary environment in which the systems and methods disclosed herein may operate. In some instances, vehicle 100 is configured to selectively switch between an autonomous mode, one or more semi-autonomous operating modes, and / or a manual mode. Such switching may be implemented in any suitable manner now known or later developed. "Manual mode" means that all or most of the navigation and / or operation of the vehicle is performed according to input received from a user (e.g., a human driver). In one or more arrangements, vehicle 100 may be a conventional vehicle configured to operate only in manual mode.

[0053] In one or more embodiments, vehicle 100 is an autonomous vehicle. As used herein, "autonomous vehicle" refers to a vehicle operating in an autonomous mode. "Autonomous mode" refers to using one or more computing systems to control vehicle 100 to navigate and / or operate vehicle 100 along a travel route with minimal or no input from a human driver. In one or more embodiments, vehicle 100 is highly automated or fully automated. In one embodiment, vehicle 100 is configured with one or more semi-autonomous modes of operation in which one or more computing systems perform a portion of the navigation and / or operation of the vehicle along a travel route, and an operator (i.e., driver) of the vehicle provides input to the vehicle to perform a portion of the navigation and / or operation of vehicle 100 along a travel route.

[0054] Vehicle 100 may include one or more processors 110. In one or more arrangements, processor 110 may be the main processor of vehicle 100. For example, processor 110 may be an electronic control unit (ECU). Vehicle 100 may include one or more data stores 115 for storing one or more types of data. Data stores 115 may include volatile and / or non-volatile memory. Examples of suitable data stores 115 include RAM (random access memory), flash memory, ROM (read-only memory), PROM (programmable read-only memory), EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), registers, magnetic disks, optical disks, hard drives, or any other suitable storage medium, or any combination thereof. Data store 115 may be a component of processor 110, or data store 115 may be operatively connected to and used by processor 110. As used throughout this specification, the term "operably connected" can include direct or indirect connections, including connections without direct physical contact.

[0055] In one or more arrangements, one or more data stores 115 may include map data 116. The map data 116 may include maps of one or more geographic regions. In some cases, the map data 116 may include information or data about roads, traffic control devices, road markings, structures, features, and / or landmarks within one or more geographic regions. The map data 116 may be in any suitable form. In some cases, the map data 116 may include an aerial photograph of an area. In some cases, the map data 116 may be a terrestrial photograph of an area, including a 360-degree terrestrial photograph. The map data 116 may include measurements, dimensions, distances, and / or information about one or more items included within the map data 116 and / or about other items included within the map data 116. The map data 116 may include a digital map with information about road geometry. The map data 116 may be of high quality and / or high definition.

[0056] In one or more arrangements, the map data 116 may include one or more topographical maps 117. The topographical maps 117 may include information about the ground, terrain, roads, surfaces, and / or other features of one or more geographical regions. The topographical maps 117 may include elevation data within one or more geographical regions. The map data 116 may be of high quality and / or high detail. The topographical maps 117 may define one or more terrain surfaces, which may include paved roads, unpaved roads, land masses, and other features that define the terrain.

[0057] In one or more arrangements, the map data 116 may include one or more stationary obstacle maps 118. The stationary obstacle map 118 may include information about one or more stationary obstacles located within one or more geographic regions. A “stationary obstacle” is a physical object whose position does not change or substantially change over a period of time and / or whose size does not change or substantially change over a period of time. Examples of stationary obstacles include trees, buildings, curbs, fences, guardrails, medians, utility poles, statues, monuments, signs, benches, furniture, mailboxes, large rocks, and hills. A stationary obstacle may be an object that extends above the ground surface. One or more stationary obstacles included in the stationary obstacle map 118 may have location data, size data, dimension data, material data, and / or other data associated therewith. The stationary obstacle map 118 may include measurements, dimensions, distances, and / or information about one or more stationary obstacles. The stationary obstacle map 118 may be of high quality and / or high definition. The static obstacle map 118 may be updated to reflect changes in the mapped area.

[0058] One or more data stores 115 may include sensor data 119. In this context, "sensor data" means any information about sensors equipped on vehicle 100, including capabilities and other information about such sensors. As described below, vehicle 100 may include sensor system 120. Sensor data 119 may relate to one or more sensors of sensor system 120. As an example, in one or more installations, sensor data 119 may include information about one or more LIDAR sensors 124 of sensor system 120.

[0059] In some cases, at least a portion of the map data 116 and / or sensor data 119 may be located in one or more data stores 115 onboard the vehicle 100. Alternatively, or additionally, at least a portion of the map data 116 and / or sensor data 119 may be located in one or more data stores 115 located remotely from the vehicle 100.

[0060] As noted above, vehicle 100 may include sensor system 120. Sensor system 120 may include one or more sensors. A "sensor" refers to any device, component, and / or system that can detect and / or sense something. One or more sensors may be configured to detect and / or sense in real time. As used herein, the term "real time" refers to a level of processing responsiveness that a user or system perceives as being immediate enough to make a particular process or decision, or that allows a processor to keep up with some external process.

[0061] In an arrangement in which sensor system 120 includes multiple sensors, the sensors may operate independently of one another. Alternatively, two or more of the sensors may operate in combination with one another. In such cases, the two or more sensors may form a sensor network. Sensor system 120 and / or one or more sensors may be operatively connected to processor 110, data store 115, and / or other elements of vehicle 100 (including any of the elements shown in FIG. 1). Sensor system 120 may acquire data of at least a portion of the environment external to vehicle 100 (e.g., nearby vehicles).

[0062] The sensor system 120 may include any suitable type of sensor. Various examples of different types of sensors are described herein. However, it is understood that embodiments are not limited to the particular sensors described. The sensor system 120 may include one or more vehicle sensors 121. The vehicle sensors 121 may detect, determine, and / or sense information about the vehicle 100 itself. In one or more arrangements, the vehicle sensors 121 may be configured to detect and / or sense changes in the position and orientation of the vehicle 100, for example, based on inertial acceleration. In one or more arrangements, the vehicle sensors 121 may include one or more accelerometers, one or more gyroscopes, an inertial measurement unit (IMU), a dead reckoning system, a global navigation satellite system (GNSS), a global positioning system (GPS), a navigation system 147, and / or other suitable sensors. The vehicle sensors 121 may be configured to detect and / or sense one or more characteristics of the vehicle 100. In one or more arrangements, the vehicle sensors 121 may include a speedometer for determining the current speed of the vehicle 100 .

[0063] Alternatively or additionally, sensor system 120 may include one or more environmental sensors 122 configured to acquire and / or detect driving environment data. "Driving environment data" includes data or information about the external environment, or one or more portions thereof, in which the autonomous vehicle is located. For example, one or more environmental sensors 122 may be configured to detect, quantify, and / or sense obstacles within at least a portion of vehicle 100's external environment, and / or information / data about such obstacles. Such obstacles may be stationary objects and / or dynamic objects. One or more environmental sensors 122 may be configured to detect, measure, quantify, and / or sense other objects within vehicle 100's external environment, such as lane markers, signs, traffic lights, traffic signs, lane lines, crosswalks, curbs proximate to vehicle 100, off-road objects, etc.

[0064] Described herein are various examples of sensors for sensor system 120. Example sensors may be part of one or more environmental sensors 122 and / or one or more vehicle sensors 121. However, it is understood that embodiments are not limited to the particular sensors described.

[0065] By way of example, in one or more arrangements, sensor system 120 may include one or more radar sensors 123, one or more LIDAR sensors 124, one or more sonar sensors 125, and / or one or more cameras 126. In one or more arrangements, one or more cameras 126 may be high dynamic range (HDR) cameras or infrared (IR) cameras.

[0066] Vehicle 100 may include input system 130. An "input system" includes any device, component, system, element, or arrangement, or group thereof, that allows information / data to be input into a machine. Input system 130 may receive input from a vehicle occupant (e.g., a driver or passenger). Vehicle 100 may include output system 135. An "output system" includes any device, component, or arrangement, or group thereof, that allows information / data to be presented to a vehicle occupant (e.g., a person, vehicle occupant).

[0067] Vehicle 100 may include one or more vehicle systems 140. Various examples of one or more vehicle systems 140 are shown in FIG. 1 . However, vehicle 100 may include more, fewer, or different vehicle systems. While certain vehicle systems are defined separately, it should be appreciated that each or any of these systems or portions thereof may be otherwise combined or separated within vehicle 100 via hardware and / or software. Vehicle 100 may include a propulsion system 141, a braking system 142, a steering system 143, a throttle system 144, a transmission system 145, a signal system 146, and / or a navigation system 147. Each of these systems may include one or more now known or future developed devices, components, and / or combinations thereof.

[0068] Navigation system 147 may include one or more now known or later developed devices, applications, and / or combinations thereof configured to determine the geographic location of vehicle 100 and / or determine driving routes for vehicle 100. Navigation system 147 may include one or more mapping applications for determining driving routes for vehicle 100. Navigation system 147 may include a global positioning system, a local positioning system, or a geolocation system.

[0069] The processor 110, the depth system 170, and / or the autonomous driving module 160 may be operatively connected (i.e., communicatively coupled) to communicate with the various vehicle systems 140 and / or their individual components. For example, returning to FIG. 1 , the processor 110 and / or the autonomous driving module 160 may be in communication to send and / or receive information from the various vehicle systems 140 for purposes of controlling the movement, speed, operation, path, direction, etc. of the vehicle 100. The processor 110, the depth system 170, and / or the autonomous driving module 160 may control some or all of these vehicle systems 140 and, therefore, be partially or fully autonomous.

[0070] The processor 110, the depth system 170, and / or the autonomous driving module 160 may be operatively connected to communicate with various vehicle systems 140 and / or individual components thereof. For example, returning to FIG. 1 , the processor 110, the depth system 170, and / or the autonomous driving module 160 may be in communication to send and / or receive information from the various vehicle systems 140 for purposes of controlling the movement, speed, operation, path, direction, etc. of the vehicle 100. The processor 110, the depth system 170, and / or the autonomous driving module 160 may control some or all of these vehicle systems 140.

[0071] Processor 110, depth system 170, and / or autonomous driving module 160 may be operable to control the navigation and / or operation of vehicle 100 by controlling vehicle systems 140 and / or one or more of its components. For example, when operating in autonomous mode, processor 110, depth system 170, and / or autonomous driving module 160 may control the direction and / or speed of vehicle 100. Processor 110, depth system 170, and / or autonomous driving module 160 may cause vehicle 100 to accelerate (e.g., by increasing fuel supply provided to the engine), decelerate (e.g., by decreasing fuel supply to the engine and / or by braking), and / or turn (e.g., by rotating the two front wheels). As used herein, "cause" or "causing" means to directly or indirectly cause, compel, oblige, direct, command, command, and / or enable an event or action to occur or to be in a state in which such an event or action can occur.

[0072] Vehicle 100 may include one or more actuators 150. Actuator 150 may be any element or combination of elements operable to modify, adjust, and / or alter one or more of vehicle systems 140 or its components responsive to receiving signals or other outputs from processor 110 and / or autonomous driving module 160. Any suitable actuator may be used. For example, one or more actuators 150 may include motors, pneumatic actuators, hydraulic pistons, relays, solenoids, and / or piezoelectric actuators, to name a few possibilities.

[0073] Vehicle 100 may include one or more modules, at least some of which are described herein. The modules may be implemented as computer-readable program code that, when executed by processor 110, implements one or more of the various processes described herein. One or more of the modules may be a component of processor 110, or alternatively, one or more of the modules may be executed on and / or distributed among other processing systems to which processor 110 is operatively connected. The modules may include instructions (e.g., program logic) executable by one or more processors 110. Alternatively, or additionally, one or more data stores 115 may store such instructions.

[0074] In one or more arrangements, one or more of the modules described herein may include artificial or computational intelligence elements, such as neural networks, fuzzy logic, or other machine learning algorithms. Further, in one or more arrangements, one or more of the modules may be distributed among multiple of the modules described herein. In one or more arrangements, two or more of the modules described herein may be combined into a single module.

[0075] Vehicle 100 may include one or more autonomous driving modules 160. Autonomous driving module 160 may be configured to receive data from sensor system 120 and / or any other type of system capable of capturing information about vehicle 100 and the vehicle's 100's external environment. In one or more arrangements, autonomous driving module 160 may use such data to generate one or more driving scene models. Autonomous driving module 160 may determine the position and speed of vehicle 100. Autonomous driving module 160 may determine the location of obstacles, obstructions, or other environmental features, including traffic signs, trees, shrubs, adjacent vehicles, pedestrians, etc.

[0076] The autonomous driving module 160 may be configured to receive and / or determine location information about obstacles in the external environment of the vehicle 100 for use by the processor 110 and / or one or more of the modules described herein to estimate the position and orientation of the vehicle 100, the position of the vehicle in global coordinates based on signals from multiple satellites, or any other data and / or signals that may be used to determine the current state of the vehicle 100 or to determine the position of the vehicle 100 in relation to the environment for use in creating a map or determining the position of the vehicle 100 in relation to map data.

[0077] The autonomous driving module 160, independently or in combination with the depth system 170, may be configured to determine a driving path, a current autonomous driving maneuver for the vehicle 100, a future autonomous driving maneuver, and / or modifications to the current autonomous driving maneuver based on data acquired by the sensor system 120, a driving scene model, and / or data from any other suitable source. A "driving maneuver" refers to one or more actions that affect the movement of the vehicle. Examples of driving maneuvers include accelerating, decelerating, braking, turning, moving the vehicle 100 laterally, changing lanes of travel, merging into a lane of travel, and / or reversing, to name a few possibilities. The autonomous driving module 160 may be configured to implement the determined driving maneuver. The autonomous driving module 160 may directly or indirectly cause such autonomous driving maneuver to be implemented. As used herein, "cause" or "causing" means directly or indirectly causing, commanding, ordering, and / or enabling an event or action to occur or to be in a state in which such an event or action can occur. Autonomous driving module 160 may be configured to perform various vehicle functions and / or transmit data to, receive data from, interact with, and / or control vehicle 100 or one or more systems thereof (e.g., one or more of vehicle systems 140).

[0078] Detailed embodiments are disclosed herein. However, it should be understood that the disclosed embodiments are intended merely as examples. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a basis for the claims and as representative principles for teaching those skilled in the art to utilize the aspects herein in various ways in almost any appropriately detailed structure. Furthermore, the terms and phrases used herein are not intended to be limiting, but rather to provide an understandable depiction of possible implementations. While various embodiments are shown in Figures 1-5, the embodiments are not limited to the illustrated structures or fields of use.

[0079] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, including one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved.

[0080] The systems, components, and / or processes described above can be implemented in hardware or a combination of hardware and software, and can be centralized in one processing system or distributed where different elements are spread across multiple interconnected processing systems. Any type of processing system or other apparatus adapted to perform the methods described herein is suitable. A typical combination of hardware and software can be a processing system with computer-usable program code that, when loaded and executed, controls the processing system in a manner that performs the methods described herein. The systems, components, and / or processes can also be embedded in a computer-readable storage device, such as a machine-readable computer program product or other data program storage device, that tangibly embodies a program of instructions executable by a machine to perform the methods and processes described herein. These elements can also be embedded in an application product that includes all features enabling implementation of the methods described herein and that can perform these methods when loaded into a processing system.

[0081] Furthermore, the arrangements described herein may take the form of a computer program product embodied in one or more computer-readable media, for example, having computer-readable program code stored thereon. Any combination of one or more computer-readable media may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The phrase "computer-readable storage medium" refers to a non-transitory storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media may include the following: a portable computer diskette, a hard disk drive (HDD), a solid-state drive (SSD), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus or device.

[0082] Generally, as used herein, a module includes a routine, program, object, component, data structure, etc. that performs a particular task or implements a particular data type. In a further aspect, a memory generally stores the indicated module. The memory associated with a module may be a buffer or cache embedded within a processor, RAM, ROM, flash memory, or another suitable electronic storage medium. In a further aspect, the modules contemplated by this disclosure are implemented as hardware components of an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a programmable logic array (PLA), or another suitable hardware component embedded with a defined configuration set (e.g., instructions) to perform the disclosed functions.

[0083] Program code embodied on a computer-readable medium can be transmitted using any suitable medium, including, but not limited to, wireless, wired, fiber optic, cable, RF, etc., or any suitable combination thereof. Computer program code for performing operations for aspects of the present arrangement may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or may connect to an external computer (e.g., through the Internet using an Internet service provider).

[0084] The terms "a" and "an," as used herein, are defined as one or more than one. The term "plurality," as used herein, is defined as two or more than two. The term "another," as used herein, is defined as at least a second or more. The terms "including" and / or "having," as used herein, are defined as comprising (i.e., open language). As used herein, "at least one of" means and includes any and all possible combinations of one or more of the associated listed items. As an example, the phrase "at least one of A, B and C" includes A only, B only, C only, or any combination thereof (e.g., AB, AC, BC, or ABC).

[0085] The aspects herein may be embodied in other forms without departing from the spirit or essential attributes thereof, and accordingly, reference should be made to the following claims, rather than the foregoing specification, as indicating their scope. [Aspect 1] 1. A depth system comprising: one or more processors; a memory communicatively coupled to the one or more processors, the memory communicatively coupled to the one or more processors, the memory communicatively coupled to the one or more processors when executed by the one or more processors, Using a combinatorial network backbone, we generate feature maps at multiple scales from the input image. decoding a feature map using a top-down path of the combinatorial network to provide features at the multiple scales; using the head of the combinatorial network to generate a depth map from features about the scene depicted in the input image and bounding boxes that identify objects in the input image; a memory storing a network module including instructions; Depth system, including [Aspect 2] 2. The depth system of claim 1, wherein the network module includes instructions for training the combined network in a first stage by using a supervised depth loss derived from the depth map. Aspect 3 3. The depth system of claim 2, wherein the network module includes instructions for training the combined network in a second stage by using the bounding boxes and ground truth data to calculate a detection loss. Aspect 4 The depth system of aspect 1, wherein the network module includes decoding instructions including instructions for using lateral connections between the backbone and the top-down pathway to provide the multiple scales of the feature map in addition to the output of a previous level from within the top-down pathway. Aspect 5 the network module includes instructions for generating the feature map, including instructions for generating the feature map at the multiple scales as a feature hierarchy; the network module includes instructions for generating the feature map to encode features of the input image to provide a common reference for generating the depth map and the bounding box. 2. The depth system of embodiment 1. Aspect 6 The depth system of aspect 1, wherein the network module includes instructions for generating the depth map and the bounding box, including instructions for using the heads in separate layers of the top-down path at the multiple scales to generate the bounding box at the multiple scales and generate the depth map at one of the multiple scales. Aspect 7 2. The depth system of claim 1, wherein the head includes multiple sub-heads that perform 3D object detection, 2D object detection, depth estimation, and classification. Aspect 8 2. The depth system of claim 1, wherein the input image is an RGB monocular image. Aspect 9 When executed by one or more processors, the one or more processors: Using a combinatorial network backbone, we generate feature maps at multiple scales from the input image. decoding a feature map using a top-down path of the combinatorial network to provide features at the multiple scales; using the head of the combinatorial network to generate a depth map from features about the scene depicted in the input image and bounding boxes that identify objects in the input image; A non-transitory computer-readable medium containing instructions. Aspect 10 10. The non-transitory computer-readable medium of claim 9, wherein the instructions include instructions for, in a first stage, training the combinatorial network by using a supervised depth loss derived from the depth map. Aspect 11 11. The non-transitory computer-readable medium of claim 10, wherein the instructions include instructions for, in a second stage, training the combined network by using the bounding boxes and ground truth data to calculate a detection loss. Aspect 12 A non-transitory computer-readable medium as described in aspect 9, wherein the decoding instructions include instructions for using lateral connections between the backbone and the top-down pathway to provide the multiple scales of the feature map in addition to the output of a previous level from within the top-down pathway. Aspect 13 the instructions to generate the feature map include instructions to generate the feature map at the multiple scales as a feature hierarchy; instructions for generating a feature map encode features of the input image to provide a common reference for generating the depth map and the bounding box; 10. The non-transitory computer-readable medium of embodiment 9. Aspect 14 generating feature maps at multiple scales from an input image using a combinatorial network backbone; decoding a feature map using a top-down path of the combinatorial network to provide features at the multiple scales; using the head of the combinatorial network to generate a depth map from features about the scene depicted in the input image and a bounding box that identifies an object in the input image; A method comprising: Aspect 15 In a first stage, training the combined network by using a supervised depth loss derived from the depth map; 15. The method of embodiment 14, further comprising: Aspect 16 In a second stage, training the combined network by using the bounding boxes and ground truth data to calculate a detection loss; 16. The method of embodiment 15, further comprising: Aspect 17 15. The method of claim 14, wherein decoding includes using lateral connections between the backbone and the top-down pathway to provide the multiple scales of the feature map in addition to outputs of previous levels from within the top-down pathway. Aspect 18 generating the feature maps includes generating the feature maps at the multiple scales as a feature hierarchy; generating the feature map encodes features of the input image to provide a common reference for generating the depth map and the bounding box; 15. The method of embodiment 14. Aspect 19 The method of claim 14, wherein generating the depth map and the bounding box includes using the head in a separate layer of the top-down path at the multiple scales to generate the bounding box at the multiple scales and the depth map at one of the multiple scales. Aspect 20 providing the depth map and the bounding box for causing navigation of a device according to the depth map and the bounding box; further comprising The method of claim 14, wherein the head includes multiple sub-heads that perform 3D object detection, 2D object detection, depth estimation, and classification, and the input image is an RGB monocular image.

Claims

1. 1. A depth system comprising: one or more processors; a memory communicatively coupled to the one or more processors, the memory communicatively coupled to the one or more processors, the memory communicatively coupled to the one or more processors when executed by the one or more processors, Using a combinatorial network backbone, we generate feature maps at multiple scales from the input image. decoding a feature map using a top-down path of the combinatorial network to provide features at the multiple scales; using the head of the combinatorial network to generate a depth map from features about the scene depicted in the input image and bounding boxes that identify objects in the input image; a memory storing a network module including instructions; Including, The network module includes instructions for, in a first stage, training the combined network by using a supervised depth loss derived from the depth map, and instructions for, in a second stage, training the combined network by using the bounding box and ground truth data to compute a detection loss. Depth system.

2. 2. The depth system of claim 1, wherein the network module includes decoding instructions including instructions for using lateral connections between the backbone and the top-down pathway to provide the multiple scales of the feature map in addition to outputs of previous levels from within the top-down pathway.

3. the network module includes instructions for generating the feature map, including instructions for generating the feature map at the multiple scales as a feature hierarchy; the network module includes instructions for generating the feature map to encode features of the input image to provide a common reference for generating the depth map and the bounding box. The depth system of claim 1 .

4. 2. The depth system of claim 1, wherein the network module includes instructions for generating the depth map and the bounding box, the instructions including instructions for using the heads in separate layers of the top-down path at the multiple scales to generate the bounding box at the multiple scales and generate the depth map at one of the multiple scales.

5. The depth system of claim 1 , wherein the head includes multiple sub-heads that perform 3D object detection, 2D object detection, depth estimation, and classification.

6. The depth system of claim 1 , wherein the input image is an RGB monocular image.

7. When executed by one or more processors, the one or more processors: Using a combinatorial network backbone, we generate feature maps at multiple scales from the input image. decoding a feature map using a top-down path of the combinatorial network to provide features at the multiple scales; using the head of the combinatorial network to generate a depth map from features about the scene depicted in the input image and bounding boxes that identify objects in the input image; Contains instructions, The instructions include, in a first stage, instructions for training the combined network by using a supervised depth loss derived from the depth map, and in a second stage, instructions for training the combined network by using the bounding box and ground truth data to compute a detection loss. Non-transitory computer-readable medium.

8. 8. The non-transitory computer-readable medium of claim 7, wherein the decoding instructions include instructions for providing the multiple scales of the feature map in addition to an output of a previous level from within the top-down path using lateral connections between the backbone and the top-down path.

9. the instructions to generate the feature map include instructions to generate the feature map at the multiple scales as a feature hierarchy; instructions for generating a feature map encode features of the input image to provide a common reference for generating the depth map and the bounding box; The non-transitory computer-readable medium of claim 7.

10. generating feature maps at multiple scales from an input image using a combinatorial network backbone; decoding a feature map using a top-down path of the combinatorial network to provide features at the multiple scales; using the head of the combinatorial network to generate a depth map from features about the scene depicted in the input image and bounding boxes that identify objects in the input image; In a first stage, training the combined network by using a supervised depth loss derived from the depth map; In a second stage, training the combined network by using the bounding boxes and ground truth data to calculate a detection loss; A method comprising:

11. 11. The method of claim 10, wherein decoding comprises using lateral connections between the backbone and the top-down pathway to provide the multiple scales of the feature map in addition to outputs of previous levels from within the top-down pathway.

12. generating the feature maps includes generating the feature maps at the multiple scales as a feature hierarchy; generating the feature map encodes features of the input image to provide a common reference for generating the depth map and the bounding box; The method of claim 10.

13. 11. The method of claim 10, wherein generating the depth map and the bounding box includes using the heads in separate layers of the top-down path at the multiple scales to generate the bounding box at the multiple scales and the depth map at one of the multiple scales.

14. providing the depth map and the bounding box for causing navigation of a device according to the depth map and the bounding box; further comprising The method of claim 10 , wherein the head includes multiple sub-heads that perform 3D object detection, 2D object detection, depth estimation, and classification, and the input image is an RGB monocular image.

Images