Self-supervised depth estimation method and system

Abstract

As deep neural networks are increasingly used for generating dense depth maps, depth perception has become an increasingly popular topic in the image community. However, the application of depth perception estimation can still be limited due to the need for large amounts of dense ground truth depth for training. It is contemplated that a self-supervised control strategy can be developed for estimating depth maps using color images and data provided by a sensor system (e.g., sparse lidar data). Such a self-supervised control strategy can utilize superpixels (i.e., groups of pixels that share common characteristics (e.g., pixel intensity)) as locally planar regions to regularize surface normal derivatives according to estimated depth and photometric loss. The control strategy can be operable to produce dense depth maps that do not require dense ground truth supervision.

Description

Technical Field

[0001] The following generally relates to self-supervised depth estimation systems and methods for generating dense depth maps. Background Technology

[0002] Applications similar to autonomous driving generally require dense and accurate depth maps. However, this typically necessitates high-quality and costly sensors and acquisition devices to generate such dense and accurate depth maps. The cost of these sensors and acquisition devices makes the use of dense and accurate depth maps prohibitive for many applications. Summary of the Invention

[0003] Systems and methods for self-supervised depth estimation receive digital images of the environment. One or more deep superpixel segments can be extracted from the digital image, and these segments can be partitioned to represent homogeneous regions of the digital image. The deep superpixel segments can also be operable as local planar regions that constrain the second derivatives of the local normal orientation and depth within one or more deep superpixel segments. Finally, a dense depth map can be generated using one or more deep superpixel segments.

[0004] The system and method can also derive surface normal maps using a deep regression neural network that regresses a full-resolution depth map from digital images and sparse depth map samples received from a depth sensor. The deep regression neural network can also be designed using an encoder-decoder architecture with encoding layers, decoding layers, and multiple skip connections. The encoding layer may include one or more convolutional layers, one or more ReLU layers, one or more ResNet layers, and one or more pooling layers. Furthermore, the decoding layer may include one or more deconvolutional layers, one or more unpooling layers, one or more ResNet layers, and one or more ReLU layers. The final convolutional layer can also be operated to produce a non-negative grayscale depth image for deriving the surface normal map.

[0005] The system and method can be further operable to compute gradients of sparse depth map samples in four directions. The sparse depth map samples can be transformed into one or more 3D vectors. The system and method can average one or more normalized cross products of the one or more 3D vectors to determine vertex normals.

[0006] The system and method are operable to determine the relative transformation between a digital image and a correlated image using a simultaneous localization and mapping system. Photometric loss can be determined using the relative transformation, the digital image, and the correlated image. The system and method are also operable to smooth and suppress inconsistencies within dense depth maps by minimizing the depth second derivative within one or more deep superpixel segments. The system and method can determine local normal directions derived using estimated depth. The system and method can invert boundaries and edges within one or more superpixel segments. Finally, the system and method can apply consistency of normal directions within each of one or more deep superpixel segments. Attached Figure Description

[0007] Figure 1 An exemplary computing system is illustrated.

[0008] Figure 2 An exemplary self-supervised depth estimation network is shown.

[0009] Figure 3A and Figure 3B An encoder-decoder architecture that can be implemented by a deep regression network is shown.

[0010] Figure 4 A computing system for controlling at least a partially autonomous robot is shown. Detailed Implementation

[0011] Detailed embodiments are disclosed herein as requested; however, it should be understood that the disclosed embodiments are merely illustrative and may be implemented in various and alternative forms. The drawings are not necessarily to scale; some features may be enlarged or minimized to show details of specific components. Therefore, the specific structural and functional details disclosed herein are not to be construed as limiting, but rather serve only as a representative basis for teaching those skilled in the art to employ these embodiments in various ways.

[0012] Depth perception is generally understood as the visual ability to perceive the distance to the world and objects in three dimensions (3D). Depth perception can be determined from a variety of depth images and cues. For example, depth perception can include binocular images (which may be based on sensing information received in three dimensions) and monocular images (which include images represented in only two dimensions).

[0013] Depth maps, including images or image channels containing information about the distances of surfaces of scene objects from the viewpoint, can also be used to determine depth perception. For example, dense and accurate depth maps can be used to capture indoor or outdoor environments. It is envisioned that dense depth maps could be useful for applications including 3-D object reconstruction, augmented reality, robot manipulation, and autonomous driving. It is also envisioned that 3-D image capture technologies and sensors (e.g., LiDAR) could be operational to provide 3-D data for generating dense depth maps.

[0014] However, for some applications, LiDAR sensors operable to generate high-quality dense depth maps can be prohibitively expensive. On the other hand, lower-cost LiDAR sensors may not provide the accuracy and resolution required for accurate dense depth maps. It is also envisioned that alternative 3D capture solutions (e.g., depth cameras) could have similar cost and performance trade-offs to LiDAR sensors. To increase the number of applications for 3D capture, it is envisioned that lower-cost and lower-quality depth sensors will be desired. However, to implement lower-cost and higher-quality depth sensors, more robust algorithms may need to be developed that estimate depth maps at higher resolution from sparse depth map samples 204 (or even as small as no samples) to compensate for the lower-quality resolution of the 3D capture device.

[0015] With the rapid development of deep learning, depth estimation algorithms can leverage deep neural networks to generate depth maps from monocular color images and / or sparse depth datasets. These algorithms can rely on dense ground-based depth maps as supervision during network training. Such ground-based data generally includes information that can be provided through direct observation (i.e., empirical evidence), as opposed to information provided through inference.

[0016] However, the reliance on dense ground-based depth maps is not optimal, as the initial goal of depth-aware estimation was to reduce the need for such dense depth maps. When applying estimation algorithms to new applications or environments, performance can be limited due to the lack of these dense ground-based maps. Self-supervised learning can sometimes overcome these performance constraints by leveraging geometric constraints between multiple pairs of consecutive images. However, self-supervised depth estimation can have much lower accuracy because geometric constraints are not as accurate as ground-based depth maps.

[0017] To enhance dense depth maps from self-supervised algorithms, stronger supervision of ground-based depth maps has been developed, in addition to geometrically constrained solutions based solely on images. For example, sparse depth measurements captured by low-end LiDAR are one approach that can be employed. With supervision provided by sparse depth measurements, image appearance blurring (e.g., repetitive patterns) can be overcome. Alternatively, multi-task learning trained to learn multiple modalities (e.g., normal, optical flow) can be used to improve depth quality and reduce model overfitting.

[0018] Figure 1 An exemplary system 100 is illustrated, which can be used to implement a self-supervised depth estimation network for estimating depth maps based on color images and sparse lidar samples. System 100 may include at least one computing device 102. Computing system 102 may include at least one processor 104 operatively connected to memory unit 108. Processor 104 may be one or more integrated circuits implementing the functions of a central processing unit (CPU) 106. It should be understood that CPU 106 may also be one or more integrated circuits implementing the functions of a general-purpose processing unit or a special-purpose processing unit (e.g., a graphics processing unit, ASIC, FPGA).

[0019] CPU 106 may be a commercially available processing unit implementing an instruction set (e.g., one of the x86, ARM, Power, or MIPS instruction set families). During operation, CPU 106 may execute stored program instructions retrieved from memory unit 108. The stored program instructions may include software that controls the operation of CPU 106 to perform the operations described herein. In some examples, processor 104 may be a system-on-a-chip (SoC) that integrates the functionality of CPU 106, memory unit 108, network interface, and input / output interface into a single integrated device. Computing system 102 may implement an operating system for managing various aspects of operation.

[0020] Memory cell 108 may include volatile and non-volatile memory for storing instructions and data. Non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 102 is disabled or loses power. Volatile memory may include static and dynamic random access memory (RAM) for storing program instructions and data. For example, memory cell 108 may store a machine learning model 110 or algorithm, a training dataset 112 for the machine learning model 110, and / or raw source data 115.

[0021] The computing system 102 may include a network interface device 122 configured to provide communication with external systems and devices. For example, the network interface device 122 may include wired and / or wireless Ethernet interfaces, as defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards. The network interface device 122 may include a cellular communication interface for communicating with cellular networks (e.g., 3G, 4G, 5G). The network interface device 122 may be further configured to provide a communication interface to an external network 124 or the cloud.

[0022] External network 124 may be referred to as the World Wide Web or the Internet. External network 124 can establish standard communication protocols between computing devices. External network 124 allows information and data to be easily exchanged between computing devices and the network. One or more servers 130 can communicate with external network 124.

[0023] The computing system 102 may include an input / output (I / O) interface 120 that can be configured to provide digital and / or analog inputs and outputs. The I / O interface 120 may include an additional serial interface (e.g., a Universal Serial Bus (USB) interface) for communicating with external devices.

[0024] The computing system 102 may include a human-machine interface (HMI) device 118, which may include any means that enable the system 100 to receive control input. Examples of input devices may include HMI inputs, such as a keyboard, mouse, touchscreen, voice input device, and other similar devices. The computing system 102 may include a display device 132. The computing system 102 may include hardware and software for outputting graphical and textual information to the display device 132. The display device 132 may include an electronic display screen, projector, printer, or other suitable means for displaying information to a user or operator. The computing system 102 may be further configured to allow interaction with a remote HMI and a remote display device via a network interface device 122.

[0025] System 100 may be implemented using one or more computing systems. Although the example illustrates a single computing system 102 implementing all the described features, the aim is that the various features and functions can be decoupled and implemented through multiple computing units that communicate with each other. The specific system architecture selected may depend on various factors.

[0026] System 100 may implement machine learning algorithm 110, which is configured to analyze raw source data 115. Raw source data 115 may include raw or unprocessed sensor data that can represent an input dataset for the machine learning system. Raw source data 115 may include video, video clips, images, and raw or partially processed sensor data (e.g., data from a digital camera or LiDAR sensor). In some examples, machine learning algorithm 110 may be a neural network algorithm designed to perform a predetermined function. For example, in automotive applications, a neural network algorithm may be configured to identify objects (e.g., pedestrians) from images provided by a digital camera and / or depth maps from a LiDAR sensor.

[0027] System 100 may store a training dataset 112 for use with machine learning algorithm 110. Training dataset 112 may represent a set of previously constructed data used to train machine learning algorithm 110. Training dataset 112 may be used by machine learning algorithm 110 to learn weighting factors associated with a neural network algorithm. Training dataset 112 may include a set of source data having corresponding outcomes or results that machine learning algorithm 110 attempts to replicate via the learning process. In this example, training dataset 112 may include source images and depth maps from various scenes in which identifiable objects (e.g., pedestrians) are present.

[0028] Machine learning algorithm 110 can operate in a learning mode using training dataset 112 as input. Machine learning algorithm 110 can be executed through multiple iterations using data from training dataset 112. In each iteration, machine learning algorithm 110 can update its internal weighting factors based on the achieved results. For example, machine learning algorithm 110 can compare its output results with those included in training dataset 112. Since training dataset 112 includes expected results, machine learning algorithm 110 can determine when performance is acceptable. After machine learning algorithm 110 achieves a predetermined performance level (e.g., 100% consistency with results associated with training dataset 112), machine learning algorithm 110 can be executed using data not in training dataset 112. The trained machine learning algorithm 110 can be applied to a new dataset to generate annotated data.

[0029] Machine learning algorithm 110 can also be configured to identify specific features in raw source data 115. Raw source data 115 may include multiple scenarios or input datasets in which annotation results are expected. For example, machine learning algorithm 110 may be configured to identify the presence of a pedestrian in an image and annotate the occurrence. Machine learning algorithm 110 may be programmed to process raw source data 115 to identify the presence of specific features. Machine learning algorithm 110 may be configured to identify features in raw source data 115 as predetermined features. Raw source data 115 may be derived from various sources. For example, raw source data 115 may be actual input data collected by a machine learning system. Raw source data 115 may be machine-generated for testing the system. As an example, raw source data 115 may include raw digital images from a camera.

[0030] In the example, machine learning algorithm 110 processes raw source data 115 and generates output. Machine learning algorithm 110 may generate a confidence level or factor for each generated output. For example, a confidence value exceeding a predetermined high confidence threshold may indicate that machine learning algorithm 110 is confident that the identified feature corresponds to a specific feature. A confidence value below a low confidence threshold may indicate that machine learning algorithm 110 has some uncertainty regarding the existence of a specific feature.

[0031] Figure 2 A self-supervised depth estimation network 200 representing a machine learning algorithm 110 is shown. It is envisioned that the self-supervised depth estimation network 200 is operable to estimate depth maps from color images including constraints from sources of variation and modalities, thereby improving depth quality and reducing model overfitting.

[0032] Constructed, network 200 may receive one or more color images 202 (i.e., two-dimensional or three-dimensional RGB color images) from a digital camera or camcorder (e.g., a DSLR camera or a mirrorless digital camera) operable to capture and / or generate RGB images. Network 200 may also receive sparse depth map samples 204, which may be provided by a high-resolution or low-resolution depth sensor (e.g., a LiDAR sensor). Network 200 may be operable to utilize color images 202, sparse depth map samples 204, extracted deep superpixels 206, and closely related color images 208 as data 115 provided for training machine learning algorithm 110.

[0033] Network 200 may also include several network paths. For example, network 200 may include a depth loss path 210, wherein a color image 202 and its corresponding sparse depth map sample 204 are passed to a depth regression path 212, which may produce a dense depth map 214 and a derived surface normal map 216. It is envisioned that the depth regression path 212 can output a dense depth map 214 (which includes an estimated depth (d) for each pixel) by utilizing the color image 202 and the sparse depth map sample 204 (i.e., samples provided by a LiDAR sensor). It is also envisioned that the depth regression path 212 may be regularized using several cues obtained from the input during training routines. For example, cues may include superpixels, neighboring images, and camera pose.

[0034] Figure 3A and Figure 3B A convolutional neural network (CNN) 300 is shown that can be used as a deep regression path 212. The CNN 300 is designed to provide enhanced performance for image-to-image regression using encoder-decoder structures and skip connections.

[0035] As shown, CNN 300 includes one or more encoder layers 302 and one or more decoder layers 304. Encoder layer 302 may include one or more convolutional layers followed by an activation function (e.g., a Corrected Linear Unit (ReLU) function). For example, CNN 300 may input a color image 202 into layer 306, which includes a convolutional layer and a ReLU activation function. Similarly, CNN 300 may input sparse depth map samples 204 into layer 308, which may include a single convolutional layer and a ReLU activation function. It is envisioned that layers 306-308 may include convolutional layers with kernels of the same size (e.g., 64x304). It is also envisioned that the color image 202 and the sparse depth map samples 204 may be provided to a single layer (e.g., layer 306) with the same convolutional layer and ReLU activation function, instead of a separate layer with a separate convolutional layer and ReLU activation function. Layers 306 and 308 can then be provided to layer 310, which may include a residual neural network (ResNet) and a ReLU activation function. Layer 310 may also be designed to have the same size and dimensions as layers 306-308 (e.g., 64x304).

[0036] Encoding layer 302 may also include one or more additional layers 312-318. Each additional layer (e.g., layer 312) may include a ResNet layer, a ReLU activation function, and a pooling layer. It is envisioned that the size of each layer 312-318 may vary, and the size of each layer 312-318 may depend on the size of image 202 and sparse depth map sample 204. For example, color image 202 and sparse depth map sample 204 may include 64 channel features, which are concatenated and fed into layers 312-318. Layers 312-318 may then be designed to downsample each received feature map by half. Layer 310 may provide a feature map of size 324, which is downsampled from layer 312 to size 212. Layer 312 may provide a feature map of size 212, which is downsampled from layer 314 to size 56. Layer 314 may provide a feature map of size 56, which is downsampled from layer 316 to size 28. Layer 316 provides a feature map of size 28, which is downsampled to size 14 by layer 318. Layer 320, with additional convolutional layers and ReLU activation function layers, receives the feature map from layer 318 and further downsamples the number of features to 512.

[0037] Then, convolutional layer 320 provides the features to decoder layer 304. It is envisioned that decoder layer 3014 comprises one or more layers 322-360. Each layer (e.g., layer 322) may be designed to include an unpooling layer, a deconvolution layer, a ResNet layer, and a ReLU activation function. It is also envisioned that decoder layer 304 operates using a transposed convolution process that upsamples the feature map back to the original resolution of image 202 and sparse depth map samples 204. For the upsampling process, CNN 300 may also include one or more skip connections 344-352 that extend from the outputs of layers 310-318 to the inputs of layers 324, 328, 332, 336, and 340 (i.e., the corresponding upsampling unit layers). The envisioned approach is that CNN 300 could be designed to fuse color images 202 and sparse depth map samples 204 at layers 306-308 to prevent unwanted noise in the depth branches from flowing into the later stages of the decoder via skip connections 344-352.

[0038] Layer 354 may be designed as a final convolutional layer, operable to compress all channels into one, thereby forming a non-negative grayscale depth image. It is also contemplated that layers 306, 308, and 320 may comprise convolutional layers using a batch normalization design with an attached ReLU activation function. Meanwhile, layer 354 may comprise a convolutional layer simply including a ReLU activation function.

[0039] Layer 356 may be designed to derive a surface normal map of the depth image. Inherent parameters of the apparatus used to generate the color image 202 (e.g., a color camera) are used. Layer 356 may be operable to compute gradients of the depth map in four directions and to convert the depth map into 3D vectors. Layer 356 may further be operable to compute vertex normals by averaging the normalized cross products of adjacent 3D vectors.

[0040] refer to Figure 2 Network 200 may further include a photometric loss path 218, which utilizes the image alignment error between color image 202 and nearby color images 208 to further supervise the training of depth regression path 212. Photometric loss path 218 may utilize sparse depth map samples 204, the inherent properties of the device acquiring image 202 (e.g., a camera), and relative transformations (T0). 1→2 And supervise the deep regression path 212.

[0041] The envisioned issue is that the photometric loss from appearance matching can be a common image distortion problem in RGBD visual range estimation. Instead of estimating the relative pose of the camera system, network 200 is designed to train depth estimation network 212 using a fixed camera transformation. For example, network 200 can use the intrinsic parameters K of image 202, nearby images 208, sparse depth map samples 204, and the capturing device (i.e., the digital camera), as well as the relative transformation 226 (i.e., T). 1→2 The system retrieves all pixels in the image space from image 204 to image 202, thereby generating a warped image (I1′). It should be understood that the more accurate the input to the sparse depth map sample 204 (i.e., the more accurate the depth sensor), the smaller the difference between the warped image (I1′) and image 202. The system 102 is also operable to determine the matching error between the warped image (I1′) and image 202 as supervision for determining the dense depth map 214 (i.e., d1).

[0042] Network 200 may also be operable to use a known simultaneous localization and mapping (SMR) or SLAM system 224 (e.g., ORBSLAM) to generate a relative transformation 226 between a pair of selected frames. For example, network 200 may receive and provide image 202 (i.e., frame 1) and sparse depth map samples 204 (i.e., frame 2) to SLAM system 224, which then determines the absolute camera trajectory T. i The relative transformation of pose between frames 226 (i.e., T) 1→2 It can be equal to T1T2 −1 The envisioned approach is that if the SLAM system 224 cannot provide a robust estimate for any provided frame, then the network 200 can simply not calculate the photometric loss (L). p ).

[0043] It is also envisioned that, with sparse depth map samples 204 as input, the estimation error may increase slightly with increasing noise. However, if the network 200 is not provided with sparse depth map samples 204, the error may be large, and the network 200 may not be able to converge to noise levels exceeding 0.1 m. Therefore, it is envisioned that the SLAM system 224 assists the network 200 by training the depth regression path 212. The network 200 can further use the SLAM system 224 to generate reliable image poses (e.g., 6DoF poses for indoor datasets) instead of a less complex PnP solver or pose estimation network.

[0044] After determining the transformation between the two images, network 200 can be operational to project the pixels (p1) of the first image into 3D space and then return to the second image plane (p1′) using the following equation (1):

[0045] (Equation 1)

[0046] Where K is an inherent parameter of the capturing device (e.g., a camera); T 1→2 It is a relative transformation; d1 is the dense depth map 214; and p1 is a pixel from the color image 202. Then, the network 200 is operable to bilinearly resample the image 208 (i.e., I2) using the second image plane (p1′) calculated by equation (1) to determine the distorted image (I1′) using the following equation (2):

[0047] (Equation 2)

[0048] Network 200 can further use the following equation (3) to determine the appearance matching loss (L). p ):

[0049] (Equation 3).

[0050] Reference Back Figure 2 The planar loss path 220 can be operated by receiving the color image 202 into a pre-trained deep superpixel path. It is envisioned that the superpixel labels (L1) generated from image 202 can be used to regularize the derived surface normal map 216 (i.e., n1) and determine the second derivative (∇) of the depth map. 2d1), which allows network 200 to further improve the training of deep regression path 212. It is also envisioned that deep superpixel paths 222 may not be updated during the training of deep regression path 212. It is envisioned that when deep regression path 212 is optimally trained, network 200 may be operable to generate dense depth maps 214 from monochrome images (i.e., image 202) during operation and / or testing.

[0051] The general understanding is that surface normals are a method that can be used to help predict depth. However, previous methods typically regress surface normals through supervised training using ground truth normal maps derived from depth images. Network 200 regularizes surface normals to improve depth estimation in a self-supervised framework without requiring ground truth normal maps. It is envisioned that, to avoid requiring ground truth normal maps, Network 200 can require reliable constraints more closely related to the surface normals. Network 200 can use semantic image segmentation operations to avoid requiring ground truth normal maps. Network 200 can extract semantic information (e.g., roads and trees) to guide the direction of the estimated normals. For example, estimated normals with road labels should face upwards. However, such use of semantic information is highly dependent on the provided dataset. For example, training dataset 112 could be an indoor dataset, such as NYUv2. For this training dataset 112, the orientation of the digital camera may not be perpendicular to the ground. In order to adjust the camera positioning, the network 200 can use a deep superpixel path 222 to represent local homogeneous regions as indicators of local planar regions to regularize the normal direction.

[0052] For each provided image 202, network 200 can use a mask encoded as a multi-label. The network 200 determines the superpixel image 206 based on a known model. The network 200 may assume that superpixels represent planar regions, and that surface normals within these planar regions are in the same direction. Using this method, the network 200 can minimize the variance of normal map pixels belonging to the same mask label, as shown by the following equation (4):

[0053] (Equation 4).

[0054] It is envisioned that edge-aware smoothing of the depth map could be included in the loss function for additional depth estimation. Known methods for implementing smoothing generally exclude boundaries of different objects because depth can be discontinuous (i.e., geometric edges or occluded edges). It is also envisioned that analytical edges could be used for edge-aware smoothing, but Laplacian edges could include redundant edges caused by appearance (e.g., texture and lighting) rather than by geometric relationships.

[0055] Network 200 envisions using superpixels to guide the smoothing process. The boundaries of adjacent superpixel regions can be defined as superpixel edges (E). S When network 200 operates on a suitable number of superpixels, the geometric edges (E) G Superpixel edge (E) S ) and Laplace edge (E I The following containment relationship can be followed: E G ⊂ E S ⊂ E I Based on this relationship, Network 200 can use superpixel edges (E... S ) as for geometric edge (E) G The network 200 then operates on an approximation of the L1 normal of the generated depth map using the following equation (5):

[0056] (Equation 5).

[0057] The envisioned feature is that network 200 can ignore superpixel edges to promote sharpness along object edges. Moreover, network 200 operates to suppress additional depth discontinuities to prevent unintended noise and errors in the estimated depth (e.g., salt-and-pepper noise, hairline noise in planar regions).

[0058] Finally, network 200 can use the following equation (6) to compare the estimated depth map with the ground reality sparse depth map sample 204 at each non-zero pixel:

[0059] (Equation 6).

[0060] It is envisioned that several options exist for the distance function ‖·‖ (e.g., L1 norm and L2 norm), and the performance of different loss functions can depend on the dataset. It should also be understood that, empirically, L1 can provide enhanced performance for indoor RGBD-based datasets, and L2 can provide enhanced performance for outdoor LiDAR-based datasets, due to the noise level in depth measurements. Finally, it is envisioned that as the number of samples in the dataset decreases, Network 200 can rely more heavily on the loss function ‖·‖. P L N and L C .

[0061] Then, network 200 can use the following equation (7) to convert the final loss cost function (L) N ) is calculated as four losses (L) P L N L D and L C Weighted combination of () and scaling factor (λ):

[0062] (Equation 7)

[0063] Among them, the superscript L (i) Indicator Proportions. Network 200 can use a known four-level pyramid to compute the last three terms. For the final loss cost function, Network 200 can use different proportions with the same weights. For example, the weights for different loss terms can be empirically set to λ2 = 0.2, λ3 = 0.8, and λ4 = 0.5.

[0064] Figure 4 An embodiment is shown in which the computing system 440 can be used to control at least a partially autonomous robot (e.g., at least a partially autonomous vehicle 400). The computing system 440 can be similar to Figure 1 The system 100 described herein. Sensor 430 may include one or more video / camera sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more lidar sensors and / or one or more location sensors (such as, for example, GPS). Some or all of these sensors are preferred, but are not necessarily integrated into the vehicle 400.

[0065] Optionally, sensor 430 may include an information system for determining the state of the actuator system. Sensor 430 may collect sensor data or other information that will be used by computing system 440. An example of such an information system is a weather information system that determines the current or future state of weather in the environment. For example, using an input signal x, a classifier may, for example, detect an object near at least part of the autonomous robot. The output signal y may include information characterizing the location of the object near at least part of the autonomous robot. Control command A may then be determined based on this information, for example, to avoid a collision with the detected object.

[0066] The actuator 410, which can be integrated into the vehicle 400, can be provided by the vehicle 400's brakes, propulsion system, engine, powertrain, or steering system. An actuator control command can be determined to control the actuator (or multiple actuators) 410 so that the vehicle 400 avoids a collision with the detected object. The detected object can also be classified according to what a classifier deems it most likely to be (e.g., a pedestrian or a tree), and the actuator control command A can be determined based on the classification.

[0067] The processes, methods, or algorithms disclosed herein can be delivered to / implemented by a processing device, controller, or computer, which may include any existing programmable electronic control unit or dedicated electronic control unit. Similarly, processes, methods, or algorithms can be stored in many forms as data, logic, and instructions executable by a controller or computer, including but not limited to information permanently stored on non-writable storage media (e.g., ROM devices) and information variablely stored on writable storage media (e.g., floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media). Processes, methods, or algorithms can also be implemented in a software executable object. Optionally, processes, methods, or algorithms can be implemented wholly or partially using suitable hardware components (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or combinations of hardware, software, and firmware components).

[0068] While exemplary embodiments have been described above, these embodiments are not intended to describe all possible forms of the invention. Rather, the terms used in this specification are descriptive rather than limiting, and it should be understood that various changes may be made without departing from the spirit and scope of the invention. Additionally, features of various embodiments may be combined to form other embodiments of the invention.

Claims

1. Methods for self-supervised depth estimation, including: Receive digital images of the environment; Extract one or more deep superpixel segments from the digital image, wherein the one or more deep superpixel segments are divided to represent homogeneous regions of the digital image, and wherein the one or more deep superpixel segments are operable as local planar regions constraining the second derivative of the local normal direction and depth within the one or more deep superpixel segments. A dense depth map is generated using one or more deep superpixel segmentations; and Inconsistencies within the dense depth map are smoothed and suppressed by minimizing the depth second derivative within the one or more deep superpixel segments.

2. The method according to claim 1, further comprising: Receive sparse depth map samples; as well as A surface normal map is derived using a deep regression neural network that regresses a full-resolution depth map from the digital image and the sparse depth map samples.

3. The method according to claim 2, wherein, The deep regression neural network is designed using an encoder-decoder structure with an encoding layer, a decoding layer, and multiple skip connections.

4. The method according to claim 3, wherein, The encoding layer includes one or more convolutional layers, one or more ReLU layers, one or more residual neural networks (ResNet), and one or more pooling layers.

5. The method according to claim 3, wherein, The decoding layer includes one or more deconvolutional layers, one or more unpooling layers, one or more ResNet layers, and one or more ReLU layers.

6. The method according to claim 5, wherein, The decoding layer includes a final convolutional layer operation with a ReLU activation function to generate a non-negative grayscale depth image for deriving the surface normal map.

7. The method according to claim 2, further comprising: The gradients of the sparse depth map samples are calculated in four directions; The sparse depth map samples are converted into one or more 3D vectors; as well as The vertex normal is determined by averaging one or more normalized cross products of the one or more 3D vectors.

8. The method according to claim 1, further comprising: The relative transformation between the digital image and the related image is determined using a simultaneous localization and mapping system.

9. The method according to claim 8, further comprising: The photometric loss is determined using the relative transformation, the digital image, and the related image.

10. The method according to claim 1, further comprising: Invert the boundaries and edges within the one or more deep superpixel segments.

11. The method according to claim 1, wherein, The local normal direction is derived using the estimated depth.

12. The method according to claim 1, further comprising: The consistency of the normal direction is applied to each of the one or more deep superpixel segments.

13. Methods for self-supervised depth estimation include: Receive digital images of the environment; Extract one or more deep superpixel segments from the digital image, wherein the one or more deep superpixel segments are divided to represent homogeneous regions of the digital image; A dense depth map is generated using one or more deep superpixel segmentations; and Inconsistencies within the dense depth map are smoothed and suppressed by minimizing the depth second derivative within the one or more deep superpixel segments.

14. Systems for self-supervised depth estimation, including: A sensor, operable to receive digital images of the environment; Controller, operable, used for: Extract one or more deep superpixel segments from the digital image, wherein the one or more deep superpixel segments are divided to represent homogeneous regions of the digital image, and wherein the one or more deep superpixel segments are operable as local planar regions constraining the second derivative of the local normal direction and depth within the one or more deep superpixel segments. A dense depth map is generated using one or more deep superpixel segmentations; and Inconsistencies within the dense depth map are smoothed and suppressed by minimizing the depth second derivative within the one or more deep superpixel segments.

15. The system of claim 14, further comprising: A depth sensor, operable to receive sparse depth map samples; as well as The controller is further operable to: A surface normal map is derived using a deep regression neural network that regresses a full-resolution depth map from the digital image and the sparse depth map samples.

16. The system according to claim 15, wherein, The sensor that receives the digital image is a digital camera, and the depth sensor is a lidar sensor.

17. The system of claim 15, wherein the controller is further operable to: invert the boundaries and edges within the one or more deep superpixel segments.

18. The system of claim 15, wherein the controller is further operable to: determine the relative transformation between the digital image and the associated image using the simultaneous localization and mapping system of the system for self-supervised depth estimation.

Images