Methods for improving scale consistency and / or scale awareness in self-supervised deep and ego-motion prediction neural network models

Abstract

A method for improving scale consistency and / or scale awareness in models of self-supervised depth and ego-motion prediction neural networks processing video streams of monocular images, wherein complementary GPS coordinates synchronized with the images are used to compute a GPS scale loss to enforce scale consistency and / or scale awareness for monocular self-supervised ego-motion and depth estimation. The relative weight assigned to the GPS scale loss grows exponentially as training proceeds. The depth and ego-motion prediction neural networks are trained using an appearance-based photometric loss between real and synthetic object images and using a smoothness loss on depth prediction.

Description

Technical Field

[0001] The present invention relates to a method for improving scale consistency and / or scale perception in a model of a self-supervised depth and ego-motion prediction neural network processing a video stream of a monocular image. Background Art

[0002] Autonomous driving systems require scene understanding for planning and navigation. Therefore, spatial awareness is essential to achieve complex behaviors in unconstrained environments, which is achieved through depth and ego-motion estimation. Even though sensors such as LiDARs can sense depth and can be used to compute ego-motion at a metric scale [1,2], their output depth is sparse and they are expensive to use. In contrast, monocular color cameras are small, low-cost, and consume little energy. Traditional camera-based methods rely on handcrafted features from multiple views [3,4], while deep learning-based methods can predict depth from a single image. Traditional methods address this problem by exploiting the differences between multiple views within a nonlinear optimization framework [3,4]. Supervised methods that can generate high-quality estimates from a single image [8,9,10] require accurate ground truth and sensor cross-calibration for training. Alternatively, using view synthesis as a signal, self-supervised methods produce accurate depth maps from stereo image pairs [24,25] or monocular video clips [5,6,7].

[0003] However, the problem with the latter approach is that monocular vision inherently suffers from scale ambiguity. In addition, self-supervised methods introduce scale inconsistencies in the estimated depth and ego-motion between different video clips

[12] . This is because most existing monocular methods only utilize appearance-based losses that assume appearance consistency, which restricts training to small video subsequences without any long sequence constraints. Summary of the invention

[0004] Therefore, the purpose of this paper is to address the scale inconsistency problem and introduce scale awareness into monocular self-supervised depth and ego-motion estimation.

[0005] According to the method of the invention, the training of a neural network is performed according to one or more of the appended claims.

[0006] The essence of the invention is that complementary GPS coordinates are synchronized with the image during the training of the neural network and are used to compute a "GPS to scale loss" (G2S) to enhance scale consistency and / or scale perception for monocular self-supervised self-motion and depth estimation.

[0007] It has been found that optimal results may be achieved when the relative weight assigned to the GPS scale loss increases exponentially as training progresses.

[0008] Suitably, the depth and ego-motion prediction neural network is trained using an appearance-based photometric loss between real and synthetic target images and a smoothness loss on depth prediction. Preferably, a final loss function is calculated, said calculation comprising an appearance-based photometric loss and a smoothness loss, plus said GPS scale loss function multiplied by relative weights.

[0009] The accuracy of "GPS Loss to Scale" (G2S) can be improved by converting GPS coordinates (comprising latitude, longitude and optionally altitude) to local coordinates.

[0010] Suitably, the GPS scale loss is calculated using the ratio of the relative translation measured by GPS and the relative translation predicted by the network. By forming this loss on the magnitude of the translation rather than on individual translation components, any noise or systematic bias that may be present in the GPS measurements is taken into account

[16] .

[0011] In a preferred embodiment, the input to the neural network is a sequence of temporally consecutive image triplets {I -1 ,l0,l1}∈R H×W×3 and synchronized GPS coordinates {G -1 , G0, G1}∈R 3 .

[0012] Suitably, the central image of the image triplet is the target, the model is arranged to synthesize the target image from the first and last source images of the image triplet, and then compare the original central target image and the synthesized target image to train the network.

[0013] Preferably, the deep neural network learning model f D : R H×W×3 →R H×W To output dense depth or disparity for each pixel coordinate p of a single image.

[0014] Preferably, further, the ego-motion neural network also learns the model f E : R 2×H×W×3 →R 6 Output relative translation (t x , t y , t z ) and rotation (r x , r y , r z ), thus forming an affine transformation between a pair of overlapping images

[0015] Advantageously, the deep neural network and the ego-motion neural network run simultaneously.

[0016] Further suitably, the output of a deep neural network is densely packed with depth or disparity and ego-motion derived from the ego-motion neural network By converting the source image I s ∈{I -1 , I1} is distorted to the target image I t The projection models of ∈{I0} are linked together.

[0017] The present invention is also embodied in a path planning and navigation method in autonomous driving, wherein in order to improve scale consistency and / or scale perception of scene understanding, depth estimation is used for positioning according to the training method described herein.

[0018] In the following, the invention will be further elucidated with reference to the accompanying drawings related to the features of the method according to the invention, which drawings do not limit the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In the attached picture:

[0020] Figure 1 A network architecture using a dynamically weighted g2s loss according to the present invention is shown;

[0021] Figure 2 Box plots visualizing the mean and standard deviation of the scale factors used for dense depth and ego-motion estimation are shown;

[0022] Figure 3 Quantitative results of dense depth prediction for each image scale without post-processing are shown;

[0023] Figure 4 shows quantitative results for unscaled dense depth prediction; and

[0024] Figure 5 A quantitative comparison of ego-motion estimation on scaled and unscaled trajectories is shown. DETAILED DESCRIPTION

[0025] Figure 1 The network architecture using the proposed dynamically weighted g2s loss according to the present invention is shown. Given a set of n images from a video sequence and m loosely corresponding GPS coordinates, the input to the network is a series of temporally continuous (RGB) image triplets {I -1 , I0, I1}∈R H×W×3 and synchronized GPS coordinates {G -1 , G0, G1}∈R 3 .

[0026] Deep network learning model f D : R H×W×3 →R H×W to output a dense depth (or disparity) for each pixel coordinate p of a single image. At the same time, the ego-motion network learns a model f E : R 2×H×W×3 →R 6 Output relative translation (t x , t y , t z ) and rotation (r x , r y , r z ), thus forming an affine transformation between a pair of overlapping images

[0027] Given the camera intrinsic parameter K, the predicted depth and self-movement Linked together by a perspective projection model [7], the perspective projection model transforms the source image I s ∈{I -1 , I1} is distorted into the target image I t ∈{I0}. The network is then trained using a photometric loss based on the appearance between the real and synthetic target images and a smoothness loss on the depth prediction [6]. To this end, the proposed exponentially weighted g2s loss is added to the previously mentioned losses, which enforces scale consistency and / or scale awareness using the ratio of the measured and estimated translations.

[0028] It is worth noting that appearance-based losses provide a supervisory signal for short monocular subsequences. This can lead to scale inconsistencies in the predicted depth and self-motion in long videos. However, the signal provided by methods that solve this problem through 3D geometry-based losses depends on the camera settings and scene distribution [References 12, 13]. The GPS scale (g2s) loss introduced in the present invention provides an independent cross-modal signal, which leads to estimates of scale consistency and scale perception. GPS information, which is commonly present in videos, consists of the latitude, longitude and optionally altitude of the vehicle. First, these geodetic coordinates are converted to local coordinates using the Mercator projection, so that:

[0029]

[0030] y g =alt (2)

[0031]

[0032] where x g ,yg , z g are the XYZ coordinates in the local coordinate system, lat is the longitude and latitude of the earth's surface in the geodetic coordinate system, lon is radians, alt is the height, r e = 6378137m is taken as the radius of the earth. Since the GPS frequency may be different from the frame rate of the captured video, these local coordinates are also synchronized with the image using their respective timestamps through the following algorithm.

[0033]

[0034] (The original English corresponding to the relevant content of Algorithm 1 above is:

[0035]

[0036] Using the ratio of the relative distance measured by GPS to the relative distance predicted by the network, the additional loss can be derived, which is given by the following formula:

[0037]

[0038] Among them, L g2s is the loss function, G s→t is the relative distance measured by GPS, is the relative distance predicted by the network model, s∈{-1, 1} and t∈{0}. By forming this loss on the translation amplitude rather than on the individual translation components, any noise or systematic bias that may be present in the GPS measurements is taken into account

[16] . This loss according to equation (4) forces the ego-motion estimate to be closer to the common metric scale in the image triplet, thereby introducing scale consistency and scale awareness. This scale consistency and scale awareness is then also introduced in the depth estimate, which is associated with the ego-motion through a perspective projection model.

[0039] The network learns to synthesize more plausible views of the target image by improving its depth and self-motion predictions over the training period. It has been observed that training with stochastic gradient descent (SGD) and its variants biases the neural network towards learning simpler functions

[17] . Since the g2s loss (Equation 4) of the present invention is much simpler than complex appearance-based losses, severely penalizing incorrect scales in the network during early training interferes with learning individual translations, rotations, and pixel depths. Alternatively, in the present invention, dynamic weights of the g2s loss are applied exponentially to provide a scale signal that starts low and increases as training progresses. Therefore, the g2s loss L g2s The weight w is:

[0040] w = exp(epoch-epoch max ) (5)

[0041] Epoch is the number of training times. max is the maximum number of trainings. The final training loss is the sum of the appearance-based loss and the proposed exponentially weighted g2s loss:

[0042] L=L appearance +w*L g2s (6)

[0043] L is the loss function of the entire model, L appearance is the appearance loss function, which includes the photometric loss based on appearance and the smoothness loss for depth prediction. This L is the average value of each batch of images.

[0044] Figure 2 Box plots are provided to visualize the mean and standard deviation of the scaling factors used for dense depth and ego-motion estimation. Depth was estimated on a test set of eigen splitting

[11] . Ego-motion has been estimated on a test sequence of 10 of the odometry segmentation method [7]. Prior art methods scale the estimated depth and ego-motion using ground truth for evaluation. The present invention allows for consistent estimation of depth and ego-motion with metric scale.

[0045] Figure 3 Quantitative results are shown for per-image scaled dense depth prediction (without post-processing) on ​​KITTI Original

[14] and ground truth depth for the improved

[15] intrinsic split. The best result for each metric is in bold. The underlined one is the second best result. * indicates the result when training on Cityscapes together with KITTI.

[0046] Figure 4 Quantitative results are shown for unscaled dense depth prediction against eigen-split ground truth depth on KITTI Original

[14] . M and HR denote methods trained on monocular image sequences and high-resolution images, respectively. 'S' denotes stereo unsupervised methods, which scale the generated depth. 'pp' [6] denotes post-processing during inference. The best result for each metric is in bold. The second best result is crossed out. * indicates results when training on Cityscapes together with KITTI.

[0047] Figure 5A quantitative comparison of ego-motion estimation on scaled and unscaled trajectories based on the KITTI odometry segmentation method [7] is shown. Results include the mean and standard deviation for ATE-5. Results on ORB-SLAM based on multi-view geometry [4] have been provided for comparison.

[0048] Although the present invention has been discussed above with reference to exemplary embodiments of the method of the present invention, the present invention is not limited to the specific embodiments described, which may be varied in many ways without departing from the present invention. Therefore, the exemplary embodiments discussed should not be strictly used to interpret the appended claims accordingly. On the contrary, the embodiments are only intended to interpret the wording of the appended claims, and are not intended to limit the claims to the exemplary embodiments. Therefore, the scope of protection of the present invention should be interpreted only in accordance with the appended claims, wherein the exemplary embodiments should be used to resolve possible ambiguities in the wording of the claims.

[0049] References

[0050] [1] Ioan Andrei Barsan, Shenlong Wang, Andrei Pokrovsky, and Raquel Urtasun. Learning to localize using a lidar intensity map. In Conference on Robot Learning (CoRL), pages 605–616, 2018.

[0051] [2] Ji Zhang and Sanjiv Singh. Loam: Lidar odometry and mapping in real-time. In Robotics: Science and Systems, 2014.

[0052] [3] Kevin Karsch, Ce Liu, and Sing Bing Kang. Depth extraction from video using non-parametric sampling. In European Conference on Computer Vision, pages 775–788. Springer, 2012.

[0053] [4] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D. Tardos. Orb-slam: aversatile and accurate monocular slam system. IEEE transactions on robotics, 31(5): 1147–1163, 2015.

[0054] [5] Vincent Casser, Soeren Pirk, Reza Mahjourian, and Anelia Angelova. Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8001–8008, 2019.

[0055] [6] Clement Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow. Digging into self-supervised monocular depth estimation. In Proceedings of the IEEE international conference on computer vision, pages 3828–3838, 2019.

[0056] [7] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and egomotion from video, 2017.

[0057] [8] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2002–2011, 2018.

[0058] [9] Chengzhou Tang and Ping Tan. Ba-net: Dense bundle adjustment network. arXiv preprint arXiv:1806.04807, 2018.

[0059]

[10] Huizhong Zhou, Benjamin Ummenhofer, and Thomas Brox. Deeptam: Deep tracking and mapping. In Proceedings of the European conference on computer vision (ECCV), pages 822–838, 2018.

[0060]

[11] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Advances in neural information processing systems, pages 2366–2374, 2014.

[0061]

[12] Jiawang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, and Ian Reid. Unsupervised scale-consistent depth and ego-motion learning from monocular video. In Advances in Neural Information Processing Systems, pages 35–45, 2019.

[0062]

[13] Reza Mahjourian, Martin Wicke, and Anelia Angelova. Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5667–5675, 2018.

[0063]

[14] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The KITTI dataset. The International Journal of Robotics Research, 32(11): 1231–1237, 2013.

[0064]

[15] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger. Sparsity invariant cnns. In International Conference on 3DVision (3DV), 2017.

[0065]

[16] Anweshan Das and Gijs Dubbelman. An experimental study on relative and absolute posegraph fusion for vehicle localization. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 630–635. IEEE, 2018.

[0066]

[17] Robert Geirhos, Jorn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix AWichmann. Shortcut learning in deep neural networks. arXiv preprint arXiv:2004.07780, 2020.

[0067]

[18] Hemang Chawla, Matti Jukola, Terence Brouns, Elahe Arani, and Bahram Zonooz. Crowdsourced 3d mapping: A combined multi-viewgeometry and self-supervised learning approach. In 2020 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.

[0068]

[19] Ariel Gordon, Hanhan Li, Rico Jonschkowski, and Anelia Angelova. Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras, 2019.

[0069]

[20] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The KITTI dataset. The International Journal of Robotics Research, 32(11): 1231–1237, 2013.

[0070]

[21] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger. Sparsity invariant CNNs. In International Conference on 3D Vision (3DV), 2017.

[0071]

[22] Ashutosh Saxena, Min Sun, and Andrew Y Ng. Make3d: Learning 3d scene structure from a single still image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(5): 824–840, 2008.

[0072] IEEE transactions on pattern analysis and machine intelligence, 31(5):824–840, 2008).

[0073]

[23] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.

[0074]

[24] Ravi Garg, Vijay Kumar BG, Gustavo Carneiro, and Ian Reid. Unsupervised CNN for single view depth estimation: Geometry to the rescue. In European Conference on Computer Vision, pages 740–756. Springer, 2016.

[0075]

[25] Clement Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estimation with left-right consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 270–279, 2017.

[0076]

[26] Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon. 3d packing for self-supervised monocular depth estimation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 2485–2494, 2020.

[0077]

[27] Anweshan Das and Gijs Dubbelman. An experimental study on relative and absolute pose graph fusion for vehicle localization. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 630–635. IEEE, 2018.

[0078]

[28] Jason Jo and Yoshua Bengio. Measuring the tendency of cnns to learn surface statistical regularities. ArXiv preprint arXiv:1711.11561, 2017.

[0079]

[29] Robert Geirhos, Jorn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. arXiv preprint arXiv:2004.07780, 2020.

[0080]

[30] Zhichao Yin and Jianping Shi. Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1983–1992, 2018.

[0081]

[31] Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black. Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 12240–12249, 2019.

[0082]

[32] Marvin Klingner, Jan-Aike Termohlen, Jonas Mikolajczyk, and Tim Fingscheidt. Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance. In ECCV, 2020.

[0083]

[33] Sudeep Pillai, Rares Ambrus, and Adrien Gaidon. Superdepth: Self-supervised, super-resolved monocular depth estimation. In 2019 International Conference on Robotics and Automation (ICRA), pages 9250–9256. IEEE, 2019.

[0084]

[34] Matan Goldman, Tal Hassner, and Shai Avidan. Learn stereo, infer mono: Siamese networks for self-supervised, monocular, depth estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019.

Claims

1. A method for improving scale consistency and / or scale perception in a model of a self-supervised deep neural network and an ego-motion prediction neural network for processing a video stream of a monocular image, characterized in that Complementary GPS coordinates synchronized with the monocular image are used to compute a GPS scale loss to enforce scale consistency and / or scale awareness in monocular self-supervised ego-motion and depth estimation; Wherein, based on the complementary GPS coordinates synchronized with the monocular image, the GPS scale loss is obtained by the following formula: Among them, L g2s is the GPS scale loss function, G s→t is the relative distance measured by GPS, is the relative distance predicted by the model of the ego-motion prediction neural network, s∈{-1,1} and t∈{0}; The output of the self-supervised deep neural network is densely packed with or disparity with the ego-motion derived from the ego-motion prediction neural network Linked together by a projection model that transforms the source image I s ∈{I -1 , I1} is distorted to the target image I t ∈{I0}, where the central image of the input image triplet is the real target image, and the target image I is distorted by the projection model. t ∈{I0} is a synthetic target image, the real target image is compared with the synthetic target image, and the self-supervised deep neural network and the self-motion prediction neural network are trained using the appearance-based photometric loss and the smoothness loss of the depth prediction between the real target image and the synthetic target image; Then, the appearance-based photometric loss and the smoothness loss are added to the GPS scale loss function multiplied by a relative weight to calculate a final loss function, thereby enhancing scale consistency and / or scale perception using a ratio between the measured translation and the estimated translation.

2. The method according to claim 1, characterized in that As training progresses, the relative weight assigned to the GPS scale loss increases exponentially.

3. The method according to claim 1, characterized in that The complementary GPS coordinates comprising latitude, longitude and optionally altitude are converted into local coordinates.

4. The method according to any one of claims 1 to 3, characterized in that The calculation of the GPS scale loss utilizes the ratio of the relative translation measured by GPS to the relative translation predicted by the network.

5. The method according to claim 1, characterized in that The input of the model of the self-supervised deep neural network and the self-motion prediction neural network for processing the video stream of the monocular image is a series of time-continuous image triplets {I -1 , I0, I1}∈R H×W×3 and synchronized GPS coordinates {G -1 , G0, G1}∈R 3 .

6. The method according to claim 1, characterized in that The self-supervised deep neural network learning model f D : R H ×W×3 →R H×W To output dense depth or disparity for each pixel coordinate p of a single image.

7. The method according to claim 1, characterized in that The self-motion prediction neural network learning model f E : R 2 ×H×W×3 →R 6 Output relative translation (t x , t y , t z ) and rotation (r x , r y , r z ), thus forming an affine transformation between a pair of overlapping images 8. The method according to claim 6 or 7, characterized in that: The self-supervised deep neural network and the ego-motion prediction neural network run simultaneously.

9. A method for planning and navigating in autonomous driving, wherein: In order to improve scale consistency and / or scale perception for scene understanding, a depth estimation model trained according to the method of claim 1 is used for localization.

Images