Self-supervised multi-camera depth prediction method with structural consistency

Abstract

The invention provides a self-supervised multi-camera depth prediction method with structural consistency, and the method comprises the steps: S1, inputting RGB images of adjacent visual angles into a depth estimation network, and obtaining depth bases of the corresponding RGB images; s2, optimizing the depth base weight of the RGB image, and carrying out linear summation on the depth base weight and the depth base to obtain an initial depth map; s3, on the basis of the initial depth map, projecting features of the feature images of the RGB images of the adjacent visual angles to the same visual angle, and obtaining combined projection features; s4, inputting the combined projection features into a weight updating network for optimization; s5, performing linear summation calculation on the updated projected feature weight and the depth base to obtain a depth map after primary optimization; and S6, repeating the steps S1 to S5, carrying out iterative optimization on the depth map until the depth base weight is converged, and obtaining a self-supervised multi-camera collaborative depth prediction result. According to the invention, the structure consistency of depth prediction can be realized by using small overlapping information between cameras.

Description

Technical Field

[0001] The invention relates to a self-supervised multi-camera depth prediction method with structural consistency, and belongs to the technical field of image processing. Background Art

[0002] Modern autonomous vehicles, such as those produced by Tesla, are equipped with multiple cameras to capture a full 360° panoramic view. This capability has given rise to a new depth estimation method that leverages a multi-camera setup to exploit tiny cross-camera overlaps. Unlike traditional methods that rely on stereo cameras and require either stereo calibration or extensive overlapping images, this innovative approach can exploit minimal overlap (as low as 10%) between cameras at arbitrary locations. Several existing works are currently exploring self-supervised depth estimation in this innovative framework. The first method, called omnidirectional monocular depth, was proposed by Guizilini et al., which leverages temporal context and spatiotemporal photometric constraints across cameras to increase the overlap between cameras. By leveraging known extrinsic parameters between cameras, omnidirectional monocular depth imposes pose consistency constraints to ensure that all cameras follow the same rigid body motion.

[0003] However, these existing methods usually estimate the depth of each view independently and fail to ensure consistent depth prediction across camera overlap regions, as shown. While omnidirectional monocular depth estimation can recover the scale of depth by leveraging photometric errors in adjacent views, experimental analysis shows that photometric errors are greatly affected by lighting conditions and cannot reconstruct accurate geometry. In multi-camera systems, this inconsistency leads to decreased depth prediction accuracy, incorrect 3D structure interpretation, and challenges in downstream tasks. Summary of the invention

[0004] The present invention aims to solve the problem of self-supervised depth estimation in small overlapping areas in a multi-camera system in the prior art, and further proposes a self-supervised multi-camera depth prediction method with structural consistency.

[0005] The technical solution adopted by the present invention to solve the above problems is: the present invention proposes a self-supervised multi-camera depth prediction method with structural consistency, comprising:

[0006] Step 1: Input the RGB images of adjacent perspectives into the depth estimation network to obtain the depth basis of the corresponding RGB images;

[0007] Step 2: Optimize the depth basis weights of the RGB image using a deep learning-based convolutional neural network;

[0008] Step 3: Linearly sum the weights of the optimized weight basis and depth basis of the RGB image to obtain the initial depth map;

[0009] Step 4: Based on the initial depth map, the features of the feature images of the adjacent perspective RGB images are projected to the same perspective to obtain the merged projection features;

[0010] Step 5: Input the merged projected features into the weight update network for optimization to obtain the updated projected feature weights;

[0011] Step 6: Perform linear sum calculation on the updated projected feature weights and depth basis to obtain the depth map after optimization.

[0012] Step 7: Repeat steps 1 to 6 to iteratively optimize the depth map until the depth basis weights converge to obtain the self-supervised multi-camera collaborative depth prediction result.

[0013] Optionally, optimizing the depth basis weight of the RGB image in step 2 specifically includes:

[0014] The depth-based weights of the RGB images are optimized using the overlapping views between the target camera and neighboring cameras.

[0015] Optionally, in step 4, the step of projecting features of feature images of adjacent viewing angle RGB images to the same viewing angle includes:

[0016] Step 4.1: Obtain the external calibration parameters and internal parameters of the target camera and adjacent cameras. The external calibration parameters include the six degrees of freedom parameters of camera rotation and translation, and the internal parameters include focal length and distance from the optical center.

[0017] Step 4.2: Combine the external calibration parameters and internal parameters of the target camera and the adjacent cameras to project the features of the adjacent images to the same viewing angle to obtain the merged projection features.

[0018] Optionally, the step of obtaining the merged projection features in step 4.2 includes:

[0019] Step 4.2.1: Combine the internal parameters and the estimated depth of the depth estimation network to obtain the pixel 3D projection operation between the target camera and the adjacent cameras;

[0020] Step 4.2.2: Obtain multi-scale feature maps of the target camera and adjacent cameras based on the encoder in the depth estimation network, align the multi-scale feature maps to the same size using the difference algorithm and connect them along the channel;

[0021] Step 4.2.3: Based on the pixel 3D projection operation between the target camera and the adjacent camera, the multi-scale features of the target camera are distorted into the adjacent camera to obtain the distorted features, and the overlapping area of ​​the multi-scale feature map of the target camera and the multi-scale feature map of the adjacent camera in the adjacent camera is filled with 0;

[0022] Step 4.2.4: Connect the distorted features along the channels to obtain the merged projection features.

[0023] Optionally, the step of obtaining the updated projected feature weights in step 5 includes:

[0024] Step 5.1: Merge the features of the target camera and the adjacent cameras in the channel layer of the weight update network;

[0025] Step 5.2: Input the merged projection features into the gated recurrent unit in the weight update network;

[0026] Step 5.3: Send the output of the gated recurrent unit to a 3×3 convolutional layer for convolution, and input the output of the convolutional layer to the pooling layer. The input of the pooling layer is pulled into an n×1 dimension, and the output of the pooling layer is input to the fully connected layer.

[0027] Step 5.4: The output of the fully connected layer is calculated by the Sigmoid function to optimize the projection feature weights and obtain the updated projected feature weights.

[0028] The beneficial effects of the present invention are:

[0029] 1. The self-supervised multi-camera collaborative depth prediction method provided by the present invention can utilize the small overlapping information between cameras to achieve structural consistency of depth prediction.

[0030] 2. The present invention can improve the efficiency of deep refinement through a cyclic refinement strategy. The weight network provided by the present invention can reduce the number of parameters while maintaining the depth prediction accuracy. At the same time, a deep learning gating network is added to realize multiple iterative calculations, which can effectively improve the computing efficiency without losing accuracy.

[0031] 3. The present invention can make more reasonable use of overlapping images and use the information of adjacent cameras in the inference stage to improve the accuracy of overall depth estimation. The performance improvement only requires a small number of additional parameters. Inspired by CodeSlam, dense depth maps can be adjusted by a small number of basic states. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a self-supervised multi-camera depth prediction method with structural consistency provided by the present invention;

[0033] Figure 2 A schematic diagram of the autonomous driving data set and the overlapping area of ​​each camera provided by the present invention;

[0034] Figure 3 A flow chart of a weight updating network provided by the present invention;

[0035] Figure 4 This is the self-supervised multi-camera depth prediction result diagram provided by the present invention. DETAILED DESCRIPTION

[0036] Combination Figure 1-4 This embodiment is described as follows. Figure 1 As shown, the steps of a self-supervised multi-camera depth prediction method with structural consistency described in this embodiment include:

[0037] S1: Input the RGB images of adjacent perspectives into the depth estimation network to obtain the depth basis of the corresponding RGB images;

[0038] This embodiment defines a depth basis as a possible probability distribution of the real depth in a 3D scene. Each depth basis is like a template, representing a possible distribution state of the depth. The predicted depth map is calculated by combining multiple bases and corresponding weights. This embodiment sets multiple initial weights and uses the depth basis to calculate the initial depth. Multiple depth bases have greater flexibility to represent depth, which provides room for optimization. This embodiment can adjust the weights of the basis to optimize the combination and improve the quality of the depth map.

[0039] As an optimization problem, another advantage is that this implementation avoids recalculating the depth map pixel by pixel, which requires a lot of computation, about 3000 pixel parameters. This implementation only needs to optimize the weights of each depth basis, a total of 32 variables, greatly reducing the number of parameters and computational complexity. When multiple cameras are available, the overlapping views between the target camera and the adjacent cameras can be used to further optimize the weights of the depth basis.

[0040] S2: Optimize the depth base weight of the RGB image, and linearly sum the optimized weight base weight of the RGB image and the depth base to obtain an initial depth map;

[0041] The embodiment designs a recursive optimization strategy to optimize the weights of the depth basis multiple times using a deep learning-based convolutional neural network. The goal of this embodiment is to optimize the weights of the depth basis using richer feature information from adjacent cameras, such as Figure 2 As shown, this embodiment assumes that in a multi-camera setting, the cameras are rigidly connected, that is, the spatial relationship between the cameras does not change during movement. Take the i-th camera and the j-th camera as an example. From the data set, this embodiment can obtain external calibration parameters including the cameras, that is, the six degrees of freedom parameters of rotation and translation. Combined with the internal parameter matrix of the camera model and the estimated depth, this embodiment can obtain the pixel 3D projection operation between the i-th camera and the j-th camera. The camera internal parameters mainly include focal length and distance from the optical center, and these projection operations follow the standard pinhole camera model.

[0042] The self-supervised multi-camera collaborative depth prediction method provided by the present invention can utilize small overlapping information between cameras to achieve structural consistency in depth prediction.

[0043] S3: Based on the initial depth map, the features of the feature images of the adjacent perspective RGB images are projected to the same perspective to obtain the merged projection features;

[0044] In order to utilize more useful information in the overlapping area, this embodiment projects the feature map of the reference view to the target view. This embodiment first obtains a multi-scale feature map in the encoder of the deep network. This embodiment uses an interpolation algorithm to align the multi-scale features to the same size and connect them along the channel. Combined with the internal parameters and the estimated depth of the depth estimation network, the pixel 3D projection operation between the target camera and the adjacent camera is obtained. Based on the pixel 3D projection operation between the cameras, this embodiment warps the multi-scale features of camera j to camera i, and obtains the warped features in the view of camera i. The non-overlapping areas of the image are filled with 0.

[0045] This embodiment connects the warped features along the channel to obtain the projection feature F. In the view of camera i, the projection feature F not only contains all the features of the i-view image, but also contains the multi-scale features of the overlapping area in the j-view camera. Using these more informative projection features F, this embodiment proposes to optimize the depth basis combination to generate a higher quality depth map.

[0046] S4: Figure 3 As shown, the combined projected features are input into the weight update network for optimization to obtain the updated projected feature weights;

[0047] S401: Merging the features of the target camera and the adjacent cameras at the channel layer of the weight update network;

[0048] S402: Inputting the combined projection features into the gated recurrent unit in the weight update network;

[0049] S403: Send the output of the gated recurrent unit to a 3×3 convolutional layer for convolution, input the output of the convolutional layer to a pooling layer, the input of the pooling layer is pulled into an n×1 dimension, and the output of the pooling layer is input to a fully connected layer;

[0050] S404: The output of the fully connected layer is calculated by the Sigmoid function to optimize the projection feature weights and obtain the updated projected feature weights.

[0051] S5: Perform linear sum calculation on the updated projected feature weights and the depth basis to obtain a depth map after optimization;

[0052] S6: Repeat S1-S5 to iteratively optimize the depth map until the depth basis weights converge to obtain the self-supervised multi-camera collaborative depth prediction result.

[0053] The present invention can improve the depth refinement efficiency through a cyclic refinement strategy, and the weight update network provided by the present invention can reduce the number of parameters while maintaining the depth prediction accuracy.

[0054] like Figure 4 As shown, this embodiment optimizes the depth map to ensure consistent depth prediction across camera overlapping areas, is able to capture a complete 360° panoramic view, and the prediction results are not affected by light intensity, so accurate geometric shapes can be reconstructed in a multi-camera system.

[0055] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A self-supervised multi-camera depth prediction method with structural consistency, characterized in that The steps of the self-supervised multi-camera depth prediction method with structural consistency include: Step 1: Input the RGB images of adjacent perspectives into the depth estimation network to obtain the depth basis of the corresponding RGB images; Step 2: Optimize the depth basis weights of the RGB image using a deep learning-based convolutional neural network; Step 3: Linearly sum the weights of the optimized weight basis and depth basis of the RGB image to obtain the initial depth map; Step 4: Based on the initial depth map, the features of the feature images of the adjacent perspective RGB images are projected to the same perspective to obtain the merged projection features; Step 5: Input the merged projected features into a weight update network for optimization to obtain updated projected feature weights; Step 6: Perform linear sum calculation on the updated projected feature weights and depth basis to obtain the depth map after optimization. Step 7: Repeat steps 1 to 6 to iteratively optimize the depth map until the depth basis weights converge to obtain the self-supervised multi-camera collaborative depth prediction result.

2. The method for self-supervised multi-camera depth prediction with structural consistency according to claim 1, characterized in that: The optimization of the depth basis weights of the RGB image in step 2 specifically includes: The depth-based weights of the RGB images are optimized using the overlapping views between the target camera and neighboring cameras.

3. The method for self-supervised multi-camera depth prediction with structural consistency according to claim 1, characterized in that: The step of projecting the features of the feature images of the adjacent viewing angle RGB images to the same viewing angle in step 4 includes: Step 4.1: Obtain external calibration parameters and internal parameters of the target camera and adjacent cameras, wherein the external calibration parameters include six degrees of freedom parameters of camera rotation and translation, and the internal parameters include focal length and distance from the optical center; Step 4.2: Combining the external calibration parameters and internal parameters of the target camera and the adjacent cameras, the features of the adjacent images are projected to the same viewing angle to obtain the merged projection features.

4. The method for self-supervised multi-camera depth prediction with structural consistency according to claim 3, characterized in that: The steps of obtaining the merged projection features in step 4.2 include: Step 4.2.1: Combine the internal parameters and the estimated depth of the depth estimation network to obtain the pixel 3D projection operation between the target camera and the adjacent cameras; Step 4.2.2: Based on the encoder in the depth estimation network, obtain the multi-scale feature maps of the target camera and the adjacent cameras, and use the difference algorithm to align the multi-scale feature maps to the same size and connect them along the channel; Step 4.2.3: Based on the pixel 3D projection operation between the target camera and the adjacent camera, the multi-scale features of the target camera are distorted into the adjacent camera to obtain distorted features, and the overlapping area of ​​the multi-scale feature map of the target camera and the multi-scale feature map of the adjacent camera in the adjacent camera is filled with 0; Step 4.2.4: Connect the distorted features along the channels to obtain the merged projection features.

5. The method for self-supervised multi-camera depth prediction with structural consistency according to claim 1, characterized in that: The steps for obtaining the updated projected feature weights in step 5 include: Step 5.1: Merge the features of the target camera and the adjacent cameras in the channel layer of the weight update network; Step 5.2: Input the merged projection features into the gated recurrent unit in the weight update network; Step 5.3: Send the output of the gated recurrent unit to a 3×3 convolutional layer for convolution, and input the output of the convolutional layer to the pooling layer. The input of the pooling layer is pulled into an n×1 dimension, and the output of the pooling layer is input to the fully connected layer. Step 5.4: The output of the fully connected layer is calculated by the Sigmoid function to optimize the projection feature weights and obtain the updated projected feature weights.

Images