A self-supervised multi-camera depth prediction method with structural consistency
By optimizing the depth basis weights and feature projections in a multi-camera system and using a deep learning network for iterative optimization, the problem of inconsistency in depth prediction across overlapping areas of cameras is solved, achieving efficient and accurate depth map reconstruction.
Patent Information
- Application Number
- CN202411291373.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2044-09-14
AI Technical Summary
Existing self-supervised depth estimation methods in small overlapping areas of multi-camera systems cannot ensure the consistency of depth prediction across camera overlapping areas, resulting in decreased depth prediction accuracy and erroneous 3D structure interpretation.
By inputting RGB images of adjacent perspectives into the depth estimation network, optimizing the depth basis weights, and projecting the feature images to the same perspective, iterative optimization is performed using deep learning convolutional neural networks and weight update networks to achieve structural consistency prediction of the depth map.
It improves the structural consistency and accuracy of depth prediction, reduces the number of calculation parameters, improves computational efficiency, and reconstructs accurate geometric shapes in multi-camera systems.
Smart Images
Figure CN119228862B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a self-supervised multi-camera depth prediction method with structural consistency, and belongs to the technical field of image processing. Background Art
[0002] Modern autonomous vehicles, such as those produced by Tesla, are equipped with multiple cameras to capture a full 360° panoramic view. This capability has given rise to a new depth estimation method that utilizes a multi-camera setup to exploit small cross-camera overlaps. Unlike traditional methods that rely on binocular cameras that require binocular correction or extensive overlapping images, this innovative method can exploit minimal overlap (as low as 10%) between cameras at any position. Currently, several existing works are exploring self-supervised depth estimation under this innovative framework. Guizilini et al. proposed the first method, called omnidirectional monocular depth, which utilizes temporal context and spatiotemporal photometric constraints across cameras to increase the overlap between cameras. By leveraging known external parameters between cameras, omnidirectional monocular depth imposes pose consistency constraints to ensure that all cameras follow the same rigid body motion.
[0003] However, these existing methods typically estimate depth for each view independently, failing to ensure consistent depth prediction across overlapping camera regions. While omnidirectional monocular depth estimation can recover depth scale by leveraging photometric errors in adjacent views, experimental analysis shows that photometric errors are significantly affected by lighting conditions and cannot accurately reconstruct geometry. In multi-camera systems, this inconsistency leads to degraded depth prediction accuracy, incorrect 3D structure interpretation, and challenges in downstream tasks. Summary of the Invention
[0004] The present invention aims to solve the problem of self-supervised depth estimation in small overlapping areas in a multi-camera system in the prior art, and further proposes a self-supervised multi-camera depth prediction method with structural consistency.
[0005] The technical solution adopted by the present invention to solve the above problems is: the present invention proposes a self-supervised multi-camera depth prediction method with structural consistency, comprising:
[0006] Step 1: Input the RGB images of adjacent perspectives into the depth estimation network to obtain the depth basis of the corresponding RGB images;
[0007] Step 2: Optimize the depth basis weights of the RGB image using a deep learning-based convolutional neural network;
[0008] Step 3: Linearly sum the weights of the optimized RGB image weight basis and the depth basis to obtain the initial depth map;
[0009] Step 4: Based on the initial depth map, the features of the feature images of the adjacent view RGB images are projected to the same view to obtain the merged projected features;
[0010] Step 5: Input the merged projected features into the weight update network for optimization to obtain the updated projected feature weights;
[0011] Step 6: Perform linear sum calculation on the updated projected feature weights and depth basis to obtain the depth map after optimization.
[0012] Step 7: Repeat steps 1 to 6 to iteratively optimize the depth map until the depth basis weights converge to obtain the self-supervised multi-camera collaborative depth prediction result.
[0013] Optionally, optimizing the depth basis weight of the RGB image in step 2 specifically includes:
[0014] The depth-based weights of the RGB images are optimized using the overlapping views between the target camera and neighboring cameras.
[0015] Optionally, the step of projecting the features of the feature images of the adjacent viewing angle RGB images to the same viewing angle in step 4 includes:
[0016] Step 4.1: Obtain the external calibration parameters and internal parameters of the target camera and adjacent cameras. The external calibration parameters include the six degrees of freedom parameters of camera rotation and translation, and the internal parameters include focal length and distance from the optical center.
[0017] Step 4.2: Combine the external calibration parameters and internal parameters of the target camera and the adjacent cameras to project the features of the adjacent images to the same viewing angle to obtain the merged projected features.
[0018] Optionally, the step of obtaining the merged projection features in step 4.2 includes:
[0019] Step 4.2.1: Combine the internal parameters and the estimated depth of the depth estimation network to obtain the pixel 3D projection operation between the target camera and the adjacent cameras;
[0020] Step 4.2.2: Obtain multi-scale feature maps of the target camera and adjacent cameras based on the encoder in the depth estimation network, align the multi-scale feature maps to the same size using the interpolation algorithm, and connect them along the channel;
[0021] Step 4.2.3: Based on the pixel 3D projection operation between the target camera and the adjacent camera, the multi-scale features of the target camera are warped to the adjacent camera to obtain the warped features, and the overlapping area between the target camera multi-scale feature map and the adjacent camera multi-scale feature map in the adjacent camera is filled with zeros;
[0022] Step 4.2.4: Connect the distorted features along the channel to obtain the merged projection features.
[0023] Optionally, the step of obtaining the updated projected feature weights in step 5 includes:
[0024] Step 5.1: Merge the features of the target camera and the adjacent cameras in the channel layer of the weight update network;
[0025] Step 5.2: Input the combined projected features into the gated recurrent unit in the weight update network;
[0026] Step 5.3: Feed the output of the gated recurrent unit into a 3×3 convolutional layer for convolution. The output of the convolutional layer is fed into a pooling layer, where the input is scaled to n×1 dimensions. The output of the pooling layer is then fed into a fully connected layer.
[0027] Step 5.4: The output of the fully connected layer is calculated through the Sigmoid function to optimize the projected feature weights and obtain the updated projected feature weights.
[0028] The beneficial effects of the present invention are:
[0029] 1. The self-supervised multi-camera collaborative depth prediction method provided by the present invention can utilize the small overlapping information between cameras to achieve structural consistency in depth prediction.
[0030] 2. The present invention can improve the efficiency of depth refinement through a cyclic refinement strategy. The weight network provided by the present invention can reduce the number of parameters while maintaining the depth prediction accuracy. At the same time, a deep learning gating network is added to realize multiple iterative calculations, which can effectively improve the computing efficiency without losing accuracy.
[0031] 3. This paper can make more reasonable use of overlapping images and use the information of adjacent cameras to improve the accuracy of overall depth estimation in the inference stage. The performance improvement only requires a small number of additional parameters. Inspired by CodeSlam, dense depth maps can be adjusted with a small number of basic states. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 A flowchart of a self-supervised multi-camera depth prediction method with structural consistency provided by the present invention;
[0033] Figure 2 This is a schematic diagram of the autonomous driving dataset provided by the present invention and the overlapping area of each camera;
[0034] Figure 3 A flow chart of the weight update network provided by the present invention;
[0035] Figure 4 This is the self-supervised multi-camera depth prediction result diagram provided by the present invention. DETAILED DESCRIPTION
[0036] Combine Figure 1-4 This embodiment is described as follows. Figure 1 As shown, the steps of a self-supervised multi-camera depth prediction method with structural consistency described in this embodiment include:
[0037] S1: Input the RGB images of adjacent perspectives into the depth estimation network to obtain the depth basis of the corresponding RGB images;
[0038] This embodiment defines a depth basis as a possible probability distribution of the true depth within a 3D scene. Each depth basis acts like a template, representing a possible distribution state of depth. The predicted depth map is calculated by combining multiple bases and corresponding weights. This embodiment sets multiple initial weights and uses the depth basis to calculate the initial depth. Multiple depth bases have greater flexibility in representing depth, which provides room for optimization. This embodiment can adjust the weights of the bases to optimize the combination and improve the quality of the depth map.
[0039] As an optimization problem, another advantage of this implementation is that it avoids recalculating the depth map pixel by pixel, which requires a large amount of computation, requiring approximately 3,000 pixel parameters. This implementation only needs to optimize the weights of each depth basis, a total of 32 variables, significantly reducing the number of parameters and computational complexity. When multiple cameras are available, the overlapping views between the target camera and adjacent cameras can be exploited to further optimize the depth basis weights.
[0040] S2: Optimize the depth basis weight of the RGB image, and linearly sum the optimized weight basis weight of the RGB image and the depth basis to obtain the initial depth map;
[0041] The implementation scheme designs a recursive optimization strategy that uses a deep learning-based convolutional neural network to optimize the weights of the depth basis multiple times. The goal of this implementation scheme is to utilize richer feature information from adjacent cameras to optimize the weights of the depth basis, such as Figure 2 As shown, this embodiment assumes that in a multi-camera setting, the cameras are rigidly connected, that is, the spatial relationship between the cameras does not change during movement. Take the i-th camera and the j-th camera as an example. From the data set, this embodiment can obtain external calibration parameters including the camera, that is, the six degrees of freedom parameters of rotation and translation. Combined with the internal parameter matrix of the camera model and the estimated depth, this embodiment can obtain the pixel 3D projection operation between the i-th camera and the j-th camera. The camera internal parameters mainly include focal length and distance from the optical center, and these projection operations follow the standard pinhole camera model.
[0042] The self-supervised multi-camera collaborative depth prediction method provided by the present invention can utilize small overlapping information between cameras to achieve structural consistency in depth prediction.
[0043] S3: Based on the initial depth map, the features of the feature images of the adjacent view RGB images are projected to the same view to obtain the merged projected features;
[0044] To utilize more useful information in the overlapping region, this embodiment projects the feature map of the reference view onto the target view. This embodiment first obtains a multi-scale feature map in the encoder of the deep network. This embodiment uses an interpolation algorithm to align the multi-scale features to the same size and connects them along the channels. This combines internal parameters with the estimated depth of the depth estimation network to obtain a pixel-wise 3D projection operation between the target camera and the adjacent camera. Based on this pixel-wise 3D projection operation between the cameras, this embodiment warps the multi-scale features of camera j onto camera i, obtaining the warped features in camera i's view. Non-overlapping regions of the image are padded with zeros.
[0045] This implementation connects the warped features along the channels to generate projected features F. For camera view i, projected features F not only include all features from view i but also multi-scale features from the overlapping region within camera view j. Leveraging these more informative projected features F, this implementation proposes optimizing the depth basis combination to generate higher-quality depth maps.
[0046] S4: As Figure 3 As shown, the combined projected features are input into the weight update network for optimization to obtain the updated projected feature weights;
[0047] S401: Merging the features of the target camera and the adjacent cameras in the channel layer of the weight update network;
[0048] S402: Inputting the combined projection features into the gated recurrent unit in the weight update network;
[0049] S403: Send the output of the gated recurrent unit to a 3×3 convolutional layer for convolution. The output of the convolutional layer is input to a pooling layer. The input of the pooling layer is expanded into an n×1 dimension. The output of the pooling layer is input to a fully connected layer.
[0050] S404: The output of the fully connected layer is calculated by the Sigmoid function to optimize the projected feature weights and obtain the updated projected feature weights.
[0051] S5: Perform linear sum calculation on the updated projected feature weights and the depth basis to obtain the depth map after optimization.
[0052] S6: Repeat S1-S5 and iteratively optimize the depth map until the depth basis weights converge to obtain the self-supervised multi-camera collaborative depth prediction result.
[0053] The present invention can improve the depth refinement efficiency through a cyclic refinement strategy, and the weight update network provided by the present invention can reduce the number of parameters while maintaining the depth prediction accuracy.
[0054] like Figure 4 As shown, this embodiment optimizes the depth map to ensure consistent depth prediction across camera overlapping areas, can capture a complete 360° panoramic view, and the prediction results are not affected by light intensity, and can reconstruct accurate geometric shapes in a multi-camera system.
[0055] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the present profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical content disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement of the above embodiments made according to the technical essence of the present invention, within the spirit and principles of the present invention, without departing from the content of the technical solution of the present invention, shall still fall within the scope of protection of the technical solution of the present invention.
Claims
1. A self-supervised multi-camera depth prediction method with structural consistency, characterized by: The steps of the self-supervised multi-camera depth prediction method with structural consistency include: Step 1: Input the RGB images of adjacent perspectives into the depth estimation network to obtain the depth basis of the corresponding RGB images; Step 2: Optimize the depth basis weights of the RGB image using a deep learning-based convolutional neural network; Step 3: Linearly sum the depth basis weight and depth basis after RGB image optimization to obtain the initial depth map; Step 4: Based on the initial depth map, the features of the feature images of the adjacent view RGB images are projected to the same view to obtain the merged projected features; Step 5: Input the merged projected features into the weight update network for optimization to obtain updated projected feature weights; Step 6: Perform linear sum calculation on the updated projected feature weights and depth basis to obtain the depth map after optimization. Step 7: Repeat steps 1 to 6 to iteratively optimize the depth map until the depth basis weights converge to obtain the self-supervised multi-camera collaborative depth prediction result.
2. The self-supervised multi-camera depth prediction method with structural consistency according to claim 1, characterized in that The optimization of the depth basis weights of the RGB image in step 2 specifically includes: The depth-based weights of the RGB images are optimized using the overlapping views between the target camera and neighboring cameras.
3. The self-supervised multi-camera depth prediction method with structural consistency according to claim 1, characterized in that The step of projecting the features of the feature images of the adjacent view RGB images to the same view in step 4 includes: Step 4.1: Obtain the external calibration parameters and internal parameters of the target camera and adjacent cameras. The external calibration parameters include the six degrees of freedom parameters of camera rotation and translation, and the internal parameters include focal length and distance from the optical center; Step 4.2: Project the features of the adjacent images to the same viewing angle by combining the external calibration parameters and internal parameters of the target camera and the adjacent cameras to obtain merged projected features.
4. The self-supervised multi-camera depth prediction method with structural consistency according to claim 3, characterized in that The steps for obtaining the merged projection features in step 4.2 include: Step 4.2.1: Combine the internal parameters and the estimated depth of the depth estimation network to obtain the pixel 3D projection operation between the target camera and the adjacent cameras; Step 4.2.2: Obtain multi-scale feature maps of the target camera and adjacent cameras based on the encoder in the depth estimation network, align the multi-scale feature maps to the same size using a difference algorithm, and connect them along the channel; Step 4.2.3: Based on the pixel 3D projection operation between the target camera and the adjacent camera, the multi-scale features of the target camera are warped to the adjacent camera to obtain a warped feature, and the overlapping area of the multi-scale feature map of the target camera and the multi-scale feature map of the adjacent camera in the adjacent camera is filled with zeros; Step 4.2.4: Connect the distorted features along the channel to obtain the merged projection features.
5. The self-supervised multi-camera depth prediction method with structural consistency according to claim 1, characterized in that The steps for obtaining the updated projected feature weights in step 5 include: Step 5.1: Merge the features of the target camera and the adjacent cameras in the channel layer of the weight update network; Step 5.2: Input the merged projection features into the gated recurrent unit in the weight update network; Step 5.3: Feed the output of the gated recurrent unit into a 3×3 convolutional layer for convolution. The output of the convolutional layer is fed into a pooling layer, where the input is scaled to n×1 dimensions. The output of the pooling layer is then fed into a fully connected layer. Step 5.4: The output of the fully connected layer is calculated through the Sigmoid function to optimize the projected feature weights and obtain the updated projected feature weights.