A Depth Completion Method for Outdoor Scenes Based on Continuous Video Streams

Abstract

This invention provides a method for depth completion of outdoor scenes based on continuous video streams, comprising: preprocessing sensor input data to form RGB-D data; the image branch of the depth completion module receives the RGB data, extracts image features through the image branch's codec, and concatenates the output of the image branch decoder with the input of the encoder of the same size in the depth branch; the depth branch of the depth completion module receives the D data and the guide map of the previous frame in the data stream, and outputs an initial completed depth map, the guide map of the current frame, and an affinity matrix through the depth branch's codec; the depth map optimization module receives the initial completed depth map and the affinity matrix output by the depth branch's codec, and outputs an optimized completed depth map for outdoor scene depth completion. This invention effectively utilizes contextual information in the continuous video stream, improving the accuracy of depth completion prediction.

Description

Technical Field

[0001] This invention relates to video image processing technology and deep learning technology, specifically to a deep learning technology for depth map completion based on continuous video streams. Background Technology

[0002] Existing methods for solving the depth completion problem in outdoor scenes can be divided into two categories: traditional machine learning methods that use manually designed features for completion, and deep learning methods based on convolutional neural network encoder-decoder structures. While both methods can solve the depth completion problem in outdoor scenes to some extent, their inherent limitations result in poor actual completion performance, making them unsuitable for practical applications.

[0003] A depth completion algorithm based on hand-designed features, grounded in traditional machine learning techniques, extracts features from images and depth maps using hand-designed feature operators, ultimately forming a depth completion model for outdoor scenes. However, this type of method exhibits poor prediction performance, primarily due to the following reasons: firstly, hand-designed 3D descriptors have varying effects across different scenes, exhibiting certain limitations; secondly, models based on hand-designed features have poor generalization ability.

[0004] Algorithms based on deep convolutional neural networks (CNNs) have addressed some of the problems mentioned above, with mainstream approaches using CNN encoder-decoder structures. However, since depth maps are two-dimensional projections of 3D point clouds, effective depth values ​​are very sparse, and extracting features from invalid depth values ​​using traditional convolution methods presents numerous difficulties. To address these issues, improved convolution operators can be used for feature extraction. An additional channel can be used to mark whether the depth map has valid values, or the spatial representation of depth values ​​can be encoded into the depth map. By introducing this positional information, the model can more effectively capture 3D feature information. Another approach is depth completion using guided filtering. This method treats the image as a separate branch, generating a guided map from the image, and then using the guided map to complete the depth map. This type of method performs better than manually designed features, but the actual prediction results are still not ideal. Summary of the Invention

[0005] The technical problem this invention aims to solve is that the applicant has found the actual prediction effect of depth map completion using guide maps to be unsatisfactory because the input for depth completion in outdoor scenes is usually a continuous video stream. Existing methods typically shuffle the data stream before inputting it into the model for training, failing to fully exploit the temporal relationship between images in the continuous video stream and the depth map. Therefore, this invention addresses the shortcoming of not utilizing temporal information in continuous video streams by proposing a method to solve the technical problem of existing deep learning-based depth completion methods for outdoor scenes failing to utilize the contextual relationships in continuous video streams, ultimately achieving accurate depth completion.

[0006] The technical solution adopted by this invention to solve the above-mentioned technical problems is a method for depth completion of outdoor scenes based on continuous video streams, comprising the following steps:

[0007] Step 1: Preprocess the data input from the sensor:

[0008] First, the point cloud data is projected onto the image based on the extrinsic parameter matrix to generate a sparse depth map D. Then, the sparse depth map D is combined with the image data RGB acquired by the RGB camera to form RGB-D data through frame synchronization.

[0009] Step 2, Depth Completion Module Processing Steps: The image branch of the trained depth completion module receives image data RGB from the RGB-D data and extracts image features through the image branch's encoder and decoder. The output of the image branch decoder is concatenated with the input of the encoder of the same size in the depth branch. The depth branch of the depth completion module receives the sparse depth map D from the RGB-D data and the guide map of the previous frame in the data stream. The codec of the depth branch outputs the initial completed depth map, the guide map of the current frame, and the affinity matrix. By applying the guide map of the previous frame in the RGB-D data stream to the generation of the initial completed depth map of the current frame, the contextual information in the continuous video stream is fully utilized.

[0010] Step 3: Depth Map Optimization Module Processing Steps: The trained depth map optimization module receives the initial completed depth map and affinity matrix output by the encoder and decoder of the depth branch. Based on the affinity matrix, iteratively optimizes the initial completed depth map and outputs the optimized completed depth map for outdoor scene depth completion.

[0011] The beneficial effects of this invention are: it effectively utilizes contextual information in continuous video streams, making up for the shortcomings of existing methods that only use convolutional neural networks for prediction, thereby improving the prediction effect; and it further optimizes the model output through the depth map optimization module, which further improves the accuracy of depth completion prediction. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the depth completion algorithm of the present invention;

[0013] Figure 2 This is a structural diagram of the depth completion module of the present invention;

[0014] Figure 3 This is a schematic diagram of the depth map optimization module of the present invention;

[0015] Figure 4 This is a schematic diagram of the ensemble convolution structure of this invention. Detailed Implementation

[0016] In this specific embodiment, the outdoor scene depth completion algorithm based on continuous video streams comprises two parts: training and prediction. The training part mainly obtains a well-trained (i.e., fully trained) outdoor scene depth completion method model based on the continuous video stream. The prediction part executes steps S1-S4 based on the trained model.

[0017] The following section, in conjunction with the implementation steps, details the data preprocessing, the training of the depth completion module, and the training of the depth map optimization module, with illustrations.

[0018] See Figure 1 For an outdoor point cloud to be completed, the model is trained and depth completion is performed through the following steps:

[0019] Step S1, data preprocessing mainly involves obtaining a data stream as input to the depth completion module. Data preprocessing includes the following steps:

[0020] First, the point cloud and image data frames transmitted by the sensor need to be synchronized. The frame synchronization method is to match the time. Image and point cloud frames with an absolute time difference of less than 0.02s are considered as matched frames.

[0021] Next, the calibrated extrinsic matrix is ​​used to project the point cloud data onto the image plane to obtain a sparse depth map. The image data and the sparse depth map data are then compared in RGB-D format. Points with negative depth D and points whose coordinates are outside the camera's range are removed. The sparse depth map data and the RGB data acquired by the RGB camera are then combined through frame synchronization to form RGB-D data, which serves as the input data for the depth completion module.

[0022] During the training phase, the video stream needs to be grouped according to a fixed number of frames and sliced. This invention uses a group size of 200 frames. Finally, data augmentation is used to increase the experimental data. Methods that can be used include random rotation, horizontal flipping, scaling, and brightness adjustment. At the same time, the projection matrix also needs to be transformed accordingly.

[0023] Specifically, the projection is performed in the following manner:

[0024]

[0025] Where (u,v) are the pixel coordinates of the point cloud data, f is the focal length of the RGB camera, α and β are the components of the focal length in the x and y directions of the camera coordinate system, respectively, and c x ,c y R is the offset of the camera's principal axis, and R is a rotation matrix used to align the camera and LiDAR coordinates. T Let X represent the extrinsic rotation matrix. c The point cloud x-axis coordinate Y c Let Z be the y-axis coordinate of the point cloud. c Let R be the z-axis coordinate of the point cloud, and R be the extrinsic parameter matrix.

[0026] During the training phase, the input data for the depth completion module also needs to be grouped into frames and augmented. Data augmentation methods include rotation, scaling, and image brightness adjustment. The same transformation strategy should be used for the same group of data. When the image is rotated or scaled, the extrinsic parameter matrix used for the corresponding projection also needs to be changed.

[0027] The steering filter matrix generated in the previous frame of the current frame is also used as input to the depth completion module.

[0028] Steps S2 and S3 are respectively for building the model and training the depth completion network. The sparse depth map, image data and steering filter matrix obtained in step S1 are used as inputs to train the depth completion model.

[0029] Then, using the trained depth completion model, the completed depth map is predicted, and an initial completed depth map, affinity matrix, and guide map for the next frame are generated.

[0030] The training mainly consists of two parts: training of the depth completion module and training of the depth map optimization module.

[0031] Those skilled in the art will understand that the training of the depth completion module primarily aims to obtain a backbone network capable of outputting dense depth maps, such as... Figure 2 As shown, the overall structure of the depth completion module model is a two-branch encoder-decoder structure. This module uses the RGB image output from data preprocessing, the sparse depth map Depth, and the guide matrix Guide generated from the previous frame as input to predict missing depth values.

[0032] The upper part of the deep completion module model's overall structure is the image branch, used to extract image features, and it is cascaded with the lower deep branch. In the encoder part, the number of feature channels in each layer of the network is doubled compared to the previous layer, while the feature map is scaled down, such as... Figure 2The encoder section has 32, 64, 128, 256, and 512 feature channels, respectively, and feature map sizes of 352*1216, 176*608, 88*304, 44*152, and 22*76, respectively. The decoder section, conversely, gradually reduces the number of channels and upsamples the feature maps. The lower half of the depth completion module model is the depth branch, used to output a dense depth map. Its encoding part cascades with the decoding part of the image branch to achieve data fusion, such as... Figure 2 The output branches (1) of the 64@176*608 module, (2) of the 128@88*304 module, and (3) of the 256@44*152 module in the decoding part of the image branch are concatenated with the input branches (1) of the 64@176*608 module, (2) of the 128@88*304 module, and (3) of the 256@44*152 module in the encoding part of the depth branch, respectively. The decoding part of the depth branch is similar to that of the image branch, and the depth value is recovered by continuously reducing the number of channels and upsampling.

[0033] The final output of the depth completion module consists of three parts: first, a steering matrix for the next frame; second, the output of the initial completed depth map; and finally, an affinity matrix for the depth map optimization module.

[0034] During training of the depth completion module, a batch size of 32 is used, employing an absolute error-based loss function and a batch gradient descent optimizer. The absolute distance between the predicted value and the effective depth in the ground truth label is used as the loss function during training.

[0035]

[0036] in Represents the true depth value. Represents the predicted depth value. It represents the valid portion of the true depth value. This is the difference between the predicted depth and the actual depth.

[0037] The convolutions in the depth branch use convolutions with accompanying geometric information features, which encode the spatial coordinates (x, y, z) of the point cloud into the depth, such as... Figure 4 As shown, this is done to improve the overall effect.

[0038] The three-dimensional extension of the guided filtering operation is as follows:

[0039]

[0040] Where I represents the input feature, c is the number of channels of the input feature, I′ represents the output feature, c′ is the number of channels of the output feature, p represents the pixel position of the feature, and I′p,c′ This represents the output feature, where m is the total number of points, k represents the neighborhood of pixel p, and I... p+k,c This represents the feature value of the input feature at pixel p+k in the c-th channel. This represents the guide graph generated by the depth branch, where D is the depth. The parameters are used. However, this method has an excessively high computational load. This invention divides the guided filtering into two parts, as shown below.

[0041]

[0042]

[0043] The first step is to perform guided filtering on each channel to obtain I′. p，c , This is the guide map for the first step. The parameters in the guide diagram; the second step is to use I′ p,c Performing a regular convolution yields I″ p,c′ , This is the guide map for the second step. The parameters in the guide graph are used. The output I″ is used as the point cloud feature for the next level of calculation.

[0044] Finally, the first frame of each group does not contain guidance information from the previous frame; in this embodiment, the depth map itself is used as the guidance matrix for the first frame.

[0045] Step S4 takes the initial completed depth map and affinity matrix obtained in the previous step as input data, and iterates through the trained depth map optimization module to output the final completed depth map.

[0046] Those skilled in the art will understand that the training of the depth map optimization module mainly involves obtaining a module that iteratively optimizes the initially predicted depth map using an affinity matrix, such as... Figure 3 As shown.

[0047] Affinity Matrix Explanation: Taking an H×W depth map and a k×k local neighborhood as an example. The initially predicted single-channel depth map is D0. The affinity matrix A output by the affinity network is A, with k×k channels and dimensions equal to the depth. Figure 1 To, among them Represents the neighborhood of pixel (m,n) Each channel represents the affinity of a neighborhood of the pixel at that location. The depth map optimization module iteratively optimizes the depth map D0 based on the following method:

[0048]

[0049] in, Let represent the values ​​of pixel (m, n) in the depth map obtained from the t-th and (t-1)-th iterations, respectively. Let represent the value of pixel (m+i,n+j) in the depth map within the neighborhood of pixel (m,n) obtained in the (t-1)th iteration. For the current coordinate affinity, For proximity affinity, the number of iterations t for depth map optimization is fixed at 6 in this embodiment.

[0050] The depth map optimization module is relatively simple, mainly involving iterative optimization of the depth map through an affinity matrix. This invention uses a local neighborhood affinity matrix for depth map optimization, which is an 8-channel matrix used to represent the affinity of a 3×3 interval surrounding a corresponding element.

[0051] The loss function of the depth map optimization module adds a smoothing loss module to the depth completion module: L = L1 + L2, which includes the absolute distance between the predicted value and the effective depth in the true label, as well as a smoothing loss, where L1 is the same as that in the depth completion module.

[0052]

[0053] L2 is the smoothing operator:

[0054]

[0055] in This represents the depth map output by the depth completion model. This represents calculating the gradient, where I represents the RGB image, ||||1 represents the 1-norm, and Ns represents the number of valid points in the depth map.

[0056] During training, a batch size of 32 is used. In each training iteration, the loss function value of the model is calculated. Based on the loss function value, the model parameters are adjusted using the Adam optimization algorithm. When the current loss function value is higher than the previous loss function value, the model training terminates.

[0057] After obtaining a well-trained model, use step S5 to make predictions. Input the corresponding image and the sparse depth data stream, and you can get a dense depth map data stream as the output.

[0058] The above description is merely a specific embodiment of the present invention. Any feature disclosed in this specification may be replaced by other equivalent or similar features unless otherwise specified. All disclosed features, or steps in all methods or processes, may be combined in any way except for mutually exclusive features and / or steps.

Claims

1. A depth completion method for outdoor scenes based on continuous video streams, characterized in that, Includes the following steps: Step 1: Preprocess the data input from the sensor: First, the point cloud data is projected onto the image based on the extrinsic parameter matrix to generate a sparse depth map D. Then, the sparse depth map D is combined with the image data RGB acquired by the RGB camera through frame synchronization to form RGB-D data. Points with negative depth and points whose coordinates are not within the camera range will be removed. Step 2, Depth Completion Module Processing Steps: The image branch of the trained depth completion module receives image data RGB from the RGB-D data, extracts image features through the image branch's encoder-decoder, and concatenates the output of the image branch decoder with the input of the encoder of the same size in the depth branch; the depth branch of the depth completion module receives the sparse depth map D from the RGB-D data and the guide map of the previous frame in the data stream, and outputs the initial completed depth map, the guide map of the current frame, and the affinity matrix through the depth branch's encoder-decoder; the depth map itself is used as the guide map of the first frame; Step 3: Depth Map Optimization Module Processing Steps: The trained depth map optimization module receives the initial completed depth map and affinity matrix output by the encoder and decoder of the depth branch. Based on the affinity matrix, iteratively optimizes the initial completed depth map and outputs the optimized completed depth map for outdoor scene depth completion.

2. The method as described in claim 1, characterized in that, During the training of the depth completion module and the depth map optimization module, after generating the RGB-D data, the following processing is required in the preprocessing stage: The RGB-D data is divided into segments of fixed length, forming a group; data augmentation is then performed on each group of data. Data augmentation techniques include random rotation, horizontal flipping, scaling, and brightness adjustment.

3. The method as described in claim 1, characterized in that, The loss function used during training of the depth completion module is calculated using the absolute error between the predicted depth and the effective depth in the true label. The loss function used during training of the depth map optimization module incorporates a smoothing loss based on the absolute error between the predicted depth and the effective depth in the true label.

Images