A deep completion method based on sparse representation

Abstract

The application discloses a kind of deep completion methods based on sparse representation, belong to image processing technical field.The application designs a kind of adaptive sampling mode capture important sampling point, it is favorable to network reconstruction to be more dense depth map, specifically includes the following steps: S1, the RGB image is output by sampling network uncertainty map;S2, based on the sampling process of uncertainty to obtain the sparse depth map after sampling;S3, the neural network structure of reconstruction is built, simultaneously the RGB image and the sparse depth map after sampling are input into reconstruction network, training is carried out, and the dense depth map is recovered.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a depth completion method based on sparse representation. BACKGROUND

[0002] In recent years, with the continuous development of the field of computer vision, it is particularly important to obtain effective depth information for the fields of autonomous driving, robots, augmented reality, etc. In actual scenarios, depth information is often obtained through RGB-D depth cameras or laser radars, etc. For laser radars, due to the limitations of hardware devices, the depth map projected to the image plane from the original laser radar scanning obtained point cloud is relatively sparse, and in real application scenarios, a dense depth map is more needed to facilitate further perception of environmental information. The process of obtaining a dense depth map from a sparse depth map is called depth completion.

[0003] For the task of depth completion, with the vigorous development of deep learning, in recent years, many researchers have also researched and innovated in this task, mainly divided into two aspects: on the one hand, only the sparse depth map is taken as the input of the neural network to reconstruct the dense depth map; on the other hand, the rich semantic information of RGB is combined as a guide to realize the completion of the sparse depth map. In actual application scenarios, the method of combining the semantic information of RGB as a guide for depth completion is more popular. Usually, the RGB picture and the sparse depth map are directly taken as the input of the network, and if better results can be reconstructed with relatively few sampling points, the cost will be saved, thereby making the cost performance higher. Past researchers usually randomly sample the original real depth map to obtain some sparse sampling points, but in this case, the sampling points cannot well reflect the edge information of some objects in the real depth map, so there is a slight flaw in the reconstruction effect.

[0004] In summary, the present application provides a depth completion method based on sparse representation. SUMMARY

[0005] The present application aims to provide a depth completion method based on sparse representation to solve the problems proposed in the background art.

[0006] In order to achieve the above-mentioned purpose, the present application adopts the following technical scheme:

[0007] A depth completion method based on sparse representation, specifically comprising the following steps:

[0008] S1, selecting an RGB image from a plurality of public depth completion data sets and inputting the same into a sampling network to output an uncertainty map;

[0009] S2, an uncertainty map based sampling process to obtain a sampled sparse depth map;

[0010] S3, designing a reconstructed neural network structure, building an image reconstruction network, inputting the RGB image and the sampled sparse depth map obtained in S2 into the reconstruction network to train and recover a dense depth map. Figure One

[0011] Preferably, the sampling network in S1 adopts a U-net type structure design, which is composed of an encoder and a decoder.

[0012] The encoder includes four residual modules, each of which is composed of a feature extraction down-sampling module and a feature preservation module. The feature extraction down-sampling module is used to extract high latitude features of the image and complete the down-sampling of the image; the feature preservation module is used to further deepen the feature map without losing resolution.

[0013] The decoder is composed of four receptive field guided up-sampling modules.

[0014] A skip connection is arranged between the encoder and the decoder to further promote feature fusion, and finally output an uncertainty map.

[0015] Preferably, the uncertainty map represents a sampling logic relationship, and high uncertainty means low sampling probability. Based on this, S2 specifically includes the following contents:

[0016] S2.1, assuming that the size of the depth map D is m x n, the elements in the binary sampling mask M are defined as:

[0017]

[0018] In the formula, p i,j represents the sampling probability of the pixel at position (i, j);

[0019] The sampling process of the depth map D is defined as:

[0020] S = D M

[0021] In the formula, S represents the sparse depth map obtained by sampling the depth map D; and represents pixel-level dot product.

[0022] S2.2, assuming that the size of the uncertainty map P is also m x n, based on S2.1, a binary sampling mask M' is generated from the uncertainty map: first, generate a random matrix R with size m x n, and the element value in the matrix is a random value between 0 and 1, denoted as r i,j ∈ [0, 1];

[0023] ​S2.3, the sampling probability of the pixel of the element in the uncertainty map is denoted as p' i,j , p' i,j ∈[0, 1]; r i,j is compared with p' i,j , the corresponding binary mask position meeting the condition r i,j ≤p' i,j is set to 1, otherwise, it is set to 0, which can be expressed as:

[0024]

[0025] S2.4, based on the contents described in S2.1-S2.3, the sampling process of the uncertainty map P is defined as:

[0026] S' = P·M'

[0027] In the formula, S' represents the sparse depth map after sampling of the uncertainty map P;· represents pixel-level dot multiplication.

[0028] Preferably, the reconstruction network in S3 is composed of a multi-scale convolution module, a parallel double-flow encoding-decoding structure, and an adaptive fusion mechanism, and the reconstruction network realizes dense depth map reconstruction through cross-channel information interaction;

[0029] The loss function design of the sampling part in the training process of the reconstruction network specifically includes the following contents:

[0030] ①L2Loss loss function l prob : the l prob is used to supervise the generated uncertainty map, and the specific function is expressed as:

[0031]

[0032] Wherein, represents the Sobel gradient operator symbol;

[0033] ②Regular loss function l reg : the l reg is used to constrain the training process, and the specific function is expressed as:

[0034]

[0035] Wherein, N represents the total number of effective depth image pixels; s represents the number of sampling points for sampling;

[0036] ③Total loss function l sample of the sampling part: the l sample specific function is expressed as:

[0037] l sample =l prob+ alpha1 reg

[0038] Wherein, alpha represents a weighting coefficient;

[0039] The loss function design of the reconstruction part in the reconstruction network training process specifically includes the following contents:

[0040] 1. The gradient loss item l based on L1 loss between the reconstructed depth map D and the real depth map D * grad : The l is used to reduce the error of calculating the depth gradient, and the specific function is represented as: grad

[0041]

[0042] 2. The surface normal loss function l norm : The l is used to further deepen the localized details, and the specific function is represented as: norm

[0043]

[0044]

[0045]

[0046] Wherein, < > represents the inner product of the vector;

[0047] 3. The total loss function l of the reconstruction part rec : The l is specifically represented as: rec

[0048]

[0049] Wherein, w1, w2, w3 and w4 represent the weighting coefficients of different parts;

[0050] The loss function design of the comprehensive sampling part and the reconstruction part can obtain the total loss function l of the reconstruction network final :

[0051] l final = l rec + beta1 sample .

[0052] Wherein, beta represents a weighting coefficient.

[0053] Compared with the prior art, the depth completion method based on sparse representation provided by the present application has the following beneficial effects:

[0054] ​​​​(1) The application can greatly improve the sampling strategy, and effectively utilize the sampling points of the edge part in the case that the traditional random sampling method cannot effectively capture the edge depth information of the object.

[0055] (2) The method provided by the application can reconstruct a relatively dense effect, thereby opening a new train of thought for reducing the device cost in the actual application scene. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 A whole block diagram of a depth completion method based on sparse representation is provided in the application;

[0057] Figure 2 For the example effect picture mentioned in embodiment 1 of the application, from left to right, each column is input RGB, randomly sampled sampling points, uncertainty map, sampling points after adaptive sampling, reconstructed dense depth map, and original real depth map. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments.

[0059] The application uses indoor dataset NYUDepthV2 and outdoor dataset KITTI as the experimental dataset. The NYUDepthV2 dataset is a video sequence of indoor scenes captured by the RGBD camera of Microsoft Kinect, and contains about 50,000 indoor RGB-D image pairs collected in 464 different indoor scenes. The official dataset division is used in the application, in which 249 scenes are used for training and the remaining 215 are used for testing. The application first down-samples the RGB-D image pairs from the original resolution of 640*480 to 320*240. Since the boundary of the original depth map does not contain measurement values, the application only evaluates the center clipping area of 304*228. In the KITTI dataset, the training set contains 85898 frames of RGB-D image pairs, the verification set contains 1000 frames of RGB-D image pairs, and another 1000 frames are used for the test set. The dataset provides RGB images and aligned sparse depth maps, which are obtained by projecting 3D LiDAR points onto the corresponding image frames, and the resolution of the color image and the depth image is the same, that is, 352*1216. The depth map obtained by the original 64-line laser radar scanning has about 5% valid pixels, and the ground true semi-dense depth map has about 15% valid pixels. Since there are invalid pixel points in the upper boundary of the depth map, the size of 256*1216 is clipped for the training and testing stages. The specific example content is as follows.

[0060] Embodiment 1

[0061] Please refer to Figures 1-2 The application provides a deep completion method based on sparse representation, which specifically comprises the following steps:

[0062] S1, select RGB images from several public depth completion datasets, input them into a sampling network to output an uncertainty map;

[0063] The sampling network adopts a U-net type structure design and is composed of an encoder and a decoder;

[0064] The encoder includes four residual modules, each of which is composed of a feature extraction down-sampling module and a feature preservation module. The feature extraction down-sampling module is used to extract high latitude features of the image and complete the down-sampling of the image; the feature preservation module is used to further deepen the feature map without losing resolution; mainly using 1*1 convolution and 3*3 convolution, and the corresponding batch normalization and RELU activation function;

[0065] The decoder is composed of four receptive field guided up-sampling modules;

[0066] A skip connection is arranged between the encoder and the decoder to further promote feature fusion, and finally an uncertainty map is output;

[0067] S2, based on the sampling process of the uncertainty map to obtain a sparse depth map after sampling;

[0068] The uncertainty map represents a sampling logic relationship, and high uncertainty means low sampling probability. Based on this, S2 specifically includes the following contents:

[0069] S2.1, assuming that the size of the depth map D is m*n, the elements in the binary sampling mask M of the depth map D are defined as:

[0070]

[0071] In the formula, p i,j represents the sampling probability of the pixel at the (i,j) position;

[0072] The sampling process of the depth map D is defined as:

[0073] S=D*M

[0074] In the formula, S represents the sparse depth map after sampling of the depth map D; and represents pixel-level dot multiplication;

[0075] S2.2, assuming the size of the uncertainty map P is also mxn, based on S2.1, a binary sampling mask M' is generated from the uncertainty map: first, a random matrix R of size mxn is generated, the element value in the matrix is a random value between 0 and 1, denoted as r i,j ∈[0,1];

[0076] S2.3, the sampling probability of the pixel of the element in the uncertainty map is denoted as p' i,j , p' i,j ∈[0,1];r i,j and p' i,j are compared, the corresponding binary mask position satisfying the condition r i,j ≤p' i,j is set to 1, otherwise it is set to 0, which can be expressed as:

[0077]

[0078] S2.4, based on the contents of S2.1-S2.3, the sampling process of the uncertainty map P is defined as:

[0079] S' = P*M'

[0080] In the formula, S' represents the sparse depth map after sampling of the uncertainty map P; * represents pixel-level dot multiplication;

[0081] S3, design the neural network structure of reconstruction, build the image reconstruction network, input the RGB image and the sparse depth Figure One obtained in S2 into the reconstruction network for training to recover the dense depth map; wherein the multi-scale convolution module mainly fuses the convolution kernels with sizes of 1 and 3, including not only element-level addition but also channel direction cascade, so as to more effectively extract features. The encoding part of the parallel double-flow encoding-decoding network is mainly the pre-trained MobileNetV3-large, the main advantage of which is lightweight while ensuring the effectiveness of feature extraction, and the network of the decoding part also adopts the receptive field guided up-sampling module. The channel attention mechanism is adopted between the encoder and the decoder, which is helpful for further fusion of features. Finally, the feature weights of the double flow are calculated through the sigmoid activation function, and the reconstructed depth map is finally obtained.

[0082] The sparse depth map and the RGB image obtained after the foregoing sampling are connected in the channel dimension, and the image with a channel number of 4 is input into the reconstruction network;

[0083] The reconstruction network is composed of a multi-scale convolution module, a parallel double-flow encoding-decoding structure and a self-adaptive fusion mechanism, and the reconstruction network realizes dense depth map reconstruction through cross-channel information interaction;

[0084] The loss function design of the sampling part in the reconstruction network training process specifically includes the following contents:

[0085] ① L2Loss loss function l prob : l prob The uncertainty map generated is used for supervision, and the specific function is represented as:

[0086]

[0087] Wherein, Sobel gradient operator symbol;

[0088] ② Regularization loss function l reg : l reg Used to constrain the training process, and the specific function is represented as:

[0089]

[0090] Wherein, N represents the total number of effective depth image pixels; s represents the number of sampling points for sampling;

[0091] ③ Total loss function l sample : l sample of the sampling part, the specific function is represented as:

[0092] l sample = l prob + αl reg

[0093] Wherein, α represents the weighting coefficient;

[0094] The loss function design of the reconstruction part in the reconstruction network training process specifically includes the following contents:

[0095] ① The gradient loss term l * based on L1 loss between the reconstructed depth map D grad : l grad Used to reduce the error of calculating depth gradient, and the specific function is represented as:

[0096]

[0097] ② Surface normal loss function l norm : l norm Used to further deepen the localized details, and the specific function is represented as:

[0098]

[0099]

[0100]

[0101] where <,> denotes the inner product of vectors;

[0102] ③Reconstruction part total loss function l rec : rec The specific function is represented as:

[0103]

[0104] where w1, w2, w3, w4 represent the weighting coefficients of different parts;

[0105] The loss function design of the sampling part and the reconstruction part is integrated, and the total loss function l of the reconstruction network is obtained final :

[0106] l final = l rec + βl sample .

[0107] where β represents the weighting coefficient.

[0108] In the present application, the Adam optimizer is used, and the parameter settings are β1=0.9, β2=0.999, and the weight decay is set to 10 -5 . The initial learning rate is set to 0.001. The present application uses the deep learning framework Pytorch to train the model, and iteratively trains the whole training set for 20 cycles, and the learning rate is adjusted to 80% of the previous one every 2 cycles. In addition, in order to improve the overall performance of the model while preventing overfitting, two data enhancement strategies are used in this paper to increase the diversity of training data, including:

[0109] Random horizontal flip: color images and depth images are horizontally flipped with a probability of 50%;

[0110] Random channel exchange: the RGB three channels of the color image are randomly exchanged with a probability of 50%.

[0111] Example 2

[0112] Based on example 1 but with some differences:

[0113] The present application selects 10 advanced comparative methods trained on the KITTI dataset, including: ACMNet, Sparse-to-Dense, CSPN, DeepLIDAR, NConv-CNN, MSG-CHN, GuideNet, Uncertainty, PENet and AdaptiveLIDAR. For the two cases of 256 sampling points and 512 sampling points, please refer to Table 1 and Table 2 for specific results.

[0114] Table 1 256 sampling point result comparison

[0115]

[0116]

[0117] Table 2 512 sampling point result comparison

[0118] Method RMSE MAE iRMSE iMAE REL delta 1.25 ]]> ACMNet 3417.34 1413.75 15.54 7.91 0.086 90.4 Sparse-to-Dense 2151.12 659.15 4.41 2.29 0.033 98.2 CSPN 1828.93 506.39 4.04 2.16 0.023 99.0 DeepLIDAR 1735.29 543.62 3.96 1.97 0.025 98.9 NConv-CNN 1973.10 508.64 4.27 2.14 0.025 98.7 MSG-CHN 1862.38 591.95 4.13 2.14 0.029 98.6 GuideNet 1787.59 554.37 3.98 2.06 0.028 99.0 Uncertainty 1771.60 568.18 4.08 2.16 0.026 98.8 PENet 1842.54 597.16 4.31 2.29 0.030 98.5 AdaptiveLIDAR 1789.41 590.62 3.92 1.89 0.027 98.7 Ours 1346.79 446.57 4.70 2.15 0.025 99.2

[0119] As shown in Table 1 and Table 2, quantitative comparison results on RMSE, MAE, iRMSE, REL and δ 1.25 indicators are shown in the table, wherein RMSE, MAE, iRMSE, REL are all the smaller the better, and δ 1.25 the larger the better; it can be seen from the table that the method of the present application can achieve the best effect in important RMSE, MAE, δ 1.25 indicators.

[0120] The above is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited to this, any skilled person in the art can make equivalent replacement or change according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A method for deep completion based on sparse representation, characterized in that, Specifically comprising the following steps: S1, selecting an RGB image from a plurality of public depth completion datasets, inputting the RGB image into a sampling network to output an uncertainty map; the sampling network adopts a structure designed in the form of a U-net, and is composed of an encoder and a decoder; The encoder comprises four residual modules, each of which is composed of a feature extraction down-sampling module and a feature preservation module; the feature extraction down-sampling module is used to extract high latitude features of the image and complete down-sampling of the image; and the feature preservation module is used to further deepen the feature map without losing resolution; The decoder is composed of four receptive field guided up-sampling modules; A skip connection is arranged between the encoder and the decoder to further promote feature fusion, and finally an uncertainty map is outputted; The uncertainty map represents a sampling logical relationship, and high uncertainty means low sampling probability; S2, a sampling process based on the uncertainty map is used to obtain a sampled sparse depth map; specifically comprising the following contents: S2.1, assuming that the size of a depth map D is m x n, the elements in a binary sampling mask M are defined as: wherein p i,j representing a sampling probability of a pixel at position i , j ) position; The sampling process of the depth map D is defined as: S = D · M wherein S denotes the sparse depth map after sampling of the depth map D; denotes a pixel-wise point multiplication; S2.2, assuming the size of the uncertainty map P is also mxn, based on S2.1, generating a binarized sampling mask M' from the uncertainty map: first, generate a random matrix R of size mxn, the element values in the matrix are random values between 0 and 1, denoted as r i,j ∈[0,1] S2.3, record the sampling probability of the pixel of the element in the uncertainty map as p i,j , p i,j ∈ [0, 1]; record the sampling probability of the pixel of the element in the uncertainty map as r i,j and p i,j , satisfy r i,j ≤ p i,j The corresponding binary mask position of the condition is set to 1, otherwise it is set to 0, which can be expressed as:​​​​ S2.4, based on the contents of S2.1-S2.3, the sampling process of the uncertainty map P is defined as: S ’= P · M ’ In the formula, S' represents a sparse depth map obtained by sampling the uncertainty map P; and • represents pixel-level point multiplication; S3, a neural network structure for reconstruction is designed, an image reconstruction network is built, and the RGB image and the sampled sparse depth map obtained in S2 are inputted into the reconstruction network for training to restore a dense depth map.

2. The method of claim 1, wherein, The reconstruction network in S3 is composed of a multi-scale convolution module, a parallel double-flow encoding-decoding structure and a self-adaptive fusion mechanism, and the reconstruction network realizes dense depth map reconstruction through cross-channel information interaction; The loss function design of the sampling part in the training process of the reconstruction network specifically comprises the following contents: ①L2 Loss loss function l prob : the l prob The uncertainty map to supervise the generation is specifically represented as: wherein, ∇ D represents the Sobel gradient operator sign; ii. Regularization loss function l reg : said l reg to constrain the training process, a specific function is represented as: wherein, N represents the total number of valid depth image pixels; s represents the number of sampling points sampled. ③ Total loss function of sampling part l sample : The l sample The specific function is represented as: l sample = l prob + αl reg wherein α denotes a weighting factor; The loss function design of the reconstruction part in the training process of the reconstruction network specifically comprises the following contents: ① reconstructed depth map D * L1 loss-based gradient loss term between the real depth map D l grad : the l grad to reduce the error of calculating the depth gradient, and the specific function is represented as:​ l grad ( D * , D )= l 1(▽ x D * ,▽ x D )+ l 1(▽ y D * ,▽ y D ) ii. Surface normal loss function l norm : said l norm To further deepen the localization details, the specific function is represented as: Wherein, < > represents the inner product of a vector. iii. Reconstructing the total loss function l rec : the l rec The specific function is represented as: wherein w 1、 w 2、 w 3、 w 4 denotes weighting factors for different parts; The loss function design of the integrated sampling part and the reconstruction part can obtain the total loss function of the reconstruction network l final For: l final = l rec + βl sample wherein β denotes a weighting factor.

Images