Real depth completion method and device based on pre-training and scale recovery

Abstract

The invention discloses a real depth completion method and device based on pre-training and scale recovery, and the method carries out the completion of the sparse laser radar depth according to the geometric clues in an RGB image, and can obtain a dense and accurate real depth image. The method comprises the following steps: carrying out calculation in a scale space through scale decomposition, carrying out scale recovery by utilizing an existing relative depth estimation method, and in a scale recovery module, firstly, carrying out pre-filling on an average scale of sparse scale / global estimation by utilizing cross attention according to spatial similarity among local points of an image; the method comprises the following steps: firstly extracting a scale image from a pre-training encoder, then performing feature fusion decoding by using image features extracted by the pre-training encoder and scale features extracted from the scale image, and finally performing iterative tuning on the scale image by adaptively adjusting neighborhood selection and affinity calculation of a spatial propagation network so as to obtain a more accurate scale result. The method not only has better robustness for different scenes, but also can finally realize a real depth completion effect with low complexity, high robustness and high precision.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a real depth completion method and device based on pre-training and scale recovery. Background Art

[0002] Acquiring dense depth information plays an important role in applications such as autonomous driving, robotics, and scene reconstruction. However, the reliable depth information obtained by common depth sensors such as LiDAR, depth cameras, and iToF is usually sparse and cannot meet the requirements of practical applications. In addition, the depth can be estimated by obtaining scene information through high-resolution RGB images. Due to the characteristics of projection, two-dimensional images lack real scale information, and the estimated depth only represents the front-to-back relationship of the depth and cannot reflect the real scale. Therefore, using high-resolution RGB images to extract scene structure information and complete the sparse depth is a better method for obtaining dense depth.

[0003] Most existing depth completion methods directly regress the depth with an absolute scale. There are large differences between different pixels in the entire image, which makes it difficult for the model to learn. In terms of network structure, the pre-fusion or feature fusion method is usually adopted, and the network is used to extract features separately. The generalization for different scenes is poor, and the feature expression needs to be re-learned for different categories of scenes. For the initial densification of sparse depth information, it is usually filled in gradually through learning methods or directly through non-learning methods, which may result in excessive calculation or inaccurate completed depth maps. Summary of the invention

[0004] The purpose of the present invention is to provide a real depth completion method and device based on pre-training and scale recovery in view of the deficiencies in the prior art.

[0005] The objective of the present invention is achieved through the following technical solutions: The first aspect of the present invention: a real depth completion method based on pre-training and scale recovery, comprising the following steps: (1) Obtain the KITTI depth completion dataset, which contains the color image of the current scene and the sparse depth map obtained by the depth sensor projection. Aggregate the sparse depth of consecutive frames to obtain the dense depth truth value; (2) Use the pre-trained visual feature encoder on the dataset to encode the input image, select the outputs of four intermediate layers, and obtain multi-scale image features; (3) Obtain relative depth map using monocular depth estimation method; (4) Divide the sparse depth map by the relative depth map pixel by pixel at the effective depth point to obtain a sparse scale map. Calculate the pooling weights based on the multi-scale image feature convolution and downsample the sparse scale map to four different scales using weighted pooling. (5) Taking multi-scale image features and sparse scale maps as input, fusion decoding is performed to complete the scale, and a fine-tuned dense scale map is output; (6) Transform from scale space to depth space, multiply the dense scale map and the relative depth map obtained in step (2) pixel by pixel to obtain the final dense absolute depth map; (7) Use the dense depth truth and the relative depth map obtained in step (2) to calculate the loss between the predicted dense absolute depth to supervise the learning of the depth completion method.

[0006] Furthermore, the step (5) is implemented by the following sub-steps: (5.1) Pre-fill the sparse scale map, calculate the global single average scale for filling, output the initial dense scale map and obtain the confidence map; (5.2) Using a lightweight convolutional network, extract features from the initial dense scale map output in step (5.1) to obtain scale features that are aligned with the image features; (5.3) Perform level-by-level fusion decoding on the scale features and image features at the four scales and output a dense scale map; (5.4) Fine-tune using the adaptive spatial propagation network: For all target pixels to be updated in the dense scale map output by step (5.3), the neighborhood is adaptively calculated based on the mixed features and the output scale map, and the affinity between the target pixels and the current pixel is directly predicted from the network, i.e., the correlation weight; finally, the current pixel scale value is weightedly updated according to the affinity to obtain the final dense scale map.

[0007] Furthermore, the step (5.1) specifically includes the following contents: at four scales, for unknown scale points on the sparse scale map, searching for their N nearest valid neighbor points and calculating the pixel distances between them; setting a distance threshold, for unknown scale points whose distances to the nearest neighbor pixels are less than the threshold, using a cross-attention method, calculating the attention score between the unknown scale point and the image features extracted in step (2) of the neighboring points, and weighting the scale values ​​of the known scale points to obtain the scale of the current point; for unknown scale points whose distances are greater than the threshold, calculating the global single average scale for filling, and outputting an initial dense scale map; and at the same time extracting a confidence map for the sparse scale map, the confidence map indicating whether the measurement value on the depth map is valid.

[0008] Furthermore, the loss calculation in step (7) includes the supervision of the true depth value and the relative depth. The loss expression is as follows: ; in Represents the depth at the valid point , It is the edge supervision of relative depth to absolute depth; the two losses are used as hyperparameters Combined together, it is the total loss.

[0009] Specifically, the The expression is as follows: ; in, represents the dense depth truth, Represents the absolute depth of the prediction.

[0010] Furthermore, the edge supervision of the relative depth on the absolute depth is specifically as follows: firstly, the absolute depth is normalized, and then the partial derivative of the residual between the relative depth and the absolute depth is calculated in the x and y directions. The expression is as follows: ; in, Indicates the relative depth. represents the absolute depth of the normalized prediction.

[0011] The second aspect of the present invention: a real depth completion device based on pre-training and scale recovery, comprising the following modules: Acquisition and aggregation module: obtains the KITTI depth completion dataset, which contains the color image of the current scene and the sparse depth map obtained by the depth sensor acquisition projection, and aggregates the sparse depth of consecutive frames to obtain the dense depth truth value; Image encoding module: uses the visual feature encoder pre-trained on the dataset to encode the input image, selects the output of four intermediate layers, and obtains multi-scale image features; Estimation module: Use monocular depth estimation method to obtain relative depth map; Pooling sampling module: The sparse depth map is divided pixel by pixel with the relative depth map at the effective depth point to obtain a sparse scale map, and the pooling weight is calculated according to the multi-scale image feature convolution, and the sparse scale map is downsampled to four different scales using weighted pooling; Decoding and completion module: takes multi-scale image features and sparse scale maps as input, performs fusion decoding for scale completion, and outputs a fine-tuned dense scale map; Pixel calculation module: performs the transformation from scale space to depth space, multiplies the dense scale map and the relative depth map obtained by the image encoding module pixel by pixel, and obtains the final dense absolute depth map; Loss supervision module: Use the relative depth map obtained by the dense depth truth and image encoding module to calculate the loss between the predicted dense absolute depth to supervise the learning of the depth completion method.

[0012] A third aspect of the present invention: an electronic device, comprising: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the true depth completion method based on pre-training and scale recovery.

[0013] The fourth aspect of the present invention: a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the true depth completion method based on pre-training and scale recovery.

[0014] The beneficial effects of the present invention are as follows: The present invention performs calculations in scale space through scale decomposition and uses the existing relative depth estimation method for scale recovery, which significantly reduces the difficulty and complexity of model learning and can obtain the absolute depth of the real scale. When fusion decoding is performed on image features and sparse scale maps, cross attention or global average scale is first used to pre-fill the sparse scale, which reduces the computational overhead compared to directly applying convolution iterations on the sparse map and obtains more accurate dense outputs compared to non-learning methods; and the image features extracted by the pre-trained encoder are used for feature fusion decoding with the scale features extracted from the scale map, which has better robustness for different scenes; finally, by adaptively adjusting the neighborhood selection and affinity calculation of the spatial propagation network, the scale map is iteratively tuned to obtain more accurate scale results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is an overall flow chart of the real depth completion method based on pre-training and scale recovery of the present invention; Figure 2 It is a design structure diagram of the scale recovery decoder of the present invention. DETAILED DESCRIPTION

[0016] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0017] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0018] The present invention will be described in detail below with reference to the accompanying drawings.

[0019] like Figure 1 As shown, the present invention is a real depth completion method based on pre-training and scale recovery, comprising the following steps: Step 1: Get dataset information: Here we use the open source KITTI depth completion dataset, which contains the color image of the current scene, the sparse depth map obtained by the depth sensor acquisition projection, and the dense depth truth obtained by aggregating the sparse depth of consecutive frames; Step 2: Use the pre-trained feature encoder of Depth Anything V2 (based on DINOv2) to encode the input image, select the output of four intermediate layers, and obtain multi-scale image features: ;in Indicates four different scales.

[0020] Step 3: Use existing monocular depth estimation methods, such as Depth Anything V2, to obtain a relative depth map: ; Step 4: Input sparse depth map (Sparse depth map) Divide the relative depth map pixel by pixel at the effective point to obtain a sparse scale map (sparse scale map): ; Using weighted pooling, according to the image features at each scale , convolution calculates the pooling weights and downsamples the sparse scale map to four different scales: ; Step 5: Multi-scale image features , sparse scalar graph As input, fusion decoding is performed to complete the scale and output a fine-tuned dense scale.

[0021] This step is the core of the present invention. Figure 2 As shown, it is divided into the following sub-steps.

[0022] (5.1) Pre-fill the sparse scale map and obtain the confidence map For invalid points in the scalar graph, search for their nearest neighbors Point, here , and calculate the pixel distance between them, and select the minimum distance . Set the distance threshold , the distance between the nearest neighbor pixel Less than threshold For unknown scale points, the cross-attention method is used to calculate the attention score between the unknown scale point and the image features extracted in step 2 of the neighboring points, and the scale value of the known scale point is weighted to obtain the scale of the current point. Here, the target point Image features As query vector Q, is the valid value point of the nearest neighbor of the current point, and the neighboring point Image features Make K, scale value Make V, perform cross-attention calculations, and obtain a pre-filled dense scale map For distance Greater than threshold Unknown scale points, calculate the global single average scale Fill and output the initial dense scale map , the expression is as follows: ; At the same time, for sparse scaling graph Extract the confidence map, which indicates whether the measurement value on the sparse depth map is valid. The expression is as follows: ; (5.2) Extracting scale features The scale input here is the scale map, which needs to be semantically aligned with the image features. The initial dense scale map output from step (5.1) and confidence map Perform feature splicing and use lightweight convolutional networks for feature extraction to obtain scale features aligned with image features , the expression is as follows: ; (5.3) Feature fusion decoding The information here includes image features , scale characteristics , and the mixed features of the previous scale output , convolute the three features one by one and output the mixed features of the current scale The expression is as follows: ;in Represents three convolutional residual networks with the same structure and no shared parameters.

[0023] The last level of mixed features Output dense scale map through prediction head : ; (5.4) Fine-tuning the adaptive spatial propagation network At the same time, the hybrid features and from the scale diagram Features extracted from To guide the spatial propagation process: the mixed features With the features extracted from the scalogram For splicing, two branches are used to decode and predict the offset and affinity weight of the neighboring pixels of the current point. When the neighborhood kernel size is k, the offset prediction output here is , where the number of channels It represents the offset of each domain point in the x and y directions. The weight prediction output is , represents the weighted weight of each neighborhood point. Spatial propagation outputs dense residual scale map and Add together to get the final dense scale output , the expressions of offset prediction output and weight prediction output are as follows: ; ; Step 6: Transform the scale space to the depth space and transform the dense scale map And the relative depth map obtained in step 3 Multiply pixel by pixel to get the final dense absolute depth map , the expression is as follows: .

[0024] Step 7: Use dense depth truth And the relative depth map obtained in step 3 , and the dense absolute depth predicted in step 6 The loss is calculated between the two, and the learning of the supervised depth completion method is supervised. The total loss is as follows: ;in, Represents the true value at depth At the effective point , since the true depth value is usually distributed inside the object, this loss is used to supervise the flat surface of the object: ;Since the edge of relative depth is better, it can be used to supervise the edge of absolute depth. First, the absolute depth is normalized to obtain , and then calculate the partial derivatives of the residual between the relative depth and the x and y directions: ; Use hyperparameters for the two losses Combined together, it is the total loss.

[0025] The present invention also discloses a real depth completion device based on pre-training and scale recovery, comprising the following modules: Acquisition and aggregation module: obtains the KITTI depth completion dataset, which contains the color image of the current scene and the sparse depth map obtained by the depth sensor acquisition projection, and aggregates the sparse depth of consecutive frames to obtain the dense depth truth value; Image encoding module: uses the visual feature encoder pre-trained on the dataset to encode the input image, selects the output of four intermediate layers, and obtains multi-scale image features; Estimation module: Use monocular depth estimation method to obtain relative depth map; Pooling sampling module: The sparse depth map is divided pixel by pixel with the relative depth map at the effective depth point to obtain a sparse scale map, and the pooling weight is calculated according to the multi-scale image feature convolution, and the sparse scale map is downsampled to four different scales using weighted pooling; Decoding and completion module: takes multi-scale image features and sparse scale maps as input, performs fusion decoding for scale completion, and outputs a fine-tuned dense scale map; Pixel calculation module: performs the transformation from scale space to depth space, multiplies the dense scale map and the relative depth map obtained by the image encoding module pixel by pixel, and obtains the final dense absolute depth map; Loss supervision module: Use the relative depth map obtained by the dense depth truth and image encoding module to calculate the loss between the predicted dense absolute depth to supervise the learning of the depth completion method.

[0026] The present invention also discloses an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the real depth completion method based on pre-training and scale recovery; and discloses a computer-readable storage medium, on which computer instructions are stored, which, when executed by the processor, implement the steps of the real depth completion method based on pre-training and scale recovery.

[0027] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0028] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A true depth completion method based on pre-training and scale recovery, characterized in that: The following steps are involved: (1) Obtain the KITTI depth completion dataset, which contains the color image of the current scene and the sparse depth map obtained by the depth sensor projection. Aggregate the sparse depth of consecutive frames to obtain the dense depth truth value; (2) Use the pre-trained visual feature encoder on the dataset to encode the input image, select the outputs of four intermediate layers, and obtain multi-scale image features; (3) Obtain relative depth map using monocular depth estimation method; (4) Divide the sparse depth map by the relative depth map pixel by pixel at the effective depth point to obtain a sparse scale map. Calculate the pooling weights based on the multi-scale image feature convolution and downsample the sparse scale map to four different scales using weighted pooling. (5) Taking multi-scale image features and sparse scale maps as input, fusion decoding is performed to complete the scale, and a fine-tuned dense scale map is output; (6) Transform from scale space to depth space, multiply the dense scale map and the relative depth map obtained in step (2) pixel by pixel to obtain the final dense absolute depth map; (7) Use the dense depth truth and the relative depth map obtained in step (2) to calculate the loss between the predicted dense absolute depth to supervise the learning of the depth completion method.

2. The depth completion method according to claim 1, characterized in that: The step (5) is implemented by the following sub-steps: (5.1) Pre-fill the sparse scale map, calculate the global single average scale for filling, output the initial dense scale map and obtain the confidence map; (5.2) Using a lightweight convolutional network, extract features from the initial dense scale map output in step (5.1) to obtain scale features that are aligned with the image features; (5.3) Perform level-by-level fusion decoding on the scale features and image features at the four scales and output a dense scale map; (5.4) Fine-tune using the adaptive spatial propagation network: For all target pixels to be updated in the dense scale map output by step (5.3), the neighborhood is adaptively calculated based on the mixed features and the output scale map, and the affinity between the target pixels and the current pixel is directly predicted from the network, i.e., the correlation weight; finally, the current pixel scale value is weightedly updated according to the affinity to obtain the final dense scale map.

3. The depth completion method according to claim 2, characterized in that: The step (5.1) specifically includes the following contents: At four scales, for unknown scale points on the sparse scale map, search for their N nearest valid neighbor points and calculate the pixel distance between them; set a distance threshold, and for unknown scale points whose distance to the nearest neighbor pixel is less than the threshold, use a cross-attention method to calculate the attention score between the unknown scale point and the image features extracted in step (2) of the neighboring point, and weight the scale value of the known scale point to obtain the scale of the current point; for unknown scale points whose distance is greater than the threshold, calculate the global single average scale for filling, and output the initial dense scale map; at the same time, extract a confidence map for the sparse scale map, and the confidence map indicates whether the measurement value on the depth map is valid.

4. The depth completion method according to claim 1, characterized in that: The loss calculation in step (7) includes the supervision of the true depth value and the relative depth. The loss expression is as follows: ; in Represents the depth at the valid point , It is the edge supervision of relative depth to absolute depth; the two losses are used as hyperparameters Combined together, it is the total loss.

5. The depth completion method according to claim 4, characterized in that: Said The expression is as follows: ; in, represents the dense depth truth, Represents the absolute depth of the prediction.

6. The depth completion method according to claim 4, characterized in that: The edge supervision of the relative depth on the absolute depth is specifically as follows: first, the absolute depth is normalized, and then the partial derivatives of the residual between the relative depth and the absolute depth are calculated in the x and y directions. The expression is as follows: ； in, Indicates the relative depth. represents the absolute depth of the normalized prediction.

7. A real depth completion device based on pre-training and scale recovery, characterized in that: Includes the following modules: Acquisition and aggregation module: obtains the KITTI depth completion dataset, which contains the color image of the current scene and the sparse depth map obtained by the depth sensor acquisition projection, and aggregates the sparse depth of consecutive frames to obtain the dense depth truth value; Image encoding module: uses the visual feature encoder pre-trained on the dataset to encode the input image, selects the output of four intermediate layers, and obtains multi-scale image features; Estimation module: Use monocular depth estimation method to obtain relative depth map; Pooling sampling module: The sparse depth map is divided pixel by pixel with the relative depth map at the effective depth point to obtain a sparse scale map, and the pooling weight is calculated according to the multi-scale image feature convolution, and the sparse scale map is downsampled to four different scales using weighted pooling; Decoding and completion module: takes multi-scale image features and sparse scale maps as input, performs fusion decoding for scale completion, and outputs a fine-tuned dense scale map; Pixel calculation module: performs the transformation from scale space to depth space, multiplies the dense scale map and the relative depth map obtained by the image encoding module pixel by pixel, and obtains the final dense absolute depth map; Loss supervision module: Use the relative depth map obtained by the dense depth truth and image encoding module to calculate the loss between the predicted dense absolute depth to supervise the learning of the depth completion method.

8. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the true depth completion method based on pre-training and scale recovery as described in any one of claims 1-6.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by the processor, the steps of the true depth completion method based on pre-training and scale recovery as described in any one of claims 1 to 6 are implemented.

Images