Learning to Guide Deformable Convolutions for Depth Completion

Abstract

The present invention relates to a depth completion method of learning guided deformable convolution, belongs to the field of computer vision, and comprises the following steps: S1: given a pair of inputs, including a sparse depth map and an RGB image; S2: extracting depth features and image features from the sparse depth map and the RGB image respectively, and further fusing them to obtain multimodal features; S3: taking the multimodal features as input, using image feature information to adaptively guide depth feature aggregation, and finally predicting a rough depth map; S4: generating a depth residual map through adaptively predicted spatial changes and content-related kernel weights and offsets, and further obtaining a refined depth map.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and relates to a depth completion method of learning guided deformable convolution. Background Art

[0002] At present, deep learning-based methods have shown excellent performance in depth completion tasks. These methods gradually map sparse depth maps to dense depth maps through a large number of stacked filters. Since RGB images contain rich semantic clues, image-guided methods have shown unimaginable performance in filling unknown depths. For example, GuideNet proposed a feature fusion module based on a guided dynamic convolutional network to better utilize the guided features of RGB images; CSPN studied affinity matrices to refine coarse depth maps through spatial propagation networks (SPNs); DySPN further developed dynamic SPNs by assigning different attention levels to neighboring pixels at different distances. ACMNet introduced a symmetric gated fusion strategy to perform the fusion of two modal data, RGB image features and depth features; FCFR-Net proposed channel shuffling and energy-based fusion strategies between the two modalities.

[0003] Although existing methods have made significant progress in the depth completion task, there are still some problems that need to be solved. Due to the fixed geometric structure of the convolution module, the convolution unit can only sample the input feature map at a fixed position, which may contain irrelevant mixed information. Therefore, in challenging environments and sparse depth measurements, existing methods have difficulty in producing clear structural features, resulting in the problem of deep mixing, that is, the depth map boundaries are blurred and artifacts appear. In addition, due to the huge differences in depth and RGB image information, in order to fully fuse multimodal data, existing methods usually use tens of millions of learnable parameters to ensure the ability of the model to learn robust features. However, such large-scale networks usually require a lot of computing resources, which is impractical in practical applications. Simply reducing the size of the network will significantly reduce the performance of the method. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an image guidance module that can adaptively perceive the context structure of each pixel to assist the depth completion process. At the same time, in order to reduce the complexity of the model, a low-coupling and lightweight network structure is provided to complete this task.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A depth completion method for learning guided deformable convolutions, comprising the following steps:

[0007] S1: Given a pair of inputs, including a sparse depth map and an RGB image;

[0008] S2: Extract depth features and image features from the sparse depth map and RGB image respectively, and further fuse them to obtain multimodal features;

[0009] S3: Taking multimodal features as input, we use image feature information to adaptively guide deep feature aggregation and finally predict a rough depth map.

[0010] S4: Generate a depth residual map by adaptively predicting spatial changes and content-related kernel weights and offsets, and further obtain a refined depth map.

[0011] Furthermore, in step S2, for a stack of input sparse depth maps S and RGB images I, an initial convolutional layer and a lightweight encoder-decoder process are performed on S and I respectively to obtain the depth feature F S′ and image features F I′ , then, these two features are fused through pixel-level addition operation to obtain the multimodal feature F IS , expressed as:

[0012]

[0013]

[0014] F IS =F I ′+F S '

[0015] in and They represent the encoder-decoder for extracting image features and deep features respectively.

[0016] Further, in step S3, two encoder-decoder branches are constructed: an image-guided branch and a deep regression branch, both of which take multimodal features as input; the encoder stage of the image-guided branch receives the input from The decoder’s skip connection realizes information transfer; the encoder stage of the deep regression branch receives The jump connection of the decoder realizes information transfer; a deformable guidance module is embedded after the encoder scale features of the deep regression branch to aggregate relevant semantic information from the neighborhood range. The output of the deep regression branch is a rough depth map.

[0017] Further, the deformable guiding module processing steps are as follows:

[0018] Given the image features f from the image guidance branch i and the deep features f from the deep regression branch s ,First, pixel-level addition is performed to fuse image features and depth features;

[0019] Next, a standard convolution operation is performed to learn the pixel-level offset feature map, which includes the offsets in the x and y coordinate directions, representing the position deviation on the regular grid, with a total of 2×k 2 channels, where k is the convolution kernel size;

[0020] Then, the deep feature map is sampled on a regular grid based on these offsets to obtain relevant semantic information in the neighborhood;

[0021] Subsequently, a standard convolution with a kernel size of k is performed on the sampled features to aggregate this information and learn the deep feature residuals;

[0022] Finally, the deep feature residual is added to the deep feature to get the final output;

[0023] The deformable guiding module is described as follows:

[0024] Offsets = Conv(f s +f i )

[0025] Output = f i +DeConv(f i ,Offsets)

[0026] DeConv() represents deformable convolution, whose kernel weights are shared in space and obtained by random initialization.

[0027] Further, step S4 specifically includes:

[0028] Given the last layer features F of the image guided branch decoder and the deep regression branch decoder respectively I and F S ;

[0029] First, a pixel-level addition is performed to fuse the dual-branch features;

[0030] Then, two separate standard convolutions are used to learn the pixel-level convolution kernel weights and offset feature maps.

[0031] The sigmoid layer makes the weight greater than 0 and less than 1, and subtracts its mean to make the sum of the weights 0;

[0032] Based on the given offset and kernel weight, a deformable convolution is performed on the rough depth map CD to obtain the depth residual map ΔD;

[0033] Finally, the rough depth map is added to the depth residual map to obtain the final fine depth map D;

[0034] The specific expression is:

[0035] Weights=Conv(F I +F S )

[0036] Offsets=Conv(F I +F S )

[0037] ΔD=DeConv(D0,Weights,Offsets)

[0038] D=CD+ΔD

[0039] DeConv() represents deformable convolution, whose kernel weights are spatially varying and content-dependent, and are learned from image-guided features and deep features.

[0040] Furthermore, the mean square error (MSE) is used to calculate the loss between the true depth and the predicted depth, which is expressed as:

[0041] L(D pred )=‖(D pred -D gt )⊙m( Dg t>0)‖ 2

[0042] Where D pred represents the predicted depth map, D gt represents the true depth map used for supervision, ⊙ refers to element-wise multiplication, and only pixels with valid depth values ​​are considered;

[0043] The rough depth map CD also needs to be supervised, so the final loss function is:

[0044] Loss=L(D)+λL(CD)

[0045] where λ is a hyperparameter set empirically.

[0046] The beneficial effects of the present invention are as follows: the present invention enables the depth map to better capture relevant information through a method that can adaptively perceive the context structure of each pixel, thereby significantly improving the structural clarity of the generated depth map; the present invention adaptively adjusts the perception range according to the pixel position to ensure that the depth map no longer contains irrelevant information, effectively avoiding the aggregation of irrelevant information; the present invention predicts the sampling position of depth-related information by making full use of the rich semantic information of RGB, avoids the irrelevant information brought by the fixed local neighborhood, improves the accuracy of the depth map, and performs better in the face of complex and challenging scenes; the low-coupling and lightweight network structure provided by the present invention not only improves the computing efficiency and reduces the resource requirements, but also makes the depth completion method easier to deploy and promote in various application scenarios; the present invention introduces a dual-branch stacked hourglass-based network structure, decouples the single encoder structure of the previous method into a stacked encoder-decoder structure, balances the learning of the model, and gradually obtains clearer and richer context semantics, and each encoder-decoder maintains a very lightweight feature.

[0047] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0049] Figure 1 Flowchart of the depth completion method for learning guided deformable convolution;

[0050] Figure 2 Flowchart for multimodal feature extraction;

[0051] Figure 3 To guide the deep regression flow chart;

[0052] Figure 4 It is a structural diagram of the deformable guidance module;

[0053] Figure 5 To deeply refine the flow chart. DETAILED DESCRIPTION

[0054] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0055] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0056] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0057] In the face of challenging environments and sparse depth measurements, traditional depth completion methods, especially fixed neighborhood convolution modules and affinity matrices, are difficult to meet the needs of depth completion tasks. There is a problem with current methods, that is, during the depth map generation process, due to the fixed neighborhood operation, it is easy to aggregate mixed information that is not related to the surroundings, resulting in a fuzzy structure of the generated feature map. Therefore, the present invention aims to solve this problem and focus on improving the structural clarity of the depth map.

[0058] To achieve the above objectives, the present invention designs an image guidance module that can adaptively perceive the context structure of each pixel. The module can adaptively adjust the perception range according to the pixel position to better capture relevant information, thereby improving the accuracy and clarity of the depth map. The introduction of this module aims to effectively deal with complex and challenging scenes in the depth completion process, ensuring that the generated depth feature map has a stronger sense of structure. Specifically, in order to avoid irrelevant information brought by a fixed local neighborhood, the rich semantic information of RGB is fully utilized, and the sampling position of depth-related information is predicted, where the sampling position is obtained by learning the offset of a regular grid. In order to meet the requirements of practical applications for model complexity, a low-coupling and lightweight network structure is designed. This lightweight design not only helps to improve computational efficiency and reduce resource requirements, but also makes the depth completion method easier to deploy and promote in various application scenarios.

[0059] The specific network structure includes a dual-branch stacked hourglass-based network structure, which decouples the single encoder structure of the previous method into a stacked encoder-decoder structure to balance the learning of the model while gradually obtaining clearer and richer contextual semantics. Due to the decoupled structure, a large number of learnable parameters are not required to balance the learning ability of the model, thereby maintaining the robustness of the model, making each encoder-decoder very lightweight.

[0060] Based on the above scheme, a depth completion method for learning guided deformable convolution is provided, such as Figure 1 As shown, the method pipeline includes three steps: multimodal feature extraction, guided deep regression and deep refinement.

[0061] Multimodal feature extraction: Figure 2 As shown in the figure, given a pair of inputs, a sparse depth map S and an RGB image I, we aim to extract a multimodal representation that seamlessly integrates the image's texture, edge and other semantic information as well as depth information. By performing an initial convolutional layer and a lightweight encoder-decoder on S and I respectively, we obtain the deep feature F S′ and image features F I′ Then, these two features are fused through pixel-level addition operation to obtain the multimodal feature F IS Specifically, the process can be described as:

[0062]

[0063]

[0064] F IS =F I ′+F S '

[0065] in and They represent the encoder-decoder modules for extracting image features and deep features, respectively.

[0066] Guided deep regression: Figure 3 As shown in , this stage is divided into two encoder-decoder branches: image guidance branch and depth regression branch. The encoder stage of the image guidance branch accepts The decoder’s skip connections enable information transfer. The encoder stage of the deep regression branch also uses skip connections, this time from A deformable guidance module is embedded after the encoder scale features of the deep regression branch to aggregate relevant semantic information from the neighborhood. The output of this branch is a rough depth map CD. Both branches are based on multimodal features F IS As input, it ensures that multi-source information is fully utilized for depth completion in subsequent steps.

[0067] like Figure 4 As shown in the figure, the design of the deformable guidance module is as follows: given the image feature f from the image guidance branch i and the deep features f from the deep regression branch s First, pixel-level addition is performed to fuse image features and depth features. Next, a standard convolution operation is used to learn a pixel-level offset feature map, which includes offsets in the x and y coordinate directions, representing position deviations on a regular grid, with a total of 2×k 2 channels, where k is the convolution kernel size. Then, the deep feature map is sampled on a regular grid based on these offsets to obtain relevant semantic information in the neighborhood. Subsequently, a standard convolution with a kernel size of k is performed on the sampled features to aggregate this information and learn the deep feature residuals. Finally, the deep feature residuals are added to the deep features to obtain the final output. Specifically, this module can be expressed as:

[0068] Offsets = Conv(f s +f i )

[0069] Output=f+DeConv(f i ,Offsets)

[0070] Among them, DConv() represents deformable convolution, whose kernel weights are shared in space and obtained by random initialization.

[0071] Depth Refinement: Figure 5 As shown, the last layer features F of the image guided branch decoder and the deep regression branch decoder are given respectively I and F S. First, a pixel-level addition is performed to fuse the dual-branch features, and then the pixel-level convolution kernel weights and offset feature maps are learned through two separate standard convolutions. In order to make the model converge stably, the weights are made greater than 0 and less than 1 through the sigmoid layer, and their mean is subtracted to make the sum of the weights 0. Then, based on the given offset and kernel weights, a deformable convolution is performed on the rough depth map CD to obtain the depth residual map ΔD. Finally, the rough depth map is added to the depth residual map to obtain the final refined depth map D. Specifically, the process can be expressed as,

[0072] Weights=Conv(F I +F S )

[0073] Offsets=Conv(F I +F S )

[0074] ΔD=Deconv(D0,Weights,Offsets)

[0075] D=CD+ΔD

[0076] DeConv() represents deformable convolution, whose kernel weights are spatially varying and content-dependent, and are learned from image-guided features and deep features.

[0077] The mean square error (MSE) is used to calculate the loss between the true depth and the predicted depth during training. For real-world depth data, the true depth is usually semi-dense because it is difficult to collect the true depth of each pixel. Therefore, when calculating the training loss, only the valid pixels in the true depth map are considered. Therefore, the loss function can be expressed as:

[0078] L(D pred )=‖(D pred -D gt )⊙m(D gt >0)‖ 2

[0079] Where D pred represents the predicted depth map, D gt represents the true depth map used for supervision, and ⊙ refers to element-wise multiplication. Since the true depth contains invalid pixels, we only consider pixels with valid depth values.

[0080] It is also necessary to supervise the intermediate depth prediction results (rough depth map CD), so the final loss function is:

[0081] Loss=L(D)+λL(CD)

[0082] Here, λ is a hyperparameter set empirically and is recommended to be set to 0.2.

[0083] A person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. When the program is executed, the steps of the method can be implemented. The storage medium, such as ROM / RAM, a disk, an optical disk, etc.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A depth completion method for learning guided deformable convolution, characterized by: The following steps are involved: S1: Given a pair of inputs, including a sparse depth map and an RGB image; S2: Extract depth features and image features from the sparse depth map and RGB image respectively, and further fuse them to obtain multimodal features; S3: Taking multimodal features as input, using image feature information to adaptively guide deep feature aggregation, and finally predicting a rough depth map, specifically including: constructing two encoder-decoder branches: an image guidance branch and a depth regression branch, both of which take multimodal features as input; the encoder stage of the image guidance branch receives information from The decoder’s skip connection realizes information transfer; the encoder stage of the deep regression branch receives The jump connection of the decoder realizes information transfer; a deformable guidance module is embedded after the encoder scale features of the deep regression branch to aggregate relevant semantic information from the neighborhood range. The output of the deep regression branch is a rough depth map; the processing steps of the deformable guidance module are as follows: Given the image features f from the image guidance branch i and the deep features f from the deep regression branch s ,First, pixel-level addition is performed to fuse image features and depth features; Next, a standard convolution operation is performed to learn the pixel-level offset feature map, which includes the offsets in the x and y coordinate directions, representing the position deviation on the regular grid, with a total of 2×k 2 channels, where k is the convolution kernel size; Then, the deep feature map is sampled on a regular grid based on these offsets to obtain relevant semantic information in the neighborhood; Subsequently, a standard convolution with a kernel size of k is performed on the sampled features to aggregate this information and learn the deep feature residuals; Finally, the deep feature residual is added to the deep feature to get the final output; The deformable guiding module is described as follows: Offsets=Conv(f s +f i ) Output=f i +DeConv(f i ,Offsets) DeConv() represents deformable convolution, whose kernel weights are shared in space and obtained by random initialization; S4: Generate a depth residual map through adaptively predicted spatial changes and content-related kernel weights and offsets, and further obtain a refined depth map; specifically, it includes: Given the last layer features F of the image guided branch decoder and the deep regression branch decoder respectively I and F S ; First, a pixel-level addition is performed to fuse the dual-branch features; Then, two separate standard convolutions are used to learn the pixel-level convolution kernel weights and offset feature maps. The sigmoid layer makes the weight greater than 0 and less than 1, and subtracts its mean to make the sum of the weights 0; Based on the given offset and kernel weight, a deformable convolution is performed on the rough depth map CD to obtain the depth residual map ΔD; Finally, the rough depth map is added to the depth residual map to obtain the final refined depth map D; The specific expression is: Weights=Conv(F I +F S ) Offsets=Conv(F I +F S ) ΔD=DeConv(D0,Weightsc,Offsets) D=CD+△D DeConv() represents deformable convolution, whose kernel weights are spatially varying and content-dependent, and are learned from image-guided features and deep features.

2. The depth completion method of learning guided deformable convolution according to claim 1, characterized in that: In step S2, for a stack of input sparse depth maps S and RGB images I, a depth feature F is obtained by performing an initial convolutional layer and a lightweight encoder-decoder process on S and I respectively. S′ and image features F I′ , then, these two features are fused through pixel-level addition operation to obtain the multimodal feature F IS , expressed as: F IS =F I′ +F S′ in and They represent the encoder-decoder used to extract image features and depth features respectively.

3. The depth completion method of learning guided deformable convolution according to claim 1, characterized in that: The mean square error (MSE) is used to calculate the loss between the true depth and the predicted depth, which can be expressed as: L(D pred )=||(D pred -D gt )⊙m(D gt >0)|| 2 Where D pred represents the predicted depth map, D gt represents the true depth map used for supervision, ⊙ refers to element-wise multiplication, and only pixels with valid depth values ​​are considered; The rough depth map CD also needs to be supervised, so the final loss function is: Loss=L(D)+λL(CD) where λ is a hyperparameter set empirically.

Images