An integrated quality-aware depth map dynamic inpainting method

Abstract

This invention discloses an integrated quality-aware depth map dynamic restoration method, comprising: S1, generating an RGB cue map of an RGB image; S2, constructing a depth map quality-aware algorithm model; and S3, constructing a dynamic depth enhancement module using a reference residual network, based on the RGB cue map S generated in S1. h and the quality score g of the original depth map generated in step S2 p Perform dynamic depth map restoration to generate a restored depth map D. r S4, based on the original depth map quality score g from step S2. p And the repaired depth map D of S3 r Dynamically generate depth cues m1, m2, m3, ..., m n This invention uses depth cues as a mask and performs element-wise multiplication with features from the RGB image and the depth map output by a feature extraction network, or with a feature map obtained through multimodal fusion. This allows for feature suppression and enhancement using depth cues. This invention addresses the issue of local feature enhancement when depth map quality varies, thus improving the generalization and robustness of the RGB-D algorithm.

Description

Technical Field

[0001] This invention relates to the field of target detection and image restoration technology, and in particular to a method for dynamic restoration of depth maps. Background Technology

[0002] In RGB-D salient object detection tasks, depth map quality is crucial to the detection results. However, due to hardware limitations of depth imaging devices, environmental factors during acquisition, and computational algorithms using RGB depth estimation, the quality of obtained depth images is often inconsistent, potentially exhibiting low-quality issues such as pixel loss, artifacts, and blurring. Existing techniques typically use RGB information to directly repair the depth map or directly use depth map quality perception to influence the multimodal information fusion process.

[0003] In the process of restoration or fusion, no method can globally consider quality perception and dynamic restoration issues. Existing methods do not dynamically restore the quality of the original depth map, resulting in the loss of some spatial geometric information of the original depth map during the restoration process, which affects the fusion process of multimodal information.

[0004] Existing methods perform local feature enhancement without considering the quality of the original depth map. They only provide fixed feature enhancement algorithms, which lack generalization and robustness when the depth map quality varies in the scene. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes an integrated quality-aware dynamic depth map restoration method. This method utilizes a reference residual network to construct a dynamic depth enhancement module to generate a quality score for the original depth map, thereby performing dynamic depth map restoration. Furthermore, it combines depth cues as a mask to achieve feature enhancement of the restored depth map.

[0006] This invention is achieved using the following technical solution:

[0007] The present invention provides a depth map dynamic repair method integrating quality awareness, comprising:

[0008] S1, using a multi-level feature fusion U-Net convolutional neural network algorithm to generate RGB cue maps and edge maps of an RGB image, including obtaining two highly correlated features for salient object detection and edge detection based on the RGB cue maps. These two features are input into stacked cross-refinement units (CRUs) to output multi-level salient object cue features and edge cue features. The final salient object cue features and final edge cue features are then input into the U-Net convolutional neural network, each of which further includes an upsampling layer, a convolutional layer, and a cascaded operation, fusing them in a top-down manner to obtain the final salient object cue features and final edge cue features. By decoding the final salient object cue feature H and the final edge cue feature E, a predicted RGB cue map S is obtained. h and edge graph S e ;

[0009] S2, construct a depth map quality-aware algorithm model, which uses the deepest RGB cue map S obtained in S1 as the basis. h As input, the model is trained using deep learning methods and the quality of depth maps from the training data in the dataset. The output of the model is a generated predicted quality score g for the original depth map. p g p This reflects the potential confidence level of the depth map;

[0010] S3, a dynamic depth enhancement module is constructed using the reference residual network, based on the RGB cue map S generated in S1. h and the quality score g of the original depth map generated in step S2 p Perform dynamic depth map restoration to generate a restored depth map D. r The process includes: first, the original depth map is normalized to the interval [0,1] and input into a dynamic depth enhancement module; then, the normalized depth map and the RGB cue map S are... h All of these are fed into the dynamic depth enhancement module, which represents the input as {l d ,S h}, where l d and S h These are the input depth map and RGB cue map, respectively. The dynamic depth enhancement module outputs a single-channel repaired depth map, as shown in the following formula:

[0011]

[0012] in, and These represent element-wise addition and element-wise multiplication, respectively.

[0013] S4, based on the original depth map quality score g of S2 pAnd the repaired depth map D of S3 r Dynamically generate depth cues m1, m2, m3, ..., m n By using depth cues as a mask, element-wise multiplication is performed with the features output by the feature extraction network from the RGB image and the depth map, or the feature map after multimodal fusion, to achieve feature suppression and enhancement using depth cues.

[0014] In some embodiments, S1 further includes performing multi-level cross-fusion between salient target cue features and edge features in stacked cross-refinement units (CRUs) to obtain salient target cue features and edge cue features at various levels generated by the CRUs.

[0015] In some implementations, S1 further includes each U-Net convolutional neural network further including an upsampling layer, a convolutional layer, and a cascade operation to fuse the final salient target cue features and the final edge cue features in a top-down manner.

[0016] In some implementations, S1 further includes employing two U-Net convolutional neural networks to aggregate the final saliency target cue features and the final edge cue generated by the final CRU, respectively.

[0017] In some implementations, in S1, the salience target cue features at each level and edge hints at various levels The expression is as follows:

[0018]

[0019] in, and Let represent the saliency target cue feature and edge cue feature of the i-th level of the n-th CRU, Cat(·) represents the cascade operation along the output channel axis, Conv(·) represents a 3×3 convolutional layer with 32 output channels, CU(·) is a scaling operation and a 1×1 convolutional layer with 32 output channels, ∏ represents element-wise multiplication of cue features of all levels, and k represents the cumulative multiplication count.

[0020] In some embodiments, S2 further includes downsampling the deep features C4 and D4 of the RGB image features and depth map features output by the ResNet-like feature extraction network to map the high-dimensional top-level features C4 and D4 into one-dimensional vectors, which are respectively RGB... Figure 1 3D eigenvector f r ,depth Figure 1 3D eigenvector f dThe two concatenated vectors are input into two cascaded fully connected layers and an activation function to obtain the estimated original depth map quality score g. p .

[0021] In some implementations, S4 further includes element-wise multiplication of the depth cues with the features of the RGB image and the depth map output by a ResNet-like network or a feature map fused with multiple modalities, thereby enhancing the features using the depth cues.

[0022] In some implementations, S4 further includes using T+1 maps generated by segmenting the repaired depth map with T thresholds as depth cues, and the threshold T generated by the multi-Otsu thresholding algorithm is expressed as:

[0023]

[0024] Among them, g p The interval is [0,1], the value of β represents the hyperparameter, and α represents the maximum number of thresholds generated by the multiple Otsu thresholding algorithm.

[0025] Compared with the prior art, the present invention achieves unique and beneficial technical effects:

[0026] 1) During the depth map restoration process, the spatial and geometric information of the original depth map is preserved as much as possible based on its quality. This allows the depth map quality-aware algorithm model to better utilize the restored depth map containing the original information during the subsequent fusion of RGB and depth map information, providing better geometric and spatial prior information. Furthermore, in the subsequent local feature enhancement algorithm, local feature enhancement information is dynamically generated based on the quality score of the original depth map. This enables the restored depth map to generate dynamic depth map cues, solving the problem of local feature enhancement when the depth map quality is uneven, and enhancing the generalization and robustness of the RGB-D algorithm.

[0027] 2) The quality of the original depth map is considered globally, and the quality of the depth map is scored, which solves the problem of losing geometric information of the original depth map as much as possible during the fusion process of using RGB information to repair the depth map.

[0028] 3) Using residuals can dynamically enhance useful information in the depth map while preserving the original information. Attached Figure Description

[0029] Figure 1 This is an overall flowchart of a depth map dynamic repair method integrating quality awareness according to the present invention.

[0030] Figure 2This is a diagram illustrating the implementation process of an integrated quality-aware depth map dynamic repair method according to the present invention.

[0031] Figure 3 This is a schematic diagram of the integrated depth map quality perception and dynamic repair module of the present invention;

[0032] Figure 4 A schematic diagram of the Dynamic Depth Enhancement (DE) module;

[0033] Figure 5 This diagram illustrates the regions under multiple thresholds for the original and repaired depth maps. Specifically, it uses multiple Otsu algorithms to generate T thresholds, resulting in T+1 regions for the original and repaired depth maps. The diagram shows the case when T=2. Detailed Implementation

[0034] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0035] Example 1

[0036] like Figure 1 and Figure 2 The diagram illustrates a flow chart of a depth map dynamic repair method integrating quality awareness according to the present invention, which specifically includes the following steps:

[0037] Step S1: Generate RGB cue maps for RGB images using the multi-level feature fusion U-Net convolutional neural network algorithm; such as... Figure 3 As shown, the RGB image is processed by a ResNet-like network, which outputs four levels of raw features C1, C2, C3, and C4. These four levels of raw features correspond to four different resolutions × number of channels, and are arranged from shallow to deep. For shallow features with smaller channel numbers and higher resolutions, the deep features have larger channel numbers and lower resolutions. Then, the number of channels in each level of raw features is compressed to the same number, resulting in two features highly correlated with salient object detection and edge detection. These two features are then input into stacked Cross Refinement Units (CRUs). CRUs output multi-level cue and edge features. For example, the raw features are processed through a 1×1 and two 3×3 convolutions to reduce the number of feature channels to 32, resulting in two features highly correlated with salient object detection and edge detection: salient object cue feature H and edge cue feature E. For example, H includes features with resolutions from large to small. i ,i=1,2,3,4} and {E i{i = 1, 2, 3, 4} represent salient target cue features and edge cue features with a fixed number of channels, respectively. These features are then input into four stacked Cross Refinement Units (CRUs) to perform multi-level cross-fusion between the salient target cue features and edge features, resulting in salient target cue features at various levels generated by the CRUs. and edge hints at various levels The expression is as follows:

[0038]

[0039] in, and Let represent the saliency target cue feature and edge cue feature of the i-th level of the n-th CRU, Cat(·) represents the cascade operation along the output channel axis, Conv(·) represents a 3×3 convolutional layer with 32 output channels, CU(·) is a scaling operation and a 1×1 convolutional layer with 32 output channels, ∏ represents element-wise multiplication of cue features of all levels, and k represents the cumulative multiplication count.

[0040] Then, two U-Net convolutional neural networks are used to aggregate the final saliency target cue features generated by the fourth CRU. and final edge hint The specific processing includes: inputting the final salient target cue features and the final edge cue features into a U-Net convolutional neural network, and for each U-Net convolutional neural network, further including upsampling layers, convolutional layers, and cascading operations to fuse four layers of final salient target cue features and four layers of final edge cue features in a top-down manner; specifically, firstly, a simple bilinear upsampling layer is used to adjust the spatial resolution of the (i+1)th level to be the same as that of the i-th level; then, the output features of the upsampling layer are fed into a 3×3 convolutional layer; then, the features of the (i+1)th level are directly concatenated with the features of the i-th level to form new features of the i-th level, and these features are fed into the next bilinear upsampling process; this upsampling, convolution, and cascading operation is performed sequentially to integrate the features in a top-down manner. and The predicted RGB cue map S is obtained by decoding the final saliency target cue feature H and the final edge cue feature E. h and edge graph S e During training, the cue map and edge map use the ground truth (GT) value (G) for saliency. s ) and processing G s The edge GT(G) generated afterward e ) for supervision; before using it to enhance the depth map, Sh Normalize to the interval [0,1];

[0041] Step S2, as follows Figure 3 As shown, a depth map quality-aware algorithm model is constructed, which uses the deepest RGB cue map S obtained in step 1. h As input, the model is trained using deep learning methods and by labeling and analyzing the depth map quality of the training data in the dataset. The output of this model is a quality score g for the generated original depth map. p Specifically, to achieve depth map quality perception, a pseudo-label g is provided to measure image quality. score To guide the learning process, the trained depth map quality-aware algorithm model is used to predict the confidence level g of the depth map. p For example, the deep features C4 and D4 of the RGB image features and depth map features output by the ResNet-like feature extraction network are processed by downsampling operations (such as Global Average Pooling, GAP) to operate on each channel of the input data, mapping the high-dimensional top-level features C4 and D4 into one-dimensional vectors, which are RGB... Figure 1 3D eigenvector f r ,depth Figure 1 3D eigenvector f d The two concatenated vectors are input to two cascaded fully connected layers (FC) and activation functions (ReLU and Sigmoid, respectively), where the output of the second fully connected layer has 1 feature count, thus obtaining the estimated original depth map quality score g. p g p This reflects the potential confidence level of the depth map, when g p When g is close to 1, it means the depth map has high reliability and good quality; when g... p A value close to 0 indicates poor depth map quality.

[0042] Step S3, as follows Figure 4 As shown, a dynamic depth enhancement module (DE) is constructed using a reference residual network, based on the RGB cue map S generated in step S1. h and the quality score g of the original depth map generated in step S2 p Perform dynamic depth map restoration to generate a restored depth map D. r Specifically, the original depth map is first normalized to the interval [0,1] and input into the dynamic depth enhancement module DE; then, the normalized depth map and the RGB cue map S are... h All inputs are fed into the DE module. The input is represented as {l} d ,S h}, where l d and S h These are the input depth map and RGB cue map, respectively. The output of the DE module is a single-channel repaired depth map, as shown in the following formula:

[0043]

[0044] in, and These represent element-wise addition and element-wise multiplication, respectively. The output of DE is a single-channel repaired depth map. The basic principle behind using this residual form is that it is combined with the second term (i.e., simple element-wise multiplication). Compared to the first item, l d Features that might be incorrectly suppressed were added. Simultaneously, g was predicted based on depth map awareness. p It can dynamically repair the depth map based on the quality of the original depth map to ensure it is better suited for salient object detection tasks. When the depth map quality is good, information will not be lost after using the depth map enhancement module. Therefore, this dynamic residual form can dynamically enhance useful information in the depth map while preserving the original information.

[0045] Step S4, based on the original depth map quality score g from step S2 p And the depth map D after repair in step S3 r Dynamically generate depth cues m1, m2, m3, ..., m n This utilizes geometric prior information and spatial information from depth maps to enhance local features. For example, ... Figure 5 As shown in the repair depth map l enhence Using a multi-Otsu thresholding algorithm, T thresholds are generated. The resulting T+1 images, segmented using these T thresholds, can serve as depth cues. During training, initialization is performed with T=3. During both training and inference, after passing through an integrated depth map quality awareness and dynamic inpainting module, a repaired depth map (l) is obtained. enhence And depth map quality perception score g p , The number of thresholds generated by the multi-Otsu thresholding algorithm during training and inference is expressed by the formula:

[0046]

[0047] Among them, g p The interval is [0,1], where β represents a hyperparameter and α represents the maximum number of thresholds generated by the multiple Otsu thresholding algorithm. pThe higher the value, the more depth cues are generated. Depth cues can be used as masks to suppress or enhance features. For example, depth cues can be used as masks to perform element-wise multiplication with features output from a ResNet-like network or a feature map fused from multiple modalities, thereby enabling the suppression and enhancement of features using depth cues.

[0048] Those skilled in the art can make improvements and variations based on the above-described invention. Any modifications, improvements, and variations made on the basis of this invention should fall within the protection scope of this invention.

Claims

1. A depth map dynamic repair method integrating quality awareness, characterized in that, include: S1, using a multi-level feature fusion U-Net convolutional neural network algorithm to generate RGB cue maps and edge maps of an RGB image, including obtaining two highly correlated features for salient object detection and edge detection based on the RGB cue maps. These two features are input into stacked cross-refinement units (CRUs) to output multi-level salient object cue features and edge cue features. The final salient object cue features and final edge cue features are then input into the U-Net convolutional neural network, each of which further includes an upsampling layer, a convolutional layer, and a cascaded operation, fusing them in a top-down manner to obtain the final salient object cue features and final edge cue features. By decoding the final salient object cue feature H and the final edge cue feature E, a predicted RGB cue map S is obtained. h and edge graph S e ; S2, construct a depth map quality-aware algorithm model, which uses the deepest RGB cue map S obtained in S1 as the basis. h As input, the model is trained using deep learning methods and the quality of depth maps from the training data in the dataset. The output of the model is a generated predicted quality score g for the original depth map. p g p This reflects the potential confidence level of the depth map; S3, a dynamic depth enhancement module is constructed using the reference residual network, based on the RGB cue map S generated in S1. h and the quality score g of the original depth map generated in step S2 p Perform dynamic depth map restoration to generate a restored depth map D. r ; include First, the original depth map is normalized to the interval [0,1] and then input into the dynamic depth enhancement module; Then, the normalized depth map and RGB cue map S h All of these are fed into the dynamic depth enhancement module, which represents the input as {l d ,S h }, where l d and S h These are the input depth map and RGB cue map, respectively. The dynamic depth enhancement module outputs a single-channel repaired depth map, as shown in the following formula: in, and These represent element-wise addition and element-wise multiplication, respectively. S4, based on the original depth map quality score g of S2 p And the repaired depth map D of S3 r Dynamically generate depth cues m1, m2, m3, ..., m n By using depth cues as a mask, element-wise multiplication is performed with the features output by the feature extraction network from the RGB image and the depth map, or the feature map after multimodal fusion, to achieve feature suppression and enhancement using depth cues.

2. The depth map dynamic repair method integrating quality awareness according to claim 1, characterized in that, S1 further includes performing multi-level cross-fusion between salient target cue features and edge features in stacked cross-refinement units (CRUs) to obtain salient target cue features and edge cue features at each level generated by the CRUs.

3. The depth map dynamic repair method integrating quality awareness according to claim 1, characterized in that, S1 further includes each U-Net convolutional neural network further including an upsampling layer, a convolutional layer, and a cascade operation, which fuses the final salient target cue features and the final edge cue features in a top-down manner.

4. The depth map dynamic repair method integrating quality awareness according to claim 1, characterized in that, S1 further includes using two U-Net convolutional neural networks to aggregate the final saliency target cue features and the final edge cue generated by the final CRU, respectively.

5. The depth map dynamic repair method integrating quality awareness according to claim 1, characterized in that, In S1, the salience of target cue features at each level and edge hints at various levels The expression is as follows: in, and Let represent the saliency target cue feature and edge cue feature of the i-th level of the n-th CRU, Cat(·) represents the cascade operation along the output channel axis, Conv(·) represents a 3×3 convolutional layer with 32 output channels, CU(·) is a scaling operation and a 1×1 convolutional layer with 32 output channels, ∏ represents element-wise multiplication of cue features of all levels, and k represents the cumulative multiplication count.

6. The depth map dynamic repair method integrating quality awareness according to claim 1, characterized in that, S2 further includes, after passing through a ResNet-like feature extraction network, deep features C4 and D4 of the RGB image features and depth map features are downsampled to map the high-dimensional top-level features C4 and D4 into one-dimensional vectors, which are respectively the one-dimensional feature vectors f of the RGB image. r One-dimensional feature vector f of the depth map d The two concatenated vectors are input into two cascaded fully connected layers and an activation function to obtain the estimated original depth map quality score g. p .

7. The depth map dynamic repair method integrating quality awareness according to claim 1, characterized in that, S4 further includes element-wise multiplication of the depth cues with the features output by the RGB image and the depth map after passing through a ResNet-like network or a feature map fused with multiple modalities, thereby realizing feature enhancement using depth cues.

8. The depth map dynamic repair method integrating quality awareness according to claim 1, characterized in that, S4 further includes using T+1 maps generated by segmenting the repaired depth map with T thresholds as depth cues, and the threshold T generated by the multi-Otsu thresholding algorithm is expressed as: Among them, g p The interval is [0,1], the value of β represents the hyperparameter, and α represents the maximum number of thresholds generated by the multiple Otsu thresholding algorithm.

Images