Depth map dynamic restoration method integrating quality perception

Abstract

The invention discloses a depth map dynamic restoration method integrated with quality perception. The method comprises the following steps: S1, generating an RGB prompt graph of an RGB image; s2, constructing a depth map quality perception algorithm model; s3, constructing a dynamic depth enhancement module by using a reference residual network, and carrying out dynamic depth image restoration according to the RGB prompt image Sh generated in the S1 and the original depth image quality score gp generated in the S2 to generate a restored depth image Dr; and S4, dynamically generating depth cues m1, m2, m3,..., mn according to the quality score gp of the original depth image in the step S2 and the repaired depth image Dr in the step S3, taking the depth cues as masks, and performing element multiplication with features output by the RGB image and the depth image through the feature extraction network or a feature image after multi-modal fusion, thereby realizing suppression and enhancement of the features by using the depth cues. According to the invention, the local feature enhancement of the algorithm when the quality of the depth map is uneven is realized, and the generalization and robustness of the RGB-D algorithm are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection and image restoration, and in particular to a depth map dynamic restoration method. Background Art

[0002] In the RGB-D salient object detection task, the quality of the depth map is crucial to the detection results. However, due to the hardware limitations of the depth imaging device, the influence of environmental factors during the acquisition process, and the calculation algorithm of the RGB depth estimation, the quality of the obtained depth image is often uneven, and low-quality problems such as pixel value loss, artifacts, and blur may occur. Existing technologies usually use RGB information to directly repair the depth map or directly use the depth map quality perception to affect the multi-modal information fusion process.

[0003] In the restoration or fusion process, there is no method that can globally consider the quality perception and dynamic restoration issues. Existing methods do not dynamically restore the quality of the original depth map, resulting in the loss of some of the spatial geometric information of the original depth map during the restoration process, affecting the fusion process of multimodal information.

[0004] Existing methods do not consider the quality of the original depth map during the local feature enhancement process, and only provide a fixed feature enhancement algorithm during local feature enhancement. When the quality of the depth map in the scene is uneven, the algorithm lacks generalization and robustness. Summary of the invention

[0005] In view of the existing technology and its defects, the present invention aims to propose a quality-aware dynamic depth map restoration method, which uses a reference residual network to construct a dynamic depth enhancement module to generate an original depth map quality score for dynamic depth map restoration, and then combines depth cues as a mask to achieve feature enhancement of the repaired depth map.

[0006] The present invention is achieved by the following technical solutions:

[0007] A method for dynamic restoration of depth images with integrated quality perception of the present invention comprises:

[0008] S1, using a multi-level feature fusion U-Net convolutional neural network algorithm to generate an RGB prompt map and an edge map of an RGB image, including obtaining two highly correlated features of salient target detection and edge detection according to the RGB prompt map, and inputting the two features into stacked cross-refinement units CRUs to output multi-level salient target prompt features and edge prompt features; respectively inputting the final salient target prompt features and the final edge prompt features into the U-Net convolutional neural network, and for each U-Net convolutional neural network further including an upsampling layer, a convolution layer and a cascade operation, respectively fusing in a top-down manner to obtain the final salient target prompt features and the final edge prompt features; decoding the final salient target prompt features H and decoding the final edge prompt features E to obtain a predicted RGB prompt map S h And the edge graph S e ;

[0009] S2, builds a depth map quality perception algorithm model, which uses the deepest RGB hint map S obtained in S1 h As input, the model is trained using deep learning methods and the depth map quality of the training data in the dataset. The output of the model is to generate a predicted raw depth map quality score g p , g p Reflects the potential confidence of the depth map;

[0010] S3, using the reference residual network to build a dynamic depth enhancement module, based on the RGB prompt map S generated in S1 h and the original depth map quality score g generated in step S2 p Perform dynamic depth map repair to generate the repaired depth map D r ; Including, first the original depth map is normalized to the interval [0,1] and input into the dynamic depth enhancement module; then, the normalized depth map and RGB prompt map S h are all input into the dynamic depth enhancement module, and the input is represented as {l d ,S h}, where l d and S h They are the input depth map and RGB prompt map respectively, and the dynamic depth enhancement module outputs a single-channel repaired depth map, as shown in the following formula:

[0011]

[0012] in, and Represent element-wise addition and element-wise multiplication respectively;

[0013] S4, based on the original depth map quality score g of S2 pAnd the repaired depth map D of S3 r , dynamically generate depth clues m1,m2,m3,..,.m n , the depth clues are used as masks, and the features of the RGB image and the depth map output by the feature extraction network or the feature map after multimodal fusion are element-wise multiplied to achieve the suppression and enhancement of features using depth clues.

[0014] In some embodiments, S1 further includes, in stacked cross refinement units CRUs, performing multi-level cross fusion between salient object hint features and edge features to obtain salient object hint features of each level and edge hints of each level generated by the CRU.

[0015] In some embodiments, the S1 further includes that each U-Net convolutional neural network further includes an upsampling layer, a convolution layer and a cascade operation, which respectively fuses the final salient target hint feature and the final edge hint feature in a top-down manner.

[0016] In some embodiments, the S1 further includes using two U-Net convolutional neural networks to respectively aggregate the final salient object hint features and the final edge hints generated by the last CRU.

[0017] In some embodiments, in S1, each level of salient target prompt features and edge hints at all levels The expression is as follows:

[0018]

[0019] in, and represents the salient target cue features and edge cue features of the i-th level of the n-th CRU, Cat(·) represents the cascade operation along the output channel axis, Conv(·) represents a 3×3 convolutional layer with 32 output channels, CU(·) is a scale transformation operation and a 1×1 convolutional layer with 32 output channels, ∏ represents element-wise multiplication of cue features of all levels, and k represents the number of cumulative multiplications.

[0020] In some embodiments, S2 further includes: after the output of the ResNet-like feature extraction network, the deep features C4 and D4 of the RGB image features and the depth map features are downsampled, and the high-dimensional top-level features C4 and D4 feature data are mapped into one-dimensional vectors, which are RGB Figure 1 dimensional feature vector f r ,depth Figure 1 dimensional feature vector f d, the two cascaded vectors are input into two series-connected fully connected layers and activation functions to obtain the estimated original depth map quality score g p .

[0021] In some embodiments, S4 further includes element-wise multiplication of the depth clues with the features of the RGB image and the depth map output by a ResNet-like network or a feature map after multimodal fusion, thereby enhancing the features using the depth clues.

[0022] In some embodiments, S4 further includes segmenting the repaired depth map with T thresholds to generate T+1 maps as depth clues, and generating a threshold T through a multi-Otsu threshold algorithm, which is expressed as:

[0023]

[0024] Among them, g p The interval is [0,1], the value of β represents the hyperparameter, and α represents the maximum number of thresholds generated by the multi-Otsu threshold algorithm.

[0025] Compared with the prior art, the present invention has achieved unique and beneficial technical effects:

[0026] 1) In the process of depth map restoration, according to the quality of the original depth map, the spatial and geometric information of the original depth map is retained as much as possible, so that in the subsequent fusion process of RGB information and depth map information, the restored depth map containing the original information is better utilized based on the depth map quality perception algorithm model, so that the restored depth map provides better geometric prior information and spatial information; and in the subsequent local feature enhancement algorithm, local feature enhancement information is dynamically generated, and according to the quality score of the original depth map, the restored depth map generates dynamic depth map clues, which solves the problem of local feature enhancement when the quality of the depth map is uneven, and enhances the generalization and robustness of the RGB-D algorithm;

[0027] 2) The quality of the original depth map is considered globally, and the depth map quality is scored, which solves the problem of geometric information loss of the original depth map as much as possible in the process of fusion of the depth map repaired using RGB information;

[0028] 3) The use of residual form can dynamically enhance the useful information in the depth map while maintaining the original information. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is an overall flow chart of a method for dynamic restoration of depth images with integrated quality perception according to the present invention.

[0030] Figure 2It is a diagram of the implementation process of an algorithm of a depth image dynamic restoration method with integrated quality perception of the present invention;

[0031] Figure 3 It is a schematic diagram of the integrated depth map quality perception and dynamic restoration module of the present invention;

[0032] Figure 4 It is a schematic diagram of the dynamic depth enhancement module (DE);

[0033] Figure 5 It is a schematic diagram of the regions under multiple thresholds of the original image and the repaired depth map. Specifically, multiple Otsu algorithms are used to generate T thresholds so that the original depth map and the repaired depth map generate T+1 regions. The figure shows the situation when T=2. DETAILED DESCRIPTION

[0034] The technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0035] Embodiment 1

[0036] like Figure 1 and Figure 2 The figure shows a flow chart of a method for dynamic restoration of depth images with integrated quality perception according to the present invention, which specifically includes the following steps:

[0037] Step S1, using a multi-level feature fusion U-Net convolutional neural network algorithm to generate an RGB prompt map of the RGB image; Figure 3 As shown, the RGB image is output through a ResNet-like network with 4 levels of original features C1, C2, C3, and C4. These 4 levels of original features correspond to feature maps of 4 different resolutions × channel numbers, and there are 4 levels of original features from shallow to deep levels, for shallow features with small channel numbers / large resolutions, deep features with large channel numbers / small resolutions; then the number of channels of the original features at each level is compressed to the same number, and two features highly correlated with salient target detection and edge detection are obtained, and then these two features are input into the stacked cross refinement units (CRUs). CRUs output multi-level prompts and edge features. For example, the original features are uniformly reduced to 32 feature channels through a 1×1 and two 3×3 convolutions, and two features highly correlated with salient target detection and edge detection are obtained, namely, salient target prompt feature H and edge prompt feature E, for example, including {H from large to small resolution}. i ,i=1,2,3,4} and {E i, i=1,2,3,4} represent the salient target hint features and edge hint features of a fixed number of channels respectively, and then these features are input into four stacked cross refinement units (CRU), so that the salient target hint features and edge features are cross-fused at multiple levels to obtain the salient target hint features of each level generated by CRU and edge hints at all levels The expression is as follows:

[0038]

[0039] in, and represents the salient target cue features and edge cue features of the i-th level of the n-th CRU, Cat(·) represents the cascade operation along the output channel axis, Conv(·) represents a 3×3 convolutional layer with 32 output channels, CU(·) is a scale transformation operation and a 1×1 convolutional layer with 32 output channels, ∏ represents element-wise multiplication of cue features of all levels, and k represents the number of cumulative multiplications.

[0040] Then two U-Net convolutional neural networks are used to aggregate the final salient target hint features generated by the fourth CRU. And the final edge tip The specific processing includes: inputting the final salient target cue features and the final edge cue features into the U-Net convolutional neural network respectively, and further including an upsampling layer, a convolution layer and a cascade operation for each U-Net convolutional neural network, and fusing the 4 layers of final salient target cue features and the 4 layers of final edge cue features in a top-down manner respectively; specifically, firstly using a simple bilinear upsampling layer to adjust the spatial resolution of the i+1th level to be the same as that of the i-th level; then, feeding the output features of the upsampling layer into a 3×3 convolutional layer; then, directly connecting the i+1th level features with the i-th level features to form new i-th level features, which are fed into the next bilinear upsampling process; performing such upsampling, convolution and cascade operations in sequence, integrating in a top-down manner and By decoding the final salient target cue feature H and decoding the final edge cue feature E, a predicted RGB cue map S is obtained. h And the edge graph S e During the training process, the hint map and edge map use the saliency GT (Ground Truth) value (G s ) and process G s The edge GT(G e ) for supervision; before using it to enhance the depth map, Sh Normalized to the interval [0,1];

[0041] Step S2, such as Figure 3 As shown, a depth map quality perception algorithm model is constructed, which uses the deepest RGB prompt map S obtained in step 1 h As input, the deep learning method is used to annotate and train the depth map quality of the training data in the dataset. The output of the model is to generate the original depth map quality score g p Specifically, in order to achieve depth map quality perception, by providing a pseudo label g that measures image quality score To guide the learning process, the trained depth map quality perception algorithm model is used to predict the confidence g of the depth map p For example, the deep features C4 and D4 of the RGB image features and depth map (Depth) features output by the ResNet-like feature extraction network are downsampled (such as the global average pooling operation, GlobalAverage Pooling, GAP) to operate on each channel of the input data and map the high-dimensional top-level features C4 and D4 feature data into one-dimensional vectors, which are RGB Figure 1 dimensional feature vector f r ,depth Figure 1 dimensional feature vector f d , the two cascaded vectors are input into two series-connected fully connected layers (FC) and activation functions (Relu and Sigmoid, respectively), where the number of features output by the second fully connected layer is 1, thereby obtaining the estimated original depth map quality score g p , g p reflects the potential confidence of the depth map. p When it is close to 1, it means that the depth map is highly reliable and of good quality. p When it is close to 0, it means the depth map quality is poor;

[0042] Step S3, such as Figure 4 As shown, a dynamic depth enhancement module (DE) is constructed using the reference residual network, according to the RGB prompt map S generated in step S1. h and the original depth map quality score g generated in step S2 p Perform dynamic depth map repair to generate the repaired depth map D r Specifically, the original depth map is first normalized to the interval [0,1] and input into the dynamic depth enhancement module DE; then, the normalized depth map and RGB hint map S h are all input to the DE module. The input is represented as {l d ,S h}, where l d and S h They are the input depth map and RGB hint map respectively, and the output of the DE module is a single-channel restored depth map. As shown in the following formula:

[0043]

[0044] in, and denote element-wise addition and element-wise multiplication, respectively. The output of DE is a single-channel inpainted depth map. The rationale for using this residual form is that, compared to the second term (i.e., simple element-wise multiplication ) compared to the first term l d It supplements the features that may be suppressed by mistake. At the same time, g is obtained based on the depth map perception prediction p , according to the quality of the original depth map, the depth map can be dynamically repaired to ensure that it is more suitable for the salient object detection task. When the depth map quality is good, the information will not be lost after using the depth map enhancement module. Therefore, this dynamic residual form can dynamically enhance the useful information in the depth map while maintaining the original information;

[0045] Step S4: Based on the quality score g of the original depth map in step S2 p and the restored depth map D of step S3 r , dynamically generate depth clues m1,m2,m3,..,.m n , using the geometric prior information and spatial information in the depth map to enhance local features. Figure 5 As shown, the repair depth map l enhence Using the multi-Otsu threshold algorithm, T thresholds are generated. The T+1 images generated by segmentation with T thresholds can be used as depth clues. In the training, T=3 is used for initialization. During the training and reasoning process, after the integrated depth map quality perception and dynamic restoration module, the restored depth map l is obtained. enhence and depth map quality perceptual score g p , Represents the number of thresholds generated by the multi-Otsu threshold algorithm during training and inference, expressed as:

[0046]

[0047] Among them, g p The interval is [0,1], the value of β represents the hyperparameter, and α represents the maximum value of the number of thresholds generated by the multi-Otsu threshold algorithm. pThe higher the value, the more depth cues will be generated, and the depth cues can be used as masks to suppress or enhance features. For example, the depth cues are used as masks to perform element-wise multiplication with the features of the RGB image and depth map output by a ResNet-like network or a feature map after multimodal fusion, thereby suppressing and enhancing features using depth cues.

[0048] For those skilled in the art, improvements and changes can be made according to the above invention content. Any modification, improvement and change made on the basis of the present invention should fall within the protection scope of the present invention.

Claims

1. A quality-aware depth map dynamic restoration method, characterized in that: include: S1, using a multi-level feature fusion U-Net convolutional neural network algorithm to generate an RGB prompt map and an edge map of an RGB image, including obtaining two highly correlated features of salient target detection and edge detection according to the RGB prompt map, and inputting the two features into stacked cross-refinement units CRUs to output multi-level salient target prompt features and edge prompt features; respectively inputting the final salient target prompt features and the final edge prompt features into the U-Net convolutional neural network, and for each U-Net convolutional neural network further including an upsampling layer, a convolution layer and a cascade operation, respectively fusing in a top-down manner to obtain the final salient target prompt features and the final edge prompt features; decoding the final salient target prompt features H and decoding the final edge prompt features E to obtain a predicted RGB prompt map S h And the edge graph S e ; S2, builds a depth map quality perception algorithm model, which uses the deepest RGB hint map S obtained in S1 h As input, the model is trained using deep learning methods and the depth map quality of the training data in the dataset. The output of the model is to generate a predicted raw depth map quality score g p , g p Reflects the potential confidence of the depth map; S3, using the reference residual network to build a dynamic depth enhancement module, based on the RGB prompt map S generated in S1 h and the original depth map quality score g generated in step S2 p Perform dynamic depth map repair to generate the repaired depth map D r ; include ,First, the original depth map is normalized to the interval [0,1] and input into the dynamic ,depth enhancement module; Then, the normalized depth map and RGB cue map S h are all input into the dynamic depth enhancement module, and the input is represented as {l d ,S h }, where l d and S h They are the input depth map and RGB prompt map respectively, and the dynamic depth enhancement module outputs a single-channel repaired depth map, as shown in the following formula: in, and Represent element-wise addition and element-wise multiplication respectively; S4, based on the original depth map quality score g of S2 p And the repaired depth map D of S3 r , dynamically generate depth clues m1,m2,m3,..,.m n , the depth clues are used as masks, and the features of the RGB image and the depth map output by the feature extraction network or the feature map after multimodal fusion are element-wise multiplied to achieve the suppression and enhancement of features using depth clues.

2. The method for dynamic restoration of depth images with integrated quality perception according to claim 1, characterized in that: The S1 further includes, in the stacked cross refinement units CRUs, performing multi-level cross fusion between the salient object prompt features and the edge features to obtain the salient object prompt features of each level and the edge prompts of each level generated by the CRU.

3. The method for dynamic restoration of depth images with integrated quality perception according to claim 1, characterized in that: The S1 further includes that each U-Net convolutional neural network further includes an upsampling layer, a convolution layer and a cascade operation, which respectively fuses the final salient target prompt features and the final edge prompt features in a top-down manner.

4. The method for dynamic restoration of depth images with integrated quality perception according to claim 1, characterized in that: The S1 further includes using two U-Net convolutional neural networks to respectively aggregate the final salient target hint features and the final edge hints generated by the last CRU.

5. The method for dynamic restoration of depth images with integrated quality perception according to claim 1, characterized in that: In S1, the salient target cue features at each level and edge hints at all levels The expression is as follows: in, and represents the salient target cue features and edge cue features of the i-th level of the n-th CRU, Cat(·) represents the cascade operation along the output channel axis, Conv(·) represents a 3×3 convolutional layer with 32 output channels, CU(·) is a scale transformation operation and a 1×1 convolutional layer with 32 output channels, ∏ represents element-wise multiplication of cue features of all levels, and k represents the number of cumulative multiplications.

6. The method for dynamic restoration of depth images with integrated quality perception according to claim 1, characterized in that: The S2 further includes the deep features C4 and D4 of the RGB image features and the depth map features output by the ResNet-like feature extraction network. After downsampling operation, the high-dimensional top-level features C4 and D4 feature data are mapped into one-dimensional vectors, which are the one-dimensional feature vectors f of the RGB image, respectively. r , one-dimensional feature vector f of the depth map d , the two cascaded vectors are input into two series-connected fully connected layers and activation functions to obtain the estimated original depth map quality score g p .

7. The method for dynamic restoration of depth images with integrated quality perception according to claim 1, characterized in that: The S4 further includes element-wise multiplication of the depth clues with the features of the RGB image and the depth map output by a ResNet-like network or a feature map after multimodal fusion, thereby enhancing the features using the depth clues.

8. The method for dynamic restoration of depth images with integrated quality perception according to claim 1, characterized in that: The S4 further includes segmenting the repaired depth map with T thresholds to generate T+1 maps as depth clues, and generating a threshold T through a multi-Otsu threshold algorithm, which is expressed as: Among them, g p The interval is [0,1], the value of β represents the hyperparameter, and α represents the maximum number of thresholds generated by the multi-Otsu threshold algorithm.

Images