Depth map super-resolution method guided by color image based on diffusion model

Abstract

The invention discloses a color image guided depth map super-resolution method based on a diffusion model, and belongs to the technical field of image super-resolution. The method comprises the following steps: constructing a color-depth image data set, and dividing the color-depth image data set into a training set and a test set; designing and training a low-frequency reconstruction model and a high-frequency reconstruction model, and performing characteristic decomposition and collaborative reconstruction on the low-resolution depth image and the high-resolution color image by using wavelet transform to obtain a preliminarily reconstructed super-resolution depth image; migrating a pre-trained monocular depth estimation diffusion generation model to a depth map super-resolution task guided by a color image by adopting transfer learning, and training a diffusion model-based fine super-resolution depth map generation model by combining a ControlNet mechanism and taking the preliminarily reconstructed depth map as a control condition; and training and testing the whole network model by designing a loss function, a fine tuning strategy and a distribution alignment method to obtain a completely trained end-to-end integration network model, and outputting a high-precision super-divided high-resolution depth map.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and image super-resolution, and particularly relates to a deep image super-resolution method based on a diffusion model and guided by color images. BACKGROUND

[0002] In the field of computer vision, deep image super-resolution technology is committed to restoring low-resolution depth images to high-resolution versions to meet the needs of high-precision depth information in application scenarios such as three-dimensional reconstruction, robot navigation, and autonomous driving. The core problem of deep image super-resolution is how to accurately restore rich detail information and accurate depth structure from limited low-resolution input.

[0003] At present, deep image super-resolution methods can be mainly divided into traditional methods and deep learning-based methods. Traditional methods mostly rely on interpolation algorithms (such as bilinear interpolation and bicubic interpolation), sparse representation, or model-based optimization strategies. These methods perform up-sampling on low-resolution depth images through pre-designed mathematical models, but due to the lack of adaptive learning ability for depth image content, they often result in edge blurring and detail loss in the reconstructed depth image, making it difficult to meet the high-precision needs in complex scenarios.

[0004] Deep learning-based methods have made significant progress in recent years, including end-to-end models based on convolutional neural networks (CNN), generative adversarial networks (GAN), and Transformer-based architectures. These methods can learn the mapping relationship between low-resolution and high-resolution depth images through large amounts of data training, but still have the following limitations: first, most models directly process low-resolution images with a single network, lacking hierarchical analysis of image multi-scale features, making it difficult to effectively separate and reconstruct structural information and detail information of the image; second, existing methods mostly perform super-resolution based on depth image information alone, even if some methods introduce color image assistance, they only simply fuse or use a small amount of features, failing to fully exploit the guiding potential of rich texture, color, and other semantic information in color images in the depth super-resolution task; third, when dealing with complex scenarios, models lack sufficient utilization of prior knowledge of depth information, resulting in structural errors or detail distortions in the reconstructed depth image in scenarios such as object edges and occluded areas. Therefore, there is an urgent need for a deep image super-resolution method that can fully utilize multi-modal information, hierarchically analyze image features, and effectively fuse prior knowledge to improve the quality and practicality of the reconstructed depth image. SUMMARY

[0005] In view of the limitation that existing deep image super-resolution methods mostly use a single model for direct processing and rely on only a single modality or a small amount of prior information, the present application provides a deep image super-resolution method based on a diffusion model and guided by color images.

[0006] To achieve the above object, the present application adopts the following technical scheme:

[0007] A depth map super-resolution method based on a diffusion model and guided by a color image, comprising the following steps:

[0008] Step 1, collect public synthetic data sets containing color images and high-resolution depth maps and real scene data sets containing color images, real low-resolution depth maps and high-resolution depth maps, construct color-depth image data sets, and divide the data sets into training sets and test sets;

[0009] The color-depth image data set in step 1 is composed of a low-resolution depth map d, a high-resolution depth map D, and a high-resolution color image R.

[0010] Step 2, design and train a low-frequency reconstruction model, which receives the wavelet low-frequency approximation component a of the low-resolution depth map d as input and predicts the wavelet low-frequency approximation component A of the high-resolution depth map as output, and design the training optimization scheme and loss function of the low-frequency reconstruction model;

[0011] The loss function of the low-frequency reconstruction model in step 2 is:

[0012]

[0013] wherein, indicates the loss function of the wavelet low-frequency approximation component A of the high-resolution depth map, and respectively indicate the real low-frequency approximation component and the predicted low-frequency approximation component, indicates the number of samples used to train the model, indicates the index of the i-th training sample.

[0014] Step 3, design and train a high-frequency reconstruction model, which receives the wavelet high-frequency detail component b of the low-resolution depth map d and the wavelet high-frequency detail component y of the high-resolution color image R as input, and predicts the wavelet high-frequency detail component B of the high-resolution depth map as output, and design the training optimization scheme and loss function of the high-frequency reconstruction model;

[0015] The loss function of the high-frequency reconstruction model in step 3 is:

[0016]

[0017] wherein, indicates the loss function of the wavelet high-frequency detail component B of the high-resolution depth map, and ​respectively represent the real high-frequency detail component and the predicted high-frequency detail component, represents the number of samples used to train the model, represents the index of the th training sample.

[0018] The wavelet transform processing steps in steps 2 and 3 include: wavelet decomposition of the low-resolution depth map d to obtain a low-frequency approximation component a and a high-frequency detail component b, and wavelet decomposition of the high-resolution color image R to obtain a low-frequency approximation component x and a high-frequency detail component y.

[0019] Step 4: performing two-dimensional wavelet inverse transform on the low-frequency approximation component A output by the low-frequency reconstruction model and the high-frequency detail component B output by the high-frequency reconstruction model to obtain a preliminary reconstructed super-resolution depth map C;

[0020] Step 5: designing and training a fine super-resolution depth map generation model based on a diffusion model, using transfer learning to migrate a pre-trained monocular depth estimation diffusion generation model to a color image guided depth map super-resolution task, inputting a high-resolution color image R, receiving the preliminary reconstructed super-resolution depth map C as an additional control condition through a ControlNet mechanism, outputting a final super-resolved high-resolution depth map D, and designing a fine-tuning strategy of the model, a distribution alignment method during inference, and a loss function;

[0021] The specific training process of the fine super-resolution depth map generation model based on the diffusion model in step 5 is as follows: first, based on transfer learning, use the pre-trained monocular depth estimation diffusion generation model to initialize the UNet model parameters of the diffusion model, the input of the model is the joint feature obtained by splicing the color image R hidden space feature encoded by the pre-trained VAE encoder and the noisy hidden space feature of the current time step high-resolution depth map D .

[0022]

[0023] represents the current time step, is a diffusion coefficient, is Gaussian noise, and the output is the current time step noise prediction value . represents the noise-free hidden space feature of the high-resolution depth map D; .

[0024] Only the Q, K, and V matrices of the UNet attention layer are decomposed during the training process, only the low-rank adapter parameters after decomposition are updated, and the ControlNet architecture is introduced. The preliminary reconstructed super-resolution depth map C is encoded as an additional condition input through the pre-trained VAE encoder, and the spatial structure features of the preliminary reconstructed super-resolution depth map C are fused between the corresponding layers of the UNet decoder through the zero convolution layer to control the noise prediction direction during the diffusion process. According to the preliminary reconstructed super-resolution depth map C and the color image R, a high-resolution depth map D is generated.

[0025] The specific inference process of the fine super-resolution depth map generation model based on the diffusion model in step 5 is as follows:

[0026] Step 501, input the preliminary reconstructed super-resolution depth map C and the color image R into the pre-trained VAE encoder to compress into hidden space features;

[0027] Step 502, input the VAE encoder v- encoded color image R hidden space feature, Gaussian noise , and the current time step (initially 1000) as input, and input the hidden space feature of the preliminary reconstructed super-resolution depth map C through the ControlNet to predict the noise of the current time step;

[0028] Step 503, starting from pure noise, iteratively update the noisy hidden space feature of the super-resolution depth map D' through the reverse diffusion process, the formula is:

[0029]

[0030] Step 504, repeat steps 502 and 503 until t = 0, and convert the noise-free hidden space feature of the final iteration generated high-resolution depth map D to pixel space through the VAE decoder to output the super-resolution depth map D';

[0031] Step 505, calculate the mean and variance of the preliminary reconstructed super-resolution depth map C and the super-resolution depth image D' obtained in step 504, eliminate the distribution offset through linear transformation, and obtain the final super-resolution high-resolution depth map D.

[0032] The mathematical expression of the loss function of the diffusion model in step 5 is as follows:

[0033]

[0034] where, represents the pre-trained VAE model, and the parameters remain frozen during the training of the diffusion model, represents a UNet model in a diffusion model, represents low-rank decomposed low-rank adapter parameters in UNet that need to be trained and updated, represents a ControlNet model, represents all parameters in ControlNet that need to be trained and updated.

[0035] The distribution alignment method in step 5 is: first, calculate the mean of the preliminary reconstructed super-resolution depth map C and variance Then calculate the mean and variance of the super-resolution depth map D' output by the VAE decoder, normalize and re-parameterize the super-resolution depth map D' output by the VAE decoder to obtain the final super-resolution high-resolution depth map D after distribution alignment, and the formula is: .

[0036] Step 6, using public synthetic data sets and real scene data sets, input low-resolution depth map d and high-resolution color image R, train and test the overall network model, obtain a complete end-to-end integrated network model after training, and finally obtain a super-resolution high-resolution depth map.

[0037] Compared with the prior art, the present application has the following advantages:

[0038] The present application expands the monocular depth estimation method based on the diffusion model to further explore the optimization of the visual effect of depth map super-resolution using the rich prior knowledge of the diffusion model. The present application innovatively realizes the deep integration of discriminative models and generative models. In the first stage, a discriminative model architecture is adopted, the image features are decomposed into low-frequency approximation and high-frequency details through wavelet transform, and a targeted reconstruction network is designed respectively, the complementary nature of color image high-frequency texture and depth map low-frequency structure is utilized for collaborative learning, and multi-modal feature prior is accurately extracted; in the second stage, a generative diffusion model is introduced, forming a two-stage processing paradigm of "discriminative feature analysis-generative prior enhancement", which not only retains the precise capture ability of the discriminative model for structural information, but also takes advantage of the modeling advantage of the generative model for detail distribution, significantly improving the comprehensiveness of feature utilization and the reliability of reconstruction results.

[0039] The application adopts a controllable diffusion generation model architecture, and embeds a preliminary reconstruction result as a spatial structure constraint condition into a diffusion process through a ControlNet mechanism to realize accurate guidance on a generation direction. The mechanism not only retains strong generation capability of the diffusion model, but also ensures that a generation result strictly conforms to semantic logic of an input color image and a structure framework of preliminary reconstruction, effectively solves a structure deviation problem prone to occurring in a traditional generation model, and enables an edge integrity and rationality of a blocked area of a super-resolution result to be maintained under a complex scene.

[0040] The application fully utilizes strong generation prior advantages of the diffusion model, migrates depth knowledge of a pre-trained monocular depth estimation diffusion model to a super-resolution task through migration learning, and enables a large amount of depth scene prior built-in the diffusion model to be reused. The generation prior can provide rich semantic context information for a super-resolution process, make up for detail loss of a low-resolution input, especially generate depth details conforming to real physical laws in a sparse texture area, and significantly improve detail richness and visual authenticity of a reconstruction result compared with a traditional method.

[0041] The application constructs a hierarchical feature prior extraction and fusion system: high-frequency and low-frequency features are decoupled and extracted through wavelet transform, so that low-frequency structure prior and high-frequency texture prior can be optimized respectively; structure prior output by a discriminative model and generation prior of the diffusion model are cross-modal fused through the ControlNet; and feature distribution differences in different stages are eliminated through a distribution alignment mechanism to realize consistency of feature transmission. Compared with a traditional method of simply splicing multi-modal features, the multi-dimensional fusion strategy can more efficiently mine internal correlations of color images and depth maps, and fully release guiding potential of multi-modal information. BRIEF DESCRIPTION OF DRAWINGS

[0042] FIG. 1 is a general model structure diagram of a depth map super-resolution method guided by a color image based on a diffusion model;

[0043] Figure 2 FIG. 2 is a structure diagram of a diffusion model-based fine super-resolution depth map generation model training process. DETAILED DESCRIPTION

[0044] In order to understand the application in depth, it will be described in detail. However, the application has various implementation manners and is not limited to the specific examples listed in the text. The presentation of these examples aims to deepen the overall understanding of the disclosed content of the application.

[0045] A depth map super-resolution method guided by a color image based on a diffusion model includes the following steps:

[0046] Step 1, Collect public synthetic dataset (NYUv2, Middlebury, Lu et al.) containing color image and high-resolution depth map, and real scene dataset containing color image, real low-resolution depth map and high-resolution depth map, to construct color-depth image dataset composed of low-resolution depth map d, high-resolution depth map D, and high-resolution color image R. Divide the dataset into training set and test set in the ratio of 7:3;

[0047] Step 2, design and train low-frequency reconstruction model, perform two-dimensional wavelet decomposition (using Haar wavelet) on low-resolution depth map d to obtain low-frequency approximation component a and high-frequency detail component b. Design low-frequency reconstruction model (using U-Net architecture), input low-frequency approximation component a, and predict output wavelet low-frequency approximation component A of high-resolution depth map. Use Adam optimizer, initialize learning rate to 2e-4, decay by 0.5 every 50 epochs, and train for 200 epochs. At the same time, design the loss function of the low-frequency reconstruction model:

[0048]

[0049] wherein, represents the loss function of the wavelet low-frequency approximation component A of the high-resolution depth map, and respectively represent the real low-frequency approximation component and the predicted low-frequency approximation component, represents the number of samples used to train the model, represents the index of the th training sample.

[0050] Step 3, design and train high-frequency reconstruction model, perform wavelet decomposition on high-resolution color image R as in step 2 to obtain its high-frequency detail component y; perform channel splicing on the high-frequency detail component b of the low-resolution depth map d and the high-frequency detail component y of the color image R as the input of the high-frequency reconstruction model (using residual convolutional network), and predict the output wavelet high-frequency detail component B of the high-resolution depth map. Use Adam optimizer, initialize learning rate to 2e-4, decay by 0.5 every 50 epochs, and train for 200 epochs. At the same time, design the loss function of the high-frequency reconstruction model:

[0051]

[0052] wherein, represents the loss function of the wavelet high-frequency detail component B of the high-resolution depth map, and respectively represent the real high-frequency detail component and the predicted high-frequency detail component, represents the number of samples used to train the model, an index representing the

[0053] Step 4, perform two-dimensional wavelet inverse transform on the low-frequency approximation component A output by the low-frequency reconstruction model and the high-frequency detail component B output by the high-frequency reconstruction model to obtain a preliminary reconstructed super-resolution depth map C, which has the same size as the high-resolution color image R.

[0054] Step 5, design and train a fine super-resolution depth map generation model based on a diffusion model, use the Stable Diffusion framework, and based on transfer learning, use the pre-trained monocular depth estimation diffusion generation model Marigold to initialize the UNet model parameters of the diffusion model. At the same time, introduce the ControlNet architecture, encode the preliminary reconstructed super-resolution depth map C through the pre-trained VAE encoder as an additional condition input, and fuse the spatial structure features of the preliminary reconstructed super-resolution depth map C through zero convolution layers between the corresponding layers of the decoder of the UNet to control the noise prediction direction in the diffusion process.

[0055] During training, the color image R hidden space features encoded by the pre-trained VAE encoder and the noisy hidden space features of the high-resolution depth map D at the current time step The spliced joint features input the UNet, and the noisy hidden space features The formula is as follows:

[0056]

[0057] wherein, represents the current time step, is the diffusion coefficient, is the Gaussian noise, and the output is the current time step noise prediction value , During training, only the Q, K, and V matrices of the attention layer of the UNet are subjected to low-rank decomposition, and only the low-rank adapter parameters after decomposition and the ControlNet parameters are updated. The mathematical expression of the loss function of the diffusion model is as follows:

[0058]

[0059] wherein, represents the pre-trained VAE model, and the parameters remain frozen during the training of the diffusion model, represents the UNet model in the diffusion model, represents the low-rank adapter parameters after low-rank decomposition in the UNet that need to be trained and updated, represents the ControlNet model, ​All the parameters in ControlNet that need to be trained and updated.

[0060] During inference, the preliminary reconstructed super-resolution depth map C is input into the pre-trained VAE encoder along with the color image R, compressed into latent space features, and then the latent space features of the color image R, Gaussian noise and the current time step t (initially 1000) are input as inputs, and the latent space features of the preliminary reconstructed super-resolution depth map C are input through the ControlNet, starting from pure noise, the noisy latent space features of the super-resolution depth image D' are iteratively updated through the process of back diffusion , the formula is:

[0061]

[0062] Until t = 0, the noise-free latent space features of the super-resolution depth image D' generated by the final iteration are converted into pixel space by the VAE decoder, and the super-resolution depth image D' is output. Then the mean and variance of the preliminary reconstructed super-resolution depth map C are calculated and variance of the super-resolution depth image D' output by the VAE decoder, and the super-resolution depth image D' output by the VAE decoder is normalized and re-parameterized to obtain the final super-resolution high-resolution depth map D after distribution alignment, the formula is: .

[0063] Step 6, using the public synthetic dataset and real scene dataset, input the low-resolution depth map d and high-resolution color image R, train and test the overall network model, obtain the trained end-to-end integrated network model, and finally obtain the super-resolution high-resolution depth map.

[0064] The contents not described in detail in the specification of the present application belong to the prior art known to those skilled in the art. Although the above describes the specific embodiments of the present application in order to facilitate those skilled in the art to understand the present application, it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

Claims

1. A depth map super-resolution method based on a diffusion model and guided by color images, characterized in that: Includes the following steps: Step 1: Collect publicly available synthetic datasets containing color images and high-resolution depth maps, as well as real-world scene datasets containing color images, real low-resolution depth maps, and high-resolution depth maps, to construct a color-depth image dataset. Divide this dataset into a training set and a test set. Step 2: Design and train a low-frequency reconstruction model. The model takes the wavelet low-frequency approximation component a of the low-resolution depth map d as input and predicts the wavelet low-frequency approximation component A of the high-resolution depth map as output. At the same time, design the training optimization scheme and loss function of the low-frequency reconstruction model. Step 3: Design and train a high-frequency reconstruction model. The model takes the wavelet high-frequency detail b of the low-resolution depth map d and the wavelet high-frequency detail component y of the high-resolution color image R as input, and predicts the wavelet high-frequency detail component B of the high-resolution depth map as output. At the same time, design the training optimization scheme and loss function of the high-frequency reconstruction model. Step 4: Perform a two-dimensional wavelet inverse transform on the low-frequency approximation component A output by the low-frequency reconstruction model and the high-frequency detail component B output by the high-frequency reconstruction model to obtain the preliminary reconstructed super-resolution depth map C. Step 5: Design and train a fine super-resolution depth map generation model based on the diffusion model. Use transfer learning to transfer the pre-trained monocular depth estimation diffusion generation model to the color image-guided depth map super-resolution task. Input a high-resolution color image R, and receive the initially reconstructed super-resolution depth map C as additional control conditions through the ControlNet mechanism. Output the final super-resolution high-resolution depth map D. At the same time, design the model's fine-tuning strategy, distribution alignment method during inference, and loss function. Step 6: Using publicly available synthetic datasets and real-world scene datasets, input a low-resolution depth map d and a high-resolution color image R, train and test the overall network model to obtain a fully trained end-to-end integrated network model, and finally obtain the super-resolution depth image.

2. The depth map super-resolution method based on a diffusion model and guided by color images according to claim 1, characterized in that: The color-depth image dataset in step 1 consists of a low-resolution depth map d, a high-resolution depth map D, and a high-resolution color image R.

3. The depth map super-resolution method based on a diffusion model and guided by color images according to claim 2, characterized in that: The wavelet transform processing steps in steps 2 and 3 include: performing wavelet decomposition on the low-resolution depth map d to obtain the low-frequency approximation component a and the high-frequency detail component b, and performing wavelet decomposition on the high-resolution color image R to obtain the low-frequency approximation component x and the high-frequency detail component y.

4. The depth map super-resolution method based on a diffusion model and guided by a color image according to claim 3, characterized in that: The specific training process of the fine super-resolution depth map generation model based on the diffusion model in step 5 is as follows: First, based on transfer learning, the UNet model parameters of the diffusion model are initialized using the pre-trained monocular depth estimation diffusion generation model parameters. The input of this model is the latent space features of the color image R encoded by the pre-trained VAE encoder and the noisy latent space features of the high-resolution depth map D at the current time step. Joint features after splicing, noisy latent space features The formula is as follows: ,in, Indicates the current time step. Where is the diffusion coefficient. The noise is Gaussian, and the output is the noise prediction value at the current time step. ; This represents the noise-free latent space features of the high-resolution depth map D; ; During training, only the Q, K, and V matrices of the UNet attention layer are decomposed into low rank, and only the parameters of the decomposed low-rank adapter are updated. At the same time, the ControlNet architecture is introduced, and the initially reconstructed super-resolution depth map C is encoded as an additional conditional input through a pre-trained VAE encoder. The spatial structure features of the initially reconstructed super-resolution depth map C are fused between the corresponding layers of the UNet decoder through zero convolutional layers to control the noise prediction direction during the diffusion process. A high-resolution depth map D is generated based on the initially reconstructed super-resolution depth map C and the color image R.

5. The depth map super-resolution method based on a diffusion model and guided by a color image according to claim 4, characterized in that: The specific reasoning process of the fine super-resolution depth map generation model based on the diffusion model in step 5 is as follows: Step 501: Input the initially reconstructed super-resolution depth map C and the color image R into the pre-trained VAE encoder and compress them into latent space features. Step 502: Extract the latent space features of the color image R encoded by the VAE encoder v, and add Gaussian noise. and the current time step As input, the latent space features of the initially reconstructed super-resolution depth map C are also input through ControlNet to predict the noise at the current time step. ; Step 503: Starting from pure noise, iteratively update the noisy latent space features of the super-resolution depth map D' through a back-diffusion process. The formula is: ； Step 504: Repeat steps 502 and 503 until t=0, and extract the noise-free latent space features of the final iteratively generated high-resolution depth map D. The super-resolution depth map D' is output by converting the data to pixel space using a VAE decoder. Step 505: Calculate the mean and variance of the initially reconstructed super-resolution depth map C and the super-resolution depth map D' obtained in step 504. Eliminate the distribution shift through linear transformation to obtain the final super-resolution high-resolution depth map D.

6. The depth map super-resolution method based on a diffusion model and guided by a color image according to claim 5, characterized in that: The loss function of the low-frequency reconstruction model in step 2 is: ,in, The loss function representing the wavelet low-frequency approximation component A of a high-resolution depth map. and These represent the true low-frequency approximation component and the predicted low-frequency approximation component, respectively. This indicates the number of samples used to train the model. Indicates the first The index of each training sample.

7. The depth map super-resolution method based on a diffusion model and guided by a color image according to claim 6, characterized in that: The loss function of the high-frequency reconstruction model in step 3 is: ,in, The loss function representing the wavelet high-frequency detail component B of a high-resolution depth map. and These represent the true high-frequency detail components and the predicted high-frequency detail components, respectively. This indicates the number of samples used to train the model. Indicates the first The index of each training sample.

8. The depth map super-resolution method based on a diffusion model and guided by a color image according to claim 7, characterized in that: The mathematical expression of the loss function of the diffusion model in step 5 is as follows: ,in, This indicates that the parameters of the pre-trained VAE model remain frozen during the training of the diffusion model. This represents the UNet model in the diffusion model. This indicates the low-rank adapter parameters that need to be trained and updated after low-rank decomposition in UNet. Represents the ControlNet model. This represents all parameters in ControlNet that need to be trained and updated.

9. A depth map super-resolution method based on a diffusion model and guided by a color image, as described in claim 8, characterized in that: The distribution alignment method in step 5 is as follows: First, calculate the mean value of the initially reconstructed super-resolution depth map C. and variance Then, the mean of the super-resolution depth map D' output by the VAE decoder is calculated. and variance The super-resolution depth map D' output by the VAE decoder is normalized and reparameterized to obtain the final super-resolution high-resolution depth map D after distribution alignment, as shown in the formula: .

Images