Depth estimation method and system based on indirect diffusion model

Abstract

The application provides a depth estimation method and system based on an indirect diffusion model, and relates to the technical field of depth estimation. The method comprises the following steps: inputting a to-be-processed image into an encoder of a U-NET to obtain high-level features and low-level features; inputting the high-level features into a denoising diffusion network after splicing to recover the features, and obtaining recovered features; inputting the recovered features into a high-layer decoder of a reversible transform enhanced decoder to extract features layer by layer, and obtaining high-layer decoding features; splicing the low-level features and the high-layer decoding features to obtain spliced input features, and inputting the spliced input features into a low-layer decoder of the reversible transform enhanced decoder to obtain spliced output features; and obtaining a predicted depth map based on the spliced output features. The application recovers the features in the denoising diffusion network, optimizes the high-level features, ensures the accuracy of the features input into the decoder, and constructs the reversible transform enhanced decoder through an affine coupling layer to optimize the features into assumed feature true values.

Description

Technical Field

[0001] This invention belongs to the field of depth estimation technology, and particularly relates to a depth estimation method and system based on an indirect diffusion model. Background Technology

[0002] Monocular depth estimation (MDE) is a fundamental task in the field of 3D vision. It is crucial for many applications, such as autonomous driving and 3D reconstruction. MDE is well-known as an ill-posed problem because depth prediction is inherently a three-dimensional problem, and depth values ​​cannot be accurately determined from a single image. However, with the development of neural networks and their powerful learning capabilities, depth values ​​can be estimated using visual cues such as relative size, perspective distortion, occlusion relationships, and motion blur.

[0003] Deep learning-based depth estimation algorithms (MDEs) typically employ an encoder-decoder architecture, which has yielded significant improvements. However, due to their ill-posedness, it remains unclear whether MDEs can be further improved. In other words, whether effective features can be learned to predict high-quality depth maps under current network architectures remains a question. Furthermore, how different levels of encoder features affect the final depth prediction has not been thoroughly investigated. On the other hand, diffusion models have achieved excellent results in image generation tasks, and many works have applied diffusion models to dense prediction tasks such as semantic segmentation, image super-resolution, and deblurring. Recent research has also investigated the application of denoising diffusion models to MDE tasks. Some studies use pre-trained StableDiffusion as a depth feature generator, where a general encoder such as a VAE is used as the backbone, and CLIP features or other semantic features are used as conditional maps to predict depth maps. Other studies use virtual color image-dense depth map pairs and use the images as conditional maps to train denoising diffusion models. These works demonstrate the potential of using diffusion models for depth estimation.

[0004] However, existing methods still face some challenges. In real-world scenarios, ground truth depth maps are typically obtained from LiDAR point clouds and are sparse. This complicates the diffusion process when adding noise to the ground truth depth map during forward propagation, thus reducing the effectiveness of backpropagation. Furthermore, the aforementioned StableDiffusion-based diffusion methods rely on supervised training of the final depth prediction loss, neglecting the iterative nature of the diffusion model. The lack of supervision in the latent features used iteratively during diffusion can lead to biases in these features during iterations, affecting the final result during multi-step inference. Figure 3 As shown, if deep supervision is used as the indirect supervision signal for the diffusion model during training, it is difficult to ensure the correct iteration because each step is optimized independently and there is a lack of direct supervision to optimize the gradient. This will lead to feature shift (green circle) during testing. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a depth estimation method and system based on an indirect diffusion model. It recovers features through a denoising diffusion network, optimizes high-level features, ensures the accuracy of features in the input decoder, and significantly improves depth prediction performance. At the same time, it selectively optimizes low-level features and constructs an invertible transformation-enhanced decoder by using an affine coupling layer. By minimizing the depth map loss, it optimizes the features in the denoising diffusion network to the optimal ground truth feature values.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of this invention provides a depth estimation method based on an indirect diffusion model.

[0008] The depth estimation method based on the indirect diffusion model includes the following steps:

[0009] Obtain the image to be processed;

[0010] The image to be processed is input into the encoder of the U-NET network for layer-by-layer feature extraction, resulting in multiple high-level features and multiple low-level features.

[0011] A reversible transform-enhanced indirect diffusion depth estimation network was constructed, comprising a denoising diffusion network and a reversible transform-enhanced decoder.

[0012] Multiple high-level features are concatenated and then input into a denoising diffusion network for feature recovery, predicting degradation and added noise, and obtaining the final recovered features.

[0013] The recovered features are input into the higher-level decoder of the reversible transform enhancement decoder for layer-by-layer feature extraction, and finally the higher-level decoded features are obtained.

[0014] The low-level features and high-level decoded features are concatenated to obtain the concatenated input features. The concatenated input features are then input into the low-level decoder of the reversible transform enhancement decoder. The concatenated input features are divided into a first part and a second part using an affine coupling layer. In each part, the forward propagation of the current part and the affine mapping of the other part are performed. The outputs of each part are then concatenated to obtain the concatenated output features.

[0015] Based on the spliced ​​output features, a predicted depth map is obtained.

[0016] A second aspect of the present invention provides a depth estimation system based on an indirect diffusion model.

[0017] Depth estimation systems based on indirect diffusion models include:

[0018] The image acquisition module is configured to acquire the image to be processed.

[0019] The encoder feature processing module is configured to: input the image to be processed into the encoder of the U-NET network, perform layer-by-layer feature extraction, and obtain multiple high-level features and multiple low-level features;

[0020] The network model building module is configured to build an indirect diffusion depth estimation network with reversible transform enhancement, which includes a denoising diffusion network and a reversible transform enhancement decoder.

[0021] The recovery module is configured to: concatenate multiple high-level features and input them into a denoising diffusion network for feature recovery, predict degradation and added noise, and obtain the final recovered features;

[0022] The high-level decoder module is configured to input the recovered features into the high-level decoder of the reversible transform enhancement decoder, perform layer-by-layer feature extraction, and finally obtain the high-level decoded features.

[0023] The low-level decoder module is configured to: concatenate low-level features and high-level decoded features to obtain concatenated input features; input the concatenated input features into the low-level decoder of the reversible transform enhancement decoder; use an affine coupling layer to divide the concatenated input features into a first part and a second part; in each part, perform forward propagation of the current part and affine mapping processing of the other part; and concatenate the outputs of each part to obtain the concatenated output features.

[0024] The prediction module is configured to obtain a predicted depth map based on the spliced ​​output features.

[0025] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps in the depth estimation method based on the indirect diffusion model as described in the first aspect of the present invention.

[0026] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the depth estimation method based on the indirect diffusion model as described in the first aspect of the present invention.

[0027] The above one or more technical solutions have the following beneficial effects:

[0028] This invention provides a depth estimation method and system based on an indirect diffusion model. Multiple high-level features are concatenated and then input into a denoising diffusion network for feature recovery, predicting degradation and added noise, and obtaining the final recovered features. This optimizes the high-level features, ensures the accuracy of the features input into the decoder, and significantly improves the depth prediction performance.

[0029] To ensure that the optimization at each step in the denoising diffusion network does not deviate from the assumed true features to be recovered, the decoder function is designed to be invertible and satisfy bidirectional Lipshitz continuity. A reversible neural layer is proposed to construct the reversible decoder module: the recovered features are input into the reversible transform enhancement decoder, and the concatenated input features are divided into a first part and a second part using an affine coupling layer. In each part, the forward propagation of the current part and the affine mapping of the other part are performed, and the outputs of each part are concatenated to obtain the concatenated output features, which in turn yields the predicted depth map. Furthermore, the features are optimized to the assumed true features by minimizing the depth map loss.

[0030] Furthermore, this invention develops an auxiliary viewpoint-based low-level feature enhancement module (AV-LFE) that improves the skip connections used in the UNet architecture to enhance low-level spatial features when an auxiliary viewpoint is available.

[0031] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0032] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0033] Figure 1 This diagram illustrates the impact of different feature levels on the final depth prediction performance in terms of RMSE.

[0034] Figure 2 This diagram illustrates the impact of different levels of features on the final depth prediction performance in Abs Rel.

[0035] Figure 3 This is a schematic diagram illustrating the characteristic deviation problem during iterative diffusion.

[0036] Figure 4 This is a diagram of the overall network structure of Embodiment 1 of the present invention. Detailed Implementation

[0037] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0038] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0039] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0040] Terminology Explanation:

[0041]

[0042]

[0043] Example 1

[0044] Diffusion models are a type of generative model that can sample from random noise to generate new images. The basic principle is to add noise to images in a dataset, and then use a diffusion network to predict the added noise. This training method enables the network to denoise, and in the process, the diffusion network also learns some intrinsic knowledge about the images in the dataset.

[0045] Training phase of a typical diffusion network:

[0046] Adding noise ε to the input F yields features Ft at different time steps (different noise levels), which represents the forward process. These features Ft are then fed into a diffusion network to predict the added noise ε for training. The training process is performed independently at each step, not iteratively.

[0047] Testing phase of general diffusion networks:

[0048] The initial F is directly added to the maximum noise level and then input into the diffusion model for gradual denoising (gradual refinement is better than direct one-step denoising). This is the reverse process.

[0049] Conditional diagrams in diffusion models: Existing diffusion models include both conditional and unconditional ones.

[0050] Unconditional: The input features are denoised (x+noise) and then fed into the diffusion model for denoising (the predicted result is noise, which can be restored to the original image after subtraction), with no additional input.

[0051] Conditional input: Not only are the input features with added noise (x+noise) fed into the diffusion model, but there are also other conditional inputs. Typically, these are concatenated with the current input features to predict the added noise together.

[0052] Diffusion models have achieved great success in generative tasks, and many works have extended them to various domains, such as image augmentation, semantic segmentation, and depth estimation. Different diffusion architectures have been developed, including the Denoising Diffusion Probabilistic Model (DDPM), the Denoising Diffusion Implicit Model (DDIM), the Latent Diffusion Model (LDM), and the Residual Denoising Diffusion Model (RDDM).

[0053] In depth estimation, due to the sparsity of LiDAR-based ground reality, it is difficult to directly train a diffusion model by propagating the sparse ground reality to noise according to a noise scheme and performing the corresponding back-inference process. Some studies have used diffusion training by generating virtual depth-color image pairs, using noisy depth images as input and color images as conditions. Other methods typically take encoder features from the depth estimation network as input and add noise, then use the final sparse depth map as supervision for back-inference training. In some studies, DDP proposes using a diffusion model to generate depth maps from noise, where features extracted from the input image serve as both input (adding noise) and conditions. ECoDepth extracts image detail embeddings with ViT as the diffusion model conditional map, providing more semantic information for depth estimation.

[0054] However, since the denoising features are not directly supervised and can be optimized in different directions in different iterations, this indirect optimization may lead to feature bias in the final iterative inference.

[0055] This embodiment leverages the powerful noise modeling capabilities of diffusion models, treating feature degradation as a form of noise and enhancing it through the diffusion model. However, unlike typical diffusion which adds known noise, the noise we add is unknown, so supervision can only be indirectly achieved through the final predicted depth map. Analysis shows that this indirect supervision requires certain conditions to achieve optimal results; therefore, a reversible decoding network is proposed.

[0056] The diffusion model in this embodiment is the U-net architecture, composed of convolutional neural networks. The encoder and decoder each have four layers with feature dimensions of c, 2c, 4c, and 8c, respectively. During encoding, the spatial dimensions are successively downsampled to half the input by a downsampling module. During decoding, the skip connections and the layer above the decoder are used as input, and then upsampled to twice the input by an upsampling module.

[0057] Furthermore, MDE, which uses a single image to reconstruct the depth of a scene, faces the ill pose problem, making it an ill-posed problem.

[0058] Furthermore, depth maps typically contain sharp edges, and their values ​​range widely. Therefore, whether it's possible to learn and decode effective features to generate high-quality depth maps under current network architectures remains a question. Especially in architectures like UNet, the impact of features at different levels on the final depth prediction has not been fully investigated.

[0059] Based on the above description, the embodiments of the present invention aim to solve three problems:

[0060] 1) The impact of features from different encoder levels on depth prediction;

[0061] 2) The encoder features suffer from degradation.

[0062] 3) The diffusion model faces the problem of indirect optimization.

[0063] This invention utilizes an existing multi-view depth estimation (MDE) network, specifically PixelFormer, to optimize for each image. We employ a pre-trained model with frozen network parameters to explore the performance of different encoders. To investigate the impact of different levels of features on the final depth estimation, we optimize the hierarchical features obtained from different scale blocks of the encoder through backpropagation. Figure 1 and Figure 2 The results of the optimization steps are shown.

[0064] First, it can be observed that performance improves for all levels of features, even with a few optimization steps, indicating that the current network architecture can achieve better depth estimation by improving feature prediction. Furthermore, the network performance continues to improve with increasing optimization steps. Therefore, this significant performance suggests that it is worthwhile to explore new architectures to reduce depth estimation errors.

[0065] More importantly, such as Figure 1 and Figure 2 As shown, although depth estimation is a pixel-level task, high-level features are easier to optimize and can provide greater performance improvements compared to low-level features. This highlights the importance of high-level features. Figure 1 and Figure 2 In the middle, the second and third floors drop faster, so the lower the index, the better.

[0066] Definition of degradation: based on Figure 1 and Figure 2 We found that optimizing the encoder features can improve network performance, especially optimizing high-level features. Therefore, we can consider the original encoder feature (x) as a result of the theoretically optimal feature degradation (x0) being degraded (deg). That is, X = X0 + deg.

[0067] Based on this, this embodiment proposes a high-level feature enhancement network for monocular depth estimation (MDE). Considering that better high-level features (such as...) can be easily optimized in just a few steps... Figure 1 and Figure 2 As shown in the figure, we regard advanced feature enhancement as a recovery task, which aims to recover high-quality advanced features from initially degraded features.

[0068] Since diffusion can generate better features by eliminating noise, we employ an indirect diffusion model based on reversible transformation for feature recovery. Because we address the depth estimation problem from the perspective of using diffusion for feature recovery, the network constructed by the proposed method in this embodiment is called the Inverse Transform Enhanced Indirect Diffusion Depth Estimation Network (IID-RDepth).

[0069] As mentioned above Figure 3 As shown, if deep supervision is used as the indirect supervision signal for the diffusion model during training, it is difficult to ensure the correct iteration because each step is optimized independently and there is a lack of direct supervision to optimize the gradient. This will lead to feature shift (green circle) during testing.

[0070] The method in this embodiment constructs a reversible decoder and uses indirect supervision to ensure the correct execution of the network's iterative inference process.

[0071] Furthermore, in order to further address the technical problem that the indirect supervised diffusion model, which relies solely on network prediction of the final depth map, cannot provide good depth prediction results, and to ensure that the optimization at each step of the diffusion model does not deviate from the assumed true features that need to be recovered, this embodiment designs the decoder function to be reversible and satisfies bidirectional Lipshitz continuity, and proposes to use reversible neural layers to construct a reversible decoder module.

[0072] Specifically, the depth estimation method based on the indirect diffusion model provided in this embodiment includes the following steps:

[0073] Obtain the image to be processed;

[0074] The image to be processed is input into the encoder of the U-NET network for layer-by-layer feature extraction, resulting in multiple high-level features and multiple low-level features.

[0075] A reversible transform-enhanced indirect diffusion depth estimation network was constructed, comprising a denoising diffusion network and a reversible transform-enhanced decoder.

[0076] Multiple high-level features are concatenated and then input into a denoising diffusion network for feature recovery, predicting degradation and added noise, and obtaining the final recovered features.

[0077] The recovered features are input into the higher-level decoder of the reversible transform enhancement decoder for layer-by-layer feature extraction, and finally the higher-level decoded features are obtained.

[0078] The low-level features and high-level decoded features are concatenated to obtain the concatenated input features. The concatenated input features are then input into the low-level decoder of the reversible transform enhancement decoder. The concatenated input features are divided into a first part and a second part using an affine coupling layer. In each part, the forward propagation of the current part and the affine mapping of the other part are performed. The outputs of each part are then concatenated to obtain the concatenated output features.

[0079] Based on the spliced ​​output features, a predicted depth map is obtained.

[0080] like Figure 4 As shown, we propose an invertible transformation-enhanced indirect diffusion depth estimation network (IID-RDepth). This network employs a U-Net architecture based on a Transformer encoder as the feature extractor, and the Transformer-based encoder is pre-trained on the depth estimation task.

[0081] The image to be processed is input into the encoder of the U-NET network for layer-by-layer feature extraction, resulting in multiple high-level features and multiple low-level features. The multiple low-level features include the obtained layer 0 features and layer 1 features, and the high-level features include the obtained layer 2 features and layer 3 features. The specific extraction process is existing technology and will not be described in detail here.

[0082] The Inverse Transform Enhanced Indirect Diffusion Depth Estimation Network (InvT-IndDiffusion) proposed in this embodiment serves as a depth prediction decoder, specifically comprising a denoising diffusion network for recovering high-level features and an inverse transform enhanced decoder.

[0083] In this embodiment, the reversible transform enhancement decoder includes a high-level decoder and a low-level decoder. The high-level decoder corresponds to the number of layers in the high-level features, and the low-level decoder corresponds to the number of layers in the low-level features. That is, as... Figure 4 As shown, since high-level features include features extracted by two encoders and low-level features also include features extracted by two encoders, the corresponding high-level decoder and low-level decoder also each include two layers.

[0084] Therefore, for the convenience of subsequent description, the decoder in this embodiment includes a third-layer high-level decoder, a second-layer high-level decoder, a first-layer low-level decoder, and a zero-layer low-level decoder.

[0085] In the actual data processing, high-level features from the encoder are concatenated together and input into InvT-IndDiffusion as a multi-scale cross-conditional feature map. The final depth is obtained using a method similar to that of Adabin, where the top-level decoder layer with the Transformer architecture, the decoder query initializer (DQI), and the box center predictor (BCP) are kept frozen to account for the generality of the high-level features.

[0086] Meanwhile, the low-level decoder adopts a reversible structure design based on the coupling layer. To further enhance low-level features, we also propose an auxiliary viewpoint-based low-level feature enhancement module (AV-LFE) and present it as a plug-and-play module to explore auxiliary viewpoints when conditions permit.

[0087] The modules described above in this embodiment will now be explained in detail.

[0088] (a) The method of this embodiment will be explained in detail according to the direction of data flow.

[0089] (1) First, in this embodiment of the invention, a Transformer-based encoder that has been pre-trained on the depth estimation task is used to extract features layer by layer. This part of the Transformer-based encoder is kept frozen during the training of this network.

[0090] The image to be processed is input into the encoder to obtain multi-level features, specifically layer 0, layer 1, layer 2, and layer 3 features. Among them, layer 0 and layer 1 features are used as low-level features, and layer 2 and layer 3 features are used as high-level features.

[0091] (2) Processing of advanced features.

[0092] For the high-level features of layers 2 and 3, we upsample these two features to the same spatial resolution and then concatenate them as the original feature input F. The high-level features of layers 2 and 3 are features at different scales, and they serve as each other's conditional maps.

[0093] During the training phase, noise is added to F, and the enhanced features, i.e., the recovered features, are generated through a diffusion model. Specifically, these include:

[0094] Add noise ε to F to obtain the feature Ft at different time steps, and define the forward process;

[0095] The degenerate feature of F F deg It is also regarded as a kind of noise. A denoising diffusion network is used for feature enhancement and denoising to obtain the recovered features at each step. The reverse process is defined to obtain the final recovered features.

[0096] The decoder function is designed to be invertible and satisfy bidirectional Lipshitz continuity. The predicted depth map is used as indirect supervision of the denoising diffusion network. By minimizing the depth map loss, F is optimized to the assumed feature ground truth.

[0097] (3) After the enhanced features (i.e. the restored features) of the diffusion model are output, they are downsampled to restore the original size and the restored features are split into two parts corresponding to the number of layers of the higher-level decoder. In this embodiment, for ease of description, the restored features are split into enhanced third-layer features and enhanced second-layer features.

[0098] As mentioned above, the decoder in this embodiment includes a third-layer high-level decoder, a second-layer high-level decoder, a first-layer low-level decoder, and a zero-layer low-level decoder.

[0099] The enhanced third-layer features are input into DQI. The enhanced third-layer features are then transformed by DQI to enhance their expressive power (they are also frozen and consist of multiple convolutional layers), resulting in the third-layer features output by DQI.

[0100] Then, for the third layer high-level decoder of the fixed transformer, the third layer features output by DQI and the enhanced third layer features are concatenated together as input and fed into the third layer high-level decoder to obtain the output features of the third layer high-level decoder.

[0101] Layer 2 High-Level Decoder:

[0102] For a fixed transformer layer 2 high-level decoder, the enhanced layer 2 features and the output features of the layer 3 high-level decoder are concatenated together as input and fed into the layer 2 high-level decoder to obtain the output features of the layer 2 high-level decoder.

[0103] It can be understood that in the data flow process of the higher-level decoder, the output of the previous higher-level decoder serves as the input of the next higher-level decoder, and both are used for feature extraction in the next higher-level decoder. This is existing technology and will not be elaborated further.

[0104] In this way, the output features of the third-layer high-level decoder and the enhanced features of the second layer are input together into the second-layer high-level decoder to obtain the output of the second-layer high-level decoder, which is the high-level decoding feature of the entire high-level decoder.

[0105] (4) Processing of low-level features.

[0106] (401) If the image to be processed comes only from the main viewpoint:

[0107] The low-level features extracted by the encoder are then directly input into the corresponding layer of the reversible transform enhancement decoder through skip connections.

[0108] In this embodiment, the features of layer 0 are input to the layer 0 low-level decoder via a skip connection, and the features of layer 1 are input to the layer 1 low-level decoder via a skip connection.

[0109] In the Layer 0 and Layer 1 low-level decoders, the high-level decoding features obtained above, along with the Layer 0 and Layer 1 features, are combined for feature extraction to obtain the final predicted depth map. Specifically, the Layer 1 features and the output of the Layer 2 high-level decoder are used as inputs to the Layer 1 low-level decoder, where features are extracted to obtain its output. Similarly, the Layer 0 features and the output of the Layer 1 low-level decoder are used as inputs to the Layer 0 low-level decoder, where features are extracted to obtain its output.

[0110] It can be understood that in the data flow process of the lower-level decoder, the output of the second-level higher-level decoder is used as the input of the first-level lower-level decoder, and the output of the first-level lower-level decoder is used as the input of the 0th-level lower-level decoder. Combined with the features of the 0th-level and 1st-level features of the encoder jump connection, they are used together to extract features in the next layer of the decoder. This is the existing technology and will not be elaborated further.

[0111] (402) If the image to be processed comes from both the main viewpoint and the auxiliary viewpoint:

[0112] Then, multiple low-level features from the main viewpoint and auxiliary viewpoint encoders are obtained respectively;

[0113] After concatenating multiple low-level features, they are input into the low-level feature enhancement module based on the auxiliary viewpoint. Deformable convolution is used to align the extracted low-level features of the main viewpoint and the auxiliary viewpoint, and after feature fusion, they are input into the corresponding layer of the reversible transform enhancement decoder.

[0114] In this embodiment, multiple low-level features are aligned and fused to obtain fused low-level features;

[0115] The fused low-level features and the high-level decoding features of the high-level decoder are concatenated and used as concatenated input features. The concatenated input features are then input into the low-level decoder to obtain the concatenated output features.

[0116] Specifically, since the low-level decoder includes a first-level low-level decoder and a zero-level low-level decoder, in the first-level low-level decoder, the fused low-level features and the high-level decoding features of the high-level decoder are concatenated together as the first concatenated input features. The first concatenated input features are then input into the first-level low-level decoder to obtain the first concatenated output features.

[0117] In the 0th layer low-level decoder, after the fused low-level features and the first concatenated output features are concatenated again, they are used as the second concatenated input features, and the second concatenated input features are input into the 0th layer low-level decoder to obtain the second concatenated output features.

[0118] The second concatenated output features obtained above are the concatenated output features of the entire low-level decoder.

[0119] It can be understood that in the case of binocular vision, the auxiliary view point and the main view point are only for better description and distinction. The monocular input is the main view point.

[0120] Based on the foregoing explanation, the low-level decoder in this embodiment is designed as multiple affine coupling layers, and the specific data processing process is as follows:

[0121] The low-level features and the high-level decoded features are concatenated to obtain the concatenated input features, and the concatenated input features are input into the low-level decoder of the invertible transformation enhancement decoder. The affine coupling layer is used to divide the concatenated input features into a first part and a second part. In each part, the forward propagation of the current part and the affine mapping processing of the other part are performed, and the outputs of each part are concatenated to obtain the concatenated output features.

[0122] Combination of internal data flow and formula in the invertible network:

[0123]

[0124] The input features are divided into two parts at position s < S. In an affine coupling layer, the first s channels are used to obtain the affine transformation parameters g1 and h1 for the last s channels, and for the operation results in Then is used to predict The affine transformation parameters h2 and g2 of

[0125] Finally, are concatenated together as the output of an affine coupling layer.

[0126] In the invertible network proposed in this embodiment, 3 layers of affine coupling layers are used in sequence, where s is set to 1 / 4S. In addition, in order not to destroy the reversible property of the network, a 1*1 convolution is used at the end of each layer of the invertible decoder to change the number of channels of the output features.

[0127] (5) BCP part

[0128] We use a neural network (i.e., the Figure 4 BCP module in to predict the depth center (bin center) of each image. The depth center is obtained by the following formula:

[0129]

[0130] Where Feature refers to the input features of the high-level decoder (i.e., the features enhanced by the DQI module, specifically the third-layer features output by the aforementioned DQI), and b i It is the predicted depth center width.

[0131]

[0132] Where d min and d max These represent the minimum and maximum depth distances, respectively.

[0133] (6) Prediction section

[0134] In the final depth map generation process, the output of the last decoder layer is passed through a convolutional network to obtain pixel-by-pixel results. Finally, SoftMax probabilistics are used to obtain the probability of each pixel at different depth centers, and then the depth map is obtained using the following formula:

[0135]

[0136] in For the final depth map, c(b) k For different depth centers, p k This represents the probability value of each pixel at different depth centers.

[0137] Based on the descriptions in (5) and (6) above, in this embodiment, the predicted depth map is obtained based on the spliced ​​output features:

[0138] The features enhanced by the DQI module are input into the BCP module to predict the depth center of each image;

[0139] In the process of generating the final depth map, the spliced ​​output features of the encoder (the output features of the last reversible decoder) are passed through a convolutional network to obtain pixel-by-pixel results, and SoftMax is used to probabilize them to obtain the probability of each pixel at different depth centers.

[0140] A depth map is obtained based on the probability of each pixel at different depth centers and the different depth centers.

[0141] It can be understood that:

[0142] The reason why reversible modules are trained at lower levels is:

[0143] 1. Indirect supervision must be satisfied;

[0144] 2. Low-level features directly correlated with predictions; open training enhances network capabilities.

[0145] The decoder freezes at higher levels because:

[0146] The higher layers are general and versatile, and they use transformers to increase the receptive field and global feature representation capabilities, so they don't need to be changed. They can be viewed as a fixed function that guarantees the network's invertibility; otherwise, the open layer wouldn't guarantee that the one-dimensional structure isn't invertible.

[0147] The data processing flow of each module in the network of this embodiment will be described in the following sections.

[0148] (II) Reversible Transformation Enhanced Indirect Diffusion (InvT-IndDiffusion) Module

[0149] From the perspective of depth estimation recovery, denoising diffusion models can be used to directly improve predicted depth maps or intermediate features. However, for supervised monocular depth estimation (MDE), the ground truth depth map is typically derived from LiDAR point clouds and exists in a sparse form. This makes it unsuitable as input to diffusion models, as the diffusion process trains the model by adding noise to the input and then denoising it based on spatial features. On the other hand, latent space diffusion models, such as Stablediffusion, have been extensively studied for feature enhancement, training feature denoising capabilities iteratively in the latent space.

[0150] However, in the MDE task, we only have sparse radar ground truth without corresponding feature map ground truth, and we can only indirectly supervise the diffusion model with the final depth map predicted by the network, which cannot provide good depth prediction results.

[0151] Therefore, in this embodiment of the invention, we propose a reversible transform-enhanced indirect diffusion (InvT-IndDiffusion) module, namely the aforementioned reversible transform-enhanced indirect diffusion depth estimation network that includes a denoising diffusion network and a reversible transform-enhanced decoder. This facilitates latent feature diffusion through indirect optimization of depth prediction. This process can be represented as:

[0152] D = f m (f ID (F in Eq1

[0153] Among them, D and F in These represent the predicted depth map and the input features, respectively. m and f ID Let represent the decoder function and the indirect diffusion function, respectively, for decoding features to predict the depth map.

[0154] The key problem with this indirect diffusion is the inconsistency between the optimization trajectories of features and predicted depth, such as... Figure 3 As shown. (Note that the diffusion model is trained and inferred iteratively, with the output of each iteration serving as the input for the next.) When using indirect supervision of diffusion, this problem can be mitigated by using a reversible decoder under the double Lipshitz condition.

[0155] Suppose there exists a true feature F gt To generate a true depth map, in each diffusion step, the feature should be directed towards F. gt Optimization is performed so that the results of each step can be further optimized in subsequent steps without deviating from F. gt However, in indirect diffusion, it is only indirectly supervised by the depth map, and the optimization at each step may deviate from F at different steps. gt Therefore, iterative reasoning faces challenges in the testing phase.

[0156] To address the aforementioned indirect diffusion problem, the decoder function f... m The following conditions must be met:

[0157]

[0158] Among them, F gt This represents the feature ground truth corresponding to the existing depth ground truth map from the perspective of restoration. And F... n This represents the feature predictions generated in the different steps of indirect diffusion. ||·|| is the norm that measures the distance between them. Therefore, f m (F gt ) corresponds to the depth truth map, while f m (F n This is a predicted depth map. In other words, a decrease in the distance between depth values ​​leads to a decrease in the distance between features. This can be achieved through a distance-preserving mapping |||f m (F n )-f m (F gt )||=||F n -F gt The condition can be relaxed if the distance after mapping is positively correlated with the distance.

[0159] However, it is very difficult to design a neural network that satisfies the above strict constraints while maintaining strong nonlinear learning capabilities.

[0160] To alleviate this situation, the constraints can be relaxed using Lipschitz's theorem, which is defined as follows:

[0161] |f(x1)-f(x2)|≤K|x1-x2| Eq3

[0162] For a Lipschitz continuous function f, the distance between function outputs is always less than the distance between the original variables multiplied by a real constant K≥0.

[0163] By applying the above Lipschitz theorem to Eq. 3, it becomes:

[0164]

[0165] Assume the function f m is invertible and satisfies the bi-Lipschitz condition, then the above equation Eq. 4 can be further constrained as:

[0166]

[0167] In this way, as long as the decoder function is invertible and satisfies bi-Lipschitz continuity, the features can be optimized to the hypothesized feature truth values by minimizing the depth map loss. Such functions can be implemented by invertible residual networks or any invertible neural layers with Lipschitz constraints.

[0168] Therefore, in this embodiment, we propose to use invertible neural layers to construct an invertible decoder module.

[0169] Specifically, an affine coupling layer is used to construct the invertible decoder. Each coupling layer divides the input features into two parts at position s < S. One part is directly connected to the output as the key, while the other part is processed by an affine mapping, where the affine factors are generated together with the key through a neural network.

[0170] This process is alternately executed within the coupling layer:

[0171]

[0172] where ⊙ is the Hadamard product, exp(·) and σ c (·) represent the exponential function and the activation function respectively. g(·) and h(·) represent the networks for generating the affine factors.

[0173] It can be understood that the restored features are divided into the first part and the second part by the affine coupling layer. In each part, the forward propagation of the current part and the affine mapping of the other part are performed, and the processing results of each part are concatenated to obtain the concatenated features.

[0174] This embodiment employs a three-layer coupling layer to construct a reversible decoder module. The three affine coupling layers are connected in series, with the output of the previous affine coupling layer serving as the input of the next affine coupling layer. Combined with a diffusion model, this forms a reversible transformation-enhanced indirect diffusion (InvT-IndDiffusion). The proposed InvT-IndDiffusion can be used not only for depth estimation but also for other potential diffusion models with only indirect supervision from the final task.

[0175] (III) Depth estimation based on feature recovery from InvT-IndDiffusion

[0176] As mentioned earlier, optimizing high-level features can significantly improve performance. Our goal is to design a denoising diffusion process for high-level semantic features to achieve feature recovery. Therefore, we concatenate two high-level features together as input to InvT-IndDiffusion.

[0177] In traditional DDPM, the input is progressively transformed into random noise during the forward process, and inference begins from this random noise. However, for depth estimation, the task is to recover a specific target, and considering that the original features contain all the encoder information necessary for the final depth estimation, noise is progressively added according to a noise schedule while the features remain unchanged without scaling. The forward process is defined as follows:

[0178]

[0179] Where F gt and F deg These are the hypothetical true features that need to be recovered and the noise features contained in the input encoder features, respectively. The noise is added from a Gaussian distribution, with the corresponding coefficient α. t It is a noise strategy. Where at the maximum forward step size T F t F represents the feature map with added noise up to step t; t-1 F represents the feature map with added noise up to step t-1; t-2 This represents the feature map with noise added up to step t-2.

[0180] By introducing noise, diverse degenerate features can be generated, and the structure of the original features can be partially disrupted, allowing them to escape potential local optima. These features can then be recovered through back-reasoning.

[0181] When recovering the hypothetical ground truth features from the input noisy features, features from different layers are treated as cross-conditional maps to fully utilize multi-scale information. Multi-scale features are upsampled to the same scale using Pixel Shuffle, and then noise is added as a cross-conditional graph (C) of the InvT-IndDiffusion network. mul ).

[0182] Specifically, the diffusion model is used as the recovery network R θ (F t ,t,C mul To predict degradation and added noise F deg and ∈ t Then, as can be seen from Eq.7, the features recovered at each step can be obtained through... Obtain, among which θ represents the enhanced feature predicted using the result of step t; θ represents the network parameters. and These represent feature degradation noise predicted using network parameters and added noise predicted using network parameters, respectively.

[0183] Therefore, the inverse probability can be expressed as:

[0184]

[0185] Where q θ (·) indicates that from F t To F t-1 The transition probability. The deterministic sampling process resuming to the next step t-1 can be expressed as:

[0186]

[0187] By iteratively performing this reverse reasoning, the final recovery features can be obtained.

[0188] The predicted depth map is supervised using SiLog loss, as shown below:

[0189] L(θ)=E[‖‖D gt -D pre ||], Eq10

[0190] Where D gt and D pre Let represent the actual ground depth and the predicted ground depth, respectively, and ||·|| represent the SiLog distance.

[0191] (iv) Low-level feature enhancement module based on auxiliary viewpoint (AV-LFE)

[0192] The InvT-IndDiffusion described above recovers high-level features, thereby improving the performance of depth estimation. On the other hand, low-level features can also improve depth estimation to some extent, such as those in the motivation layer. Figure 1 and Figure 2 As shown. However, low-level features mostly provide local details, which InvT-IndDiffusion cannot effectively handle.

[0193] Therefore, we developed an auxiliary viewpoint-based low-level feature enhancement module (AV-LFE) to improve the skip connections used in the UNet architecture by enhancing low-level spatial features when an auxiliary viewpoint is available.

[0194] Given that some scenarios lack an auxiliary viewpoint, our AV-LFE module can operate in two modes: a compatible mode and a fully trainable mode. In compatible mode, AV-LFE is used as a plug-and-play module, with the encoder and decoder parameters frozen during training, training only this module. In this case, AV-LFE is an optional module that can improve performance with an auxiliary viewpoint without causing a performance degradation in monocular estimation without an auxiliary viewpoint. In fully trainable mode, the AV-LFE module will be trained along with the entire network to achieve optimal performance.

[0195] Specifically, the low-level features of the auxiliary viewpoint share the encoder with the main viewpoint, and only pass through the first two layers of the encoder. The proposed AV-LFE module concatenates the encoder features of the main and auxiliary viewpoints as input, and passes the processing result as a skip connection to the decoder to enhance the original skip connection.

[0196] First, deformable convolutions are used to align the features of the auxiliary viewpoint with those of the main viewpoint to reduce the differences between them. Then, features are fused through 3×3 and 1×1 convolutional layers, reducing the dimensionality to the same level as the encoder features of the main viewpoint. This way, the entire network can be replaced without changing it, which is compatible with using only the main viewpoint.

[0197] (V) Experiment Setup and Introduction

[0198] Our model was implemented on the PyTorch platform. Some parameters of the backbone network were pre-trained on the relevant dataset and remained unchanged, including the top two layers of the encoder and decoder, DQI, and BCP. During feature recovery training, only InvT-IndDiffusion and AV-LFE were activated. Our batch size was set to 4, and training was performed for 30 epochs. The training process used the RAdam optimizer, with a learning rate policy of 4×10⁻⁶. -5 Linearly reduced to 4×10 -6The performance evaluation metrics used include root mean square error (RMSE), absolute relative error (Abs Rel), relative squared error (Sq Rel), root mean square logarithmic error (RMSE log), and outlier pixel percentage (δ). k ).

[0199] Comparative analysis of experimental results:

[0200] The present invention was compared with other existing related methods on different datasets, and the results are shown in Tables 1 and 2. Table 1 shows the results on the KITTI dataset, Table 2 shows the results on the DDAD dataset, and Table 3 shows the generalization results on the Make3d dataset.

[0201] We compared the performance of the proposed method both indoors and outdoors. For a comprehensive comparison, we compared it with several other solutions:

[0202] In Table 1, we categorize the models into regression-based models and classification-regression-based models. DDP and EcoDepth are methods based on Denoise Diffusion.

[0203] Table 2 shows our testing on the novel outdoor depth estimation benchmark DDAD. AFnet uses its publicly available results, while Newcrf and Pixelforemer, for which no results were available on this dataset, were trained using their open-source code.

[0204] In Table 3, all models were trained on the KITTI dataset and then tested on the Make3D dataset for generalization without additional training.

[0205] As can be seen, the proposed solution achieves significantly better results and improved performance metrics compared to other current algorithms on major outdoor autonomous driving datasets. Furthermore, the proposed method shows a marked performance improvement after using the AVE-FM assisted viewpoint enhancement module, and this module does not degrade the performance of the network's monocular depth estimation when running in compatibility mode.

[0206] Therefore, the algorithm proposed in this invention can improve performance to varying degrees on different datasets, proving its effectiveness.

[0207] Table 1

[0208]

[0209] Table 2

[0210]

[0211] Table 3

[0212]

[0213] Table 1 compares the quantitative results of the proposed method with existing methods, using the Eigen segmentation strategy on the KITTI

[19] dataset. Here, '↓' indicates smaller is better, while '↑' indicates larger is better. ΔRMSE is calculated by comparing with the baseline PixelFormer. This indicates the results obtained using InvT-IndDiffusion and AV-LFE in compatibility mode. This indicates the results obtained using InvT-IndDiffusion and AV-LFE in fully trainable mode.

[0214] Table 2 shows a comparison of quantitative results on the DDAD dataset. '↓' indicates smaller is better, and '↑' indicates larger is better. ΔRMSE is calculated by comparing with the baseline PixelFormer. {'*' indicates the use of time information. Ours-N(-P) indicates the use of NeWCRFs (PixelFormer) as the backbone.}

[0215] Table 3 shows the zero-shot generalization experiments performed on Make3D. ΔRMSE was calculated by comparing with Monodepth2. ours-N(-P) indicates that NeWCRFs (PixelFormer) were used as the backbone.

[0216] Example 2

[0217] This embodiment discloses a depth estimation system based on an indirect diffusion model.

[0218] Depth estimation systems based on indirect diffusion models include:

[0219] The image acquisition module is configured to acquire the image to be processed.

[0220] The encoder feature processing module is configured to: input the image to be processed into the encoder of the U-NET network, perform layer-by-layer feature extraction, and obtain multiple high-level features and multiple low-level features;

[0221] The network model building module is configured to build an indirect diffusion depth estimation network with reversible transform enhancement, which includes a denoising diffusion network and a reversible transform enhancement decoder.

[0222] The recovery module is configured to: concatenate multiple high-level features and input them into a denoising diffusion network for feature recovery, predict degradation and added noise, and obtain the final recovered features;

[0223] The high-level decoder module is configured to input the recovered features into the high-level decoder of the reversible transform enhancement decoder, perform layer-by-layer feature extraction, and finally obtain the high-level decoded features.

[0224] The low-level decoder module is configured to: concatenate low-level features and high-level decoded features to obtain concatenated input features; input the concatenated input features into the low-level decoder of the reversible transform enhancement decoder; use an affine coupling layer to divide the concatenated input features into a first part and a second part; in each part, perform forward propagation of the current part and affine mapping processing of the other part; and concatenate the outputs of each part to obtain the concatenated output features.

[0225] The prediction module is configured to obtain a predicted depth map based on the spliced ​​output features.

[0226] Example 3

[0227] The purpose of this embodiment is to provide a computer-readable storage medium.

[0228] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps in the depth estimation method based on an indirect diffusion model as described in Embodiment 1 of this disclosure.

[0229] Example 4

[0230] The purpose of this embodiment is to provide an electronic device.

[0231] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the depth estimation method based on the indirect diffusion model as described in Embodiment 1 of this disclosure.

[0232] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0233] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0234] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A depth estimation method based on an indirect diffusion model, characterized in that, Includes the following steps: Obtain the image to be processed; The image to be processed is input into the encoder of the U-NET network for layer-by-layer feature extraction, resulting in multiple high-level features and multiple low-level features. A reversible transform-enhanced indirect diffusion depth estimation network was constructed, comprising a denoising diffusion network and a reversible transform-enhanced decoder. Multiple high-level features are concatenated and then input into a denoising diffusion network for feature recovery, predicting degradation and added noise, and obtaining the final recovered features. The input to the denoising diffusion network for feature recovery specifically involves adding noise to F. We obtain the features Ft at different time steps and define the forward process; we then define the degenerate features of F. It is also regarded as a kind of noise. A denoising diffusion network is used for feature enhancement and denoising to obtain the recovered features at each step, and then to obtain the final recovered features; where F is the original feature input obtained by concatenating multiple high-level features; The recovered features are input into the higher-level decoder of the reversible transform-enhanced decoder for layer-by-layer feature extraction, ultimately yielding the higher-level decoded features. Specifically, the recovered features are split into multiple parts corresponding to the number of layers in the higher-level decoder. One of these split parts is input into a DQI module for transform enhancement to improve expressive power, resulting in the enhanced output features of the DQI module, which includes multiple convolutional layers. The enhanced output features are then concatenated with the corresponding split parts and input into the highest layer of the higher-level decoder for feature extraction. The remaining split parts are input into other corresponding layers of the higher-level decoder for feature extraction. In the higher-level decoder, the output of the previous layer serves as the input of the next layer. After feature extraction layer by layer, the final higher-level decoded features of the entire higher-level decoder output are obtained. The low-level features and high-level decoded features are concatenated to obtain the concatenated input features. The concatenated input features are then input into the low-level decoder of the reversible transform enhancement decoder. The concatenated input features are divided into a first part and a second part using an affine coupling layer. In each part, the forward propagation of the current part and the affine mapping of the other part are performed. The outputs of each part are then concatenated to obtain the concatenated output features. Based on the spliced ​​output features, a predicted depth map is obtained.

2. The depth estimation method based on the indirect diffusion model as described in claim 1, characterized in that, Also includes: Determine whether the image to be processed comes from the main viewpoint and the secondary viewpoint: If the image to be processed comes only from the main viewpoint: The low-level features extracted by the encoder are then fed into the corresponding layer of the reversible transform enhancement decoder via skip connections. If the image to be processed comes from both the main viewpoint and the secondary viewpoint: The low-level features of the main viewpoint and auxiliary viewpoint encoders are obtained separately, concatenated, and then input into the low-level feature enhancement module based on the auxiliary viewpoint. Deformable convolution is used to align the extracted low-level features of the main viewpoint and auxiliary viewpoint, and after feature fusion, they are input into the corresponding layer of the reversible transform enhancement decoder. The encoder of the U-NET network was pre-trained on a depth estimation task; The low-level features are features extracted from the pre-set layer of the encoder, and the high-level features are features extracted from the post-set layer of the encoder. The reversible transform enhancement decoder includes a high-level decoder and a low-level decoder. The high-level decoder corresponds to the number of layers of high-level features, and the low-level decoder corresponds to the number of layers of low-level features.

3. The depth estimation method based on the indirect diffusion model as described in claim 1, characterized in that, Multiple high-level features are concatenated and then input into a denoising diffusion network for feature recovery, predicting degradation and added noise, and obtaining the final recovered features, specifically including: Multiple high-level features are upsampled to the same spatial resolution and then concatenated to form the original feature input F; Noise is added to F and input into a denoising diffusion network. High-level features from different layers are treated as mutual conditional mappings. Enhanced features, i.e. restored features, are generated through the denoising diffusion network. The generation of enhanced features through the denoising diffusion network includes: designing the decoder function to be invertible and satisfying bidirectional Lipshitz continuity; using the predicted depth map as indirect supervision of the denoising diffusion network; and optimizing F to the hypothetical ground truth of features by minimizing the depth map loss.

4. The depth estimation method based on the indirect diffusion model as described in claim 3, characterized in that, The defined forward process is specifically as follows: ； in, and These are the hypothetical true features and the degenerate features that need to be restored; The noise is added from a Gaussian distribution, and the corresponding coefficients are... It is a noise strategy; ; This represents the feature map with noise added up to step t; This represents the feature map with added noise up to step t-1; This represents the feature map with added noise up to step t-2; The features recovered at each step are obtained using the following formula: ； in, This represents the enhanced feature predicted using the result of step t; Indicates network parameters; and These represent feature degradation noise predicted using network parameters and added noise predicted using network parameters, respectively.

5. The depth estimation method based on the indirect diffusion model as described in claim 1, characterized in that, The recovered features are divided into a first part and a second part using an affine coupling layer. In each part, forward propagation of the current part and affine mapping processing of the other part are performed, specifically including: Restore features Divided into the first part Part Two The location is And calculate them separately: ； in, It is the Hadamard product. and These represent the exponential function and the activation function, respectively. and This represents the network used to generate affine factors; and These represent the outputs of the first part and the second part, respectively. The outputs of the first part and the second part are concatenated to form the output of an affine coupling layer.

6. The depth estimation method based on the indirect diffusion model as described in claim 1, characterized in that, Based on the concatenated output features, a predicted depth map is obtained, specifically including: The output features enhanced by the DQI module are input into the BCP module to predict the depth center of each image; In the process of generating the final depth map, the output features, namely the output features of the last layer of the decoder, are spliced ​​together and passed through a convolutional network to obtain pixel-by-pixel results. SoftMax is then used to probabilize the data to obtain the probability of each pixel at different depth centers. A depth map is obtained based on the probability of each pixel at different depth centers and the different depth centers; Using SiLog loss Supervise the predicted depth map: in and These represent the actual depth and the predicted depth on the ground, respectively. This represents the SiLog distance.

7. A depth estimation system based on an indirect diffusion model, characterized in that, include: The image acquisition module is configured to acquire the image to be processed. The encoder feature processing module is configured to: input the image to be processed into the encoder of the U-NET network, perform layer-by-layer feature extraction, and obtain multiple high-level features and multiple low-level features; The network model building module is configured to build an indirect diffusion depth estimation network with reversible transform enhancement, which includes a denoising diffusion network and a reversible transform enhancement decoder. The recovery module is configured to: concatenate multiple high-level features and input them into a denoising diffusion network for feature recovery, predict degradation and added noise, and obtain the final recovered features; specifically, inputting them into the denoising diffusion network for feature recovery involves: adding noise to F. We obtain the features Ft at different time steps and define the forward process; we then define the degenerate features of F. It is also regarded as a kind of noise. A denoising diffusion network is used for feature enhancement and denoising to obtain the recovered features at each step, and then to obtain the final recovered features; where F is the original feature input obtained by concatenating multiple high-level features; The high-level decoder module is configured to: input the recovered features into the high-level decoder of the reversible transform-enhanced decoder, perform layer-by-layer feature extraction, and finally obtain the high-level decoded features. Specifically, the recovered features are split into multiple parts corresponding to the number of layers in the high-level decoder. One of the split parts is input into the DQI module to enhance its expressive power, resulting in the output features enhanced by the DQI module, which includes multiple convolutional layers. The output features enhanced by the DQI module are concatenated with the corresponding split parts and then input into the highest layer of the high-level decoder to extract features. The remaining split parts are input into other corresponding layers of the high-level decoder to extract features respectively. In the high-level decoder, the output of the previous layer is used as the input of the next layer. After extracting features layer by layer, the high-level decoded features of the entire high-level decoder output are finally obtained. The low-level decoder module is configured to: concatenate low-level features and high-level decoded features to obtain concatenated input features; input the concatenated input features into the low-level decoder of the reversible transform enhancement decoder; use an affine coupling layer to divide the concatenated input features into a first part and a second part; in each part, perform forward propagation of the current part and affine mapping processing of the other part; and concatenate the outputs of each part to obtain the concatenated output features. The prediction module is configured to obtain a predicted depth map based on the spliced ​​output features.

8. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by a processor, the program implements the steps in the depth estimation method based on the indirect diffusion model as described in any one of claims 1-6.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the depth estimation method based on the indirect diffusion model as described in any one of claims 1-6.

Images