Flight-spot-free monocular depth estimation method and device based on pixel space diffusion generation

Abstract

The invention relates to the technical field of computer vision depth estimation, in particular to a flying-spot-free monocular depth estimation method and a flying-spot-free monocular depth estimation device based on pixel space diffusion generation. The high-level semantic representation is received as a guide signal through a semantic guide DiT block, so that modeling of a global image structure and fine-grained details is enhanced, and simplicity and expandability of the DiT block are kept at the same time; the semantic guidance DiT block effectively simulates a global image structure and fine-grained details, so that the accuracy of depth prediction is remarkably improved; by integrating high-level semantic representation, diffusion depth prediction is directly executed in a pixel space instead of a potential space, pixel distribution of depth is allowed to be directly simulated, the situation that a depth map is compressed into the potential space through VAE and then depth estimation is conducted is avoided, and therefore the influence of flying spots on a depth estimation map is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision depth estimation, in particular to a no-flying-point monocular depth estimation method and device based on pixel space diffusion generation. BACKGROUND

[0002] Monocular depth estimation is a fundamental task with a wide range of downstream applications, such as 3D reconstruction, novel view synthesis, and robot manipulation. Due to its importance, a large number of depth estimation models have emerged. These depth estimation models achieve high-quality results in most zero-mirror scenes or regions, but are affected by flying points around the boundaries of objects and fine details when converted into point clouds, which limits their practical application in tasks such as free-viewpoint broadcasting, robot manipulation, and immersive content creation.

[0003] Existing models suffer from the problem of flying points for several different reasons. For discriminative models, flying points mainly arise from their tendency to output the average value between foreground and background depths at depth discontinuities in order to minimize the regression loss. In contrast, generative depth models bypass direct regression by modeling the depth distribution of pixels, which allows them to preserve sharp edges and recover fine structures. However, current generative depth models typically fine-tune Stable Diffusion, a latent diffusion model, for depth estimation, which requires a Variational Autoencoder (VAE) to compress depth maps into a latent space. This compression inevitably leads to a loss of edge sharpness and structure fidelity, resulting in a large number of flying points.

[0004] In view of this, overcoming the defects of the prior art is a problem urgently to be solved in the technical field. SUMMARY

[0005] The technical problem to be solved by the present application is how to solve the problem that the depth estimation map of the prior art is prone to flying points.

[0006] The present application adopts the following technical solutions: In a first aspect, a no-flying-point monocular depth estimation method based on pixel space diffusion generation is provided, comprising: extracting high-level semantic representations in the input image through a visual base model; adding Gaussian noise to the real depth image of the input image to obtain a noise sample, and splicing the noise sample with the input image to obtain a joint image; An intermediate feature map is obtained by processing global features of the joint image through a standard DiT block, the high-level semantic representation is received by a semantic-guided DiT block as a guiding signal, and fine-grained details in the intermediate feature map are modeled to obtain a depth estimation image; Parameters in the standard DiT block and the semantic-guided DiT block are updated according to the depth estimation image and the real depth image to obtain a trained depth prediction model, and monocular depth estimation is performed on an input image according to the trained depth prediction model.

[0007] Preferably, the intermediate feature map is obtained by processing global features of the joint image through a standard DiT block, specifically comprising: The joint image is input into the standard DiT block, a first layer of the standard DiT block converts the joint image into a one-dimensional sequence of tokens by performing a linear embedding operation on each local token with a size of p x p in the joint image, and the dimension of each token is ; ; The tokens in the one-dimensional sequence are processed by a sequence of converter blocks of the standard DiT block to mine global feature correlations between the tokens to generate the intermediate feature map.

[0008] Preferably, the method further comprises: A real depth image of an input image is combined with Gaussian noise to generate a noise sample , where ∈[0, 1], and specifically represented as: t

[0009] A velocity field defining a noise adding process is learned by a neural network to obtain a velocity field , where is a feature representation of the input image; In the inference stage, the real depth image is gradually recovered from the noise sample based on the predicted velocity field by a discretized ordinary differential equation solver, and the specific iterative formula is:

[0010] Where the noise sample is converted into the real depth image by multiple iterations.

[0011] ​​​Preferably, the expression formula of the high-level semantic representation is:

[0012] wherein, is the high-level semantic representation, is a visual base model, is the number of tokens, is the embedding dimension of the visual base model, is a real number set.

[0013] Preferably, the method further comprises: normalizing the high-level semantic representation, and the normalized high-level semantic representation is:

[0014] wherein, is the high-level semantic representation, is the normalized high-level semantic representation.

[0015] Preferably, the high-level semantic representation is received by the semantic-guided DiT block as a guide signal, specifically comprising: through a multi-layer perceptron network integrating the normalized high-level semantic representation set into the tokens of the semantic-guided DiT block as a guide signal:

[0016] wherein, denotes a bilinear interpolation operator, denotes DiT tokens enhanced with semantics, is the normalized high-level semantic representation.

[0017] Preferably, the method further comprises: normalizing the real depth image of the input image.

[0018] Preferably, the method further comprises: extracting an edge mask from the real depth image using an edge detection operator; projecting the real depth image as a real point cloud and projecting the depth estimation image as a predicted point cloud; calculating chamfer distance between the predicted point cloud and the real point cloud near the edge based on the edge mask, and taking the chamfer distance as a loss term to minimize the geometric difference between the predicted point cloud and the ground truth point cloud.

[0019] In a second aspect, a pixel space diffusion generation based fly point free monocular depth estimation device is provided, comprising a processor and a memory for storing processor executable instructions. The processor is configured to perform the pixel space diffusion generation based fly point free monocular depth estimation method.

[0020] In a third aspect, a non-volatile computer storage medium is provided, which stores computer executable instructions executed by one or more processors to complete the pixel space diffusion generation based fly point free monocular depth estimation method of the first aspect.

[0021] In a fourth aspect, a chip is provided, comprising a processor and an interface for calling and running a computer program stored in a memory to perform the pixel space diffusion generation based fly point free monocular depth estimation method of the first aspect.

[0022] In a fifth aspect, a computer program product comprising instructions which, when executed on a computer or processor, cause the computer or processor to perform the pixel space diffusion generation based fly point free monocular depth estimation method of the first to fourth aspects and any one thereof.

[0023] Compared with the prior art, the present application has the following beneficial effects: The present application extracts high-level semantic representation in the input image through the visual base model, and receives the high-level semantic representation as a guide signal through the semantic guided DiT block, enhances the modeling of global image structure and fine-grained details, while retaining the simplicity and scalability of the DiT block; the semantic guided DiT block effectively simulates the global image structure and fine-grained details, thereby significantly improving the accuracy of depth prediction; by integrating high-level semantic representation, depth prediction is directly performed in pixel space rather than latent space, allowing direct simulation of the pixel distribution of depth rather than compressing the depth map into latent space through VAE for depth estimation, thereby avoiding the influence of fly points on the depth estimation map; in summary, the present application takes the diffusion model as the core, integrates the high-level semantic representation extracted by the visual base model, and realizes accurate prediction from image to depth map through the semantic guided cascading diffusion process, and the training relies on depth ground truth, and the inference can directly generate depth estimation image from image. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0025] Figure 1 is a flowchart of a pixel space diffusion generation-based no-flying point monocular depth estimation method provided by an embodiment of the present application; Figure 2 is a principle diagram of a pixel space diffusion generation-based no-flying point monocular depth estimation method provided by an embodiment of the present application; Figure 3 is a flowchart of generating an intermediate feature map provided by an embodiment of the present application; Figure 4 is another flowchart of a pixel space diffusion generation-based no-flying point monocular depth estimation method provided by an embodiment of the present application; Figure 5 is a comparison diagram of the present application and prior models provided by an embodiment of the present application; Figure 6 is a qualitative comparison diagram provided by an embodiment of the present application; Figure 7 is a qualitative point cloud result diagram in a complex scene provided by an embodiment of the present application; Figure 8 is a qualitative ablation diagram of SG-DiT provided by an embodiment of the present application; Figure 9 is a qualitative comparison diagram of the present application and MoGe provided by an embodiment of the present application; Figure 10 is a flying point diagram in different types of VAE provided by an embodiment of the present application; Figure 11 is a diagram of comparing the present application with recent depth estimation models on five real-world benchmarks provided by an embodiment of the present application; Figure 12 is an ablation study diagram of zero-shot learning benchmark testing provided by an embodiment of the present application; Figure 13 is an ablation diagram on a visual basis model provided by an embodiment of the present application; Figure 14 is a quantitative result diagram provided by an embodiment of the present application; Figure 15 is a quantitative comparison diagram of the present application and representation alignment provided by an embodiment of the present application; Figure 16 is a structural schematic diagram of a non-flying-point monocular depth estimation device based on pixel space diffusion generation provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0027] Unless otherwise required by context, the term "comprises" in the specification and claims is to be construed as open-ended, i.e. as "comprises but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example" or "some examples" are intended to mean that the specific feature, structure, material or characteristic associated with that embodiment or example is included in at least one embodiment or example of the present disclosure. The illustrative representations of the above terms do not necessarily mean the same embodiment or example. In addition, the specific features, structures, materials or characteristics described can be included in any one or more embodiments or examples in any appropriate manner, i.e. although they are carried in the embodiments or examples of the above terms due to the order of appearance and position, they are not limited to being carried by one embodiment or example in a combined manner.

[0028] In the description of the present application, the terms "first", "second" are only used for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "a plurality of" is two or more. In addition, for example, in the description, the same type of nouns can also be described as two independent individuals by adding "A", "B" at the end, in which case the features limited by "A", "B" are only used for the purpose of distinguishing the same type of individual description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features.

[0029] In describing some embodiments, the terms "coupled," "coupled," and "connected," and their derivative expressions, may be used. For example, the term "connected" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact with each other. Similarly, the term "coupled" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact. However, the terms "connected" or "coupled" may also refer to two or more components that do not have direct contact with each other but still cooperate or interact with each other, such as "optical coupling," "wireless connection," etc. The embodiments disclosed herein are not necessarily limited to the scope of this invention.

[0030] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0031] Example 1: Existing depth prediction models suffer from the flying point problem for various reasons. A simple solution is to train a diffusion-based depth model in pixel space, thereby bypassing the use of VAEs. However, training a diffusion-based depth model in pixel space is difficult to achieve because the complexity and instability of modeling global image structure and fine-grained details are significant, resulting in low depth prediction quality.

[0032] To address the problem of flying points (depth anomaly discrete points) that easily occur in traditional monocular depth estimation, this embodiment proposes a flying point-free monocular depth estimation method based on pixel spatial diffusion generation. In one embodiment, such as... Figure 1 and Figure 2 As shown, this method integrates visual semantic guidance and diffusion model feature modeling. Starting from the input image, it extracts semantics, fuses noise, and optimizes features in a two-stage process to output accurate and spot-free depth prediction results. This includes constructing and training a depth prediction model, and finally using the trained model for depth prediction. The spot-free monocular depth estimation method based on pixel spatial diffusion includes: Step 101: Extract high-level semantic representations from the input image using a visual base model.

[0033] The input image whose depth is to be predicted (e.g.) Figure 2a scene graph Image containing a puppy, a kitten, etc., containing visual information such as object shape, scene structure, etc.). A visual base model is called, which can be a Masked Autoencoder (MAE), DINOv2 (English full name: Deformable Instance and Semantic Contrastive Learning v2), Depth Anything v2, or VGGT (English full name: Visual Geometry Grounded Transformer). It is trained in advance on a large amount of image data and has strong general feature extraction and semantic understanding capabilities. It encodes the input image.

[0034] In one embodiment, the expression formula of the high-level semantic representation is:

[0035] wherein, is the high-level semantic representation, is the visual base model, is the number of tokens, is the embedding dimension of the visual base model, is the set of real numbers.

[0036] In one embodiment, the high-level semantic representation may be unstable due to model or input differences and feature distribution. Therefore, after obtaining the high-level semantic representation, the pixel space diffusion-based generation of non-flying point monocular depth estimation method further includes: performing a normalization operation on the high-level semantic representation, and the normalized high-level semantic representation is:

[0037] wherein, is the high-level semantic representation, is the normalized high-level semantic representation.

[0038] Normalization can unify the feature mean and variance (such as standardization to mean 0 and variance 1), improve the stability of subsequent DiT block processing, and avoid training difficulty convergence or prediction of flying points due to feature scale differences. That is, in the subsequent process of receiving the high-level semantic representation as a guide signal through the semantic guide DiT block, the high-level semantic representation is the normalized high-level semantic representation.

[0039] In one embodiment, the visual base model splits the input image into several local patches, encodes each patch into a high-dimensional feature vector (i.e., token) through a multi-layer Transformer or convolutional network, and outputs a high-level semantic representation (a feature matrix with a shape of T' x D', T' is the number of tokens, and D' is the feature dimension, which contains abstract semantics of the image, such as "which regions are objects, which are backgrounds, and the class relationship between objects"). The semantic prior is provided for subsequent depth prediction, so that the model knows how the depth of different semantic regions in the scene should be distributed, and avoids flying points from a high-level cognitive perspective.

[0040] Step 102: Add Gaussian noise to the real depth image of the input image to obtain a noise sample, and splice the noise sample with the input image to obtain a joint image.

[0041] wherein the real depth image corresponding to the input image (such as Figure 2 Depth GT in the figure) is only used in the training stage, which is a precise annotation of the scene depth, recording the actual depth value of each pixel.

[0042] The real depth image is subjected to Gaussian noise addition (following the diffusion model logic, simulating the noise addition process from clean depth to noisy depth), to generate a noise sample (depth data with random noise, used to let the model learn the association between denoising and depth restoration).

[0043] Then the input image and the noise sample are spliced (i.e., Concat) in the channel dimension or the spatial dimension to obtain a joint image (a new input that integrates the scene semantics and the noisy depth dual modal information). To let the subsequent DiT model perceive the image semantics and the depth noise distribution at the same time, and provide complete information for learning depth denoising prediction under semantic constraints.

[0044] In one embodiment, the goal of the present embodiment is to estimate the depth map of the perfect depth of the pixels, to produce a clean point cloud without flying points. Flying points often occur at the edges of objects because the depth prediction of these regions is not accurate. However, existing evaluation benchmarks and metrics are generally difficult to reflect the flying points at the edges of objects. For example, metrics such as AbsRel (English full name: Absolute Relative Error) and d1 are mainly flat areas, so it is difficult to evaluate the depth accuracy of the edges. To solve these limitations, in one embodiment, the official test split of the Hypersim dataset is evaluated, which provides high-quality ground truth point clouds (i.e., the real depth image corresponding to the input image described above).

[0045] Step 103: Obtain an intermediate feature map by processing the global features of the joint image through a standard DiT block, receive the high-level semantic representation as a guiding signal through a semantic-guided DiT block, and model the fine-grained details in the intermediate feature map to obtain a depth estimation image.

[0046] wherein the standard DiT block is a Transformer-based basic module for extracting global multi-modal features of the joint image to generate an intermediate feature map; the intermediate feature map is the global feature output by the standard DiT block, containing multi-modal information but needing semantic optimization; the semantic-guided DiT block is used to optimize the fine-grained depth details of the intermediate feature map by injecting the high-level semantic representation, and suppress the flying points; and the depth estimation image is the final output of the depth prediction result, with consistent resolution with the input image and no obvious flying points.

[0047] This step works cooperatively through the standard DiT block and the semantic-guided DiT block, first extracts multi-modal global features, and then optimizes fine-grained details with semantic information to avoid flying points.

[0048] In one embodiment, the embodiment proposes a new cascaded DiT design to reduce the computational burden of the model. Specifically: a large patch is used in the early standard DiT block. This design significantly reduces the number of patches that need to be processed, thereby reducing the computational cost. In the later semantic-guided DiT block, the number of patches is increased, which is equivalent to using smaller patches. This allows the model to better focus on fine-grained spatial details.

[0049] For example: for a diffusion model with a total of N DiT blocks, the first N / 2 blocks constitute a coarse level with a larger tokens size, and the remaining N / 2 blocks use a fine level with a smaller tokens size. The resulting cascading design from coarse to fine reflects the hierarchical nature of visual perception, optimizing the efficiency and effectiveness of the process and improving the efficiency and accuracy of depth estimation. Specifically: a total of N=24 DiT blocks are used, each operating at a hidden dimension of D=1024. The first 12 blocks are standard DiT blocks with a patch size of 16, corresponding to (H / 16)×(W / 16) patches for an input of size H×W. The remaining 12 blocks are the proposed semantic-guided DiT blocks, which use a more fine-grained patch size of 8, resulting in (H / 8)×(W / 8) patches. After the 12th block, a Multilayer Perceptron (MLP) network is used to increase the hidden dimension by a factor of 4, and then reshaped to support more patches.

[0050] Step 104: updating the parameters in the standard DiT block and the semantic-guided DiT block according to the depth estimation image and the real depth image to obtain a trained depth prediction model, and performing monocular depth estimation on an input image according to the trained depth prediction model.

[0051] wherein the depth prediction model finally outputs a depth estimation image of the current training step (i.e., Depth Pred) after being processed by the standard DiT block and the semantic-guided DiT block. The depth estimation image and the corresponding real depth image are compared to calculate the difference (loss) therebetween, and common losses include mean square error or absolute value error, etc. All learnable parameters in the depth prediction model, including the parameters in the standard diffusion transformation block and the semantic-guided diffusion transformation block, are updated using the loss and through a gradient descent algorithm. The specific training process will be described below.

[0052] The above process is repeated until a preset number of training times is reached or a preset effect (e.g., the difference between the depth estimation image and the real depth image is minimized) is reached, and a trained depth prediction model is obtained. When the depth prediction model is trained, it can be used to predict the depth of a new input image.

[0053] The present application extracts high-level semantic representations in the input image through a visual base model, and receives the high-level semantic representations as guide signals through the semantic-guided DiT block, thereby enhancing the modeling of global image structure and fine-grained details while retaining the simplicity and scalability of the DiT block. The semantic-guided DiT block effectively simulates global image structure and fine-grained details, thereby significantly improving the accuracy of depth prediction. By integrating high-level semantic representations, depth prediction is performed directly in pixel space rather than latent space, allowing direct simulation of the pixel distribution of depth rather than compressing the depth map into latent space through VAE and then performing depth estimation, thereby avoiding the influence of flying points on the depth estimation image. In summary, the present application takes a diffusion model as the core, integrates high-level semantic representations extracted by a visual base model, and realizes accurate prediction from image to depth map through a semantic-guided cascading diffusion process. The training relies on depth ground truth, and the inference can directly generate a depth estimation image from an image.

[0054] In one embodiment, as shown in FIG. 1, the method comprises the following steps: Figure 3 As shown in FIG. 1, the method comprises the following steps: Step 1031: inputting the joint image into the standard DiT block, and performing a linear embedding operation on each local token with a size of p x p in the input joint image to convert the joint image into a one-dimensional sequence of T tokens in the first layer of the standard DiT block, and the dimension of each token is D.

[0055] where the joint image is divided into non-overlapping local patches of size p x p (e.g., 16 x 16 pixels), and each patch is treated as a local token.

[0056] For example, if the input image size is 224 x 224, then (224 / p) x (224 / p) local tokens are generated (e.g., when p = 16, there are 14 x 14 = 196 tokens).

[0057] For each local token of size p x p x 4, a linear transformation matrix W (dimension 4p2 x D) is used to map it to a feature vector of dimension D (i.e., 1 token).

[0058] After linear embedding of all local tokens, a one-dimensional sequence of length T = (H / p) x (W / p) is formed, with dimension T x D (e.g., when H = 224, p = 16, and D = 768, the sequence is 196 x 768).

[0059] This step is used to split the 2D image structure into a 1D token sequence, which facilitates the processing of global relationships by the Transformer. By linear projection, the features of different modalities (image + depth) are mapped to the same high-dimensional space (D dimensions), which facilitates subsequent fusion.

[0060] Step 1032: The tokens of the one-dimensional sequence are processed by the transformer block sequence of the DiT block to mine the global feature correlation between the tokens, to generate the intermediate feature map.

[0061] where the transformer block sequence includes L Transformer blocks in cascade, and each block contains: Multi-head self-attention: used to calculate the association weight of each token with all other tokens, to mine the global dependency relationship (e.g., the depth correlation between "dog head" and "dog foot").

[0062] Layer normalization: used to stabilize training and accelerate convergence.

[0063] Feedforward network: used to perform nonlinear transformation of features through two fully connected layers, to enhance expression ability.

[0064] Global correlation mining: used to learn the dependency relationship between tokens in different positions (e.g., depth consistency of semantically related regions) through self-attention mechanism, to suppress the generation of outliers.

[0065] The processed token sequence T x D is reshaped and transposed to restore the spatial resolution matching the input image: For example: T = (H / p) x (W / p), reshape T x D to (H / p) x (W / p) x D, and then adjust the channel through convolution or linear layer to obtain the intermediate feature map of H x W x D. The intermediate feature map integrates the global correlation of image semantics and noise depth, contains basic depth clues, but needs subsequent semantic guidance for further optimization.

[0066] In one embodiment, the pixel space diffusion-based non-flying point monocular depth estimation method further comprises: combining the real depth image of the input image with Gaussian noise to generate a noise sample , wherein t ∈[0, 1], and is specifically expressed as:

[0067] defining a velocity field of the noise adding process , and learning and predicting the velocity field through a neural network to obtain a velocity field , wherein is a feature representation of the input image.

[0068] In the inference stage, the real depth image is gradually recovered from the noise sample based on the predicted velocity field through a discretized ordinary differential equation solver . The specific iterative formula is:

[0069] wherein the noise sample is converted into the real depth image through multiple iterations . .

[0070] In one embodiment, the above process includes a noise adding process, a velocity field defining process, and depth prediction model training and inference stage. The noise adding process is used to simulate the continuous change from clean depth (real depth image) to noisy depth (noise sample), so that the depth prediction model learns the depth distribution rule. The time-dependent interpolation function , when t = 0, = (noiseless); when t = 1, = (all noise); the greater t is, the stronger the noise is.

[0071] The velocity field defining process is used to define the change speed of the noise adding process, so that the neural network learns to predict this speed, thereby mastering the rule of denoising to depth restoration. The neural network ​is the speed predictor that the model is to learn, a neural network The input to the model is a noisy sample , a time step t, input image features c (e.g., high-level semantic representations extracted in step 101), and the output is a predicted velocity field .

[0072] where the training objective of the depth prediction model is to make the velocity field predicted by the model (corresponding to the depth prediction image) as close as possible to the true velocity field (corresponding to the true depth image). In one embodiment, the training can be performed by the following equation:

[0073] The inference stage refers to using the trained depth prediction model to start from a pure noise and gradually denoise to obtain a clean depth (i.e., a depth estimation image). The core operation is: the noise adding process is continuous (controlled by time step t), and the [0, 1] time interval is divided into discrete time steps (e.g., (t1, t2,..., tn), iterating from t = 1 to t = 0) during inference. is the noise sample for the current time step (initially , is the velocity field predicted by the model for the current time step (corresponding to the depth estimation image). The iteration process is: starting from t = 1 (pure noise ), gradually decreasing t (e.g., t = 0.9, 0.8,..., 0), and each time using the velocity field predicted by the model to update the current noise sample in the direction of the clean depth, finally obtaining (i.e., the depth estimation image).

[0074] In one embodiment, the semantic-guided DiT block receives the high-level semantic representation as a guidance signal, specifically comprising: integrating the normalized high-level semantic representation into the tokens of the semantic-guided DiT block as a guidance signal:

[0075] wherein, denotes a bilinear interpolation operator that aligns the spatial resolution of the high-level semantic representation e with the spatial resolution of the DiT tokens. denotes the DiT tokens enhanced with semantics, is the normalized high-level semantic representation.

[0076] In one embodiment, as Figure 4 ​As shown, the pixel space diffusion based flypoint-free monocular depth estimation method further comprises: Step 201: Normalize the ground truth depth image of the input image to match the scale expected by the diffusion model.

[0077] The numerical range of the ground truth depth image can vary greatly. Normalization can make the depth value distribution more uniform (e.g., scaled to [0, 1] or standardized to mean 0, variance 1), making the model training more stable.

[0078] In one embodiment, before normalization, the depth values of the ground truth depth image are converted to a logarithmic scale to ensure a more balanced capacity allocation between indoor and outdoor scenes. Specifically, the transformation is applied, where represents the transformed depth, d is the original depth value, is a small normal number (e.g., 1) to ensure numerical stability. Then, the log-scaled depth is normalized using the following method:

[0079] where and may be the and depth percentiles of each image, respectively.

[0080] Step 202: Extract an edge mask from the ground truth depth image of the input image using an edge detection operator, and calculate the chamfer distance between the predicted point cloud near the edge and the ground truth point cloud.

[0081] In one embodiment, an edge detection operator is used to extract an edge mask from the ground truth depth image; the ground truth depth image is projected as a ground truth point cloud, and the depth estimation image is projected as a predicted point cloud; the chamfer distance between the predicted point cloud near the edge and the ground truth point cloud is calculated based on the edge mask, and the chamfer distance is taken as a loss term to minimize the geometric difference between the predicted point cloud and the ground truth point cloud.

[0082] In one embodiment, flypoints in depth prediction often occur at object edges (where the model has difficulty distinguishing between object and background depth jumps). By extracting an edge mask of the ground truth depth image (marking pixels with depth jumps); calculating the chamfer distance between the depth estimation image and the ground truth depth image near the edge (measuring the prediction error at the edge), and adding it to the loss function, the depth prediction model pays more attention to edge accuracy and reduces flypoints.

[0083] In one implementation, an edge detection operator (such as Canny) is applied to the real depth image to generate a binary edge mask (edge ​​pixels = 1, non-edge pixels = 0). The real depth image is then projected onto a real point cloud using camera intrinsics (only the points projected from the mask = 1 pixels are retained). The depth estimate image predicted by the depth prediction model is then projected onto a predicted point cloud (again, edge region points are retained). The chamfer distance between the two edge point clouds is then calculated and used as a loss term to directly measure the difference in 3D geometry at the boundary.

[0084] Example 2: In one embodiment, such as Figure 5 The image shows a comparison between a monocular depth estimation model with pixel space (i.e., Ours) and existing models (Marigold, Depth Anything v2, and Depth pro), which can generate high-quality point clouds without flying points.

[0085] In one embodiment, Figure 6 This is a qualitative comparison. GT (VAE) represents the ground reality depth map reconstructed by VAE. Existing generative models use VAE to compress the input into the latent space, inevitably introducing flying points at edges and details. In contrast, the model proposed in this invention (i.e., Ours) performs diffusion in pixel space, avoiding these problems. The depth map is visualized on the point cloud.

[0086] In one embodiment, Figure 7 It provides qualitative point cloud results for complex scenes. Compared to other depth estimation models (Marigold, Depth Anything v2, and Depth pro), the model proposed in this invention (i.e., Ours) produces significantly fewer flying points, and the depth map is overlaid on the point cloud for visualization.

[0087] In one embodiment, Figure 8 This is the qualitative ablation of the proposed SG-DiT. From top to bottom: input images from five benchmarks, results without SG-DiT, and results with SG-DiT. Without SG-DiT, pixel-space DiT models struggle to effectively model global image structure and fine-grained details.

[0088] In one embodiment, Figure 9 This is a qualitative comparison between the present invention and MoGe (Monocular Geometry). Among them, Figure 9 The image above shows the input images taken from four test sets: Hypersim, DIODE, ScanNet, and ETH3D. Figure 9 The middle image in the image shows the results from MoGe. Figure 9The lower plot in Figure 6: As a result of the present invention, MoGe, like previous discriminative models, suffers from flyby artifacts at the edges and in the details.

[0089] In an embodiment, Figure 10 is validation of flyby artifacts in different types of VAEs. Further qualitative comparisons are presented, showing that increasing the latent dimensionality in VAEs does not eliminate flyby artifacts. VAE-d4 (SD2) denotes the reconstruction of ground truth depth maps using the VAE from Stable Diffusion 2 with a latent dimensionality of 4, which was also used for Marigold. VAE-d16 (SD3.5) uses the VAE from Stable Diffusion 3.5 with a latent dimensionality of 16.

[0090] In an embodiment, to evaluate the zero-shot generalization of the model proposed by the present invention, it is compared to recent depth estimation models on five real-world benchmarks. As shown in Figure 11 all evaluation metrics outperform all other generative depth estimation models. Unlike previous generative models, the present invention does not rely on image priors from pre-trained Stable Diffusion models. Instead, the diffusion model is trained from scratch and still achieves superior performance. It generalizes well to a wide range of real-world scenarios, even if it is only trained on synthetic depth datasets. Furthermore, it outperforms discriminative models trained on a similar amount of training data. Unlike previous models using convolutional architectures, e.g., the denoising U-Net for the generative model and DPT (Dense Prediction Transformer) for the discriminative model, the present invention is purely based on transformers without convolutional layers.

[0091] In an embodiment, as shown in Figure 12 is an ablation study for the zero-shot learning benchmark. All metrics are expressed in percentages, and the best results are in bold numbers. Inference times were tested on an RTX 4090 GPU.

[0092] In an embodiment, as shown in Figure 13 is an ablation on the visual base model, using pre-trained visual encoders from different visual base models to evaluate the performance of SG-DiT, including MAE, DINOv2, Depth Anything v2, and VGGT.

[0093] In an embodiment, Figure 14Quantitative results in Table 2 show that the method proposed by the present application achieves the best performance. Discriminative models like Depth Pro and Depth Anything v2 tend to smooth edges, leading to fly points. Marigold et al. generative model relies on VAE compression, which blurs edges and details, leading to artifacts in the reconstructed point cloud. To illustrate this, the present application encodes and decodes ground-truth depth using VAE (GT(VAE)) without any generative process.

[0094] In one embodiment, Figure 15 For quantitative comparison with Representation Alignment (REPA) of the present application. To ensure fairness, the pre-trained visual encoder used in DiT+REPA and DiT+Ours remains the same.

[0095] Embodiment 3: A pixel space diffusion generation based fly point free monocular depth estimation method is provided in embodiment 1, and in the present embodiment a pixel space diffusion generation based fly point free monocular depth estimation device will be proposed. The pixel space diffusion generation based fly point free monocular depth estimation device comprises a processor and a memory for storing processor executable instructions; wherein the processor is configured to execute the pixel space diffusion generation based fly point free monocular depth estimation method described in embodiment 1.

[0096] As Figure 16 shown, the pixel space diffusion generation based fly point free monocular depth estimation device comprises a processor 21 and a memory 22, wherein the processor 21 and the memory 22 can be connected through a bus or other means.

[0097] The processor 21 can be a central processing unit (CPU). The processor 21 can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. chips, or combinations of the above various types of chips.

[0098] The memory 22, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the pixel space diffusion based generation of fly point free monocular depth estimation method in the embodiment 1 of the present application. The processor executes various functions of the processor and training processing by running the non-transitory software programs, instructions and modules stored in the memory.

[0099] The memory 22 can include a program storage area and a training storage area, wherein the program storage area can store an operating system and at least one application required by a function; and the training storage area can store training created by the processor. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 22 can optionally include a memory remotely arranged with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The one or more modules are stored in the memory 22, and when executed by the processor 21, the pixel space diffusion based generation of fly point free monocular depth estimation method in the embodiment 1 shown in the figure is executed. The specific details of the pixel space diffusion based generation of fly point free monocular depth estimation method can be understood by referring to the corresponding related descriptions and effects in the embodiments shown in Figure 1 Figure 1 、 Figure 2 and Figure 3 , and the like, which will not be described here in detail.

[0100] The embodiment also provides a computer storage medium, which stores a computer program executable by a processor to complete the pixel space diffusion based generation of fly point free monocular depth estimation method described in the embodiment 1.

[0101] The computer storage medium stores computer executable instructions, and the computer executable instructions can execute the pixel space diffusion based generation of fly point free monocular depth estimation method in any method embodiment described above. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; and the storage medium can also include a combination of the above-mentioned types of memories.

[0102] ​The specific steps of the pixel space diffusion-based non-flying point monocular depth estimation method are described in Embodiment 1, which will not be repeated herein.

[0103] The above merely provides the preferred embodiments of the application, and is not intended to limit the application. Any modification, equivalent replacement, and improvement made within the spirit and principle of the application shall fall within the protection scope of the application.

Claims

1. A monocular depth estimation method without flying points based on pixel spatial diffusion generation, characterized in that, include: High-level semantic representations are extracted from input images using a visual basic model; Gaussian noise is added to the true depth image of the input image to obtain a noise sample, and the noise sample is concatenated with the input image to obtain a joint image; The global features of the joint image are processed by a standard DiT block to obtain an intermediate feature map. The high-level semantic representation is received as a guiding signal by a semantically guided DiT block, and the fine-grained details in the intermediate feature map are modeled to obtain a depth estimation image. The parameters in the standard DiT block and the semantically guided DiT block are updated based on the depth estimation image and the real depth image to obtain the trained depth prediction model, and monocular depth estimation is performed on the input image based on the trained depth prediction model.

2. The monocular depth estimation method without flying points based on pixel spatial diffusion generation according to claim 1, characterized in that, The process of obtaining an intermediate feature map by processing the global features of the joint image using standard DiT blocks specifically includes: The joint image is input into the standard DiT block, and the first layer of the standard DiT block converts the joint image into a linear embedding operation on each local token of size p×p in the joint image. A one-dimensional sequence of tokens, where each token has a dimension of . ; The tokens in the one-dimensional sequence are processed by the converter block sequence of the standard DiT block to mine global feature associations between tokens in order to generate the intermediate feature map.

3. The monocular depth estimation method without flying points based on pixel spatial diffusion generation according to claim 1, characterized in that, The method further includes: The true depth image of the input image is obtained through a time-dependent interpolation function. With Gaussian noise Combine to generate noise samples ,in t ∈[0,1], specifically represented as: Define the velocity field of the noise-adding process And through neural networks Learning to predict and obtain the velocity field ,in, The feature representation of the input image; During the inference phase, a discretized ordinary differential equation solver is used to solve the predicted velocity field. From noise samples Gradually restore the true depth image The specific iterative formula is as follows: Among them, noise samples are processed through multiple iterations. Convert to true depth image .

4. The monocular depth estimation method without flying points based on pixel spatial diffusion generation according to claim 1, characterized in that, The expression formula for the high-level semantic representation is as follows: in, For the high-level semantic representation, As a basic model for vision, For the number of tokens, The embedding dimension of the visual basic model. It is the set of real numbers.

5. The monocular depth estimation method without flying points based on pixel spatial diffusion generation according to claim 1, characterized in that, The method further includes: The high-level semantic representation is normalized, and the normalized high-level semantic representation is as follows: in, For the high-level semantic representation, This is the high-level semantic representation after normalization.

6. The monocular depth estimation method without flying points based on pixel spatial diffusion generation according to claim 1, characterized in that, The step of receiving the high-level semantic representation as a guiding signal through the semantically guided DiT block specifically includes: Through multilayer perceptron network The normalized high-level semantic representation is integrated into the tokens of the semantically guided DiT block. As a guiding signal: in, This represents the bilinear interpolation operator. This indicates the use of semantically enhanced DiT tokens. This is the high-level semantic representation after normalization.

7. The monocular depth estimation method without flying points based on pixel spatial diffusion generation according to claim 1, characterized in that, The method further includes: The true depth image of the input image is normalized.

8. The monocular depth estimation method without flying points based on pixel spatial diffusion generation according to claim 1, characterized in that, The method further includes: Edge masks are extracted from the true depth image using edge detection operators; The true depth image is projected into a true point cloud, and the depth estimation image is projected into a predicted point cloud; The chamfer distance between the predicted point cloud and the ground truth point cloud near the edge is calculated based on the edge mask, and the chamfer distance is used as a loss term to minimize the geometric difference between the predicted point cloud and the ground truth point cloud.

9. A monocular depth estimation device without flying points based on pixel spatial diffusion generation, characterized in that, The non-flying-point monocular depth estimation device based on pixel spatial diffusion includes: a processor and a memory for storing processor-executable instructions; The processor is configured to execute the non-flying-point monocular depth estimation method based on pixel spatial diffusion generation as described in any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the non-flying-point monocular depth estimation method based on pixel spatial diffusion generation as described in any one of claims 1-8.

Images