Depth prediction distillation method based on Diffusion model

Abstract

The invention belongs to the technical field of computer vision, and relates to a depth prediction distillation method based on a Diffusion model, which comprises the following steps: 1, acquiring a real sparse point cloud and projecting to generate a sparse depth map; 2, inputting the RGB image into a deep network to extract an initial depth feature map; 3, using the sparse pixel depth map to construct a diffusion condition; 4, performing de-noising generation of the depth feature map by using a Layout Diffusion model; 5, decoding diffusion features and performing supervised training based on real sparse depth; 6, using the depth prediction network after diffusion model optimization as a teacher model, and guiding the lightweight student model to perform knowledge distillation; according to the method, the prediction precision, the structure consistency and the practical application efficiency of the model are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a deep prediction distillation method based on a Diffusion model. BACKGROUND

[0002] In the field of computer vision, depth prediction is one of the core tasks in three-dimensional reconstruction and scene understanding. The mainstream method is mainly based on monocular image (RGB image) input, and the depth neural network (such as DepthNet structure based on convolution and Transformer) is used to model the image content, so as to directly predict the depth value of each pixel in the scene. This kind of method has the advantages of end-to-end and fast inference speed, but still has the problem of inaccurate prediction in the area lacking of geometric information (such as texture flat area and occlusion area), especially in the face of complex scenes with large-scale changes and overlapping of objects at different distances, which is easy to cause depth jump and structure disorder.

[0003] In order to improve the geometric accuracy of depth prediction, some studies try to introduce point cloud data collected by real sensors, such as sparse three-dimensional point cloud obtained by LiDAR or structured light camera. These methods usually project the real point cloud to the image plane through the camera internal and external parameter matrix to form a sparse depth map, and take it as one of the inputs of the network to guide the depth prediction network to enhance spatial perception. This kind of method provides part of the real depth sampling, which helps to alleviate the problem of insufficient information in the image itself. However, in actual operation, the depth map generated by projection is severely sparse, and the pixel position after point cloud projection is irregular, which is difficult to be directly used to regress high-quality dense depth map, so it still needs to be modeled and predicted by depth neural network.

[0004] On the other hand, as a generative modeling framework, diffusion model has strong step-by-step inference and structure restoration ability in image generation, image editing and three-dimensional reconstruction tasks. Diffusion model has the ability to model complex structure prior by gradually restoring the real data distribution in Gaussian noise space. In recent years, some studies have begun to try to introduce diffusion model into the field of depth prediction, usually taking the intermediate feature map as the initial noise input, and learning to restore it to a depth prediction feature map with more complete structure and more reasonable geometry. Representative works such as LayoutDiffusion try to use scene layout information as a condition to guide image or structure generation. But most of these methods still take image or image features as input, and do not effectively introduce real point cloud information collected in the real world as a structural constraint.

[0005] Therefore, the existing depth prediction method has the following disadvantages: (1) lack of three-dimensional geometric perception ability, difficult to handle complex structure scene; (2) low utilization efficiency of sparse point cloud, no deep modeling mechanism is formed; (3) the diffusion modeling process lacks real structure condition input, and the generation effect stability is poor.

[0006] Therefore, there is an urgent need for a method that can realize accurate and structurally coherent dense depth prediction under sparse conditions to solve the above technical problems. SUMMARY

[0007] The present application aims to embed real point cloud into diffusion model as structure condition, and combine depth feature map for step-by-step generation and optimization, to improve the structure rationality and spatial consistency of depth prediction, and overcome the shortcomings of existing depth prediction method in structure expression and geometric modeling.

[0008] The present application provides the following technical scheme: a depth prediction distillation method based on Diffusion model, comprising the following steps:

[0009] Step 1, obtaining real sparse point cloud and projecting to generate sparse depth map;

[0010] Step 2, inputting RGB image into depth network to extract initial depth feature map;

[0011] Step 3, constructing diffusion condition using sparse pixel depth map;

[0012] Step 4, using LayoutDiffusion model to denoise and generate depth feature map;

[0013] Step 5, decoding diffusion feature and supervising training based on real sparse depth;

[0014] Step 6, using the depth prediction network optimized by diffusion model as teacher model to guide the knowledge distillation of light student model.

[0015] Preferably, in step 1, the three-dimensional point cloud data obtained in the real space is projected to the image plane to form a sparse depth map consistent with the image resolution, which is used as the feature modeling input of the subsequent depth network;

[0016] In step 2, multi-scale structure and semantic features in RGB image are extracted by visual backbone network to form initial feature map required for depth prediction, which is used as the basis for structure expression of subsequent diffusion modeling;

[0017] In step 3, based on the sparse depth map formed by projecting the point cloud to the image plane in step 1, the image position of the effective pixel point and its corresponding depth value are extracted, and the condition guiding information is constructed in the feature map space, which is used as the structure guide of the subsequent diffusion model;

[0018] In step 4, the initial depth feature map extracted by the image backbone network is conditionally generated by a diffusion model to gradually guide it to denoise and reconstruct a structure-complete and geometric-consistent depth feature map under the sparse structure constraint provided by the real point cloud;

[0019] In step 5, after completing the denoising and reconstruction of the depth feature map based on LayoutDiffusion, the depth feature map is decoded into a final dense depth map by a depth decoder module to perform physical supervision and subsequent distillation;

[0020] In step 6, by constructing a knowledge distillation mechanism, using the complete depth prediction network containing the LayoutDiffusion model as the teacher model, the lightweight network without sparse point cloud input is guided to learn depth structure perception ability, so as to achieve the purpose of getting rid of high computational overhead model in the inference stage and realizing efficient deployment.

[0021] More preferably, in step 1, the point cloud coordinates are converted into lidar coordinates and then into image coordinates; step 1 sequentially includes: point cloud coordinate conversion, calculation of row and column numbers of points on the image;

[0022] The point cloud coordinates are converted as follows: Wherein, represents the camera coordinate system, T lidar→cam represents the extrinsic transformation matrix, represents the homogeneous coordinates in the lidar coordinate system;

[0023] In the calculation of the row and column numbers of points on the image, the normalized two-dimensional pixel coordinates are obtained as follows: Wherein, u i , v i respectively represent the pixel coordinates after rounding, Z i represents the depth value in the camera coordinate system, f x represents the focal length in the x direction, f y represents the focal length in the y direction, X i represents the x coordinate in the camera coordinate system, Y i represents the Y coordinate in the camera coordinate system, c x , c y represent the principal point coordinates.

[0024] More preferably, step 2 sequentially includes: extracting visual features by an image backbone network, and extracting an initial depth feature map;

[0025] The feature extraction process of the image backbone network extracting visual features is as follows: F img =Backbone(I; θ b ), wherein F img represents the feature map, I represents the input image, and θb representing Image Backbone parameters;

[0026] extracting a feature map F0 of the initial depth feature map as F0 = DepthFeatExtractor(F img ), wherein the DepthFeatExtractor is a convolutional neural network.

[0027] More preferably, the step 3 sequentially comprises: obtaining a sparse depth observation point set, and constructing a sparse depth guidance map.

[0028] In the sparse depth observation point set, the sparse observation set is composed of:

[0029]

[0030] In the construction of the sparse depth guidance map, a sparse depth template C pc (u' i , v' i ) = Z i is initialized on a two-dimensional plane of the feature map size, wherein u' i , v' i represent the feature map coordinates corresponding to the points.

[0031] More preferably, the step 4 sequentially comprises: input composition, conditional fusion mode, reverse generation process, and loss function design.

[0032] The input in the input composition includes: noise feature map, structure condition map, and time encoding.

[0033] The conditional fusion mode includes:

[0034] Channel concatenation: concatenating with the current feature at each U-Net layer;

[0035] FiLM modulation: using C cond to control the normalization parameters of the intermediate layer;

[0036] Attention guidance: adding a Cross-Attention module in the Transformer layer, taking C cond as Key / Value, and guiding the feature to focus on the known structure.

[0037] More preferably, the step 5 sequentially comprises: input feature map, decoder structure design, sparse depth ground truth supervision construction, sparse depth loss design, and training target function.

[0038] More preferably, the step 6 sequentially comprises: teacher model composition and freezing, student model construction, distillation strategy design, and total loss function.

[0039] In the teacher model configuration and freezing, the teacher model module configuration comprises: an RGB image input backbone network, a sparse point cloud condition input module, a LayoutDiffusion condition generation model and a depth map decoder module.

[0040] In the student model construction, the student model is a lightweight network, and the input of the student model comprises an RGB image, and the student model outputs a predicted depth map.

[0041] More preferably, the distillation strategy design comprises: intermediate feature distillation, output depth map distillation and sparse point true value auxiliary supervision.

[0042] In the intermediate feature distillation, the feature distillation loss is:

[0043]

[0044] Wherein, The student model encoder output feature is represented as F s, The optimized feature map extracted from the teacher model is represented as F t ;

[0045] In the output depth map distillation, the distillation loss is:

[0046]

[0047] Wherein, The student output is represented as D s, The teacher output dense depth map is represented as D t ;

[0048] In the sparse point true value auxiliary supervision, the sparse supervision loss is calculated on the projection point position as:

[0049]

[0050] Wherein, The sparse depth point set is represented as P.

[0051] The present application has the following advantages:

[0052] 1. The present application proposes a structure-aware depth prediction method and its distillation mechanism based on a diffusion model, aiming at the problems of poor structure recovery ability, insufficient robustness and low inference efficiency of existing depth estimation technology under sparse supervision, and introduces innovative designs in the overall system architecture, training paradigm and deployment path, thereby significantly improving the prediction accuracy, structural consistency and actual application efficiency of the model.

[0053] 2、The application fuses point cloud geometry prior, guides structure recovery, introduces a LayoutDiffusion model for structure denoising reconstruction, decodes and supervises training, realizes physical scale alignment, constructs a teacher-student network distillation mechanism, realizes lightweight and efficient deployment, and has good expansibility. BRIEF DESCRIPTION OF DRAWINGS

[0054] Fig. 1 A depth prediction model structure diagram of the deep prediction distillation method based on the Diffusion model is provided.

[0055] Fig. 2 A student-teacher network structure diagram of the application is provided.

[0056] Fig. 3 A method step diagram of the application is provided. DETAILED DESCRIPTION

[0057] The related technologies in the application will be described clearly and completely in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0058] As shown in the embodiment of the application, the implementation steps are as follows: Figs. 1-3

[0059] 1. Obtain real sparse point cloud and project to generate sparse depth map;

[0060] 2. Input the RGB image into the depth network to extract the initial depth feature map;

[0061] 3. Use the sparse pixel depth map to construct the diffusion condition;

[0062] 4. Use the LayoutDiffusion model to denoise and generate the depth feature map;

[0063] 5. Decode the diffusion feature and supervise the training based on the real sparse depth;

[0064] 6. Use the depth prediction network optimized by the diffusion model as the teacher model to guide the knowledge distillation of the lightweight student model.

[0065] The implementation details of each step are as follows:

[0066] Step 1: This step aims to accurately project the three-dimensional point cloud data obtained in the real space to the image plane to form a sparse depth map consistent with the image resolution, providing input for the subsequent feature modeling of the depth network. ​

[0067] 1.1 Point cloud coordinate conversion (lidar coordinate system to image coordinate system)

[0068] Let be the homogeneous coordinates in the lidar coordinate system, and use the extrinsic transformation matrix:

[0069]

[0070] The point can be transformed into the camera coordinate system:

[0071]

[0072] where R is the rotation matrix and t is the translation vector.

[0073] 1.2 Calculate the row and column numbers of the point on the image

[0074] Use the camera intrinsic matrix to project the three-dimensional point onto the image plane to obtain the homogeneous pixel coordinates: Normalize to obtain the two-dimensional pixel coordinates: where Z i represents the depth value of the point in the camera coordinate system. Normalize to obtain the two-dimensional pixel coordinates: where (u i , v i ) is rounded to an integer pixel coordinate, and Z i represents the depth value of the point in the camera coordinate system.

[0075] Step 2: This step extracts multi-scale structures and semantic features in the RGB image through a visual backbone network (image backbone), forming the initial feature map required for depth prediction, and providing a structural expression basis for subsequent diffusion modeling.

[0076] 2.1 Image backbone (Image Backbone) extracts visual features

[0077] Input an image First, input the image backbone network to extract multi-level spatial semantic features. This patent uses Swin Transformer as the Image Backbone.

[0078] The feature extraction process can be represented as: F img = Backbone(I; θ b ), where θ b is the Image Backbone parameter, H' = H / s, W' = W / s represents the scale of the image after spatial downsampling (s = 8), and C is the channel dimension (C = 256).

[0079] 2.2 Extract initial depth feature map

[0080] After the backbone network and the optional structure enhancement module, the output depth perception feature map F0 is obtained: wherein DepthFeatExtractor is a convolutional neural network. The feature map is the initial state input for subsequent diffusion modeling, and has the following characteristics: in the spatial dimension, it reflects the image geometric structure and occluded area; in the channel dimension, it encodes disparity signals, depth textures, scene layouts and other information, which can be directly used for initial prediction.

[0081] Step 3: Based on the sparse depth map formed by projecting the laser radar point cloud to the image plane in step 1, the image positions (row and column numbers) of the effective pixel points and their corresponding depth values are extracted, and the condition guidance information is constructed in the feature map space for the structure guidance of the subsequent diffusion model.

[0082] 3.1 Obtain sparse depth observation point set

[0083] Through the projection conversion operation in step 1, a sparse depth map is obtained Most of the pixels are empty, and only a few pixels have valid depth values Z i . The two-dimensional indexes (u i , v i ) and the corresponding depth values Z i of all effective pixels are extracted to form a sparse observation set: wherein N << H x W.

[0084] 3.2 Construct sparse depth guidance map

[0085] In order to align with the image feature map , the sparse depth observation points need to be mapped to the feature map space. Assuming that the image downsampling factor is s = H / H', the corresponding feature map coordinates of each point are: Then, a sparse depth template is initialized on the two-dimensional plane of the feature map size: C pc (u i ', v i ') = Z i , and the remaining pixels are empty or set to 0, and a mask M ∈ {0, 1} H’×W’ indicating which pixel positions have real depth can be constructed synchronously.

[0086] Step 4: This step aims to conditionally generate the initial depth feature map extracted by the image backbone network through the diffusion model, and gradually guide it to denoise and reconstruct a structure complete and geometric consistent depth feature map under the sparse structure constraint provided by the real point cloud.

[0087] 4.1 Input Composition

[0088] The input of LayoutDiffusion consists of three parts:

[0089] - Noise feature map (initial random):

[0090]

[0091] - Structure condition map (constructed by sparse depth projection):

[0092]

[0093] where is the set of projected points.

[0094] - Temporal encoding:

[0095] Each time step t is encoded into a vector e t using a sinusoidal embedding or learnable embedding.

[0096] 4.2 Condition Fusion Methods

[0097] LayoutDiffusion incorporates the sparse depth condition C cond into the denoising process in multiple ways: ① Channel concatenation: concatenate with the current feature at each U-Net layer; ② FiLM modulation: control the normalization parameters of the intermediate layer with C cond ; ③ Attention guidance: add a Cross-Attention module in the Transformer layer, taking C cond as Key / Value to guide the feature to focus on the known structure.

[0098] 4.3 Backward Generation (Denoising) Process

[0099] In the generation process, the model starts from pure noise feature F T , and gradually performs time step denoising operations: Finally, the reconstructed depth feature map is obtained: This feature map retains the geometric structure information provided by the sparse point cloud, while removing the noise and uncertainty in the original features of the RGB network.

[0100] 4.4 Loss Function Design

[0101] During training, LayoutDiffusion usually uses the denoising reconstruction loss:

[0102]

[0103] The loss function calculates the difference between the original feature map and the feature map after adding noise and denoising.

[0104] Step 5: After completing the LayoutDiffusion-based deep feature map denoising reconstruction, it needs to be decoded into the final dense depth map by a depth decoder module to perform physical supervision and subsequent distillation.

[0105] 5.1 Input feature map

[0106] The output structure-guided depth feature map after denoising is: The feature map structure is rich, but it does not directly correspond to the pixel-level depth value.

[0107] 5.2 Decoder structure

[0108] Use a structure-preserving depth convolution module Map to a dense depth map at image resolution: This module ensures that the dense depth map retains the geometric structure learned in the diffusion stage.

[0109] 5.3 Sparse depth ground truth supervision construction

[0110] The training supervision signal comes from the point cloud obtained by the 128-line laser radar in step 1, which is projected to the image plane through the camera extrinsic and intrinsic parameters, forming a sparse depth point set: The sparse depth map is represented as: D gt (u i , v i ) = Z i This sparse supervision signal is an important basis for the geometric alignment and physical scale preservation of the entire network structure.

[0111] 5.4 Sparse depth loss design

[0112] In order to only train supervision at effective observation positions, the supervision loss is:

[0113]

[0114] 5.5 Training objective function

[0115] The overall loss function is combined as follows:

[0116]

[0117] Among them Used to optimize the LayoutDiffusion model in the early training;

[0118] Used for sparse supervision. λ1 and λ2 are weight coefficients.​

[0119] Step 6: In this step, a knowledge distillation mechanism is constructed to guide a lightweight network (student model, Student) that does not require sparse point cloud input to learn depth structure perception ability using a complete deep prediction network containing the LayoutDiffusion model as the teacher model (Teacher), so as to achieve the purpose of getting rid of high computational overhead models in the inference stage and realizing efficient deployment.

[0120] 6.1 Teacher model composition and freezing

[0121] The teacher model is composed of the following modules: an RGB image input backbone network (image encoder); a sparse point cloud condition input module (projection construction in step 1); a LayoutDiffusion condition generation model; and a depth map decoder module.

[0122] After training, the model freezes all parameters, and the input is an RGB image and the corresponding sparse point cloud, and the output is a high-quality dense depth map Intermediate features

[0123] 6.2 Student model construction

[0124] The student model is designed as a lightweight network S φ , which takes only an RGB image I as input without point cloud assistance, and outputs a predicted depth map

[0125] 6.3 Distillation strategy design

[0126] In order to realize the knowledge transfer between the teacher and the student, the following distillation mechanism is designed:

[0127] 6.3.1 Intermediate feature distillation

[0128] Extract the optimized feature map from the teacher model Align with the student model encoder output feature Calculate the feature distillation loss:

[0129] 6.3.2 Output depth map distillation

[0130] The dense depth map output by the teacher is The student output is Define the L1 distillation loss: It can only be supervised in the structure area or the point cloud projection area.

[0131] 6.3.3 Sparse point truth auxiliary supervision

[0132] Use the sparse depth map Dgt Auxiliary training student model, calculating sparse supervision loss on projection points:

[0133]

[0134] 6.4 Total loss function

[0135] The overall training goal of the student model is:

[0136] λ i is a weighting coefficient.

[0137] The embodiment is based on sparse point cloud to construct diffusion condition and guide depth feature generation.

[0138] The embodiment uses sparse point cloud from 128-line laser radar, projects to image plane through camera extrinsic and intrinsic parameters, generates sparse depth map, and represents it as "row number + depth value" pair to construct condition input of LayoutDiffusion model.

[0139] The embodiment uses real sparse point cloud to construct diffusion model condition, guides depth feature to gradually converge to structure consistent distribution, and improves structure restoration ability under weak supervision or no GT, which is the core source of structure perception performance of the whole system.

[0140] The embodiment fuses LayoutDiffusion model and image backbone network to realize depth structure denoising modeling.

[0141] The embodiment combines diffusion model with sparse point cloud as condition on the basis of image backbone extracting initial image features, performs time-series multi-step reconstruction, and finally outputs structure-optimized feature map.

[0142] The embodiment avoids the blur problem of relying only on RGB image to restore structure in traditional network, realizes dynamic modeling and denoising generation of geometric structure, and improves spatial coherence of depth map.

[0143] The embodiment uses diffusion depth network as teacher model to guide student network without point cloud input to perform knowledge distillation (deployment optimization).

[0144] The embodiment freezes structure perception type teacher network, constructs lightweight student network, relies only on RGB input, and introduces feature distillation and output distillation loss on multi-level structure.

[0145] The embodiment realizes an inference path that considers both depth estimation accuracy and speed without relying on point cloud, meets the needs of industrial deployment, and is a key embodiment of engineering value.

[0146] To sum up, the structure perception depth prediction method based on the diffusion model and the distillation mechanism thereof improve the overall system architecture, training paradigm and deployment path, thereby significantly improving the prediction accuracy, structural consistency and actual application efficiency of the model. It should be emphasized that the above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification, equivalent change and modification of the above embodiment according to the technical essence of the present application still belongs to the scope of the technical solutions of the present application.

Claims

1. A depth-predictive distillation method based on a diffusion model, characterized in that, Includes the following steps: Step 1: Obtain the real sparse point cloud and project it to generate a sparse depth map; Step 2: Input the RGB image into the deep network to extract the initial depth feature map; Step 3: Construct diffusion conditions using a sparse pixel depth map; Step 4: Use the LayoutDiffusion model to generate denoised depth feature maps; Step 5: Decode the diffusion features and perform supervised training based on the real sparse depth; Step 6: Use the deep prediction network optimized by the diffusion model as the teacher model to guide the lightweight student model to perform knowledge distillation.

2. The depth prediction distillation method based on the Diffusion model according to claim 1, characterized in that, In step 1, the 3D point cloud data obtained in the real space is projected onto the image plane to form a sparse depth map with the same resolution as the image, which is used as the feature modeling input for the subsequent deep network. In step 2, multi-scale structural and semantic features in RGB images are extracted through a visual backbone network to form an initial feature map required for depth prediction, which serves as the structural representation basis for subsequent diffusion modeling. In step 3, based on the sparse depth map formed by projecting the point cloud onto the image plane in step 1, the image positions of effective pixels and their corresponding depth values ​​are extracted, and conditional guidance information is constructed in the feature map space as structural guidance for the subsequent diffusion model. In step 4, the initial depth feature map extracted by the image backbone network is conditionally generated through the diffusion model, and gradually guided to denoise and reconstruct a structurally complete and geometrically consistent depth feature map under the sparse structure constraints provided by the real point cloud. In step 5, after completing the denoising and reconstruction of the depth feature map based on LayoutDiffusion, the depth feature map is decoded into the final dense depth map by the depth decoder module. In step 6, a knowledge distillation mechanism is constructed, and a complete deep prediction network containing a LayoutDiffusion model is used as a teacher model to guide a lightweight network that does not require sparse point cloud input to learn deep structure perception capabilities.

3. The depth prediction distillation method based on the Diffusion model according to claim 2, characterized in that, In step 1, the point cloud coordinates are converted to the LiDAR coordinate system and then to the image coordinate system; step 1 includes in sequence: point cloud coordinate conversion, and calculation of the row and column numbers of the points on the image. The point cloud coordinates are converted to: in, T represents the camera coordinate system. lidar→cam Represents the extrinsic transformation matrix. Represents homogeneous coordinates in the lidar coordinate system; The normalized two-dimensional pixel coordinates of the calculated point in the image's row and column indices are: Among them, u i v i Z represents the pixel coordinates after being rounded to the nearest integer. i f represents the depth value in the camera coordinate system. x f represents the focal length in the x-direction. y The focal length in the y-direction is represented by X. i Represents the x-coordinate and y-coordinate in the camera coordinate system. i c represents the Y-coordinate in the camera coordinate system. x c y Indicates the coordinates of the principal point.

4. The depth prediction distillation method based on the Diffusion model according to claim 3, characterized in that, Step 2 includes, in sequence: extracting visual features from the image backbone network and extracting an initial depth feature map; The feature extraction process for visual features extracted by the image backbone network is as follows: F img =Backbone(I; θ) b ), where F img Let I represent the feature map, and θ represent the input image. b This represents the Image Backbone parameter; The feature map F0 for extracting the initial depth feature map is: F0 = DepthFeatExtractor(F img ), where DepthFeatExtractor is a convolutional neural network.

5. The depth prediction distillation method based on the Diffusion model according to claim 4, characterized in that, Step 3 includes, in sequence: obtaining a sparse depth observation point set and constructing a sparse depth guidance map; The sparse observation set constitutes the sparse observation set as follows: In the construction of the sparse depth guide graph, a sparse depth template is initialized on a two-dimensional plane the size of the feature map: C pc (u i ′,v i ′)=Z i , where u i ′,v i ′ represents the coordinates of the feature map corresponding to the point.

6. The depth prediction distillation method based on the Diffusion model according to claim 5, characterized in that, Step 4 includes, in sequence: input composition, conditional fusion method, reverse generation process, and loss function design. The inputs in the input structure include: noise feature map, structural condition map, and time code; The conditional fusion methods include: Channel concatenation: Concatenate with the current feature at each U-Net layer; FiLM modulation: using C cond Control the normalization parameters of the intermediate layer; Attention guidance: Add a Cross-Attention module to the Transformer layer to... cond As a key / value pair, the guiding features focus on known structures.

7. The depth prediction distillation method based on the Diffusion model according to claim 6, characterized in that, Step 5 includes, in sequence: input feature map, design decoder structure, sparse depthwise ground truth supervision construction, sparse depthwise loss design, and training objective function.

8. The depth prediction distillation method based on the Diffusion model according to claim 7, characterized in that, Step 6 includes, in sequence: teacher model construction and freezing, student model construction, distillation strategy design, and total loss function; In the teacher model composition and freezing process, the teacher model module consists of: RGB image input backbone network, sparse point cloud conditional input module, LayoutDiffusion conditional generation model, and depth map decoder module. In the construction of the student model, the student model is a lightweight network, the input of the student model includes RGB images, and the output of the student model is a predicted depth map.

9. A depth prediction distillation method based on a diffusion model according to claim 8, characterized in that, The distillation strategy design includes: intermediate feature distillation, output depth map distillation, and sparse point ground value-assisted supervision. The characteristic distillation loss in the intermediate characteristic distillation is: in, This represents the output features of the student model encoder. This represents the optimized feature map extracted from the teacher model; The distillation loss in the output depth map distillation is: in, This represents the student's output. This represents the dense depth map output by the teacher. In the sparse point truth-aided supervision, the sparse supervision loss is calculated at the projected points as follows: in, This represents a sparse depth point set.

Images