A spatio-temporal consistent video depth completion method under a zero-shot unified diffusion framework

Abstract

The application discloses a kind of spatiotemporal consistent video depth completion method under zero sample unified diffusion framework, comprising: constructing the depth completion model including variational autoencoder, semantic coding network and spatiotemporal diffusion generation network;Prepare training data, generate frame-level semantic feature vector for video frame and condition latent variable fused original depth and relative depth;With the input of condition latent variable sequence, as a condition, spatiotemporal diffusion generation network is trained in stages;In inference stage, the video sequence to be completed is processed by sliding window fusion mechanism, and the trained network obtains denoised depth latent variable, and finally the completed depth sequence is output by the decoding of variational autoencoder and scale recovery;The advantage is that when the long sequence video of sparse, noise or structural damage exists is completed, depth sequence with metric consistency, structural integrity and time stability can be generated.

Description

Technical Field

[0001] This invention relates to video depth completion methods, and more particularly to a spatiotemporally consistent video depth completion method under a zero-shot unified diffusion framework. Background Technology

[0002] With the development of applications such as autonomous driving, robot navigation, AR / VR, video generation, and video understanding, the engineering demand for video depth completion is becoming increasingly prominent. In real video streams, depth observations often exhibit sparse, irregular, and noisy characteristics (e.g., low-beam LiDAR, SfM sparse points, downsampled RGBD), accompanied by complex degradations such as occlusion variations, sensor frame drops, exposure variations, motion drift, and structural holes. Against this backdrop, the goal of video depth completion is to recover geometrically complete, metrically consistent, and temporally stable dense depth sequences from the input video and its sparse or damaged depth observations, while maintaining robustness and efficiency under conditions of cross-domain appearance variations and long time spans.

[0003] There are three main types of existing video depth completion methods: single-frame-based depth completion methods, video-based depth estimation completion methods, and single-frame-based relative depth estimation depth completion methods.

[0004] The single-frame-based depth completion method fuses the sparse or damaged depth measurement of each frame with the corresponding color image, achieving spatial completion using local texture, edge, and structural cues. It can also perform a degree of scale calibration at the single-frame scale using effective metric anchors. This method exhibits good detail restoration and boundary preservation capabilities at the single-frame level and is suitable for diverse sparse metric input formats and resolution settings. However, due to independent optimization on a frame-by-frame basis, it lacks cross-frame temporal constraints and self-correction mechanisms. When the video sequence contains rapid motion, occlusion changes, large holes, or extremely low sampling density, it is prone to flickering, unstable boundaries, and global scale inconsistencies in cross-frame results. Especially under long video sequences (input sequences of 50 frames or more), this independent frame-by-frame processing can also lead to structural discontinuities and detail jumps between consecutive frames, making it difficult to meet the application requirements for long-term stability.

[0005] Video-based depth estimation completion methods explicitly introduce mechanisms such as sliding windows, cross-frame feature aggregation, caching, or temporal attention in the temporal dimension to enhance cross-frame geometric consistency, mitigate temporal jitter, and improve inference coherence over long time spans. By utilizing temporal information, this method can achieve relatively stable cross-frame performance under motion scenes, occlusion changes, and appearance variations, and can propagate and complete depth information within a certain range. However, because its model design and training data focus on relative geometric cues (relying more on appearance and motion consistency than explicit metric calibration), it is more prone to scale instability under different scenes and shooting conditions. Furthermore, it often employs a unidirectional temporal propagation strategy; when there are estimation biases in early frames, the errors may accumulate unidirectionally along time, gradually amplifying boundary biases and scale drift in long video sequences, affecting overall metric consistency.

[0006] Depth completion methods based on single-frame relative depth estimation directly infer the relative geometric relationships of a scene from color images, providing prior information such as global layout, object boundaries, and hierarchical structure. They also exhibit good cross-domain generalization ability under conditions of no or weak metric constraints. This method can serve as an important structural prior for subsequent completion and fusion, supporting the collaboration of multi-source cues. However, because its model targets and training samples primarily learn relative ordering and boundary continuity (weakening explicit metric calibration and cross-frame scale constraints), it is prone to scale fluctuations and inconsistencies in object boundary structure under complex occlusion, exposure variations, and cross-domain appearance differences. When combined with sparse metric observations without a reasonable prior fusion and scale alignment mechanism, it may also introduce problems such as local oversmoothing or global scale bias.

[0007] In recent years, researchers have proposed a representative depth completion method called PriorDA (Prior DepthAnything, a general depth estimation method based on priors) in the paper "Zehan Wang, Siyu Chen, Lihe Yang, JialeiWang, Ziang Zhang, Hengshuang Zhao, and Zhou Zhao. Depth anything with anyprior. arXiv preprint arXiv:2505.10565, 2025." PriorDA aims to generate accurate and dense metric depth maps by combining incomplete but accurate metric observation information (metric priors) with relatively complete geometric structures (i.e., depth prediction models). Its core technology lies in employing a coarse-to-fine processing pipeline: First, using pixel-level metric alignment and distance-aware weighting, sparse metric priors are explicitly pre-filled with depth predictions; then, implicit fusion and noise restoration of the pre-filled depth are performed under the guidance of an RGB image through a conditional monocular depth estimation model.

[0008] However, the PriorDA method still has the following significant limitations in measuring consistency, structural integrity, and temporal stability when dealing with long video depth completion tasks:

[0009] 1. Global Metric Inconsistency: The metric calibration of the PriorDA method relies heavily on local pixel-level linear fitting. While this design improves local accuracy, the lack of global and long-term scale constraints makes it highly susceptible to sensor frame drops or local observation noise when processing long video sequences. As the video progresses, this local estimation bias accumulates non-linearly over time, ultimately leading to global scale drift and drastic fluctuations in the metric standard.

[0010] II. Limited Structural Integrity Under Complex Degradation: Although the framework utilizes the structural prior of a pre-trained conditional monocular depth estimation model, its discriminative regression mechanism often struggles to reconstruct accurate geometric contours in missing regions when faced with extremely low sampling density, large-scale structural damage, or degradation regions exceeding the sensor's effective range. In the dynamic changes of video streams, this vulnerability to geometric structure recovery manifests as blurred object boundaries, loss of thin structures, and structural artifacts in depth-deficient regions, making it difficult to maintain the geometric integrity of the depth sequence.

[0011] Third, there is a severe lack of temporal stability: Because the PriorDA method adopts a single-frame optimization paradigm, its depth completion process is discrete and independent in the temporal dimension. Due to the lack of cross-frame temporal attention mechanisms, feature caching, or explicit motion compensation constraints, it cannot collaboratively optimize the geometric relationships between adjacent frames when processing continuous video streams. This leads to severe inter-frame flickering and abrupt changes in edge structures in the completed sequence when faced with rapid object movement, changes in occlusion relationships, or drastic changes in lighting. Summary of the Invention

[0012] The technical problem to be solved by the present invention is to provide a spatiotemporally consistent video depth completion method under a zero-sample unified diffusion framework that can generate depth sequences with metric consistency, structural integrity and temporal stability when performing depth completion on long video sequences that are sparse, noisy or have structural defects.

[0013] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a spatiotemporally consistent video depth completion method under a zero-sample unified diffusion framework, comprising the following steps:

[0014] Step S1: Construct a deep completion model, which includes a variational autoencoder, a semantic coding network, and a spatiotemporal diffusion generation network. The variational autoencoder and the semantic coding network are respectively connected to the spatiotemporal diffusion generation network.

[0015] Step S2: Prepare training data: Obtain multiple video sequences and their depth data and validity masks for each frame. Each video sequence consists of no less than 50 color images in chronological order. For each frame of each video sequence, generate the corresponding frame-level semantic feature vector, and calculate the conditional latent variables based on its depth data, validity mask, and relative depth generated by the pre-trained monocular depth model.

[0016] Step S3: Train the deep completion model: Use the conditional latent variables or the sequence of conditional latent variables composed of consecutive frame conditional latent variables as the input of the spatiotemporal diffusion generation network, and use the frame-level semantic feature vector as the condition to train the spatiotemporal diffusion generation network in stages, so that it learns to predict the real deep latent variables from noisy deep latent variables.

[0017] Step S4: Perform depth completion on the video sequence to be completed, which consists of no less than 50 color images arranged in chronological order: Obtain the depth data and validity mask of each frame of the video sequence to be completed; obtain the conditional latent variables and frame-level semantic feature vectors of each frame in the same way as in Step S2; input the conditional latent variable sequence into the trained spatiotemporal diffusion generation network through a sliding window fusion mechanism to obtain the denoised depth latent variables of the video sequence to be completed; finally, decode the denoised depth latent variables of the video sequence to be completed and restore the scale through the variational autoencoder to obtain the completed video depth sequence.

[0018] Compared with existing technologies, the advantages of this invention are as follows: It constructs a depth completion model comprising a variational autoencoder, a semantic coding network, and a spatiotemporal diffusion generation network to perform depth completion on long video sequences that are sparse, noisy, or structurally damaged. In the data preparation stage, it obtains the depth data and validity mask for each frame of the video sequence and combines this with the relative depth generated by a pre-trained monocular depth model to calculate the conditional latent variable. Simultaneously, it generates a corresponding frame-level semantic feature vector for each frame, thus constructing the key conditional input for the depth completion model generation process: the conditional latent variable integrates the sparse / noisy depth with absolute scale information and the relative depth providing a continuous geometric layout, providing a basis for the generation process. The process provides dual geometric anchor points; while frame-level semantic feature vectors, as independent conditions, provide global scene semantic information. This design, which decouples geometric priors and semantic conditions and uses them together as inputs to the spatiotemporal diffusion generation network, enables the depth completion model to perform depth reconstruction closely around reliable geometric cues under the guidance of semantic context. This effectively improves the structural integrity of restoring object contours and thin structures in complex scenes and avoids the appearance interference that may be introduced by directly using the original RGB image. During the model training phase, a staged training strategy is adopted, using conditional latent variables or a sequence of conditional latent variables composed of consecutive frame conditional latent variables as inputs to the spatiotemporal diffusion generation network, and frame-level semantic feature vectors as inputs. As a condition, the spatiotemporal diffusion generative network is trained to predict true depth latent variables from noisy depth latent variables. This training has a clear progressive nature: first, single-frame training is performed to enable the depth completion model to master spatial depth completion capabilities based on single-frame fusion priors; then, sequence training is introduced to enable the depth completion model to learn to utilize temporal information. This training method guides the depth completion model to gradually integrate spatial awareness and temporal coherence learning, laying the foundation for generating temporally stable sequences. During training, the variational autoencoder and semantic coding networks are frozen, and only the spatiotemporal diffusion generative network is optimized to ensure that the training focuses on learning the mapping relationship from multi-source conditions to high-quality depth latent variables. In the inference application stage, the video to be completed... The sequence is processed in the same way as the training data to obtain the conditional latent variables and frame-level semantic feature vectors for each frame. The conditional latent variable sequence is then input into the trained spatiotemporal diffusion generation network through a sliding window fusion mechanism. The sliding window fusion mechanism is the core design for processing long sequences and ensuring the temporal stability of the results. It allows the depth completion model to generate local sequences as units and then integrate them into a complete sequence through a fusion strategy. This effectively avoids the cumulative propagation of errors in long sequences and suppresses inter-frame flicker and scale drift. Finally, the generated denoised depth latent variables are decoded by a variational autoencoder and scaled back to obtain the completed video depth sequence, completing the closed loop from conditional input to final depth measurement output.Therefore, this invention constructs a decoupled and information-rich conditional system by using conditional latent variables and frame-level semantic feature vectors. Through phased training of the spatiotemporal diffusion generation network, it achieves synergistic spatial restoration and temporal consistency capabilities. Furthermore, it realizes stable generation of long sequences during inference through a sliding window fusion mechanism. Specifically, this application designs a multi-source conditional coupling architecture, aligning and fusing depth observation data from different sources (such as LiDAR, SfM sparse points, and random sampling), with varying sparsity and noise levels, using relative depths generated by a pre-trained monocular depth model. This results in conditional latent variables in a unified format. Simultaneously, frame-level semantic feature vectors extracted by a semantic coding network are used as independent conditions, decoupling geometric structure restoration from texture appearance. This design allows the trained depth completion model to be directly applied to video sequences containing unseen depth sampling patterns or structural defects without requiring fine-tuning for specific sensor types (such as 64-line or 8-line radar), specific sampling modes, or specific scenes (indoor or outdoor). It maintains robust depth completion performance under cross-domain appearance variations and diverse sparse input conditions, achieving true "zero-shot" generalization. These interconnected aspects enable the invention to generate depth sequences with metric consistency, structural integrity, and temporal stability when performing depth completion on long video sequences that are sparse, noisy, or structurally damaged, under zero-shot conditions.

[0019] Furthermore, in step S2, the specific process of calculating the conditional latent variables for a single frame image is as follows:

[0020] Step S2.1: Obtain the depth data and validity mask of the frame; denote the pixel size of a single frame image as M×N×3, where M represents the number of pixel rows, N represents the number of pixel columns, and 3 represents the number of channels; the depth data and validity mask of a single frame image are data matrices of size M×N, and in the validity mask, 1 represents validity and 0 represents invalidity; the validity mask is used to indicate whether each depth in the depth data is valid; if the data at a certain position of the validity mask of a certain depth data is 0, then the depth at the same position in the depth data is an invalid depth, and if it is 1, then the depth at the same position in the depth data is a valid depth;

[0021] Step S2.2: Using a pre-trained monocular depth model, generate the relative depth of the frame, and determine the valid and invalid depths in the relative depth based on the validity mask of the depth data of the frame.

[0022] Step S2.3: Use a pre-trained semantic coding network to extract the frame-level semantic feature vector of the frame;

[0023] Step S2.4: In the depth data of the frame, each invalid depth is subjected to inverse spatial distance weighted interpolation according to its spatial distance from the valid depth, where the weight is inversely proportional to the spatial distance, so as to obtain the coarse depth of the frame with spatial connectivity, and determine the valid depth and invalid depth in the coarse depth according to the validity mask of the depth data of the frame.

[0024] Step S2.5: Perform least-squares linear fitting on the effective depth and the effective depth in the relative depth of the depth data of the frame to obtain the fitted relative depth of the frame.

[0025] Step S2.6: Use the effective depth in the fitted relative depth of the frame to perform inverse spatial distance weighting correction on each invalid depth in its coarse depth to obtain a metric depth approximation that combines absolute scale anchor points and local structural continuity.

[0026] Step S2.7: Perform quantization cropping and linear normalization on the approximate depth and relative depth of the frame to obtain its normalized approximate depth and normalized relative depth.

[0027] Step S2.8: Use a variational autoencoder to encode the normalized metric depth approximation and normalized relative depth of the frame to obtain the corresponding coarse completion depth latent variable and relative depth latent variable;

[0028] Step S2.9: The depth data of the frame is first encoded by the variational autoencoder to obtain the target latent variable, and then Gaussian noise is injected into the target latent variable to generate a noisy target latent variable;

[0029] Step S2.10: The coarse completion depth latent variable, relative depth latent variable, and noisy target latent variable of the frame are spliced ​​along the channel dimension to obtain its conditional latent variable.

[0030] Furthermore, in step S3, the specific process of performing phased training on the spatiotemporal diffusion generation network is as follows:

[0031] Step S3.1: Take the vector composed of the conditional latent variable, target latent variable and frame-level semantic feature vector of each frame of each video sequence as a training sample, and use all training samples to form the first stage training sample set; use the first stage training sample set to train the spatial correlation module in the spatiotemporal diffusion generation network.

[0032] Step S3.2: Generate a random number p between 2 and 15. Divide each video sequence into a unit video sequence, starting from the first frame, with each p frames until the sequence is completely divided or there are fewer than p frames. Concatenate the conditional latent variables of each frame in each unit video sequence into a conditional latent variable sequence and the target latent variables into a target latent variable sequence. Use the conditional latent variable sequence, target latent variable sequence, and frame-level semantic feature vector of each frame in each unit video sequence to form a sequence training sample. Use all sequence training samples to form the second-stage training sample set. Use the second-stage training sample set to train the temporal correlation module in the spatiotemporal diffusion generation network.

[0033] Step S3.3: Generate a random number q between 2 and 40. Divide each video sequence into a unit video sequence, starting from the first frame, with each q frames until the sequence is completely divided or there are fewer than q frames. Concatenate the conditional latent variables of each frame in each unit video sequence into a conditional latent variable sequence and the target latent variables into a target latent variable sequence. Use the conditional latent variable sequence, target latent variable sequence, and frame-level semantic feature vector of each frame in each unit video sequence to form a sequence training sample. Use all sequence training samples to form the third-stage training sample set. Use the third-stage training sample set to train the temporal correlation module in the spatiotemporal diffusion generation network trained in step S3.2.

[0034] Step S3.4: Divide each video sequence into a unit video sequence starting from the first frame, with every 25 frames divided until the sequence is completely divided or there are fewer than 25 frames. Concatenate the conditional latent variables of each frame in each unit video sequence into a conditional latent variable sequence and the target latent variables into a target latent variable sequence. Use the conditional latent variable sequence, target latent variable sequence, and frame-level semantic feature vector of each frame in each unit video sequence to form a sequence training sample. Use all sequence training samples to form the fourth-stage training sample set. Use the fourth-stage training sample set to train the temporal correlation module in the spatiotemporal diffusion generation network trained in step S3.3. Finally, the trained deep completion model is obtained.

[0035] Furthermore, in step S4, the specific process of inputting the conditional latent variable sequence into the trained spatiotemporal diffusion generation network through a sliding window fusion mechanism to obtain the denoised deep latent variables is as follows:

[0036] Step S4.1: Set the window length to W frames and the overlap length to O frames, where W is an integer greater than or equal to 10 and less than or equal to 25, and O is an integer greater than or equal to 2 and less than W; based on the window length and overlap length, segment the video sequence in both the forward and reverse directions to form multiple time windows for forward processing and multiple time windows for reverse processing; the G time windows for forward processing are sequentially named from the first to the Gth time windows in the forward processing according to their segmentation order; the G time windows for reverse processing are sequentially named from the first to the Gth time windows in the reverse processing according to their segmentation order.

[0037] Step S4.2: Input the conditional latent variable sequence of the first time window of the forward processing and the frame-level semantic feature vector into the spatiotemporal diffusion generation network of the trained deep completion model to obtain the denoised deep latent variables corresponding to each frame within the time window.

[0038] Step S4.3: Add the denoised depth latent variables of all overlapping frames in the first time window of the forward processing to the corresponding overlapping frames of the conditional latent variable sequence in the second time window of the forward processing, to obtain the conditional latent variables with prior knowledge in the second time window.

[0039] Step S4.4: Input the conditional latent variables with prior knowledge and the frame-level semantic feature vector of the second time window into the spatiotemporal diffusion generation network of the trained deep completion model to obtain the denoising deep latent variables corresponding to each frame in the time window.

[0040] Step S4.5: Perform linear weighted fusion on the overlapping frames of the denoising depth latent variable of the first time window and the denoising depth latent variable of the second time window to obtain the linear denoising depth latent variable of the second time window.

[0041] Step S4.6: Sequentially obtain the linear denoising depth latent variables from the 3rd time window to the last time window. Specifically, the process for obtaining the linear denoising depth latent variable for the Kth time window (K=3,4,…,G, where G represents the total number of time windows) is as follows:

[0042] Step S4.6.1: Add the denoised depth latent variables of all overlapping frames in the (K-1)th time window of the forward processing to the corresponding overlapping frames of the conditional latent variable sequence of the Kth time window of the forward processing, to obtain the conditional latent variables with prior knowledge in the Kth time window.

[0043] Step S4.6.2: Input the conditional latent variables with prior knowledge and the frame-level semantic feature vector of the Kth time window into the spatiotemporal diffusion generation network of the trained deep completion model to obtain the denoising deep latent variables corresponding to each frame in the time window.

[0044] Step S4.6.3: Perform linear weighted fusion on the corresponding parts of the overlapping frames of the denoising depth latent variable of the (K-1)th time window and the denoising depth latent variable of the Kth time window to obtain the linear denoising depth latent variable of the Kth time window.

[0045] Step S4.7: Take the linear denoising deep latent variable of the Gth time window as the positive denoising deep latent variable sequence;

[0046] Step S4.8: Starting from the first time window of the reverse processing, process in the same way as steps S4.2 to S4.6 to obtain the linear denoising deep latent variable of the Kth time window of the reverse processing; use the linear denoising deep latent variable of the Kth time window of the reverse processing as the reverse denoising deep latent variable sequence.

[0047] Step S4.9: Perform a weighted fusion of the forward denoising deep latent variable sequence and the reverse denoising deep latent variable sequence to obtain the denoising deep latent variable of the video sequence.

[0048] Furthermore, in step S4, the specific process of obtaining the completed video depth sequence through denoising depth latent variable decoding and scale recovery of the video sequence to be completed by the variational autoencoder is as follows:

[0049] Step S4.10: For the video sequence to be completed, use the variational autoencoder in the trained deep completion model to decode its denoising depth latent variable frame by frame to obtain the decoding relative depth of each frame of the video sequence to be completed.

[0050] Step S4.11: For each frame of the video sequence to be completed, use the validity mask of the depth data of the frame to determine the position of the valid depth of the frame relative to the decoding depth. Use the least squares method to process the valid depth of the depth data of the frame and the valid depth of the frame relative to the decoding depth to obtain the global scale parameter a and the offset parameter b.

[0051] Step S4.12: For each frame of the video sequence to be completed, first multiply its decoded relative depth by a, and then add b to obtain the measured depth of the frame;

[0052] Step S4.13: Concatenate the depth measurements of all frames in the video sequence to be completed in chronological order to obtain the completed video depth sequence, thus completing the video depth completion. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the overall process of the spatiotemporally consistent video depth completion method under the zero-sample unified diffusion framework of the present invention;

[0054] Figure 2 This is a schematic diagram of the reasoning process of the sliding window fusion mechanism in the spatiotemporally consistent video depth completion method under the zero-sample unified diffusion framework of the present invention.

[0055] Figure 3 This diagram illustrates the depth completion performance of the spatiotemporally consistent video depth completion method under the zero-sample unified diffusion framework of this invention and the PriorDA method on the Virtual Kitti 2 test dataset. Detailed Implementation

[0056] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0057] Example 1: As Figure 1 As shown, a spatiotemporally consistent video depth completion method under a zero-shot unified diffusion framework includes the following steps:

[0058] Step S1: Construct a deep completion model, which includes a variational autoencoder, a semantic coding network, and a spatiotemporal diffusion generation network. The variational autoencoder and the semantic coding network are connected to the spatiotemporal diffusion generation network, respectively.

[0059] Step S2: Prepare training data: Obtain multiple video sequences and their depth data and validity masks for each frame. Each video sequence consists of no less than 50 color images in chronological order. For each frame of each video sequence, generate the corresponding frame-level semantic feature vector, and calculate the conditional latent variables based on its depth data, validity mask, and relative depth generated by the pre-trained monocular depth model.

[0060] Step S3: Train the deep completion model: Use conditional latent variables or a sequence of conditional latent variables composed of consecutive frame conditional latent variables as input to the spatiotemporal diffusion generation network, and use frame-level semantic feature vectors as conditions to train the spatiotemporal diffusion generation network in stages, so that it learns to predict the true deep latent variables from noisy deep latent variables.

[0061] Step S4: Perform depth completion on the video sequence to be completed, which consists of no less than 50 color images arranged in chronological order: Obtain the depth data and validity mask of each frame of the video sequence to be completed; obtain the conditional latent variables and frame-level semantic feature vectors of each frame in the same way as in Step S2; input the conditional latent variable sequence into the trained spatiotemporal diffusion generation network through a sliding window fusion mechanism to obtain the denoised depth latent variables of the video sequence to be completed; finally, decode the denoised depth latent variables of the video sequence to be completed and restore the scale through a variational autoencoder to obtain the completed video depth sequence.

[0062] Figure 1 In this context, CLIP encoder represents a pre-trained semantic encoding network, Diffusion U-Net represents a spatiotemporal diffusion generator network, and VAE represents a variational autoencoder.

[0063] In this embodiment, a deep completion model comprising a variational autoencoder, a semantic coding network, and a spatiotemporal diffusion generation network is constructed to achieve the design objective of deep completion for long video sequences that are sparse, noisy, or structurally damaged. Its design features are reflected in the collaborative design of the entire process, from data preparation and deep completion model training to inference applications. Each step is closely aligned with the goal of generating deep sequences that possess metric consistency, structural integrity, and temporal stability.

[0064] First, through an innovative design of conditional latent variables, the original depth data (which may be sparse, noisy, or structurally damaged with mask markings) is coupled with the relative depth generated by the monocular depth model to form a unified geometric conditional representation; simultaneously, frame-level semantic feature vectors are used as independent semantic conditional inputs. This multi-source conditional coupling architecture enables the model to cope with multiple depth degradation scenarios within a unified framework, without requiring model adjustments or fine-tuning for different degradation types, achieving true zero-shot generalization capability.

[0065] Second, a phased training strategy is adopted: first, single-frame training is performed to establish spatial depth completion capability, and then short and long sequences are gradually introduced for training to enhance the temporal modeling capability of the spatiotemporal diffusion generation network. This progressive training method enables the model to gradually learn how to effectively utilize temporal information to optimize spatial depth prediction, especially performing well in handling temporally related issues such as occlusion regions and motion blur, achieving a unified optimization of spatial structure integrity and temporal smoothness.

[0066] Third, a sliding window fusion mechanism is proposed to effectively solve the error accumulation problem in long sequence processing. The sliding window fusion mechanism breaks the unidirectional error propagation chain by segmenting the long sequence into overlapping local windows for parallel processing and employing a fusion strategy for overlapping regions. Simultaneously, this mechanism allows the model to fully utilize contextual information when processing each local window, maintaining long-range consistency while avoiding excessive computational complexity in global processing, significantly improving the temporal stability of long sequence deep completion.

[0067] Fourth, by decoupling geometric conditions (conditional latent variables) from semantic conditions (frame-level semantic feature vectors), the model can focus on restoring geometric structures while leveraging high-level semantic information to guide the understanding of complex scenes. The absolute scale information fused in the conditional latent variables complements the relative geometric layout: sparse / noisy depth provides metric anchors, while relative depth provides continuous geometric priors. This design is particularly beneficial for restoring details such as thin structures and complex boundaries, significantly improving the structural integrity of depth completion.

[0068] V. A complete technical closed loop from conditional input to metric output was established, ensuring the accuracy of depth prediction metrics. The entire technical solution forms a complete technical chain from multi-source conditional construction and spatiotemporal diffusion generation to final scale recovery. During training, the variational autoencoder and semantic coding network are frozen, focusing on optimizing the spatiotemporal diffusion generation network to ensure the learning efficiency of the conditional-to-depth mapping. During inference, the generated latent depth variables are converted into physically meaningful metric depths through decoding and scale recovery of the variational autoencoder, ensuring that the output depth sequence is not only geometrically accurate but also consistent in absolute scale, meeting the stringent requirements for metric consistency in practical applications.

[0069] Example 2: This example is basically the same as Example 1, except that in this example, the specific process of calculating the conditional latent variables for a single frame image in step S2 is as follows:

[0070] Step S2.1: Obtain the depth data and validity mask of the frame; denote the pixel size of a single frame image as M×N×3, where M represents the number of pixel rows, N represents the number of pixel columns, and 3 represents the number of channels; the depth data and validity mask of a single frame image are data matrices of size M×N, and in the validity mask, 1 represents validity and 0 represents invalidity; the validity mask is used to indicate whether each depth in the depth data is valid; if the data at a certain position of the validity mask of a certain depth data is 0, then the depth at the same position in the depth data is an invalid depth, and if it is 1, then the depth at the same position in the depth data is a valid depth;

[0071] Step S2.2: Using a pre-trained monocular depth model, generate the relative depth of the frame, and determine the valid and invalid depths in the relative depth based on the validity mask of the depth data of the frame.

[0072] Step S2.3: Use a pre-trained semantic coding network to extract the frame-level semantic feature vector of the frame;

[0073] Step S2.4: In the depth data of the frame, each invalid depth is subjected to inverse spatial distance weighted interpolation based on its spatial distance to the valid depth, where the weight is inversely proportional to the spatial distance, thereby obtaining the coarse depth of the frame with spatial connectivity. The valid and invalid depths in the coarse depth are then determined based on the validity mask of the frame's depth data. Specifically, the inverse spatial distance weighted interpolation process for a given invalid depth is as follows: For the invalid depth, the nearest valid depth in terms of spatial distance is searched within the set of all valid depths in the frame's depth data as its valid anchor point; the Euclidean distance between the invalid depth and all its valid anchor points is calculated; the weight of each valid anchor point is calculated based on the Euclidean distance, where the weight is inversely proportional to the Euclidean distance. The larger the Euclidean distance between the invalid depth and a valid anchor point, the smaller the weight of the valid anchor point. The weights of all valid anchor points are normalized so that the sum of all valid anchor point weights is 1; finally, the depth values ​​of all valid anchor points are multiplied by their corresponding weights and summed to obtain the interpolated depth value of the invalid depth. By replacing all invalid depths in the depth data of the frame with the corresponding interpolated depth values, a coarse depth with spatial connectivity for the frame is obtained.

[0074] Step S2.5: Perform least-squares linear processing on the effective depth in the depth data of the frame and the effective depth in the relative depth to obtain the global scaling parameter and the offset parameter; then, multiply the depth data value in the relative depth of the frame by the scaling parameter and add the offset parameter to obtain the fitted relative depth of the frame.

[0075] Step S2.6: Using the effective depth in the fitted relative depth of the frame, interpolate each invalid depth in its coarse depth using inverse spatial distance weighted interpolation (same method as in step S2.4) to obtain a metric depth approximation that combines absolute scale anchor points and local structural continuity.

[0076] Step S2.7: Perform quantile cropping and linear normalization on the approximate depth and relative depth of the frame to obtain the normalized approximate depth and normalized relative depth. To eliminate the interference of extreme points, a robust percentile statistical method is used for quantile cropping and linear normalization. The specific process of performing quantile cropping and linear normalization on the approximate depth of the frame to obtain the normalized approximate depth is as follows: First, statistically analyze the distribution of all pixel values ​​in the approximate depth of the frame, and calculate the 1st percentile and 99th percentile values ​​as the lower and upper bounds of the effective value range. Then, perform a cropping operation, assigning the 1st percentile value to the pixel values ​​in the approximate depth of the frame whose values ​​are less than the 1st percentile value, and assigning the 99th percentile value to the pixel values ​​whose values ​​are greater than the 99th percentile value. Finally, perform a linear mapping, linearly transforming each pixel value in the cropped approximate depth of the frame from the interval [1st percentile value, 99th percentile value] to the normalized interval [-1, ... [1], to obtain a normalized depth approximation. The relative depth of the frame is segmented, cropped, and linearly normalized to obtain its normalized relative depth in the same way as the normalized depth approximation;

[0077] Step S2.8: Use a variational autoencoder to encode the normalized metric depth approximation and normalized relative depth of the frame to obtain the corresponding coarse completion depth latent variable and relative depth latent variable;

[0078] Step S2.9: The depth data of this frame is first encoded by a variational autoencoder to obtain the target latent variable, and then Gaussian noise is injected into the target latent variable to generate a noisy target latent variable;

[0079] Step S2.10: The coarse completion depth latent variable, relative depth latent variable, and noisy target latent variable of the frame are spliced ​​along the channel dimension to obtain its conditional latent variable.

[0080] Example 3: This example is basically the same as Example 2, except that in this example, the specific process of training the spatiotemporal diffusion generation network in stages in step S3 is as follows:

[0081] Step S3.1: Take the vector composed of the conditional latent variable, target latent variable and frame-level semantic feature vector of each frame of each video sequence as a training sample, and use all training samples to form the first stage training sample set; use the first stage training sample set to train the spatial correlation module in the spatiotemporal diffusion generation network.

[0082] Step S3.2: Generate a random number p between 2 and 15. Divide each video sequence into a unit video sequence, starting from the first frame, with each p frames until the sequence is completely divided or there are fewer than p frames. Concatenate the conditional latent variables of each frame in each unit video sequence into a conditional latent variable sequence and the target latent variables into a target latent variable sequence. Use the conditional latent variable sequence, target latent variable sequence, and frame-level semantic feature vector of each frame in each unit video sequence to form a sequence training sample. Use all sequence training samples to form the second-stage training sample set. Use the second-stage training sample set to train the temporal correlation module in the spatiotemporal diffusion generation network.

[0083] Step S3.3: Generate a random number q between 2 and 40. Divide each video sequence into a unit video sequence, starting from the first frame, with each q frames until the sequence is completely divided or there are fewer than q frames. Concatenate the conditional latent variables of each frame in each unit video sequence into a conditional latent variable sequence and the target latent variables into a target latent variable sequence. Use the conditional latent variable sequence, target latent variable sequence, and frame-level semantic feature vector of each frame in each unit video sequence to form a sequence training sample. Use all sequence training samples to form the third-stage training sample set. Use the third-stage training sample set to train the temporal correlation module in the spatiotemporal diffusion generation network trained in step S3.2.

[0084] Step S3.4: Divide each video sequence into a unit video sequence starting from the first frame, with every 25 frames divided until the sequence is completely divided or there are fewer than 25 frames. Concatenate the conditional latent variables of each frame in each unit video sequence into a conditional latent variable sequence and the target latent variables into a target latent variable sequence. Use the conditional latent variable sequence, target latent variable sequence, and frame-level semantic feature vector of each frame in each unit video sequence to form a sequence training sample. Use all sequence training samples to form the fourth-stage training sample set. Use the fourth-stage training sample set to train the temporal correlation module in the spatiotemporal diffusion generation network trained in step S3.3. Finally, the trained deep completion model is obtained.

[0085] In this application, random numbers p and q are introduced to divide the unit video sequence during the construction of training samples, which improves the generalization ability of the deep completion model.

[0086] Example 4: This example is basically the same as Example 3, except that in step S4, the conditional latent variable sequence is input into the trained spatiotemporal diffusion generation network through a sliding window fusion mechanism to obtain denoised deep latent variables. Then, the denoised deep latent variables of the video sequence to be completed are decoded and scaled by a variational autoencoder to obtain the completed video depth sequence. The specific process is as follows:

[0087] Step S4.1: Set the window length to W frames and the overlap length to O frames, where W is an integer greater than or equal to 10 and less than or equal to 25, and O is an integer greater than or equal to 2 and less than W; based on the window length and overlap length, segment the video sequence in both the forward and reverse directions to form multiple time windows for forward processing and multiple time windows for reverse processing; the G time windows for forward processing are sequentially named from the first to the Gth time windows in the forward processing according to their segmentation order; the G time windows for reverse processing are sequentially named from the first to the Gth time windows in the reverse processing according to their segmentation order.

[0088] Step S4.2: Input the conditional latent variable sequence of the first time window of the forward processing and the frame-level semantic feature vector into the spatiotemporal diffusion generation network of the trained deep completion model to obtain the denoised deep latent variables corresponding to each frame within the time window.

[0089] Step S4.3: Add the denoised depth latent variables of all overlapping frames in the first time window of the forward processing to the corresponding overlapping frames of the conditional latent variable sequence in the second time window of the forward processing, to obtain the conditional latent variables with prior knowledge in the second time window.

[0090] Step S4.4: Input the conditional latent variables with prior knowledge and the frame-level semantic feature vector of the second time window into the spatiotemporal diffusion generation network of the trained deep completion model to obtain the denoising deep latent variables corresponding to each frame in the time window.

[0091] Step S4.5: Perform linear weighted fusion on the overlapping frames of the denoising depth latent variables of the first time window and the second time window to obtain the linear denoising depth latent variable of the second time window. The specific process of linear weighted fusion is as follows: determine the overlapping area of ​​two adjacent time windows on the time axis. For each frame in the overlapping area, calculate the fusion weight based on its relative time position in the overlapping area. The weights are allocated in a linearly gradual manner, that is, as the frame number moves forward in the overlapping area in chronological order, the weight of the first time window (the previous time window) decreases linearly from large to small, while the weight of the second time window (the current time window) increases linearly from small to large, and the sum of the weights of the two time windows corresponding to the same frame is always 1. Finally, multiply the denoising depth latent variables of the two time windows at the same frame position by their corresponding weights and add them together to obtain the fused latent variable.

[0092] Step S4.6: Sequentially obtain the linear denoising depth latent variables from the 3rd time window to the last time window. Specifically, the process for obtaining the linear denoising depth latent variable for the Kth time window (K = 3, 4, ..., G, where G represents the total number of time windows) is as follows:

[0093] Step S4.6.1: Add the denoised depth latent variables of all overlapping frames in the (K-1)th time window of the forward processing to the corresponding overlapping frames of the conditional latent variable sequence of the Kth time window of the forward processing, to obtain the conditional latent variables with prior knowledge in the Kth time window.

[0094] Step S4.6.2: Input the conditional latent variables with prior knowledge and the frame-level semantic feature vector of the Kth time window into the spatiotemporal diffusion generation network of the trained deep completion model to obtain the denoising deep latent variables corresponding to each frame in the time window.

[0095] Step S4.6.3: Perform linear weighted fusion on the corresponding parts of the overlapping frames of the denoising depth latent variable of the (K-1)th time window and the denoising depth latent variable of the Kth time window to obtain the linear denoising depth latent variable of the Kth time window.

[0096] Step S4.7: Take the linear denoising deep latent variable of the Gth time window as the positive denoising deep latent variable sequence;

[0097] Step S4.8: Starting from the first time window of the reverse processing, process in the same way as steps S4.2 to S4.6 to obtain the linear denoising depth latent variable of the Gth time window of the reverse processing; use the linear denoising depth latent variable of the Gth time window of the reverse processing as the reverse denoising depth latent variable sequence.

[0098] Step S4.9: Perform a weighted fusion of the forward denoising deep latent variable sequence and the reverse denoising deep latent variable sequence to obtain the denoising deep latent variable of the video sequence;

[0099] Step S4.10: For the video sequence to be completed, use the variational autoencoder in the trained deep completion model to decode its denoising depth latent variable frame by frame to obtain the decoding relative depth of each frame of the video sequence to be completed.

[0100] Step S4.11: For each frame of the video sequence to be completed, use the validity mask of the depth data of the frame to determine the position of the valid depth of the frame relative to the decoding depth. Use the least squares method to process the valid depth of the depth data of the frame and the valid depth of the frame relative to the decoding depth to obtain the global scale parameter a and the offset parameter b.

[0101] Step S4.12: For each frame of the video sequence to be completed, first multiply its decoded relative depth by a, and then add b to obtain the measured depth of the frame;

[0102] Step S4.13: Concatenate the depth measurements of all frames in the video sequence to be completed in chronological order to obtain the completed video depth sequence, thus completing the video depth completion.

[0103] In this embodiment, the inference process diagram of the sliding window fusion mechanism is shown below. Figure 2 As shown, forward and backward sliding window links with an overlap length of O are constructed to generate latent space sequences. Overlapping frames of adjacent windows are fused using linear ramp weights. The overlapping region of the next time window is reinitialized using previously obtained clean latent variables as a basis and superimposed with Gaussian noise of the matching variance. After the forward and backward latent variable sequences are generated, they are fused based on global linear time weights, providing stable anchoring in the early stages of the sequence and achieving correction in the later stages, thus obtaining a long-range consistent depth sequence.

[0104] To further clarify the design features of the method of this invention, it is compared with the PriorDA method, and the two have the following main differences:

[0105] I. Differences in Spatiotemporal Modeling Capabilities

[0106] PriorDA method: Essentially a single-frame depth estimation framework, its completion process is performed independently frame by frame. Due to the lack of cross-frame temporal constraints, it is prone to inter-frame flickering and structural jumps when processing video sequences, making it difficult to maintain long-term temporal geometric consistency.

[0107] The method of this invention constructs a spatiotemporal diffusion generation network specifically designed for video. By introducing a temporal correlation module and a bidirectional sliding window fusion mechanism, smooth transitions and scale alignment of cross-frame features are achieved within the latent space, significantly improving the temporal stability of long video completion.

[0108] II. Differences between Modeling Paradigms and Generation Mechanisms

[0109] PriorDA method: Employs a discriminative conditional single-frame depth estimation model. It incorporates prior metrics into the estimation model through a pre-filling strategy for optimized regression.

[0110] The method of this invention is a deep completion model based on a diffusion-generative framework. Utilizing the powerful modeling capabilities of the deep completion model, dense deep latent variables are progressively recovered from noise distributions. It exhibits stronger generative recovery capabilities and structural robustness, especially when facing large-scale structurally missing or out-of-range data.

[0111] III. Differences between Feature Decoupling and Guiding Methods

[0112] PriorDA method: The metric conditions and image features are simply concatenated in the input layer, and the model learns the geometric mapping and appearance association at the same time.

[0113] The method of this invention involves designing a decoupled architecture between geometric priors and semantic conditions. A CLIP encoder is used to extract frame-level semantic features as global guidance, while conditional latent variables focus on the transmission of geometric structure. This design avoids interference from image appearance details on depth reconstruction, allowing the depth completion model to focus more on restoring the geometric contours of complex objects.

[0114] IV. Differences between training strategies and generalization ability

[0115] PriorDA method: focuses on generalization on multiple static sparse patterns, and the training process usually does not involve the expansion of the time dimension.

[0116] The method of this invention employs a phased, progressive training strategy. It starts with single-frame spatial completion and gradually transitions to temporal modeling training of short and long sequences. This phased design enables the deep completion model to synergistically optimize spatial accuracy and temporal smoothness, thereby achieving stronger adaptability to real-world long video scenes under "zero-shot" conditions.

[0117] To verify the performance of the spatiotemporally consistent video depth completion method under the zero-shot unified diffusion framework of this invention, the method was tested using the KITTI dataset disclosed in Reference 2, “A. Geiger, P. Lenz and R. Urtasun, “Are we ready for autonomous driving? The KITTI vision benchmark suite,” 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, 2012, pp. 3354-3361, doi: 10.1109 / CVPR.2012.6248074.” (“Are we ready for autonomous driving? The KITTI vision benchmark suite,” 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, 2012, pp. 3354–3361, DOI: 10.1109 / CVPR.2012.6248074.). The method employed the techniques of this invention and Reference 1, “Zehan Wang, Siyu…” Chen, Lihe Yang, Jialei Wang, Ziang Zhang, Hengshuang Zhao, and Zhou Zhao. Depth anything with any prior. arXiv preprint arXiv:2505.10565, 2025. The PriorDA method, published in ("Depth estimation model with any prior. arXiv preprint, arXiv:2505.10565, 2025."), was applied to the test sequence 2011_09_26_drive_0005 in the KITTI dataset. The sparse depth was uniformly set to an 8-line sparse LiDAR, and the final video depth completion effect is shown below. Figure 3 As shown.

[0118] Depend on Figure 3 The results demonstration and comparative analysis show that:

[0119] I. Regarding single-frame structural integrity: Although the PriorDA method utilizes pre-training priors, its essence is based on single-frame optimization, lacking the utilization of video context. When processing sparse 8-line LiDAR input, it is prone to structural artifacts or blurring at object edges and fine structures (e.g., Figure 3 The PriorDA results show a lack of local detail. In contrast, the method of this invention ( Figure 3The method of this invention (as shown in the present invention) can more accurately recover clear object boundaries and geometric details by introducing the fusion of frame-level semantic feature vectors and conditional latent variables, and the generated depth map is more complete in spatial structure and closer to the real scene.

[0120] II. Regarding timing consistency and stability: Figure 3 The temporal depth contour slices (i.e., depth variation maps cropped along the time axis) visually reflect the stability of video depth. The results of the PriorDA method exhibit obvious sawtooth fluctuations on the time axis, indicating severe depth "flickering" and scale drift between consecutive frames, making it difficult to maintain metric consistency between consecutive frames. In contrast, the method of this invention benefits from the staged training of the spatiotemporal diffusion generation network and the bidirectional sliding window fusion mechanism (BOSW) in the inference stage, resulting in smooth and continuous temporal contour slices that greatly suppress inter-frame jumps.

[0121] To further evaluate the depth completion performance of the spatiotemporally consistent video depth completion method under the zero-sample unified diffusion framework of the present invention more objectively and comprehensively, the evaluation index system shown in Table 1 was first established.

[0122] Table 1: Explanation of Evaluation Indicators

[0123]

[0124] The mathematical definitions of each evaluation indicator in Table 1 are described below:

[0125] Absolute relative error (AbsRel): For all valid pixels, calculate the absolute value of the difference between the predicted depth x_p and the true depth x_d, divide it by the true depth value, and finally take the average value of the N valid pixels in the entire image.

[0126] Threshold accuracy ($\delta_1$): For each valid pixel in a video frame, calculate the ratio between its predicted depth value x_p and its true depth value x_d, and take the maximum of this ratio and its reciprocal. When this maximum ratio is less than a preset constant threshold of 1.25, the depth estimate of that pixel is considered accurate.

[0127] Temporal alignment error (TAE): used to quantify the geometric consistency and temporal stability of a video sequence. Mathematically, it is defined as follows: using the reprojection function f(·) between adjacent frames, calculate the absolute relative error between the predicted depth of frame k mapped to the position of frame k+1 and the actual predicted depth of frame k+1, and average the forward and backward mapping errors over the entire video sequence of T frames.

[0128] In addition, to verify the generalization ability of the spatiotemporally consistent video depth completion method under the zero-sample unified diffusion framework of the present invention in the "zero-sample" setting, various real-world degradation modes (i.e. test tasks) as shown in Table 2 were simulated, covering a variety of harsh scenarios from sparse sampling to structural damage.

[0129] Table 2: Degradation Modes and Technical Specifications

[0130]

[0131] Based on the above evaluation metrics and test task settings, the spatiotemporally consistent video depth completion method under the zero-sample unified diffusion framework of this invention and the PriorDA method are quantitatively compared on the KITTI dataset. The quantitative comparison results are shown in Table 3.

[0132] Table 3: Comparison of Quantitative Experiment Results for Each Task

[0133]

[0134] Analysis of the quantitative experimental results shown in Table 3 reveals that the method of this invention has the following advantages over the PriorDA method:

[0135] I. Advantages in Global Metrics and Temporal Stability: In all test tasks, the TAE (Temporal Alignment Error) of this invention is significantly lower than that of the PriorDA method. Particularly in the "Effective Range Limit Completion" task, the TAE of this invention is only 1.12, while PriorDA reaches 2.30. This quantitatively demonstrates that this invention, through the spatiotemporal diffusion generation network and sliding window fusion mechanism, greatly suppresses the inter-frame flickering phenomenon commonly found in the PriorDA method.

[0136] II. Structural Integrity in Complex Scenarios: In scenarios with large-area deficiencies, such as "effective range limit completion" and "structured damage completion," the $\delta_1$ (threshold accuracy) of this invention demonstrates a significant lead. For example, in the range limit test, this invention improves the accuracy from 74.81% to 92.85%, effectively overcoming the limitation of the PriorDA method in recovering distant geometry when global time constraints are missing.

[0137] III. Zero-Sample Generalization Accuracy: In all test scenarios, this invention remains competitive in terms of $\delta_1$, which reflects spatial structure accuracy, and temporally stable TAE. This demonstrates that this invention can not only handle diverse sparse inputs, but also ensure the coherence of video depth while maintaining physical scale realism.

[0138] Therefore, the quantitative experimental results above fully support the technological advancement of the method of this invention over the PriorDA method in the task of deep completion of long sequence videos from the three dimensions of metric consistency, structural integrity and temporal stability.

[0139] In summary, this invention not only outperforms the PriorDA method in single-frame completion accuracy, but more importantly, it effectively solves the temporal inconsistency problem that existing single-frame methods cannot overcome in video sequence processing, achieving video depth completion that combines high-precision geometric restoration and long-sequence temporal stability under zero-sample settings.

Claims

1. A spatiotemporally consistent video depth completion method under a zero-shot unified diffusion framework, characterized in that, Includes the following steps: Step S1: Construct a deep completion model, including a variational autoencoder, a semantic coding network, and a spatiotemporal diffusion generation network. The variational autoencoder and the semantic coding network are connected to the spatiotemporal diffusion generation network, respectively. Step S2: Obtain multiple video sequences and their depth data and validity masks for each frame. Each video sequence consists of no less than 50 color images. For each frame, generate a frame-level semantic feature vector and calculate the conditional latent variable based on its depth data, validity mask, and relative depth generated by the pre-trained monocular depth model. Step S3: Using conditional latent variables or a sequence of conditional latent variables composed of consecutive frame conditional latent variables as input to the spatiotemporal diffusion generation network, and using frame-level semantic feature vectors as conditions, the spatiotemporal diffusion generation network is trained in stages to learn to predict real depth latent variables from noisy depth latent variables. Step S4: Perform depth completion on the video sequence to be completed, which consists of no less than 50 color images: obtain the depth data and validity mask for each frame; The conditional latent variables and frame-level semantic feature vectors for each frame are obtained in the same manner as in step S2. The conditional latent variable sequence is input into the trained spatiotemporal diffusion generation network through a sliding window fusion mechanism to obtain the denoised depth latent variable of the video sequence to be completed. After being decoded by a variational autoencoder and scaled back, the completed video depth sequence is obtained. In step S3, the specific steps for phased training of the spatiotemporal diffusion generation network are as follows: Step S3.1: Take the vector composed of the conditional latent variable, target latent variable and frame-level semantic feature vector of each frame of each video sequence as a training sample, and use the first-stage training sample set composed of all training samples to train the spatial correlation module in the spatiotemporal diffusion generation network. Step S3.2: Generate a random number p between 2 and 15. Divide each video sequence into a unit video sequence starting from the first frame, with each p frame divided into a unit video sequence, until the sequence is completely divided or there are fewer than p frames. Concatenate the conditional latent variables of each frame in each unit video sequence into a conditional latent variable sequence and the target latent variables into a target latent variable sequence. Use the conditional latent variable sequence, target latent variable sequence, and frame-level semantic feature vector of each frame in each unit video sequence to form a sequence training sample. Use the second-stage training sample set composed of all sequence training samples to train the temporal correlation module in the spatiotemporal diffusion generation network. Subsequently, the temporal correlation module in the spatiotemporal diffusion generation network is trained to enhance the temporal modeling capability of the deep completion model. The specific steps for continuing to train the time-related module in the spatiotemporal diffusion generation network are as follows: Step S3.3: Generate a random number q between 2 and 40, and divide each video sequence into a unit video sequence starting from the first frame, with each q frames divided into a unit video sequence, until the sequence is completely divided or there are fewer than q frames. The third-stage training sample set is constructed using the same method as in step S3.2, and the time-related module in the spatiotemporal diffusion generation network trained in step S3.2 is trained using the third-stage training sample set. Step S3.4: Divide each video sequence into a unit video sequence starting from the first frame, with every 25 frames divided until the sequence is completely divided or there are fewer than 25 frames. Concatenate the conditional latent variables of each frame in each unit video sequence into a conditional latent variable sequence and the target latent variables into a target latent variable sequence. Use the conditional latent variable sequence, target latent variable sequence, and frame-level semantic feature vector of each frame in each unit video sequence to form a sequence training sample. Use all sequence training samples to form the fourth-stage training sample set. Use the fourth-stage training sample set to train the temporal correlation module in the spatiotemporal diffusion generation network trained in step S3.

3. The final result is a trained deep completion model.

2. The spatiotemporally consistent video depth completion method under the zero-sample unified diffusion framework according to claim 1, characterized in that, In step S2, the specific steps for calculating the conditional latent variables of a single frame image are as follows: Step S2.1: Obtain the depth data and validity mask of the frame; denote the number of pixel rows × number of columns × channel size of a single frame image as M×N×3; the depth data and validity mask of a single frame image are data matrices of size M×N, where 1 in the validity mask represents valid depth and 0 represents invalid depth; Step S2.2: Using a pre-trained monocular depth model, generate the relative depth of the frame. Based on the pixel position indicated by the effective depth in the validity mask, determine the depth value at the corresponding position in the relative depth as the effective depth, and the rest as invalid depths. Step S2.3: Use a pre-trained semantic coding network to extract the frame-level semantic feature vector of the frame; Step S2.4: For each invalid depth in the depth data of the frame, perform inverse spatial distance weighted interpolation according to its spatial distance from the valid depth, where the weight is inversely proportional to the spatial distance, to obtain the coarse depth of the frame, and determine the valid depth and invalid depth in its coarse depth according to the validity mask. Step S2.5: Perform least-squares linear fitting on the effective depth and the effective depth in the relative depth of the frame's depth data to obtain the fitted relative depth of the frame; correct the coarse depth based on the fitted relative depth to generate a metric depth approximation, and process the metric depth approximation with the relative depth to finally generate conditional latent variables.

3. The spatiotemporally consistent video depth completion method under the zero-sample unified diffusion framework according to claim 2, characterized in that, The specific steps for correcting the coarse depth based on the fitted relative depth to generate a metric depth approximation, and then processing the metric depth approximation with the relative depth to finally generate the conditional latent variable are as follows: Step S2.6: Using the effective depth in the fitted relative depth of the frame, each invalid depth in its coarse depth is weighted and interpolated according to the inverse spatial distance to obtain a metric depth approximation that combines absolute scale anchor points and local structural continuity. Step S2.7: Perform quantization cropping and linear normalization on the approximate depth and relative depth of the frame to obtain its normalized approximate depth and normalized relative depth. Step S2.8: Use a variational autoencoder to encode the normalized metric depth approximation and normalized relative depth of the frame to obtain the corresponding coarse completion depth latent variable and relative depth latent variable; Step S2.9: The depth data of the frame is first encoded by the variational autoencoder to obtain the target latent variable, and then Gaussian noise is injected into the target latent variable to generate a noisy target latent variable; Step S2.10: The coarse completion depth latent variable, relative depth latent variable, and noisy target latent variable of the frame are spliced ​​along the channel dimension to obtain its conditional latent variable.

4. The spatiotemporally consistent video depth completion method under the zero-sample unified diffusion framework according to claim 1, characterized in that, Step S4 obtains the denoising depth latent variable of the video sequence as follows: Step S4.1: Set the window length W to 10-25 frames and the overlap length O to 2 to W-1 frames. Perform segmentation processing on the video sequence in both forward and reverse directions to form multiple time windows for forward and reverse processing. The total number of time windows for forward or reverse processing is denoted as G. The G time windows for forward processing are arranged from the first to the Gth time windows in the forward processing, in order of segmentation. The G time windows for reverse processing are arranged from the first to the Gth time windows in the reverse processing, in order of segmentation. Step S4.2: Obtain the denoising depth latent variable for the first time window of the forward processing; Step S4.3: Obtain the conditional latent variables with prior knowledge for the second time window of the forward processing; Step S4.4: Obtain the denoising depth latent variable for the second time window of the forward processing; Step S4.5: Obtain the linear denoising depth latent variable for the second time window of the forward processing; Step S4.6: Obtain the linear denoising depth latent variables for the 3rd to Gth time windows of the forward processing; Step S4.7: Take the linear denoising deep latent variable of the Gth time window as the positive denoising deep latent variable sequence; Step S4.8: Obtain the linear denoising deep latent variable of the Gth time window of the reverse processing, as the reverse denoising deep latent variable sequence; Step S4.9: Weighted fusion of the forward denoising deep latent variable sequence and the reverse denoising deep latent variable sequence to obtain the denoising deep latent variable of the video sequence.

5. The spatiotemporally consistent video depth completion method under the zero-sample unified diffusion framework according to claim 4, characterized in that, Step S4.2 specifically involves inputting the conditional latent variable sequence of the first time window of the forward processing and the frame-level semantic feature vector into the spatiotemporal diffusion generation network of the trained deep completion model to obtain the denoising deep latent variables corresponding to each frame within the time window. Step S4.3 specifically involves: superimposing the denoised depth latent variables of all overlapping frames in the first time window of the forward processing onto the corresponding overlapping frames of the conditional latent variable sequence in the second time window of the forward processing, to obtain the conditional latent variables with prior knowledge in the second time window. Step S4.4 specifically involves inputting the conditional latent variables with prior knowledge and the frame-level semantic feature vector of the second time window into the spatiotemporal diffusion generation network of the trained deep completion model to obtain the denoising deep latent variables corresponding to each frame within the time window. Step S4.5 specifically involves: performing linear weighted fusion on the overlapping frames of the denoising depth latent variables of the first time window and the second time window to obtain the linear denoising depth latent variables of the second time window.

6. The spatiotemporally consistent video depth completion method under the zero-sample unified diffusion framework according to claim 5, characterized in that, In step S4.6, setting K=3, 4, ..., G, the specific steps for obtaining the linear denoising depth latent variable of the Kth time window of the forward processing using the sliding window fusion mechanism are as follows: Step S4.6.1: Add the denoised depth latent variables of all overlapping frames in the (K-1)th time window of the forward processing to the corresponding overlapping frames of the conditional latent variable sequence of the Kth time window of the forward processing, to obtain the conditional latent variables with prior knowledge in the Kth time window. Step S4.6.2: Input the conditional latent variables with prior knowledge and the frame-level semantic feature vector of the Kth time window into the spatiotemporal diffusion generation network of the trained deep completion model to obtain the denoising deep latent variables corresponding to each frame in the time window. Step S4.6.3: Perform linear weighted fusion of the overlapping frames of the denoising depth latent variable of the (K-1)th time window and the denoising depth latent variable of the Kth time window to obtain the linear denoising depth latent variable of the Kth time window.

7. The spatiotemporally consistent video depth completion method under the zero-sample unified diffusion framework according to claim 6, characterized in that, Step S4.8 specifically involves: starting from the first time window of the reverse processing, using the same method as steps S4.2 to S4.6, sequentially obtaining the linear denoising depth latent variables for the second to Gth time windows of the reverse processing.

8. The spatiotemporally consistent video depth completion method under the zero-sample unified diffusion framework according to claim 1, characterized in that, In step S4, the specific steps for obtaining the completed video depth sequence after denoising depth latent variable decoding and scale recovery of the video sequence to be completed by the variational autoencoder are as follows: Step S4.10: For the video sequence to be completed, use the variational autoencoder in the trained deep completion model to decode its denoising depth latent variable frame by frame to obtain the decoding relative depth of each frame of the video sequence to be completed. Step S4.11: For each frame of the video sequence to be completed, use the validity mask of the depth data of the frame to determine the position of the valid depth of the frame relative to the decoding depth. Use the least squares method to process the valid depth of the depth data of the frame and the valid depth of the frame relative to the decoding depth to obtain the global scale parameter a and the offset parameter b. Step S4.12: For each frame of the video sequence to be completed, first multiply its decoded relative depth by a, and then add b to obtain the measured depth of the frame; Step S4.13: Concatenate the depth measurements of all frames in the video sequence to be completed in chronological order to obtain the completed video depth sequence, thus completing the video depth completion.

Images