Depth estimation method and device for any video, and storage medium

Abstract

The invention discloses a depth estimation method and device for any video and a storage medium, and belongs to the field of visual depth estimation. The method comprises the following steps: carrying out annotation processing on a scene video sample to obtain a deep annotation video data set; screening based on the depth labeling video data set and the TartanAir data set to obtain a spatio-temporal joint training sample; the method comprises the following steps: performing time sequence embedding on a multi-head attention layer in an encoder of a DepthAnything model to obtain a space-time combined multi-head attention layer, and constructing an initial TC-DepthAnything model; training the initial TC-DepthAnything model by adopting a space-time joint training sample, and performing constraint by adopting an overall training loss function formed by space consistency loss and time domain regularization loss in the training process to obtain a target TC-DepthAnything model; and inputting any video into the target TC-DepthAnything model to obtain a predicted depth video. The problem of time sequence jitter of DepthAnything in video depth estimation is solved, and flicker artifacts and motion blur in a dynamic scene are inhibited. And video depth estimation with a large application range and an accurate estimation result is realized.

Description

Technical Field

[0001] This invention belongs to the field of visual depth estimation, and more specifically, relates to a depth estimation method, apparatus and storage medium for arbitrary videos. Background Technology

[0002] Monocular depth estimation (MDE), an important research direction in computer vision, aims to recover the spatial distance between each pixel and the camera from a single image—a task falling under the category of pixel-level prediction. With the increasing maturity of MDE technology, research in video depth estimation (VDE) has also emerged. As a core technology in 3D vision, VDE provides autonomous system environment capabilities for various autonomous devices. It reconstructs 3D coordinate information through depth calculation algorithms, directly supporting fields such as autonomous driving, robotics, 3D reconstruction, and unmanned navigation systems. Furthermore, it can be combined with other computer vision tasks as an auxiliary technology to achieve functions such as object detection and semantic segmentation.

[0003] With the iterative upgrades of current computing hardware and breakthroughs in deep learning theory, deep learning-based methods have gradually become the mainstream research direction for VDE (Video Depth Assessment). Deep neural networks, through end-to-end learning mechanisms, can autonomously extract multi-level features from massive amounts of data, achieving accurate mapping from single-frame images to dense depth maps, significantly improving the robustness of algorithms under varying lighting conditions and occlusion scenarios. Researchers have gradually built a theoretical framework and technical system for video temporal depth perception by continuously overcoming key bottlenecks. In 2020, Luo et al. proposed CVD (Continuous Visualization) to improve the temporal continuity of depth prediction models by using camera pose and optical flow. The main limitation of this method is the need for accurate camera pose, and pose estimation itself is a challenging problem, which will greatly affect the depth estimation effect in the presence of dynamic object motion. To solve this problem, Kopf et al. proposed Robust-CVD to co-optimize camera pose and depth alignment. Although the method improves the robustness of video depth estimation, the computational efficiency bottleneck brought by the joint optimization mechanism is difficult to meet the needs of real-time applications. To address this issue, Wang et al. proposed FMNet, which leverages the characteristic that missing frames in a video can be recovered from adjacent frames. By masking parts of the video frame and having the model predict the depth of the occluded frame based on neighboring frames, it learns the intrinsic correlation between video frames. This mechanism allows FMNet to naturally learn temporal consistency during the reconstruction of the occluded frame, thereby enhancing the stability of video depth estimation. Xian et al., based on a pre-trained DPT model, introduced additional temporal embeddings into the Transformer structure and adopted spatiotemporal consistency loss, enabling the model to perform stable and reliable depth prediction for video input. Wang et al. proposed a plug-and-play video depth stabilizer, NVDS+, which consists of a depth predictor and a stabilization network. The depth predictor can be any single-image depth model that generates the initial flicker disparity map, while the stabilization network uses a cross-attention module to extract depth-aware features from the temporal information of relevant frames and operates in a sliding window manner. To further enhance consistency, a flow-guided consistency fusion strategy was proposed, adaptively fusing bidirectional results based on the motion amplitude between relevant frames and pixels. At the data level, this work has constructed VDW, the largest natural scene video depth dataset to date, which has solved some of the problems of lack of depth annotation in large-scale natural scene videos under supervised learning. However, the problem of low accuracy of its depth labels still needs to be solved.

[0004] With technological advancements, mainstream video depth estimation relies on RGB-D datasets to learn the mapping function from video to depth. The training of video depth estimation models heavily depends on the quality and diversity of labeled data. However, existing RGB-D datasets commonly suffer from sensor noise interference. Whether it's the NYUDv2 indoor dataset acquired using Kinect or the KITTI street scene data obtained using LiDAR, hardware limitations lead to point cloud sparsity and outlier noise issues. These outliers disrupt the spatial continuity of the depth map, affecting the model's ability to model scene geometric features and thus reducing the final prediction accuracy. Furthermore, mainstream datasets exhibit significant inter-domain distribution differences due to sensor physical characteristics. Indoor scenes rely on Kinect sensors with a range of less than 10 meters, while road environment data is mostly acquired by vehicle-mounted LiDAR with a range of 80 meters. This scene fragmentation makes it difficult for models to establish cross-domain generalization capabilities and adapt to the depth perception needs of unstructured scenes.

[0005] For input video in a scene, current mainstream methods focus on robust depth estimation of single-frame images. While they demonstrate strong generalization ability in depth estimation of various scene videos, their design paradigm lacks explicit modeling of the spatiotemporal consistency between video frames. This leads to the model performing frame-by-frame depth prediction and then simply stitching them together when processing a large number of consecutive video frames. This results in flickering artifacts and inter-frame jitter in the overall depth video, affecting the system's robustness. Summary of the Invention

[0006] In view of the shortcomings of related technologies, the purpose of this invention is to provide a depth estimation method, device and storage medium for arbitrary videos, aiming to solve the problem of temporal jitter artifacts in existing depth estimation models for video depth estimation.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a depth estimation method for arbitrary videos, comprising: Depth is labeled on scene video samples to obtain a depth-labeled video dataset; video data containing dynamic targets or dynamic scenes are selected from the depth-labeled video dataset and the TartanAir dataset, and data augmentation processing is performed to obtain spatiotemporal joint training samples. Temporal embedding is performed on the multi-head attention layer in the encoder of the DepthAnything model to obtain a spatiotemporal combined multi-head attention layer, and the initial TC-DepthAnything model is constructed. The initial TC-DepthAnything model is trained using the spatiotemporal joint training samples. During training, a global training loss function consisting of spatial consistency loss and temporal regularization loss is used for constraint to obtain the target TC-DepthAnything model. The spatial consistency loss includes affine invariance loss, multi-scale gradient matching loss, and direction-amplitude normal loss. The temporal regularization loss is a constraint on the optical flow changes between the forward / backward optical flow obtained from the optical flow estimation model and the forward / backward mask. Input any video into the target TC-DepthAnything model to obtain the predicted depth video.

[0008] Optionally, the DepthAnything model uses DinoV2 as the encoder and DPT as the decoder; learnable temporal position encoding is embedded in the DinoV2 encoder to capture motion image changes between adjacent video frames through an explicit spatiotemporal attention mechanism, and to learn the temporal features of the deep video used for training.

[0009] Optionally, capturing motion image changes between adjacent video frames through an explicit spatiotemporal attention mechanism includes: The input image frame is flattened, and then word vectors are obtained by combining the word embedding and position encoding methods. The word vectors are subjected to a layer normalization operation, and then input into the spatiotemporal combined multi-head attention layer of the Transformer module for temporal embedding. The output of the spatiotemporal combined multi-head attention layer is then subjected to layer normalization and a multilayer perceptron to obtain the final output.

[0010] Optionally, the expression for the affine invariance loss is:

[0011] Where K represents the number of effective pixels. H is the length of the image, and W is the width of the image. , For the predicted depth map, For the corresponding truth value result; The expression for the multi-scale gradient matching loss is:

[0012] in, , This represents the difference between the predicted and actual values ​​under the S-scale. The expression for the direction-amplitude normal loss is:

[0013] in, These are the weighting coefficients for cosine similarity and L1 loss, respectively. Denotes the Euclidean norm. These are the normal vectors corresponding to the predicted depth map and the ground truth depth value, respectively. ; The formula for the spatial consistency loss is:

[0014] in, The weight parameters are set.

[0015] Optionally, the temporal regularization loss is used to constrain the temporal relationship between adjacent frames; The expression for the temporal regularization loss is:

[0016] in, , This represents the number of valid pixels in the mask, t is the time, T is the maximum time, and T-2 is the maximum time minus two frames. for Prediction depth after time-normalization for Two-way mask for time, ; For forward optical flow Depth map at time step mapped to Depth map at any given time. For backward optical flow Depth map at time step mapped to The depth map at time step, where k is the position of a pixel in the image.

[0017] Optionally, the training process for the initial TC-DepthAnything model may also include: The NYUDv2, Sintel, and KITTI datasets were input into the trained TC-DepthAnything model for qualitative and quantitative validation.

[0018] Optionally, the depth annotation process for the scene video samples to obtain a depth-annotated video dataset includes: Multiple scene video samples were selected from the DL3DV-10K dataset as input. Frame sampling was performed on the input scene video samples to construct a set of video frame image sequences, and the camera transformation matrix corresponding to each frame image was obtained synchronously. The image sequence set is sparsely reconstructed according to the COLMAP standard format. The reconstruction process is performed using confidence-weighted constraint volume rendering to obtain a 3D scene containing depth information at each location and output the depth map of the 3D scene from each viewpoint. By analyzing the optical flow motion patterns between consecutive frames, the dense optical flow field between adjacent frames is calculated. Based on the optical flow amplitude thresholding, static backgrounds and dynamic regions are identified, a consistency mask is generated, and the error of the dynamic region in the depth map is repaired based on the consistency mask. The repaired depth map is stitched frame by frame to obtain a ground truth depth video. The view is then expanded by interpolation using the camera according to the required frame rate to obtain a depth-annotated video dataset.

[0019] Secondly, the present invention also provides a depth estimation apparatus for arbitrary videos, comprising: The sample data processing module is used to annotate the depth of scene video samples to obtain a depth-annotated video dataset; and to filter video data containing dynamic targets or dynamic scenes from the depth-annotated video dataset and the TartanAir dataset, and to perform data augmentation processing to obtain spatiotemporal joint training samples. The model building module is used to perform temporal embedding in the multi-head attention layer of the encoder of the DepthAnything model to obtain a spatiotemporal combined multi-head attention layer and build the initial TC-DepthAnything model. The model training module is used to train the initial TC-DepthAnything model using the spatiotemporal joint training samples. During the training process, a global training loss function consisting of spatial consistency loss and temporal regularization loss is used for constraints to obtain the target TC-DepthAnything model. The spatial consistency loss includes affine invariance loss, multi-scale gradient matching loss, and direction-magnitude normal loss. The temporal regularization loss is a constraint on the changes in optical flow in consecutive frames of the predicted depth map obtained by the optical flow estimation model based on the forward / backward optical flow and the forward / backward masks. The prediction module is used to input any video into the target TC-DepthAnything model to obtain a predicted depth video.

[0020] Thirdly, the present invention also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the depth estimation method for arbitrary video as described in any one of the first aspects.

[0021] Compared with the prior art, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: 1. This invention provides a depth estimation method for arbitrary videos. It obtains spatiotemporal joint training samples based on depth-annotated video datasets and TartanAir datasets, achieving data-driven optimization. Addressing the insufficient modeling of temporal continuity in current DepthAnything models, it proposes the TC-DepthAnything model with enhanced spatiotemporal consistency. This model embeds learnable temporal position encoding into its architecture to learn temporal features and utilizes a composite loss function for constraints. It integrates optical flow projection loss, implicit temporal regularization terms, and planar normal vector geometric constraints to ensure the consistency of dynamic target edges and maintain scene structural smoothness. Simultaneously, implicit temporal regularization loss is added to constrain adjacent frames in the temporal domain. Through the construction of spatiotemporal joint training samples, explicit spatiotemporal attention mechanisms, and composite loss function optimization, the method solves the temporal jitter problem in DepthAnything video depth estimation and suppresses flicker artifacts and motion blur in dynamic scenes.

[0022] 2. This invention provides a depth estimation method for arbitrary videos. Addressing the problem that the lack of large-scale labeled depth video data restricts the generalization ability of depth estimation models in various open environments, this invention proposes a depth labeling method for open scenes that integrates neural radiation field multi-view 3D reconstruction and motion compensation mechanisms. It performs multi-view dense 3D reconstruction using current 3D reconstruction methods, and achieves depth error correction for dynamic object regions through confidence map preprocessing and consistency mask optimization guided by optical flow. It also adaptively obtains the best depth information for each angle of the reconstructed scene, thereby constructing a DL3DV-Depth dataset covering the diversity of indoor and outdoor scenes. Attached Figure Description

[0023] Figure 1 This is an overall flowchart of the video depth annotation method for open scenes provided in the embodiments of the present invention; Figure 2 This is a block diagram of the spatiotemporally consistent video depth estimation method based on the DepthAnything model provided in this embodiment of the invention; Figure 3 This is a schematic diagram of the encoder timing embedding in the TC-DepthAnything model provided in this embodiment of the invention; Figure 4 This is a schematic diagram of the spatiotemporal combined multi-head attention layer framework provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the scaled dot product attention layer provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the qualitative experimental results using the NYUDv2 dataset provided in an embodiment of the present invention; Figure 7This is a schematic diagram illustrating the qualitative experimental results using the Sintel dataset provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the qualitative experimental results using the KITTI dataset provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the temporal consistency visualization results of depth estimation results of different methods on the Sintel dataset provided in the embodiments of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0025] The following description, in conjunction with a preferred embodiment, illustrates the content involved in the above embodiments.

[0026] Example 1 This invention provides a depth estimation method for arbitrary videos, comprising: Depth is labeled on scene video samples to obtain a depth-labeled video dataset; video data containing dynamic targets or dynamic scenes are selected from the depth-labeled video dataset and the TartanAir dataset, and data augmentation processing is performed to obtain spatiotemporal joint training samples. Temporal embedding is performed on the multi-head attention layer in the encoder of the DepthAnything model to obtain a spatiotemporal combined multi-head attention layer, and the initial TC-DepthAnything model is constructed. The initial TC-DepthAnything model is trained using the spatiotemporal joint training samples. During training, a global training loss function consisting of spatial consistency loss and temporal regularization loss is used for constraint to obtain the target TC-DepthAnything model. The spatial consistency loss includes affine invariance loss, multi-scale gradient matching loss, and direction-amplitude normal loss. The temporal regularization loss is a constraint on the changes in optical flow across consecutive frames of the predicted depth map obtained by the optical flow estimation model, based on the forward / backward optical flow and the forward / backward masks. Input any video into the target TC-DepthAnything model to obtain the predicted depth video.

[0027] To address the limitation of existing technologies on the generalization ability of depth estimation models due to the lack of large-scale labeled depth video data, this solution selects spatiotemporal joint training samples based on depth-labeled video datasets and the TartanAir dataset to achieve data-driven optimization. While existing models demonstrate strong generalization ability in various video depth estimation scenarios, their design paradigms lack explicit modeling of spatiotemporal consistency between video frames, resulting in temporal jitter artifacts in the predicted video depth estimation results. This solution proposes the TC-DepthAnything model to enhance spatiotemporal consistency. It embeds learnable temporal location codes into the model architecture, learns temporal features, and uses a composite loss function for constraints, ensuring the consistency of dynamic target edges and maintaining the smoothness of scene structure.

[0028] To address the aforementioned issues in existing solutions, data-driven optimizations are implemented at the data construction level, and the prediction model is improved at the algorithm design level.

[0029] Specifically, at the data construction level, this solution proposes a depth annotation method that integrates Neural Radiance Fields (NeRF) multi-view 3D reconstruction with motion compensation mechanisms. It utilizes state-of-the-art 3D reconstruction methods to perform dense multi-view 3D reconstruction and adaptively obtains the best depth information from each angle of the reconstructed scene. Through confidence map preprocessing and optical flow-guided consistency mask optimization, depth error correction for dynamic object regions is achieved, resulting in reliable ground truth depth values. Based on this, the DL3DV-Depth dataset, containing approximately 1.9 million frames of high-precision depth annotations, is constructed from 1000 scene videos selected from the DL3DV-10K dataset. Its scene diversity and motion complexity surpass existing datasets.

[0030] At the algorithm design level, this solution addresses the shortcomings of the DepthAnything model in temporal continuity by proposing a multi-layered spatiotemporal consistency enhancement strategy from three perspectives, resulting in a spatiotemporal consistent DepthAnything model (TC-DepthAnything): First, from a data-driven optimization perspective, highly dynamic data sequences rich in motion and scene changes are selected based on the DL3DV-Depth dataset, and spatiotemporal joint training samples are constructed using data augmentation techniques. Next, improvements are made through hierarchical temporal modeling, embedding learnable temporal position encoding into the model architecture and capturing inter-frame motion patterns through explicit spatiotemporal attention to learn temporal features. Finally, a composite loss function is used for constraints, fusing optical flow projection loss, implicit temporal regularization terms, and planar normal vector geometric constraints to ensure the consistency of dynamic target edges and maintain the smoothness of scene structure. The training hyperparameters and loss function are optimized by adding optical flow loss to promote temporal consistency. Implicit temporal regularization loss is added to constrain adjacent frames in the temporal domain. By introducing planar normals to the scene, maintaining the flatness of the ground and walls, and optimizing the geometric scene, better depth estimation results can be achieved.

[0031] refer to Figure 1 This paper proposes a video depth annotation method for open scenes that integrates multi-view 3D reconstruction of neural radiation fields with motion compensation mechanisms. The overall algorithm framework is as follows: Figure 1 As shown, the process can be mainly divided into four stages: data preprocessing, 3D reconstruction, data postprocessing, and depth video generation. Based on this, the annotation method constructs the DL3DV-Depth dataset by selecting 1000 scene videos from the DL3DV-10K dataset. This dataset contains approximately 1.9 million high-precision depth-annotated images, obtaining large-scale scene-level depth annotation ground truth values ​​that combine scene diversity and data reliability, providing a data foundation for subsequent model training in this scheme.

[0032] In the above steps, the annotation process for the scene video samples to obtain a depth-annotated video dataset includes: Multiple scene video samples were selected from the DL3DV-10K dataset as input. Frame sampling was performed on the input scene video samples to construct a set of video frame image sequences, and the camera transformation matrix corresponding to each frame image was obtained synchronously. The image sequence set is sparsely reconstructed according to the COLMAP standard format. The reconstruction process is performed using confidence-weighted constraint volume rendering to obtain a 3D scene containing depth information at each location and output the depth map of the 3D scene from each viewpoint. By analyzing the optical flow motion patterns between consecutive frames, the dense optical flow field between adjacent frames is calculated. Based on the optical flow amplitude thresholding, static backgrounds and dynamic regions are identified, a consistency mask is generated, and the error of the dynamic region in the depth map is repaired based on the consistency mask. The repaired depth map is stitched frame by frame to obtain a ground truth depth video. The view is then expanded by interpolation using the camera according to the required frame rate to obtain a depth-annotated video dataset.

[0033] The process involves frame sampling of the input scene video to construct a video frame image sequence set, and synchronously acquiring the camera transformation matrix corresponding to each frame image. This matrix represents the rotation and translation information of the image relative to the world coordinate system.

[0034] This calibration method uniformly reconstructs image sequences sparsely according to the COLMAP standard format. COLMAP is a powerful 3D reconstruction tool, and its feature matching function is crucial for generating accurate 3D models. In the feature extraction stage, COLMAP uses a scale-invariant feature transformation algorithm to extract scale- and rotation-invariant keypoints from the image, calculating a descriptor for each keypoint to measure the similarity between two feature points. (1) in, These are the feature descriptors in image i and image j, respectively. It is the Euclidean distance between these two feature descriptors. The smaller the descriptor distance, the more likely the feature points are to match.

[0035] To improve the accuracy and robustness of the matching, COLMAP further employs a random sampling consensus algorithm for geometric consistency verification. This is achieved by estimating the fundamental matrix. Constrained matching point pairs The geometric relationships, and the relationship between the fundamental matrix and the matching points, are as follows: (2) Let the error of the matching point be... Then the error metric for matching point pairs can be expressed as: (3) If the error is less than a preset threshold, it is determined to be an interior point, meaning the match is considered valid. After iterative optimization, high-confidence matches that satisfy geometric consistency are retained, significantly improving cross-view matching accuracy.

[0036] Based on feature matching results, COLMAP employs an incremental motion structure recovery framework to recover the sparse structure of a scene. First, it extracts and matches image features to calculate the 3D points and camera pose of the initial image pairs. As more images are added, new 3D points are estimated using triangulation techniques, and the camera pose is continuously optimized. To improve accuracy, COLMAP uses bundle adjustment to globally optimize camera parameters and 3D points, thereby minimizing reprojection errors and generating a more accurate 3D model. (4) in, For camera projection function, For Huber robust kernel function, Let k be the k-th three-dimensional point.

[0037] Through this process, high-precision sparse 3D point clouds and camera intrinsic parameters are output after optimization. and camera external parameters These results can serve as the basis for subsequent processing.

[0038] To eliminate lens distortion (such as radial distortion) With tangential distortion The impact on reconstruction: Before performing subsequent dense 3D reconstruction, image distortion correction is necessary. Distortion is usually caused by non-ideal characteristics of the camera lens, which can cause straight lines in the image to become curved, thus affecting the accuracy of 3D reconstruction. COLMAP performs geometric correction on the original image based on calibration parameters: (5) The corrected distortion-free image is then optimized using a combination of photometric and geometric consistency constraints to restore the undistorted image, ensuring the accuracy of subsequent 3D scene reconstruction. This preprocessing yields a high-precision sparse scene structure and a distortion-free image sequence. The sparse reconstruction results from COLMAP are then used to obtain the camera's intrinsic and extrinsic parameter matrices and sparse point clouds, which serve as geometric priors for subsequent dense scene reconstruction.

[0039] In the basic NeRF framework, all observation data are equally weighted for volume rendering optimization, ignoring the differences in multi-view coverage across different spatial regions. However, in real-world 3D reconstruction scenarios, due to camera trajectories or occlusion, some regions may only be observed by a few views, leading to noise or over-smoothing in the reconstruction results. To further improve reconstruction efficiency and accuracy, this proposal suggests a confidence map-guided neural radiation field reconstruction method, which uses confidence weighting to constrain the volume rendering reconstruction process. Specifically, the confidence score of a point is constructed using the number of cameras observed and the total number of cameras in the observation space. The confidence score is positively correlated with the number of cameras observed. This defines the spatial point... confidence level : (6) in, Representing a spatial point The number of times the camera observes the data during the sparse reconstruction phase. This represents the total number of cameras in the scene. To better utilize confidence levels to optimize the NeRF volume rendering process, the method adjusted two main components: color contribution and transmissivity.

[0040] (7) (8) (9) Equation (7) indicates that the low-confidence region is more transparent, the smaller the effective density at that point, and the easier it is for light to pass through that point. Equations (8) and (9) indicate that confidence affects both geometry and color, and the probability of light termination and color contribution of low-confidence points are doubly suppressed.

[0041] During the initialization phase, the camera parameters provided by COLMAP provide NeRF with initial viewpoint information, avoiding the problem that NeRF needs to optimize the camera pose from scratch. During the training process, NeRF can further optimize the camera parameters to compensate for possible estimation errors of COLMAP and improve the consistency of multi-viewpoints. Then, the confidence weighting of formula (9) is used to correct the volume rendering equation. For different indoor and outdoor scene inputs, the results obtained by different 3D reconstruction methods are also different. The annotation method introduces four of the best current 3D reconstruction methods. For the scene reconstructed by different methods, the PSNR and SSIM indices of the scene reconstruction are calculated, and the best result is selected and retained. After obtaining the dense reconstruction of the scene, the depth map of each viewpoint of the scene can be output through the camera pose to complete the depth data acquisition.

[0042] In dynamic scenes, the depth map obtained through 3D reconstruction inevitably suffers from poor reconstruction quality in areas that are out of camera range or difficult to capture. Geometric inconsistencies caused by inter-frame motion significantly reduce the accuracy of the depth map. To address this issue, this solution further employs a temporal consistency mask generation method based on optical flow estimation. By analyzing the optical flow motion patterns between consecutive frames, it detects and repairs errors in dynamic regions and details within the depth map. The main method includes the following core steps: calculating the dense optical flow field between adjacent frames using optical flow estimation, identifying static backgrounds and dynamic regions based on optical flow amplitude thresholding, generating a consistency mask, and using the mask to repair the depth map.

[0043] Firstly, regarding the optical flow algorithm, to ensure the overall annotation method can handle large-scale scenes quickly, this study employs the Farneback dense optical flow algorithm to calculate the optical flow between adjacent frames. sports fields between This algorithm, based on a polynomial expansion model, achieves sub-pixel-level motion estimation through second-order polynomial fitting of local pixel neighborhoods. Its core advantage lies in its robustness to illumination variations and noise. It leverages the horizontal component of the optical flow field... With vertical component Calculate the optical flow amplitude: (10) Set different thresholds for different indoor and outdoor scenarios. Obtain the consistency mask: (11) The original depth map is optimized based on a consistent mask. Depth values ​​outside the checksum are marked as invalid. The Telea algorithm is then applied to invalid regions, using texture and structural information from known regions outside the mask to fill the missing areas. The depth values ​​at the mask locations are filled with information from these known regions. This yields high-quality, reliable depth information.

[0044] After obtaining high-quality depth information from various perspectives of the reconstructed scene, reliable ground-proof depth videos are stitched together frame by frame. Depending on the required frame rate, the view can be expanded using camera interpolation to obtain scene-level high-precision depth-annotated videos. Applying the annotation method proposed in this scheme, 1000 scene depth videos were obtained and the DL3DV-Depth dataset was constructed, resulting in approximately 1.9 million frames of high-precision depth annotations.

[0045] refer to Figure 2 This solution addresses the shortcomings of current DepthAnything models with strong generalization capabilities in terms of temporal continuity. It proposes a multi-level temporal consistency enhancement strategy from three levels, innovatively introducing a temporal consistency-enhanced DepthAnything model (TC-DepthAnything). Its overall framework is as follows: Figure 2 .

[0046] First, from the perspective of data-driven optimization, we selected highly dynamic data sequences rich in motion and scene changes based on the DL3DV-Depth and TartanAir datasets. Combined with data augmentation techniques, we constructed spatiotemporal joint training samples to improve the robustness of the model in dynamic scenes.

[0047] Secondly, improvements are made to hierarchical temporal modeling. The basic DepthAnything model uses DinoV2 as the encoder and DPT as the decoder to obtain robust single-image depth estimation. The DinoV2 encoder is implemented based on the VisionTransformer (ViT) architecture, so learnable temporal position encoding can be embedded in the multi-layer attention mechanism of ViT. By capturing inter-frame motion patterns through an explicit spatiotemporal attention mechanism, temporal features can be effectively learned.

[0048] Finally, a composite loss function is used for constraints, integrating optical flow projection loss, implicit temporal regularization, and geometric constraints from plane normals. These constraints ensure edge consistency of dynamic targets and maintain the smoothness of scene structure. By optimizing training hyperparameters and the loss function, optical flow loss is added to promote temporal consistency, while implicit temporal regularization loss is introduced to constrain the temporal relationship between adjacent frames. Furthermore, by introducing plane normals of the scene to maintain the flatness of the ground and walls, the geometric scene is further optimized, resulting in better depth estimation performance.

[0049] As above Figure 3 As shown, the proposed TC-DepthAnything improves upon the encoder part of DepthAnything by explicitly embedding temporal information in the multi-head attention part and replacing the original attention layer with a constructed video attention layer, enabling the model to learn the temporal features between the input data.

[0050] Specifically, the DepthAnything model uses DinoV2 as the encoder and DPT as the decoder; learnable temporal position encoding is embedded in the DinoV2 encoder to capture motion image changes between adjacent video frames through an explicit spatiotemporal attention mechanism, and to learn the temporal features of the deep video used for training.

[0051] To enable the model to learn not only spatial attention between pixel blocks in a single frame of an image, but also temporal attention between different frames of a video, the input data needs to be adjusted to achieve temporal embedding. Assume the input is... Flatten the input data (image frames) to obtain After word embedding, we get ,in, , s This means the patch size is set to 14. D It is the feature dimension of each token (word), obtained after positional encoding. The input data for subsequent Transformer modules, obtained by adding the results of word embedding and positional encoding, can be represented as follows: .

[0052] A spatiotemporal multi-head attention layer framework, such as Figure 4 As shown, the scaled dot product attention layer is as follows: Figure 5 As shown.

[0053] Before entering the spatio-temporal multihead self-attention (STMSA) layer of each Transformer module, a layer normalization operation is performed first. After obtaining the output, it goes through a second-stage layer normalization operation and a multilayer perceptron. Each multilayer perceptron contains two linear mapping layers and a GELU activation function.

[0054] In this scheme, TC-DepthAnything is an improvement on the DepthAnythingV2-vitl model, with 28 layers of video transformer modules, and each module contains 16 output heads. For the... The videotransformer module, the specific process is as follows: (12) (13) The spatiotemporal combined multi-head attention layer aims to capture long-range interactions of tokens in spatiotemporal space. Given the input sequence for the STMSA layer, attention weights are calculated for each token using different headers. For each header, the spatiotemporal attention weights are calculated as follows: (14) in, , After normalization linear projection and , Indicates every b Insert a timing attention embedding into each block.

[0055] Spatiotemporal attention consists of data terms and residual terms. The former can be viewed as the original spatial self-attention computation, while the latter is a residual function incorporating temporal embedding. By using the residual form, the information of the original pre-trained model can be preserved to the greatest extent.

[0056] To generate the temporal embedding for each head The following methods are used: (1) For a bias vector Parameterize the model and initialize it to all zeros. The purpose of this is to initialize the bias to zero so that it does not affect the embedding in the initial state, but can be gradually learned to a suitable value during training, thereby fine-tuning the model.

[0057] (2) Calculate the relative position index along the time axis This index represents the relative positional relationship between various time steps in the time dimension, and these index values ​​are located at... Then offset the index value to To ensure that relative position indexing starts from 0, and then is based on relative position... index from Get the value and obtain the time embedding. The introduction of this relative position index helps the model capture the dynamic relationships between different time steps in the sequence and encodes the time position, which explicitly simulates the temporal ordering of the input frames, thus better integrating temporal information into the final embedding.

[0058] (3) The output of the head is connected with the linear projection to obtain the final spatiotemporal combined multi-head attention layer output as follows: (15) in, h Indicates the first h Size, .

[0059] During model training, to better address the spatial loss of prediction results, this scheme introduces a weighted affine-invariant loss, a multi-scale gradient matching loss, and a direction-magnitude normal loss to form a spatial consistency loss. (16) In the formula, The weight parameters are manually set to 0.5 in this scheme. Affine invariant loss is used to handle the ambiguity of scale and offset in the training data, multi-scale gradient matching loss is used to enhance the smoothness of gradients and sharp discontinuities in the predicted depth map, and orientation-magnitude normal loss optimizes the geometric scene from both the magnitude and angle aspects by independently optimizing the orientation angle and magnitude of the surface normal vector, thereby enhancing the geometric continuity of object edges.

[0060] To address the blurring issues caused by scale and offset, z-score normalization is used to align the predicted and ground truth results. Given a predicted depth map... and the corresponding truth results We calculate the scale and offset using the following formula: (17) in, K Indicates the number of valid pixels H is the image length, and W is the image width. Align the predicted video depth with the corresponding ground truth value: (18) Given the aligned predicted depth and the ground truth depth, the affine invariant loss can be calculated as follows: (19) Multi-scale gradient matching loss is added as an additional term to the spatial loss to improve the smoothness of uniform regions and the sharpness of discontinuous regions: (20) in, , This represents the difference between the predicted and actual values ​​under the S scale, where S is set to 4.

[0061] The normal vector of a depth map is related to the spatial variation of the depth map. At each pixel location in the image, the normal vector is calculated based on the change in depth values ​​of neighboring pixels at that location, by calculating the depth map in the horizontal direction. x Axis and vertical direction y The gradient of the axis is used to obtain the two components of the normal vector.

[0062] (twenty one) With the gradients in each direction, the normal vector is: (twenty two) The negative sign is used to ensure that the direction of the normal vector is consistent with the direction of the surface normal in the depth map.

[0063] To ensure consistent scaling, the normal vectors are normalized to obtain the normal vector map corresponding to the depth map: (twenty three) With the normal vector map, to further enhance the model's prediction accuracy in space, a Directional Magnitude Normal Loss is proposed to constrain the normal vectors. By independently optimizing the direction angle and magnitude of the surface normal vectors, the geometric continuity of the object's edges is enhanced. (twenty four) in, These are the weight coefficients for cosine similarity and L1 loss, respectively, controlling the relative importance of the two loss terms during training. Set to 1, Set it to 0.5. Denotes the Euclidean norm. These are the normal vectors corresponding to the predicted depth map and the ground truth depth value, respectively.

[0064] The time regularization loss is mainly constrained by the optical flow variation obtained offline from the optical flow estimation model and the preceding and following term masks.

[0065] refer to Figure 2 Assuming the input consists of three consecutive video frames GMflow, an optical flow estimation model

[80] The forward optical flow corresponding to the generated video frame is Reverse optical flow is The forward optical flow mask is The reverse optical flow mask is .

[0066] First obtain The mask of time, Determined by the intersection of the bidirectional masks: (25) Through forward optical flow Depth map at time step mapped to The depth map and backward optical flow at each moment will Depth map at time step mapped to The depth map at time t is shown in the following formula: (26) (27) In the formula, The process of representing the deformation layer, represent Time and Prediction depth after time-normalization.

[0067] The overall time-domain regularization loss can be expressed by the following formula: (28) in, , This represents the number of valid pixels in the mask, t is the time, T is the maximum time, and T-2 is the maximum time minus two frames. for Prediction depth after time-normalization for A two-way mask for each moment, where k is the position of a pixel in the image.

[0068] The overall training loss function can be expressed as follows: (29) After training is complete, the model's performance needs to be validated to obtain the final target TC-DepthAnything model. The target TC-DepthAnything model trained through the above steps can be applied to depth estimation for any video.

[0069] To verify the effectiveness of the obtained TC-DepthAnything model, qualitative and quantitative experiments will be conducted based on the NYUDv2, Sintel, and KITTI datasets. These three datasets cover real-world image sequences and synthetic video clips, ranging from static indoor scenes to dynamic outdoor traffic scenes, and from real-world to synthetic scenes, validating the algorithm from various perspectives. It is important to note that data from these three datasets were not used during model training; therefore, the model will undergo zero-shot cross-dataset performance evaluation on these three datasets to verify the model's spatial accuracy and temporal consistency.

[0070] Specifically, this embodiment selects several representative deep learning-based algorithms, including the supervised learning method FMNet; the self-supervised learning methods SC-DepthV1, SC-DepthV2, and SC-DepthV3; the test-time training method Robust-CVD; the depth prediction method DPT based on the ViT framework; the Transformer adapter ViTA with temporal embedding added to the attention module of DPT; and the optical flow-guided consistency fusion method NVDS+. The method of this invention uses the depth_anything-v2_vitl model as the baseline model; therefore, for methods based on the ViT framework with multiple pre-trained models, this invention selects vitl as the corresponding test model.

[0071] The results of the qualitative experiment are as follows Figure 6 , Figure 7 and Figure 8 As shown, the results of the method of this embodiment of the invention and other methods with relatively better performance are compared on three datasets. The parts marked in the boxes prove that the model of this invention can recover a globally scale-consistent depth map with fine-grained details, and can also predict the corresponding depth well for weak texture regions such as smooth walls.

[0072] On three test datasets, the proposed target TC-DepthAnything model outperforms the current state-of-the-art video depth estimation methods in terms of spatial accuracy. It can effectively address the shortcomings of ViTA and NVDS models in terms of insufficient depth estimation capabilities at detailed locations and restore a globally consistent depth scene.

[0073] To further investigate the performance of these methods in the time dimension, such as Figure 9 As shown, the visualization results of four consecutive frames and corresponding XT slices of different methods are displayed, mainly showing the methods with better temporal consistency and their corresponding basic models.

[0074] Figure 9 The first to fourth columns show the depth estimation maps for different depth estimation methods at corresponding time points. For different depth estimation results, after standardizing the resolution, the same row is used to plot scan lines to compare the temporal consistency of each method. The scan lines are in... Figure 9 The first column highlights the points indicated by horizontal lines, and the final result is shown in the last column, the XT map. The fewer the zigzag patterns in the XT map, the better the temporal consistency of the model. Compared with other methods, the depth map predicted by TC-DepthAnything is more temporally consistent. Moreover, compared with methods based on DPT, it can also successfully predict objects in scene details spatially, achieving more refined depth estimation results.

[0075] Quantitative analysis compares the performance of our algorithm with some of the best current methods under the same experimental conditions. Qualitative analysis, on the other hand, evaluates the algorithm's effectiveness by visualizing the depth estimation output, leveraging the observer's intuitive perception of the depth estimation effect in a scene. This comparison method is more intuitive and easier to understand, providing a more direct demonstration of the algorithm's performance. For ease of understanding, the table below uses bold black numbers to represent the best quantitative results for the current evaluation metrics, and underlines to represent the second-best results. Furthermore, "↑" next to the performance metrics indicates that a larger value indicates better performance, and "↓" indicates that a smaller value indicates better performance.

[0076] Table 1 Results of quantitative comparative analysis

[0077] Analyzing the results in the table above, the supervised learning-based method FMNet highly relies on the diversity and quality of the deep ground truth graphs. Given the limited diversity of training data, this model cannot adequately meet the requirements of generalization and performance in intelligent transportation scenarios.

[0078] The SC-Depth series of methods based on self-supervised learning have achieved promising results on specific datasets. However, these methods are typically trained on specific indoor or outdoor scenes. Their performance degrades significantly when applied to more general scenarios. While SC-DepthV2 and SC-DepthV3 incorporate smoothing constraints into their loss functions, which enhances the temporal consistency of the predicted depth map to some extent, excessive temporal smoothing can lead to a degradation in spatial depth prediction capabilities. Therefore, when measuring temporal smoothness using the OPW metric, spatial prediction metrics such as absolute relative error and threshold measures should be considered comprehensively.

[0079] Robust-CVD, a test-time training method, achieves geometrically and temporally consistent depth maps by fine-tuning the network for each video during the inference phase. While it achieves better performance in spatiotemporal metrics compared to previous methods, it exhibits noticeable artifacts during moving object prediction, and optical flow estimation and pose optimization incur significant computational costs when applied to depth estimation for long videos.

[0080] The DPT model, a depth prediction model based on the ViT architecture, achieves excellent results in terms of absolute relative error and threshold metrics. It utilizes the Transformer's attention mechanism to obtain the global receptive field, resulting in outstanding depth estimation capabilities for single images. However, it exhibits poor inter-frame temporal consistency on the OPW metric. This can lead to flickering artifacts in predictions when applied to scenarios requiring real-time depth assessment of video, such as intelligent transportation systems, thus impacting subsequent downstream tasks. Of particular note is the DPT model's use of the Sintel dataset as part of its training data. While it achieves excellent results on the AbsRel metric on the Sintel dataset, its performance on temporal consistency is very poor.

[0081] Both ViTA and NVDS+ methods start with optical flow information, utilizing forward and backward optical flow to guide the forward and backward processes of depth estimation, achieving better performance than the aforementioned methods. ViTA and NVDS+ use DPT as their base model, effectively addressing the issue of poor inter-frame temporal consistency in DPT. Therefore, in experiments on the Sintel dataset, NVDS+ achieved the best performance on the AbsRel metric. On the KITTI dataset, the NVDS+ method was fine-tuned for this scene, thus achieving the best spatial accuracy on the KITTI data. However, from the above... Figure 6-8The qualitative results show that ViTA loses some detailed information about moving objects in the video, such as a moving person's hand or a distant street lamp. NVDS+, on the other hand, relies on optical flow information for consistent fusion, so it may produce some minor errors in areas with poor lighting conditions, such as blurred outlines of dogs at night, depth errors in transparent windows, and incomplete spires under excessive lighting.

[0082] The underlying model used in this invention, DepthAnything-v2, is a highly generalizable model trained on large-scale unlabeled data, achieving good depth estimation results across multiple datasets. However, due to the lack of direct training on the Sintel dataset, it is relatively weaker than the DPT model in the AbsRel metric. It is worth noting, however, that the DepthAnything-v2 model achieved excellent results in threshold accuracy, indicating that due to zero-sample testing, its error estimation bias for the true depth values ​​on the Sintel dataset is larger than that of DPT. Nevertheless, it performs better in terms of relative consistency in depth estimation, meaning it can better capture proportional changes in depth and thus better estimate details in the scene.

[0083] In summary, on the NYUDv2 dataset, the target TC-DepthAnything model obtained in this embodiment achieves the best performance in both spatial and temporal metrics. Figure 7 and Figure 9 The visualization results show that on the Sintel dataset, TC-DepthAnything achieves improved performance in depth estimation across both spatiotemporal metrics, while retaining the advantages of the base model in relative depth estimation and estimation of scene details. This invention effectively addresses the shortcomings of ViTA and NVDS models in depth estimation at detailed locations, restoring a globally consistent depth scene.

[0084] In this embodiment of the invention, addressing the problem that the lack of large-scale labeled depth video data severely restricts the generalization ability of depth estimation models in various open environments, a depth video annotation method for open scenes is proposed, which integrates multi-view 3D reconstruction of neural radiation fields and motion compensation mechanisms. Through confidence map preprocessing and optical flow-guided consistency mask optimization, depth error correction for dynamic object regions is achieved, and a DL3DV-Depth dataset covering diverse indoor and outdoor scenes is constructed. With data support, to address the insufficient modeling of temporal continuity in the current basic model DepthAnything, a TC-DepthAnything model with enhanced spatiotemporal consistency is further proposed. Through spatiotemporal joint training sample construction, explicit spatiotemporal attention mechanism, and composite loss function optimization, the temporal jitter problem in video depth estimation of DepthAnything is solved, flicker artifacts and motion blur in dynamic scenes are suppressed, and accurate and efficient depth estimation applicable to videos in any scene is achieved.

[0085] The video depth estimation implemented in this solution is a core technology in the field of environmental perception for autonomous systems. Its value lies in providing 3D environmental cognition capabilities for various autonomous devices by establishing the spatial distance relationships between points in the scene and the camera equipment. By reconstructing 3D coordinate information through depth computing algorithms, it directly supports fields such as autonomous driving, robotics, 3D reconstruction, and unmanned navigation systems. Furthermore, it can be combined with other computer vision tasks as an auxiliary technology to achieve functions such as vehicle and pedestrian target detection and semantic segmentation.

[0086] Example 2 The present invention also provides a depth estimation apparatus for arbitrary videos, comprising: The sample data processing module is used to annotate the depth of scene video samples to obtain a depth-annotated video dataset; and to filter video data containing dynamic targets or dynamic scenes from the depth-annotated video dataset and the TartanAir dataset, and to perform data augmentation processing to obtain spatiotemporal joint training samples. The model building module is used to perform temporal embedding in the multi-head attention layer of the encoder of the DepthAnything model to obtain a spatiotemporal combined multi-head attention layer and build the initial TC-DepthAnything model. The model training module is used to train the initial TC-DepthAnything model using the spatiotemporal joint training samples. During the training process, a global training loss function consisting of spatial consistency loss and temporal regularization loss is used for constraints to obtain the target TC-DepthAnything model. The spatial consistency loss includes affine invariance loss, multi-scale gradient matching loss, and direction-magnitude normal loss. The temporal regularization loss is a constraint on the changes in optical flow in consecutive frames of the predicted depth map obtained by the optical flow estimation model based on the forward / backward optical flow and the forward / backward masks. The prediction module is used to input any video into the target TC-DepthAnything model to obtain a predicted depth video.

[0087] The depth estimation device for arbitrary videos provided in this embodiment of the invention is used to execute the depth estimation method for arbitrary videos in Embodiment 2, and has the same beneficial effects, which will not be repeated here.

[0088] Example 3 The present invention also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the depth estimation method for arbitrary video as described in Embodiment 1.

[0089] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A depth estimation method for arbitrary videos, characterized in that, include: The depth of the scene video samples is labeled to obtain a depth-labeled video dataset; Video data containing dynamic targets or dynamic scenes are selected from the deep-annotated video dataset and the TartanAir dataset, and data augmentation processing is performed to obtain spatiotemporal joint training samples. Temporal embedding is performed on the multi-head attention layer in the encoder of the DepthAnything model to obtain a spatiotemporal combined multi-head attention layer, and the initial TC-DepthAnything model is constructed. The initial TC-DepthAnything model is trained using the spatiotemporal joint training samples. During training, a global training loss function consisting of spatial consistency loss and temporal regularization loss is used for constraint to obtain the target TC-DepthAnything model. The spatial consistency loss includes affine invariance loss, multi-scale gradient matching loss, and direction-amplitude normal loss. The temporal regularization loss is a constraint on the changes in optical flow across consecutive frames of the predicted depth map obtained by the optical flow estimation model, based on the forward / backward optical flow and the forward / backward masks. Input any video into the target TC-DepthAnything model to obtain the predicted depth video.

2. The depth estimation method as described in claim 1, characterized in that, The DepthAnything model uses DinoV2 as the encoder and DPT as the decoder. Learnable temporal position codes are embedded in the DinoV2 encoder to capture motion image changes between adjacent video frames through an explicit spatiotemporal attention mechanism, and to learn the temporal features of the deep video for training.

3. The depth estimation method as described in claim 2, characterized in that, The method of capturing motion image changes between adjacent video frames through an explicit spatiotemporal attention mechanism includes: The input image frame is flattened, and then word vectors are obtained by combining the word embedding and position encoding methods. The word vectors are subjected to a layer normalization operation, and then input into the spatiotemporal combined multi-head attention layer of the Transformer module for temporal embedding. The output of the spatiotemporal combined multi-head attention layer is then subjected to layer normalization and a multilayer perceptron to obtain the final output.

4. The depth estimation method as described in claim 1, characterized in that, The expression for the affine invariant loss is: Where K represents the number of effective pixels. H is the length of the image, and W is the width of the image. , For the predicted depth map, For the corresponding truth value result; The expression for the multi-scale gradient matching loss is: in, , This represents the difference between the predicted and actual values ​​under the S-scale. The expression for the direction-amplitude normal loss is: in, These are the weighting coefficients for cosine similarity and L1 loss, respectively. Denotes the Euclidean norm. These are the normal vectors corresponding to the predicted depth map and the ground truth depth value, respectively. ; The formula for the spatial consistency loss is: in, The weight parameters are set.

5. The depth estimation method as described in claim 1, characterized in that, The temporal regularization loss is used to constrain the temporal relationship between adjacent frames; The expression for the temporal regularization loss is: in, , This represents the number of valid pixels in the mask, t is the time, T is the maximum time, and T-2 is the maximum time minus two frames. for Prediction depth after time-normalization for Two-way mask for time, ; For forward optical flow Depth map at time step mapped to Depth map at any given time. For backward optical flow Depth map at time step mapped to The depth map at time step, where k is the position of a pixel in the image.

6. The depth estimation method as described in claim 1, characterized in that, The training process for the initial TC-DepthAnything model also includes: The NYUDv2, Sintel, and KITTI datasets were input into the trained TC-DepthAnything model for qualitative and quantitative validation.

7. The depth estimation method as described in claim 1, characterized in that, The depth annotation process for the scene video samples yields a depth-annotated video dataset, including: Multiple scene video samples were selected from the DL3DV-10K dataset as input. Frame sampling was performed on the input scene video samples to construct a set of video frame image sequences, and the camera transformation matrix corresponding to each frame image was obtained synchronously. The image sequence set is sparsely reconstructed according to the COLMAP standard format. The reconstruction process is performed using confidence-weighted constraint volume rendering to obtain a 3D scene containing depth information at each location and output the depth map of the 3D scene from each viewpoint. By analyzing the optical flow motion patterns between consecutive frames, the dense optical flow field between adjacent frames is calculated. Based on the optical flow amplitude thresholding, static backgrounds and dynamic regions are identified, a consistency mask is generated, and the error of the dynamic region in the depth map is repaired based on the consistency mask. The repaired depth map is stitched frame by frame to obtain a ground truth depth video. The view is then expanded by interpolation using the camera according to the required frame rate to obtain a depth-annotated video dataset.

8. A depth estimation device for arbitrary videos, characterized in that, include: The sample data processing module is used to annotate the depth of scene video samples to obtain a depth-annotated video dataset. Video data containing dynamic targets or dynamic scenes are selected from the deep-annotated video dataset and the TartanAir dataset, and data augmentation processing is performed to obtain spatiotemporal joint training samples. The model building module is used to perform temporal embedding in the multi-head attention layer of the encoder of the DepthAnything model to obtain a spatiotemporal combined multi-head attention layer and build the initial TC-DepthAnything model. The model training module is used to train the initial TC-DepthAnything model using the spatiotemporal joint training samples. During the training process, a global training loss function consisting of spatial consistency loss and temporal regularization loss is used for constraints to obtain the target TC-DepthAnything model. The spatial consistency loss includes affine invariance loss, multi-scale gradient matching loss, and direction-magnitude normal loss. The temporal regularization loss is a constraint on the changes in optical flow in consecutive frames of the predicted depth map obtained by the optical flow estimation model based on the forward / backward optical flow and the forward / backward masks. The prediction module is used to input any video into the target TC-DepthAnything model to obtain a predicted depth video.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the depth estimation method for arbitrary video as described in any one of claims 1 to 8.

Images