Time consistency video depth estimation system based on data driving method

Abstract

The invention discloses a time consistency video depth estimation system based on a data driving method. The time consistency video depth estimation system comprises a depth map sequence generation module, a depth prediction module and a time consistency module. A reliable video depth sequence can be generated in the absence of a high-quality training data set, and the time consistency and precision of depth estimation are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and specifically relates to a time-consistent video depth estimation system based on a data-driven method. BACKGROUND

[0002] Monocular depth estimation aims to infer the depth information of a three-dimensional scene from two-dimensional images. However, when dealing with dynamic scene videos, existing depth estimation methods face serious problems of time consistency and accuracy. Traditional methods rely on camera pose and optical flow information in the image processing stage, which is not only time-consuming but also difficult to adapt to changing environments. At the same time, due to the lack of sufficient high-quality depth datasets, especially for videos under low-light and complex lighting conditions, existing methods have many limitations in practical applications.

[0003] With the increasing demand for high-precision depth information in application scenarios such as autonomous driving, augmented reality, and robot navigation, it is crucial to develop a depth estimation system that can maintain time consistency and accuracy in these complex scenarios. Although existing research has explored the use of synthetic datasets and machine learning methods to improve the accuracy of depth estimation, these methods often cannot effectively handle depth prediction under complex lighting changes and high-noise environments. SUMMARY

[0004] The present application proposes a time-consistent video depth estimation system based on a data-driven method, aiming to solve the problems of inconsistent depth prediction and insufficient accuracy in traditional methods in dynamic scenes.

[0005] A time-consistent video depth estimation system based on a data-driven method includes a depth map sequence generation module, a depth prediction module DPM, and a time consistency module TC.

[0006] The depth map sequence generation module is used to generate training set video depth data.

[0007] The DPM is a single-frame depth prediction module that processes input RGB images to generate corresponding depth maps.

[0008] The TC is an improved recurrent ResCNN network equipped with skip connections to preserve multi-layer features and capture the spatio-temporal correlation of input sequences. In the time dimension, the coherence of depth maps is optimized to ensure the generation of consistent depth map sequences in complex scenarios.

[0009] In a possible implementation, the depth map sequence generation module proposes a method for inferring scene motion from a single image and generating a time-consistent depth map sequence, including two strategies: (1) simulating the translational motion of the camera by predicting a global affine transformation; and (2) estimating the local motion of a dynamic region by predicting a sparse motion-guided optical flow. According to the two strategies, the global and local motions are comprehensively considered, so that the motion that can occur in a real scene is more accurately inferred, and a time-consistent depth map sequence is generated.

[0010] In a possible implementation, the generation of the time-consistent depth map sequence specifically includes the following steps.

[0011] First, the original image and the corresponding depth map are obtained by the depth camera. For any input original image, a global affine transformation is predicted, and a camera translation in any direction within Q pixels (Q = 8, 9, 10, 11, 12) is randomly set, that is, the entire picture is translated, and the edge pixels of the image are filled in the corresponding blank part. Then, the Detectron2 algorithm is used to calculate a binary segmentation mask S of dynamic pixels, and 15 guide motion vectors are randomly sampled from the mask. Subsequently, the vectors are input to a local optical flow prediction CMP model together with the original image. According to the predicted optical flow result, a warped image is generated by warping operation in combination with the original image. At the same time, according to the predicted optical flow result, a warped depth map is generated by warping operation in combination with the original image corresponding to the depth map:

[0012] The generated warped image is replaced by the original image to go through the above generation process again, and T pairs of continuous warped images and depth maps are generated, each image having the same camera translation pixels, and finally a depth map sequence with high time consistency is obtained.

[0013] In a possible implementation, a time consistency module TC is designed to learn the time consistency of adjacent depth maps.

[0014] The TC module inputs the unoptimized depth map into a ConvLSTM network to calculate an optimized depth map, and the ConvLSTM network shares weights when processing other depth maps. The specific optimization process is as follows: based on the optimized depth map at the previous moment and the unoptimized initial depth map generated by the DPM module for the original image, the TC module performs a depth refinement operation through the ConvLSTM network in a loop to obtain an optimized depth map at time t (t = 1, 2,..., T), which is expressed by the following formula:

[0015] (2) ​

[0016] The input of the next time step.

[0017] In one possible implementation, during the training process of the TC module, multiple loss functions are introduced to optimize the temporal consistency and accuracy of the depth map:

[0018] Min-max normalization loss : In order to make the optimized depth map close to the ground truth (GT), a depth value constraint is added between the two. Specifically, the TC module is trained in the disparity space using min-max normalization (0-1), which is defined as follows:

[0019] (3)

[0020] where, is the i-th pixel value in the normalized depth map, is the i-th pixel value in the normalized ground truth depth map, is the number of pixels whose pixel value in the residual map is less than 20% of the maximum pixel value. The normalization operation is as follows:

[0021] , (4)

[0022] where, is the i-th pixel value in the depth map; is the i-th pixel value in the ground truth depth map; and represent the mean and standard deviation of the depth map, respectively.

[0023] Temporal consistency loss : used to constrain the temporal consistency between adjacent depth maps. Since the optical flow can constrain the pixel area of adjacent RGB images, the corresponding depth values also remain consistent in the static area. Specifically, the temporal consistency loss is defined by the following formula:

[0024] (5)

[0025] where, is a weight term calculated according to the RGB image, used to measure the inter-frame similarity. is the i-th pixel optimized depth map of the m+1 frame, is the depth value obtained by GMFlow optical flow calculation.

[0026] Symmetric consistency loss : To avoid the distorted results of errors, further constrain the temporal consistency of adjacent depth maps. Specifically, the input T unoptimized depth maps are reconstructed into (2T-1) unoptimized depth maps, and the arrangement order is{ ,..., } becomes{ ,... , , ,... } Therefore, we will get{ ,... , , ,... } According to the output of the symmetric depth map is close, the symmetric consistency loss is defined by the following formula:

[0027] (5)

[0028] Wherein, is the weight term calculated according to the RGB image, which is used to measure the inter-frame similarity. is the optimized depth map of the i-th pixel of the u-th frame, is the optimized depth map obtained after reconstruction.

[0029] Total loss: Finally, the above loss functions are combined into a total loss function, which is defined as:

[0030] (6)

[0031] Wherein, is the weight factor of the temporal consistency loss and the symmetric consistency loss.

[0032] The present application has the following advantages:

[0033] Multi-model compatibility: During the training process, the DPM module of the system can adopt any single-frame depth estimation model to enhance its temporal consistency and can process video data in various real-world scenarios in real time.

[0034] Online processing capability: Compared with the existing method which needs time-consuming testing and training stage, the system can directly process video data in real-time environment, meeting the needs of practical applications.

[0035] Data generation method: In order to solve the problem of lack of temporal consistent video depth training dataset in dynamic scenes, the present application proposes a method for generating training video sequence dataset based on the inferred motion field from a single image, which provides sufficient data support for the training process. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description only represent some of the embodiments of the present application, and other embodiments can be obtained by those skilled in the art based on these drawings.

[0037] Figure 1 The structural schematic diagram of the deep cycle network of the embodiment of the present application.

[0038] Figure 2 The process schematic diagram of the video depth data generation of the embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely in the following description with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application belong to the scope of protection of the present application.

[0040] The embodiment of the present application provides a time consistency video depth estimation system based on a data-driven method, which comprises a depth map sequence generation module, a depth prediction module (DPM) and a time consistency module (TC).

[0041] The depth map sequence generation module is used to generate training set video depth data, infer scene motion from a single image, and generate a time-consistent depth map sequence.

[0042] The DPM is a single-frame depth prediction module, which is used to process the input RGB image and generate a corresponding depth map. The module can adopt any single-frame depth prediction model and does not participate in training.

[0043] The TC is an improved cycle ResCNN network, which is equipped with a skip connection to preserve multi-layer features and capture the spatio-temporal correlation of the input sequence. The main function of the TC is to optimize the continuity of the depth map in the time dimension and ensure the generation of consistent depth map sequences in complex scenes.

[0044] The embodiment of the present application also provides a time consistency video depth estimation method based on a data-driven method, which comprises the following steps:

[0045] Step (1) generating training set video depth data;

[0046] Since video depth datasets in real scenes are still scarce, to make up for this deficiency, we propose a method to infer scene motion from a single image and generate temporally consistent depth map sequences to better simulate dynamic scenes in the real world.

[0047] Specifically, this method mainly includes two strategies: (1) simulating the camera's translational motion by predicting the global affine transformation. (2) Estimating the local motion of dynamic regions by predicting the optical flow guided by sparse motion. Based on these two strategies, the global and local motions are comprehensively considered to more accurately infer the motion that may occur in real scenes and generate temporally consistent depth map sequences.

[0048] First, the original image and the corresponding depth map are obtained through the depth camera. For any input original image, the global affine transformation is predicted, and the camera translation in any direction within 10 pixels is randomly set, that is, the entire picture is translated, and the edge pixels of the image fill the corresponding blank parts. Then, the Detectron2 algorithm is used to calculate the binary segmentation mask S of the dynamic pixels, and 15 guided motion vectors are randomly sampled from it. Subsequently, these vectors are input into the local optical flow prediction CMP model together with the original image. According to the predicted optical flow results, the distorted image is generated by the distortion operation in combination with the original image. At the same time, according to the currently predicted optical flow results, the distorted depth map is generated by the distortion operation in combination with the depth map corresponding to the original image:

[0049] = W( , CMP ( ,V )) (1)

[0050] Among them, and Represent the original image and the distorted image respectively. W is the distortion operation and V is the guided motion vector.

[0051] The warped image is then used to replace the original image and run through the above generation process again, generating T (T = 10) pairs of consecutive warped images and depth maps. The camera translation for each image is the same pixel by pixel, ultimately resulting in a highly temporally consistent depth map sequence. This provides reliable training data and ground truth depth maps for the temporal consistency module. These generated depth sequences not only cover a wide range of motion patterns likely encountered in the real world, but also reduce errors introduced by repeated warping operations by guiding motion vectors.

[0052] Step (2) inputs the generated distorted image into the depth prediction module DPM to generate an unoptimized initial depth map.

[0053] The depth recurrent network used in video depth estimation is composed of a DPM module and a TC module. The DPM module can use any single-frame depth estimation model.

[0054] Step (3) learns the temporal consistency of adjacent depth maps through the temporal consistency module TC.

[0055] The temporal consistency module TC inputs the unoptimized depth map into the ConvLSTM network to calculate the optimized depth map. The ConvLSTM network shares weights when processing other depth maps. The specific optimization process is as follows: based on the optimized depth map at the previous time The DPM module generates a depth map for the original image) and the unoptimized initial depth map The TC module performs depth refinement operations through the ConvLSTM network in a loop to obtain the optimized depth map at time t (t = 1, 2,..., T), which is expressed as follows:

[0056] (2)

[0057] As the input of the next time step.

[0058] In the training process of the TC module, several loss functions are introduced to optimize the temporal consistency and accuracy of the depth map:

[0059] Min-max normalization loss : In order to make the optimized depth map close to the real depth map (GT), a depth value constraint is added between the two. Specifically, the TC module is trained in the disparity space using min-max normalization (0-1), which is defined as follows:

[0060] (3)

[0061] Where, is the i-th pixel value in the normalized depth map, is the i-th pixel value in the normalized real depth map, is the number of pixels whose pixel value in the residual map is less than 20% of the maximum pixel value. The normalization operation is as follows:

[0062] , (4)

[0063] Where, is the i-th pixel value in the depth map; is the i-th pixel value in the real depth map; and represent the mean and standard deviation of the depth map, respectively.​​

[0064] Temporal consistency loss : To constrain the temporal consistency between adjacent depth maps. Since the optical flow can constrain the pixel region of adjacent RGB images, the corresponding depth values also remain consistent in the static region. Specifically, the temporal consistency loss is defined by the following formula:

[0065] (5)

[0066] where, is the weight term calculated according to the RGB image, which is used to measure the inter-frame similarity. is the optimized depth map of the i-th pixel in the m+1-th frame, is the depth value obtained by GMFlow optical flow calculation.

[0067] Symmetric consistency loss : To avoid false distortion results and further constrain the temporal consistency of adjacent depth maps. Specifically, the input T unoptimized depth maps are reconstructed into (2T-1) unoptimized depth maps, and the arrangement order is changed from { ,... } to { ... , , ... }, so that { ... , , ... } will be obtained. According to the output of the symmetric depth map is close, the symmetric consistency loss is defined by the following formula:

[0068] (5)

[0069] where, is the weight term calculated according to the RGB image, which is used to measure the inter-frame similarity. is the optimized depth map of the i-th pixel in the u-th frame, is the optimized depth map obtained after reconstruction.

[0070] Total loss: Finally, the above loss functions are combined into a total loss function, which is defined as:

[0071] (6)

[0072] where, and weight factors for temporal consistency loss and symmetric consistency loss, respectively, in an embodiment = 0.5, = 1.

[0073] The MPI Sintel video dataset, which covers both dynamic and static scenes, was evaluated in the experiment. The predicted depth was scaled based on the median to align them with the ground truth depth in inverse depth space when the predicted depth reaches an unknown scale. The regions with depth values greater than 80 meters were removed. The final evaluation results of the present application were compared with other common video depth estimation methods, as shown in Table 1.

[0074] Table 1: Comparison results of depth estimation methods

[0075]

[0076] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one from another entity or action, without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0077] Each of the embodiments in the specification is described in a related manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts between the embodiments can be referred to each other.

[0078] The above description is merely preferred embodiments of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A temporally consistent video depth estimation system based on a data-driven approach, characterized in that: It includes a depth map sequence generation module, a depth prediction module DPM and a temporal consistency module TC; The depth map sequence generation module is used to generate training set video depth data; The DPM is a single-frame depth prediction module that processes the input RGB image to generate a corresponding depth map; The TC is an improved recurrent ResCNN network equipped with skip connections to preserve multi-layer features and capture the spatiotemporal correlation of the input sequence; it optimizes the coherence of the depth map in the temporal dimension to ensure the generation of consistent depth map sequences in complex scenarios.

2. The temporally consistent video depth estimation system based on a data-driven approach according to claim 1, characterized in that: The depth map sequence generation module proposes a method for inferring scene motion from a single image and generating a temporally consistent depth map sequence, which includes two strategies: (1) simulating the camera's translational motion by predicting the global affine transformation; (2) estimating the local motion of the dynamic area by predicting the optical flow guided by sparse motion; according to these two strategies, the global and local motions are comprehensively considered to more accurately infer the motion that may occur in the real scene and generate a temporally consistent depth map sequence.

3. The temporally consistent video depth estimation system based on a data-driven method according to claim 2, characterized in that: Generation of temporally consistent depth map sequences, specifically including: First, the original image and the corresponding depth map are obtained through the depth camera. For any input original image, the global affine transformation is predicted, and the camera translation in any direction within Q pixels is randomly set, that is, the entire image is translated, and the edge pixels of the image fill the corresponding blank parts; then the Detectron2 algorithm is used to calculate the binary segmentation mask S of the dynamic pixels, and 15 guided motion vectors are randomly sampled from it; then, these vectors are input into the local optical flow prediction CMP model together with the original image; based on the predicted optical flow results, the distorted image is generated by the warping operation in combination with the original image; at the same time, based on the current predicted optical flow results, the distorted depth map is generated by the warping operation in combination with the depth map corresponding to the original image: The generated distorted image replaces the original image and goes through the above generation process again to generate T pairs of continuous distorted images and depth maps. The camera translation pixels of each image are the same, and finally a depth map sequence with high temporal consistency is obtained.

4. The temporally consistent video depth estimation system based on a data-driven approach according to claim 3, characterized in that: Q=8，9，10，11，12。 5. The temporally consistent video depth estimation system based on a data-driven approach according to claim 3, characterized in that: The temporal consistency module TC is used to learn the temporal consistency of adjacent depth maps, including: The TC module inputs the unoptimized depth map into the ConvLSTM network to calculate the optimized depth map. The ConvLSTM network shares weights when processing other depth maps. The specific optimization process is as follows: based on the depth map optimized at the previous moment and the unoptimized initial depth map , the TC module performs depth refinement operations through the ConvLSTM network loop to obtain the optimized depth map at time t, which is expressed as follows: (2) Will As input for the next time step; is the depth map generated by the DPM module of the original image, t=1, 2, ..., T.

6. The temporally consistent video depth estimation system based on a data-driven approach according to claim 5, characterized in that: During the training of the TC module, multiple loss functions are introduced to optimize the temporal consistency and accuracy of the depth map: Min-max normalized loss : In order to make the optimized depth map close to the real depth map, a depth value constraint is added between the two; specifically, the TC module is trained in the disparity space using min-max normalization, which is defined as follows: (3) in, is the i-th pixel value in the normalized depth map, is the i-th pixel value in the normalized true depth map, It is the number of pixels in the residual image whose pixel value is less than 20% of the maximum pixel value; the normalization operation is as follows: ， (4) in, is the i-th pixel value in the depth map; is the i-th pixel value in the real depth map; and represent the mean and standard deviation of the depth map respectively; Temporal consistency loss : It is used to constrain the temporal consistency between adjacent depth maps. Since the optical flow can constrain the pixel areas of adjacent RGB images, the corresponding depth values ​​are also consistent in the static area. Specifically, the temporal consistency loss is defined by the following formula: (5) in, It is a weight term calculated based on the RGB image and is used to measure the similarity between frames; is the optimized depth map of the i-th pixel in the m+1-th frame, yes The depth value obtained by GMFlow optical flow calculation; Symmetry consistency loss : It is used to avoid erroneous distortion results and further constrain the temporal consistency of adjacent depth maps; specifically, the input T unoptimized depth maps are reconstructed into 2T-1 unoptimized depth maps, and the order is { ,..., } becomes { ,... , , ,... }, so we get { ,... , , ,... }; According to the output of the symmetric depth map is close, the symmetry consistency loss is defined by the following formula: (5) in, It is a weight term calculated based on the RGB image and is used to measure the similarity between frames; is the depth map after optimization of the i-th pixel in the u-th frame, is the optimized depth map obtained after reconstruction; Total loss: Finally, the above loss functions are combined into a total loss function, which is defined as: (6) in, is the weight factor of temporal consistency loss and symmetric consistency loss.

Images