A refinement distillation method for monocular depth estimation
By combining local and cross-context distillation methods, and integrating global depth structure and local detail information, the problem of insufficient utilization of pseudo-labels in monocular depth estimation is solved, thereby improving the model's depth estimation accuracy and generalization ability in real-world scenarios.
Patent Information
- Application Number
- CN202411800080.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-09
- Publication Date
- 2026-01-02
- Estimated Expiration
- 2044-12-09
AI Technical Summary
Existing monocular depth estimation methods struggle to effectively utilize pseudo-labels in the absence of labeled data, resulting in insufficient model generalization ability, particularly in the recovery of local details.
We employ local and cross-context distillation methods, combining global depth structure and local detail information. We guide student models to perform depth estimation using depth pseudo-labels from the teacher model. We train the teacher model using synthetic data and apply it to unlabeled real image data, thereby improving the depth estimation accuracy of the model in real-world scenarios.
It significantly improves the generalization ability and accuracy of monocular depth estimation models in real-world scenarios, especially in images with significant contextual changes, surpassing the accuracy of traditional global normalization methods.
Smart Images
Figure CN119722769B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of computer vision, in particular to the technology related to monocular depth estimation, which is suitable for the task of estimating the depth of a three-dimensional scene from a single RGB image without the need for depth sensors. Specific applications include but are not limited to the fields of autonomous driving, robot navigation, augmented reality (AR), virtual reality (VR), and map generation, etc.
[0002] BACKGROUND Monocular depth estimation is a key research direction in computer vision, aiming to predict the depth information of a scene by analyzing a single RGB image. Unlike stereo vision or multi-camera depth estimation methods, monocular depth estimation has significant advantages: it does not rely on additional depth sensors or multi-camera devices, so it has broad application prospects in practical applications, especially in the fields of autonomous driving, robot navigation, augmented reality (AR), virtual reality (VR), etc. However, although deep learning technology has made significant progress in monocular depth estimation, how to improve the accuracy and generalization ability of depth estimation in the absence of a large amount of labeled data remains a challenge to be solved.
[0003] Traditional depth estimation methods usually rely on hand-designed feature extraction and geometric analysis, and these methods have low accuracy in dealing with complex scenes. In recent years, deep learning methods based on convolutional neural networks (CNN) have gradually become mainstream. Through end-to-end training, these methods can automatically learn effective feature representations from images, thereby significantly improving the accuracy of monocular depth estimation. However, one of the main challenges of these deep learning methods is the dependence on a large amount of labeled data. The acquisition of labeled data usually requires high cost, and in some application scenarios, labeled data is difficult to obtain.
[0004] To solve this problem, researchers have proposed unsupervised learning and semi-supervised learning methods. These methods use proxy tasks or geometric constraints of images themselves to enable the model to be trained without labeled data. For example, unsupervised methods that use the geometric consistency of image pairs for training, and depth estimation methods based on self-supervised learning, have made significant progress. However, these methods often rely on a large number of image pairs or synthetic data, and still have certain limitations. Especially in the absence of labeled data, how to better use pseudo-labels to improve the generalization ability of the model remains a problem to be solved.
[0005] Distillation technology as a knowledge transfer method has been widely used in the field of deep learning in recent years. In monocular depth estimation, distillation technology enhances the learning ability of the student model by transferring knowledge from the teacher model, thereby improving its generalization ability on unlabelled data. Although existing distillation methods in the field of depth estimation have achieved certain success, most of these methods still rely on global normalization strategies (such as scale and shift invariant (SSI) depth representation), and these methods often have a problem of accuracy decline in the recovery of local details. The reason is that due to global normalization, the depth predicted by each pixel is no longer determined by the pixel itself, but is affected by all other pixels, so the position that can originally predict the correct depth will also be affected by the depth information of the extreme error position. This problem still exists when the model is distilled. Therefore, when the model is distilled, how to balance the learning of global depth structure and local details is still a problem to be explored. SUMMARY
[0006] In order to overcome the shortcomings of the prior art, the present application provides a refinement distillation method for monocular depth estimation, which solves the problems of ineffective supervision and insufficient pseudo-label utilization in the prior art by a new refinement distillation method. The core innovation of this method is that it can effectively combine global depth structure and local detail information during the distillation process, so that the student model not only relies on the global information provided by the teacher model, but also supplements and strengthens the depth estimation of local regions during the distillation process, thereby improving the depth estimation accuracy of the model.
[0007] The technical scheme adopted by the present application to solve its technical problems is:
[0008] A refinement distillation method for monocular depth estimation, comprising the following steps:
[0009] Step 1) First, train a teacher model with labelled high-precision synthetic data. The teacher model is trained to estimate the depth information of the entire image.
[0010] Step 2) Use the depth pseudo-label output by the existing teacher model as a direct supervision signal to distill the student model.
[0011] Step 3) On this basis, use the methods of local context distillation and cross-context distillation to improve the utilization of unlabelled data by the teacher model, supplement and strengthen the depth estimation of local regions during the distillation process, and thereby improve the depth estimation details and accuracy of the model.
[0012] Further, in step 1), by using synthetic data, we can obtain large-scale, high-quality annotated depth information, which comes from virtual environments (such as 3D engine simulations). Although these data are virtually synthesized, they have very dense depth annotations and are highly accurate.
[0013] Further, in step 2), first, the depth pseudo-label generated by the teacher model is applied to the unlabeled real image data, which guides the student model during the distillation process, allowing the student model to gradually learn how to handle real-world data.
[0014] Further, in step 3), the key to local context distillation is how to handle local regions in the image. The local context distillation method generates local depth maps for supervision by inputting local regions of the image into the teacher and student models. According to the input original image, a local region is randomly cropped and input into the teacher and student models, and the depth pseudo-label predicted by the teacher model is used to supervise the student model's depth estimation of the local region.
[0015] In step 3), in cross-context distillation, the teacher assistant crops the original image multiple times and predicts local depth information for each cropped region, while the student model performs global depth prediction on the entire image. During training, the local depth prediction generated by the teacher assistant and the global depth prediction of the student model differ in the overlapping area, and this difference is the local loss. By calculating these local losses, the student model's depth prediction in specific regions is further refined, allowing the student model to more accurately capture detailed local information.
[0016] In the present application, by comprehensively analyzing the effects of global normalization, local normalization, and multi-layer normalization strategies, a new depth information representation method is proposed. Unlike traditional global normalized depth representation, the method of the present application does not perform global normalization of depth information, but directly uses the depth pseudo-label output by the teacher model as a supervision signal to guide the student model to perform depth estimation. This method is not only simple and effective, but also surpasses traditional global normalization methods in accuracy. In addition, the present application also proposes local context distillation and cross-context distillation methods, which effectively improve the performance of monocular depth estimation models in real-world scenarios by improving the teacher model's utilization of unlabeled data, enhancing the model's generalization ability and accuracy. In particular, in multiple popular monocular depth estimation test benchmarks, the method of the present application has shown strong advantages.
[0017] The present application effectively makes up for the deficiency of the traditional method in detail recovery by combining local context distillation and cross-context distillation, and can more accurately estimate the depth information in the image. The teacher model is trained on synthetic data, combined with pseudo-label distillation of unannotated real image data, which further improves the generalization ability and accuracy of the model in the actual scene.
[0018] The present application has the following advantages:
[0019] 1. The present application has carried out extensive experiments on multiple popular benchmark datasets. The experimental results show that our method is significantly better than the benchmark model, creating a new state-of-the-art level. We are surprised to find that the student model only relies on unannotated data to surpass the performance of the benchmark model.
[0020] 2. The present application has carried out in-depth analysis on various depth normalization strategies, including global normalization, local normalization, mixed normalization and no normalization method. Based on these analyses, we explore a more adaptive loss strategy that is more effective than the traditional SSI loss method in the distillation process.
[0021] 3. The present application proposes a refinement distillation framework using a cross-context teacher mechanism, which operates on two scales of global depth and local region, enabling the student model to learn both macro depth structure and fine detail information, thereby improving accuracy and robustness, especially in images with significant context changes. BRIEF DESCRIPTION OF DRAWINGS
[0022] Fig. 1 The present application is a general distillation flowchart;
[0023] Fig. 2 The present application is a local context distillation flowchart;
[0024] Fig. 3 The present application is a cross-context distillation flowchart. DETAILED DESCRIPTION
[0025] The present application will be further described below with reference to the accompanying drawings.
[0026] Reference Figs. 1-3 A refinement distillation method for monocular depth estimation aims to combine global structure information and local details through multi-scale supervision by cross-context distillation and local context distillation, significantly improving the accuracy of depth estimation.
[0027] The refined distillation method comprises the following steps:
[0028] Step 1) First, a powerful teacher model is trained by a large amount of labeled high-precision synthetic data. The teacher model is fully trained and can effectively estimate the depth information of the entire image. By using synthetic data, we can obtain large-scale depth information with high-quality annotations. Synthetic data come from virtual environments (such as 3D engine simulations), which, although virtually synthesized, have very dense depth annotations and extremely high accuracy. Compared with traditional depth data relying on manual annotation (such as lidar data), synthetic data have obvious advantages in annotation quality and quantity. Artificially annotated depth data not only has high cost, but also often has poor depth quality due to measurement errors or environmental complexity. Therefore, using synthetic data to train the teacher model can not only effectively overcome the limitations of traditional annotated data, but also greatly improve the accuracy of the model in depth estimation, thereby improving the model's ability to predict depth details.
[0029] Step 2) Use the depth pseudo-label output by the teacher model as a direct supervision signal to distill the student model; First, the teacher model trained based on synthetic data has certain data distribution differences with real scene data, which leads to poor performance in actual application. In order to solve this problem, using a wide range of real unlabeled image data has become an effective solution. Although the teacher model performs well on synthetic data, its performance in real scenes is poor. In order to make the teacher model adapt to real scenes, we use the depth pseudo-label generated by the teacher model and apply it to unlabeled real image data. These pseudo-labels guide the student model during the distillation process, allowing the student model to gradually learn how to handle real-world data. Unlike traditional methods, the present invention does not use global normalization (such as scale and translation invariant loss) for depth supervision, but directly uses the depth pseudo-label output by the teacher model as a supervision signal to guide the student model to estimate depth. This method not only has a simple structure, but also performs better in accuracy than traditional methods.
[0030] Step 3) On this basis, by using the method of local context distillation and cross-context distillation, the depth estimation of local areas is supplemented and strengthened in the distillation process by improving the utilization of teacher models for unlabeled data, so as to improve the details and accuracy of depth estimation of the model. The key of the local context distillation lies in how to deal with the local area in the image. The local context distillation method generates a local depth map for supervision by inputting the local area of the image into the teacher and student models respectively. Specifically, we randomly crop a local area from the input original image and input it into the teacher and student models respectively, and supervise the depth estimation of the student model for the local area by the depth pseudo-label predicted by the teacher model. Compared with the full image input, this local area-based input can provide more detailed and detailed depth prediction, so that the depth estimation of the student model in the local area is more accurate. This method helps the student model to achieve higher accuracy in fine-grained depth estimation by improving the attention to local details.
[0031] The core idea of the cross-context distillation is to further expand the context understanding ability of the model on the basis of the local context distillation. Specifically, in the cross-context distillation, the teacher assistant will crop the original image multiple times and predict local depth information for each cropped area, while the student model predicts the global depth of the whole image. In the training process, the local depth prediction generated by the teacher assistant and the global depth prediction of the student model will produce differences in the overlapping areas. This part of the difference is the local loss. By calculating these local losses, the depth prediction of the student model in a specific area is further refined, so that the student model can more accurately capture the local information with rich details. Through cross-context distillation, the student model can not only retain the global depth structure, but also significantly improve the recovery of local details.
[0032] As Fig. 1As shown, the refined distillation framework includes a teacher model (Teacher) and a student model (Student) and a teacher assistant model (Teacher Assistant), and distills through depth information across context and local context. Specifically, for local context distillation, the same image is input into the teacher and student models, and depth maps are predicted by them respectively, and the depth map predicted by the teacher model will guide the depth map predicted by the student model. And for cross-context distillation, the present application is implemented through a teacher assistant, and the same image is input, and multiple regions are cropped from the image and input into the teacher assistant, which is responsible for predicting the local depth information of the multiple regions cropped from the input image, while the student model accepts the input of the whole image and generates a global depth map of the whole image. In the training process, the overlapping area between the local depth prediction of the teacher assistant and the global prediction of the student model is used to calculate the loss, so as to realize distillation. In this process, the depth prediction of the teacher is used as a pseudo-label to guide the student model to further refine the depth prediction in the local area, and it is expected that the depth prediction of multiple local areas can alleviate the noise caused by single local pseudo-label prediction, so as to make the supervision more robust.
[0033] As shown in Fig. 2 The method of local context distillation of the present application, according to the input original image, randomly crops a local region as the input of the model. We respectively input it into the teacher and student models to predict the depth map, and the loss function is:
[0034]
[0035] The pseudo-label predicted by the teacher Will be used to supervise the depth map predicted by the student model This way makes the teacher and student models accept the same input, and can predict better pseudo-labels through the cropped pictures to supervise the student model and help the student model also predict better detailed features.
[0036] As shown in Fig. 3 In order to further improve the accuracy of the student model in predicting local details, the present application introduces cross-context distillation. Specifically, the teacher assistant will crop the original image input multiple times to predict local depth information for each cropped region At the same time, the student model generates a global predicted depth map for the original image input In the training process, the overlapping area between the local depth prediction of the teacher assistant and the global depth prediction of the student will be used to calculate the local loss, and the loss function is:
[0037]
[0038] These local losses are used to refine the student model's depth prediction in these specific regions, enabling the student to capture more accurate local details.
[0039] To balance the contributions of cross-context distillation and local context distillation, the invention defines a unified loss function.
[0040]
[0041] The loss function includes the following parts:
[0042] Cross-context distillation loss: The loss is used to refine the student model's depth prediction in these specific regions, enabling the student to capture more accurate local details.
[0043] Local context distillation loss: By supervising the fine-grained depth estimation loss of local regions, the teacher model and student model estimate consistent results for the same local context.
[0044] Feature alignment loss: According to existing methods, the invention also introduces a feature alignment loss to ensure that the features of the teacher model and student model are consistent.
[0045] Gradient preservation loss: To enhance the details of the model's depth prediction, the invention also adds a gradient preservation term.
[0046] The weighted sum of these loss terms will constitute the final loss function, where λ1, λ2, λ3 are hyperparameters used to balance the contributions of each part of the loss. The final loss function can help the student model capture both detailed local depth information and global depth structure, thereby improving the accuracy and robustness of depth estimation.
[0047] During the training process, the teacher model and student model are optimized through the distillation process. The teacher model is trained using synthetic data to generate accurate depth predictions, while the student model learns how to perform depth estimation on unlabeled data under the guidance of the teacher model's pseudo-labels, as follows:
[0048] Teacher model training: First, train the teacher model using a large amount of synthetic data to ensure that it can accurately predict the depth information of the entire image and local regions. After sufficient training, the teacher model can generate high-quality depth labels.
[0049] Student model initialization: The student model is initialized with random parameters and is ready to receive depth information guidance from the teacher model.
[0050] Distillation Process: In the distillation process, the teacher model generates pseudo-labels through global and local depth predictions. The student model uses these pseudo-labels as supervisory signals to optimize its depth estimation capabilities.
[0051] Loss Calculation and Optimization: In each training iteration, the difference between the student model's predicted depth map and the pseudo-labels generated by the teacher model is used to calculate the loss function. Through the backpropagation algorithm, the student model's parameters are updated.
[0052] Through repeated training, the student model can accurately estimate the depth information in the scene without a large amount of labeled data.
[0053] The embodiments of the present specification are merely enumerations of implementation forms of the inventive concept, and are only for illustrative purposes. The protection scope of the present application should not be regarded as being limited to the specific forms presented in the present embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.
Claims
1. A refined distillation method for monocular depth estimation, characterized in that, The method includes the following steps: Step 1) First, a teacher model is trained using labeled, high-precision synthetic data. This teacher model is then used to estimate the depth information of the entire image. Step 2) Use the depth pseudo-labels output by the teacher model directly as supervision signals to distill the student model; Step 3) Based on this, local context distillation and cross-context distillation methods are used to improve the teacher model's utilization of unlabeled data, supplement and strengthen the depth estimation of local regions during the distillation process, thereby improving the model's depth estimation details and accuracy. In step 3), the local context distillation method generates local depth maps by inputting local regions of the image into the teacher and student models respectively for supervision. Based on the input original image, a local region is randomly cropped and input into the teacher and student models respectively. The depth pseudo-label predicted by the teacher model is used to supervise the student model's depth estimation of the local region. In step 3), during cross-context distillation, the teacher assistant performs multiple cropping operations on the original image and predicts local depth information for each cropped region. Simultaneously, the student model performs global depth prediction for the entire image. During training, the local depth predictions generated by the teacher assistant and the global depth predictions of the student model differ in overlapping regions; this difference is the local loss. By calculating these local losses, the depth predictions of the student model in specific regions are further refined, enabling the student model to capture rich local information more accurately.
2. The refined distillation method for monocular depth estimation as described in claim 1, characterized in that, In step 1), by using synthetic data, it is possible to obtain large-scale, high-quality labeled depth information, which comes from a virtual environment.
3. A refined distillation method for monocular depth estimation as described in claim 1 or 2, characterized in that, In step 2), firstly, the depth pseudo-labels generated by the teacher model are used and applied to unlabeled real image data. The depth pseudo-labels guide the student model during the distillation process, thereby enabling the student model to gradually learn how to process real-world data.