Refining distillation method for monocular depth estimation

Abstract

A refining distillation method for monocular depth estimation comprises the following steps: step 1) firstly, training a teacher model through labeled high-precision synthetic data, and using the teacher model to estimate depth information of a whole image after full training; 2) utilizing an existing teacher model, and distilling the student model by adopting a deep pseudo label output by the teacher model as a supervision signal directly; and step 3) using a local context distillation method and a cross-context distillation method, and supplementing and strengthening depth estimation of a local area in the distillation process by improving utilization of the teacher model on unlabeled data, thereby improving depth estimation details and precision of the model. According to the method, the utilization of the monocular depth estimation model on unlabeled data can be effectively improved, so that the method has relatively strong generalization ability and higher accuracy in a zero sample scene, and is particularly suitable for application scenes needing high-precision and strong-generalization depth models, such as automatic driving, robots, augmented reality and the like.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to technologies related to monocular depth estimation, which is applicable to tasks that do not require a depth sensor and can infer the depth of a three-dimensional scene using only a single RGB image. Specific applications include, but are not limited to, autonomous driving, robot navigation, augmented reality (AR), virtual reality (VR), and map generation.

[0002] BACKGROUND Monocular depth estimation is a key research direction in computer vision, which aims to predict the depth information of a scene by analyzing a single RGB image. Unlike stereo vision or multi-camera depth estimation methods, monocular depth estimation has significant advantages: it does not rely on additional depth sensors or multi-camera devices, so it has broad application prospects in practical applications, especially in autonomous driving, robot navigation, augmented reality (AR), virtual reality (VR) and other fields. However, although deep learning technology has made significant progress in monocular depth estimation, how to improve the accuracy and generalization ability of depth estimation in the absence of a large amount of annotated data remains a challenge that needs to be solved.

[0003] Traditional depth estimation methods usually rely on hand-crafted feature extraction and geometric analysis, which have low accuracy when dealing with complex scenes. In recent years, deep learning methods based on convolutional neural networks (CNNs) have gradually become mainstream. Through end-to-end training, these methods can automatically learn effective feature representations from images, thereby significantly improving the accuracy of monocular depth estimation. However, one of the main challenges of these deep learning methods is their reliance on large amounts of annotated data. The acquisition of annotated data usually requires high costs, and in some application scenarios, annotated data is difficult to obtain.

[0004] To solve this problem, researchers have proposed unsupervised learning and semi-supervised learning methods. These methods enable the model to be trained without labeled data by designing proxy tasks or exploiting the geometric constraints of the image itself. For example, unsupervised methods that use the geometric consistency of image pairs for training and depth estimation methods based on self-supervised learning have made significant progress. However, these methods often rely on a large number of image pairs or synthetic data and still have certain limitations. Especially in the absence of labeled data, how to better use pseudo-labels to improve the generalization ability of the model remains an urgent problem to be solved.

[0005] As a knowledge transfer method, distillation technology has been widely used in the field of deep learning in recent years. In monocular depth estimation, distillation technology enhances the learning ability of the student model by transferring knowledge from the teacher model, thereby improving its generalization ability on unlabeled data. Although existing distillation methods in the field of depth estimation have achieved certain success, most methods still rely on global normalization strategies (such as scale and translation invariance (SSI) depth representation), which often suffer from the problem of reduced accuracy in the recovery of local details. The reason is that due to global normalization, the depth predicted by each pixel is no longer determined by the pixel itself, but is also affected by all other pixels. Therefore, the position that could have predicted the correct depth will also be affected by the depth information of the extremely wrong position. This problem still exists when performing model distillation. Therefore, when performing model distillation, how to balance the learning of global depth structure and local details is still a problem that needs to be explored. Summary of the invention

[0006] In order to overcome the shortcomings of the existing technology, the present invention provides a refinement distillation method for monocular depth estimation. In particular, a new refinement distillation method is used to solve the problems of ineffective supervision and insufficient utilization of pseudo labels in the prior art. The core innovation of this method is that it can effectively combine the global depth structure and local detail information in the distillation process, so that the student model not only relies on the global information provided by the teacher model, but also can supplement and strengthen the depth estimation of the local area during the distillation process, thereby improving the depth estimation accuracy of the model.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A refinement and distillation method for monocular depth estimation comprises the following steps:

[0009] Step 1) First, a teacher model is trained by annotated high-precision synthetic data, and the teacher model is trained to estimate the depth information of the entire image;

[0010] Step 2) Using the existing teacher model, the deep pseudo labels output by the teacher model are directly used as supervision signals to distill the student model;

[0011] Step 3) On this basis, using local context distillation and cross-context distillation methods, by improving the teacher model's utilization of unlabeled data, the depth estimation of local areas is supplemented and strengthened during the distillation process, thereby improving the model's depth estimation details and accuracy.

[0012] Furthermore, in step 1), by using synthetic data, we can obtain large-scale depth information with high-quality annotations. The synthetic data comes from a virtual environment (such as a 3D engine simulation). Although these data are virtually synthesized, they have very dense depth annotations and are extremely accurate.

[0013] Furthermore, in the step 2), first, the deep pseudo labels generated by the teacher model are used and applied to unlabeled real image data. The deep pseudo labels guide the student model in the distillation process, so that the student model gradually learns how to process real-world data.

[0014] Furthermore, in the step 3), the key to local context distillation is how to process the local area in the image. The local context distillation method generates a local depth map by inputting the local area of ​​the image into the teacher and student models respectively for supervision; according to the input original image, a local area is randomly cropped out and input into the teacher and student models respectively, and the depth pseudo-label predicted by the teacher model is used to supervise the student model's depth estimation of the local area.

[0015] In step 3), in the cross-context distillation, the teacher assistant will crop the original image multiple times and predict the local depth information for each cropped area respectively. At the same time, the student model performs a global depth prediction for the entire image. During the training process, the local depth prediction generated by the teacher assistant and the global depth prediction of the student model produce differences in the overlapping areas, and this part of the difference is the local loss. By calculating these local losses, the depth prediction of the student model in a specific area is further refined, so that the student model can more accurately capture the detailed local information.

[0016] In the present invention, a new method for representing depth information is proposed by comprehensively analyzing the effects of global normalization, local normalization, and multi-layer normalization strategies. Unlike the traditional global normalized depth representation, the method of the present invention no longer performs global normalization of depth information, but directly uses the depth pseudo-label output by the teacher model as a supervision signal to guide the student model to perform depth estimation. This method is not only simple and effective, but also surpasses the traditional global normalization method in accuracy. In addition, the present invention also proposes local context distillation and cross-context distillation methods, which effectively improve the performance of the monocular depth estimation model in actual scenarios by improving the utilization of unlabeled data by the teacher model, and enhances the generalization ability and accuracy of the model. In particular, in multiple popular monocular depth estimation test benchmarks, the method of the present invention has shown strong advantages.

[0017] The present invention effectively makes up for the shortcomings of traditional methods in detail recovery by combining local context distillation and cross-context distillation, and can more accurately estimate the depth information in the image. The teacher model is trained on synthetic data and combined with unlabeled real image data for pseudo-label distillation, which further improves the generalization ability and accuracy of the model in actual scenes.

[0018] The beneficial effects of the present invention are mainly manifested in:

[0019] 1. We conduct extensive experiments on multiple popular benchmark datasets. The experimental results show that our method significantly outperforms the baseline model and creates a new state-of-the-art. We are surprised to find that the student model surpasses the performance of the baseline model by relying only on unlabeled data.

[0020] 2. This paper conducts an in-depth analysis of multiple deep normalization strategies, including global normalization, local normalization, hybrid normalization, and no normalization method. Based on these analyses, we explore a more adaptive loss strategy that is more effective than the traditional SSI loss method in the distillation process.

[0021] 3. The present invention proposes a refined distillation framework using a cross-context teacher mechanism, which operates at two scales: global depth and local area, so that the student model can simultaneously learn macroscopic deep structure and fine detail information, thereby improving accuracy and robustness, especially in images with significant contextual changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is the overall distillation flow chart of the present invention;

[0023] Figure 2 It is a local context distillation flow chart of the present invention;

[0024] Figure 3 This is the cross-context distillation flow chart adopted by the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described below in conjunction with the accompanying drawings.

[0026] Reference Figure 1 to Figure 3 , a refined distillation method for monocular depth estimation, aims to perform multi-scale supervision through cross-context distillation and local context distillation, thereby effectively combining global structural information and local details and significantly improving the accuracy of depth estimation.

[0027] The refinement distillation method comprises the following steps:

[0028] Step 1) First, a powerful teacher model is trained through a large amount of annotated high-precision synthetic data. The teacher model is fully trained and can effectively estimate the depth information of the entire image; by using synthetic data, we can obtain large-scale, high-quality annotated depth information. Synthetic data comes from a virtual environment (such as a 3D engine simulation). Although these data are virtually synthesized, they have very dense depth annotations and are extremely accurate. Compared with traditional depth data that relies on manual annotations (such as LiDAR data), synthetic data has obvious advantages in annotation quality and quantity. Manually annotated depth data is not only costly, but also often has poor depth quality due to measurement errors or environmental complexity. Therefore, using synthetic data for teacher model training can not only effectively overcome the limitations of traditional annotated data, but also greatly improve the accuracy of the model in depth estimation, thereby improving the model's ability to predict depth details.

[0029] Step 2) Using the existing teacher model, the deep pseudo-labels output by the teacher model are directly used as supervisory signals to distill the student model; first, there is a certain difference in data distribution between the teacher model trained based on synthetic data and the real scene data, which leads to its poor performance in practical applications. In order to solve this problem, the use of extensive, real unlabeled image data has become an effective solution. Although the teacher model performs well on synthetic data, its performance in real scenes is poor. In order to make the teacher model adapt to real scenes, we use the deep pseudo-labels generated by the teacher model and apply them to unlabeled real image data. These pseudo-labels guide the student model during the distillation process, so that the student model gradually learns how to process real-world data. Unlike traditional methods, the present invention does not use global normalization (such as scale and translation invariance loss) for deep supervision, but directly uses the deep pseudo-labels output by the teacher model as supervisory signals to guide the student model to perform depth estimation. This method is not only simple in structure, but also performs better in accuracy than traditional methods.

[0030] Step 3) On this basis, the local context distillation and cross-context distillation methods are used to improve the teacher model's use of unlabeled data, supplement and strengthen the depth estimation of local areas during the distillation process, thereby improving the model's depth estimation details and accuracy. The key to the local context distillation is how to process local areas in the image. The local context distillation method generates local depth maps by inputting local areas of the image into the teacher and student models respectively for supervision. Specifically, we randomly crop a local area based on the input original image, and input it into the teacher and student models respectively, and supervise the student model's depth estimation of the local area through the depth pseudo-label predicted by the teacher model. Compared with the full image input, this local area-based input can provide a finer and more detailed depth prediction, making the student model's depth estimation in the local area more accurate. This method helps the student model achieve higher accuracy in fine-grained depth estimation by increasing attention to local details.

[0031] The core idea of ​​the cross-context distillation is to further expand the contextual understanding ability of the model on the basis of local context distillation. Specifically, in cross-context distillation, the teacher assistant will crop the original image multiple times and predict the local depth information for each cropped area respectively. At the same time, the student model performs a global depth prediction for the entire image. During the training process, the local depth prediction generated by the teacher assistant and the global depth prediction of the student model differ in the overlapping areas, and this part of the difference is the local loss. By calculating these local losses, the depth prediction of the student model in specific areas is further refined, so that the student model can more accurately capture detailed local information. Through cross-context distillation, the student model can significantly improve the recovery of local details while retaining the global depth structure.

[0032] like Figure 1As shown, the refined distillation framework includes a teacher model (Teacher), a student model (Student) and a teacher assistant model (Teacher Assistant), and distills through the depth information of cross-context and local context. Specifically, for local context distillation, the same image is input, and we input it into the teacher and student models at the same time, and the depth map is predicted by them respectively. The depth map predicted by the teacher model will guide the depth map predicted by the student model. For cross-context distillation, the present invention is implemented by the teacher assistant. When the same image is input, we will crop out multiple regions and input them into the teacher assistant (Teacher Assistant). The teacher assistant is responsible for predicting the local depth information of the multiple regions cropped from the input image, while the student model accepts the input of the whole image and generates a global depth map of the entire image. During the training process, the overlapping area between the local depth prediction of the teacher assistant and the global prediction of the student model is used to calculate the loss, thereby realizing distillation. In this process, the teacher's depth prediction is used as a pseudo-label to guide the student model to further refine the depth prediction on the local area, and it is expected that the depth prediction of multiple local areas can alleviate the noise caused by the prediction of a single local pseudo-label, thereby making the supervision more robust.

[0033] like Figure 2 As shown in the figure, the local context distillation method of the present invention randomly cuts out a local area from the input original image as the input of the model. We input it into the teacher and student models respectively to predict the depth map, and its loss function is:

[0034]

[0035] Pseudo labels predicted by the teacher Will be used to supervise the depth map predicted by the student model This method allows the teacher and student models to receive the same input and predict better pseudo-labels through cropped images, thereby supervising the student model and helping the student model to predict better detailed features.

[0036] like Figure 3 As shown in Figure 1, in order to further improve the accuracy of the student model in predicting local details, the present invention introduces cross-context distillation. Specifically, the teacher assistant will crop the original image input multiple times and predict the local depth information for each cropped area. At the same time, the student model generates a global predicted depth map of the original image input During training, the overlap between the teacher's local depth prediction and the student's global depth prediction is used to calculate the local loss, and the loss function is:

[0037]

[0038] These local losses are used to refine the depth prediction of the student model in these specific areas, so that the student can capture the detailed information of the local areas more accurately.

[0039] In order to balance the contributions of cross-context distillation and local context distillation, the present invention defines a unified loss function.

[0040]

[0041] The loss function consists of the following parts:

[0042] Cross-context distillation loss: The loss is used to refine the depth prediction of the student model in these specific areas, so that the student can capture the detailed information of the local area more accurately.

[0043] Local context distillation loss: By supervising the refined depth estimation loss of the local area, the teacher model and the student model estimate consistent results for the same local context.

[0044] Feature alignment loss: Based on the existing methods, the present invention also introduces feature alignment loss to ensure that the features of the teacher model and the student model are consistent.

[0045] Gradient preservation loss: In order to enhance the deep prediction details of the model, the present invention also adds a gradient preservation term.

[0046] The weighted sum of these loss terms will constitute the final loss function, where λ1, λ2, λ3 are hyperparameters used to balance the contribution of each part of the loss. The final loss function can help the student model capture both detailed local depth information and global depth structure, thereby improving the accuracy and robustness of depth estimation.

[0047] During the training process, the teacher model and the student model are jointly optimized through the distillation process. The teacher model is trained with synthetic data to generate accurate depth predictions, while the student model gradually learns how to perform depth estimation on unlabeled data under the guidance of the pseudo-labels provided by the teacher model. The process is as follows:

[0048] Teacher model training: First, a large amount of synthetic data is used to train the teacher model to ensure that it can accurately predict the depth information of the entire image and local areas. After the teacher model is fully trained, it can generate high-quality depth labels.

[0049] Student model initialization: The student model is initialized with random parameters and is ready to be guided by the deep information provided by the teacher model.

[0050] Distillation process: During the distillation process, the teacher model generates pseudo labels through global depth prediction and local depth prediction. The student model uses these pseudo labels as supervision signals to optimize its own depth estimation ability.

[0051] Loss calculation and optimization: In each training iteration, the difference between the predicted depth map of the student model and the pseudo-label generated by the teacher model is used to calculate the loss function. Through the back-propagation algorithm, the parameters of the student model are updated.

[0052] Through repeated training, the student model can accurately estimate the depth information in the scene without a large amount of labeled data.

[0053] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A refinement and distillation method for monocular depth estimation, characterized in that: The method comprises the following steps: Step 1) First, a teacher model is trained by annotated high-precision synthetic data, and the teacher model is trained to estimate the depth information of the entire image; Step 2) Using the existing teacher model, the deep pseudo labels output by the teacher model are directly used as supervision signals to distill the student model; Step 3) On this basis, using local context distillation and cross-context distillation methods, by improving the teacher model's utilization of unlabeled data, the depth estimation of local areas is supplemented and strengthened during the distillation process, thereby improving the model's depth estimation details and accuracy.

2. A refinement and distillation method for monocular depth estimation according to claim 1, characterized in that: In the step 1), large-scale depth information with high-quality annotations can be obtained by using synthetic data. The synthetic data comes from a virtual environment. Although the synthetic data is virtually synthesized, it has very dense depth annotations and extremely high accuracy.

3. A refinement and distillation method for monocular depth estimation according to claim 1 or 2, characterized in that: In the step 2), first, the deep pseudo labels generated by the teacher model are used and applied to unlabeled real image data. The deep pseudo labels guide the student model in the distillation process, so that the student model gradually learns how to process real-world data.

4. A refinement and distillation method for monocular depth estimation according to claim 1 or 2, characterized in that: In the step 3), the local context distillation method generates a local depth map by inputting the local area of ​​the image into the teacher and student models respectively for supervision. According to the input original image, a local area is randomly cropped and input into the teacher and student models respectively. The depth pseudo label predicted by the teacher model is used to supervise the student model's depth estimation of the local area.

5. The refinement and distillation method for monocular depth estimation according to claim 4, characterized in that: In step 3), in the cross-context distillation, the teacher assistant will crop the original image multiple times and predict the local depth information for each cropped area respectively. At the same time, the student model performs global depth prediction for the entire image. During the training process, the local depth predictions generated by the teacher assistant and the global depth predictions of the student model differ in the overlapping areas. This difference is the local loss. By calculating these local losses, the depth predictions of the student model in specific areas are further refined, so that the student model can more accurately capture detailed local information.

Images