Reliable robust monocular depth estimation method for severe weather
By generating multivariate degraded datasets and employing a multi-granularity knowledge distillation strategy, the performance degradation problem of monocular depth estimation under adverse weather conditions is addressed, achieving stable and efficient depth estimation under adverse weather conditions, applicable to fields such as autonomous driving and robot navigation.
Patent Information
- Application Number
- CN202510899685.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-01
- Publication Date
- 2025-11-18
AI Technical Summary
Existing monocular depth estimation methods suffer from significant performance degradation under adverse weather conditions, lack high-quality datasets covering multiple degradation types, and have loose constraints between teacher and student models, resulting in incomplete knowledge transfer, unnatural generated images, and poor generalization ability.
A dual-path learning system is constructed by generating a multivariate degradation dataset based on a pre-trained single-step diffusion model and a LoRA adapter. A multi-granularity knowledge distillation strategy and an ordinal-guided distillation mechanism are adopted, combined with a feature consistency alignment strategy, to train student models to improve their robustness under severe weather conditions.
It generates high-quality degraded data samples, enables stable and efficient depth estimation of student models under severe weather conditions, significantly improves generalization ability, reduces computing resource requirements, and is applicable to fields such as autonomous driving and robot navigation.
Smart Images
Figure CN120976280A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer vision, in particular to a reliable and robust monocular depth estimation method for severe weather, and the present application is applicable to automatic driving, robot navigation and three-dimensional scene reconstruction and other application scenarios. BACKGROUND TECHNOLOGY
[0002] With the rapid development of computer vision technology, monocular depth estimation plays an important role in automatic driving, robot navigation and three-dimensional scene reconstruction. However, existing monocular depth estimation methods perform well in normal scenes, but their performance significantly decreases in severe weather conditions (such as rain, fog, low light), and the main problems include: 1. Lack of high-quality datasets covering multiple degradation types; 2. The constraints between the teacher model and the student model are loose, resulting in incomplete knowledge transfer.
[0003] In summary, existing technologies such as data generation methods based on GAN or diffusion models have problems such as unnatural generated images and poor generalization ability. Therefore, there is an urgent need for a monocular depth estimation method that can work stably in severe weather conditions. SUMMARY
[0004] To solve the problem of unnatural generated images and poor generalization ability in existing data generation methods, a reliable and robust monocular depth estimation method for severe weather is proposed.
[0005] The technical solution adopted by the present application to solve the above problems is: The reliable and robust monocular depth estimation method for severe weather comprises the following steps: Generating a multi-element degradation dataset based on a pre-trained single-step diffusion model and a LoRA adapter to simulate scene images under severe weather conditions; Using a multi-granularity knowledge distillation strategy, a pre-trained Depth Anything V2 model as a priori teacher network, a lightweight depth estimation network as a basic teacher network, a two-way learning system, and training a student model; Using ordinal guided distillation mechanism, by establishing the depth ordinal space relationship, helping the student network to focus more effectively on the feature learning of the depth boundary area and uncertain pixel points; Feature consistency alignment strategy, through multiple levels of feature representation, so that the student model can generate consistent and reliable feature representation under various input conditions.
[0006] Further, the specific steps of generating a multi-element degradation dataset include: By using a pre-trained single-step diffusion model, the pre-trained single-step diffusion model is fine-tuned by a low-rank adapter to establish an adjustable degradation parameter space; and precise modeling of complex weather conditions such as light attenuation, particle scattering, rain and fog blocking, etc.
[0007] The cyclic consistency constraint and the adversarial training joint optimization strategy are introduced, and a degradation-recovery bidirectional mapping network is constructed to ensure the consistency of the synthesized data in the degradation feature distribution and the original scene semantics.
[0008] Further, the multi-granularity knowledge distillation strategy comprises: A double distillation system is constructed, and the double distillation system takes the pre-trained Depth Anything V2 model as the prior teacher network; and a lightweight depth estimation network is introduced as the basic teacher network. Further, the specific implementation of the ordinal guided distillation mechanism comprises: Local information is sampled from the depth error maps of the teacher model and the student model, and the error area of the student model is dynamically optimized by using a soft mask; For global information, the relative depth map is randomly sampled, the depth information sampled from the relative depth map is defined as an ordinal pair, and the ordinal pair is distilled into the student model through an ordering loss; Further, the feature consistency alignment strategy comprises: Consistency constraints are applied on shallow, intermediate and deep features respectively to ensure that the student model can generate similar feature representations to the teacher model under different input conditions; The knowledge transfer mode is adjusted by dynamic weight distribution to adapt to adverse weather conditions.
[0009] The beneficial effects of the present application are: 1、The present application generates a multi-element degradation data set, simulates a variety of complex environmental conditions (such as low light, haze, rain and snow, etc.), and can generate high-quality degraded data samples. These data not only cover a wealth of scene information, but also fully consider various uncertainty factors in actual applications, thereby providing comprehensive and detailed data support for model training.
[0010] 2、The present application realizes complete transfer and alignment of the ability from the teacher model to the student model through the multi-granularity knowledge distillation strategy. By constructing a double distillation system, the student fully utilizes the powerful depth estimation capability of the pre-trained Depth Anything V2 model as the prior teacher network and the high efficiency of the lightweight depth estimation network as the basic teacher network. This strategy not only improves the generalization ability of the student model, but also ensures its stable performance in various complex environments, significantly reduces the demand for computing resources, and makes it more suitable for actual deployment.
[0011] 3、The multi-element degradation data set generation scheme and the multi-granularity knowledge distillation strategy constitute a new type of robust monocular depth estimation framework, which not only breaks through the limit of visual perception in the technical field, but also provides new methodological support for building a reliable machine vision system.
[0012] 4、The application is not only suitable for automatic driving, unmanned vehicle navigation and the like, but also can be widely applied to industrial detection, security monitoring and a plurality of industries, and has important academic value and broad application prospect. BRIEF DESCRIPTION OF DRAWINGS
[0013] Figure 1 is a structural schematic diagram of a reliable robust monocular depth estimation model for severe weather of the application; Figure 2 is a structural schematic diagram of a multi-granularity distillation learning module of the application. DETAILED DESCRIPTION
[0014] Specific implementation one: combined with Figure 1 and Figure 2 The present embodiment proposes a reliable robust monocular depth estimation model for severe weather, which mainly includes two modules, as shown in Figure 1 , respectively, a multi-element degradation data generation module and a multi-granularity distillation learning module, The multi-element degradation data generation module is given normal input captured under good lighting and visibility conditions. Previous studies generate degraded images based on GAN or diffusion-based models, including various weather scenarios such as rain, night and fog. However, these methods have two main problems: first, they require a large number of real multi-element pairs for training, which is often difficult to obtain in practical applications; second, the translated images generated by these techniques have obvious differences and unnaturalness from real samples, and this quality gap directly affects the generalization ability of the depth estimation model in different scenarios. The application explores the stable diffusion content generation capability, and simultaneously adopts adversarial learning and cycle consistency loss to train the LoRA adapter to promote the naturalness and consistency of the translated images. Taking normal images as input, together with the corresponding text prompts, learning the specific LoRA adapter for each scene conversion can complete the conversion from the source domain to the target domain. This method not only maintains the structural information of the original scene, but also generates natural images that meet the target weather characteristics.
[0015] The multi-granularity distillation learning module, as shown in Figure 2The model training level, the present application innovatively proposes a multi-granularity knowledge distillation architecture. This architecture aims to fully utilize the knowledge of the pre-trained model to improve the performance of the lightweight student model by constructing a double distillation system. Specifically, the double distillation system takes the pre-trained Depth Anything V2 model as the prior teacher network, which is known for its powerful monocular depth estimation ability and can provide high-precision depth prediction results. At the same time, a lightweight depth estimation network is introduced as the basic teacher network. This design not only retains the ability to capture details in complex scenes, but also significantly reduces the demand for computing resources. In order to further optimize the learning effect of the student network, the present application specially designs an ordinal guided distillation mechanism, the core idea of which is to establish a depth ordinal space relationship to help the student network focus more effectively on the feature learning of depth boundary areas and uncertain pixels. The key of this method is to use depth ordinal information to guide the student model to understand the relative depth relationship between different objects, thereby improving its understanding ability of complex scenes. In this way, the student network can better grasp the subtle changes in the depth boundary area and handle those pixels with uncertain depth values. In addition, the present application also proposes a dynamic feature alignment strategy to promote the student model to learn the scene degradation content from the teacher model. The core of the feature consistency alignment strategy is to adaptively adjust the way of knowledge transfer according to the characteristics of different scenes. For example, in the face of low light, rain and other adverse conditions, the dynamic feature alignment strategy can enhance the learning ability of the student model for these specific scenes, making it better adapt to the diversified needs in actual application. This strategy not only improves the generalization ability of the student model, but also ensures its stable performance in various complex environments.
[0016] Specific implementation method two: this implementation method proposes a reliable and robust monocular depth estimation method for adverse weather, including multi-element degradation data generation and multi-granularity knowledge distillation strategy, as follows: Multi-element degradation data generation, the principle of data generation is image generation, which involves image translation. Image translation is an important technology in the field of computer vision, aiming to map input images from one domain to another while preserving their semantic information. For example, converting images of sunny scenes to rainy or foggy scenes, or converting photo-style images to artistic-style images. In this implementation, image translation technology is used to generate a multi-element degradation dataset to simulate scenes under various adverse weather conditions, thereby laying a foundation for the optimization of the subsequent robust monocular depth estimation model.
[0017] The core of the image translation task lies in building an adversarial learning framework of a generator and a discriminator. The generator is responsible for converting input images from the source domain to the target domain, while the discriminator is used to distinguish the differences between the generated images and the real target domain images. Through continuous optimization of the game process between the generator and the discriminator, the generator can finally generate highly realistic target domain images. The conventional approach is to complete the translation task based on Gan's image unpaired training, but such methods often require a large amount of real degraded data as a training set, making it difficult to generalize to some extremely harsh scenarios.
[0018] Diffusion models, as powerful base models for generative tasks, have received increasing attention. The basic principle can be divided into forward diffusion process and reverse diffusion process. In the forward diffusion process, the original image is gradually added with noise until it finally becomes completely random noise. This process is usually carried out through a series of predefined steps, with each step adding a certain amount of noise; the reverse diffusion process is the process of learning how to gradually recover the original image or target image from the noise. The goal of training is to learn a model that can reverse each step in the forward diffusion process; in our method, the diffusion model is used as the basic image generator, and the style of the generated image is adjusted by a set of LoRA parameters. LoRA is a parameter-efficient fine-tuning technique designed to adapt a pre-trained model to a specific task by updating only a small number of newly added parameters, rather than fine-tuning the entire model.
[0019] Multi-granularity knowledge distillation strategy, including ordinal guided distillation and feature consistency alignment, specifically includes the following: Ordinal guided distillation: Knowledge distillation is a model compression and transfer learning technique that aims to train a smaller "student" model through a larger "teacher" model. The core idea is to use the knowledge of the teacher model to guide the learning process of the student model, so that the student model has a smaller model size and higher computational efficiency while maintaining high performance. For example Figure 2As shown, training an excellent teacher model is the primary task of the entire distillation learning strategy, and the performance of the teacher model will directly affect the upper limit of the entire learning, thereby determining the final performance of the student model. In the present embodiment, the pre-trained deep large model is innovatively used as an additional teacher prior model, and a ordinal guided distillation method is designed to distill the knowledge of the large model into the student model. In addition, the teacher model of the conventional self-supervised learning will also serve as a distillation signal, which provides the student model with basic scale information. In the training of the basic teacher model and the pre-trained deep model, the final student model not only inherits the absolute scale information of the teacher model, but also obtains the representation of local details from the pre-trained deep large model. In order to introduce this combination idea into the distillation in the harsh scene, we propose an uncertain region learning strategy, which defines the error between the teacher model and the student model output as the distillation target region, introduces a mask to determine the sampling region of the ordinal distillation loss, and guides the distillation direction of the pre-trained deep large model. This strategy not only transfers the knowledge learned by the teacher model to the student model in a certain form, but also continuously optimizes the gap information between the teacher model and the student model, thereby improving the learning efficiency of the student model. The detailed distillation structure is shown in Figure 2 .
[0020] Since the supervision information provided by the pre-trained deep large model is relative depth, it cannot be directly used in the absolute depth with scale, so additional distillation strategies must be designed. In the present method, we design an ordinal guided distillation based on ranking loss. Although relative depth does not carry any absolute scale information, it provides the position relationship of the scene. This relationship can directly define the position information of the scene object, and it is feasible to improve the error region of the student model using the position prior in the scene. Specifically, we sample local information from the depth error map of the teacher model and the student model, which uses soft mask information to continuously optimize the student model to learn poor regions. For global information, we sample from the relative depth map, and the sampling is completely random, which can improve the student model from the perspective of global depth map. The depth information sampled from the relative depth map is defined as ordinal pairs, which are distilled into the student model through ranking loss, thereby improving the performance of the student model. In the depth estimation task, ranking loss is a loss function used to ensure the correctness of the relative order between predicted depth values. It is particularly suitable for tasks that require to maintain the relative depth relationship between objects in the scene. Since calculating the ranking loss does not require any scale information, it is very suitable for our task.
[0021] It is worth noting that there may be some mutual conflicts between the distillation of relative depth maps and the distillation of absolute depth maps, the most common of which is that relative depth can cause the student model to overfit, and for this, the ordinal window sampling method is introduced to solve this problem, which modifies the sampling of a single pixel to the sampling of a window area of pixels, and distills this area of pixels to the student model to improve the performance of the student model.
[0022] Feature Consistency Alignment: To further enhance the robustness of the student model under different scenarios, we introduce an innovative feature consistency constraint mechanism, as shown in Figure 2 Specifically, feature consistency constraint is achieved through multiple levels of feature representation, enabling the student model to generate consistent and reliable feature representations under various input conditions. The core idea of feature consistency constraint is to force the student model to generate similar feature representations to the teacher model under different input conditions, thereby improving its stability and accuracy in complex environments. This constraint mechanism not only acts on a single level, but also permeates multiple feature levels of the model, from shallow features to deep features, ensuring that the entire network structure maintains high consistency. Shallow features usually contain basic edges, textures, and other low-level information of images. By introducing feature consistency constraints at the shallow level, we can ensure that the student model can accurately capture these basic features, even under adverse weather conditions (such as foggy and rainy days), and maintain an understanding of basic visual elements. This consistency of shallow features helps to improve the robustness of the model under low light or blurred conditions, ensuring the quality of the basic data for subsequent processing steps. Intermediate-level features contain more semantic information, such as object shapes, contours, etc. By imposing feature consistency constraints at the intermediate level, the student model can better understand the objects in the scene and their relationships. In complex weather conditions, such as fog and heavy rain, the consistency of intermediate-level features can help the model distinguish between foreground and background and identify important target areas, thereby improving the accuracy of depth estimation. Deep features usually involve high-level semantic information, such as the overall layout of the scene and object categories. By introducing feature consistency constraints at the deep level, the student model can mimic the behavior of the teacher model at a high level, ensuring consistent understanding of the overall scene. This deep-level feature consistency is particularly important for handling extreme weather conditions (such as heavy rain and thick fog), as it helps the model maintain a correct understanding of the global structure in complex environments.
[0023] Through the feature consistency constraint, the knowledge of the teacher model is efficiently transferred to the student model. Not only the specific values of depth estimation, but also the understanding of scene structure and object relationship are included. This knowledge transfer mechanism enables the student model to obtain strong depth estimation ability in a short time, while maintaining low computational cost and storage requirements.
[0024] The above merely describes preferred embodiments of the present application, and is not intended to limit the present application in any form. Although the present application has been disclosed as above with preferred embodiments, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to the above disclosed technical content without departing from the technical solution range of the present application, and equivalent embodiments with equivalent changes are made. Any simple modification, equivalent replacement and improvement of the above embodiments within the technical solution content of the present application, according to the technical essence of the present application, within the spirit and principles of the present application, are still within the protection scope of the technical solution of the present application.
Claims
1. A reliable and robust monocular depth estimation method for severe weather, characterized in that, Includes the following steps: A multivariate degradation dataset is generated based on a pre-trained single-step diffusion model and a LoRA adapter to simulate scene images under severe weather conditions; The student model is trained by using a multi-granularity knowledge distillation strategy, which employs a dual-path learning system with a pre-trained Depth Anything V2 model as the prior teacher network and a lightweight deep estimation network as the base teacher network. By employing an ordinal-guided distillation mechanism, and establishing deep ordinal spatial relationships, the student network can more effectively focus on feature learning of deep boundary regions and uncertain pixels. The feature consistency alignment strategy is implemented through multiple levels of feature representation, enabling the student model to generate consistent and reliable feature representations under various input conditions.
2. The reliable and robust monocular depth estimation method for severe weather as described in claim 1, characterized in that, The specific steps for generating the multivariate degradation dataset include: By using a pre-trained single-step diffusion model, the parameters of the pre-trained single-step diffusion model are fine-tuned through a low-rank adapter to establish an adjustable degradation parameter space; thus achieving accurate modeling of complex meteorological conditions such as light attenuation, particle scattering, and rain and fog obstruction.
3. A joint optimization strategy of cyclic consistency constraint and adversarial training is introduced. By constructing a degradation-restoration bidirectional mapping network, the consistency between the degradation feature distribution of synthetic data and the semantics of the original scene is ensured.
4. The reliable and robust monocular depth estimation method for severe weather as described in claim 1, characterized in that, The multi-granularity knowledge distillation strategy includes: A dual-channel distillation system is constructed, with a pre-trained Depth Anything V2 model as the prior teacher network; a lightweight deep estimation network is introduced as the base teacher network.
5. A reliable and robust monocular depth estimation method for severe weather conditions according to claim 1, characterized in that, The specific implementation of the ordinal-guided distillation mechanism includes: Local information is sampled from the depth error maps of the teacher and student models, and the error region of the student model is dynamically optimized using soft masks. For global information, random sampling is performed from the relative depth map. The depth information sampled from the relative depth map is defined as ordinal pairs, and then distilled into the student model using a sorting loss.
6. The reliable and robust monocular depth estimation method for severe weather conditions according to claim 1, characterized in that, The feature consistency alignment strategy includes: Consistency constraints are applied to shallow, intermediate, and deep features to ensure that the student model can generate feature representations similar to the teacher model under different input conditions. The knowledge transfer method is adjusted by dynamically allocating weights to adapt to severe weather conditions.