A nighttime depth estimation method based on a self-supervised framework

Abstract

The present invention discloses a nighttime depth estimation method based on a self-supervised framework, comprising the following steps: obtaining a daytime data set, inputting the daytime data set into a pre-trained estimation network model to obtain a daytime depth map; obtaining a target frame and a source frame of a nighttime RGB image, inputting the target frame into a depth estimation network model to obtain a nighttime depth map; splicing the channels of the target frame and the source frame, and obtaining the posture changes of the spliced ​​target frame and the source frame according to a posture network model; enhancing the target frame and the source frame using an adaptive image enhancement module to obtain a first source frame and a first target frame; reconstructing the nighttime depth map, the posture changes, and the first source frame to obtain a reconstructed target frame; inputting the daytime depth map and the nighttime depth map into an adversarial neural network model to obtain a self-supervised nighttime depth map whose estimated distribution conforms to the depth distribution under normal daytime lighting scenes. The present invention can achieve excellent recognition results for specific nighttime scenes.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving or robotics technology, and in particular relates to a nighttime depth estimation method based on a self-supervisory framework. Background Art

[0002] Monocular depth estimation is an important task in computer vision, crucial for applications such as autonomous driving, robotic navigation, and 3D reconstruction. Collecting high-quality depth data in a wide range of environments using expensive depth sensors such as LiDAR and Time-of-Flight (TOF) is prohibitively expensive. Self-supervised monocular depth estimation, which exploits photometric limitations between monocular image sequences without requiring the collection of ground-truth depth maps, is gaining increasing attention. In addition to research on daytime datasets such as KITTI and Make3D, numerous efforts have been made to develop self-supervised methods. These methods train deep networks to estimate depth maps by exploiting geometric cues in videos, essentially reconstructing a target view (or frame) from another view, rather than leveraging high-quality depth data. Furthermore, their performance is comparable to supervised methods in well-lit environments. However, few methods can handle the more challenging nighttime scenes. Nighttime scenes, however, involve two significant challenges: low visibility and varying illumination, leading most existing self-supervised methods to produce erratic depth outputs. Low visibility often results in textureless regions. Dark regions with indistinguishable visual textures exacerbate this problem by producing depth maps with large holes, exacerbated by cropping two patches of varying brightness from the same location in two temporally adjacent frames. This brightness inconsistency leads to imperfect target view reconstruction, i.e., large training loss.

[0003] At the same time, the issue of lightweight models in monocular depth estimation has also attracted attention. Recent research has often focused on reducing model parameters and exploring new convolutional operations through techniques such as knowledge distillation. This allows models to be deployed on edge devices to solve real-world problems. However, few lightweight methods have been proposed to address the challenges posed by more complex nighttime scenes. The deep networks used in current nighttime methods are primarily adapted from Monodepth2, exhibiting relatively high model complexity. Existing lightweight methods are primarily trained and evaluated in daytime scenes.

[0004] In summary, the shortcomings of the prior art are as follows:

[0005] 1. Data collection requires manual labor, which results in high labor and time costs;

[0006] 2. The use of multiple sensors increases hardware costs and makes it impossible to collect accurate nighttime depth information.

[0007] 3. The current model cannot effectively process data from low-light and high-noise scenarios that are common in undesirable nighttime scenes.

[0008] 4. The computational cost of using the complex backbone network is too high, and real-time performance cannot be guaranteed when deployed in real scenarios.

[0009] In summary, it is of great significance to propose a lightweight deep network and integrate it into the nighttime self-supervised depth estimation architecture to deal with the low light and high noise problems in nighttime scenes. Summary of the Invention

[0010] To solve the above technical problems, the present invention proposes a nighttime depth estimation method based on a self-supervised framework, which is suitable for application on resource-constrained edge devices and can achieve good recognition effects on specific nighttime scenes.

[0011] The present invention provides a nighttime depth estimation method based on a self-supervised framework, comprising:

[0012] Obtaining a daytime data set, inputting the daytime data set into a pre-trained estimation network model to obtain a daytime depth map;

[0013] Obtain a target frame and a source frame of a nighttime RGB image, input the target frame into a depth estimation network model, and obtain a nighttime depth map;

[0014] splicing the channels of the target frame and the source frame, and obtaining the posture changes of the spliced ​​target frame and the source frame according to the posture network model;

[0015] The target frame and the source frame are enhanced by using an adaptive image enhancement module to obtain a first source frame and a first target frame;

[0016] Reconstructing the nighttime depth map, the posture change, and the first source frame to obtain a reconstructed target frame, and supervising the reconstructed target frame and the first target frame using a first loss function;

[0017] The daytime depth map and the nighttime depth map are input into an adversarial neural network model, and the estimated distribution of the self-supervised nighttime depth map is consistent with the depth distribution under normal daytime lighting scenes, wherein the adversarial neural network uses a second loss function for supervised training.

[0018] Optionally, the depth estimation network model includes: a depth encoder and a depth decoder;

[0019] The depth encoder is used to encode the target frame;

[0020] The depth decoder is used to decode the encoded target frame to obtain a nighttime depth map.

[0021] Optionally, the depth encoder includes: a convolution layer, several downsampling layers, and several feature fusion modules;

[0022] The convolutional layer is used to refine feature extraction and further enhance the ability to capture local details;

[0023] The downsampling layer is used to reduce the spatial resolution of the feature map layer by layer, increase the receptive field, and retain the main scene structure information;

[0024] The feature fusion module is used to simultaneously realize the perception of local details and scene structure and integrate local and global image features.

[0025] Optionally, the depth decoder includes: several upsampling layers and an output module;

[0026] The upsampling layer is used for upsampling by bilinear interpolation;

[0027] The output module is used to output the upsampling result.

[0028] Optionally, splicing the channels of the target frame and the source frame, and obtaining the posture changes of the spliced ​​target frame and the source frame according to the posture network model includes:

[0029] The target frame and the source frame channels are spliced, and the spliced ​​target frame and the source frame are input into a posture network model to obtain the relative posture between the target frame and the source frame.

[0030] Optionally, enhancing the target frame and the source frame by using an adaptive image enhancement module to obtain the first source frame and the first target frame includes:

[0031] Divide the target frame and the source frame into blocks to obtain a plurality of image blocks;

[0032] Introducing a contrast limit parameter to obtain a histogram of the image block;

[0033] The histogram is equalized, and the image blocks after the splicing process are smoothed using bilinear interpolation to obtain the first source frame and the first target frame.

[0034] Optionally, the first loss function is:

[0035]

[0036] Among them, I' t is the enhanced target frame, is the reconstructed target frame, a is a parameter, and ||·||1 is the L1 norm.

[0037] Compared with the prior art, the present invention has the following advantages and technical effects:

[0038] (1) This paper designs a lightweight architecture for monocular depth estimation under nighttime conditions, which achieves a good balance between model complexity and accuracy and is particularly suitable for application on resource-constrained edge devices.

[0039] (2) Using unpaired daytime and nighttime RGB inputs, depth estimation is performed by fitting an adversarial neural network.

[0040] (3) No need to rely on complex and expensive depth cameras for data collection, training can be performed through self-supervision;

[0041] (4) By training or fine-tuning the model, depth estimation can be performed on real scenes in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0043] Figure 1 This is a flow chart of a nighttime depth estimation method based on a self-supervised framework according to an embodiment of the present invention;

[0044] Figure 2 This is a structural diagram of a lightweight depth estimation network model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0046] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] This paper proposes a nighttime depth estimation method based on a self-supervised framework, such as Figure 1 As shown, the specific steps include:

[0048] Obtain a daytime data set, input the daytime data set into the pre-trained estimation network model, and obtain a daytime depth map;

[0049] Obtain the target frame and source frame of the nighttime RGB image, input the target frame into the depth estimation network model, and obtain the nighttime depth map;

[0050] The channels of the target frame and the source frame are spliced ​​together, and the posture changes of the spliced ​​target frame and the source frame are obtained according to the posture network model;

[0051] Enhance the target frame and the source frame using an adaptive image enhancement module to obtain a first source frame and a first target frame;

[0052] Reconstructing the nighttime depth map, the posture change, and the first source frame to obtain a reconstructed target frame, and supervising the reconstructed target frame and the first target frame using a first loss function;

[0053] The daytime depth map and the nighttime depth map are input into the adversarial neural network model, and the estimated distribution of the self-supervised nighttime depth map is consistent with the depth distribution under normal daytime lighting scenes, wherein the adversarial neural network uses the second loss function for supervised training.

[0054] Specifically, the nighttime self-supervised depth estimation architecture: the system mainly consists of a depth estimation network Φ d and posture network Φ p However, due to the particularity of night scenes, relying solely on the above method may produce a large number of abnormal depth values. In order to solve this problem, the present invention pre-trains the depth estimation network Φ′ on the daytime dataset. d , and then use the adversarial neural network Patch-GANΦ A To guide night training. A The discriminator in Φ d Generated night depth map D t and Φ′ d Generated daytime depth map D d .

[0055] Given an RGB image I t , we can use the trainable network Φ d :D t =Φ d (I t ) to predict the depth map D t In order to achieve self-supervision, it is necessary to use the geometric relationship T t→s =Φ p (I t , I s ) from source frame I s Reconstruct target frame I t The reconstruction process involves using the posture network to obtain I s with I t The relative posture T t→s Then, D t Point p in t Projection to I s Point p in s superior:

[0056] p s ~KT t→s D t (p t )K -1 p t

[0057] Among them, ~ represents the homogeneous equivalence relation, and K represents the camera intrinsic parameter.

[0058] Then, I s Through differentiable bilinear sampling operation

[0059]

[0060] Subsequently, the present invention combines the L1 regularization loss and the SSIM (structural similarity) loss into the photometric loss L P , the formula is:

[0061]

[0062] Where ||·||1 represents the L1 norm, and the parameter α is set to 0.85 in all experiments. In addition, the present invention uses edge-aware smoothing loss to reduce noise and discontinuities in the depth map or surface normal vector, making the depth estimation result more accurate and stable. The expression is as follows:

[0063]

[0064] in, and are the gradients of the image in the horizontal and vertical directions respectively. Finally, the present invention trains the generator Φ by minimizing the loss function of GAN d and the discriminator Φ A , the generator loss function is expressed as:

[0065]

[0066] The discriminator loss function is expressed as:

[0067]

[0068] Among them, |I d | represents the number of daytime training images, |I t | represents the number of nighttime training images. It should be noted that I t and I d does not correspond one-to-one to daytime and nighttime images, so the generated depth map D d =Φ′ d (I d ) and D t=Φ d (I t ) No pairing required.

[0069] Furthermore, the depth estimation network model includes: a depth encoder and a depth decoder;

[0070] A deep encoder for encoding the target frame;

[0071] The depth decoder is used to decode the encoded target frame to obtain a nighttime depth map.

[0072] Furthermore, the deep encoder includes: a convolutional layer, several downsampling layers, and several feature fusion modules;

[0073] Convolutional layer, used to refine feature extraction and further enhance the ability to capture local details;

[0074] The downsampling layer is used to gradually reduce the spatial resolution of the feature map, increase the receptive field, and retain the main scene structure information;

[0075] The feature fusion module is used to simultaneously realize the perception of local details and scene structure and integrate local and global image features.

[0076] Furthermore, the deep decoder includes: several upsampling layers and an output module;

[0077] Upsampling layer, used for upsampling by bilinear interpolation;

[0078] Output module, used to output upsampling results.

[0079] Specifically, this lightweight depth estimation network is implemented as follows Figure 2 As shown. The main module designs are as follows:

[0080] Lightweight Depth Estimation Network: Subsequent reasoning only requires the use of DepthNet. Therefore, designing a lightweight depth network can help the present invention effectively reduce the complexity of the model and the speed of reasoning. The present invention designs a lightweight depth estimation network that includes a feature fusion block and cross connections:

[0081] Deep encoder: Deep encoder. Using a shallower network can effectively reduce the complexity of the model, so the present invention adopts a four-level encoder. The convolution layer consists of a 3×3 convolution with a stride of 2 and two 3×3 convolutions with a stride of 1. The downsampling module is a 3×3 convolution with a stride of 2. Since the depth estimation task has high requirements for the perception of local details and scene structures, the present invention introduces a feature fusion block that can simultaneously realize the perception of local details and scene structures and integrate local and global image features. First, the feature fusion block uses 3×3 expanded convolution to expand the receptive field to realize the extraction of local features. Assuming a feature map x with a dimension of H×W×C, this process is as follows Figure 2 As shown:

[0082] First: a dilated convolution (Dconv) is used to expand the receptive field to perceive local features in a larger range. The output of the dilated convolution will undergo a batch normalization (BN) operation to standardize the features; then, the feature map is adjusted point by point; finally, the processed features are residually connected with the input features to ensure the effective integration of the newly extracted features with the original information.

[0083] In the encoder’s feature fusion block, the above operations for extracting local features are repeated multiple times to gradually enhance feature extraction capabilities. Typically, these operations are repeated 3, 3, and 9 times, respectively, from top to bottom.

[0084] The process can be expressed as:

[0085] L(x)=Pω2(Pω1(BN(Dconv(x))))+x

[0086] Where Dconv(·) represents dilated convolution, BN represents batch normalization, and Pω1(·) and Pω2(·) represent point-by-point operations for dimensionality expansion and reduction, respectively. In the feature fusion block of the encoder, the operation of extracting local features L(·) is repeated N times, 3, 3, and 9 times from top to bottom.

[0087] The present invention replaces the original self-attention with Cross-Covariance Attention (XCA), which has lower computational complexity, so as to more effectively model the global context. Specifically, it includes: global feature modeling: first, the global context features are extracted through the XCA module; then layer normalization (LN) is used to ensure the stability of the feature distribution; the features are expanded and reduced in dimension through point-by-point operations, and finally a global feature representation is obtained. Integration of local and global features: In order to enhance the integration and propagation of local features and global features, the present invention introduces a cross-connection mechanism in the downsampling module and the feature fusion block. Through cross-connection, local detail features are retained while global feature modeling, ensuring the effective transmission of context information throughout the network. The final output of the feature fusion block: The output of the feature fusion block consists of two parts: the result of multiple extractions of local features and the result of global feature modeling. The final output merges the two through the feature concatenation operation (Concat) to form a richer feature expression. Such a design effectively improves the comprehensive perception of local details and global context while reducing computational complexity.

[0088] The present invention denotes this operation as G(·), and the corresponding expression is:

[0089] G(x′)=Pω2(Pω1(LN(XcA(x′))))

[0090] Where x′ is the input feature and LN represents layer normalization. To enhance the integration and propagation of local and global features, the present invention uses cross connections in both the downsampling module and the feature fusion block. The final output y of the feature fusion block can be expressed as:

[0091] y=Concat[N·L(x),G(N·L(x))]

[0092] Deep Decoder: Deep decoder. To reduce the complexity of the model, this paper only uses a decoder composed of convolutional layers. The upsampling module uses bilinear interpolation for upsampling and is then connected to the disparity head of each output. These outputs are respectively and full resolution generation.

[0093] Furthermore, the channels of the target frame and the source frame are spliced ​​together, and the posture changes of the spliced ​​target frame and the source frame are obtained according to the posture network model, including:

[0094] The target frame and the source frame channels are spliced ​​together, and the spliced ​​target frame and source frame are input into the posture network model to obtain the relative posture between the target frame and the source frame.

[0095] Furthermore, reconstructing the nighttime depth map, the posture change and the first source frame includes: given an RGB image I t, we can use the trainable network Φ d :D t =Φ d (I t ) to predict the depth map D t In order to achieve self-supervision, it is necessary to use the geometric relationship T t→s =Φ p (I t , I s ) from source frame I s Reconstruct target frame I t The reconstruction process involves using the posture network to obtain I s with I t The relative posture T t→s Then, D t Point p in t Projection to I s Point p in s superior:

[0096] p s ~KT t→s D t (p t )K -1 p t

[0097] Among them, ~ represents the homogeneous equivalence relation, and K represents the camera intrinsic parameter.

[0098] Then, I s The reconstructed target frame is obtained by the differentiable bilinear sampling operation s(·,·)=

[0099]

[0100] Subsequently, the reconstructed target frame and the target frame are supervised by using the photometric loss, which includes: the present invention combines the L1 regularization loss and the SSIM (structural similarity) loss into the photometric loss L P , the formula is:

[0101]

[0102] Where ||·||1 represents the L1 norm, and the parameter α is set to 0.85 in all experiments. In addition, the present invention uses edge-aware smoothing loss to reduce noise and discontinuities in the depth map or surface normal vector, making the depth estimation result more accurate and stable. The expression is as follows:

[0103]

[0104] in and are the gradients of the image in the horizontal and vertical directions, respectively.

[0105] Further, enhancing the target frame and the source frame by using an adaptive image enhancement module to obtain the first source frame and the first target frame includes:

[0106] Divide the target frame and the source frame into blocks to obtain several image blocks;

[0107] Introduce contrast limit parameters to obtain the histogram of the image block;

[0108] The histogram is equalized, and the processed image blocks are smoothed using bilinear interpolation to obtain a first source frame and a first target frame.

[0109] Specifically, noise-constrained adaptive image enhancement: In nighttime images, the photometric consistency between the target and source frames is often not maintained, accompanied by low light and high noise levels. Therefore, inspired by adaptive histogram equalization, this paper proposes a noise-constrained adaptive image enhancement (NCAIE) module. This avoids the use of an additional image enhancement network, which is consistent with the goal of lightweight model design.

[0110] The present invention first converts the target frame I t and source frame I s Divide into I×J small blocks. Target frame I t Each block in the partition is denoted as t i,j , source frame I s Each block in the partition is denoted as s i，j Assuming that each block has M pixels and N gray levels, the present invention calculates the histogram h of each small block i，j (n), and at the same time, in order to control the noise amplification caused by excessive contrast, the present invention introduces a contrast limit parameter β, and obtains the clipped histogram h′ i，j (n), then, the present invention calculates the distribution function CDF of the cumulative histogram as follows:

[0111]

[0112] The above expression represents the histogram equalization operation of each small block. Finally, the image blocks after smoothing and splicing are used to obtain the enhanced target frame I′ t and source frame I′ s .

[0113] In the experiment of the present invention, the present invention also introduces the reconstruction after the image enhancement module. The difference between this method and the present invention is that the size of each small block I×J is set to 8×8 and the contrast limit parameter β is set to 4. The entire enhancement process can be expressed as:

[0114] I′ t =∈(I t), I′ s =∈(I s )

[0115] Where ∈ represents the mapping function of image enhancement. Therefore, I′ s The reconstructed RGB image I′ can be obtained t :

[0116]

[0117] Finally, the overall modified definition of the luminosity loss function of the present invention is as follows:

[0118]

[0119] In the normal reconstruction process, of course, it can be completed without image enhancement. The present invention also deduces the reconstruction formula based on this.

[0120] Considering that in dark light scenes, the luminosity consistency between the target frame and the source frame is often not maintained, the quality of reconstruction will be poor. This invention designs this image enhancement module, which first enhances the image and then reconstructs it, which can alleviate some of this phenomenon.

[0121] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A nighttime depth estimation method based on a self-supervised framework, characterized in that: include: Obtaining a daytime data set, inputting the daytime data set into a pre-trained estimation network model to obtain a daytime depth map; Obtain a target frame and a source frame of a nighttime RGB image, input the target frame into a depth estimation network model, and obtain a nighttime depth map; splicing the channels of the target frame and the source frame, and obtaining the posture changes of the spliced ​​target frame and the source frame according to the posture network model; The target frame and the source frame are enhanced by using an adaptive image enhancement module to obtain a first source frame and a first target frame; Reconstructing the nighttime depth map, the posture change, and the first source frame to obtain a reconstructed target frame, and supervising the reconstructed target frame and the first target frame using a first loss function; The daytime depth map and the nighttime depth map are input into an adversarial neural network model, and the estimated distribution of the self-supervised nighttime depth map is consistent with the depth distribution under normal daytime lighting scenes, wherein the adversarial neural network uses a second loss function for supervised training.

2. A nighttime depth estimation method based on a self-supervised framework according to claim 1, characterized in that: The depth estimation network model includes: a depth encoder and a depth decoder; The depth encoder is used to encode the target frame; The depth decoder is used to decode the encoded target frame to obtain a nighttime depth map.

3. The method for nighttime depth estimation based on a self-supervised framework according to claim 2, characterized in that: The depth encoder includes: a convolution layer, several downsampling layers, and several feature fusion modules; The convolutional layer is used to refine feature extraction and further enhance the ability to capture local details; The downsampling layer is used to reduce the spatial resolution of the feature map layer by layer, increase the receptive field, and retain the main scene structure information; The feature fusion module is used to simultaneously realize the perception of local details and scene structure and integrate local and global image features.

4. The method for nighttime depth estimation based on a self-supervised framework according to claim 3, characterized in that: The depth decoder includes: several upsampling layers and an output module; The upsampling layer is used for upsampling by bilinear interpolation; The output module is used to output the upsampling result.

5. The method for nighttime depth estimation based on a self-supervised framework according to claim 1, characterized in that: The channels of the target frame and the source frame are spliced ​​together, and the posture changes of the spliced ​​target frame and the source frame are obtained according to the posture network model, including: The target frame and the source frame channels are spliced, and the spliced ​​target frame and the source frame are input into a posture network model to obtain the relative posture between the target frame and the source frame.

6. The method for nighttime depth estimation based on a self-supervised framework according to claim 1, characterized in that: Enhancing the target frame and the source frame by using an adaptive image enhancement module to obtain a first source frame and a first target frame includes: Divide the target frame and the source frame into blocks to obtain a plurality of image blocks; Introducing a contrast limit parameter to obtain a histogram of the image block; The histogram is equalized, and the image blocks after the splicing process are smoothed using bilinear interpolation to obtain the first source frame and the first target frame.

7. The method for nighttime depth estimation based on a self-supervised framework according to claim 1, characterized in that: The first loss function is: Among them, I' t is the enhanced target frame, is the reconstructed target frame, a is a parameter, and ||·||1 is the L1 norm.

Images