Video defogging and depth estimation method in real mobile scene

Abstract

The invention discloses a video defogging and depth estimation method in a real mobile scene, which provides a clearer visual field and distance perception for a real fog scene, and combines a brightness consistency constraint and an atmospheric scattering model to jointly optimize a depth estimation network; according to the brightness consistency constraint, defogged adjacent frames are used as input, a more accurate camera pose is obtained, re-projection is completed by using an estimated depth map, and a self-supervised estimation mode is constructed; along with self-supervised learning of the depth estimation network, effective decoupling is carried out on each component of the whole atmospheric scattering model, meanwhile, each component is used for reconstructing a fog frame, and a reconstruction loss constraint is formed to supervise learning of the whole frame; a common optimized learning framework is constructed by using brightness consistency constraint and depth information shared by an atmospheric scattering model. In a test and use stage, a clear scene can be quickly repaired by using the defogging network and the depth estimation network only by inputting a real fog test image.

Description

Technical Field

[0001] The present invention relates to the field of video restoration and depth estimation, and in particular to a video defogging and depth estimation algorithm in a real moving scene. Background Art

[0002] Early image dehazing methods mainly focused on combining atmospheric scattering models (ASM) with different prior knowledge. However, recent progress in this field has shown that deep learning techniques have achieved better performance on a large number of blurry / clear image datasets. These methods use deep neural networks to learn physical model parameters or directly map blurry images to clear images / videos. These methods mainly rely on aligned synthetic data for training, which leads to unsatisfactory performance in real scenes due to distribution differences. To narrow the gap between domains, some methods adopt domain adaptation and unpaired dehazing models to handle real scenes. Despite these efforts, when image dehazing models are applied to videos, the lack of brightness consistency constraints leads to inconsistent brightness between adjacent frames, resulting in flickering. Video dehazing techniques use the temporal information of adjacent frames to improve the restoration quality. Early methods mainly focus on post-processing to ensure temporal consistency by optimizing transmission maps and suppressing artifacts. Some methods solve multiple tasks simultaneously, including depth estimation and detection in blurry videos. Recently, the Real World Video Dehazing Dataset (REVIDE) was introduced and an improved deformation network based on confidence guidance was proposed. Based on REVIDE, a phase-based memory network is designed for video dehazing. Similarly, a memory-based physical prior guidance module is proposed to incorporate features related to the prior into long-term memory for video dehazing. In addition, some image restoration methods have shown excellent performance on the REVIDE dataset, especially under severe weather conditions. However, these methods are mainly trained and evaluated on indoor smoke scenes, which limits their effectiveness in real-world outdoor hazy conditions. Compared with dehazing and depth estimation works, our proposed DCL is trained on real hazy videos instead of synthetic hazy images, and optimizes video dehazing and depth estimation simultaneously by integrating the ASM model and brightness consistency constraints (BCC). Self-supervised monocular depth estimation (SMDE) has sparked a growing interest in self-supervised methods due to the limited perception capabilities of lidar in extreme weather conditions. After the pioneering work of Zhou et al. demonstrated that superior performance can be achieved by simply exploiting geometric constraints between consecutive frames, researchers further explored clues and methods to train self-supervised models using video or stereo image pairs. Recently, several studies have utilized advanced networks to estimate depth in challenging scenes (such as rain, snow, fog, and low-light environments) and explored SMDE. These degraded images, especially in areas with weak textures, significantly affect depth estimation. For weak textures, some methods mainly adopt image enhancement and domain adaptive learning to address this problem. However, these image enhancement methods rely on operator-based methods rather than learnable methods, which only enhance depth estimation. In addition, most domain adaptive learning based on synthetic data may not be effectively applied to real scenes. Summary of the invention

[0003] The purpose of this invention is to propose a video defogging and depth estimation method in a real mobile scene. On the one hand, the loss function is designed by using non-aligned clear images. The supervised training of the desmog network, on the other hand, redefines the mean-variance description of atmospheric light and the three-channel method of the transmission medium map, and proposes a corresponding neural network structure to better learn atmospheric light and transmission medium maps, making the reconstruction loss function Efficiently train a haze removal network.

[0004] A technical solution to achieve the purpose of the present invention: a video defogging and depth estimation method for a real moving scene, comprising:

[0005] Step 1: Preprocess the data set; use the non-aligned matching algorithm to obtain the matching information between all foggy video frames and clear reference frames, dedistort the data according to the calibrated camera internal parameters, and perform cropping;

[0006] Step 2: Transform the continuous fog video frames I [t-n:t+n] ∈R 3×h×w Input to the dehazing network Φ J The dehazing result is obtained in [J t ,J s ], J t represents the current defogging frame, J s Indicates J t Neighboring defogging frames, where n = 1; the current foggy video frame I t Input into the depth estimation network Φ respectively d Medium estimate d t ∈R 1×h×w , and the scattering coefficient estimation network Φ β Estimated β t ∈R 1×h×w , using the dark channel method to calculate the value of the infinite atmospheric light A ∞ ;

[0007] Step 3: The defogging result of step 2 t Then input into the camera pose prediction network Φ p To predict the camera’s pose information P x→y ∈R 4×4 , and then use Φ d The estimated d t and Φ p Get P x→y By reprojecting the adjacent dehazed frames onto the current frame, the depth d is estimated by self-supervised learning by calculating the photometric loss constrained depth estimation. t ;

[0008] Step 4: J generated from step 2 to step 3t ,A ∞ ,d t and β t Reconstruct the current foggy video frame I′ using the atmospheric scattering model t , then define I′ t and I t The reconstruction loss function Contains the sum of three loss functions, namely the first-order norm loss Perceived loss and structural similarity loss

[0009] Step 5: Build about J t and J′ t The non-aligned reference loss function H t and H′ t The non-aligned reference loss function Regularize the color and texture feature distribution of the dehazing network;

[0010] Step 6: Use the discriminator network to analyze the dehazing network Φ J and the depth estimation network Φ d Perform regularization constraints;

[0011] Step 7: Based on the two loss functions of steps 4, 5, and 6 and And two discriminator networks D MFIR and D MDR The adversarial loss regularization optimizes the network parameters of the entire DCL framework to obtain the defogging result; finally, the real RGB fog video frame I is input for testing t , respectively input into the dehazing network and the depth estimation network, and directly generate a clear scene image result J t and the depth map d t .

[0012] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the video defogging and depth estimation method in a real mobile scene is implemented.

[0013] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned video defogging and depth estimation method in a real mobile scene.

[0014] A computer program product includes a computer program, which implements the above-mentioned video defogging and depth estimation method in a real mobile scene when executed by a processor.

[0015] Compared with the prior art, the present invention has the following significant advantages: (1) The present invention combines the atmospheric scattering model and brightness consistency constraint to propose a real scene video defogging and depth estimation learning framework with depth as the core. (2) The present invention proposes two discriminator networks D MFIR and D MDR To enhance the high-frequency details and weak textures of the dehazing results respectively, the depth map "black hole" problem is generated.

[0016] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a model network architecture diagram of the method of the present invention.

[0018] Figure 2 It is a comparative schematic diagram of the method of the present invention, which shows that the combined video defogging and depth estimation has a better effect than the separate foggy depth estimation and the defogging followed by depth estimation.

[0019] Figure 3 The figures show the comparison between the proposed method and other methods on three real foggy video datasets: GoProHazy, DrivingHazy and InternetHazy. Our method also gives the effects of dehazing and depth estimation.

[0020] Figure 4 The figures show the comparison between the proposed method and other methods on three real foggy video datasets: GoProHazy, DENS-Fog (dense) and DENS-Fog (light). Our method also gives the effects of dehazing and depth estimation.

[0021] Figure 5 The ablation experimental results of the proposed core module were visualized on DENS-Fog (light) and verified on GoProHazy. MFIR and D MDR Impact on dehazing results and estimated depth.

[0022] Figure 6 The influence of different β types on the proposed model is visualized on the DENS-Fog (light) dataset for the method of the present invention. Here, only the influence on depth estimation is visualized.

[0023] Figure 7 The figure shows the defogging and depth estimation effects of the method of the present invention on a continuous video frame. DETAILED DESCRIPTION

[0024] The present invention proposes a video defogging and depth estimation method in a real mobile (driving) scene, which provides a clearer field of view and distance perception for real foggy scenes, such as driving scenes and mobile phone photography in real foggy weather. Figure 2 As shown in Figure 2, the joint video defogging depth estimation has better results than the separate depth estimation in foggy days and the defogging first and then the depth estimation. This method is named DCL (video defogging and self-supervised depth estimation with depth as the core optimization), which includes training and testing stages. In the training stage, three adjacent foggy images I and three adjacent non-aligned and clear reference images J are input. ref (corresponding to the fog image), where the three adjacent non-aligned reference frames are obtained by matching the non-aligned reference frames. This method uses three main loss functions and two regularization constraints to train four sub-networks, namely, the defogging network Φ J , the depth estimation network Φ d , scattering coefficient estimation network Φ β and the camera pose prediction network Φ p , reconstruction loss function, non-aligned reference loss function and brightness consistency constraint loss. Based on the above four sub-networks, the reconstruction loss function is composed of I rec The perceptual, structural similarity and L1 loss of I are composed of rec To reconstruct the smoke image (calculated by the atmospheric scattering model). More importantly, this method combines the brightness consistency constraint (BCC) and the atmospheric scattering model (ASM) to jointly optimize the depth estimation network Φ d , and it is completed in a self-supervised manner. The brightness consistency constraint here uses the defogging adjacent frames as input to obtain a more accurate camera pose, so as to more accurately use the estimated depth map to complete the reprojection and construct a self-supervised estimation method. With the self-supervised learning of the depth estimation network, we have completed the effective decoupling of each component of the entire ASM respectively. At the same time, we use each component to reconstruct the fog frame, forming a reconstruction loss constraint to supervise the learning of the entire framework. In this learning framework, our main innovation is to use the depth information shared by ASM and BCC to build a jointly optimized learning framework. In the test use stage, you only need to input a real foggy test image to use the defogging network Φ J and the depth estimation network Φ d Quickly repair its clear scene. The attached figure shows the video defogging effect of DCL in a real scene, and it has achieved the current best effect.

[0025] The method of the present invention is described in detail below with reference to the accompanying drawings.

[0026] Combination Figure 1, a video defogging and depth estimation method in a real moving scene, the real moving scene of the present invention refers to a driving environment. Input sample pair (real RGB smoke image I and non-aligned and clear reference image J′ t ), the steps are as follows:

[0027] Step 1: Preprocess the dataset. First, use the non-aligned matching algorithm (NRFM) to obtain the matching information between all foggy video frames and clear reference frames. Then, according to the calibrated camera intrinsic parameters, dedistort the data and crop it from 1920×1080 to 1600×512.

[0028] Step 2: Transform the continuous fog video frames I [t-n:t+n] ∈R 3×h×w (n=1) Input to the dehazing network Φ J The dehazing result is obtained in [J t ,J s ](J t represents the current defogging frame, J s Indicates J t The neighboring defogging frame), the current foggy video frame I t Input into the depth estimation network Φ respectively d Medium estimate d t ∈R 1×h×w , and the scattering coefficient estimation network Φ β Estimated β t ∈R 1×h×w Then use the dark channel method to calculate the value of the infinite atmospheric light A ∞ ;

[0029] Step 3: The defogging result of step 2 t Then input into the camera pose prediction network Φ p To predict the camera’s pose information P x→y ∈R 4×4 , and then use Φ d The estimated d t and Φ p Get P x→y The depth d is estimated by reprojecting the adjacent dehazed frames onto the current frame and then performing self-supervised learning by calculating the photometric loss constrained depth estimation. t .

[0030] Step 4: J generated according to steps 2-3 t ,A ∞ ,d t and β t Reconstruct the current foggy video frame I′ using the atmospheric scattering model t , then define I′ t and I tThe reconstruction loss function Contains the sum of three loss functions, namely the first-order norm loss Perceived loss and structural similarity loss in Use VGG16 to calculate the perceived content.

[0031] Step 5: In addition to the reconstruction loss in step 4 In addition, this method also uses the t and J′ t The non-aligned reference loss function

[0032] Step 6. In addition to the loss constraints in steps 4 and 5, this method also proposes to use the discriminator network to respectively J and the depth estimation network Φ d Regular constraints are performed, namely D MFIR and D MDR . They are used to constrain Φ J Recover more high-frequency details and constrain Φ d The "black holes" problem that occurs in weak texture areas.

[0033] Step 7: Based on the two loss functions of steps 4, 5, and 6 and and two discriminator networks (D MFIR and D MDR ) adversarial loss regularization is used to optimize the network parameters (model training) of the entire DCL framework to obtain the defogging result. Finally, the real RGB fog video frame image I is input for testing t , respectively input into the dehazing network and the depth estimation network, and directly generate a clear scene image result J t and the depth map d t Finally, we show the effect of dehazing and depth estimation on a continuous video. Figure 7 shown.

[0034] Preferably, the data set is preprocessed in step 1. First, the matching information between all foggy video frames and clear reference frames is obtained using a non-aligned matching algorithm (matching the most similar clear reference frame), and then the data is dedistorted according to the calibrated camera internal parameters, and the size is cropped from 1920×1080 to 1600×512. (Ensure that the camera calibration parameters are accurate)

[0035] Preferably, step 2 uses three networks to estimate the atmospheric scattering model parameters respectively, using the depth estimation network Φ d Estimated t , dehazing network Φ J Prediction Jt and the scattering coefficient estimation network Φ β Estimated β t , where d t is the depth map of the current frame, β t is the scattering coefficient of the current frame.

[0036] The continuous fog video frames I [t-n:t+n] ∈R 3×h×w (n=1) Input to the dehazing network Φ J The dehazing result is obtained in [J t ,J s ](J t represents the current defogging frame, J s Indicates J t The neighboring defogging frame), the current foggy video frame I t Input into the depth estimation network Φ respectively d Medium estimate d t ∈R 1×h×w , and the scattering coefficient estimation network Φ β Estimated β t ∈R 1×h×w Then, the atmospheric light value A is calculated using the dark channel (DCP) method. ∞ . (Φ d and Φ β is a shared encoder network).

[0037] Preferably, step 3 uses the adjacent frames after defogging to estimate the camera posture, and constrains the depth estimation network Φ by reprojection. d Self-supervised learning.

[0038] The dehazing result J in step 2 t Then input into the camera pose prediction network Φ p To predict the camera’s pose information P x→y ∈R 4×4 , and then use Φ d The estimated d t and Φ p Get P x→y The depth d is estimated by reprojecting the adjacent dehazed frames onto the current frame and then performing self-supervised learning by calculating the photometric loss constrained depth estimation. t (x,y refer to the pixel coordinates of the current frame and the adjacent frame respectively).

[0039] Preferably, in step 4, J generated by steps 1-3 is used t ,A ∞ ,d t and β t , using the atmospheric scattering model to calculate and reconstruct fog video frames Then, using the real fog video frame I t Define the reconstruction loss function And supervised learning on texture and color, where Use VGG16 to calculate the perceived content (select the feature calculation content loss after the Relu layer).

[0040] Preferably, in step 5, t and J′ t The non-aligned reference loss function Regular constraints are imposed on the feature distribution of color and texture of the dehazing network, where Using VGG16 network to calculate J t and J′ t The content loss between .

[0041] Step 6. In addition to the loss constraints in steps 4 and 5, this method also proposes to use the discriminator network to respectively J and the depth estimation network Φ d Regular constraints are performed, namely D MFIR and D MDR . They are used to constrain Φ J Recover more high-frequency details and constrain Φ d The "black holes" problem that occurs in weak texture areas.

[0042] Preferably, step 7 is based on the two loss functions of steps 4, 5 and 6 and and two discriminator networks (D MFIR and D MDR ) adversarial loss regularization is used to optimize the network parameters (model training) of the entire DCL framework to obtain video defogging and depth estimation results. Finally, the real RGB fog video frame I is input for testing t To the defogging network Φ J and the depth estimation network Φ d , directly generate clear dehazing J t And the depth estimation result d t .

[0043] The present invention establishes a video dehazing and depth estimation framework (ASM-BCC) based on the atmospheric scattering model with brightness consistency constraint by sharing the same depth information between the atmospheric scattering model and the self-supervised depth estimation, and optimizes the two tasks at the same time. And two regularized discriminator networks D MFIR and D MDR , respectively to enhance the high-frequency details of the dehazing results and to avoid inaccurate depth values ​​produced by weak textures, such as the “black hole” problem of weak road textures.

[0044] The present invention is described in detail below with reference to the embodiments.

[0045] Example

[0046] like Figure 1 As shown in the figure, a video defogging and depth estimation algorithm in a real mobile (driving) scene is given. First, a continuous fog video frame I is given. [t-n:t+n] ∈R 3×h×w (n=1), and then use the non-aligned reference frame matching algorithm to match a clear reference frame with the highest similarity for each frame in this continuous video frame sequence. Then we input the foggy video frames into different atmospheric scattering model parameter prediction networks. First, input to the defogging network Φ J The dehazing result is obtained in t ,J s ](J t represents the current defogging frame, J s Indicates J t Next, the current foggy video frame I t Input into the depth estimation network Φ respectively d Medium estimate d t ∈R 1×h×w , and then the scattering coefficient estimation network Φ β Estimated β t ∈R 1×h×w Finally, the atmospheric light value A is calculated using the dark channel (DCP) method. ∞ Next, we will construct a reconstruction loss function by using the predicted atmospheric scattering model (ASM) parameters to reconstruct the foggy video frame. To constrain the learning of the entire framework, in order to ensure that each parameter variable can be fully disentangled, we established a t and J′ t The non-aligned reference loss function As a further supervision training signal for the dehazing network, the joint and The network parameters are optimized to obtain high-quality desmog images (in terms of color, texture, and brightness).

[0047] More importantly, we combine the atmospheric scattering model (ASM) and brightness consistency constraint (BCC) to build a joint optimization paradigm for video dehazing and depth estimation. We use the dehazed video frames as the camera pose network Φ p Input to predict more accurate camera pose information P x→y ∈R 4×4 The more accurate camera pose provides a more effective supervision signal for constructing the self-supervisory constraints of reprojection. And the reconstruction of atmospheric scattering provides a more accurate supervision signal for the depth estimation network Φd It also provides a reconstruction constraint based on the atmospheric scattering model, so the learning process between video dehazing and depth estimation is depth-centric and mutually compensated. We named this learning framework that uses depth information as the center to simultaneously optimize video dehazing and depth estimation on real foggy videos as DCL. In addition, we also proposed two discriminator networks D MFIR and D MDR , are respectively used to enhance the high-frequency details in the dehazed video frames, and to constrain the "black hole" problem that often occurs when predicting depth in weak texture areas, such as pure black road areas.

[0048] The specific implementation steps of DCL are as follows: (Note that the size representation of the fog video frame is for a batch size of 1.)

[0049] Step 1: Preprocess the training data set. First, take the camera that collects the data, obtain the camera internal parameters, use the camera internal parameters to dedistort the video frames of the data set, and then crop the original video frame from 1920×1080 to 1600×512 resolution that meets the normal driving perspective.

[0050] Step 2: For a given fog video frame segment I [t-n:t+n] ∈R 3×h×w (n=1), the current frame is I t , its adjacent frame is I s∈[t-n:t+n],s≠t ∈R 3×h×w , where h is height and w is width. In this step, we combine the atmospheric scattering model and brightness consistency to build an ASM-BCC joint model. We define this ASM-BCC as follows:

[0051]

[0052] Among them, J t is the defogging result corresponding to the current foggy video frame, which is what the model wants to predict. s isI s Corresponding neighboring frames for dehazing. Represents a bilinear interpolation sampling operation. x and y are the position coordinates of the corresponding pixel points in the current frame and the adjacent frame respectively. K represents the intrinsic parameter of the camera.

[0053] Step 3: Figure 1 Here we will define some different networks to predict the parameters J of the atmospheric scattering model respectively. t ,d t and β t We define three different networks Φ J , Φ d and Φ βTo estimate these three key parameters separately, as well as a posture estimation network Φ p To predict the camera’s pose information P x→y ∈R 4×4 Here the atmospheric light A ∞ It is obtained by DCP calculation. We formulate this process as follows:

[0054]

[0055] Among them, Φ J is the defogging network, Φ d is the depth estimation network, Φ β is the scattering coefficient estimation network. Next, we use the estimated depth information d t and the camera pose P x→y Go to reconstruct and get the current defogging frame J t The specific process is as follows:

[0056]

[0057] Among them, SSIM represents the structural similarity loss, Indicates bilinear interpolation sampling operation, α = 0.85 which is the default setting. a is a mask obtained by the automatic masking strategy. In addition, in order to obtain a smoother depth estimation result in this step, we use the edge smoothing loss to constrain Φ d The estimated depth, the specific constraints are expressed as follows:

[0058]

[0059] here, represents the mean normalized inverse depth, and Represent the gradients in the horizontal and vertical directions respectively.

[0060] Step 4: Figure 1 As shown, in step 3, the J predicted in the above step is used t ,d t and β t Bring it into the atmospheric scattering model to calculate and reconstruct the fog video frame Reusing real fog video frames I t , construct three loss functions ( and ) for the reconstructed I′ t Apply texture and color constraints Get realistic reconstruction results. The content loss is calculated using VGG16 (select the Relu layer and then calculate the content loss, the specific number of layers: 3, 8, 15, 22, 29). Note: The loss formula is for a sample pair.

[0061]

[0062] The parameters θ, λ, and η are set to 1 by default. l (I′ t ) and Φ l (I) represent VGG16 network extraction I rec and the feature map corresponding to I at layer l, N represents the number of extracted feature layers, and Represents image I′ t The mean of and Represents image I t The variance of Represents image I′ t and I t The covariance of .

[0063] Step 5: Figure 1 As shown, the present invention uses non-aligned reference loss Constrained generated J t The color and texture of the image are closer to the reference image. The same VGG16 network is used to calculate the clear scene graph J t and the non-aligned reference frame J′ t The content loss between .

[0064]

[0065] Among them, J′ t is a non-aligned clear reference frame image, J t It is a clear scene graph generated by the model of the present invention. is the content similarity between image features, Ω l (J t ) and Ω l (J′ t ) represent images extracted by VGG16 network J t and J′ t The corresponding feature map at layer l.

[0066] Step 6: Figure 1 As shown in Figure 2, we propose two identification networks D MFIR and D MDR Results of deregularized video dehazing and depth estimation. MFIR Used to enhance the high-frequency details of the dehazing results, DMDR The depth used for regular estimation is the "black hole" problem that will appear in weak texture areas, such as pure black road surfaces. We define D MFIR and D MDR as follows:

[0067]

[0068] here It is a normalization operation to remove the scale and make the training model more stable. cat(a,b) means concatenating a and b. Denotes the dehazed video frame J extracted using Haar wavelet t High frequency characteristics and And cascade up. Denotes the extraction of non-aligned clear reference frames using Haar wavelet t High frequency characteristics and And cascade up. J is the dehazing result discriminator network, D d is the depth estimation result discriminator network, is an expectation function, is the loss of the dehazing result on the discriminator, is the loss of the dehazing result on the generator, is the loss of depth estimation on the discriminator, is the loss of depth estimation on the generator.

[0069] Step 7: Figure 1 As shown in the figure, according to the loss and regularization constraints of steps 4, 5 and 6, the network parameters of the entire DCL framework are optimized (model training) to obtain a clear scene graph after training (J t ) to generate the network.

[0070] This paper compares several existing SOTA dehazing methods, namely DCP, RefineNet, CDD-GAN, D 4, PSD, RIDCP, PM-Net, MAP-Net, NSDNet and DVD. We first evaluate the proposed DCL on three video dehazing benchmark datasets, namely GoProHazy, DrivingHazy and InternetHazy. It is worth noting that during testing, our DCL model only requires the dehazing network. The results under unpaired, paired and non-aligned settings are summarized in Table 1. Overall, our DCL ranks first among all methods in terms of FADE and NIQE. For example, the NIQE score of DCL is 22.04% higher than that of the second-place non-aligned method DVD (Fan et al. 2024), and it also significantly surpasses other top methods in paired and unpaired methods. Figure 3 Visual comparisons with PM-Net (Liu et al. 2022b), RIDCP (Wu et al. 2023), NSDNet (Fan et al. 2023), MAPNet (Xu et al. 2023), and DVD (Fan et al. 2024) are shown. While these methods generally produce visually pleasing dehazing results, our DCL recovers sharper predictions with more accurate content and contours. In addition, our DCL is also able to estimate effective depth, a feature not present in other dehazing methods.

[0071] Table 1 shows the quantitative results of three real-world blurry video datasets. ↓ indicates that lower is better. For the dehazing methods that rely on real values ​​for training, we use It is trained on the GoPro dataset. The DrivingHazy and InternetHazy datasets are tested using the dehazing model trained on GoProHazy. Note that all quantitative results are evaluated at an output resolution of 640×192.

[0072] Table 1: Quantitative results on three real-world blurry video datasets

[0073]

[0074]

[0075] In addition to the image dehazing and video dehazing methods compared in the above table, in order to verify the effect of our proposed model on depth estimation, we also compare it with the current SOTA self-supervised depth estimation method and the self-supervised depth estimation method under extreme environments. To further verify the performance on the depth estimation task, we compare DCL with well-known self-supervised depth estimation methods, including MonoDepth2 (Godard et al. 2019), MonViT (Zhao et al. 2022), RobustDepth (Saunders, Vogiatzis, and Manso 2023), Mono-ViFI (Liu et al. 2024) and Lite-Mono (Zhang et al. 2023). Since blurry data with depth annotations are very scarce, we train these methods on the GoProHazy benchmark and then test them on the DENSE-Fog dataset. It is worth noting that in the test phase, our DCL only uses the depth estimation subnetwork. Table 2 reports the numerical results. Overall, our DCL shows the best performance on almost all five metrics in both light and dense fog scenes. In dense fog scenes, the accuracy and error metrics are inconsistent because the blurred depth is closer to the mean of the true value, resulting in smoother results. Figure 4 Visual results of MonoViT, Mono-ViFI and DCL are shown. It can be found that the depth estimated by MonoViT and Mono-ViFI is blurry and unreliable. In contrast, our DCL is able to predict more reasonable and accurate depth results. In addition, DCL can also provide clean dehazed images.

[0076] Table 2 shows the quantitative results. We compare our framework with previous state-of-the-art methods on the DENSE-Fog dataset. All methods are trained on the GoProHazy dataset. It is worth noting that RobustDepth is trained on clear reference videos in GoProHazy because it uses its own synthetic fog for training.

[0077] Table 2: Comparison of quantitative results

[0078]

[0079] In addition to the above comparisons, we also compared the efficiency of the models. We compared the number of parameters, FLOPs, and inference time of cutting-edge methods in image / video dehazing and self-supervised depth estimation tasks, and the tests were performed on an NVIDIA RTX4090 GPU. The running time is calculated when the input size is 640×192. As shown in Tables 1 and 2, our method achieves the shortest inference time of 0.075 seconds and 0.009 seconds in image / video dehazing and self-supervised estimation tasks, respectively, which shows that DCL performs well in fast inference. All these facts prove that the proposed DCL performs very well based on our ASM-BCC model.

[0080] Table 3: Ablation core module on DENSE-Fog (light) dataset

[0081] Method BCC <![CDATA[D MFIR ]]> <![CDATA[D MDR ]]> Abs Real↓ RMSE log↓ <![CDATA[δ1↑]]> DCL w / o BCC √ √ 0.636 0.569 0.439 <![CDATA[DCLw / oD MFIR ]]> √ √ 0.320 0.366 0.621 <![CDATA[DCL w / oD MDR ]]> √ √ 0.340 0.392 0.562 DCL(Ours) √ √ √ 0.311 0.364 0.623

[0082] In order to evaluate our proposed BCC,D MFIR and D MDR To test the effectiveness of , we conducted experiments by excluding each component and training our model. Table 3 and Figure 5 The results in show that the video dehazing effect is significantly improved after integrating our BCC module. This improvement comes from using the dehazed image to enforce the brightness consistency constraint, which allows better depth estimation (d) and more effective haze removal. In addition, D MFIR and D MDR It also contributes to the improvement of dehazing and depth estimation.

[0083] Table 4: Ablation of different loss functions on the GoProHazy dataset.

[0084]

[0085] In order to evaluate the impact of different loss functions on the proposed model, we conducted corresponding ablation experiments to verify the loss function and Effectiveness on GoProHazy. The results of FADE and NIQE are reported in Table 4. Obviously, plays a key role because the ASM model is an important physical mechanism for video dehazing in formula (1), which ensures the independence between the dehazing result and the unaligned clear reference frame. In addition, the smoothness loss It also made important contributions to video dehazing and depth estimation.

[0086] Table 5: Comparison of different β types on the DENSE-Fog (light) dataset

[0087] Shape of β Type Abs Real↓ RMSE log↓ <![CDATA[δ1↑]]> (1,1,1) Constant 0.325 0.371 0.621 (1,192,640)(Ours) Non-uniform 0.311 0.364 0.623

[0088] In the description of the method section, we assume that β is a non-uniform variable, mainly because the haze in real scenes is usually non-uniform, such as local haze. Therefore, the scattering coefficient varies between different regions. Since existing studies mainly use constant scattering coefficients, in this study, we conducted comparative experiments on different beta values ​​(i.e., constant or non-uniform) in Table 5. The results show that non-uniform β can lead to better depth estimation accuracy. We Figure 6 A visual comparison is further shown in .

Claims

1. A video defogging and depth estimation method in a real mobile scene, characterized in that: include: Step 1: Preprocess the data set; The matching relationship between all foggy video frames and clear reference frames is obtained by using the non-aligned reference matching algorithm. The data is dedistorted and cropped according to the calibrated camera intrinsic parameters. Step 2: Transform the continuous fog video frames I [t-n：t+n] ∈R 3×h×w Input to the dehazing network Φ J The dehazing result is obtained in [J t , J s ], J t represents the current defogging frame, J s Indicates J t Neighboring defogging frames, n = 1; current foggy video frame I t Input into the depth estimation network Φ respectively d Medium estimate d t ∈R 1×h×w , and the scattering coefficient estimation network Φ β Estimated β t ∈R 1×h×w , using the dark channel method to calculate the value of the infinite atmospheric light A ∞ ; Step 3: The defogging result of step 2 t Then input into the camera pose prediction network Φ p To predict the camera’s position information p x→y ∈R 4×4 , and then use Φ d The estimated d t and Φ p Get p x→y By reprojecting the adjacent dehazed frames onto the current frame, the depth d is estimated by self-supervised learning by calculating the photometric loss constrained depth estimation. t ; Step 4: J generated from step 2 to step 3 t , A ∞ , d t and β t , using the atmospheric scattering model to calculate and reconstruct the current fog video frame I′ t , then define I′ t and I t The reconstruction loss function It contains the sum of three loss functions, namely the first-order norm loss Perceived loss and structural similarity loss Step 5: Build about J t and J′ t The non-aligned reference loss function J t and J′ t The non-aligned reference loss function Regularize the color and texture feature distribution of the dehazing network; Step 6: Use the discriminator network to analyze the dehazing network Φ J and the depth estimation network Φ d Perform regularization constraints; Step 7: Based on the two loss functions of step 4, step 5, and step 6 and And two discriminator networks D MFIR and D MDR The adversarial loss regularization optimizes the network parameters of the entire DCL framework to obtain the defogging result; finally, the real RGB fog video frame image I is input for testing t , respectively input into the dehazing network and the depth estimation network, and directly generate a clear scene image result J t and the depth map d t .

2. The video defogging and depth estimation method in a real mobile scene according to claim 1, characterized in that: In step 1, the calibrated camera intrinsic parameters are used to dedistort and crop the video frame to a size of 1600×512.

3. The video defogging and depth estimation method in a real mobile scene according to claim 2, characterized in that: Step 2 uses three networks to estimate the atmospheric scattering model parameters respectively, using the depth estimation network Φ d Estimated t , dehazing network Φ J Prediction J t and the scattering coefficient estimation network Φ β Estimated β t , where d t is the depth map of the current frame, β t is the scattering coefficient of the current frame.

4. The video defogging and depth estimation method in a real mobile scene according to claim 3, characterized in that: Step 3 uses the adjacent frames after defogging to estimate the camera posture and constrains the depth estimation network Φ by reprojection. d Self-supervised learning.

5. The video defogging and depth estimation method in a real mobile scene according to claim 4, characterized in that: In step 4, the predicted J t , A ∞ , d t and β t Bring it into the atmospheric scattering model to calculate and reconstruct the fog video frame Reusing real fog video frames I t and reconstructed fog video frame I′ t , construct the reconstruction loss function to I' t Perform unsupervised learning of texture and color, where Use VGG16 to calculate the perceived content.

6. The method for video defogging and depth estimation in a real mobile scene according to claim 5, characterized in that: Multi-scale non-aligned reference loss in step 5 Supervised learning of color and texture for the dehazing network, where The VGG16 network is used to calculate J t and J′ t The feature distribution loss between .

7. The method for video defogging and depth estimation in a real mobile scene according to claim 6, characterized in that: Step 6 uses D MFIR and D MDR The dehazing results with richer high-frequency details are obtained by separately constraining the model, and accurate depth information is produced in the weak texture dehazing of the road surface.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the video defogging and depth estimation method in a real mobile scene as described in any one of claims 1-7 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the video defogging and depth estimation method in a real mobile scene as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for video defogging and depth estimation in a real mobile scene described in any one of claims 1 to 7 is implemented.

Images