Real scene image defogging method based on dual-branch CNN-Transformer and depth information fusion

Abstract

The invention discloses a real scene image defogging method based on dual-branch CNN-Transform and depth information fusion, and relates to the field of artificial intelligence image processing and machine vision. The method comprises the following steps: generating a depth map: inputting an RGB image into a monocular depth estimator to generate an estimated depth map, wherein the RGB image is a fog-containing image; designing an image defogging model architecture, wherein an image defogging model comprises a feature extraction decoder, a multi-mode feature fusion module and a feature recovery encoder and is used for extracting and synthesizing global features and local features from an RGB mode and a depth mode to carry out image defogging; designing a loss function to carry out model optimization; collecting a data set for model training; and verifying the effectiveness of the trained model by adopting an evaluation index. According to the invention, after the model is designed and trained through a deep learning method, haze removal of the fog-containing image can be quickly and effectively realized.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence image processing and machine vision technology, and specifically relates to a real scene image defogging method based on a double-branch CNN-Transformer and depth information fusion. Background Art

[0002] Haze is a ubiquitous atmospheric condition caused by light scattering caused by various suspended particles such as dust, smoke, and moisture. This scattering effect can significantly degrade the quality of images captured under such conditions, leading to problems such as reduced contrast, faded colors, and decreased clarity. This phenomenon not only damages the aesthetic value of photos, but also poses a huge challenge to visual processing systems, especially in tasks involving precision and accuracy, such as image recognition and analysis. Therefore, the field of image dehazing has become an important research area in computer vision and imaging science.

[0003] In the early stages of research, traditional methods used to enhance the visual quality of images affected by haze include techniques such as histogram equalization, partial histogram equalization, wavelet transform, and homomorphic filtering. These methods mainly focus on adjusting the contrast level to mitigate the effects of haze. However, a significant limitation of these methods is that they generally ignore the underlying formation process of haze in the image, and the improved images still have poor visual appeal and obvious loss of details.

[0004] Based on traditional image enhancement technology, the emergence of atmospheric scattering model (ASM) marks a major progress in the field of image defogging. By obtaining accurate transmission map and atmospheric illumination intensity, foggy images can be restored to clean images. However, methods based on atmospheric scattering model are usually difficult to adapt to all scenarios. When the actual situation violates the prior conditions, inaccurate transmission map estimation will be generated, resulting in poor defogging effect.

[0005] The development of deep learning theory has brought new ideas to the field of image defogging. By constructing an intermediate layer between foggy and fog-free images, clear fog-free images can be directly output in an end-to-end manner. However, their training often requires a large number of paired clear and foggy image samples. In practical application scenarios, it is extremely difficult to obtain such paired data. For this reason, Koschmieder's law is usually used to artificially simulate foggy images, and a large training data set is constructed in this way. However, there are still significant differences between the foggy images generated by Koschmieder's law and the foggy images in the real world. Therefore, although these end-to-end deep learning models perform well on synthetic data sets, it is still difficult for them to achieve satisfactory defogging results when processing real-world scenes.

[0006] Due to the incompleteness of scene datasets and the performance limitations of neural networks, it is difficult to accurately obtain transmission maps and atmospheric light based on atmospheric scattering models. However, atmospheric scattering models have important guiding significance for the image dehazing task itself. The scene depth is essentially related to the haze distribution, and there is a close relationship between the haze distribution and the image depth. However, existing models usually focus on the color space and ignore the depth information, resulting in these models being unable to effectively process images with uneven haze distribution or high concentration in the real world. This shows that single-modal methods have limitations in capturing the multidimensional nature of haze, and incorporating depth information into the dehazing process is the key to improving the effectiveness of applications in complex real-world scenes.

[0007] With the continuous advancement of depth cameras and the continuous improvement of monocular depth estimation accuracy, obtaining scene depth is no longer a difficult problem. In the past two years, some scholars have been filling the gaps in related research. However, these methods mostly use multi-scale deep fusion technology based on the U-Net architecture, which is easy to lose important information such as subtle textures and edge details when extracting features, and also perform poorly when processing spatial information. What's more serious is that when fusing information from different sources of RGB images and depth map information, the complementarity between the two modalities is not effectively utilized, which limits the improvement of the dehazing effect.

[0008] Most existing image dehazing methods mainly rely on a single image color space to learn haze features. This approach often leads to unsatisfactory dehazing effects when dealing with complex real scenes, especially in terms of color and edge information retention. In fact, the distribution of haze is closely related to image depth information, which reveals the light attenuation and depth perception in the scene. In order to solve this problem, the present invention proposes an innovative image dehazing technology, which is based on the structure of a dual-branch CNN and a Transformer to achieve effective fusion of multimodal features. This method aims to efficiently integrate the feature space of foggy color images and the feature space of depth information. By mining the complementarity of these two modal information, it can more accurately guide the network to identify haze features in the depth space, especially in complex real environments, especially in the face of non-uniform fog and dense fog, to improve the recognition and processing capabilities of the model. Summary of the invention

[0009] The purpose of the present invention is to provide a real scene image dehazing method based on a dual-branch CNN-Transformer and depth information fusion, so as to solve the problem that when fusing different source information from RGB image and depth map information in the prior art, the complementarity between the two modalities cannot be effectively utilized.

[0010] To achieve the above purpose, the technical solution adopted by the present invention is:

[0011] The first aspect of the present invention proposes a real scene image dehazing model based on a dual-branch CNN-Transformer and deep information fusion, the model includes a feature extraction decoder, a multimodal feature fusion module and a feature recovery encoder; namely, DehazeFuse;

[0012] The feature extraction decoder includes a shallow feature extraction unit and a deep feature extraction unit; the shallow feature extraction unit adopts a reversible neural network module with an affine coupling layer; the deep feature extraction unit adopts a Restormer module, and the Restormer module has an MDTA module and a GDFN module as basic components;

[0013] The multimodal feature fusion module adopts a multimodal deep cross attention mechanism and a convolutional layer with a spatially invariant filter;

[0014] The feature recovery encoder includes a deep feature reconstruction unit including a P-layer Restormer module and an image reconstruction unit consisting of a reversible neural network module.

[0015] The second aspect of the present invention proposes a real scene image defogging method based on a dual-branch CNN-Transformer and depth information fusion, comprising the following steps:

[0016] S1. Generate a depth map. Input the RGB image into a monocular depth estimator to generate an estimated depth map. The RGB image is a foggy image.

[0017] S2. Design the image defogging model architecture; the image defogging model includes a feature extraction decoder, a multimodal feature fusion module and a feature recovery encoder, which are used to extract and integrate global features and local features from RGB modality and depth modality for image defogging;

[0018] Among them, the multimodal feature fusion module adopts a multimodal deep cross-attention mechanism for global information exchange between the RGB modality and the depth modality, and uses a convolutional layer with a spatially invariant filter to aggregate the local information of the RGB modality and the depth modality;

[0019] S3. Design loss function to optimize the model;

[0020] S4, collect data sets for model training;

[0021] S5. Use evaluation indicators to verify the effectiveness of the trained model.

[0022] Preferably, the feature extraction decoder comprises a shallow feature extraction unit and a deep feature extraction unit;

[0023] The shallow feature extraction unit extracts shallow features from the foggy image H and its corresponding depth map D and The details are as follows:

[0024]

[0025] Among them, S(.) is the shallow feature extraction unit;

[0026] The deep feature extraction unit extracts features from the shallow features and Further extract deep features and The details are as follows:

[0027]

[0028] in, It is a deep feature extraction unit.

[0029] Furthermore, the shallow feature extraction unit adopts a reversible neural network module with an affine coupling layer, and the feature extraction is specifically as follows:

[0030]

[0031] where ⊙ is the Hadamard dot product, are the 1st to cth channels of the input features of the kth reversible layer (k=1,…,K), is a channel connection operation, is an arbitrary mapping function; in each reversible layer, Can be set to any mapping without affecting the lossless information transmission of the reversible layer; It can also be obtained in the same way.

[0032] Furthermore, the deep feature extraction unit adopts the Restormer module, which has the MDTA module and the GDFN module as basic components. The feature extraction is specifically as follows:

[0033] The MDTA module first generates The subsequent process is summarized as follows:

[0034]

[0035] Among them, F and are the feature maps of input and output; It is a 1×1 point-by-point convolution; The matrix is ​​in the original size The result after reshaping the tensor; α is a learnable scaling parameter used to control the size of the dot product of K and Q before the softmax function;

[0036] Given an input tensor The GDFN module is represented as:

[0037]

[0038] Among them, ⊙ represents element-by-element multiplication, φ represents GELU nonlinearity, and LN is the normalization layer; is a 1×1 point-wise convolution, It is a 3×3 depth convolution;

[0039] The complete unfolding of the deep feature extraction process is as follows:

[0040]

[0041] Among them, Z is the output of the deep feature extraction unit.

[0042] Preferably, the multimodal feature fusion module adopts a multimodal deep cross attention mechanism for RGB modality F H With the depth modality F D The global information exchange between them is as follows:

[0043] For the RGB modality By combining the and Attention weights are performed to merge cross-modal information, while the information in the RGB modality is retained through residual connections to obtain the fused feature Z H , and vice versa, we get the fusion feature Z D ; The process is as follows:

[0044]

[0045] Among them, MDTCA is a multimodal deep cross attention mechanism; LN is a normalization layer.

[0046] Furthermore, a convolutional layer with spatially invariant filters is deployed to aggregate local information of the RGB modality and the depth modality as follows:

[0047] Z DF =Conv(Concat(Z H , Z D )),

[0048] Where Conv(·) represents a convolutional layer with a spatially invariant filter, Concat(·) represents a connection in the channel dimension; Z DF Represents the fused deep features.

[0049] Preferably, the feature recovery encoder comprises a deep feature reconstruction unit and an image reconstruction unit;

[0050] The deep feature reconstruction unit deploys a P-layer Restormer module, which is used to refine the fused deep features and restore the fused shallow features from a global perspective. The details are as follows:

[0051] Z SF =DR(Z DF )

[0052] Among them, DR(.) is the deep feature reconstruction unit;

[0053] The image reconstruction unit is composed of reversible neural network modules; the details are as follows:

[0054]

[0055] in, is the dehazed image, and IR(.) is the image reconstruction unit.

[0056] Preferably, the loss function in S3 is a comprehensive loss function including SSIM loss, Gradient loss and Color loss; its formula is:

[0057]

[0058] Among them, λ1, λ2, and λ3 are hyperparameters that control the trade-off of each sub-loss term.

[0059] Furthermore, SSIM loss Used to constrain the dehazed image The structural consistency between the original clear image I is as follows:

[0060]

[0061] Gradient loss Lgrad is used to maintain the sharpness of the image edge by minimizing the difference between the gradient of the dehazed image and the original haze-free image, as follows:

[0062]

[0063] in, represents the Sobel gradient operator;

[0064] Color loss L color First, Gaussian filtering is applied to the dehazed image and the haze-free image to obtain their blur representations, and then the color loss is achieved by calculating the mean square error between the two, as follows:

[0065]

[0066] Here, G(·) represents the image representation after applying the Gaussian filter, and MSE is the mean square error.

[0067] Preferably, the data set in S4 adopts Dense-HAZE, NH-HAZE, I-HAZE, and O-HAZE data sets, selects samples from the data set, uses the RGB images in the samples and the corresponding depth maps generated by the monocular depth estimator as training sets, and uses the training sets to train the image dehazing model.

[0068] Preferably, the evaluation index in S5 includes peak signal-to-noise ratio PSNR and structural similarity index SSIM.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] (1) The real scene image dehazing method based on dual-branch CNN-Transformer and deep information fusion in the present invention adopts DehazeFuse, which is an advanced multimodal feature fusion dehazing framework based on dual-branch CNN and Transformer structure. This framework can efficiently extract and integrate global and local features from multiple modalities, significantly improving the dehazing performance in real environments. Experimental results show that the method in the present invention performs well in multiple complex real scenes, showing excellent image clarity restoration and detail preservation capabilities.

[0071] (2) The present invention designs a multimodal feature fusion module based on the cross-attention mechanism, which can effectively fuse the common features and unique features of foggy images and depth images, thereby enhancing the model's adaptability to complex scenes and its ability to capture details.

[0072] (3) The present invention also proposes a new composite loss function, which aims to fully restore the true color, contrast and texture details of the image affected by fog, thereby improving the quality and authenticity of the dehazed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a flow chart of the real scene image defogging method in the present invention;

[0074] Figure 2 It is a structural block diagram of a shallow feature extraction unit in a feature extraction encoder in the present invention;

[0075] Figure 3 It is a structural block diagram of a deep feature extraction unit in a feature extraction encoder of the present invention;

[0076] Figure 4 It is a structural block diagram of the multimodal feature fusion module in the present invention;

[0077] Figure 5A schematic diagram of a foggy image and its corresponding depth map and a real image and its corresponding depth map in the present invention;

[0078] Figure 6 A visual comparison of the results of various dehazing methods on the I-HAZE dataset;

[0079] Figure 7 A visual comparison of the results of various dehazing methods on the O-HAZE dataset;

[0080] Figure 8 A visual comparison of the results of various dehazing methods on the NH-HAZE dataset;

[0081] Fig. 9 A visual comparison of the results of various dehazing methods on the Dense-HAZE dataset;

[0082] Fig.10 The figure is a comparison chart of the subjective visual effects of different configuration models. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] Embodiment 1:

[0085] See also Figure 1 , a real scene image dehazing method based on dual-branch CNN-Transformer and depth information fusion, adopts a monocular depth estimator and an image dehazing model (DehazeFuse). DehazeFuse aims to restore the haze-free image I by merging the local and global complementary information in the hazy RGB image H and the depth image D. and Represent the foggy RGB image and its depth map respectively. Represents the restored haze-free image.

[0086] In this embodiment, a monocular depth estimator;

[0087] The monocular depth estimator adopts a model based on RGB estimated depth map. The foggy image H is sent into the network to generate an estimated depth map, namely, the depth map D.

[0088] The DehazeFuse model is divided into three parts: feature extraction decoder, multimodal feature fusion module, and image restoration decoder. The overall network architecture is as follows: Figure 2-4 shown.

[0089] In this embodiment, the feature extraction decoder includes a shallow feature extraction module and a deep feature extraction module, which are as follows:

[0090] Specifically, the shallow feature extraction unit:

[0091] The ability to capture local semantic information in a concise and effective way and transform this information is due to the high efficiency of the convolutional layer in early visual processing, which brings stability to the optimization process and improves the optimization results. The convolutional layer is a representation in a high-dimensional feature space. Based on this advantage, the present invention initially uses a convolutional neural network (CNN) to extract shallow features from the foggy image H and its corresponding depth map D and The process can be summarized by the following expression:

[0092]

[0093] Among them, S(.) is a shallow feature extraction unit. The shallow feature extraction unit adopts the INN module.

[0094] In view of the importance of detail features such as edges and textures in the image dehazing process, the shallow feature extraction module designed by the present invention utilizes the CNN architecture, aiming to retain these key detail information to the greatest extent. Therefore, the present invention introduces an invertible neural network (INN) module, such as Figure 2 The uniqueness of the INN module is that it can realize the mutual generation between input and output features, ensuring that the integrity of the input information is well protected. Therefore, the module can be regarded as a lossless feature extraction tool, which is very suitable for use in this application scenario. The present invention uses an INN block with an affine coupling layer, such as Figure 2 As shown, at each reversible layer, the transformation process is as follows:

[0095]

[0096] where ⊙ is the Hadamard dot product, are the 1st to cth channels of the input features of the kth reversible layer (k=1,…,K), is a channel connection operation, is an arbitrary mapping function. In each reversible layer, It can be set to any mapping without affecting the lossless information transmission of the reversible layer. Considering the trade-off between computational consumption and feature extraction capability, the present invention adopts the bottleneck residual block (BRB) in MobileNetV2 as It can also be obtained in the same way, just change H in the formula to D.

[0097] Specifically, the deep feature extraction unit:

[0098] After this, the present invention starts from the shallow characteristics and Further extract deep features and The details are as follows:

[0099]

[0100] in, It is a deep feature extraction unit.

[0101] Restormer is considered to be very suitable for high-resolution foggy image restoration tasks in real scenes because it can extract global features across feature dimensions through the self-attention mechanism without significantly increasing the computational burden. Figure 3 As shown in Figure 1, Multi-Dconv Head Transposed-based Attention (MDTA) is the basic component of designing this deep feature extraction unit. Given a feature F of size H×W×C, the idea of ​​generating F is similar to the self-attention mechanism (SA). Q , F K and F V , then MDTA is first generated by projection

[0102]

[0103] MDTA further enhances the richness of local context. It is generated by applying 1×1 convolution to aggregate pixel-level cross-channel context, followed by 3×3 depth-wise convolution to encode channel-level spatial context. is a 1×1 point-wise convolution, is a 3×3 depth convolution. Subsequently, the query and key projections are reshaped so that their dot products interact to construct a convolution of size The transposed attention map A, rather than the conventional huge regular attention map The MDTA process is outlined as follows:

[0104]

[0105] Among them, F and are the feature maps of input and output; The matrix is ​​in the original size Here α is a learnable scaling parameter used to control the size of the dot product of K and Q before the softmax function. Similar to the traditional multi-head SA, the number of channels is divided into heads and separate attention maps are learned in parallel.

[0106] Next, the Gated-Dconv Feed-Forward Network (GDFN) is used to further refine the features produced by MDTA. The gating mechanism is implemented by the element-wise product of linear transformation layers on two parallel paths, one of which integrates the GELU nonlinear activation. Like MDTA, GDFN also contains deep convolution operations to encode information from spatially neighboring pixel locations, which is crucial for effectively recovering local image structures. Given an input tensor GDFN is represented as:

[0107]

[0108] Among them, ⊙ represents element-by-element multiplication, φ represents GELU nonlinearity, and LN is a normalization layer. Overall, GDFN ensures that each layer can effectively supplement and enhance the detailed information of other layers by precisely controlling the information flow between layers in the processing flow. This is different from MDTA, which mainly focuses on using contextual information to enrich feature representation. Therefore, the complete development of the deep feature extraction process is as follows:

[0109]

[0110] Among them, Z is the output of the deep feature extraction unit.

[0111] In this embodiment, the multimodal feature fusion module is specifically as follows:

[0112] After successfully extracting deep features rich in unimodal global semantic information, the present invention further designs a multimodal fusion module based on the attention mechanism, aiming to explore and integrate the guiding role of depth information on foggy RGB information. Figure 4 As shown in Figure 1, the core of this module is the cross-attention mechanism. The key difference between this module and the single-modal deep feature extraction unit is that multi-modal deep cross-attention (MDTCA) is used here instead of MDTA to achieve global context exchange between the RGB modality and the depth modality. Therefore, given the features F from the RGB modality and the depth modality H and F D , the whole process of the attention mechanism-guided adaptive multimodal fusion unit is defined as:

[0113]

[0114] As the formula indicates, for the RGB modality It does this by combining and Attention weights are performed to merge cross-modal information, while the information in the RGB modality is retained through residual connections to obtain the fused feature Z H ,vice versa.

[0115] After the cross self-attention guided multimodal fusion module, the present invention deploys a convolutional layer with a spatially invariant filter to aggregate local information of different modalities, which is intended to enhance the translation equivariance of the DehazeFuse of the present invention, which can be expressed as:

[0116] Z DF =Conv(Concat(Z H , Z D )),

[0117] Where Conv(·) represents a convolutional layer with spatially invariant filters and Concat(·) represents a connection in the channel dimension. DF Represents the fused deep features and is the input of the feature reconstruction module.

[0118] In this embodiment, the feature recovery encoder is as follows:

[0119] After fully integrating the complementary information of different modalities, the present invention further designs a Transformer-based deep feature reconstruction module and a CNN-based image reconstruction module to remap the fused deep features to the image space. This process is first implemented by deploying a deep feature reconstruction unit DR(·) containing a P-layer Restormer block, which aims to refine the fused deep features and restore the fused shallow features from a global perspective. The specific process can be described as follows:

[0120] Z SF =DR(Z DF )

[0121] Among them, in order to fully utilize the global context in the deep features to restore the fused shallow features, P is set to 4.

[0122] Next, the CNN-based image reconstruction unit IR(·) module is used to reduce the number of channels to achieve the reconstruction of a clear and fog-free image. The specific process is as follows:

[0123]

[0124] The image reconstruction unit is also composed of an invertible neural network INN.

[0125] The present invention further optimizes the above DehazeFuse using a loss function, and the loss function is specifically as follows:

[0126] Image dehazing is a complex problem, and the loss can only be determined by calculating the spatial distance between two tensors. Unlike most existing dehazing solutions, this paper comprehensively considers all positive factors that can improve the quality of the final dehazed images, making them clearer and more realistic. Therefore, this paper uses a comprehensive loss including SSIM loss, Gradient loss and Color loss, and its formula is:

[0127]

[0128] Among them, λ1, λ2, and λ3 are hyperparameters that control the trade-off of each sub-loss term.

[0129] SSIM loss: Considering that the structural similarity (SSIM) index is the most widely used metric, it reflects the image distortion from three aspects: light, contrast, and structure. To constrain the dehazed image and the structural consistency between the original clear image I. Specifically, the SSIM loss can be defined as:

[0130]

[0131] Among them, SSIM(·) represents the structural similarity operation, which measures the similarity between two images.

[0132] Gradient loss: In the task of image dehazing, it is crucial to maintain or restore image details. The gradient loss function aims to ensure that the dehazed image is as close to the original image as possible in terms of edge details and texture features, thereby retaining more detail information. By minimizing the difference between the gradient of the dehazed image and the original haze-free image, the gradient loss helps maintain the sharpness of the image edge and mitigate the blurring effect that may be introduced by the dehazing process. The specific formula of the gradient loss is as follows:

[0133]

[0134] in, Represents the Sobel gradient operator.

[0135] Color loss: Given that foggy images usually show lower brightness and contrast than fog-free images, the present invention uses this phenomenon as a priori knowledge and introduces color loss to evaluate the color difference between defogging images and fog-free images. This loss function aims to ensure that the defogging image meets human visual expectations in terms of brightness, thereby improving the realism of the image. Specifically, the present invention first applies Gaussian filtering to the defogging image and the fog-free image to obtain their blurred representations, and then realizes color loss by calculating the mean square error (MSE) between the two. The calculation formula is as follows:

[0136]

[0137] Here, G(·) represents the image representation after applying the Gaussian filter.

[0138] Using the above-mentioned model DehazeFuse, a real scene image dehazing method based on dual-branch CNN-Transformer and depth information fusion includes the following steps:

[0139] S1: Send the RGB image in the public haze dataset to a public depth estimation model to generate an estimated depth map. The public haze datasets required for the training and testing stages of the present invention are I-HAZE, O-HAZE, NH-HAZE and Dense-HAZE. Figure 5 shown.

[0140] The specific method of generating an estimated depth map is to find an open source model based on RGB estimated depth map, and send all RGB images in the public haze dataset into the network to generate an estimated depth map.

[0141] S2: Design a model according to the model architecture provided by the present invention, and design a feature extraction encoder, a multimodal fusion module of the attention mechanism, and a feature recovery decoder.

[0142] The specific approach to designing the model architecture and three modules is as follows: the feature extraction decoder selects the INN network to extract shallow features of RGB and depth maps, and uses Reformer to further extract deep features of RGB and depth maps. According to the model architecture diagram of the present invention, the feature map extracted by the dual-branch feature extractor is sent to the multimodal fusion module based on the attention mechanism to integrate the guiding effect of depth information on foggy RGB information. After fully integrating the complementary information of different modalities, the fused deep features are remapped to the image space through the feature recovery encoder.

[0143] S3: Design a loss function for learning according to the loss function designed by the present invention.

[0144] The specific method of generating the saliency map predicted by the model is as follows: structural similarity loss, gradient loss and color loss are all recognized and effective loss functions open in this field. According to the supervision method mentioned in the present invention, the clear images restored by RGB modality, depth modality and fusion modality are supervised.

[0145] S4: Send the RGB images of the haze data training set used for model training and the estimated depth map generated by S1 corresponding to the training set to this model for feature fusion, and restore it to a clear image through the decoder. The required haze training set contains 250 samples of NH-Haze, 250 samples of Dense-Haze, 140 samples of I-Haze, and 140 samples of O-Haze.

[0146] The specific method for generating the saliency map predicted by the model is as follows: the estimated depth obtained by S1 and the RGB and depth images in the haze dataset are cropped to 224×224 size and sent to the network, and the model is trained using a public training method.

[0147] S5: After the model training is completed, the effectiveness of this model is verified using the public haze dataset RGB images and public evaluation indicators.

[0148] The specific method to verify the effectiveness of the model is: the foggy RGB images in the public dataset and the real fog-free images are fed into the model, and the fog-free images generated by the feature recovery encoder are the model prediction results. The effectiveness of the prediction results of this model is verified using two public evaluation indicators. The following are the evaluation indicators that need to be used:

[0149] Peak signal-to-noise ratio (PSNR) is used to measure the difference between the reconstructed image and the original image. The higher the value, the better the reconstruction quality. Its mathematical expression is:

[0150]

[0151] Structural Similarity Index (SSIM): SSIM evaluates the brightness, contrast, and structural similarity of images. Its value is between 0 and 1. The closer the value is to 1, the higher the similarity. Its expression is

[0152]

[0153] Experimental verification:

[0154] The present invention summarizes and compares the performance of various dehazing methods on real data sets. Specifically, the objective evaluation indicators of DehazeFuse proposed in the present invention and other dehazing models in the prior art are obtained based on experiments on data sets Dense-HAZE, NH-HAZE, I-HAZE, and O-HAZE. See Table 1.

[0155] Table 1 Quantitative evaluation on real-world dehazing datasets

[0156]

[0157] The results in Table 1 show that the DehazeFuse method proposed in the present invention is in a leading position in the comprehensive evaluation. Each column in the table represents the average of the test results on different data sets. As expected, the dehazing method DCP based on prior knowledge lags behind the data-driven method in both PSNR and SSIM. In addition, those end-to-end direct estimation dehazing methods implemented using deep learning technology show better performance than those based on indirect estimation. When the test results of all data sets are combined, the DehazeFuse algorithm is generally superior to other algorithms in terms of PSNR and SSIM, because other algorithms do not consider the key factors affecting the imaging of foggy images during the model construction process. In contrast, the method based on scene deep fusion proposed in the present invention is more effective.

[0158] Figure 6-Figure 9 The visual comparison of the results of various dehazing methods on the datasets I-HAZE, O-HAZE, NH-HAZE, and Dense-HAZE are shown respectively. It can be clearly seen that the dehazed image processed by the DehazeFuse algorithm proposed in the present invention has more advantages in the real color, contrast, and texture details of the image.

[0159] In order to verify the effectiveness of the model using scene depth information, the present invention conducted an ablation experiment. In this experiment, the present invention compared two model configurations: one is a model without scene depth information (labeled as "no depth") and the other is a final model with scene depth information (labeled as "depth"). Through experiments on various data sets, the present invention obtained objective evaluation indicators, as shown in Table 2.

[0160] Table 2 Test the fusion scene depth strategy in different datasets to enhance the model

[0161]

[0162] The results in Table 2 show that the “deep” model significantly outperforms the “no depth” model on all datasets.

[0163] Fig.10The visual comparison of the two models with different configurations is shown. It can be clearly seen that the images processed by the "depth" model are significantly better than those processed by the "no depth" model in terms of color restoration, residual fog level, and image detail retention. The results of this experiment clearly show that fusing scene depth information into the model can effectively improve the performance of image dehazing. This finding emphasizes the importance of depth information in the image dehazing process, especially in maintaining image details and color authenticity. The fusion of scene depth information not only helps to more accurately estimate the distribution and density of haze, but also improves the overall quality of the final dehazed image, thus confirming the key role of depth information in improving the dehazing effect.

[0164] The above description is only used to help understand the method of the present invention and its core essence, but the protection scope of the present invention is not limited thereto. For those skilled in the art in the art, equivalent replacement or change according to the technical solution and inventive concept of the present invention within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A real scene image dehazing method based on dual-branch CNN-Transformer and depth information fusion, characterized in that: The following steps are involved: S1. Generate a depth map. Input the RGB image into a monocular depth estimator to generate an estimated depth map. The RGB image is a foggy image. S2. Design the image defogging model architecture; the image defogging model includes a feature extraction decoder, a multimodal feature fusion module and a feature recovery encoder, which are used to extract and integrate global features and local features from RGB modality and depth modality for image defogging; Among them, the multimodal feature fusion module adopts a multimodal deep cross-attention mechanism for global information exchange between the RGB modality and the depth modality, and uses a convolutional layer with a spatially invariant filter to aggregate the local information of the RGB modality and the depth modality; S3. Design loss function to optimize the model; S4, collect data sets for model training; S5. Use evaluation indicators to verify the effectiveness of the trained model.

2. The real scene image defogging method based on dual-branch CNN-Transformer and depth information fusion according to claim 1 is characterized in that: The feature extraction decoder includes a shallow feature extraction unit and a deep feature extraction unit; The shallow feature extraction unit extracts shallow features from the foggy image H and its corresponding depth map D and The details are as follows: Among them, S(.) is the shallow feature extraction unit; The deep feature extraction unit extracts features from the shallow features and Further extract deep features and The details are as follows: in, It is a deep feature extraction unit.

3. The real scene image defogging method based on dual-branch CNN-Transformer and depth information fusion according to claim 2 is characterized in that: The shallow feature extraction unit adopts a reversible neural network module with an affine coupling layer; The deep feature extraction unit adopts a Restormer module, which has an MDTA module and a GDFN module as basic components.

4. The real scene image defogging method based on dual-branch CNN-Transformer and depth information fusion according to claim 3 is characterized in that: The multimodal feature fusion module adopts a multimodal deep cross attention mechanism for the RGB modality F H With the depth modality F D The global information exchange between them is as follows: For the RGB modality By combining the and Attention weights are performed to merge cross-modal information, while the information in the RGB modality is retained through residual connections to obtain the fused feature Z H , and vice versa, we get the fusion feature Z D ; The process is as follows: Among them, MDTCA is a multimodal deep cross attention mechanism; LN is a normalization layer.

5. The real scene image defogging method based on dual-branch CNN-Transformer and depth information fusion according to claim 4, characterized in that: A convolutional layer with spatially invariant filters is further deployed to aggregate local information of the RGB modality and the depth modality as follows: Z DF =Conυ(Concat(Z H ,Z D )), Where Conυ(·) represents a convolutional layer with a spatially invariant filter, Concat(·) represents a connection in the channel dimension; Z DF Represents the fused deep features.

6. The real scene image defogging method based on dual-branch CNN-Transformer and depth information fusion according to claim 5, characterized in that: The feature recovery encoder includes a deep feature reconstruction unit and an image reconstruction unit; The deep feature reconstruction unit deploys a P-layer Restormer module, which is used to refine the fused deep features and restore the fused shallow features from a global perspective. The details are as follows: WITH SF =DR(Z DF ) Among them, DR(.) is the deep feature reconstruction unit; The image reconstruction unit is composed of reversible neural network modules; the details are as follows: in, is the dehazed image, and IR(.) is the image reconstruction unit.

7. The real scene image defogging method based on dual-branch CNN-Transformer and depth information fusion according to claim 1, characterized in that: The loss function in S3 is a comprehensive loss function including SSIM loss, Gradient loss and Colorloss; its formula is: Among them, λ1, λ2, and λ3 are hyperparameters that control the trade-off of each sub-loss term.

8. The real scene image defogging method based on dual-branch CNN-Transformer and depth information fusion according to claim 7, characterized in that: SSIM loss Used to constrain the dehazed image The structural consistency between the original clear image I is as follows: Gradient loss L grad The sharpness of the image edge is maintained by minimizing the difference between the gradient of the dehazed image and the original haze-free image, as follows: in, represents the Sobel gradient operator; Color loss L color First, Gaussian filtering is applied to the dehazed image and the haze-free image to obtain their blur representations, and then the color loss is achieved by calculating the mean square error between the two, as follows: Here, G(·) represents the image representation after applying the Gaussian filter, and MSE is the mean square error.

9. The real scene image defogging method based on dual-branch CNN-Transformer and depth information fusion according to claim 1, characterized in that: The data set in S4 uses Dense-HAZE, NH-HAZE, I-HAZE, and O-HAZE data sets, selects samples from the data sets, uses the RGB images in the samples and the corresponding depth maps generated by the monocular depth estimator as training sets, and uses the training sets to train the image dehazing model.

10. The real scene image defogging method based on dual-branch CNN-Transformer and depth information fusion according to claim 1, characterized in that: The evaluation indicators in S5 include peak signal-to-noise ratio PSNR and structural similarity index SSIM.

Images