A method for constructing a partial twin dehazing network based on U-Net

Abstract

A method for constructing a partially Siamese dehazing network based on U-Net is disclosed. This framework comprises two weight-sharing sub-networks. A hazy image is input into the first sub-network for dehazing, and the resulting image is then used as prior information for dehazing, which is passed to the second sub-network. The second sub-network, by inputting the hazy image and prior guidance information, possesses stronger dehazing and image reconstruction capabilities. Furthermore, this invention designs a dual encoding / decoding module within the sub-networks of this framework. The hazy image is input into the sub-network, and convolutional downsampling is performed to obtain encoded features. These features are then further encoded and fused using a secondary encoding module to obtain fused secondary encoded features. The fused secondary encoded features are then passed through a secondary decoding module, and subsequently fused with features at the corresponding scales of the encoding and decoding stages, respectively. Finally, upsampling is performed to obtain the final dehazed image.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically relating to a method for constructing a partially twinned dehazing network based on U-Net. Background Technology

[0002] With rapid industrialization, severe weather, such as smog, occurs frequently. Smog often results in blurry, color-distorted, or even images lacking crucial information. Therefore, dehazing of acquired images has become a critical issue. The effectiveness of image dehazing directly impacts the quality of images acquired in this invention and the accuracy of results in advanced vision tasks such as object detection, object tracking, face recognition, and image classification. How to recover a clear image from a blurry one has become a hot topic in computer vision research.

[0003] Currently, existing image dehazing methods can be divided into two categories: parameter-prior-based dehazing methods and learning-based dehazing methods. Parameter-prior-based image dehazing methods rely on atmospheric scattering models. These methods achieve image dehazing by predicting parameters such as global atmospheric light and transmission maps. Although these prior-based methods have achieved significant progress in many cases, they may face challenges under specific conditions. For example, in situations with uneven illumination or strong haze, dehazing methods based on intermediate parameter estimation may exhibit large errors, leading to artifacts, color distortion, and other degradation phenomena in the image. With the development of deep learning, end-to-end dehazing methods have become mainstream in recent years. These methods directly learn the mapping relationship between fog and clear images without using atmospheric scattering models. Wu et al. published "Contrastive Learning for Compact Single Image Dehazing," proposing the AECR dehazing method. This method fuses information from the upper encoding and decoding layers with the corresponding scale on the decoding layer and introduces negative samples into the frame's loss function to form a contrastive regularization loss. Qing et al. published "FFA-Net: Feature Fusion Attention Network for Single Image Dehazing," which enhances the network's ability to constrain foggy regions by stacking channel attention and pixel attention modules in the network, and proposes the dehazing method FFA-Net accordingly. Done et al. published "Multi-Scale Boosted Dehazing Network with Dense Feature Fusion," which proposed an MSBDN dehazing method. MSBDN designs dense skip connections between the encoding and decoding layers and achieves good dehazing performance. Some methods improve the constraint ability of the framework by modifying the network and designing a two-stage dehazing framework. Wei et al. designed a two-stage image dehazing framework based on a recurrent structure. Jiang et al. designed a two-stage image dehazing framework that treats dehazing and detail refinement as two independent tasks.

[0004] Although these methods can achieve good dehazing results, they still suffer from artifacts and color distortion when processing complex hazy images due to a lack of constraint capabilities. Specifically, they have the following problems.

[0005] (1) This paper mainly addresses the constraint loss problem in two-stage frameworks. Although two-stage frameworks can enhance the constraint capability of networks, they use the output of the first stage as the input of the second stage. Therefore, the information lost in the first stage cannot be reconstructed in the second stage. As a result, the dual constraints formed by the two stages are ineffective, and artifacts often appear in the generated images.

[0006] (2) Most of these methods fuse information by skipping between the encoding and decoding layers to reduce information loss. However, these skip connections ignore the importance of each level in the encoding stage, which contains different information. If information from different encoding layers can be effectively fused during image reconstruction at the decoding layer, not only can feature representation be enhanced, but the dilution of feature information can also be effectively reduced. Summary of the Invention

[0007] The purpose of this invention is to solve the technical problems of existing image dehazing techniques when using U-shaped networks for image dehazing, such as the lack of constraints on foggy regions and the failure of constraints on foggy regions due to the excessive number of parameters in the two-stage framework. In addition, this invention can also solve the technical problems of traditional U-shaped networks directly fusing the decoding layer with the corresponding scale coding layer, which lacks effective utilization of information from different levels of coding layers, and the technical problems of downsampling based on the U-shaped network structure model destroying spatial detail information of the image and feature dilution during upsampling.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] A method for constructing a partially Siamese dehazing network based on U-Net includes the following steps:

[0010] Step S1: Construct a partial twin dehazing framework, which includes two weight-shared U-shaped networks and an efficient feature extraction module;

[0011] Step S2: Construct a U-shaped image dehazing network, which includes: a feature extraction module at the encoding layer, a feature inpainting module, and an image restoration module at the decoding layer;

[0012] Step S3: Construct a dual-encoding and decoding multi-scale feature fusion module, and fuse features from different levels using the features extracted by the encoding layer feature extraction module in S1;

[0013] Step S4: Feed the foggy image into the U-shaped image dehazing network. Through the multi-level feature fusion module and the channel non-local information enhancement attention module, a clear fog-free image is output. Finally, the loss is calculated using the output clear image to constrain the training of the network.

[0014] In step S1, as Figure 1As shown, the dehazing framework partially consists of two sub-networks with the same weights and shared structure. Each sub-network includes an efficient feature extraction module and a dehazing generator. In order to preserve and extract the hidden information under fog in the input image to the maximum extent, the same foggy image is used as the input of the two sub-networks. The outputs of the two sub-networks are constrained by a series of loss functions, so that the networks can learn from each other better and improve the image dehazing ability.

[0015] The partially constructed twin dehazing framework is as follows:

[0016] Input image I → Dehazing network U1 → First dehazing image → High-efficiency feature extraction module EFECG → Prior information F;

[0017] Input image I, prior information F → dehazing network U2 → dehazing image.

[0018] The defogging network U1 is specifically as follows:

[0019] Foggy image → First coding layer → First coding feature layer → Second coding layer → Second coding feature layer → Third coding layer → Third coding feature layer;

[0020] First layer coding features, second layer coding features → first multi-scale feature extraction module → first secondary coding fusion feature;

[0021] First secondary coding feature, third layer coding feature → second multi-scale feature extraction module → second secondary coding fusion feature;

[0022] The multi-scale feature extraction module operates as follows.

[0023] First layer encoding features → First Conv layer (3*3) → First secondary encoding feature bf1, Second encoding layer → Second Conv layer (3*3) → First InstanceNorm layer → First residual feature extraction module → Second residual feature extraction module → First secondary encoding multi-scale feature;

[0024] Second layer coding features → Third Conv layer (3*3) → Second secondary coding features bf1, Third coding layer → Fourth Conv layer (3*3) → Second InstanceNorm layer → Third residual feature extraction module → Fourth residual feature extraction module → Second secondary coding multi-scale features;

[0025] Second secondary encoding fusion feature → First deconvolution layer → First secondary decoding feature → Second deconvolution layer → Second secondary decoding feature;

[0026] Third layer coding features → Residual feature extraction group → First decoding layer → First decoding layer features, first coding layer features, first secondary coding features → Second decoding layer → Second decoding layer features, second coding layer features, second secondary coding features → Third decoding layer → First dehazing image.

[0027] The efficient feature extraction module EFECG is as follows:

[0028] like Figure 3 As shown, the first dehazed image → the first GlobalAvagepool layer → the second GlobalAvagepool layer;

[0029] First dehazed image → First convolutional layer (3*3) → Second convolutional layer (3*3) → Convolutional layer 1*1 cb4 → Prior features F1;

[0030] First GlobalAvagepool layer GAP → First convolutional layer (3*3) → Second convolutional layer (3*3) → Third convolutional layer (3*3) → Fourth convolutional layer (1*1) → Prior features F2;

[0031] The second GlobalAvagepool layer (GAP) → the first convolutional layer (3*3) → the second convolutional layer (3*3) → the third convolutional layer (3*3) → the fourth convolutional layer (1*1) → prior features F3.

[0032] The defogging network U2 is specifically:

[0033] Foggy image → First coding layer → First coding feature layer → Second coding layer → Second coding feature layer → Third coding layer → Third coding feature layer;

[0034] The fusion features of the first layer coding features and the prior feature F1, the fusion features of the second layer coding features and the prior feature F2 → the first multi-scale feature extraction module → the first secondary coding fusion feature;

[0035] The first secondary coding feature, the fusion feature of the third layer coding feature and the prior feature F3 → the second multi-scale feature extraction module → the second secondary coding fusion feature;

[0036] The multi-scale feature extraction module operates as follows.

[0037] First layer encoding features → First Conv layer (3*3) → First secondary encoding feature bf1, Second encoding layer → Second Conv layer (3*3) → First InstanceNorm layer → First residual feature extraction module → Second residual feature extraction module → First secondary encoding multi-scale feature;

[0038] Second layer coding features → Third Conv layer (3*3) → Second secondary coding features bf1, Third coding layer → Fourth Conv layer (3*3) → Second InstanceNorm layer → Third residual feature extraction module → Fourth residual feature extraction module → Second secondary coding multi-scale features;

[0039] Second secondary encoding fusion feature → First deconvolution layer → First secondary decoding feature → Second deconvolution layer → Second secondary decoding feature;

[0040] Third layer coding features → Residual feature extraction group → First decoding layer → First decoding layer features, first coding layer features, first secondary coding features → Second decoding layer → Second decoding layer features, second coding layer features, second secondary coding features → Third decoding layer → Dehazed image.

[0041] In step S4, during training, the network is trained using four different loss constraints as follows:

[0042] (1) SmoothL1 loss, the specific formula is as follows:

[0043]

[0044]

[0045] x i and y i These represent the values ​​at pixel i in the hazy image and the ground truth image, respectively. G() represents the parameters of the dehazing network. i ) represents the pixel value at position i in the input image, which is then processed by the dehazing network parameters to obtain the value; N represents the number of pixels in the image.

[0046] (2) Perceptual loss: The perceptual loss is calculated using the VGG16 pre-trained model on ImageNet. The specific formula is as follows:

[0047]

[0048] Where x and y represent the foggy image and the ground truth image, respectively, i represents the i-th layer of the feature map, H represents the length of the feature map, W represents the width of the feature map, and C represents the channels of the feature map, where C i W represents the channel of the i-th layer of the feature map. i H represents the width of the i-th layer of the feature map. i φ represents the length of the i-th layer of the feature map. i (x) represents the input foggy image processed by the VGG16 pre-trained model to obtain the i-th layer with size and length H. i Width is Wi The number of channels is C i The feature map, |||| represents the L2 norm, and N represents the number of feature layers of the VGG16 pre-trained model used in the perceptual loss;

[0049] (3) Multi-scale structural similarity loss, the specific formula is as follows:

[0050]

[0051] Where x represents the generated image, y represents the sharp image, μx and μy represent the mean of the generated image and the ground truth image, respectively, σx and σy represent the standard deviation, σxy represents the covariance of the generated image and the sharp image, βm and γm represent the relative importance of the two terms, C1 and C2 are constant terms, and M represents the total number of scales.

[0052] (4) To counteract losses, the specific formula is:

[0053]

[0054] Where D(y) represents the probability of classifying the dehazed image y as a clear image, N represents the total number of images, and the overall network loss function is expressed as:

[0055] loss sod =λ1L smoothL1 +λ2L adv +λ3L perc +λ4L ms-ssim

[0056] Where λ1, λ2, λ3, and λ4 are hyperparameters of each loss function.

[0057] Compared with the prior art, the present invention has the following technical effects:

[0058] 1) By using a well-designed partial twin dehazing framework, a "twin network" can be introduced to construct "twin constraints" to enhance the network's ability to constrain foggy areas, thereby improving the overall image dehazing capability of the network, which is particularly suitable for remote sensing images with dense fog.

[0059] 2) A method called the dual-encoder-decoder multi-scale information fusion module is proposed. This method explores how to improve the extraction of fog features in the network through dual encoders and further enhances the feature representation ability by fusing multi-scale information, effectively reducing the dilution of feature information. In addition, when the dataset is contaminated by dense fog, the dual-encoder-decoder multi-scale information fusion module method can effectively enhance the robustness of dehazing.

[0060] 3) This invention develops a novel method called Partial Siamese Dehazing Network, which seamlessly integrates the designed partial siamese dehazing framework and a dual-encoder / decoder multi-scale information fusion module into the partial siamese dehazing network for processing image dehazing in remote sensing scenes. By imposing more effective constraints on foggy regions, our method outperforms most state-of-the-art methods on challenging benchmark datasets such as StateHazelk and HRSD. Attached Figure Description

[0061] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0062] Figure 1 This is a diagram of the overall partial twin network framework structure of an embodiment of the present invention;

[0063] Figure 2 for Figure 1 A schematic diagram of the structure of a single sub-network and a schematic diagram of its dual encoding / decoding module;

[0064] Figure 3 for Figure 2 A schematic diagram of the structure of a medium-efficiency feature extraction module. Detailed Implementation

[0065] like Figures 1 to 3 As shown, a method for constructing a partially Siamese dehazing network based on U-Net includes the following steps:

[0066] S1. Construct a partially twin dehazing framework, which includes two U-shaped networks with shared weights and an efficient feature extraction module.

[0067] S2. Construct a U-shaped image dehazing network, which includes: a feature extraction module at the encoding layer, a feature repair module, and an image restoration module at the decoding layer.

[0068] S3. Construct a dual-encoding and decoding multi-scale feature fusion module, which uses the features extracted by the encoding layer feature extraction module in S1 to fuse features at different levels.

[0069] S4. The foggy image is fed into the U-shaped image dehazing network. Through the multi-level feature fusion module and the channel non-local information enhancement attention module, a clear fog-free image is output. Finally, the loss is calculated using the output clear image to constrain the training of the network.

[0070] Step S1 specifically includes:

[0071] like Figure 1As shown, the partial Siamese framework comprises two sub-networks with the same weight-sharing structure. Each sub-network includes an efficient feature extraction module and a dehazing generator. To maximize the preservation and extraction of hidden information under fog in the input image, this invention uses the same foggy image as the input to both sub-networks. A series of loss functions constrain the outputs of the two sub-networks, enabling them to learn from each other better and improve image dehazing capabilities. Specifically, a given foggy image is input into two Siamese sub-networks, U1 and U2. The invention then uses the dehazed image generated by U1 to create prior information. First, the invention performs three kernel=2 average pooling operations on the result of U1 to obtain three dehazed images at different scales. Subsequently, the invention inputs these three dehazed images of different sizes into the efficient feature extraction module, which can quickly and efficiently extract features to form prior information for dehazing, guiding the second-stage Siamese sub-networks in dehazing.

[0072] Steps S2 and S3 specifically include:

[0073] like Figure 1 As shown, the feature extraction module of the encoding layer uses three convolutional operations to perform 4x downsampling for feature extraction. Each layer in the encoding stage consists of a convolutional layer, an activation layer, and an instance-norm. Next, six consecutive residual blocks are combined to further enhance feature extraction on the extracted low-resolution features. Finally, the features extracted from the consecutive residual blocks, which contain a large amount of semantic scene information, are decoded and reconstructed using deconvolution and convolution operations to restore the original image resolution.

[0074] A dual encoding module is constructed in the dual encoding and decoding process. Utilizing the features extracted by the feature extraction module of the encoding layer in S1, the features from the previous layer are encoded a second time, and then fused using the FE module. The FE module is constructed as follows: Figure 1 As shown in the FE module section, the information is then encoded and fused layer by layer in a second cycle, as expressed by the following formula:

[0075]

[0076]

[0077] Where EBi represents the coding feature of the i-th layer, and bi_EB i Represents the secondary encoding fusion feature of the i-th layer.

[0078] A dual-decoding module is constructed in the dual encoding and decoding process. It utilizes the features extracted by the feature extraction module in the encoding layer of S1, and then performs a second decoding on the features obtained from the second encoding. Afterwards, the information is fused through a layer-by-layer cyclical second decoding process. The specific formula is as follows:

[0079] bi_DB1 = TConv(bi_EB) n )

[0080] bi_DB i =TConv(bi_DB) i-1 ),

[0081] Among them bi_EB n Bi_DB represents the last layer of secondary encoding features. i The second-order decoding features represent layer i. Conv and Tconv represent convolution and deconvolution, respectively. This invention not only fully integrates shallow detail textures and deep semantic features of the encoding layer into each decoding layer, but also effectively alleviates the feature dilution problem caused by upsampling during the decoding stage, ultimately improving the ability of the dehazing network to recover details, semantics, and scene information.

[0082] Step S4 specifically includes:

[0083] A foggy image is input into a U-shaped image dehazing network. The network is then processed through a multi-level feature fusion module, followed by a multi-level feature fusion module and a channel non-local information enhancement attention module to output a clear, fog-free image. Finally, the loss is calculated using the clear output image. The network training process is then constrained by four loss mechanisms, as follows:

[0084] The first type is the SmoothL1 loss, and the specific formula is as follows:

[0085]

[0086]

[0087] x i and y i These represent the values ​​at pixel i in the hazy image and the ground truth image, respectively. G() represents the parameters of the dehazing network. i ) represents the pixel value at position i in the input image, which is then processed by the dehazing network parameters. N represents the number of pixels in the image.

[0088] The first method is perceptual loss, which calculates the perceptual loss using a pre-trained VGG16 model on ImageNet. The specific formula is as follows:

[0089]

[0090] Where x and y represent the foggy image and the ground truth image, respectively. i represents the i-th layer of the feature map, H represents the length of the feature map, W represents the width of the feature map, and C represents the number of channels in the feature map. i W represents the channel of the i-th layer of the feature map. iH represents the width of the i-th layer of the feature map. i φ represents the length of the i-th layer of the feature map. i (x) represents the input foggy image processed by the VGG16 pre-trained model to obtain the i-th layer with size and length H. i Width is W i The number of channels is C i The feature map. |||| represents the L2 norm, and N represents the number of feature layers of the VGG16 pre-trained model used in the perceptual loss.

[0091] The third type of loss is the multi-scale structural similarity loss, and the specific formula is as follows:

[0092]

[0093] Where x represents the generated image and y represents the sharpened image. μx and μy represent the means of the generated and ground truth (GT) images, respectively. σx and σy represent the standard deviations, and σxy represents the covariance of the generated and sharpened images. βm and γm indicate the relative importance of the two terms, and C1 and C2 are constant terms. M represents the total number of scales.

[0094] The fourth type of loss is adversarial loss, and the specific formula is as follows:

[0095]

[0096] Where D(y) represents the probability of classifying the dehazed image y as a clear image, and N represents the total number of images. The overall network loss function is expressed as:

[0097] loss sod =λ1L smoothL1 +λ2L adv +λ3L perc +λ4L ms-ssim

[0098] Where λ1, λ2, λ3, and λ4 are hyperparameters of each loss function.

[0099] The network constructed in this invention is detailed as follows:

[0100] The partially constructed twin dehazing framework is as follows:

[0101] Input image I → Dehazing network U1 → First dehazing image → High-efficiency feature extraction module EFECG → Prior information F;

[0102] Input image I, prior information F → dehazing network U2 → dehazing image.

[0103] The defogging network U1 is specifically as follows:

[0104] Foggy image → First coding layer 1 → First coding feature layer → Second coding layer 2 → Second coding feature layer → Third coding layer 3 → Third coding feature layer;

[0105] First layer coding features, second layer coding features → first multi-scale feature extraction module 4 → first secondary coding fusion features;

[0106] First secondary coding feature, third layer coding feature → second multi-scale feature extraction module 5 → second secondary coding fusion feature;

[0107] The multi-scale feature extraction module operates as follows.

[0108] First layer coding feature → First Conv layer → First secondary coding feature bf1, Second coding layer 2 → Second Conv layer → First InstanceNorm layer → First residual feature extraction module → Second residual feature extraction module → First secondary coding multi-scale feature;

[0109] Second layer coding features → Third Conv layer → Second secondary coding features bf1, Third coding layer 2 → Fourth Conv layer → Second InstanceNorm layer → Third residual feature extraction module → Fourth residual feature extraction module → Second secondary coding multi-scale features;

[0110] Second secondary encoding fusion feature → First deconvolution layer 8 → First secondary decoding feature → Second deconvolution layer 10 → Second secondary decoding feature;

[0111] Third layer coding features → Residual feature extraction group 6 → First decoding layer 7 → First decoding layer features, first coding layer features, first secondary coding features → Second decoding layer 9 → Second decoding layer features, second coding layer features, second secondary coding features → Third decoding layer 11 → First dehazing image.

[0112] The efficient feature extraction module EFECG is as follows:

[0113] First dehazed image → First GlobalAvagepool layer 12 → Second GlobalAvagepool layer 38;

[0114] First dehazed image → First convolutional layer 27 → Second convolutional layer 28 → Convolutional layer 1*1 29 → Prior features F1;

[0115] First GlobalAvagepool layer GAP12 → First convolutional layer 30 → Second convolutional layer 31 → Third convolutional layer 32 → Fourth convolutional layer 33 → Prior features F2;

[0116] The second GlobalAvagepool layer GAP38 → the first convolutional layer 34 → the second convolutional layer 35 → the third convolutional layer 36 → the fourth convolutional layer 37 → prior features F3.

[0117] The defogging network U2 is specifically as follows:

[0118] Foggy image → First coding layer 16 → First coding feature layer → Second coding layer 17 → Second coding feature layer → Third coding layer 18 → Third coding feature layer;

[0119] The fusion feature of the first layer coding feature and the prior feature F1, the fusion feature of the second layer coding feature and the prior feature F2 → the first multi-scale feature extraction module 19 → the first secondary coding fusion feature;

[0120] The first secondary coding feature, the fusion feature of the third layer coding feature and the prior feature F3 → the second multi-scale feature extraction module 20 → the second secondary coding fusion feature;

[0121] The multi-scale feature extraction module operates as follows.

[0122] First layer coding feature → First Conv layer → First secondary coding feature bf1, Second coding layer 2 → Second Conv layer → First InstanceNorm layer → First residual feature extraction module → Second residual feature extraction module → First secondary coding multi-scale feature;

[0123] Second layer coding features → Third Conv layer → Second secondary coding features bf1, Third coding layer 2 → Fourth Conv layer → Second InstanceNorm layer → Third residual feature extraction module → Fourth residual feature extraction module → Second secondary coding multi-scale features;

[0124] Second secondary encoding fusion feature → First deconvolution layer 23 → First secondary decoding feature → Second deconvolution layer 24 → Second secondary decoding feature;

[0125] Third layer coding features → Residual feature extraction group 21 → First decoding layer 22 → First decoding layer features, first coding layer features, first secondary coding features → Second decoding layer 25 → Second decoding layer features, second coding layer features, second secondary coding features → Third decoding layer 26 → Dehazed image.

[0126] To facilitate a better understanding of the present invention by those skilled in the art, further explanation is provided below:

[0127] 1) Parameter settings

[0128] The code for this invention is implemented using the PyTorch framework, and experiments were conducted on an NVIDIA RTX 3090T GPU. The Adam optimizer was used to optimize the network, with the learning rate and batch size set to 0.001 and 8, respectively, and momentum decay exponents β1 = 0.9 and β2 = 0.999. The initial learning rate was set to 0.001, and a cosine annealing strategy was used to adjust the learning rate, with the cosine function half-cycle set to 5. Furthermore, this invention evaluated various dehazing algorithms on the synthetic dataset RESIDE and the real-world datasets from the NTIRE dehazing challenge: DenseHaze, NHHaze21, and SateHaze1k remote sensing datasets. In the RESIDE dataset, the network was trained using the outdoor training set OTS, and the outdoor SOTS dataset was used as the test set. OTS contains 8970 clear images and 313950 foggy images, while SOTS contains 500 indoor test images and an outdoor test dataset. DenseHaze contains 45 dense fog images, including 35 training images, 5 validation images, and 5 test images. NHHaze2021 contains 25 non-uniform fog images. Since its validation and test ground truth (GT) images have not yet been released, this invention selects the first 20 images as the training set and the remaining 5 as the test set for evaluation. The publicly available remote sensing dataset SateHaze1k contains three subsets representing different fog concentrations: Thin represents the thin fog dataset, Moderate represents the medium-dense fog dataset, and Thick represents the dense fog dataset. Each training subset contains 320 images, the validation set contains 35 images, and the test set contains 45 images. To verify the correctness and effectiveness of the method in this invention, it is compared with current excellent defogging algorithms, including: DCP, AOD-Net, GCA-Net, EPDN, Grideehaze-Net, MSBDN, FFA, AECR, and TBN.

[0129] 2) Experimental Results

[0130] Table 1 compares the results with the SOTA method on the SOTS-outdoor dataset and real-world scene datasets.

[0131]

[0132] This invention achieved excellent performance on the SOTS-outdoor dataset, ranking first in PSNR and second in SSIM. As shown in column 2 of Table 1, end-to-end dehazing algorithms generally outperform parameter estimation-based algorithms. Among learning-based methods, the algorithm of this invention improves PSNR by 11.65dB, 2.06dB, and 3.32dB respectively compared to U-shaped structure-based EDPN, MSBDN, and AECR, and improves SSIM by 0.116, 0.003, and 0.011 respectively. These algorithms only fuse their corresponding encoding and decoding features or perform feature fusion through dense connections at the encoding and decoding layers, neglecting the utilization between different layers in the encoding stage. The algorithm of this invention fuses encoding features from different levels and utilizes the proposed attention mining to discover channels that are more important for dehazing, thereby improving the network dehazing performance and generating higher-quality dehazed images. Furthermore, the dehazing algorithm proposed in this invention achieves PSNR of 18.20 dB and SSIM of 0.610 on dense fog dataset and PSNR of 23.34 dB and SSIM of 0.852 on non-uniform fog dataset (NH-haze21), respectively, which are all superior to all compared dehazing methods and demonstrate excellent performance.

[0133] 3. Ablation test

[0134] Table 2 shows the PSNR and SSIM results on the SOTS outdoor dataset.

[0135]

[0136] To evaluate the effectiveness of each module of the invention, the invention designed ablation experiments based on innovations in the framework and attention. There are a total of 5 experiments: (1) Base represents the U-shaped basic framework, which mainly includes two downsampling layers, six residual blocks and two upsampling layers, where the encoding and decoding layers are directly connected together through skip connections. (2) Base+PSDF without prior information. (3) Base+PSDF. (4) Base+PSDF+Dual encoding module. (5) Base+PSDF+BMIF module.

[0137] The Reside-outdoor dataset was used in this ablation experiment. The results of PSNR and SSIM are shown in the table. In Experiment (2), this invention introduces a Siamese network and designs a partial Siamese network (PSDF w / opriorinformation) which contains two sub-networks that use only the basic construction. The partial Siamese network framework can promote the ability of the two sub-networks to learn dehazing knowledge, which ultimately greatly improves the image dehazing ability. In Experiment (3), this invention uses the output of the first generator as prior information to guide the second network to perform dehazing (PSDF) to improve the constraint ability between the sub-networks. Based on this, this invention conducted Experiment (4), adding dual encoding to each of the two sub-networks of PSDF to enhance the feature extraction ability of the network. In order to better utilize the acquired multi-scale information, this invention designed Experiment (5), which more effectively utilizes multi-scale fusion information to assist image reconstruction by adding a dual decoding module to the decoding layer of the network, and the dual encoding and dual decoding together form the BMIF module. Simultaneously, this invention passes the output of the first generator to the BMIF module of the second network as prior guidance information to guide the second generator in learning dehazing knowledge. With weights shared between the two sub-networks, when one sub-network improves its understanding of dehazing knowledge, the other sub-network also benefits. The method of this invention achieves a PSNR of 34.22 dB and an SSIM of 0.979.

Claims

1. A method for constructing a partially Siamese dehazing network based on U-Net, comprising the following steps: Step S1: Construct a partial twin dehazing framework, which includes two weight-shared U-shaped networks and an efficient feature extraction module; Step S2: Construct a U-shaped image dehazing network, which includes: a feature extraction module at the encoding layer, a feature inpainting module, and an image restoration module at the decoding layer; Step S3: Construct a dual-encoding and decoding multi-scale feature fusion module, and use the features extracted by the coding layer feature extraction module in S2 to fuse features at different levels; Step S4: Feed the foggy image into the U-shaped image dehazing network. Through the multi-level feature fusion module and the channel non-local information enhancement attention module, a clear fog-free image is output. Finally, the loss is calculated using the output clear image to constrain the training of the network. The partially constructed twin dehazing framework is as follows: Input image I → Dehazing network U1 → First dehazing image → High-efficiency feature extraction module EFECG → Prior information F; Input image I, prior information F → dehazing network U2 → dehazing image; The high-efficiency feature extraction module EFECG is specifically as follows: First dehazed image → First GlobalAvagepool layer → Second GlobalAvagepool layer; First dehazed image → First convolutional layer → Second convolutional layer → 1*1 convolutional layer → Prior features F1; First GlobalAvagepool layer → First convolutional layer → Second convolutional layer → Third convolutional layer → Fourth convolutional layer → Prior features F2; Second GlobalAvagepool layer → First convolutional layer → Second convolutional layer → Third convolutional layer → Fourth convolutional layer → Prior features F3; The defogging network U2 is specifically: First dehazed image → First coding layer → First coding feature layer → Second coding layer → Second coding feature layer → Third coding layer → Third coding feature layer; The fusion features of the first layer coding features and the prior feature F1, the fusion features of the second layer coding features and the prior feature F2 → the first multi-scale feature extraction module → the first secondary coding fusion feature; The first secondary coding fusion feature, the third layer coding feature and the prior feature F3 fusion feature → the second multi-scale feature extraction module → the second secondary coding fusion feature; Second secondary encoding fusion feature → First deconvolution layer → First secondary decoding feature → Second deconvolution layer → Second secondary decoding feature; Third layer coding features → Residual feature extraction group → First decoding layer → First decoding layer features, second layer coding features, first secondary decoding features → Second decoding layer → Second decoding layer features, first layer coding features, second secondary decoding features → Third decoding layer → Dehazed image.

2. The method according to claim 1, characterized in that, In step S1, the dehazing framework partially consists of two sub-networks with the same weight-sharing structure. Each sub-network includes an efficient feature extraction module and a dehazing generator. In order to preserve and extract the hidden information under the fog in the input image to the maximum extent, the same foggy image is used as the input of the two sub-networks. The outputs of the two sub-networks are constrained by a series of loss functions so that the networks can learn from each other better and improve the image dehazing ability.

3. The method according to claim 1, characterized in that, The defogging network U1 is specifically: Foggy image → First coding layer → First coding feature layer → Second coding layer → Second coding feature layer → Third coding layer → Third coding feature layer; First layer coding features, second layer coding features → first multi-scale feature extraction module → first secondary coding fusion feature; First secondary coding fusion feature, third layer coding feature → second multi-scale feature extraction module → second secondary coding fusion feature; Second secondary encoding fusion feature → First deconvolution layer → First secondary decoding feature → Second deconvolution layer → Second secondary decoding feature; Third layer coding features → Residual feature extraction group → First decoding layer → First decoding layer features, second layer coding features, first secondary decoding features → Second decoding layer → Second decoding layer features, first layer coding features, second secondary decoding features → Third decoding layer → First dehazing image.

Images