A method for constructing a multi-level feature interaction defogging network based on U-Net

Abstract

The application discloses a construction method of a multi-level feature interaction defogging network based on U-Net, and the basic framework of the network is a U-shaped architecture, which comprises a multi-level feature interaction module and a channel non-local information enhancement attention module. A U-shaped network architecture is input with a fog image, and convolution downsampling is performed to obtain EB1, EB2 and EB3, respectively. Then, the multi-level feature module is used to fuse the features of EB1, EB2 and EB3 to obtain fused features EF1, EF2 and EF3. The fused features obtained in the last step are fused with decoding stages DB1, DB2 and DB3 through the channel non-local information enhancement attention module, and then, the final defogging image is obtained through upsampling. On thick fog, non-uniform fog images and remote sensing images with large scale changes, the method disclosed by the application can restore a defogging image with better color, brightness and detail information.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image processing, and particularly relates to a construction method of a multi-level feature interaction defogging network based on U-Net. BACKGROUND

[0002] Haze is a common atmospheric phenomenon caused by small particles such as smoke and dust in the atmosphere, and haze is an important factor causing the decline of visual quality such as object appearance and contrast. Pictures taken on a foggy day often have problems such as image blurring, information loss, and contrast reduction. Haze can seriously damage image information and have a negative impact on numerous advanced visual tasks such as face recognition, image segmentation, target detection, and target tracking. Therefore, in the past decade, the image defogging task has received extensive attention in the field of vision.

[0003] At present, image dehazing algorithms are mainly divided into two categories: parameter estimation-based dehazing methods and end-to-end dehazing methods. The parameter estimation-based dehazing method relies on the atmospheric scattering model, and realizes image dehazing by estimating global atmospheric light and transmission map parameters. Although the prior-based method has made significant progress, under unconstrained conditions, these dehazing methods based on intermediate parameter estimation are prone to large errors and may produce a large number of artifacts, color distortion and other image degradation phenomena. With the development of deep learning, in recent years, end-to-end dehazing methods have become the mainstream, researchers have proposed many end-to-end dehazing methods, which use convolutional neural networks to directly learn the mapping relationship between hazy images and haze-free images, without estimating any intermediate parameters. Qing et al. published “Enhanced Pix2pix Dehazing Network”, which proposed an enhanced pix2pix dehazing algorithm based on a U-shaped network, constructed a multi-resolution generator and a multi-scale discriminator, and designed an enhancement module at the end of the multi-resolution generator, which enhanced the recovery effect of image texture and color. Done et al. published “Multi-Scale Boosted Dehazing Network with Dense Feature Fusion”, which proposed the MSBDN dehazing network, which combines the U-shaped architecture and dense feature fusion, and performs dense skip connection on the encoding layer and the decoding layer, achieving excellent dehazing performance. Wu et al. published “Contrastive Learning for Compact Single Image Dehazing”, which proposed a dehazing network AECR-Net based on the idea of contrastive learning, which is based on the U-shaped architecture, and its algorithm pulls the dehazed image and the positive sample GT image closer in the representation space, and pushes the input hazy image further away, further improving the dehazing effect. Qing et al. published “FFA-Net: Feature Fusion Attention Network for Single Image Dehazing”, which proposed a feature fusion attention network FFA-Net, which uses channel attention and pixel attention to allocate weights in the spatial and channel dimensions of the feature map, achieving good dehazing performance.

[0004] Although the end-to-end dehazing method based on deep learning has achieved excellent dehazing performance. However, when using U-shaped networks and non-U-shaped networks for image dehazing, there are still several problems.

[0005] (1) Most of the defogging algorithms use U-shaped network structure, directly fuse the decoding layer with the corresponding scale of the encoding layer, not only ignores the effective use of different encoding layer information, but also has the problem of dilution of feature information, resulting in the edge details and overall scene (color, brightness, etc.) of the defogging image recovery are not ideal.

[0006] The dimension reduction operation of the two full connections in the channel attention has a negative impact on the prediction of feature channel weights, thereby reducing the performance of the defogging network. SUMMARY

[0007] The purpose of the present application is to solve the technical problems that the prior art directly fuses the decoding layer with the corresponding scale of the encoding layer when using the U-shaped network for image defogging, lacking effective use of different levels of encoding layer information, and the down-sampling based on the u-shaped network structure model destroys the spatial detail information of the image, and there is a feature dilution problem in the up-sampling process. In addition, the present application can also solve the technical problems that the two full connections in the SE channel attention have a negative impact on the prediction of feature channel weights, reducing the performance of the defogging network, and provide a multi-level feature interaction and efficient channel non-local information enhancement attention image defogging method.

[0008] A construction method of a multi-level feature interaction defogging network based on U-Net, comprising the following steps:

[0009] Step S1: constructing a U-shaped image defogging network, which includes: an encoding layer feature extraction module, a feature repair module, a decoding layer image recovery module, a single feature-channel non-local information enhancement attention module SF-NEA, and a multi-feature-channel non-local information enhancement attention module MF-NEA;

[0010] Step S2: constructing a channel non-local information enhancement attention module NEA added to the U-shaped network to enhance the performance of the defogging network, which includes two sub-modules, namely a single feature-channel non-local information enhancement attention module SF-NEA and a multi-feature-channel non-local information enhancement attention module MF-NEA;

[0011] Step S3: sending the hazy image into the U-shaped image defogging network, through the multi-level feature fusion module and the channel non-local information enhancement attention module, finally outputting a clear haze-free image, and finally using the output clear image to calculate the loss and constrain the training of the network;

[0012] The multi-level feature interaction defogging network based on U-Net is constructed through the above steps.

[0013] In step S1, the U-shaped image defogging network constructed is as follows:

[0014] the first layer of the encoding layer feature extraction module EB1→ the InstanceNorm layer IN1→ the second layer of the encoding layer feature extraction module EB2→ the InstanceNorm layer IN2→ the third layer of the encoding layer feature extraction module EB3→ the InstanceNorm layer IN3;

[0015] the InstanceNorm layer IN1, the InstanceNorm layer IN2, and the InstanceNorm layer IN3→ the third multi-level feature interaction module MFS3→ the fused feature EF3→ the third multi-feature channel non-local information enhancement attention MF-NEA3;

[0016] the third multi-feature channel non-local information enhancement attention MF-NEA3 and the decoding recovery layer DB3→ fusion operation→ the first deconvolution layer→ the InstanceNorm layer IN4→ the SF-NEA module→ the decoding recovery layer DB2;

[0017] the InstanceNorm layer IN2, the InstanceNorm layer IN1, and the InstanceNorm layer IN3→ the second multi-level feature interaction module MFS2→ the fused feature EF2→ the second multi-feature channel non-local information enhancement attention MF-NEA2;

[0018] the second multi-feature channel non-local information enhancement attention MF-NEA2 and the decoding recovery layer DB2→ fusion operation→ the second deconvolution layer→ the InstanceNorm layer IN5→ the SF-NEA module→ the decoding recovery layer DB1;

[0019] the InstanceNorm layer IN3, the InstanceNorm layer IN2, and the InstanceNorm layer IN1→ the first multi-level feature interaction module MFS1→ the fused feature EF1→ the first multi-feature channel non-local information enhancement attention MF-NEA1;

[0020] the first multi-feature channel non-local information enhancement attention MF-NEA1 and the decoding recovery layer DB1→ fusion operation→ the third deconvolution layer→ the haze-free image.

[0021] The specific operation of the MFS3 is: 1*1Convolution is performed on the InstanceNorm layer IN3, 3*3Convolution is performed on the InstanceNorm layer IN2, and 3*3Convolution is performed on the InstanceNorm layer IN1→ fusion operation;

[0022] The specific operations of MFS2 are as follows: perform 1*1 Convolution and 3*3 Convolution on the InstanceNorm layer IN2, and perform 3*3 Convolution transpose → fusion operation on the InstanceNorm layer IN3;

[0023] The specific operations of MSF1 are as follows: perform a 1*1 Convolution on the InstanceNorm layer IN1, perform a 3*3 Convolution transpose on the InstanceNorm layer IN2, and perform a 3*3 Convolution transpose on the InstanceNorm layer IN3 → fusion operation.

[0024] In step S2, the single-feature-channel nonlocal information enhancement attention module (SF-NEA) structure in the channel nonlocal information enhancement attention module is as follows:

[0025] Input feature F → Global average pooling (GAP) operation → Channel descriptor vector S → 1D convolution operation → Vector S containing local information lc ;

[0026] The transpose operation of the channel descriptor vector S yields the transpose vector S of the channel descriptor. T → Channel descriptor vector S, transpose of channel descriptor S T After dot product operation → Vector S containing nonlocal information gc ;

[0027] A vector S containing nonlocal information gc A vector S containing local information lc → Fusion operation → 1D convolution operation → Feature weight assignment W;

[0028] Feature assignment weights W, input feature map F → pixel-wise multiplication operation → feature map F*

[0029] The Multi-Feature-Channel Non-Local Information Enhancement Attention Module (MF-NEA) structure in step S3 is as follows:

[0030] Multi-level features EC1, EC2, EC → fusion operation → fusion feature EF → global average pooling → three 1D convolution operations → including local information

[0031] Fusion feature channel descriptor S EF → The transpose operation yields the transposed fused feature channel descriptor. → Fusion Feature Channel Descriptor S EF , Point multiplication operation → vector containing non-local information

[0032] Vector containing local information Add respectively with vector containing non-local information → three 1D convolution operations → vector containing local information → Concat operation → Softmax activation function → obtain W1, W2, W3 weights → W1 and input feature Ec1 are multiplied, W2 and input feature Ec2 are multiplied, and W3 and input feature Ec3 are respectively multiplied → fusion operation → fusion feature F.

[0033] In step S4, the multi-level feature interaction and efficient channel non-local information enhancement attention image defogging network, multi-level feature fusion module and channel non-local information enhancement attention module are used in the following steps when in use:

[0034] Step 1) input the hazy image into the U-shaped image defogging network;

[0035] Step 2) input the feature information extracted by the U-shaped network into the multi-level feature fusion module to obtain the fusion feature;

[0036] Step 3) fuse the fusion feature with the corresponding encoding layer respectively to obtain the feature with more detailed texture semantics, and improve the performance of the defogging network through the channel non-local information enhancement attention module to obtain the final output clear haze-free image;

[0037] Step 4) use four kinds of loss to constrain the training process of the network.

[0038] In step 4), the four kinds of loss are as follows:

[0039] (1) L1 loss, the specific formula is:

[0040]

[0041] x i and y i represent the values of the hazy image and the GT image at pixel i respectively, G() represents the defogging network parameters, G(x i ) represents the pixel value of the input image i, and then the value obtained by operating the defogging network parameters. N represents the number of pixels in the image.

[0042] (2) perceptual loss, the pre-trained model of VGG16 on ImageNet is used to calculate the perceptual loss, and the specific formula is:

[0043]

[0044] where x and y represent the foggy image and the GT image respectively, i represents the i-th layer of the feature map, H represents the length of the feature map, W represents the width of the feature map, and C represents the channel of the feature map. i i i i where φ (x) represents the i-th layer of the input foggy image after the VGG16 pre-trained model, and the size of the feature map is length H i , width W i , and channel C i .

[0045] (3) Multi-scale structural similarity loss, the specific formula is:

[0046]

[0047] where x represents the generated image, and y represents the clear image. μ x ,μ y respectively represent the mean of the generated image and the GT image. σ x ,σ y respectively represent the standard deviation, and σ xy represents the covariance of the generated image and the clear image. β m γ m represent the relative importance of the two terms, and C1 and C2 are constant terms. M represents the total number of scales.

[0048] (4) Adversarial loss, the specific formula is:

[0049]

[0050] where D(y) represents the probability of judging the dehazed image y as a clear image, and N represents the total number of images.

[0051] The loss function of the overall network is represented as:

[0052] L loss =λ1L adv +λ2L l1 +λ3L perc +λ4L ms-ssim

[0053] where λ1, λ2, λ3, and λ4 are hyperparameters of each loss function.

[0054] Compared with the prior art, the present application has the following technical effects:

[0055] ​​​1) This invention proposes a dehazing network with multi-level feature interaction and efficient channel nonlocal information enhancement attention. Through effective encoder-decoder layer information interaction and channel nonlocal information enhancement attention, high-quality image dehazing results are achieved, and the best performance is achieved on multiple natural image and remote sensing datasets such as RESIDE, Densehaze, NHhaze and StateHaze1k.

[0056] 2) The multi-level feature interaction module proposed in this invention integrates shallow and deep information from the encoding stage into each layer of features in the decoding stage, reducing feature information dilution and effectively improving the ability of the dehazing network to recover details, semantics, and scene information.

[0057] 3) The efficient channel nonlocal information enhancement attention mechanism proposed in this invention utilizes ID convolution and nonlocal information fusion to effectively learn channel weight allocation. While reducing the number of learning parameters, it better strengthens important channel features for dehazing and suppresses unimportant channel features, thereby improving the network dehazing performance. Attached Figure Description

[0058] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0059] Figure 1 This is an overall network structure diagram of an embodiment of the present invention;

[0060] Figure 2 for Figure 1 A schematic diagram of the structure of the Single Feature-Channel Nonlocal Information Enhancement Attention Module (SF-NEA);

[0061] Figure 3 for Figure 1 A schematic diagram of the Multi-Feature-Channel Nonlocal Information Enhanced Attention Module (MF-NEA). Detailed Implementation

[0062] like Figures 1 to 3 As shown, a method for constructing an image dehazing network with multi-level feature interaction and efficient channel non-local information enhancement attention includes the following steps:

[0063] S1. Construct a U-shaped image dehazing network, which includes: a feature extraction module in the encoding layer, a feature repair module, and an image restoration module in the decoding layer.

[0064] S2. Construct a multi-level feature fusion module, which uses the features extracted by the coding layer feature extraction module in S1 to fuse features from different levels.

[0065] S3, a channel non-local information enhancement attention module is constructed, which includes two sub-modules, namely a single feature-channel non-local information enhancement attention module (SF-NEA) and a multi-feature-channel non-local information enhancement attention module (MF-NEA).

[0066] S4, the foggy image is input into the U-shaped image defogging network, and through the multi-level feature fusion module and the channel non-local information enhancement attention module, a clear fog-free image is output, and finally the output clear image is used to calculate the loss to constrain the training of the network.

[0067] Step S1 specifically includes:

[0068] As shown in Figure 1 , the encoding layer feature extraction module adopts 3 times of convolution operation for 4 times of down-sampling for feature extraction. Each layer of the encoding stage is composed of a convolution layer, an activation layer and an Instance-Norm. Secondly, 6 consecutive residual block combinations are used on the extracted low-resolution features to further strengthen the feature extraction. Finally, the features containing a large amount of semantic scene information extracted by the consecutive residual blocks are decoded and reconstructed by deconvolution and convolution operations to restore to the original image resolution.

[0069] A multi-level feature fusion module is constructed, which fuses features of different levels by using the features extracted by the encoding layer feature extraction module in S1. As shown in Figure 1 , the multi-level feature interaction module is shown in the part of the frame selected in Figure 1 , taking the EB2 encoding layer as an example, first, a 3*3 convolution kernel is used to down-sample EB1 layer to the same resolution as EB2 layer, and a 3*3 deconvolution layer is used to restore EB3 layer to the same resolution as EB2 layer, and finally the features containing different information are fused to obtain feature EF2. The multi-level feature interaction module proposed in this paper has three multi-level interaction processes, which are expressed by the following formulas:

[0070] EF1=(Conv(EB1)+TConv(EB2)+TConv(EB3))

[0071] EF2=(Conv(EB1)+Conv(EB2)+TConv(EB3))

[0072] EF3=(Conv(EB1)+Conv(EB2)+Conv(EB3))

[0073] Wherein EBi represents the encoding feature of the i-th layer, EFi represents the fusion feature of the i-th layer, DBi represents the decoding layer feature of the i-th layer, our multi-level feature interaction module has three multi-level interaction processes, so i belongs to {1, 2, 3}, Conv and Tconv represent convolution and deconvolution respectively. In this way, the encoding layer shallow detail texture and deep semantic features are fully fused into each decoding layer, and the feature dilution problem caused by the upsampling process in the decoding stage is effectively alleviated, and finally the purpose of improving the recovery ability of the haze removal network in detail, semantic and scene information is achieved.

[0074] Step S2 specifically comprises: constructing a channel non-local information enhancement attention module, which includes two sub-modules, namely a single feature-channel non-local information enhancement attention module (SF-NEA) and a multi-feature-channel non-local information enhancement attention module (MF-NEA).

[0075] As shown in Figure 2 and Figure 3 , the SF-NEA module first performs global average pooling and one-dimensional convolution operation on the feature to capture the local channel dependency. Secondly, the non-local information is obtained by using the autocorrelation operation, and is fused into the vector representing the local channel dependency to make up for the lack of non-local information. Finally, the one-dimensional convolution operation is used again to learn the weight of the feature channel. The single feature-NEA module can mine more effective feature channel information for feature map weight allocation by introducing non-local channel information and twice 1D convolution learning. The overall process of MF-NEA is divided into three processes of aggregation-dispersion-aggregation. The first aggregation is to fuse features of different levels to obtain fusion features with more perfect texture, details and semantic information. The dispersion process is to adaptively learn the weights of different level features according to the fusion feature statistics, and to distribute the learned weights to the corresponding level features. The second aggregation is to fuse the same channel features of different level features according to the weight for the reconstruction of the defogged image.

[0076] Step S3 specifically comprises:

[0077] The hazy image is input into the U-shaped image defogging network, and then output clear haze-free image through the multi-level feature fusion module and the channel non-local information enhancement attention module. Finally, the clear image output is used to calculate the loss. Finally, four kinds of loss are used to constrain the network training process. The four loss constraints are as follows:

[0078] The first one is L1 loss, and the specific formula is:

[0079]

[0080] x i and yi respectively represent the value of the hazy image and the GT image at pixel i, G() represents the dehazing network parameters, G(x i ) represents the value of the pixel at the input image i, and then the value obtained by operating the dehazing network parameters. N represents the number of pixels in the image.

[0081] The second is the perception loss, which uses the pre-trained model of VGG16 on ImageNet to calculate the perception loss; the specific formula is:

[0082]

[0083] Where x and y represent the hazy image and the GT image respectively. i represents the i-th layer of the feature map, H represents the length of the feature map, W represents the width of the feature map, and C represents the channel of the feature map. Among them, C i represents the channel of the i-th layer of the feature map, W i represents the width of the i-th layer of the feature map, and H i represents the length of the i-th layer of the feature map. φ i (x) represents the i-th layer feature map obtained after the input hazy image is processed by the VGG16 pre-trained model, with a length of H i , a width of W i , and a channel number of C i . || || represents the L2 norm, and N represents the number of VGG16 pre-trained model feature layers used in the perception loss.

[0084] The third is the multi-scale structural similarity loss, and the specific formula is:

[0085]

[0086] Where x represents the generated image, and y represents the clear image. μ x , μ y represent the mean of the generated image and the GT image respectively. σ x , σ y represent the standard deviation, and σ xy represents the covariance of the generated image and the clear image. β m γ m represent the relative importance of the two terms, and C1 and C2 are constant terms. M represents the total number of scales.

[0087] The fourth is the adversarial loss, and the specific formula is:

[0088]

[0089] Where D(y) represents the probability of judging the dehazed image y as a clear image, and N represents the total number of images.

[0090] The loss function of the whole network is represented as:

[0091] L loss = λ1L adv + λ2L l1 + λ3L perc + λ4L ms-ssim

[0092] Wherein λ1, λ2, λ3, λ4 are hyperparameters of each loss function, λ1 = 0.5, λ2 = 1, λ3 = 1, λ4 = 1.

[0093] Embodiment

[0094] 1. Parameter setting

[0095] The code of the application is implemented based on the Pytorch framework, and the experiment is performed on an NVIDIA RTX3090Ti GPU. The network is optimized by using the Adam optimizer, and the learning rate and batch size are set to 0.001 and 8 respectively. The momentum decay index β1 = 0.9, β2 = 0.999. The initial learning rate is set to 0.001, and the cosine annealing strategy is used to adjust the learning rate, and the half cycle of the cosine function is set to 5. In addition, the application evaluates various dehazing algorithms on the synthetic dataset RESIDE and the real dataset DenseHaze, NHHaze21 and SateHaze1k of the NTIRE dehazing challenge. In the RESIDE dataset, we use the outdoor training set OTS to train the network, and the outdoor of SOTS as the test set. Among them, OTS contains 8970 clear pictures and 313950 foggy pictures, and SOTS contains 500 indoor test data and outdoor test data set. DenseHaze contains 45 dense fog data sets, including 35 training data sets, 5 validation sets and 5 test sets. NHHaze2021 contains 25 non-uniform foggy images. Since the GT images of the validation set and the test set have not been published, we select the first 20 as the training set and the remaining 5 as the test set for evaluation. The public remote sensing dataset SateHaze1k contains three sub-datasets, representing different concentrations of fog, among which Thin represents the thin fog dataset in the sub-dataset, Moderate represents the moderate fog dataset in the sub-dataset, and Thick represents the thick fog dataset in the sub-dataset. Each sub-training set contains 320 images, the validation set contains 35 images, and the test set contains 45 images. In order to verify the correctness and effectiveness of the method of the application, the currently excellent dehazing algorithms and the method of the application are compared, and these methods are as follows: DCP, AOD-Net, GCA-Net, EPDN, Grideehaze-Net, MSBDN, FFA, AECR, TBN

[0096] 2. Experimental results

[0097] Table 1 Comparison with SOTA methods on SOTS-outdoor dataset and real scene dataset

[0098]

[0099] Our invention achieves the first place in PSNR and the second place in SSIM on SOTS(outdoor) dataset. As shown in the second column of Table 1, the end-to-end based dehazing algorithms are generally superior to the parameter estimation based algorithms. Among the learning based methods, our algorithm improves 11.31db, 1.72db, 2.98db in PSNR and 0.118, 0.005, 0.013 in SSIM compared with EDPN, MSBDN, AECR based on U-shaped structure respectively. These algorithms only fuse their corresponding coding features or perform feature fusion on the encoding layer and decoding layer respectively through dense connection, ignoring the utilization between each layer in the encoding layer stage. Our algorithm fuses the encoding features of different levels and uses the proposed attention to mine the channels that are more important for dehazing, improving the network dehazing performance and generating higher quality dehazed images. In addition, the dehazing algorithm proposed in this invention achieves 18.34db PSNR, 0.609SSIM and 23.47db PSNR, 0.873SSIM on Dense-haze and NH-haze21 datasets respectively, which are superior to all the compared dehazing methods and achieve excellent performance.

[0100] 3. Ablation analysis

[0101] In order to evaluate the effectiveness of each module of the invention, the invention designs an ablation experiment according to the innovation of the framework and the innovation of attention. There are 7 experiments in total:(1) Base represents the U-shaped basic framework, which is mainly composed of two down-sampling layers, six residual blocks and two up-sampling layers, wherein the encoding layer and the decoding layer directly use the skip connection.(2) Base+1MFS: Base framework + one multi-scale feature fusion skip connection(3) Base+2MFS: Base framework + two multi-scale feature fusion skip connections(4) Base+3MFS: Base framework + three multi-scale feature fusion skip connections(5) Base+3MFS+CA: Base framework + three multi-scale feature fusion skip connections + CA + PA(6) Base+3MFS+ECA: Base framework + three multi-scale feature fusion skip connections + ECA + PA(7) Base+3MFS+NEA: Base framework + three multi-scale feature fusion skip connections + NEA + PA.

[0102] Table 2 PSNR and SSIM results on SOTS outdoor dataset

[0103]

[0104] Ablation experiments were tested on the SOTS outdoor dataset, and the seven experiments described above were compared, and the PSNR and SSIM results are shown in the table. First, the base Base framework achieved 29.55 and 0.963 in PSNR and SSIM. On the basis of the benchmark Base, we added a multi-scale feature fusion jump connection, and the indicators were improved by 2.03dB in PSNR and 0.005 in SSIM. By continuously adding multi-scale feature fusion jump connection, PSNR gradually increased, and on the basis of three multi-scale feature fusion jumps in Base, PSNR was improved by 2.66db, and SSIM was improved by 0.007. The experiment verifies the effectiveness of the multi-scale feature fusion jump. Secondly, in order to verify the performance of our attention module, on the basis of three multi-scale feature fusion jumps, we compared the results of CA, ECA and the attention model proposed in this paper respectively. As shown in Table 3, using CA on the basis of three feature fusion jumps, PSNR is improved by 1.3db, and SSIM is improved by 0.008. Using ECA to avoid channel dimension reduction, its performance is slightly improved. Finally, using the efficient channel non-local information enhancement attention model designed in this paper, compared with no attention, PSNR is improved by 1.67db, and SSIM is improved by 0.011. Compared with the CA module, PSNR is improved by 0.37db, and SSIM is improved by 0.003.

[0105] The application proposes a single image defogging algorithm based on multi-level feature interaction and efficient channel non-local information enhancement attention, which includes a multi-level feature fusion module and an efficient channel non-local information enhancement attention module. The multi-level feature fusion module fully utilizes the information of different levels of the encoding layer, and the efficient channel non-local information enhancement attention module fully mines more effective channel features by adding new information guidance. The application verifies the effectiveness of multi-scale feature fusion, and effectively improves the defogging effect of the network and restores clearer images with higher quality.

Claims

1. A method for constructing a multi-layered feature interaction dehazing network based on U-Net, comprising the following steps: Step S1: Construct a U-shaped image dehazing network; Step S2: Construct a Channel Nonlocal Information Enhanced Attention Module (NEA) and add it to the U-shaped network; Step S3: Feed the foggy image into the U-shaped image dehazing network. Through the multi-level feature fusion module and the channel non-local information enhancement attention module, a clear fog-free image is output. Finally, the loss is calculated using the output clear image to constrain the training of the network. The above steps are used to construct a multi-level feature interaction dehazing network based on U-Net; In step S1, the constructed U-shaped image dehazing network is as follows: The first layer of the coding layer feature extraction module is EB1 → InstanceNorm layer IN1 → the second layer of the coding layer feature extraction module is EB2 → InstanceNorm layer IN2 → the third layer of the coding layer feature extraction module is EB3 → InstanceNorm layer IN3; InstanceNorm layer IN1, InstanceNorm layer IN2, InstanceNorm layer IN3 → Third multi-level feature interaction module MFS3 → Obtain fused feature EF3 → Third multi-feature channel non-local information enhancement attention MF-NEA3; The third multi-feature channel non-local information enhancement attention MF-NEA3, decoding and recovery layer DB3 → fusion operation → first deconvolution layer → InstanceNorm layer IN4 → SF-NEA module → decoding and recovery layer DB2; InstanceNorm layer IN2, InstanceNorm layer IN1, InstanceNorm layer IN3 → Second multi-level feature interaction module MFS2 → Obtain fused feature EF2 → Second multi-feature channel non-local information enhancement attention MF-NEA2; The second multi-feature channel non-local information enhancement attention (MF-NEA2) is followed by the decoding and recovery layer DB2 → fusion operation → the second deconvolution layer → the instanceNorm layer IN5 → the SF-NEA module → the decoding and recovery layer DB1. InstanceNorm layer IN3, InstanceNorm layer IN2, InstanceNorm layer IN1 → First multi-level feature interaction module MFS1 → Obtain fused feature EF1 → First multi-feature channel non-local information enhancement attention MF-NEA1; The first multi-feature channel non-local information enhancement attention layer MF-NEA1, decoding and recovery layer DB1 → fusion operation → third deconvolution layer → fog-free image.

2. The method according to claim 1, characterized in that, The specific operation of the third multi-level feature interaction module MFS3 is as follows: perform 1*1 Convolution on the InstanceNorm layer IN3, perform 3*3 Convolution on the InstanceNorm layer IN2, and perform 3*3 Convolution on the InstanceNorm layer IN1 → fusion operation; The specific operation of the second multi-level feature interaction module MFS2 is as follows: perform 1*1 Convolution and 3*3 Convolution on the InstanceNorm layer IN2, and perform 3*3 Convolutiontranspose→fusion operation on the InstanceNorm layer IN3. The specific operation of the first multi-level feature interaction module MSF1 is as follows: perform 1*1 Convolution on InstanceNorm layer IN1, perform 3*3 Convolution transpose on InstanceNorm layer IN2, and perform 3*3 Convolution transpose → fusion operation on InstanceNorm layer IN3.

3. The method according to claim 1, characterized in that, In step S2, the single-feature-channel nonlocal information enhancement attention module (SF-NEA) structure in the channel nonlocal information enhancement attention module is as follows: Input feature F → Global average pooling (GAP) operation → Channel descriptor vector S → 1D convolution operation → Vector containing local information ; The transpose operation of the channel descriptor vector S yields the transpose vector of the channel descriptor. →Channel Descriptor Vector The transpose of the channel descriptor After dot product operation → a vector containing nonlocal information ; Vectors containing non-local information Vectors containing local information → Fusion operation → 1D convolution operation → Feature weight assignment W; Assign weights W to features, input feature map F → perform pixel-wise multiplication → feature map F*.

4. The method according to claim 1, characterized in that, The Multi-Feature-Channel Non-Local Information Enhancement Attention Module (MF-NEA) structure in step S3 is as follows: Multi-level features EC1, EC2, EC3 → Fusion operation → Fusion feature EF → Global average pooling (GAP) operation → Three 1D convolution operations → Vector containing local information , , ; Fusion feature channel descriptor → The transpose operation yields the transposed fused feature channel descriptor. → Fusion Feature Channel Descriptor , Performing a dot product operation results in a vector containing non-local information. Vectors containing local information , , →Concat operation→Softmax activation function→Obtain weights W1, W2, W3→W1 is multiplied by input feature Ec1, W2 is multiplied by input feature Ec2, and W3 is multiplied by input feature Ec3 respectively→Fusion operation→Fuse feature F.

5. The method according to claim 1, characterized in that, In step S1, the U-shaped image dehazing network includes: a coding layer feature extraction module, a feature inpainting module, a decoding layer image restoration module, a single feature-channel nonlocal information enhancement attention module (SF-NEA), and a multi-feature-channel nonlocal information enhancement attention module (MF-NEA); in step S2, the channel nonlocal information enhancement attention module (NEA) includes two sub-modules, namely the single feature-channel nonlocal information enhancement attention module (SF-NEA) and the multi-feature-channel nonlocal information enhancement attention module (MF-NEA).

6. The method according to claim 1, characterized in that, In step S4, the image dehazing network with multi-level feature interaction and efficient channel non-local information enhancement attention, the multi-level feature fusion module, and the channel non-local information enhancement attention module are used in accordance with the following steps: Step 1) Input the foggy image into the U-shaped image dehazing network; Step 2) Input the feature information extracted by the U-shaped network into the multi-level feature fusion module to obtain the fused features; Step 3) The fused features are fused with the corresponding encoding layers to obtain features with more detailed texture semantics. The performance of the dehazing network is improved by enhancing the attention module through channel nonlocal information, resulting in a clear and hazy final output image. Step 4) The process of training the network using four types of loss constraints; In step 4), the four loss constraints are as follows: (1) L1 loss, the specific formula is: ； and These represent the values ​​at pixel i in the foggy image and the ground truth image, respectively. The value represents the pixel value at position i in the input image, which is then processed by the dehazing network parameters; N represents the number of pixels in the image. (2) Perceptual loss: The perceptual loss is calculated using a pre-trained VGG16 model on ImageNet; the specific formula is as follows: ； Where x and y represent the foggy image and the ground truth image, respectively, i represents the i-th layer of the feature map, H represents the length of the feature map, W represents the width of the feature map, and C represents the channels of the feature map. This represents the channel of the i-th layer of the feature map. The width of the i-th layer of the feature map. This represents the length of the i-th layer of the feature map. The input foggy image, after being pre-trained by the VGG16 model, yields the i-th layer with a size and length of [value missing]. Width is The number of channels is Feature map; denoted by L2 norm, and N represents the number of feature layers of the VGG16 pre-trained model used in the perceptual loss; (3) Multi-scale structural similarity loss, the specific formula is: ； Where x represents the generated image and y represents the sharpened image. , represent the mean values ​​of the generated image and the ground truth image, respectively. , They represent standard deviation, The covariance between the generated image and the sharpened image. This indicates the relative importance of two items. , Here, M is a constant term, representing the total number of scales. (4) Countermeasures against losses, the specific formula is: ； in, This indicates the image after dehazing. The probability of judging it as a clear image, where N represents the total number of images and n indicates starting from the first image; The loss function of the overall network is expressed as: ； Where λ1, λ2, λ3, and λ4 are hyperparameters of each loss function.

Images