A construction method of a dual-branch remote sensing image defogging network

Abstract

The application discloses a construction method of a dual-branch remote sensing image defogging network, which comprises the following steps: S1, constructing a U-shaped lower branch image defogging subnetwork; S2, constructing a non-U-shaped upper branch image defogging subnetwork, which comprises two convolutional layers and a multi-scale residual hollow convolutional module, wherein the first convolutional layer is used for increasing the channel number of an input image, and the second convolutional layer is used for restoring the channel number of a feature map to the channel number of the original input image; S3, constructing an image fusion module, which is used for fusing the output of the U-shaped lower branch image defogging subnetwork and the output of the non-U-shaped upper branch image defogging subnetwork; S4, sending a hazy image into the U-shaped lower branch image defogging subnetwork and the non-U-shaped upper branch image defogging subnetwork, and the image fusion module fuses the outputs of the two branches to output a clear remote sensing image; finally, the clear remote sensing image is used to calculate a loss, so as to constrain the training of the network; and the dual-branch remote sensing image defogging network is constructed through the above steps.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically relating to a method for constructing a dual-branch remote sensing image dehazing network. Background Technology

[0002] With the rapid development of remote sensing technology, remote sensing images are widely used in meteorology, surveying and mapping, disaster monitoring and early warning, and other fields. However, affected by weather phenomena such as haze, particles suspended in the air, such as water vapor, clouds, and fog, can cause problems such as image blurring, color distortion, and reduced contrast during the acquisition of remote sensing images by remote sensing satellites and drones, seriously affecting decision-making and command in high-level missions. Therefore, effective dehazing of remote sensing images and restoration of image detail information are of great significance.

[0003] In early dehazing algorithms, most methods estimated intermediate parameters such as transmission maps and global atmospheric light values ​​based on prior information, and then used atmospheric scattering models to achieve image dehazing. Although prior-based methods have made significant progress, these methods often suffer from inaccurate estimations under unconstrained conditions, resulting in less than ideal quality recovered haze-free images. With the development of deep learning, more and more researchers are designing end-to-end dehazing network models that directly output clear images without needing to estimate intermediate parameters of the atmospheric scattering model. Deep learning-based dehazing methods use U-shaped and non-U-shaped architectures to design network models. For example, Qu et al.'s "Enhanced Pix2pix Dehazing Network" proposes an enhanced pix2pix dehazing network based on a U-shaped structure. This network is independent of the atmospheric scattering model and directly restores hazy images to clear images by designing a multi-resolution generator, a multi-scale discriminator, and an enhancement network. Dong et al.'s "FD-GAN: Generative Adversarial Networks with Fusion-Discriminator for Single Image Dehazing" proposes a fully end-to-end generative adversarial network with a fusion discriminator for image dehazing. This method uses a U-shaped structure to design the network model and fuses frequency information as additional prior information into the discriminator, making the clear images restored by the generator network model more realistic and natural. Meanwhile, many researchers have designed network models based on non-U-shaped structures. Li et al.'s paper, "A Coarse-to-Fine Two-Stage Attention Network for Haze Removal of Remote Sensing Image," proposed a two-stage image dehazing network from coarse to fine. This method uses a non-U-shaped network model, with its core structure composed of dense residual blocks. The network depth is increased by stacking a large number of dense residual block structures. Similarly, Qin et al.'s paper, "FFA-Net: Feature Fusion Attention Network for Single Image Dehazing," proposed an image dehazing network based on feature fusion attention. It leverages the characteristics of residual learning to stack a large number of channel attention and pixel attention blocks, thus achieving better dehazing results.

[0004] Although deep learning-based end-to-end dehazing methods have achieved better dehazing results than those based on physical models, several problems still exist when using U-shaped and non-U-shaped networks for image dehazing:

[0005] 1) U-shaped network-based algorithms directly pass features from the encoding layer to the corresponding decoding layer, lacking information interaction between shallow and deep features. Furthermore, downsampling based on the U-shaped network structure model destroys spatial detail information in the image, resulting in a lack of texture structure information in the recovered haze-free image.

[0006] 2) Dehazing methods based on non-U-shaped networks have limited receptive fields, which prevents them from acquiring more contextual information that is crucial for dehazing remote sensing images, resulting in insufficient dehazing effects. Summary of the Invention

[0007] The purpose of this invention is to address the technical problems in existing technologies for image dehazing using U-shaped and non-U-shaped networks. U-shaped network-based algorithms directly pass features from the encoding layer to the corresponding decoding layer, lacking information interaction between shallow and deep features. Downsampling based on the U-shaped network structure model destroys spatial detail information, resulting in a lack of texture structure information in the recovered haze-free image. Non-U-shaped network-based dehazing methods have limited receptive fields, failing to acquire more contextual information crucial for remote sensing image dehazing, thus leading to poor dehazing results.

[0008] A method for constructing a dual-branch remote sensing image dehazing network includes the following steps:

[0009] S1. Construct a U-shaped lower branch image dehazing subnet, which includes: a coding layer feature extraction module, a hierarchical feature interaction fusion module, and a decoding layer image restoration module;

[0010] S2. Construct a non-U-shaped upper branch image dehazing subnet. This subnet contains two convolutional layers and a multi-scale residual dilated convolutional module. The first convolutional layer is used to increase the number of channels in the input image, and the second convolutional layer is used to restore the number of channels in the feature map to the number of channels in the original input image.

[0011] S3. Construct an image fusion module to fuse the output of the U-shaped lower branch image dehazing subnet and the output of the non-U-shaped upper branch image dehazing subnet;

[0012] S4. The foggy image is fed into the U-shaped lower branch image defogging subnet and the non-U-shaped upper branch image defogging subnet. The image fusion module will fuse the outputs of the two branches to output a clear remote sensing image. Finally, the clear remote sensing image is used to calculate the loss, thereby constraining the training of the network.

[0013] The above steps are used to construct a dual-branch remote sensing image dehazing network.

[0014] like Figure 2 As shown, in step S1, the constructed U-shaped lower branch image dehazing subnet is as follows:

[0015] The first layer of the coding layer feature extraction module → the second layer of the coding layer feature extraction module → the third layer of the coding layer feature extraction module → the fourth layer of the coding layer feature extraction module → the first layer of the decoding layer image restoration module → the second layer of the decoding layer image restoration module;

[0016] The third layer of the coding layer feature extraction module, the fourth layer of the coding layer feature extraction module → the first layer feature interaction fusion module;

[0017] The second layer of the coding layer feature extraction module, the first layer feature interaction fusion module → the second layer feature interaction fusion module;

[0018] The first layer of the coding layer feature extraction module, the second layer feature interaction fusion module → the third layer feature interaction fusion module;

[0019] The second layer of the decoding layer image restoration module, the first layer feature interaction fusion module → the first Concat operation → the third layer of the decoding layer image restoration module;

[0020] The third layer of the decoding layer image restoration module, the second-level feature interaction fusion module → the second Concat operation → the fourth layer of the decoding layer image restoration module;

[0021] The fourth layer of the decoding layer image restoration module, the third layer feature interaction fusion module → the third Concat operation → the fifth layer of the decoding layer image restoration module → the sixth layer of the decoding layer image restoration module.

[0022] The structures of the first-level feature interaction fusion module, the second-level feature interaction fusion module, and the third-level feature interaction fusion module are as follows: Figure 3 As shown, the sequence is: low-level features → ordinary convolution operation → first 1×1 convolution operation → first ReLU operation → second 1×1 convolution operation → first sigmoid operation;

[0023] High-level features → ordinary deconvolution operation → global average pooling operation → third 1×1 convolution operation → second ReLU operation → fourth 1×1 convolution operation → second sigmoid operation;

[0024] The first sigmoid operation is followed by a normal deconvolution operation, which then becomes the first pixel-wise multiplication operation.

[0025] The second sigmoid operation is a normal convolution operation followed by the second pixel-wise multiplication operation.

[0026] The first pixel-by-pixel multiplication operation, the second pixel-by-pixel multiplication operation → pixel-by-pixel addition operation.

[0027] like Figure 2As shown, in step S2, the structure of the non-U-shaped upper branch image dehazing subnet is as follows:

[0028] The first convolutional layer → N multi-scale residual dilated convolutional modules → element-wise addition operation → second convolutional layer; N represents the number of multi-scale residual dilated convolutional modules.

[0029] The structure of the multi-scale residual dilated convolution module is as follows: Figure 4 As shown;

[0030] Input layer → d(1,1) submodule; Input layer → d(1,2) submodule; Input layer → d(1,2,3) submodule;

[0031] Submodule d(1,1), d(1,2), d(1,2,3) → Concat operation → Global average pooling operation → First 1×1 convolution operation → ReLU operation → Second 1×1 convolution operation → sigmoid operation;

[0032] Concat and sigmoid operations are element-wise multiplication operations.

[0033] Input layer: element-wise multiplication → element-wise addition.

[0034] The structure of the d(1,1) submodule is as follows: input layer → first ordinary 3×3 convolution operation → ReLU operation → second ordinary 3×3 convolution operation; input layer, second 3×3 convolution operation → element-wise addition operation;

[0035] The structure of the d(1,2) submodule is as follows: input layer → ordinary 3×3 convolution operation → ReLU operation → 3×3 convolution operation with a dilation rate of 2; input layer, 3×3 convolution operation with a dilation rate of 2 → element-wise addition operation;

[0036] The structure of the d(1,2,3) submodule is as follows: input layer → ordinary 3×3 convolution operation → ReLU operation → 3×3 convolution operation with a dilation rate of 2 → ReLU operation → 3×3 convolution operation with a dilation rate of 3; input layer, 3×3 convolution operation with a dilation rate of 3 → element-wise addition operation.

[0037] In step S3, the structure of the constructed image fusion module is: convolution operation → Tanh activation operation;

[0038] The image fusion module is used as follows:

[0039] The sixth layer of the image restoration module in the lower branch decoding layer, and the second convolutional layer in the upper branch → Concat operation → image fusion module.

[0040] In step S4, the dual-branch remote sensing image dehazing network based on hierarchical feature interaction and enhanced receptive field includes a U-shaped lower branch image dehazing subnetwork and a non-U-shaped upper branch image dehazing subnetwork; the dual-branch remote sensing image dehazing network based on hierarchical feature interaction and enhanced receptive field adopts the following steps when in use:

[0041] Step 1) Input the foggy image into the U-shaped lower branch image defogging subnet to obtain the lower branch output image;

[0042] Step 2) Input the foggy image into the non-U-shaped upper branch image defogging subnet to obtain the upper branch output image;

[0043] Step 3) Input the output images of the lower branch and the upper branch into the image fusion module to obtain the final clear remote sensing image output by the network;

[0044] Step 4) The process of training the network using four types of loss constraints.

[0045] In step 4), the four loss constraints are as follows:

[0046] (1) Smooth the L1 loss, the specific formula is:

[0047]

[0048]

[0049] in, and Let represent the value of pixel i in the c-th channel of the clear image after dehazing and the true clear image, respectively; N represents the number of pixels in each channel; L smooth-L1 To smooth the L1 loss; α() is the smoothing function.

[0050] (2) Perceptual loss: The perceptual loss is calculated using a pre-trained VGG16 model on ImageNet; the specific formula is as follows:

[0051]

[0052] in, This indicates that the length of the j-th layer in the VGG16 pre-trained model is H. j Width is W j The number of channels is C j Feature maps; N represents the number of feature layers of the VGG16 pre-trained model used in the perceptual loss function; I * I represents the image after dehazing by the network. gt Represents a true, clear image; L p To perceive loss.

[0053] (3) Multi-scale structural similarity loss, the specific formula is as follows:

[0054]

[0055] Where, μ p μ g σ represents the mean of the dehazed image and the clear image, respectively; p , σ g σ represents the standard deviation of the dehazed image and the clear image, respectively; pg β represents the covariance between the dehazed image and the clear image. m γ m The relative importance between the two terms is represented by C1 and C2, which are constant terms to prevent division by zero; Lms-ssim is the multi-scale structural similarity loss.

[0056] (4) To counteract losses, the specific formula is:

[0057]

[0058] Wherein, D(I) * ) represents the dehazed image I * The probability of identifying it as a clear image; N represents the batch size set in the network; L adv To combat the losses.

[0059] The loss function of the overall network is expressed as:

[0060] L total =L smooth-L1 +α ms-ssim +βL p +γL adv

[0061] Where α, β, and γ are hyperparameters of each loss function.

[0062] Compared with the prior art, the present invention has the following technical effects:

[0063] 1) This invention proposes a method for constructing a dual-branch remote sensing image dehazing network. Through the dual-branch architecture, each branch can process the image input to the network independently, enabling the network to extract different information from the same input. Considering the complex and varied scenes in remote sensing images, using a dual-branch network can provide more comprehensive information than a single-branch network, thereby improving the performance of image dehazing.

[0064] 2) This invention proposes a hierarchical feature interaction fusion module, which gradually integrates deep semantic information and shallow detail texture information from the encoding stage to enhance the network's expressive power and recover higher quality, clearer images.

[0065] 3) This invention proposes a multi-scale residual dilated convolution module, which can increase the receptive field of the network without changing the feature map size, and fuse contextual information of different scales. Attached Figure Description

[0066] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0067] Figure 1 This is a flowchart of the image dehazing process according to an embodiment of the present invention;

[0068] Figure 2 This is an overall network structure diagram of an embodiment of the present invention;

[0069] Figure 3 for Figure 2 Structure diagram of the mid-level feature interaction fusion module HFIM;

[0070] Figure 4 for Figure 3 Structure diagram of the multi-scale residual dilated convolution module. Detailed Implementation

[0071] like Figures 1 to 4 As shown, a method for constructing a dual-branch remote sensing image dehazing network includes the following steps:

[0072] S1. Construct a U-shaped lower branch image dehazing subnet, which includes: a coding layer feature extraction module, a hierarchical feature interaction fusion module, and a decoding layer image restoration module.

[0073] S2. Construct a non-U-shaped upper branch image dehazing subnet, which contains two convolutional layers and a multi-scale residual dilated convolutional module. The first convolutional layer is used to increase the number of channels in the input image, and the second convolutional layer is used to restore the number of channels in the feature map to the number of channels in the original input image.

[0074] S3. Construct an image fusion module, which is responsible for fusing the output of the U-shaped lower branch image dehazing subnet and the output of the non-U-shaped upper branch image dehazing subnet.

[0075] S4. The foggy image is fed into the U-shaped lower branch image defogging subnet and the non-U-shaped upper branch image defogging subnet. The image fusion module will fuse the outputs of the two branches to output a clear remote sensing image. Finally, the loss is calculated using the output clear remote sensing image to constrain the training process of the network.

[0076] Step S1 specifically includes:

[0077] like Figure 2As shown in Figure ①, the encoding layer feature extraction module uses a Res2Net network with the fully connected layers removed as the encoding layer feature extraction module, and loads a pre-trained Res2Net network model on the ImageNet dataset.

[0078] like Figure 2 As shown in ②, the hierarchical feature interaction fusion module has the following specific network structure: Figure 3 The input to the hierarchical feature interaction fusion module can be represented as S = [s1, s2, ..., s...]. n ,..s N ], D=[d1,d2,..d m ,..d M Where S represents shallow features, D represents deep features, and N and M represent the number of channels in the input features. First, the shallow and deep features are subjected to convolution and deconvolution operations to adjust the width and height of their feature maps to H and W, and the number of channels to N. The specific formula is:

[0079] S * =Conv(S)

[0080] D * =Deconv(D)

[0081] Among them, S * With D * All feature maps have dimensions H and W, and N channels. To extract semantic information from deep features, this invention compresses the deep features according to the channel dimension, where each channel represents a type of semantic information. Specifically, the deep features are first processed using a global average pooling function, with the following formula:

[0082]

[0083] Where d * n (i,j) represents the nth single-channel feature map d * n The value of H at position (i,j) gap This represents the global average pooling function. The shape of the feature map changes from H×W×C to 1×1×C. Then, the channel vector f is passed through a simple gating mechanism with sigmoid activation (Jie et al., 2017), ensuring that multiple channels are emphasized (rather than forcing a single-hot activation). The entire process can be represented as:

[0084] CRB=σ(C2(δ(C1(f))))

[0085] Where σ and δ represent the Sigmoid and ReLU activation functions, respectively. C1 and C2 represent two convolutions, which convert the number of channels in the feature map to N / 8 and N, respectively.

[0086] To extract spatial detail information from shallow features, this invention directly uses shallow features S * The input is fed into two convolutional layers with ReLU and Sigmoid activation functions. The specific formula is:

[0087] SRB=σ(C4(δ(C3(S * ))))

[0088] The feature map shape changes from H×W×C to 1×H×W. This is because semantic information extracted from deep features is embedded into shallow features, and spatial detail information extracted from shallow features is embedded into deep features. Finally, these two weighted features are fused. The specific formula is:

[0089]

[0090] Based on the above reasoning and analysis, it can be seen that this invention introduces deeper semantic information into shallow features and spatial detail information into deep features, and then fuses the shallow and deep features. This hierarchical feature interaction and fusion method facilitates the full utilization of information extracted by the network, thereby recovering a clearer image of better quality.

[0091] like Figure 2 As shown in ③, the decoding layer image restoration module consists of a pixel-shuffle operation and an attention block. The attention block comprises a global average pooling layer, a convolutional layer, a ReLU activation layer, another convolutional layer, and a sigmoid activation layer. This module is finally connected to two convolutional layers.

[0092] Step S2 specifically includes:

[0093] The non-U-shaped upper branch image dehazing subnet consists of two convolutional layers and a multi-scale residual dilated convolution module. The multi-scale residual dilated convolution module is as follows: Figure 2 As shown in Figure ④, the specific network structure diagram is as follows: Figure 4 . Figure 4 The upper part shows the residual block structure in the ResNet network. The residual block structure addresses the degradation problem inherent in deep neural networks. This structure consists of two 3×3 convolutions, an activation layer, and a skip connection. This invention uses convolutions with different dilation rates in the residual block structure, such as... Figure 4The lower half shows: d(1,1) means using two 3×3 convolutions in the residual block, each with a dilation rate of 1; d(1,2) means using two 3×3 convolutions in the residual block, the first convolution with a dilation rate of 1 and the second convolution with a dilation rate of 2; d(1,2,3) means using three 3×3 convolutions in the residual block, the first convolution with a dilation rate of 1, the second convolution with a dilation rate of 2, and the third convolution with a dilation rate of 3.

[0094] Step S3 specifically includes:

[0095] like Figure 2 As shown in Figure ⑤, the image fusion module includes a Concat operation, convolutional layers, and Tanh activation layers. The Concat operation adds the outputs of the U-shaped lower branch network and the non-U-shaped upper branch network in the channel dimension. The convolutional layer converts the number of input channels to 3. Finally, the image passes through the Tanh activation layer to output a clear remote sensing image.

[0096] Step S4 specifically includes:

[0097] The foggy image is input into the U-shaped lower branch image defogging subnet to obtain the lower branch output image; the foggy image is input into the non-U-shaped upper branch image defogging subnet to obtain the upper branch output image; the lower branch output image and the upper branch output image are input into the image fusion module to obtain the final clear remote sensing image output by the network. Finally, the network training process is constrained by four types of loss, the first of which is smooth L1 loss, the specific formula of which is:

[0098]

[0099]

[0100] in, and represents the value of pixel i in the c-th channel of the clear image after dehazing and the true clear image, respectively. N represents the number of pixels in each channel.

[0101] The second method is perceptual loss, which is calculated using VGG16 at layers 3, 5, and 15 of the ImageNet pre-trained model. The specific formula is:

[0102]

[0103] in, This indicates that the length of the j-th layer in the VGG16 pre-trained model is H. j Width is W j The number of channels is C j The feature map. N represents the number of feature layers of the VGG16 pre-trained model used in the perceptual loss function.

[0104] The third type is multi-scale structural similarity loss, the specific formula of which is:

[0105]

[0106] Where, μ p μ g σ represents the mean of the dehazed image and the clear image, respectively. p , σ g σ represents the standard deviation of the dehazed image and the clear image, respectively. pg β represents the covariance between the dehazed image and the clear image. m γ m This indicates the relative importance between two terms. C1 and C2 are constant terms to prevent division by zero.

[0107] The fourth type is counter-loss, and the specific formula is as follows:

[0108]

[0109] Wherein, D(I) * ) represents the dehazed image I * The probability of judging it as a clear image.

[0110] The loss function of the overall network is expressed as:

[0111] L total =L smooth-L1 +α ms-ssim +βL p +γL adv

[0112] Where α, β, and γ are hyperparameters of each loss function, and α, β, and γ are set to 0.2, 0.001, and 0.005, respectively.

[0113] Example

[0114] 1. Parameter settings

[0115] The code for this invention is implemented using the PyTorch framework, and the network model is trained on an NVIDIA 2080Ti GPU in an Ubuntu environment. The Adam optimizer is used to optimize the network with a momentum decay exponent. The initial learning rate is set to 0.001, and a multi-step decay method is used to adjust the learning rate, with Gamma = 0.5. Furthermore, the effectiveness of this invention is evaluated on the publicly available remote sensing dataset SateHaze1k. The SateHaze1k dataset contains three subsets, each with different fog concentrations. The Thin subset contains images with light fog, the Moderate subset contains images with moderate fog, and the Thick subset contains images with dense fog. In each subset, the training set contains 320 images, the validation set contains 35 images, and the test set contains 45 images. To verify the correctness and effectiveness of the method, current excellent dehazing algorithms are compared with the method of this invention. These methods are as follows:

[0116] Type 1: Image dehazing methods based on prior knowledge, such as image dehazing methods based on dark channel priors (DCP).

[0117] Type 2: Image dehazing methods based on U-shaped structure network models, such as Enhanced Pix2Pix Image Dehazing Network (EPDN), Satellite Image Dehazing Based on Conditional Generative Adversarial Network (Huang), and Image Dehazing Network Based on Contrastive Learning (AECR).

[0118] Type 3: Image dehazing methods based on non-U-shaped structure network models, such as Feature Fusion Attention-Based Image Dehazing Network (FFA) and Coarse-to-Fine Two-Stage Remote Sensing Image Dehazing Network (FCTF).

[0119] 2. Experimental Results

[0120] To evaluate the performance of various methods, this invention uses Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) as evaluation metrics. These two metrics are commonly used in image dehazing tasks to evaluate the image quality restored by algorithms. The evaluation metrics of the proposed method and other dehazing methods are shown in Table 1.

[0121] Table 1. Quantitative evaluation of various algorithms on the SateHaze1k dataset.

[0122]

[0123] Experimental results show that the proposed method achieves excellent performance on the SateHaze1k dataset, with the best results in terms of PSNR and SSIM. Specifically, compared with algorithms EPDN, Huang, and AECR based on U-shaped network models, the proposed method improves PSNR by 5.622 dB, 3.199 dB, and 4.457 dB, respectively, and SSIM by 0.0464, 0.0224, and 0.0507, respectively, on the Thin fog subset. Compared with algorithms FFA and FCTF based on non-U-shaped network models, the proposed method improves PSNR by 1.668 dB and 3.977 dB, respectively, and SSIM by 0.0068 and 0.0191, respectively, on the Moderate fog subset. Furthermore, in the comparison algorithms, Huang.SAR indicates that radar data is introduced as auxiliary information into the defogging network to guide the network in recovering a clear image, and this achieves a significant performance improvement. However, acquiring radar data is relatively difficult, and the method proposed in this invention achieves better quantitative indicators than Huang.SAR even without radar data.

[0124] 3. Ablation analysis

[0125] To verify the effectiveness of each module in the proposed dehazing method, this invention designed ablation experiments based on the idea of ​​network structure and module innovation, which included a total of 6 experiments: (1) OUB (ordinary up branch) represents a network model that only contains the up branch, in which the up branch only uses a normal residual block structure. (2) ODB (ordinary down branch) represents a network model that only contains the down branch, in which the down branch is a normal U-shaped network without the addition of a hierarchical feature fusion module. (3) OTB (ordinary two branch) represents a two-branch network, in which the up branch network uses a normal residual block structure and the down branch network uses a normal U-shaped network. (4) OTB_MRDCB (ordinary two branch, Multiscaleresidual dilated convolution block) represents (3) with the addition of a multiscale residual dilated convolution module. (5) OTB_HFIM (ordinary two branch, Hierarchical feature interaction block) represents (3) with the addition of a hierarchical feature interaction module. (6) OTB_MRDCB_HFIM(ordinary two branch, Multiscale residual dilated convolution block, Hierarchical feature interaction block) means that on the basis of (3), both multiscale residual dilated convolution block and hierarchical feature interaction block are added. This model is also the final model used in this invention.

[0126] Table 2 shows the PSNR and SSIM results on the Thick subset.

[0127]

[0128] The ablation experiments were conducted on the Thick subtest set of SateHaze1k. This invention compared the above six experiments, and the PSNR and SSIM results are shown in Table 2. By comparing the first three models in Table 2, it can be found that the dual-branch network can improve the model's performance in terms of PSNR and SSIM to a certain extent. In addition, by observing the third and fourth models, the model with the multi-scale dilated convolution module improved PSNR and SSIM by 1.501dB and 0.0064, respectively, compared with the model without the module. This is because the multi-scale dilated convolution module enhances the receptive field of the network and integrates contextual information without changing the structural information of the input image. Furthermore, to verify the effectiveness of the hierarchical feature fusion module, this invention added a hierarchical feature fusion module to the ordinary dual-branch network. By comparing the third and fifth models in Table 2, it can be found that the hierarchical feature fusion module plays an important role in improving PSNR and SSIM. The increased PSNR and SSIM also indicate that gradually fusing deep semantic information and shallow detail texture information in the encoding stage plays an important role in improving dehazing performance.

[0129] This invention provides a two-branch image dehazing method based on hierarchical feature interaction and enhanced receptive field. Unlike existing models based on U-shaped network designs, this invention strengthens the interaction of feature information in the coding layers within the U-shaped network, fully utilizing the shallow and deep feature information extracted by the network, thereby enhancing the network's expressive power. Simultaneously, this invention increases the network's receptive field without changing the feature map size and integrates contextual information at different scales, thereby improving the network's dehazing performance, effectively enhancing the dehazing effect, and restoring higher-quality, clearer remote sensing images.

Claims

1. A method for constructing a dual-branch remote sensing image dehazing network, characterized in that, It includes the following steps: Step S1: Construct a U-shaped lower branch image dehazing subnet, which includes: a coding layer feature extraction module, a hierarchical feature interaction fusion module, and a decoding layer image restoration module; Step S2: Construct a non-U-shaped upper branch image dehazing subnet, which contains two convolutional layers and a multi-scale residual dilated convolutional module. The first convolutional layer is used to increase the number of channels in the input image, and the second convolutional layer is used to restore the number of channels in the feature map to the number of channels in the original input image. Step S3: Construct an image fusion module to fuse the output of the U-shaped lower branch image dehazing subnet and the output of the non-U-shaped upper branch image dehazing subnet; Step S4: The foggy image is fed into the U-shaped lower branch image defogging subnet and the non-U-shaped upper branch image defogging subnet. The image fusion module will fuse the outputs of the two branches to output a clear remote sensing image. Finally, the clear remote sensing image is used to calculate the loss, thereby constraining the training of the network. The dual-branch remote sensing image dehazing network is constructed using the above steps; In step S1, the hierarchical feature interaction fusion module includes a first hierarchical feature interaction fusion module, a second hierarchical feature interaction fusion module, and a third hierarchical feature interaction fusion module; The hierarchical feature interaction and fusion module is specifically as follows: low-level features → ordinary convolution operation → first time Convolution operation → First ReLU operation → Second Convolution operation → First sigmoid operation; High-level features → Ordinary deconvolution operation → Global average pooling operation → Third time Convolution operation → Second ReLU operation → Fourth operation Convolution operation → second sigmoid operation; The first sigmoid operation is followed by a normal deconvolution operation, which then becomes the first pixel-wise multiplication operation. The second sigmoid operation is a normal convolution operation followed by the second pixel-wise multiplication operation. The first pixel-by-pixel multiplication operation, the second pixel-by-pixel multiplication operation → pixel-by-pixel addition operation; In step S2, the structure of the non-U-shaped upper branch image dehazing subnet is as follows: First convolutional layer → 3 multi-scale residual dilated convolutional modules → element-wise addition operation → second convolutional layer; The structure of the multi-scale residual dilated convolution module is as follows: Input layer → d(1,1) submodule; Input layer → d(1,2) submodule; Input layer → d(1,2,3) submodule; Submodules d(1,1), d(1,2), and d(1,2,3) → Concat operation → Global average pooling operation → First time Convolution operation → ReLU operation → Second operation Convolution operation → sigmoid operation; Concat and sigmoid operations are element-wise multiplication operations. Input layer: element-wise multiplication → element-wise addition; The structure of the d(1,1) submodule is as follows: Input layer → First normal Convolution operation → ReLU operation → Second normal operation Convolution operation; input layer, second time Convolution operation → element-wise addition operation; The structure of the d(1,2) submodule is: Input layer → Normal Convolution operation → ReLU operation → dilation rate of 2 Convolution operation; input layer, dilation rate of 2. Convolution operation → element-wise addition operation; The structure of the d(1,2,3) submodule is: Input layer → Normal Convolution operation → ReLU operation → dilation rate of 2 Convolution operation → ReLU operation → dilation rate of 3 Convolution operation; input layer, dilation rate 3 Convolution operation → element-wise addition operation.

2. The method according to claim 1, characterized in that, In step S3, the structure of the constructed image fusion module is as follows: Convolution operation → Tanh activation operation; The image fusion module is used as follows: The sixth layer of the image restoration module in the lower branch decoding layer, and the second convolutional layer in the upper branch → Concat operation → image fusion module.

3. The method according to claim 1, characterized in that, In step S4, the dual-branch remote sensing image dehazing network includes a U-shaped lower branch image dehazing subnetwork and a non-U-shaped upper branch image dehazing subnetwork. The dual-branch remote sensing image dehazing network based on hierarchical feature interaction and enhanced receptive field is used in the following steps: Step 1) Input the foggy image into the U-shaped lower branch image defogging subnet to obtain the lower branch output image; Step 2) Input the foggy image into the non-U-shaped upper branch image defogging subnet to obtain the upper branch output image; Step 3) Input the output images of the lower branch and the upper branch into the image fusion module to obtain the final clear remote sensing image output by the network; Step 4) The process of training the network using four types of loss constraints.

4. The method according to claim 3, characterized in that, In step 4), the four loss constraints are as follows: (1) Smoothing L1 loss, the specific formula is: ； ； in, and These represent the values ​​of pixel i in the c-th channel of the clear image after dehazing and the truly clear image, respectively. This indicates the number of pixels in each channel; To smooth out L1 loss; It is a smoothing function; (2) Perceptual loss: The perceptual loss is calculated using a pre-trained VGG16 model on ImageNet; the specific formula is as follows: ； in, This indicates obtaining the first [unit / item] in the VGG16 pre-trained model. The length of the layer is Width is The number of channels is Feature map; This indicates the number of feature layers of the VGG16 pre-trained model used in the perceptual loss function; This represents the image after dehazing by the network. Represents a true and clear image; To perceive loss; (3) Multi-scale structural similarity loss, the specific formula is: ； in, , represents the mean values ​​of the dehazed image and the clear image, respectively; , These represent the standard deviations of the dehazed image and the clear image, respectively. This represents the covariance between the dehazed image and the clear image; , Indicates the relative importance between two items. , This is a constant term to prevent the divisor from being zero; For multi-scale structural similarity loss; (4) Countermeasures against losses, the specific formula is: ； in, This indicates the image after dehazing. The probability of judging it as a clear image; This indicates the Batchsize setting in the network; To combat the losses; The loss function of the overall network is expressed as: ； in , , These are the hyperparameters for each loss function.

Images