Construction method of embedded U-shaped remote sensing defogging network

By introducing parallel embedded U-shaped network framework and cross-level feature adaptive fusion module in remote sensing image defog network, feature dilution and cross-level feature interaction problems are solved, and high-quality remote sensing image defog effect is achieved, especially in Under real fog and non-uniform fog conditions.

CN119006845AActive Publication Date: 2024-11-22CHINA THREE GORGES UNIV

Patent Information

Application Number
CN202411153193.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-22
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

The existing remote sensing image defog removal network model has feature dilution when utilizing the feature information of the encoding layer and ignores cross-level feature interactions, resulting in unsatisfactory defog removal effect, especially when real fog, non-uniform fog images and scene changes greatly. Not good.

Method used

Build an embedded U-shaped remote sensing defog network, extract multi-scale features through a parallel embedded U-shaped network framework, and introduce cross-level feature adaptive fusion modules between the decoding layer and the encoding layer to adaptively fuse feature information at different levels. , reduce feature dilution and enhance network expression ability.

Benefits of technology

High-quality image defog results are achieved, significantly improving the clarity and detail recovery effect of remote sensing images, achieving the best performance on StateHaze1k and RICE1 datasets, proving the effectiveness of the algorithm.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006845A_ABST
    Figure CN119006845A_ABST
Patent Text Reader

Abstract

The invention provides a construction method of an embedded U-shaped remote sensing defogging network. The construction method comprises the following steps: S1, constructing a U-shaped image defogging network; s2, a plurality of parallel embedded U-shaped network frameworks are constructed and added between a U-shaped network coding layer and a decoding layer, and multi-scale feature information is extracted; s3, constructing an adaptive fusion mechanism, wherein the mechanism can adaptively fuse multi-scale features extracted from different levels of embedded U-shaped network modules; and S4, sending the foggy remote sensing image into the U-shaped image defogging network, finally outputting a clear fogless image through a plurality of parallel embedded U-shaped network frameworks and cross-level feature adaptive fusion modules, and finally calculating loss and constraining training of the network by using the output clear image. Through the above steps, the embedded U-shaped remote sensing defogging network based on the cross-level feature adaptive fusion of the U-shaped network is constructed. The method can improve the defogging effect of the defogging network.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method for constructing an embedded U-shaped remote sensing defogging network. Background Art

[0002] With the advancement of remote sensing technology, remote sensing images are widely used in the fields of environment, meteorology, ground observation, ocean monitoring, disaster monitoring and aerospace. However, during the image acquisition process, remote sensing satellites and drones are affected by tiny particles such as smoke and dust in the atmosphere, which leads to blurred images, reduced contrast, color distortion and other quality degradation. Therefore, how to effectively remove haze information in remote sensing images and restore image detail information has attracted the attention of many researchers.

[0003] At present, the methods of image defogging are mainly divided into two categories: defogging methods based on priors and defogging methods based on deep learning. Most of the early defogging works were developed on the basis of the classic atmospheric scattering model. Physical models based on the atmospheric scattering model have been used for image defogging, such as dark channel prior (DCP) and color attenuation (CAP) prior. In recent years, methods based on deep learning have been widely used in image defogging, image classification and face recognition. In the field of image defogging, convolutional neural network (CNN) has been widely used in natural image defogging and has achieved remarkable success in natural image defogging. Among them, many researchers have made improvements and innovations based on the traditional U-type network and are committed to improving the performance and effect of deep learning algorithms. At first, the researchers improved the architecture of the U-type network. They added jump connections on the basis of the traditional U-type network and improved the expression ability of the model. Although these methods have achieved certain defogging effects on remote sensing image defogging, they still have the following problems when using the U-type network and the effective fusion of different levels of features:

[0004] The decoding layer of most existing dehazing network models fails to fully utilize the feature information extracted by the encoding layer, resulting in feature dilution in the network model and unsatisfactory effect of dehazing remote sensing images.

[0005] Most dehazing algorithms ignore the interaction of cross-level feature information, making the fused features too redundant, thus affecting the scene restoration of the image and the constraints of contextual information.

[0006] Therefore, there is an urgent need to provide a method for building a defogging network that can achieve better restoration effects on real fog, non-uniform fog images, and remote sensing images with large scene changes. Summary of the invention

[0007] The purpose of the present invention is to solve the technical problems in the above background and propose a method for constructing an embedded U-shaped remote sensing defogging network, comprising the following steps:

[0008] S1, build a U-shaped image dehazing network;

[0009] S2, construct several parallel embedded U-shaped network frameworks and add them between the U-shaped network encoding layer and decoding layer to extract multi-scale feature information;

[0010] S3. Build an adaptive fusion mechanism that can adaptively fuse multi-scale features extracted from embedded U-shaped network modules at different levels

[0011] S4, the foggy remote sensing image is sent to the U-shaped image defogging network, and a clear fog-free image is finally output through several parallel embedded U-shaped network frameworks and cross-level feature adaptive fusion modules. Finally, the output clear image is used to calculate the loss and constrain the network training;

[0012] Through the above steps, an embedded U-shaped remote sensing dehazing network based on cross-level feature adaptive fusion of U-shaped network is constructed.

[0013] In a preferred solution, the U-shaped image dehazing network in step S1 includes a coding layer feature extraction module, a feature restoration module, a decoding layer image recovery module, a parallel embedded U-shaped framework and a cross-level feature adaptive fusion module CHAF.

[0014] In the preferred solution, in step S1, the U-shaped image defogging network structure is constructed as follows:

[0015]

[0016] Features E33, E23, E13 → Cross-level feature adaptive fusion module CHAF,

[0017] Features E32, E22, E12 → Cross-level feature adaptive fusion module CHAF,

[0018] Features E31, E21, E11 → Cross-level feature adaptive fusion module CHAF,

[0019] In the preferred solution, the process of feature extraction by the embedded U-shaped network framework in step S2 is expressed as follows:

[0020]

[0021] Among them, EU1, EU2, and EU3 represent parallel embedded U-shaped network frameworks, and E11, E12, and E13 represent hierarchical features of different scales output after passing through the EU1 module.

[0022] In a preferred solution, the plurality of parallel embedded U-shaped network framework structures in step S2 include:

[0023] The specific operations of EU1 are:

[0024] E14, E13→ cross-level feature adaptive fusion operation→ 1*1Convolution→ obtain feature E12; E12, E15→ cross-level feature adaptive fusion operation→ 1*1Convolution→ obtain feature E11;

[0025] The specific operations of EU2 are:

[0026] E24, E23→ cross-level feature adaptive fusion operation→ 1*1Convolution→ obtain feature E22; E22, E25→ cross-level feature adaptive fusion operation→ 1*1Convolution→ obtain feature E21;

[0027] The specific operations of EU3 are:

[0028] E34, E33→ cross-level feature adaptive fusion operation→ 1*1Convolution→ obtain feature E32; E32, E35→ cross-level feature adaptive fusion operation→ 1*1Convolution→ obtain feature E31.

[0029] In the preferred solution, step S3 includes: constructing a cross-level feature adaptive fusion module CHAF, which combines the size of H×W×S and The hierarchical features M and N of are extracted through pixel attention and channel attention respectively, which specifically includes the following steps:

[0030] First, use 3*3 convolution and 3*3 deconvolution on M and N respectively to adjust the length, width and number of channels of the feature map to H×W×S. The formula is:

[0031] M′=Conv2D(M)

[0032] N′=TConv(N)

[0033] Where M′ and N′ are both H×W×S feature maps, TConv represents a standard transposed convolution with a kernel size of 3*3, and Conv2D represents a convolution with a kernel size of 3 and does not change the size of the feature map;

[0034] Compress N′ according to the channel dimension. Each channel represents a kind of semantic information. The formula of N′ after global average pooling is:

[0035]

[0036] where F n (i, j) represents the value of the nth single channel feature map at position (i, j), H gap Represents the global average pooling function, the shape of the feature map F is squeezed from H×W×C to 1×1×C;

[0037] The channel vector f undergoes two 1x1 convolutions and activation functions. The whole process is expressed as:

[0038] D = σ(Conv1(δ(Conv2(f))))

[0039] Where σ and δ represent Sigmoid and Relu activation functions respectively, and Conv1 and Conv2 represent two 1x1 convolutions;

[0040] Secondly, M′ is directly activated by two convolution, Sigmoid and Relu functions, and the formula is:

[0041] S=σ(Conv4(δ(Conv3(M′))));

[0042] Then, S is multiplied by the adjacent level feature N′ to obtain M″; the extracted semantic information is embedded into the shallow features, and N′ is multiplied by the adjacent level feature D to obtain N″. The process is expressed as follows:

[0043]

[0044] Finally, the cross-level features are fused through the adaptive weight allocator, and the adaptive weight allocator formula is expressed as:

[0045] f=Awa(f i ,f j )=A(θ)×f i +A(1-θ)×f j

[0046] Where Awa represents the adaptive weight allocator, f i ,f j represents two different level features, and θ represents the learnable factor in the adaptive weight allocator.

[0047] In a preferred solution, the cross-level features in step S3 are adaptively fused in a CHAF module structure including:

[0048] Input level feature M→3*3Convolution→get feature M'→1*1Convolution→activation function Relu→1*1Convolution→activation function sigmoid→feature A with 1 channel number;

[0049] Input level feature N→3*3 deconvolution→get feature N'→global average pooling→1*1Convolution→activation function Relu→1*1Convolution→activation function sigmoid→1×1×C feature B;

[0050] Feature A, feature N'→pixel-by-pixel multiplication→feature M";

[0051] Feature B, feature M'→pixel-by-pixel multiplication→feature N";

[0052] Adaptive weight allocator assigns weights θ,M"→multiply pixel by pixel;

[0053] The adaptive weight allocator assigns weights 1-θ, N"→multiply pixel by pixel;

[0054] The two features after weight multiplication → fusion addition → feature J after cross-level fusion.

[0055] In a preferred solution, in step S4, the embedded U-shaped remote sensing defogging network, the embedded U-shaped network framework and the cross-level feature adaptive fusion module adopt the following steps when used:

[0056] S41, inputting the foggy image into the U-type network image defogging network;

[0057] S42, inputting the coding layers of different levels into the embedded U-shaped network structure of the corresponding layer to obtain information features;

[0058] S43, inputting the features of different scales extracted by the embedded U-net into the cross-level feature adaptive fusion module, and further fusing the fused features with the corresponding coding layer features to obtain the final output clear and fog-free image;

[0059] S44. Perform loss constraints on the network training process.

[0060] In a preferred solution, the loss constraints in step S44 include: LI loss constraints, adversarial loss constraints, perceptual loss constraints, and multi-scale structural similarity loss constraints;

[0061] The specific formula of L1 loss is:

[0062]

[0063] Where G represents the generation of NHU-Net, x i and i They represent the values ​​of blurred image and clear image at pixel i respectively, and N represents the number of pixels in the image;

[0064] The adversarial loss constraint uses the adversarial loss of GAN. The specific adversarial loss is defined as follows:

[0065]

[0066] Where G represents the generator, which converts the haze-free image into a clear image, and D is the discriminator, which distinguishes the generated clear image from the real haze-free image;

[0067] The perceptual loss constraint retains more texture information for the restored image and calculates the L2 norm of the clear image and the haze-free image at different layers of the VGG16 pre-trained model. The perceptual loss is defined as:

[0068]

[0069] in, Indicates the size of the feature map of layer i of the VGG16 pre-trained model, N indicates the number of feature layers of the VGG16 pre-trained model used in the perceptual loss, where x and y represent the foggy image and the clear image, respectively;

[0070] Multi-scale structural similarity loss constraint, the specific formula is:

[0071]

[0072] where μ i , μ j Represent the mean of the dehazed image and the clear image, σ i , σ j Represent the standard deviation of the dehazed image and the clear image, σ ij Used to represent the covariance of the dehazed image and the clear image, β m , γ m is the equation importance coefficient, C1 and C2 are constant terms.

[0073] In the preferred solution, the total loss function of the entire network is expressed as:

[0074] L=θ1L smoothL1 +θ2L adv +θ3L pere +θ4L ms-ssim

[0075] where θ1, θ2, θ3, and θ4 are hyperparameters that balance between different losses.

[0076] The beneficial effects of the present invention are:

[0077] 1) This paper proposes an embedded U-shaped remote sensing dehazing network with cross-level feature adaptive fusion. The parallel embedded U-shaped network structure extracts the coding layer information and passes it to the decoding layer for restoring the clear image and the cross-level feature adaptive fusion module to adaptively fuse the features, achieving high-quality image dehazing results. In addition, it is proved that the algorithms proposed by us have achieved the best performance on two public remote sensing image dehazing datasets, StateHaze1k and RICE1.

[0078] 2) The parallel embedded U-shaped network framework proposed in the present invention introduces multiple embedded parallel U-shaped structures between the decoding layer and the encoding layer, enhances the feature extraction of the encoding layer so that it can be supplemented to each decoding layer as supplementary information, thereby reducing the feature dilution in the process of reconstructing a clear image in the decoding layer and increasing the receptive field of the network.

[0079] 3) The cross-level feature adaptive fusion module proposed in the present invention explores how to effectively fuse adjacent level features. It adaptively fuses adjacent level features extracted by channel attention and pixel attention through learnable factors, which can enhance the expression ability of the network and thus better restore scene, detail and color information. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 The overall network structure diagram and multiple parallel embedded U-shaped network framework diagrams of the embodiment of the present invention;

[0081] Figure 2 This is a schematic diagram of the structure of the cross-level feature adaptive fusion module CHAF in the figure. DETAILED DESCRIPTION

[0082] Example 1

[0083] like Figure 1 and Figure 2 As shown, a method for constructing an embedded U-shaped remote sensing defogging network with cross-level feature adaptive fusion includes the following steps:

[0084] S1. Construct a U-shaped image dehazing network, which includes: a coding layer feature extraction module, a feature restoration module, a decoding layer image recovery module, a parallel embedded U-shaped framework, and a cross-level feature adaptive fusion module CHAF.

[0085] S2. Build a parallel embedded U-shaped framework, use the features extracted by the feature extraction module of each coding layer in S1, and embed each layer of features into the U-shaped network for feature extraction.

[0086] S3. Construct a cross-level feature adaptive fusion module to effectively and adaptively fuse multiple adjacent level features extracted by channel attention and pixel attention through learnable factors.

[0087] S4. Send the foggy image to the U-shaped image dehazing network, and output a clear fog-free image through the parallel embedded U-shaped network and the cross-level feature adaptive fusion module. Finally, the output clear image is used to calculate the loss and constrain the network training.

[0088] S1 specifically includes:

[0089] like Figure 1 As shown in the figure, the coding layer feature extraction module uses 4 convolution operations to perform 8 times downsampling to obtain different scale coding layers. Then, the extracted low-resolution features Six consecutive residual blocks are combined to further enhance feature extraction. Finally, the features extracted by the consecutive residual blocks, which contain a large amount of semantic scene information, are decoded and reconstructed using deconvolution operations to restore the original image resolution.

[0090] Step S2 specifically includes: constructing an embedded U-shaped network framework EU-Net

[0091] Construct a parallel embedded U-shaped network framework and use the features extracted by the encoding layer feature extraction module in S1 Then further extract features. Figure 1 As shown in the part with shading color, the first layer of the coding layer is shown. After the coding layer features are input into the embedded U-shaped network, three cross-scale features are obtained. The process of feature extraction of the embedded U-shaped network framework proposed in this paper is expressed as follows:

[0092]

[0093] Among them, EU1, EU2, and EU3 represent parallel embedded U-shaped network frameworks, and E11, E12, and E13 represent hierarchical features of different scales output after passing through the EU1 module. In this way, the present invention not only further effectively extracts the information extracted by the coding layer, but also adds the features extracted from the coding layers of different scales as supplementary information to each decoding layer, which can alleviate the feature dilution in the decoding process and increase the receptive field of the network.

[0094] Step S3 specifically includes: constructing a cross-level feature adaptive fusion module CHAF.

[0095] like Figure 2 As shown, the size is H×W×S and The horizontal features M and N. Since shallow features have more edge and texture information, deep features contain more semantic information. The two cross-level features are extracted through pixel attention columns and channel attention respectively. First, M and N are adjusted to H×W×S in length, width and number of channels using 3*3 convolution and 3*3 deconvolution respectively, as shown in the following formula:

[0096] M′=Conv2D(M)

[0097] N′=TConv(N)

[0098] Where M′ and N′ are both H×W×S feature maps, TConv represents a standard transposed convolution with a kernel size of 3*3, and Conv2D represents a convolution with a kernel size of 3 without changing the size of the feature map.

[0099] Since the deep feature N′ contains more semantic information, we compress N′ according to the channel dimension, and each channel represents a kind of semantic information. The formula of N′ after global average pooling is as follows:

[0100]

[0101] where F n (i, j) represents the value of the nth single channel feature map at position (i, j), H gap represents the global average pooling function, and the shape of the feature map F is squeezed from H×W×C to 1×1×C. In order to better obtain the information of each channel, we pass the channel vector f through two 1x1 convolutions and a simple activation function. The whole process can be expressed as:

[0102] D = σ(Conv1(δ(Conv2(f))))

[0103] Where σ and δ represent Sigmoid and Relu activation functions respectively. Conv1 and Conv2 represent two 1x1 convolutions.

[0104] Secondly, in order to extract the spatial details and texture information of shallow features, we directly use two convolution, Sigmoid and Relu activation functions for M′. As shown in the following formula:

[0105] S=σ(Conv4(δ(Conv3(M′))))

[0106] Then, in order to embed the detailed texture information into the adjacent level feature N′, this paper multiplies S with the adjacent level feature N′ to obtain M″. The extracted semantic information is embedded in the shallow feature, and N′ is multiplied with the adjacent level feature D to obtain N″. The process is shown in the formula:

[0107]

[0108] Finally, in order to adaptively fuse two cross-level features M″, N″ to obtain feature J, we consider an adaptive weight allocator to fuse cross-level features. The adaptive weight allocator can be formulated as:

[0109] f=Awa(f i ,f j )=A(θ)×f i +A(1-θ)×f j

[0110] Where Awa represents the adaptive weight allocator, f i ,f j represents two different level features, and θ represents the learnable factor in the adaptive weight allocator.

[0111] Step S4 specifically includes:

[0112] The foggy image is input into the U-shaped image defogging network. Through multiple parallel embedded U-shaped network modules and cross-level feature adaptive fusion modules, a clear fog-free image is finally output. Finally, the output clear image is used to calculate the loss. The four constraints of the network training process using four loss constraints are as follows:

[0113] The first is L1 loss, the specific formula is:

[0114]

[0115] Where G represents the generation of NHU-Net, x i and i Represent the values ​​of blurred image and clear image at pixel i respectively. N represents the number of pixels in the image.

[0116] The second is adversarial loss. We use the adversarial loss of GAN, where G represents the generator, which mainly converts the haze-free image into a clear image, and D represents the discriminator, which mainly distinguishes the generated clear image from the real haze-free image. The specific adversarial loss is defined as follows:

[0117]

[0118] The third type is perceptual loss. Perceptual loss can well retain more texture information for the restored image, and can well calculate the L2 norm of clear images and fog-free images at different layers of the VGG16 pre-trained model. The perceptual loss in this article is defined as follows:

[0119] in, Indicates the size of the feature map of layer i of the VGG16 pre-trained model. N indicates the number of feature layers of the VGG16 pre-trained model used in the perceptual loss. Where x and y represent the foggy image and the clear image, respectively.

[0120] The fourth type is to constrain the network using multi-scale structural similarity loss. The specific formula is:

[0121]

[0122] where μ i , μ j Represent the mean of the dehazed image and the clear image, σ i , σ j Represent the standard deviation of the dehazed image and the clear image, σ ij Used to represent the covariance of the dehazed image and the clear image, β m , γ m are the two important terms in the equation. C1 and C2 are constant terms.

[0123] The total loss function of the overall network is expressed as:

[0124] L=θ1L smoothL1 +θ2L adv +θ3L pere +θ4L ms-ssim

[0125] where θ1, θ2, θ3, and θ4 are hyperparameters that balance between different losses, θ1=1, θ2=0.0005, θ3=0.01, θ4=0.5.

[0126] Example 2

[0127] Parameter settings

[0128] The present invention evaluates various dehazing algorithms on two public datasets, SateHaze1 and RICE. The SateHaze1k dataset contains three sub-datasets, namely SateHaze 1k Thin, SateHaze 1kMederate and SateHaze 1k Thick, each of which contains 400 pairs of synthetic remote sensing images (the training set contains 320 pairs of images, the validation set contains 35 pairs of images, and the test set contains 45 pairs of images). The RICE dataset was collected on Google Earth. The dataset has 500 pairs of images with a resolution of 512x512, and each pair of images contains a clear image and a foggy image. In order to verify the effectiveness of each method, we divide the RICE1 dataset into a training set and a test set in a ratio of 9:1. In addition, the present invention uses the Adam optimizer to optimize the proposed network, with momentum decay exponents β1=0.9, β2=0.999, and the learning rate and batch size are set to 0.0001 and 4, respectively. The initial learning rate is set to 0.001, and MultiStepLR is used to dynamically adjust the learning rate between them. During the training process, we set the parameters of the loss function in the network model to θ1=1, θ2=0.0005, θ3=0.01, and θ4=0.5. All experiments in this paper are carried out on the NVIDIARTX3090 GPU, and the code is implemented based on Pytorch. In order to better evaluate the proposed method, the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used as evaluation indicators. We compare our dehazing network with seven other recent excellent algorithms. The compared algorithms are mainly DCP, FFA, AECR, SGID, FCTF, DCRD, and DCI-Net.

[0129] 2. Experimental results

[0130] Table 1 Comparison results of the method in this invention with each method on StateHaze1k and RICE1 datasets

[0131]

[0132] The present invention is quantitatively evaluated with other 7 algorithms on the SateHaze1k and RICE1 datasets, and Table I shows the average PSNR and SSIM values ​​of the tested methods. As shown in the fifth column of Table 1, DCP has only 11.37dB PSNR on the SateHaze1k dataset, while the remaining algorithms are all above 20dB PSNR, proving that the end-to-end dehazing algorithm is superior to the traditional parameter estimation algorithm. We note that on the SateHaze1k dataset, our faming is much higher than other methods in terms of thin fog, medium fog, and thick fog. Among them, the present invention has a significant improvement over the second place in the SateHaze1k Thin, SateHaze 1k Mederate, and SateHaze1k Thick datasets, with PSNR improvements of 3.03db, 1.24db, and 2.71db, respectively, and SSIM improvements of 0.028, 0.11, and 0.04, respectively. In addition, the method proposed in this paper achieved better results on the RICE1 dataset compared with other algorithms, especially compared with the algorithms in the remote sensing field, the PSNR was higher than 4dB and the SSIM was higher than 0.01.

[0133] In summary, the defogging network of the present invention has better defogging performance in light fog, medium fog, dense fog and real fog. Through the network structure of the present invention, the effect of haze in the image can be effectively reduced, the clarity and details of the image can be improved, and a more realistic scene can be restored.

[0134] 3. Ablation experiment

[0135] In order to evaluate the effectiveness of each module, the present invention mainly designs ablation experiments on the innovative embedded U-shaped network framework and cross-level feature adaptive fusion module proposed. There are five experiments in total: (1) Base: Basic U-shaped framework, mainly composed of four layers of downsampling, six residual blocks, channel attention, pixel attention and three layers of upsampling, in which the encoding layer and the decoding layer use jump connections. (2) Base+EU1: Based on the basic framework, the first layer uses the embedded U-shaped network for feature extraction. (3) Base+EU2: The first and second layers of the basic framework + downsampling use the embedded U-shaped network for feature extraction and jump connection. (4) Base+EU3: The basic framework + the first three layers use the embedded U-shaped network for feature extraction and jump connection. (5) Base+3EU+CHAF: The basic framework + the first three layers of downsampling use the embedded U-shaped network for feature extraction + cross-level feature adaptive fusion module and feature fusion of the decoding layer.

[0136] Table 2 PSNR and SSIM results on the SateHaze 1k Thin dataset

[0137]

[0138] For the fairness of the experiment, we train the above five experiments in the same way on the SateHaze 1k Thin dataset, and the PSNR and SSIM results are shown in Table II. First, the basic framework achieved 23.56 and 0.897 in PSNR and SSIM. On the basic framework, the first layer of downsampling was adopted by the proposed EU-Net, and the PSNR and SSIM were improved by 1.24dB and 0.016 respectively. The experimental results show the effectiveness of the EU-Net we proposed. Then, we used EU-Net to extract features for all the first three layers of downsampling, and both PSNR and SSIM were greatly improved. Finally, we used the CHAF module to perform cross-level feature adaptive fusion and then fused it with upsampling. Compared with the jump connection, the PSNR was improved by 0.28dB and the SSIM was improved by 0.001. The experimental results show that the present invention has a certain effect on remote sensing image dehazing and verifies the effectiveness of the CHAF module.

[0139] The present invention proposes an embedded U-shaped remote sensing defogging network algorithm with cross-level feature adaptive fusion. The network mainly includes multiple parallel embedded U-shaped network frameworks and cross-level feature adaptive fusion modules. The embedded U-shaped network framework can alleviate the feature dilution in the encoding process and increase the receptive field of the network by making full use of the features extracted by the encoding layer and adding them to each decoding layer as supplementary information. The cross-level feature adaptive fusion module can effectively utilize learnable factors to adaptively fuse cross-level features, thereby further improving scene restoration and contextual information constraints. The present invention verifies the effectiveness of the proposed framework and module, and can restore higher quality clear remote sensing images while improving network performance.

[0140] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing an embedded U-shaped remote sensing defogging network, characterized by: The following steps are involved: S1, build a U-shaped image dehazing network; S2, construct several parallel embedded U-shaped network frameworks and add them between the U-shaped network encoding layer and decoding layer to extract multi-scale feature information; S3, build an adaptive fusion mechanism that can adaptively fuse multi-scale features extracted from embedded U-shaped network modules at different levels; S4, the foggy remote sensing image is sent to the U-shaped image defogging network, and a clear fog-free image is finally output through several parallel embedded U-shaped network frameworks and cross-level feature adaptive fusion modules. Finally, the output clear image is used to calculate the loss and constrain the network training; Through the above steps, an embedded U-shaped remote sensing dehazing network based on cross-level feature adaptive fusion of U-shaped network is constructed.

2. According to claim 1, a method for constructing an embedded U-shaped remote sensing defogging network is characterized by: The U-shaped image dehazing network in step S1 includes a coding layer feature extraction module, a feature restoration module, a decoding layer image restoration module, a parallel embedded U-shaped framework and a cross-level feature adaptive fusion module CHAF.

3. The method for constructing an embedded U-shaped remote sensing defogging network according to claim 1, characterized in that: In step S1, the U-shaped image defogging network structure is constructed as follows: The first layer of the coding layer feature extraction module → Encoding layer feature extraction second layer → Encoding layer feature extraction third layer → Coding layer feature extraction fourth layer The first layer of the coding layer feature extraction module →Embedded U-shaped network framework EU1→output three features E11, E12, E13; The second layer of the coding layer feature extraction module →Embedded U-shaped network framework EU2→output three features E21, E22, E23; The third layer of the coding layer feature extraction module →Embedded U-shaped network framework EU3→output three features E31, E32, E33; Features E33, E23, E13 → Cross-level feature adaptive fusion module CHAF, decoding recovery layer →First deconvolution layer→Fusion operation→Channel attention module CA→Pixel attention module PA→Decoding recovery layer Features E32, E22, E12 → Cross-level feature adaptive fusion module CHAF, decoding recovery layer →The second deconvolution layer →Fusing operation →Channel attention module CA →Pixel attention module PA →Get features Features E31, E21, E11 → Cross-level feature adaptive fusion module CHAF, features → Fusion operation → The third deconvolution layer → Haze-free remote sensing image.

4. The method for constructing an embedded U-shaped remote sensing defogging network according to claim 1, characterized in that: The process of feature extraction by the embedded U-shaped network framework in step S2 is expressed as follows: Among them, EU1, EU2, and EU3 represent parallel embedded U-shaped network frameworks, and E11, E12, and E13 represent hierarchical features of different scales output after passing through the EU1 module.

5. The method for constructing an embedded U-shaped remote sensing defogging network according to claim 1, characterized in that: The multiple parallel embedded U-shaped network framework structures in step S2 include: The specific operation of EU1 is: →1*1Convolution without changing the size→3*3Convolution→3*3Convolution→get features E15, E14, E13; E14, E13→cross-level feature adaptive fusion operation→1*1Convolution→get feature E12; E12, E15→cross-level feature adaptive fusion operation→1*1Convolution→get feature E11; The specific operation of EU2 is: →1*1Convolution without changing the size→3*3Convolution→3*3Convolution→get features E25, E24, E23; E24, E23→cross-level feature adaptive fusion operation→1*1Convolution→get feature E22; E22, E25→cross-level feature adaptive fusion operation→1*1Convolution→get feature E21; The specific operation of EU3 is: →1*1Convolution without changing the size→3*3Convolution→3*3Convolution→obtain features E35, E34, E33; E34, E33→cross-level feature adaptive fusion operation→1*1Convolution→obtain feature E32; E32, E35→cross-level feature adaptive fusion operation→1*1Convolution→obtain feature E31.

6. The method for constructing an embedded U-shaped remote sensing defogging network according to claim 1, characterized in that: Step S3 includes: constructing a cross-level feature adaptive fusion module CHAF, which combines the size of H×W×S and The hierarchical features M and N of are extracted through pixel attention and channel attention respectively, which specifically includes the following steps: First, use 3*3 convolution and 3*3 deconvolution on M and N respectively to adjust the length, width and number of channels of the feature map to H×W×S. The formula is: M′=Conv2D(M) N′=TConv(N) Where M′ and N′ are both H×W×S feature maps, TConv represents a standard transposed convolution with a kernel size of 3*3, and Conv2D represents a convolution with a kernel size of 3 without changing the size of the feature map; Compress N′ according to the channel dimension. Each channel represents a kind of semantic information. The formula of N′ after global average pooling is: where F n (i, j) represents the value of the nth single channel feature map at position (i, j), H gap Represents the global average pooling function, the shape of the feature map F is squeezed from H×W×C to 1×1×C; The channel vector f undergoes two 1x1 convolutions and activation functions. The whole process is expressed as: D = σ(Conv1(δ(Conv2(f)))) Where σ and δ represent Sigmoid and Relu activation functions respectively, and Conv1 and Conv2 represent two 1x1 convolutions; Secondly, M′ is directly activated by two convolution, Sigmoid and Relu functions, and the formula is: S=σ(Conv4(δ(Conv3(M ′ )))); Then, multiply S by the adjacent level feature N′ to get M ″ ′; The extracted semantic information is embedded into the shallow features, and N′ is multiplied by the adjacent level features D to obtain N ″ ′, the process is expressed by the formula: Finally, the cross-level features are fused through the adaptive weight allocator, and the adaptive weight allocator formula is expressed as: f=Awa(f i ,f j )=A(θ)×f i +A(1-θ)×f j Where Awa represents the adaptive weight allocator, f i ,f j represents two different level features, and θ represents the learnable factor in the adaptive weight allocator.

7. The method for constructing an embedded U-shaped remote sensing defogging network according to claim 1, characterized in that: The cross-level features in step S3 are adaptively fused in the CHAF module structure including: Input level feature M→3*3Convolution→get feature M'→1*1Convolution→activation function Relu→1*1Convolution→activation function sigmoid→feature A with 1 channel number; Input level feature N→3*3 deconvolution→get feature N'→global average pooling→1*1Convolution→activation function Relu→1*1Convolution→activation function sigmoid→1×1×C feature B; Feature A, feature N'→pixel-by-pixel multiplication→feature M"; Feature B, feature M'→pixel-by-pixel multiplication→feature N"; Adaptive weight allocator assigns weights θ,M"→multiply pixel by pixel; The adaptive weight allocator assigns weights 1-θ, N"→multiply pixel by pixel; The two features after weight multiplication → fusion addition → feature J after cross-level fusion.

8. According to the method for constructing an embedded U-shaped remote sensing defogging network as described in claim 1, it is characterized by: In step S4, the embedded U-shaped remote sensing defogging network, the embedded U-shaped network framework and the cross-level feature adaptive fusion module adopt the following steps when used: S41, inputting the foggy image into the U-type network image defogging network; S42, inputting the coding layers of different levels into the embedded U-shaped network structure of the corresponding layer to obtain information features; S43, inputting the features of different scales extracted by the embedded U-net into the cross-level feature adaptive fusion module, and further fusing the fused features with the corresponding coding layer features to obtain the final output clear and fog-free image; S44. Perform loss constraints on the network training process.

9. The method for constructing an embedded U-shaped remote sensing defogging network according to claim 1, characterized in that: The loss constraints in step S44 include: LI loss constraint, adversarial loss constraint, perceptual loss constraint and multi-scale structural similarity loss constraint; The specific formula of L1 loss is: Where G represents the generation of NHU-Net, x i and i They represent the values ​​of blurred image and clear image at pixel i respectively, and N represents the number of pixels in the image; The adversarial loss constraint uses the adversarial loss of GAN. The specific adversarial loss is defined as follows: Where G represents the generator, which converts the haze-free image into a clear image, and D is the discriminator, which distinguishes the generated clear image from the real haze-free image; The perceptual loss constraint retains more texture information for the restored image and calculates the L2 norm of the clear image and the haze-free image at different layers of the VGG16 pre-trained model. The perceptual loss is defined as: in, Indicates the size of the feature map of layer i of the VGG16 pre-trained model, N indicates the number of feature layers of the VGG16 pre-trained model used in the perceptual loss, where x and y represent the foggy image and the clear image, respectively; Multi-scale structural similarity loss constraint, the specific formula is: where μ i , μ j Represent the mean of the dehazed image and the clear image, σ i , σ j Represent the standard deviation of the dehazed image and the clear image, σ ij Used to represent the covariance of the dehazed image and the clear image, β m , γ m is the equation importance coefficient, C1 and C2 are constant terms.

10. The method for constructing an embedded U-shaped remote sensing defogging network according to claim 1, characterized in that: The total loss function of the overall network is expressed as: L=θ1L smoothL1 +θ2L adv +θ3L pere +θ4L ms-ssim where θ1, θ2, θ3, and θ4 are hyperparameters that balance between different losses.

Citation Information

Patent Citations

  • Construction method of double-branch remote sensing image defogging network

    CN115578280A

  • Construction method of multi-level feature interactive defogging network based on U-Net

    CN115578638A

  • Construction method of partial twinning defogging network based on U-Net

    CN117274098A

  • Defogging method based on cross-layer attention feature interaction and multi-scale channel attention

    CN117541505A

Cited By

  • Image defogging system and method for low-altitude scene

    CN120598820A