Remote sensing image defogging method based on non-uniform fog density distribution prior

Abstract

The application discloses a remote sensing image defogging method based on non-uniform fog density distribution prior. The method comprises the following contents: 1) extraction of a fog distribution prior map based on an improved HDM model; 2) simulation of a remote sensing non-uniform fog image based on the fog distribution prior; and 3) construction and verification of a defogging network based on the fog density distribution prior. The method provides a remote sensing image defogging method based on non-uniform fog density distribution prior, and a learning network based on non-uniform fog density distribution prior can accurately express the real fog features of an image, thereby effectively solving the problems of non-uniform remote sensing image defogging, color and edge degradation. The processing process of the method does not have approximation, the method has a good removal effect on the non-uniform fog features existing in the remote sensing image, can be used in the fields of remote sensing data quality inspection and remote sensing data enhancement, and can effectively improve the quality of remote sensing data.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing data quality inspection and data enhancement, and particularly relates to a remote sensing image defogging method with a non-uniform fog density distribution prior. BACKGROUND

[0002] For non-uniform haze, most scholars choose to first estimate a haze distribution prior map, and then use this prior to guide the model for precise dehazing. Makarau et al. calculated a haze density map (HTM) by local search of dark objects to eliminate non-uniform haze (Reference: A. Makarau, R. Richter, R. Muller, and P. Reinartz, “Haze detection and removal in remotely sensed multispectral imagery,” IEEE Trans. Geosci. Remote Sens., vol. 52, no. 9, pp. 5895–5905, Jan. 2014) based on the work of Chavez. Liu et al. calculated a ground radiance-suppressed haze density map (GRS-HTM) to eliminate the haze component (Reference: Q. Liu, X. Gao, L. He, and W. Lu, “Haze removal for a single visible remote sensing image,” Signal Process., vol. 137, pp. 33–43, 2017), which is more accurate than HTM. Pan et al. used the saturation component to correct the dark channel to obtain a haze distribution map (HDM), which was used for haze concentration estimation (Reference: Pan, X., Xie, F., Jiang, Z., Shi, Z., & Luo, X. (2016). No-Reference Assessment on Haze for Remote-Sensing Images. IEEE Geoscience and Remote Sensing Letters, 13(12), 1855–1859). However, the above methods are heavily dependent on the accuracy of the haze distribution prior. Gu et al. proposed a dense attentive dehazing network (DADN) based on prior by inputting HDM and haze image (Reference: Gu, Z., Zhan, Z., Yuan, Q., & Yan, L. (2019). Single Remote Sensing Image Dehazing Using a Prior-Based Dense Attentive Network. Remote Sensing, 11(24), 3008). The network uses a stacking method to fuse the distribution prior of the haze, which lacks physical interpretability and is prone to problems of haze and texture feature aliasing.In addition, most of the existing methods use dark channel prior to synthesize non-uniform fog, and do not consider the physical characteristics of remote sensing imaging, such as the fact that the imaging radiation characteristics of remote sensing fog are different in different wavebands, and the temporal and spatial characteristics of fog are random. Therefore, the existing methods have poor effect on removing real remote sensing fog, and the generalization ability of the network model is weak.

[0003] With the deepening of the network, the deep learning method will have the phenomenon of feature degradation, and often needs to enhance the features in the propagation process. Therefore, the attention mechanism should be regarded as an effect module for dehazing. Liu et al. proposed an attention-based multi-scale network (Grid Dehaze Net) based on attention mechanism (reference: X Liu, Ma Y, Shi Z, et al. Grid DehazeNet: Attention-Based Multi-Scale Network for Image Dehazing [J]. 2019), which realizes the fusion of different scale features of the image by calculating the weight on one channel. Qin et al. considered that different colors of light have different effects on the formation of fog, and the distribution of fog in the image is uneven, and proposed a feature fusion attention network (FFA-Net) with channel and spatial attention (reference: Qin X, Wang Z, Bai Y, et al. FFA-Net: Feature Fusion Attention Network for Single Image Dehazing [J]. 2019) to solve the problem of fog in different wavebands and regions of the image, respectively. The network learning method based on fog prior features will inevitably increase the redundancy of information and bring additional noise, and the above methods do not consider the multi-feature attention problem after introducing the fog distribution prior. In addition, considering the transmission characteristics of remote sensing images, the fog features are different in different wavebands and different regions, so how to design an attention module suitable for remote sensing image dehazing network and improve the texture and edge preservation ability is the key. SUMMARY

[0004] In view of the above technical problems, the purpose of the present application is to overcome the shortcomings of the prior art, and to provide a remote sensing image defogging network model with non-uniform fog density distribution priori. First, the near-infrared band of the remote sensing image is used to correct the fog density distribution priori physical model, and an end-to-end remote sensing fog training set approximating the real scene is constructed based on the model. A fog density distribution priori fog feature perception fusion module FPFM is designed based on the decoder-encoder backbone network. The module uses an affine transformation method to realize the fusion of image color and fog density distribution priori features, and improves the representation ability of the model for non-uniform fog distribution features. Then, a multi-scale texture feature enhancement module MTEM based on a residual attention mechanism is designed. The module includes residual attention, variable / differential convolution, and enhances the representation ability of the model for different scale geometric and texture features.

[0005] To achieve the above purpose, the present application comprises the following steps:

[0006] S1: Extracting a non-uniform fog distribution priori graph from the input remote sensing image taken under foggy weather;

[0007] S2: Constructing an atmospheric scattering model by taking the non-uniform fog distribution priori graph extracted in step S1 as the transmittance of the simulated fog image and randomly giving an atmospheric light parameter;

[0008] S3: Generating a simulated fog image with non-uniform distribution characteristics based on the atmospheric scattering model obtained in step S2, using the input clear remote sensing image taken under non-foggy weather; and using the generated simulated fog image and the original clear non-foggy image to prepare a training data set for remote sensing image defogging;

[0009] S4: Constructing a remote sensing image defogging network model with non-uniform fog density distribution priori, and learning and training the parameters of the network model using the training data set of step S3 to obtain the parameters of the defogging network model;

[0010] S5: Using the defogging network model and parameters in step S4 to test the input fog image, and obtaining the remote sensing image after defogging.

[0011] Further, the extraction method of the non-uniform fog distribution priori graph in step S1 comprises the following methods and steps:

[0012] (1) Expression of fog distribution priori component: extracting dark channel component graph I dark (x), saturation component graph S(x), and near-infrared band NI(x) from the real remote sensing fog image;

[0013] (2) Extracting the fog distribution priori graph FDM(x) from the real remote sensing fog image using the following model:

[0014] FDM(x)=max(Idark (x)-αS(x)-gNI(x),0)

[0015] Where S(x) represents the saturation component map of the image, I dark (x) is the dark channel map of the image, α is the scaling factor of the saturation component, FDM(x) is the fog density distribution map, NI(x) is the near-infrared band of the image, and ε is the scaling factor of the near-infrared band.

[0016] Furthermore, the transmittance calculation method in step S2 includes the following methods and steps:

[0017] (1) Transmittance of the first band of remote sensing image Expressed as:

[0018]

[0019] Wherein, FDM(x) is the fog density distribution map. This represents the transmittance of the first band.

[0020] (2) Transmittance t of other bands of remote sensing images i Expressed as:

[0021]

[0022] Where λ1 is the wavelength of the first band, λ i Let λ be the wavelength of the i-th band, γ be a fixed quantity related to the size of atmospheric suspended particles, ln be the natural logarithm, and t be the wavelength of the i-th band. i Let be the transmittance of the i-th band.

[0023] Furthermore, the method for constructing the remote sensing image dehazing network model with a priori non-uniform fog density distribution in step S4 includes the following methods and steps:

[0024] (1) Overall structure of the network model: The network is an end-to-end convolutional network structure, including a feature extraction module, a fog feature perception fusion module FPFM, a multi-scale texture feature enhancement module MTEM, and a feature output module;

[0025] (2) End-to-end input of the network: For the fog images in the training set, the fog distribution prior map is extracted using the method in step S1. The prior map and the fog images are used together as the input of one end of the network, and the clear images in the training set are used as the input of the other end of the network.

[0026] (3) Structure of the feature extraction module: This module consists of three upsampled convolutional layers, and the output is a feature map;

[0027] (4) The structure of the fog feature perception fusion module FPFM: the module has two inputs, one is the output of the last feature extraction module, and the other is the input of the network end fog distribution prior graph in step (2), and the output is a feature map; the module is composed of a spatial transformation module SFT, a channel attention layer CA and a spatial attention layer PA in series, wherein the structure of the spatial transformation module SFT is as follows:

[0028] The first module is to perform feature extraction on the input fog distribution prior graph through three convolution layers;

[0029] The second module is an upper and lower parallel structure, both of which are composed of two convolution layers, wherein two outputs are obtained respectively and The two output features are expressed as follows:

[0030]

[0031] Wherein SFT(x) is the final output of the spatial transformation module, I(x) is the output of the last feature extraction module, is a multiplication operation;

[0032] (5) The structure of the multi-scale texture feature enhancement module MTEM: the module is a residual attention structure composed of two parallel branches, one branch is an attention channel, and the other branch is a multi-scale channel that fuses variable convolution and difference convolution; the module has one input, which is the output of the last fog feature perception fusion FPFM module, and the output is a feature map;

[0033] Wherein, the structure of the attention channel is: first, a 1x1 convolution layer is used to extract the basic features of the channel, then a spatial attention and a channel attention branch are established to extract attention features, and a global attention map is obtained by fusing the features in a residual structure;

[0034] Wherein, the structure of the multi-scale channel is: first, a 3x3 variable convolution is used, then a plurality of scale pooling operations are used, and finally a difference convolution is connected;

[0035] (6) Feature output module: the module is composed of three up-sampling convolution layers, and the output is a predicted image after fog removal.

[0036] Further, the loss in the image defogging model constructed in step S4 includes a content loss L c , a perception loss L prep and an edge loss L gra , and the loss function adopts a mean square error with small calculation amount, which can be expressed as:

[0037] L=L c +λ1Lprep + λ2L gra

[0038] Wherein, λ1, λ2 are weight coefficients of perception loss and edge loss respectively;

[0039] The perception loss takes a fixed pre-training VGG16 network, and the feature before the activation of the last convolutional layer in the 4th stage is used for loss calculation, which can be shown by the formula:

[0040] L prep =∑||V1(GI))-V1(I)||2

[0041] Wherein, V1 represents a fixed VGG16 network, G represents the network in the present application, and I represents an input image;

[0042] The gradient information of the image is extracted by the sobel operator, and the edge recovery ability of the network is constrained from the x and y directions, and the edge loss can be represented by the formula:

[0043] L gra =||(E x (GI))-E x I)+(E y (GI))-E y I)||1

[0044] Wherein, E x is the edge extraction in the horizontal direction, and E y is the edge extraction in the vertical direction;

[0045] According to the above, the learning network of the non-uniform fog density distribution prior can accurately express the real fog features of the image, and effectively solve the problems of non-uniform remote sensing image defogging, color and edge degradation. The fog distribution prior is combined with the network in a novel and more explanatory way, the perception ability of the network to the fog is improved, and the attention mechanism and the feature pyramid are combined to enhance the features. In addition, two special convolutions are used to replace the traditional convolution to strengthen the recovery of edges and textures. The method has good removal effect on the non-uniform fog features existing in the remote sensing image, and can be used in remote sensing data quality inspection, remote sensing data enhancement and other fields, and can effectively improve the quality of remote sensing data. BRIEF DESCRIPTION OF DRAWINGS

[0046] The nature of the present application will become apparent and easier to understand from the following description, given by way of example only, and in conjunction with the accompanying drawings, wherein:

[0047] Figure 1 The flowchart of the non-uniform fog density distribution prior remote sensing image defogging method of the present application is shown in the figure;

[0048] Figure 2 The overall network structure diagram is shown in the figure.

[0049] Figure 3 Here is a structural diagram of the fog feature perception fusion module FPFM;

[0050] Figure 4 This is a diagram of the Spatial Feature Transform (SFT) layer structure.

[0051] Figure 5 The structure diagram of the Multi-Scale Texture Feature Enhancement Module (MTEM);

[0052] Figure 6 This is a diagram showing the effect of an image dehazing algorithm. Detailed Implementation

[0053] according to Figure 1 The steps shown below provide a detailed explanation of the remote sensing image defogging method based on a priori non-uniform fog density distribution according to the present invention.

[0054] Step 1: Extract the non-uniform fog distribution prior map from the input remote sensing image taken on a foggy day. The detailed operation is as follows:

[0055] (1) Representation of prior components of fog distribution: Extracting dark channel components from real remote sensing fog images (Figure I) dark (x), saturation component map S(x) and near-infrared band NI(x);

[0056] (2) For real remote sensing fog images, the following model is used to extract the fog distribution prior map FDM(x):

[0057] FDM(x) = max(I dark (x)-αS(x)-gNI(x),0)

[0058] Where S(x) represents the saturation component map of the image, I dark (x) is the dark channel map of the image, α is the scaling factor of the saturation component, FDM(x) is the fog density distribution map, NI(x) is the near-infrared band of the image, and ε is the scaling factor of the near-infrared band. Through extensive experimental verification, two empirical values ​​α = 0.5 and ε = 0.2 can be obtained.

[0059] Step 2: Use the non-uniform fog distribution prior image extracted in step S1 as the transmittance of the fog image to be simulated, and construct an atmospheric scattering model by randomly assigning an atmospheric light parameter. The detailed steps are as follows:

[0060] (1) Transmittance of the first band of remote sensing image Expressed as:

[0061]

[0062] Wherein, FDM(x) is the fog density distribution map. Transmittance of the first band;

[0063] (2) Transmittance of other bands of the remote sensing image i is expressed as:

[0064]

[0065] wherein λ1 is the wavelength of the first band, λ i is the wavelength of the i-th band, γ is a fixed quantity related to the size of atmospheric suspended particles, ln is the natural logarithm operation, t i is the transmittance of the i-th band, and γ ranges from 0.5 to 1, and γ is taken as 1 under thin cloud conditions.

[0066] Step 3: For the input clear remote sensing image taken under a non-foggy day, a simulated fog image with non-uniform distribution characteristics can be generated according to the atmospheric scattering model obtained in step S2, and a training data set for remote sensing image defogging is prepared by using the generated simulated fog image and the original clear non-fog image. The detailed steps are as follows:

[0067] ZY3-02 satellite multispectral images taken around January 2020 are selected as the basis images for preparing the data set, the sensor is TMS, and the image range is around 102.9° east longitude and 24.7° north latitude. First, the images are preprocessed and cropped to 512x512 size, then 15 real fog images are selected and the fog density distribution is extracted, the atmospheric light value is randomly selected from 0.6, 0.8 and 1.0, and finally 10000 simulated fog images with different effects and non-uniform distribution are generated.

[0068] Step 4: A remote sensing image defogging network model with non-uniform fog density distribution prior is constructed, and the training data set of step S3 is used to learn and train the parameters of the network model to obtain the parameters of the defogging network model. The detailed steps are as follows:

[0069] (1) The overall structure of the network model: as shown in Figure 2 , the network is an end-to-end convolutional network structure, including a feature extraction module, a fog feature perception fusion module FPFM, a multi-scale texture feature enhancement module MTEM and a feature output module;

[0070] (2) End-to-end input of the network: for the fog images in the training set, the fog distribution prior map is extracted by the method of step S1, and the prior map and the fog image are taken as the input of one end of the network, and the clear image in the training set is taken as the input of the other end of the network;

[0071] (3) Structure of the feature extraction module: the module is composed of three up-sampling convolutional layers, and the output is a feature map;

[0072] (4) The structure of the fog feature perception fusion module FPFM: As shown in Figure 3 , the module has two inputs, one is the output of the previous layer feature extraction module, and the other is the input of the network end fog distribution prior map in step (2), and the output is a feature map; the module is composed of a spatial transformation module SFT, a channel attention layer CA and a spatial attention layer PA in series structure, wherein, as shown in Figure 4 , the structure of the spatial transformation module SFT is as follows:

[0073] The first module is to perform feature extraction on the input fog distribution prior map through three convolution layers;

[0074] The second module is a parallel structure of the upper and lower two, both composed of two convolution layers, wherein two outputs and are obtained respectively.

[0075]

[0076] Wherein, SFT(x) is the final output of the spatial transformation module, I(x) is the output of the previous layer feature extraction module, is a multiplication operation;

[0077] (5) The structure of the multi-scale texture feature enhancement module MTEM: As shown in Figure 5 , the module is a residual attention structure composed of two parallel branches, one branch is an attention channel, and the other branch is a multi-scale channel that fuses variable convolution and difference convolution; the module has one input, which is the output of the previous layer fog feature perception fusion FPFM module, and the output is a feature map;

[0078] Wherein, the structure of the attention channel is: first, a 1x1 convolution layer is used to extract the basic features of the channel, then a spatial attention and a channel attention branch are established to extract attention features, and a global attention map is obtained by fusing the features in a residual structure;

[0079] Wherein, the structure of the multi-scale channel is: first, a 3x3 variable convolution is used, then a plurality of scale pooling operations are used, and finally a difference convolution is connected;

[0080] (6) Feature output module: The module is composed of three up-sampling convolution layers, and the output is a predicted map after dehazing.

[0081] (7) Expression of loss:

[0082] The loss in the image dehazing model constructed includes content loss L c , perception loss L prep and edge loss L gra, the loss function adopts the mean square error with small calculation amount, and the above can be expressed by formula:

[0083] L = L c + λ 1 L prep + λ 2 L gra

[0084] Wherein, λ 1 、λ 2 It is the weight coefficient of perception loss and edge loss respectively;

[0085] Perception loss adopts fixed pre-training VGG16 network, and the feature before the last convolution layer activation in the 4th stage is used for loss calculation, which can be shown by formula:

[0086] L prep = ∑||V 1 (G I ))-V 1 (I)|| 2

[0087] Wherein, V 1 It indicates fixed VGG16 network, G indicates the network of the present application, and I indicates input image;

[0088] The gradient information of the image is extracted by the sobel operator, and the edge recovery ability of the network is constrained from x and y two directions, and the edge loss can be expressed by formula:

[0089] L gra =||(E x (G I ))-E x I)+(E y (G I ))-E y I)|| 1

[0090] Wherein, E x It is the edge extraction in the horizontal direction, and E y It is the edge extraction in the vertical direction;

[0091] Step 5: using the dehazing network model and parameters in step S4, the inputted haze image is tested, and the remote sensing image after dehazing is obtained, and the effect is shown in Figure 6 The first column of images are all real remote sensing cloud and haze images, and the second column of images are remote sensing images after dehazing corresponding to the real remote sensing cloud and haze images.

[0092] The application provides a remote sensing image dehazing method based on non-uniform haze density distribution prior, for the current image dehazing algorithm, whether it is based on prior knowledge algorithm or based on deep learning algorithm, only dehazing operation is carried out in the single feature space of image, there are problems such as non-uniform dehazing, color and edge degradation, therefore, the application provides a new remote sensing image dehazing network model based on non-uniform haze density distribution prior.

[0093] The above merely illustrates the specific embodiments of the present application, and the protection scope of the present application is not limited thereto, any person skilled in the art can understand and conceive the transformation or replacement within the technical range disclosed by the present application, which should be covered in the protection scope of the present application, therefore the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A remote sensing image defogging method of a non-uniform fog density distribution prior, characterized in that, It comprises the following steps: S1: extracting a non-uniform fog distribution prior map from an input remote sensing image taken under a foggy day; S2: constructing an atmospheric scattering model by taking the non-uniform fog distribution prior map extracted in step S1 as the transmittance of the simulated fog image and randomly giving an atmospheric light parameter; S3: generating a simulated fog image with non-uniform distribution characteristics from a clear remote sensing image taken under a non-foggy day according to the atmospheric scattering model obtained in step S2, and using the generated simulated fog image and the original clear non-fog image to prepare a training data set for remote sensing image defogging; S4: constructing a remote sensing image defogging network model with a non-uniform fog density distribution prior, learning and training the network model with the training data set in step S3 to obtain the parameters of the defogging network model; S5: testing the input fog image using the defogging network model and parameters in step S4 to obtain the remote sensing image after defogging.

2. The remote sensing image defogging method of claim 1, wherein, The extraction method of the non-uniform fog distribution prior map in step S1 comprises the following methods and steps: (1) Expression of the prior component of fog distribution: Extract the dark channel component map I from the real remote sensing fog image dark (x), a saturation component map S(x), and a near-infrared band NI(x); (2) The following model is used to extract the fog distribution prior map FDM(x) from the real remote sensing fog image: FDM(x) = max(I dark (x) - aS(x) - eNI(x), 0) where S(x) represents the saturation component map of the image, I dark (x) the dark channel of the image, a is the scaling factor of the saturation component, FDM(x) is the fog density distribution map, NI(x) is the near-infrared band of the image, and ε is the scaling factor of the near-infrared band.

3. The remote sensing image defogging method of claim 1, wherein, The transmittance calculation method in step S2 comprises the following methods and steps: (1) Transmittance of the first band of the remote sensing image is expressed as: FDM(x) is a fog density distribution map, T1 is the transmittance of the first waveband; (2) Transmittance t of remote sensing image other bands i Expressed as: where λ1is the wavelength of the first band, λ i is the wavelength of the i-th band, γ is a fixed quantity related to the size of atmospheric suspended particles, ln is the natural logarithm operation, t i is the transmittance of the i-th band.

4. The remote sensing image defogging method of claim 1, wherein, The construction method of the remote sensing image defogging network model with a non-uniform fog density distribution prior in step S4 comprises the following methods and steps: (1) The overall structure of the network model: the network is an end-to-end convolutional network structure, including a feature extraction module, a cloud and fog feature perception fusion module FPFM, a multi-scale texture feature enhancement module MTEM, and a feature output module; (2) End-to-end input of the network: the fog distribution prior map is extracted from the fog image in the training set using the method in step S1, and the prior map and the fog image are used as the input of one end of the network, and the clear image in the training set is used as the input of the other end of the network; (3) The structure of the feature extraction module: the module is composed of three up-sampling convolutional layers, and the output is a feature map; (4) The structure of the fog feature perception fusion module FPFM: the module has two inputs, one is the output of the feature extraction module of the last layer, and the other is the input of the fog distribution prior map of one end of the network in step (2), and the output is a feature map; the module is composed of a spatial transformation module SFT, a channel attention layer CA, and a spatial attention layer PA in series, wherein the structure of the spatial transformation module SFT is as follows: The first module is to extract features of the input fog distribution prior map through three convolutional layers; The second module is an upper and lower parallel structure, both of which are composed of two convolutional layers, wherein two outputs are obtained respectively and The two output features are represented as follows: wherein SFT(x) is the final output of the spatial transformation module, I(x) is the output of the previous layer feature extraction module, is a product operation; (5) The structure of the multi-scale texture feature enhancement module MTEM: the module is a residual attention structure composed of two parallel branches, one branch is an attention channel, and the other branch is a multi-scale channel that fuses variable convolution and difference convolution; the module has one input, which is the output of the last layer of the fog feature perception fusion FPFM module, and the output is a feature map; Among them, the structure of the attention channel is: first, a 1x1 convolutional layer is used to extract the basic features of the channel, then two branches of spatial attention and channel attention are established to extract attention features, and a global attention map is obtained by fusing the features in a residual structure. Wherein, the structure of the multi-scale channel is: first, a 3*3 variable convolution is adopted, then a plurality of scale pooling operations are adopted, and finally a differential convolution is connected; (6) Feature output module: This module is composed of three up-sampling convolution layers, and the output is the predicted image after dehazing.

Images