SAR and optical remote sensing image fusion cloud removal method based on edge feature enhancement

Abstract

The invention provides an SAR and optical remote sensing image fusion cloud removal method based on edge feature enhancement, and belongs to the technical field of cross-modal remote sensing image processing, and the method comprises the following steps: S1, inputting a cloud coverage optical remote sensing image into an embedded layer, and obtaining optical shallow layer features; s2, inputting the SAR image into an edge feature extraction module to obtain SAR edge features; s3, enabling the optical shallow layer features to pass through a fusion encoder to obtain fusion encoder features of different scales; s4, obtaining a strong characterization capability feature through a bottleneck layer by the fusion encoder feature of the last layer; s5, the strong characterization capability features and output features of the fusion encoder feature extraction module pass through a fusion decoder to obtain fusion decoder features of different scales; s6, the fused decoder features of the last layer pass through the mapping layer to obtain mapping features, the features and the input cloud coverage optical remote sensing image are added to obtain a fused cloud-removed image, the pixel restoration precision of the cloud-removed area can be effectively improved, and the high-quality optical remote sensing image is obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal remote sensing image processing, and in particular to a SAR and optical remote sensing image fusion and cloud removal method based on edge feature enhancement. Background Art

[0002] Satellite remote sensing technology continues to develop and plays an important role in earth observation tasks such as environmental protection, precision agriculture, and disaster management. However, due to its passive imaging method, optical remote sensing technology is inevitably interfered and blocked by clouds of different shapes, sizes, and thicknesses, which further leads to the loss of optical remote sensing image information and seriously reduces the amount of available data for downstream image interpretation tasks. Therefore, cloud removal of optical remote sensing images is the basis for subsequent applications and has important research significance.

[0003] Existing deep learning-based SAR and optical remote sensing image fusion and cloud removal technologies usually have the following two problems: (1) They do not consider the imaging principles and essential differences between SAR and optical remote sensing images, and directly use the channel superposition data of the two modalities as the input of the deep neural network, resulting in fusion remaining at the pixel level and limited cloud removal effect; (2) SAR is more sensitive to the surface roughness, water content and dielectric properties of the ground objects, while optical remote sensing images focus on obtaining color, space and spectrum information. Therefore, the actual registration process of the two is very difficult.

[0004] In summary, the current technology for de-clouding has low accuracy and still needs to be improved and enhanced. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the existing technology and to solve the problem that the current technology does not take into account the imaging differences and registration difficulties of cross-modal data. A method for SAR and optical remote sensing image fusion and cloud removal based on edge feature enhancement is proposed.

[0006] The present invention adopts the following technical solutions to solve the above problems:

[0007] A method for removing cloud by fusing SAR and optical remote sensing images based on edge feature enhancement, the method comprising the following steps:

[0008] S1: randomly select k non-overlapping training image pairs of cloud-covered optical remote sensing images O, cloud-free ground-truth optical remote sensing images X, and SAR images S from the dataset;

[0009] S2: Build a cross-modal fusion network structure based on edge feature enhancement, and input the image pairs selected in S1 for training;

[0010] S3: Repeat S1 and S2 for t times;

[0011] S4: For the data testing process, the cloud cover optical remote sensing and SAR test images are input into the cross-modal feature fusion network with trained weight parameters to obtain the final fusion de-clouding result.

[0012] S1, input the cloud cover optical remote sensing image into the embedding layer to obtain the optical shallow layer features;

[0013] S2, inputting the SAR image into the edge feature extraction module to obtain SAR edge features;

[0014] S3, the optical shallow features are fused through the encoder to obtain fused encoder features of different scales;

[0015] S4, the fusion encoder features of the last layer are passed through the bottleneck layer to obtain strong representation capabilities;

[0016] S5, the strong representation capability features and the output features of the fusion encoder feature extraction module are combined with the fusion decoder to obtain fusion decoder features of different scales;

[0017] S6, the fusion decoder features of the last layer are passed through the mapping layer to obtain the mapping features, which are added to the input cloud cover optical remote sensing image to obtain the fused de-clouded image.

[0018] Furthermore, in S202, the edge feature extraction module mainly includes an edge extraction layer with a learnable Sobel operator, a shallow convolution layer and a deep convolution layer. The three-layer network extracts edge features, shallow features and deep features respectively, and uses shallow features and deep features to compensate for edge features to obtain the final edge features.

[0019] Furthermore, in S203, the encoder is obtained by a spectral multi-channel feature extraction module and a downsampling module for SAR edge feature enhancement.

[0020] Furthermore, in S205, the decoder is composed of an upsampling module, a channel superposition mechanism, a shallow convolution layer, and a spectral multi-channel feature extraction module for SAR edge feature enhancement.

[0021] Compared with the prior art, the beneficial effects of this application are:

[0022] 1. Compared with the existing technology, the method of the present invention can fully mine and utilize the edge information of SAR images and the global spectral information of optical remote sensing images, perform cross-modal feature-level fusion, and achieve the effect of improving the fusion declouding accuracy, improving the edge restoration effect of the image, and improving the image quality of the declouding result.

[0023] 2. A spectral multi-channel feature extraction module for SAR edge feature enhancement is designed. The SAR edge information is obtained by using the SAR edge feature extraction module with a learnable Sobel operator. The spectral multi-channel attention mechanism is used to extract the global spectral information of cloud-covered optical remote sensing images. The edge information is effectively fused with the global spectral information at different scales to improve the edge restoration effect of ground objects in de-clouded optical remote sensing images.

[0024] 3. A gradient-guided composite loss function is designed, which is composed of the mean square error L1 and the spatial gradient error L2 of the fused de-clouded remote sensing image and the cloud-free true optical remote sensing image. spa and the spectral gradient error L spe The sum is obtained to ensure the consistency of spatial edge information and the fidelity of reconstructed spectrum of the fused de-clouded remote sensing image. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the specific embodiments of the present invention, the drawings required for use in the description of the specific embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 It is a flow chart of the fusion cloud removal method of the present invention;

[0027] Figure 2 Flowchart for training and testing cross-modal feature fusion networks;

[0028] Figure 3 This is a diagram of the cross-modal feature fusion network structure;

[0029] in, Figure 3 (a) is E-SWMSA, i.e., the spectral multi-channel attention mechanism diagram for SAR edge feature enhancement;

[0030] Figure 3 (b) is EF-STB, i.e., spectral multi-channel feature extraction module for SAR edge feature enhancement;

[0031] Figure 3 (c) is the overall network structure diagram;

[0032] Figure 4 Schematic diagram of the learnable edge detection operator in Example 2. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] A method for removing cloud by fusing SAR and optical remote sensing images based on edge feature enhancement. The basic implementation process is as follows:

[0035] Cross-modal fusion network based on edge feature enhancement: A spectral multi-channel feature extraction module for SAR edge feature enhancement is designed. The SAR edge information is obtained by using the SAR edge feature extraction module with a learnable Sobel operator. The spectral multi-channel attention mechanism is used to extract the global spectral information of cloud-covered optical remote sensing images. The edge information and the global spectral information are effectively fused at different scales to improve the edge restoration effect of de-clouded optical remote sensing images. A gradient-guided composite loss function is designed. The mean square error L1 and spatial gradient error L2 of the fused de-clouded remote sensing image and the cloud-free true optical remote sensing image are used to obtain the edge information of the cloud-covered optical remote sensing image. spa and the spectral gradient error L spe The sum is obtained to ensure the consistency of spatial edge information and the fidelity of reconstructed spectrum of the fused de-clouded remote sensing image.

[0036] For the network training process, the cloud-covered optical remote sensing and SAR training images are first used to pass through the cross-modal feature fusion network to obtain the fused de-clouding result; then the result and the cloud-free optical remote sensing training image are simultaneously input into the gradient-guided composite loss function to calculate the error, and the network weight parameters are optimized while minimizing the error until the maximum number of training cycles is reached and the weight parameters are stopped and saved. For the data testing process, the cloud-covered optical remote sensing and SAR image pairs are input into the cross-modal feature fusion network with trained weight parameters to obtain the final fused de-clouding image.

[0037] In order to better understand the present embodiment, the mathematical terms are defined as follows:

[0038] ① The element representation in data X: X(i,j,k) represents the element with index i,j,k;

[0039] ②L1 norm solution: For X and Subtract the corresponding elements in , take their absolute values ​​and add them together.

[0040] Example 1

[0041] In this embodiment, the input instructions are explained as follows:

[0042] ① Training image pairs: cloud-covered optical remote sensing image O with m rows, n columns, and b band, cloud-free true value optical remote sensing image X, and SAR image S with m rows, n columns, and z band;

[0043] ② The image pair to be tested: the cloud cover optical remote sensing image O with m rows, n columns and b band and the SAR image S with m rows, n columns and z band;

[0044] ③ Input parameters: maximum number of training cycles t, learning rate l r , the number of image pairs input for one iteration training is k. The dataset consists of a training set, a validation set, and a test set. The number of image pairs in the three datasets is tra, val, and tes, respectively. Therefore, the training process, validation process, and test process require iterations of tra / k times, val / k times, and tes / k times, respectively.

[0045] like Figure 1-Figure 3 As shown, the present application provides a SAR and optical remote sensing image fusion de-clouding method based on edge feature enhancement, which specifically includes the following steps:

[0046] S1: randomly select k non-overlapping training image pairs of cloud-covered optical remote sensing images O, cloud-free ground-truth optical remote sensing images X, and SAR images S from the dataset;

[0047] S2: Build a cross-modal fusion network structure based on edge feature enhancement, input the above image pairs for training, and specifically include the following sub-steps:

[0048] S201-S202: Input the cloud cover optical remote sensing image O into the embedding layer to obtain the optical shallow layer feature Y0, and input the SAR image S into the edge feature extraction module to obtain the SAR edge feature F s ; The embedding layer consists of a convolutional layer and a LeakReLu activation layer.

[0049] The edge feature extraction module mainly includes an edge extraction layer with a learnable Sobel operator, a shallow convolution layer, and a deep convolution layer. The three-layer network extracts edge features, shallow features, and deep features respectively, and uses shallow features and deep features to compensate for edge features to obtain the final edge feature F. s .

[0050] The optical shallow features are fused through an encoder to obtain fused encoder features Y1 and Y2 of different scales. The encoder is obtained by a spectral multi-channel feature extraction module and a downsampling module for SAR edge feature enhancement. In the spectral multi-channel feature extraction module for SAR edge feature enhancement, the optical remote sensing image is input into the global spectral attention mechanism in the form of spectral multi-channels to obtain attention features V, K and Q. s Fuse cross-modal information to achieve edge feature enhancement.

[0051] S203: The fusion encoder feature Y2 of the last layer is passed through the bottleneck layer to obtain the strong representation capability feature Y'2; the bottleneck layer is EFF-STN: SAR edge feature enhanced spectral multi-channel feature extraction module.

[0052] S204: The strong representation capability feature Y'2 and the output feature of the fusion encoder feature extraction module are used to obtain fusion decoder features Y'1 and Y'0 of different scales through a fusion decoder. The decoder consists of an upsampling module, a channel superposition mechanism, a shallow convolution layer, and a spectral multi-channel feature extraction module with SAR edge feature enhancement.

[0053] S205: The fusion decoder feature Y'0 of the last layer is passed through the mapping layer to obtain the mapping feature R, which is added to the input cloud cover optical remote sensing image O to obtain the fusion declouding result.

[0054] S206: Fusion cloud removal results The gradient-guided composite loss function is input simultaneously with the cloud-free optical remote sensing training image X to calculate the error. When the error is minimized, the network weight parameters are optimized until the training converges, and the weight parameters are stopped and saved.

[0055] The gradient-guided composite loss function is composed of the mean square error L1, the spatial gradient error L spa and the spectral gradient error L spe The sum is obtained to ensure the consistency of spatial edge information and the fidelity of reconstructed spectrum of the fused de-clouded remote sensing image.

[0056] L=L1+L spa +L spe (1)

[0057] Among them, the mean square error It can constrain the pixel similarity between the cloud-free remote sensing image and the fused de-clouded remote sensing image, thereby improving image quality, minimizing artifacts, and maintaining fine details. The simplicity and effectiveness of this loss function make it an important tool for improving the effect of removing cloud occlusions; in addition, the mean square error L1 loss complements other losses, helping to stabilize training and produce visually consistent results.

[0058] Spatial gradient error L spa =λ1L spah +λ2L spav ; is the spatial gradient error in the horizontal direction, and They are the horizontal gradient images of the cloud-free remote sensing image and the fused de-clouded remote sensing image respectively; is the spatial gradient error in the vertical direction. The vertical gradient maps of the cloud-free remote sensing image and the fused de-clouded remote sensing image are and λ1=λ2 is the balance coefficient.

[0059] Spectral gradient error is the gradient error in the spectral channel direction, where E spe (i,j,k)=X(i,j,k+1)-X(i,j,k) and They are the spectral gradient maps of the cloud-free remote sensing image and the fused de-clouded remote sensing image, respectively.

[0060] S3: Repeat steps 1 and 2tra / k times as one cycle, for a total of t cycles.

[0061] S4: For the data testing process, the cloud cover optical remote sensing and SAR test images are input into the cross-modal feature fusion network with trained weight parameters to obtain the final fusion de-clouding result.

[0062] Since then, the fusion of SAR and optical remote sensing images for cloud removal has been completed / realized, which has improved the edge restoration effect of the image and the image quality of the cloud removal results.

[0063] Example 2

[0064] In this embodiment, the input instructions are explained as follows:

[0065] ① Training image pairs: cloud-covered optical remote sensing image O with m = 128 rows, n = 128 columns, and b = 13 bands, cloud-free true value optical remote sensing image X, and SAR image S with m = 128 rows, n = 128 columns, and z = 2 bands;

[0066] ② The image pair to be tested: cloud cover optical remote sensing image O with m = 128 rows, n = 128 columns, b = 13 bands and SAR image S with m = 128 rows, n = 128 columns, z = 2 bands;

[0067] ③ Input parameters: maximum number of training cycles t = 300, learning rate l r =0.0001, and the number of image pairs input for one iteration training is k=4. The data set consists of a training set, a validation set, and a test set. The number of image pairs in the three data sets is 6800, 800, and 400, respectively. Therefore, the training process, validation process, and test process require 6800 / 4=1700 iterations, 800 / 4=200 iterations, and 400 / 4=100 iterations, respectively.

[0068] This embodiment specifically includes the following steps:

[0069] S1: Four non-overlapping training image pairs of cloud-covered optical remote sensing images O, cloud-free ground-truth optical remote sensing images X, and SAR images S are randomly selected from the training set;

[0070] S2: Build a cross-modal fusion network structure based on edge feature enhancement, input the above image pairs for training, and specifically include the following sub-steps:

[0071] S201-S202: Figure 4 As shown in the figure, the cloud cover optical remote sensing image O is input into the embedding layer to obtain the optical shallow feature Y0, which has a size of 128×128×32 and a feature dimension of 32; the embedding layer consists of a 3×3 convolutional layer and a LeakReLu activation layer; the SAR image S is input into the edge feature extraction module to obtain the SAR edge feature F s , the size is 128×128×32, and the feature dimension is 32.

[0072] The edge feature extraction module mainly includes an edge extraction layer with a learnable edge detection operator, a shallow convolution layer and a deep convolution layer. The three-layer network extracts edge features, shallow features and deep features respectively, and uses shallow features and deep features to compensate for edge features to obtain the final edge feature F. s ; The edge extraction layer of the learnable Sobel operator includes a 3×3 learnable convolution kernel and a 1×1 convolution layer (as shown below, α is a learnable weight parameter); the shallow convolution layer includes a 1×1 convolution layer; the deep convolution layer includes a 1×1 convolution layer, a 5×5 convolution layer and a sigmoid activation layer.

[0073] The optical shallow features are fused through the encoder to obtain fusion encoder features Y1 and Y2 of different scales, with sizes of 64×64×64 and 32×32×128 respectively. The encoder is obtained by the spectral multi-channel feature extraction module and the downsampling module of the SAR edge feature enhancement; in the spectral multi-channel feature extraction module of the SAR edge feature enhancement, the optical remote sensing image is input into the global spectral attention mechanism in the form of spectral multi-channel to obtain the attention features V, K and Q, and V'=V⊙F s Fuse cross-modal information to achieve edge feature enhancement.

[0074] S203: The fusion encoder feature Y2 of the last layer has a size of 32×32×128, and the strong representation capability feature Y'2 with a size of 32×32×128 is obtained through the bottleneck layer; the bottleneck layer is EFF-STB: SAR edge feature enhanced spectral multi-channel feature extraction module.

[0075] S204: The strong representation capability feature Y'2 and the output features of the fusion encoder feature extraction module are fused through a decoder to obtain fusion decoder features Y'1 and Y'0 of different scales, with sizes of 64×64×64 and 128×128×32 respectively; the decoder consists of an upsampling module, a channel superposition mechanism, a shallow convolution layer, and a spectral multi-channel feature extraction module with SAR edge feature enhancement.

[0076] S205: The fusion decoder feature Y'0 of the last layer is passed through the mapping layer to obtain the mapping feature R, which is added to the input cloud cover optical remote sensing image O to obtain the fusion declouding result.

[0077] S206: Fusion cloud removal results The gradient-guided composite loss function is input simultaneously with the cloud-free optical remote sensing training image X, the error is calculated, and the network weight parameters are optimized while minimizing the error until the training converges, then the weight parameters are stopped and saved; the gradient-guided composite loss function is composed of the mean square error L1, the spatial gradient error L spa and the spectral gradient error L spe The sum of is obtained, thereby ensuring the consistency of spatial edge information and the fidelity of reconstructed spectrum of the fused de-clouded remote sensing image.

[0078] L=L1+L spa +L spe (2)

[0079] Among them, the mean square error It can constrain the pixel similarity between the cloud-free remote sensing image and the fused de-clouded remote sensing image, thereby improving image quality, minimizing artifacts, and maintaining fine details. The simplicity and effectiveness of this loss function make it an important tool for improving the effect of removing cloud occlusions; in addition, the mean square error L1 loss complements other losses, helping to stabilize training and produce visually consistent results.

[0080] Spatial gradient error L spa =λ1L spah +λ2L spav ,in is the spatial gradient error in the horizontal direction. The horizontal gradient maps of the cloud-free remote sensing image and the fused de-clouded remote sensing image are and is the spatial gradient error in the vertical direction, where is the spatial gradient error in the vertical direction. The vertical gradient maps of the cloud-free remote sensing image and the fused de-clouded remote sensing image are and λ1=λ2 is the balance coefficient, usually 0.5.

[0081] Spectral gradient error is the gradient error in the spectral channel direction. spe (i,j,k)=X(i,j,k+1)-X(i,j,k) and They are the spectral gradient maps of the cloud-free remote sensing image and the fused de-clouded remote sensing image, respectively.

[0082] S3: Repeat S1 and S2 1700 times as one cycle, for a total of 300 cycles.

[0083] S4: For the data testing process, the cloud cover optical remote sensing and SAR test images are input into the cross-modal feature fusion network with trained weight parameters to obtain the final fusion de-clouding result.

[0084] As shown in Table 1, the experimental results of different cloud removal methods on the SEN12MS-CR dataset are quantitatively compared based on five evaluation indicators: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), comprehensive global dimensionless error (ERGAS), spectral angle mapping (SAM) and correlation coefficient (CC). The best results are marked in bold and the suboptimal results are underlined.

[0085] Compared with existing deep learning-based cloud removal methods, the method of the present invention performs outstandingly in all indicators, achieving higher cloud removal accuracy while using the least number of parameters. This result highlights the practical value of the method of the present invention.

[0086] Table 1 Quantitative comparison of experimental results of different cloud removal methods on the SEN12MS-CR dataset

[0087]

[0088] The present invention is described in detail above through the embodiments, but the contents are only preferred embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made according to the scope of application of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A cloud removal method based on edge feature enhancement for SAR and optical remote sensing image fusion, characterized in that: The method comprises the following steps: S1: randomly select k non-overlapping training image pairs of cloud-covered optical remote sensing images O, cloud-free ground-truth optical remote sensing images X, and SAR images S from the dataset; S2: Build a cross-modal fusion network structure based on edge feature enhancement, input the image pairs selected in S1 for training, use the cloud cover optical remote sensing and SAR training image pairs to pass through the cross-modal feature fusion network to obtain the fused de-clouding result, and input the result and the cloud-free optical remote sensing training image into the gradient-guided composite loss function at the same time to calculate the error. When the error is minimized, optimize the network weight parameters until the maximum number of training cycles is reached and the weight parameters are stopped and saved; S3: Repeat S1 and S2 for t times; S4: For the data testing process, the cloud cover optical remote sensing and SAR test images are input into the cross-modal feature fusion network with trained weight parameters to obtain the final fusion de-clouding result.

2. The cloud removal method based on edge feature enhancement of SAR and optical remote sensing images according to claim 1 is characterized in that: S2 specifically includes the following sub-steps: S201, inputting the cloud cover optical remote sensing image into the embedding layer to obtain optical shallow layer features; S202, inputting the SAR image into an edge feature extraction module to obtain SAR edge features; S203, the optical shallow features are fused through an encoder to obtain fused encoder features of different scales; S204, the fusion encoder features of the last layer are passed through the bottleneck layer to obtain strong representation capability features; S205, the strong representation capability feature and the output feature of the fusion encoder feature extraction module are combined into a fusion decoder to obtain fusion decoder features of different scales; S6, the fusion decoder features of the last layer are passed through the mapping layer to obtain the mapping features, which are added to the input cloud cover optical remote sensing image to obtain the fused de-clouded image.

3. The cloud removal method based on edge feature enhancement of SAR and optical remote sensing images according to claim 2 is characterized in that: In S202, the edge feature extraction module mainly includes an edge extraction layer with a learnable Sobel operator, a shallow convolution layer and a deep convolution layer. The three-layer network extracts edge features, shallow features and deep features respectively, and uses the shallow features and deep features to compensate for the edge features to obtain the final edge features.

4. The cloud removal method based on edge feature enhancement of SAR and optical remote sensing images according to claim 2 is characterized in that: In S203, the encoder is obtained by a spectral multi-channel feature extraction module and a downsampling module for SAR edge feature enhancement.

5. The cloud removal method based on edge feature enhancement of SAR and optical remote sensing images by fusion according to claim 2, characterized in that: In S205, the decoder is composed of an upsampling module, a channel superposition mechanism, a shallow convolution layer, and a spectral multi-channel feature extraction module for SAR edge feature enhancement.

Images