A cloud removal method for SAR and optical remote sensing image fusion based on edge feature enhancement

Abstract

The application provides a cloud-removed SAR and optical remote sensing image fusion method based on edge feature enhancement, and belongs to the technical field of cross-modal remote sensing image processing, and comprises the following steps: S1: inputting a cloud-covered optical remote sensing image into an embedding layer to obtain optical shallow layer features; S2: inputting a SAR image into an edge feature extraction module to obtain SAR edge features; S3: obtaining different scale fusion encoder features by taking the optical shallow layer features through a fusion encoder; S4: obtaining strong representation capability features by taking the fusion encoder features of the last layer through a bottleneck layer; S5: obtaining different scale fusion decoder features by taking the strong representation capability features and the output features of the fusion encoder feature extraction module through a fusion decoder; and S6: obtaining mapping features by taking the fusion decoder features of the last layer through a mapping layer, and adding the mapping features and the input cloud-covered optical remote sensing image to obtain a fusion cloud-removed image. The method can effectively improve the pixel restoration precision of the cloud-removed area and obtain a high-quality optical remote sensing image.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal remote sensing image processing, and in particular to a SAR and optical remote sensing image fusion and cloud removal method based on edge feature enhancement. Background Art

[0002] Satellite remote sensing technology continues to advance, playing a vital role in Earth observation missions such as environmental protection, precision agriculture, and disaster management. However, due to its passive imaging approach, optical remote sensing technology is inevitably subject to interference and obstruction from clouds of varying shapes, sizes, and thicknesses. This leads to information loss in optical remote sensing images, significantly reducing the amount of data available for downstream image interpretation tasks. Therefore, cloud removal from optical remote sensing images is fundamental to subsequent applications and holds significant research significance.

[0003] Existing deep learning-based SAR and optical remote sensing image fusion and cloud removal technologies often have the following two problems: (1) They do not consider the imaging principles and essential differences between SAR and optical remote sensing images, and directly use the channel stacking data of the two modalities as the input of the deep neural network, resulting in fusion remaining at the pixel level and limited cloud removal effect; (2) SAR is more sensitive to surface roughness, water content, and dielectric properties of ground objects, while optical remote sensing images focus on obtaining color, spatial, and spectral information. Therefore, the actual registration process of the two is very difficult.

[0004] In summary, the current technology for de-clouding has low accuracy and still needs to be improved and enhanced. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the existing technology and to solve the problem that the current technology does not take into account the imaging differences and registration difficulties of cross-modal data. A method for cloud removal by fusion of SAR and optical remote sensing images based on edge feature enhancement is proposed.

[0006] The present invention adopts the following technical solutions to solve the above problems:

[0007] A cloud removal method for SAR and optical remote sensing image fusion based on edge feature enhancement, the method comprising the following steps:

[0008] S1: Randomly select k non-overlapping training image pairs of cloud-covered optical remote sensing images O, cloud-free ground-truth optical remote sensing images X, and SAR images S from the dataset;

[0009] S2: Build a cross-modal fusion network structure based on edge feature enhancement, input the image pairs selected in S1 for training, use the cloud cover optical remote sensing and SAR training image pairs to pass through the cross-modal feature fusion network to obtain the fused de-clouding result, and input the result and the cloud-free optical remote sensing training image into the gradient-guided composite loss function at the same time to calculate the error. When the error is minimized, optimize the network weight parameters until the maximum number of training cycles is reached and the weight parameters are stopped and saved. Specifically, the steps include:

[0010] S201, inputting the cloud cover optical remote sensing image into the embedding layer to obtain optical shallow features;

[0011] S202: Input the SAR image into an edge feature extraction module to obtain SAR edge features. The edge feature extraction module mainly includes an edge extraction layer with a learnable Sobel operator, a shallow convolution layer, and a deep convolution layer. The three-layer network extracts edge features, shallow features, and deep features respectively, and uses the shallow features and deep features to compensate for the edge features to obtain the final edge features.

[0012] S203, applying the optical shallow features to a fusion encoder to obtain fusion encoder features of different scales, wherein the encoder is obtained by a spectral multi-channel feature extraction module and a downsampling module with SAR edge feature enhancement;

[0013] S204, the fusion encoder features of the last layer are passed through the bottleneck layer to obtain strong representation capabilities;

[0014] S205, the strong representation capability features and the output features of the fusion encoder feature extraction module are combined into a fusion decoder to obtain fusion decoder features of different scales, wherein the decoder is composed of an upsampling module, a channel superposition mechanism, a shallow convolution layer, and a spectral multi-channel feature extraction module for SAR edge feature enhancement;

[0015] S206, the fusion decoder feature of the last layer is mapped through the mapping layer to obtain the mapping feature, which is added to the input cloud cover optical remote sensing image to obtain the fused cloud removal image. The gradient-guided composite loss function is composed of the mean square error L1, the spatial gradient error L spa and spectral gradient error L spe The sum is obtained to ensure the consistency of spatial edge information and the fidelity of reconstructed spectrum of the fused de-clouded remote sensing image;

[0016] S3: Repeat S1 and S2 for t times;

[0017] S4: For the data testing process, the cloud cover optical remote sensing and SAR test images are input into the cross-modal feature fusion network with trained weight parameters to obtain the final fusion declouding results.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] 1. Compared with existing technologies, the method of the present invention can fully mine and utilize the edge information of SAR images and the global spectral information of optical remote sensing images to perform cross-modal feature-level fusion, thereby improving the accuracy of fusion declouding, enhancing the edge restoration effect of the image, and improving the image quality of the declouding results.

[0020] 2. A spectral multi-channel feature extraction module for SAR edge feature enhancement was designed. The SAR edge information was obtained using the SAR edge feature extraction module with a learnable Sobel operator. The global spectral information of cloud-covered optical remote sensing images was extracted using a spectral multi-channel attention mechanism. The edge information was effectively fused with the global spectral information at different scales to improve the edge restoration effect of ground objects in de-clouded optical remote sensing images.

[0021] 3. A gradient-guided composite loss function is designed, which is composed of the mean square error L1 and the spatial gradient error L of the fused de-clouded remote sensing image and the cloud-free true optical remote sensing image. spa and spectral gradient error L spe The sum is obtained, thereby ensuring the consistency of spatial edge information and the fidelity of reconstructed spectrum of the fused de-clouded remote sensing image. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific embodiments of the present invention, the following will briefly introduce the drawings required for use in the description of the specific embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 This is a flow chart of the fusion cloud removal method of the present invention;

[0024] Figure 2 Flowchart for training and testing cross-modal feature fusion networks;

[0025] Figure 3 This is the network structure diagram of cross-modal feature fusion;

[0026] in, Figure 3 (a) E-SWMSA, the spectral multi-channel attention mechanism for SAR edge feature enhancement;

[0027] Figure 3 (b) is EF-STB, which is a spectral multi-channel feature extraction module for SAR edge feature enhancement;

[0028] Figure 3 (c) is the overall network structure diagram;

[0029] Figure 4 Schematic diagram of the learnable edge detection operator in Example 2. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] A cloud removal method based on edge feature enhancement for SAR and optical remote sensing image fusion is proposed. The basic implementation process is as follows:

[0032] Cross-modal fusion network based on edge feature enhancement: a spectral multi-channel feature extraction module for SAR edge feature enhancement is designed. The SAR edge information is obtained by using the SAR edge feature extraction module with a learnable Sobel operator. The global spectral information of cloud-covered optical remote sensing images is extracted using a spectral multi-channel attention mechanism. The edge information and the global spectral information are effectively fused at different scales to improve the edge restoration effect of the de-clouded optical remote sensing image. A gradient-guided composite loss function is designed. The mean square error L1 and spatial gradient error L2 of the fused de-clouded remote sensing image and the cloud-free true value optical remote sensing image are used to obtain the edge information of the SAR edge feature extraction module. spa and spectral gradient error L spe The sum is obtained, thereby ensuring the consistency of spatial edge information and the fidelity of reconstructed spectrum of the fused de-clouded remote sensing image.

[0033] During network training, cloud-covered optical remote sensing and SAR training image pairs are first fed through a cross-modal feature fusion network to obtain a fused declouding result. This result, along with cloud-free optical remote sensing training images, is then fed into a gradient-guided composite loss function to calculate the error. The network weight parameters are optimized to minimize this error until the maximum number of training cycles is reached, at which point the training stops and the weight parameters are saved. During data testing, cloud-covered optical remote sensing and SAR image pairs are fed into the cross-modal feature fusion network with the trained weight parameters to obtain the final fused declouding image.

[0034] In order to better understand the present embodiment, the mathematical definitions are explained as follows:

[0035] ①Element representation in data X: X(i,j,k) represents the element with index i,j,k;

[0036] ②L1 norm solution: For X and Subtract the corresponding elements, take their absolute values ​​and add them together.

[0037] Example 1

[0038] In this embodiment, the input instructions are explained as follows:

[0039] ① Training image pairs: cloud cover optical remote sensing image O with m rows, n columns, and b band, cloud-free true value optical remote sensing image X, and SAR image S with m rows, n columns, and z band;

[0040] ② Image pair to be tested: cloud cover optical remote sensing image O with m rows, n columns, and b band, and SAR image S with m rows, n columns, and z band;

[0041] ③ Input parameters: maximum number of training cycles t, learning rate l r , the number of image pairs input for each training iteration is k. The dataset consists of a training set, a validation set, and a test set. The number of image pairs in the three datasets is tra, val, and tes, respectively. Therefore, the training process, validation process, and testing process require tra / k iterations, val / k iterations, and tes / k iterations, respectively.

[0042] like Figure 1-Figure 3 As shown, the present application provides a method for cloud removal by fusion of SAR and optical remote sensing images based on edge feature enhancement, which specifically includes the following steps:

[0043] S1: Randomly select k non-overlapping training image pairs of cloud-covered optical remote sensing images O, cloud-free ground-truth optical remote sensing images X, and SAR images S from the dataset;

[0044] S2: Build a cross-modal fusion network structure based on edge feature enhancement, input the above image pairs for training, and specifically include the following sub-steps:

[0045] S201-S202: Input the cloud cover optical remote sensing image O into the embedding layer to obtain the optical shallow layer feature Y0, and input the SAR image S into the edge feature extraction module to obtain the SAR edge feature F s ; The embedding layer consists of a convolutional layer and a LeakReLu activation layer.

[0046] The edge feature extraction module mainly includes an edge extraction layer with a learnable Sobel operator, a shallow convolution layer, and a deep convolution layer. The three-layer network extracts edge features, shallow features, and deep features respectively, and uses shallow features and deep features to compensate for edge features to obtain the final edge feature F s .

[0047] The optical shallow features are fused through an encoder to obtain fused encoder features Y1 and Y2 of different scales. The encoder is obtained by the spectral multi-channel feature extraction module and the downsampling module of the SAR edge feature enhancement. In the spectral multi-channel feature extraction module of the SAR edge feature enhancement, the optical remote sensing image is input into the global spectral attention mechanism in the form of spectral multi-channel to obtain the attention features V, K and Q. s Fuse cross-modal information to achieve edge feature enhancement.

[0048] S203: The fusion encoder feature Y2 of the last layer is passed through the bottleneck layer to obtain the strong representation capability feature Y'2; the bottleneck layer is EFF-STN: SAR edge feature enhanced spectral multi-channel feature extraction module.

[0049] S204: The strong representation capability feature Y'2 and the output features of the fusion encoder feature extraction module are combined into a fusion decoder to obtain fusion decoder features Y'1 and Y'0 of different scales. The decoder consists of an upsampling module, a channel superposition mechanism, a shallow convolution layer, and a spectral multi-channel feature extraction module with SAR edge feature enhancement.

[0050] S205: The fusion decoder feature Y'0 of the last layer is passed through the mapping layer to obtain the mapping feature R, which is added to the input cloud cover optical remote sensing image O to obtain the fusion declouding result.

[0051] S206: Fusion cloud removal results The gradient-guided composite loss function is input simultaneously with the cloud-free optical remote sensing training image X, the error is calculated, and the network weight parameters are optimized while minimizing the error until the training converges. The weight parameters are then stopped and saved.

[0052] The gradient-guided composite loss function is composed of the mean square error L1, the spatial gradient error L spa and spectral gradient error L spe The sum is obtained, thereby ensuring the consistency of spatial edge information and the fidelity of reconstructed spectrum of the fused de-clouded remote sensing image.

[0053] L=L1+L spa +L spe (1)

[0054] Among them, the mean square error It can constrain the pixel similarity between cloud-free remote sensing images and fused de-clouded remote sensing images, thereby improving image quality, minimizing artifacts, and maintaining fine details. The simplicity and effectiveness of this loss function make it an important tool for improving the effect of removing cloud occlusions; in addition, the mean square error L1 loss complements other losses, helping to stabilize training and produce visually consistent results.

[0055] Spatial gradient error L spa =λ1L spah +λ2L spav ; is the spatial gradient error in the horizontal direction, and They are the horizontal gradient maps of the cloud-free remote sensing image and the fused de-clouded remote sensing image respectively; is the spatial gradient error in the vertical direction. The vertical gradient maps of the cloud-free remote sensing image and the fused de-clouded remote sensing image are and

[0056] λ1=λ2 is the balance coefficient.

[0057] Spectral gradient error is the gradient error in the spectral channel direction, where E spe (i,j,k)=X(i,j,k+1)-X(i,j,k) and These are the spectral gradient maps of the cloud-free remote sensing image and the fused de-clouded remote sensing image, respectively.

[0058] S3: Repeat steps 1 and 2 tra / k times as one cycle, for a total of t rounds.

[0059] S4: For the data testing process, the cloud cover optical remote sensing and SAR test images are input into the cross-modal feature fusion network with trained weight parameters to obtain the final fusion declouding results.

[0060] Since then, the fusion of SAR and optical remote sensing images for cloud removal has been completed / realized, which has improved the edge restoration effect of the image and the image quality of the cloud removal results.

[0061] Example 2

[0062] In this embodiment, the input instructions are explained as follows:

[0063] ① Training image pairs: cloud cover optical remote sensing image O with m = 128 rows, n = 128 columns, b = 13 bands, cloud-free ground-truth optical remote sensing image X and SAR image S with m = 128 rows, n = 128 columns, z = 2 bands;

[0064] ② Image pair to be tested: cloud cover optical remote sensing image O with m = 128 rows, n = 128 columns, b = 13 bands and SAR image S with m = 128 rows, n = 128 columns, z = 2 bands;

[0065] ③ Input parameters: maximum number of training cycles t = 300, learning rate l r = 0.0001, and the number of image pairs input for one training iteration is k = 4. The dataset consists of a training set, a validation set, and a test set. The number of image pairs in the three datasets is 6800, 800, and 400, respectively. Therefore, the training process, validation process, and testing process require 6800 / 4 = 1700 iterations, 800 / 4 = 200 iterations, and 400 / 4 = 100 iterations, respectively.

[0066] This embodiment specifically includes the following steps:

[0067] S1: Four non-overlapping training image pairs of cloud-covered optical remote sensing images O, cloud-free ground-truth optical remote sensing images X, and SAR images S are randomly selected from the training set;

[0068] S2: Build a cross-modal fusion network structure based on edge feature enhancement, input the above image pairs for training, and specifically include the following sub-steps:

[0069] S201-S202: Figure 4 As shown in the figure, the cloud cover optical remote sensing image O is input into the embedding layer to obtain the optical shallow feature Y0, which has a size of 128×128×32 and a feature dimension of 32; the embedding layer consists of a 3×3 convolutional layer and a LeakReLu activation layer; the SAR image S is input into the edge feature extraction module to obtain the SAR edge feature F s , the size is 128×128×32, and the feature dimension is 32.

[0070] The edge feature extraction module mainly includes an edge extraction layer with a learnable edge detection operator, a shallow convolution layer, and a deep convolution layer. The three-layer network extracts edge features, shallow features, and deep features respectively, and uses shallow features and deep features to compensate for edge features to obtain the final edge feature F s ; The edge extraction layer of the learnable Sobel operator includes a 3×3 learnable convolution kernel and a 1×1 convolution layer (as shown below, α is a learnable weight parameter); the shallow convolution layer includes a 1×1 convolution layer; the deep convolution layer includes a 1×1 convolution layer, a 5×5 convolution layer and a sigmoid activation layer.

[0071] The optical shallow features are fused through the encoder to obtain fusion encoder features Y1 and Y2 of different scales, with sizes of 64×64×64 and 32×32×128 respectively. The encoder is obtained by the spectral multi-channel feature extraction module and the downsampling module of the SAR edge feature enhancement; in the spectral multi-channel feature extraction module of the SAR edge feature enhancement, the optical remote sensing image is input into the global spectral attention mechanism in the form of spectral multi-channel to obtain the attention features V, K and Q, and V'=V⊙F s Fuse cross-modal information to achieve edge feature enhancement.

[0072] S203: The fused encoder feature Y2 of the last layer has a size of 32×32×128, and the strong representation capability feature Y'2 of size 32×32×128 is obtained through the bottleneck layer; the bottleneck layer is EFF-STB: SAR edge feature enhanced spectral multi-channel feature extraction module.

[0073] S204: The strong representation capability feature Y'2 and the output features of the fusion encoder feature extraction module are fused through a decoder to obtain fusion decoder features Y'1 and Y'0 of different scales, with sizes of 64×64×64 and 128×128×32 respectively; the decoder consists of an upsampling module, a channel superposition mechanism, a shallow convolution layer, and a spectral multi-channel feature extraction module with SAR edge feature enhancement.

[0074] S205: The fusion decoder feature Y'0 of the last layer is passed through the mapping layer to obtain the mapping feature R, which is added to the input cloud cover optical remote sensing image O to obtain the fusion declouding result.

[0075] S206: Fusion cloud removal results The gradient-guided composite loss function is input simultaneously with the cloud-free optical remote sensing training image X, and the error is calculated. When the error is minimized, the network weight parameters are optimized until the training converges, and the weight parameters are stopped and saved. The gradient-guided composite loss function is composed of the mean square error L1, the spatial gradient error L spa and spectral gradient error L spe The sum of is obtained, thereby ensuring the consistency of spatial edge information and the fidelity of reconstructed spectrum of the fused de-clouded remote sensing image.

[0076] L=L1+L spa +L spe (2)

[0077] Among them, the mean square error It can constrain the pixel similarity between cloud-free remote sensing images and fused de-clouded remote sensing images, thereby improving image quality, minimizing artifacts, and maintaining fine details. The simplicity and effectiveness of this loss function make it an important tool for improving the effect of removing cloud occlusions; in addition, the mean square error L1 loss complements other losses, helping to stabilize training and produce visually consistent results.

[0078] Spatial gradient error L spa =λ1L spah +λ2L spav ,in is the spatial gradient error in the horizontal direction. The horizontal gradient maps of the cloud-free remote sensing image and the fused de-clouded remote sensing image are and is the spatial gradient error in the vertical direction, where is the spatial gradient error in the vertical direction. The vertical gradient maps of the cloud-free remote sensing image and the fused de-clouded remote sensing image are and λ1=λ2 is the balance coefficient, usually 0.5.

[0079] Spectral gradient error The spectral gradient error is the gradient error in the direction of the spectral channel. Wherein, E spe (i,j,k)=X(i,j,k+1)-X(i,j,k)and The spectral gradient maps of the cloud-free remote sensing image and the fused cloud-free remote sensing image, respectively.

[0080] S3: Repeat S1 and S2 1700 times for one cycle, and cycle 300 times in total.

[0081] S4: For the data test process, input the cloud-covered optical remote sensing and SAR test images into the cross-modal feature fusion network with the trained weight parameters to obtain the final fused cloud-free result.

[0082] As shown in Table 1, the table shows the quantitative comparison of the experimental results of different cloud removal methods on the SEN12MS-CR dataset according to five evaluation indexes of peak signal-to-noise ratio (PSNR), structural similarity (SSIM), comprehensive global dimensionless error (ERGAS), spectral angle mapping (SAM) and correlation coefficient (CC). The optimal result is marked in bold, and the suboptimal result is marked with an underscore.

[0083] Compared with the existing deep learning-based cloud removal methods, the method of the present application performs outstandingly in all indicators, not only obtaining higher cloud removal accuracy, but also using the least parameter amount. This result highlights the practical value of the method of the present application.

[0084] Table 1 Quantitative comparison of experimental results of different cloud removal methods on SEN12MS-CR dataset

[0085]

[0086] The above has described the present application in detail through the embodiments, but the content is only the preferred embodiments of the present application, and cannot be considered as limiting the implementation scope of the present application. Any equivalent changes and improvements made according to the scope of the present application should still belong to the patent coverage range of the present application.

Claims

1. A cloud removal method for SAR and optical remote sensing image fusion based on edge feature enhancement, characterized by: The method comprises the following steps: S1: Randomly select k non-overlapping training image pairs of cloud-covered optical remote sensing images O, cloud-free ground-truth optical remote sensing images X, and SAR images S from the dataset; S2: Build a cross-modal fusion network structure based on edge feature enhancement, input the image pairs selected in S1 for training, use the cloud cover optical remote sensing and SAR training image pairs to pass through the cross-modal feature fusion network to obtain the fused de-clouding result, and input the result and the cloud-free optical remote sensing training image into the gradient-guided composite loss function at the same time to calculate the error. When the error is minimized, optimize the network weight parameters until the maximum number of training cycles is reached and the weight parameters are stopped and saved. Specifically, the steps include: S201, inputting the cloud cover optical remote sensing image into the embedding layer to obtain optical shallow features; S202: Input the SAR image into an edge feature extraction module to obtain SAR edge features. The edge feature extraction module mainly includes an edge extraction layer with a learnable Sobel operator, a shallow convolution layer, and a deep convolution layer. The three-layer network extracts edge features, shallow features, and deep features respectively, and uses the shallow features and deep features to compensate for the edge features to obtain the final edge features. S203, applying the optical shallow features to a fusion encoder to obtain fusion encoder features of different scales, wherein the encoder is obtained by a spectral multi-channel feature extraction module and a downsampling module with SAR edge feature enhancement; S204, the fusion encoder features of the last layer are passed through the bottleneck layer to obtain strong representation capabilities; S205, the strong representation capability features and the output features of the fusion encoder feature extraction module are combined into a fusion decoder to obtain fusion decoder features of different scales, wherein the decoder is composed of an upsampling module, a channel superposition mechanism, a shallow convolution layer, and a spectral multi-channel feature extraction module for SAR edge feature enhancement; S206, the fusion decoder feature of the last layer is mapped through the mapping layer to obtain the mapping feature, which is added to the input cloud cover optical remote sensing image to obtain the fused cloud removal image. The gradient-guided composite loss function is composed of the mean square error L1, the spatial gradient error L spa and spectral gradient error L spe The sum is obtained to ensure the consistency of spatial edge information and the fidelity of reconstructed spectrum of the fused de-clouded remote sensing image; S3: Repeat S1 and S2 for t times; S4: For the data testing process, the cloud cover optical remote sensing and SAR test images are input into the cross-modal feature fusion network with trained weight parameters to obtain the final fusion declouding results.

Images