Thick cloud removal method and device for SAR-optical remote sensing image fusion

Abstract

The invention discloses a thick cloud removal method and device for SAR-optical remote sensing image fusion. The method comprises the following steps: acquiring an SAR image and a corresponding optical remote sensing image; based on a remote sensing image cloud removal model, processing and fusing the SAR image and optical remote sensing image data features, and outputting a cloud-removed remote sensing image; wherein the remote sensing image cloud removal model comprises a space channel attention module, a frequency domain feature extraction module and a space frequency domain feature fusion module; according to the cloud removal method and device provided by the invention, the penetration characteristic of an SAR image to a cloud layer and rich spectral information of an optical image can be fully utilized, an image with clear details can be output, the image precision and the image fidelity are improved, the problem of limited information of a single-mode image in a thick cloud shielding scene is solved, and the cloud removal efficiency is improved. And the reconstruction precision of the model on the earth surface information of the thick cloud coverage area is improved.

Description

Technical Field

[0001] This invention relates to a method for removing thick clouds based on SAR-optical remote sensing image fusion, belonging to the field of image restoration technology. Background Technology

[0002] With the rapid development of remote sensing imaging technology, remote sensing images have achieved significant improvements in spectral resolution, spatial resolution, and observation accuracy, becoming indispensable tools for investigation, monitoring, analysis, and prediction in fields such as resource management, environmental monitoring, and disaster early warning. However, the average annual cloud cover on Earth exceeds 60%, and optical satellites, limited by the imaging mechanism of their optical sensors, cannot penetrate the cloud layer during image acquisition, affecting data acquisition. Therefore, resolving cloud interference in remote sensing images, repairing low-quality images affected by clouds, and improving data usability are of great significance for the widespread application of remote sensing data.

[0003] Existing methods for thick cloud removal are mainly divided into two categories: single-modal methods and multi-modal fusion methods. Single-modal methods rely solely on the information in the optical remote sensing image itself, using interpolation and deep learning-based image inpainting techniques to restore cloud areas. However, since the original information of ground features in the thick cloud-covered area is completely lost, it is difficult to achieve accurate reconstruction by relying solely on image context information, and problems such as blurred ground feature details and false textures are prone to occur.

[0004] Multimodal fusion-based thick cloud removal methods fully utilize the varying degrees of cloud interference experienced by different sensors. By integrating data from multiple sensors to obtain supplementary information, they achieve effective thick cloud removal. Synthetic Aperture Radar (SAR) images, based on microwave imaging principles, possess penetration characteristics unaffected by cloud cover, illumination, or other meteorological conditions, enabling stable acquisition of surface structure information and providing crucial complementary data for thick cloud removal. However, existing SAR-optical image fusion cloud removal methods often employ simple feature stitching strategies, failing to fully exploit the essential differences and complementary advantages between the two modalities, ultimately impacting the accuracy and image fidelity of thick cloud removal. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that the prior art is prone to problems such as blurred details of ground features and low image accuracy and image fidelity.

[0006] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution.

[0007] On one hand, the present invention provides a method for thick cloud removal in SAR-optical remote sensing image fusion, comprising:

[0008] Acquire SAR images and corresponding optical remote sensing images;

[0009] The trained remote sensing image cloud removal model is used to process and fuse the features of the SAR image and optical remote sensing image data, and output the cloud-removed remote sensing image.

[0010] The remote sensing image cloud removal model includes a spatial channel attention module, a frequency domain feature extraction module, and a spatial-frequency domain feature fusion module.

[0011] By leveraging the cloud penetration characteristics of SAR images and the rich spectral information of optical images, the problem of limited information in single-modal images under thick cloud cover scenarios is solved, thereby improving the model's accuracy in reconstructing surface information in areas covered by thick clouds.

[0012] The remote sensing image cloud removal model is trained, including:

[0013] The SAR images and corresponding optical remote sensing images are randomly divided into training and testing sets according to a fixed ratio;

[0014] The training set images are input into the remote sensing image cloud removal model for training until the remote sensing image cloud removal model converges, thus obtaining the network model with optimal parameters;

[0015] The test set is input into the remote sensing image cloud removal model for verification.

[0016] The steps of the remote sensing image cloud removal model include:

[0017] The SAR image data is input into the frequency domain feature extraction module to obtain a low-frequency feature map. and high-frequency feature maps ;

[0018] The optical remote sensing image is input into the spatial channel attention module to obtain the feature map. ;

[0019] During the encoding stage, the low-frequency feature map and the feature map Input spatial frequency domain feature fusion module to obtain fused feature map ;

[0020] During the decoding stage, the high-frequency feature map is... The feature map The input spatial frequency domain feature fusion module performs upsampling on the features of the fused optical remote sensing image data, restoring the spatial resolution of the feature map to its original size, and obtaining the cloud removal result.

[0021] The remote sensing image cloud removal model includes a SAR branch and an optical remote sensing image branch;

[0022] The SAR branch uses a frequency domain feature extraction module to transform the features of optical remote sensing image data from the spatial domain to the frequency domain, and extracts low-frequency and high-frequency features respectively.

[0023] The optical remote sensing image branch utilizes a spatial channel attention module to capture the spatial and channel relationships of the optical remote sensing image data features.

[0024] The extracted low-frequency and high-frequency features are fused using a dual-path fusion mechanism that combines spatial feature enhancement and frequency domain attention weighting through a spatial frequency domain feature fusion module.

[0025] The frequency domain feature extraction module implements the following steps:

[0026] The SAR image is processed by convolution and ReLU activation function to obtain a feature map. ;

[0027] The feature map The features are transformed to the frequency domain using a two-dimensional discrete wavelet transform to obtain the low-frequency feature map. Horizontal high-frequency feature map Vertical high-frequency feature map and diagonal high-frequency feature maps ;

[0028] The low-frequency feature map Low-frequency features are output after convolution. ;

[0029] The horizontal high-frequency feature map Vertical high-frequency feature map and diagonal high-frequency feature maps After concatenation along the channel dimension, high-frequency features are output through a convolution operation. .

[0030] The implementation steps of the spatial channel attention module include:

[0031] The optical remote sensing image is processed through two sets of parallel convolutions to obtain the query matrix of the channel branches. Key matrix Sum matrix and the query matrix of spatial branches Key matrix Sum matrix ;

[0032] The query matrix of the channel branch Key matrix Sum matrix Reshape into a new matrix, and then use the query matrix of the channel branches after reshaping. AND key matrix The transpose multiplication is then passed through the Softmax function and then combined with the value matrix of the channel branch. Multiplying yields the channel attention map ;

[0033] The query matrix, key matrix, and value matrix of the spatial branch are divided into non-overlapping windows of size N. For each window, the query matrix of the spatial branch is... Bond matrix After multiplying by the transpose, the value matrix after passing through the Softmax function and spatial branching is... Multiplication yields a spatial attention map ;

[0034] The channel attention map With the spatial attention map Element-wise addition is performed to obtain the output feature map of the spatial channel attention module. .

[0035] The optical remote sensing image branch utilizes a spatial channel attention module to capture the spatial and channel relationships of features, thereby enhancing feature representation.

[0036] The new matrix has dimension [ ]. The matrix, where The height of the input feature map, The width of the input feature map. The number of channels in the input feature map.

[0037] The spatial frequency domain feature fusion module implements the following steps:

[0038] The output feature map of the spatial channel attention module The input features are processed through two parallel convolutional paths, one of which applies the Gelu non-linear activation function. The outputs of the two convolutional paths are multiplied to obtain the spatial feature map. ;

[0039] The low-frequency features output by the frequency domain feature extraction module or high frequency characteristics As input features, the input features are processed through two parallel convolutional paths, and then transformed into the frequency domain by fast Fourier transform to obtain frequency domain attention weights.

[0040] The frequency domain attention weights are combined with the learnable parameter matrix. After multiplication, the feature maps of the two parallel paths are obtained by inverse Fourier transform. and feature map ;

[0041] For the feature map After applying the GeLU nonlinear activation function and the feature map Multiply to obtain frequency domain features ;

[0042] The spatial feature map With the frequency domain features After addition, the output of the spatial frequency domain feature fusion module is obtained through convolution operation. .

[0043] The spatial-frequency domain feature fusion module employs a dual-path fusion mechanism of spatial feature enhancement and frequency domain attention weighting to achieve complementary fusion of multimodal features.

[0044] Secondly, the present invention provides a thick cloud removal device for SAR-optical remote sensing image fusion, comprising:

[0045] The image acquisition module is used to acquire SAR images and corresponding optical remote sensing images;

[0046] The remote sensing image cloud removal module is used to perform multimodal feature fusion based on image data;

[0047] The image output module is used to output cloud-free remote sensing images.

[0048] Thirdly, the present invention provides a computer system, comprising:

[0049] Memory, used to store computer programs;

[0050] A processor for executing the computer program to implement the steps of the thick cloud removal method for SAR-optical remote sensing image fusion.

[0051] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0052] The cloud removal method and apparatus provided by this invention can make full use of the cloud penetration characteristics of SAR images and the rich spectral information of optical images, and can output images with clear details, improve image accuracy and image fidelity, solve the problem of limited information in single-modal images under thick cloud cover, and improve the model's reconstruction accuracy of surface information in areas covered by thick clouds. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the thick cloud removal method for SAR-optical remote sensing image fusion as shown in Embodiment 1 of the present invention.

[0054] Figure 2 This is a schematic diagram of the spatial frequency domain feature fusion module shown in Embodiment 1 of the present invention;

[0055] Figure 3 This is a schematic diagram of the spatial channel attention module shown in Embodiment 1 of the present invention;

[0056] Figure 4 This is a schematic diagram of the frequency domain feature extraction module shown in Embodiment 1 of the present invention;

[0057] Figure 5 This is a flowchart of the thick cloud removal device for SAR-optical remote sensing image fusion shown in Embodiment 1 of the present invention. Detailed Implementation

[0058] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0059] Example 1

[0060] like Figure 1 This embodiment describes a thick cloud removal method for SAR-optical remote sensing image fusion, including:

[0061] Acquire SAR images and corresponding optical remote sensing images;

[0062] The trained remote sensing image cloud removal model is used to process and fuse the features of the SAR image and optical remote sensing image data, and output the cloud-removed remote sensing image.

[0063] The remote sensing image cloud removal model includes a spatial channel attention module, a frequency domain feature extraction module, and a spatial-frequency domain feature fusion module.

[0064] Specifically, the SAR images and corresponding optical remote sensing images in this embodiment are selected from the SEN12MS-CR dataset as the benchmark.

[0065] The remote sensing image cloud removal model is trained, including:

[0066] The SAR images and corresponding optical remote sensing images are randomly divided into training and testing sets according to a fixed ratio;

[0067] The training set images are input into the remote sensing image cloud removal model for training until the remote sensing image cloud removal model converges, thus obtaining the network model with optimal parameters;

[0068] The test set is input into the remote sensing image cloud removal model for verification.

[0069] The steps of the remote sensing image cloud removal model include:

[0070] The SAR image data is input into the frequency domain feature extraction module to obtain a low-frequency feature map. and high-frequency feature maps ;

[0071] The optical remote sensing image is input into the spatial channel attention module to obtain the feature map. ;

[0072] During the encoding stage, the low-frequency feature map and the feature map Input spatial frequency domain feature fusion module to obtain fused feature map ;

[0073] During the decoding stage, the high-frequency feature map is... The feature map The input spatial frequency domain feature fusion module performs upsampling on the features of the fused optical remote sensing image data, restoring the spatial resolution of the feature map to its original size, and obtaining the cloud removal result.

[0074] The remote sensing image cloud removal model includes a SAR branch and an optical remote sensing image branch;

[0075] The SAR branch uses a frequency domain feature extraction module to transform the features of optical remote sensing image data from the spatial domain to the frequency domain, and extracts low-frequency and high-frequency features respectively.

[0076] The optical remote sensing image branch utilizes a spatial channel attention module to capture the spatial and channel relationships of the optical remote sensing image data features.

[0077] The extracted low-frequency and high-frequency features are fused using a dual-path fusion mechanism that combines spatial feature enhancement and frequency domain attention weighting through a spatial frequency domain feature fusion module.

[0078] Specifically, the remote sensing image cloud removal model has a SAR image branch and an optical remote sensing image branch. The remote sensing image cloud removal model includes a spatial channel attention module, a frequency domain feature extraction module, and a spatial frequency domain feature fusion module.

[0079] like Figure 2 As shown, the frequency domain feature extraction module includes the following steps:

[0080] The SAR image is processed by convolution and ReLU activation function to obtain a feature map. ;

[0081] The feature map The features are transformed to the frequency domain using a two-dimensional discrete wavelet transform to obtain the low-frequency feature map. Horizontal high-frequency feature map Vertical high-frequency feature map and diagonal high-frequency feature maps ;

[0082] The low-frequency feature map Low-frequency features are output after convolution. ;

[0083] The horizontal high-frequency feature map Vertical high-frequency feature map and diagonal high-frequency feature maps After concatenation along the channel dimension, high-frequency features are output through a convolution operation. .

[0084] Specifically, let the input feature map be... .in, These are the height, width, and number of channels of the input feature map, respectively; feature map pass Feature maps are obtained by convolution and ReLU activation functions. ,in, for The weight parameters of point convolution.

[0085] Specifically, low-frequency characteristics ,in, for Point convolution, It is a low-frequency characteristic.

[0086] Specifically, high-frequency features ,in, For convolution calculation, It is a horizontal high-frequency feature. It is a vertical high-frequency feature. It is a diagonal high-frequency feature.

[0087] like Figure 3 As shown, the implementation steps of the spatial channel attention module include:

[0088] The optical remote sensing image is processed through two sets of parallel convolutions to obtain the query matrix of the channel branches. Key matrix Sum matrix and the query matrix of spatial branches Key matrix Sum matrix ;

[0089] The query matrix of the channel branch Key matrix Sum matrix Reshape into a new matrix, and then use the query matrix of the channel branches after reshaping. AND key matrix The transpose multiplication is then passed through the Softmax function and then combined with the value matrix of the channel branch. Multiplying yields the channel attention map ;

[0090] The query matrix, key matrix, and value matrix of the spatial branch are divided into non-overlapping windows of size N. For each window, the query matrix of the spatial branch is... Bond matrix After multiplying by the transpose, the value matrix after passing through the Softmax function and spatial branching is... Multiplication yields a spatial attention map ;

[0091] The channel attention map With the spatial attention map Element-wise addition is performed to obtain the output feature map of the spatial channel attention module. .

[0092] Specifically, let the input feature map be... .in, These are the height, width, and number of channels of the input feature map, respectively.

[0093] Specifically, feature map After two parallel sets Point convolution and Depthwise separable convolution yields two sets of matrices for attention computation: a query matrix, a key matrix, and a value matrix.

[0094]

[0095] in, for The weight parameters of point convolution, for Weight parameters for depthwise convolution.

[0096] Specifically, channel attention map ,in, for Activation function This is the query matrix for channel branches. The key matrix of the channel branches, This is the value matrix for the channel branches.

[0097] Specifically, spatial attention maps ,in, for Activation function The query matrix for spatial branches. The key matrix of the spatial branch, The value matrix is ​​the branch matrix of the space.

[0098] Specifically, the output feature map of the attention module ,in, This is a channel attention map. This is a spatial attention map.

[0099] The new matrix has dimension [ ]. The matrix, where The height of the input feature map, The width of the input feature map. The number of channels in the input feature map.

[0100] like Figure 4 As shown, the spatial frequency domain feature fusion module includes the following steps:

[0101] The output feature map of the spatial channel attention module The input features are processed through two parallel convolutional paths, one of which applies the Gelu non-linear activation function. The outputs of the two convolutional paths are multiplied to obtain the spatial feature map. ;

[0102] The low-frequency features output by the frequency domain feature extraction module or high frequency characteristics As input features, the input features are processed through two parallel convolutional paths, and then transformed to the frequency domain using a Fast Fourier Transform to obtain frequency domain attention weights. These frequency domain attention weights are then combined with a learnable parameter matrix. After multiplication, the feature maps of the two parallel paths are obtained by inverse Fourier transform. and feature map ;

[0103] For the feature map After applying the GeLU nonlinear activation function and the feature map Multiply to obtain frequency domain features ;

[0104] The spatial feature map With the frequency domain features After addition, the output of the spatial frequency domain feature fusion module is obtained through convolution operation. .

[0105] Specifically, the output feature map After two parallel sets Point convolution and Depthwise separable convolution, where one path applies the Gelu non-linear activation function, and the outputs of the two paths are multiplied to obtain spatial features: ,in, For Gelu nonlinear activation functions, This is for element-wise multiplication.

[0106] Specifically, the low-frequency characteristics of the output or high frequency characteristics After two parallel sets Point convolution and After depthwise separable convolution, the features are transformed to the frequency domain using Fast Fourier Transform to obtain the feature map: or ,in For Fast Fourier Transform, for The weight parameters of point convolution, for Weight parameters for depthwise separable convolution.

[0107] Specifically, the feature maps of the two parallel paths and feature map : , , For inverse Fourier transform, for The weight parameters of point convolution, for Feature maps of point convolution, for Feature maps of depthwise separable convolutions.

[0108] Specifically, frequency domain features .

[0109] Specifically, the output of the spatial frequency domain feature fusion module ,in, This is a convolution operation.

[0110] Example 2

[0111] Based on the same inventive concept as Embodiment 1, such as Figure 5 As shown in the figure, this embodiment introduces a thick cloud removal device for SAR-optical remote sensing image fusion, including:

[0112] The image acquisition module is used to acquire SAR images and corresponding optical remote sensing images;

[0113] The remote sensing image cloud removal module is used to perform multimodal feature fusion based on image data;

[0114] The image output module is used to output cloud-free remote sensing images.

[0115] Example 3

[0116] Based on the same inventive concept as other embodiments, this embodiment introduces a computer system characterized by comprising:

[0117] Memory, used to store computer programs;

[0118] A processor for executing the computer program to implement the steps of a thick cloud removal method for SAR-optical remote sensing image fusion.

Claims

1. A method for thick cloud removal in SAR-optical remote sensing image fusion, characterized in that, include: Acquire SAR images and corresponding optical remote sensing images; The trained remote sensing image cloud removal model is used to process and fuse the features of the SAR image and optical remote sensing image data, and output the cloud-removed remote sensing image. The remote sensing image cloud removal model includes a spatial channel attention module, a frequency domain feature extraction module, and a spatial-frequency domain feature fusion module.

2. The thick cloud removal method for SAR-optical remote sensing image fusion according to claim 1, characterized in that, Training the cloud removal model for the remote sensing image includes: The SAR images and corresponding optical remote sensing images are randomly divided into training and testing sets according to a fixed ratio; The training set images are input into the remote sensing image cloud removal model for training until the remote sensing image cloud removal model converges, thus obtaining the network model with optimal parameters; The test set is input into the remote sensing image cloud removal model for verification.

3. The thick cloud removal method for SAR-optical remote sensing image fusion according to claim 1, characterized in that, The steps involved in processing a remote sensing image cloud removal model include: The SAR image data is input into the frequency domain feature extraction module to obtain a low-frequency feature map. and high-frequency feature maps ; The optical remote sensing image is input into the spatial channel attention module to obtain the feature map. ; During the encoding stage, the low-frequency feature map and the feature map Input spatial frequency domain feature fusion module to obtain fused feature map ; During the decoding stage, the high-frequency feature map is... The feature map The input spatial frequency domain feature fusion module performs upsampling on the features of the fused optical remote sensing image data, restoring the spatial resolution of the feature map to its original size, and obtaining the cloud removal result.

4. The thick cloud removal method for SAR-optical remote sensing image fusion according to claim 1, characterized in that, The remote sensing image cloud removal model includes a SAR branch and an optical remote sensing image branch; The SAR branch uses a frequency domain feature extraction module to transform the features of optical remote sensing image data from the spatial domain to the frequency domain, and extracts low-frequency and high-frequency features respectively. The optical remote sensing image branch utilizes a spatial channel attention module to capture the spatial and channel relationships of the optical remote sensing image data features. The extracted low-frequency and high-frequency features are fused using a spatial feature enhancement and frequency domain attention weighting dual-path fusion module.

5. The thick cloud removal method for SAR-optical remote sensing image fusion according to claim 1, characterized in that, The frequency domain feature extraction module implements the following steps: The SAR image is processed by convolution and ReLU activation function to obtain a feature map. ; The feature map The features are transformed to the frequency domain using a two-dimensional discrete wavelet transform to obtain the low-frequency feature map. Horizontal high-frequency feature map Vertical high-frequency feature map and diagonal high-frequency feature maps ; The low-frequency feature map Low-frequency features are output after convolution. ; The horizontal high-frequency feature map Vertical high-frequency feature map and diagonal high-frequency feature maps After concatenation along the channel dimension, high-frequency features are output through a convolution operation. .

6. The thick cloud removal method for SAR-optical remote sensing image fusion according to claim 1, characterized in that, The implementation steps of the spatial channel attention module include: The optical remote sensing image is processed through two sets of parallel convolutions to obtain the query matrix of the channel branches. Key matrix Sum matrix and the query matrix of spatial branches Key matrix Sum matrix ; The query matrix of the channel branch Key matrix Sum matrix Reshape into a new matrix, and then use the query matrix of the channel branches after reshaping. AND key matrix The transpose multiplication is then passed through the Softmax function and then combined with the value matrix of the channel branch. Multiplying yields the channel attention map ; The query matrix, key matrix, and value matrix of the spatial branch are divided into non-overlapping windows of size N. For each window, the query matrix of the spatial branch is... Bond matrix After multiplying by the transpose, the value matrix after passing through the Softmax function and spatial branching is... Multiplication yields a spatial attention map ; The channel attention map With the spatial attention map Element-wise addition is performed to obtain the output feature map of the spatial channel attention module. .

7. The thick cloud removal method for SAR-optical remote sensing image fusion according to claim 6, characterized in that, The new matrix has dimension [ ]. The matrix, where The height of the input feature map, The width of the input feature map. The number of channels in the input feature map.

8. The thick cloud removal method for SAR-optical remote sensing image fusion according to claim 1, characterized in that, The spatial frequency domain feature fusion module implements the following steps: The output feature map of the spatial channel attention module The input features are processed through two parallel convolutional paths, one of which applies the Gelu non-linear activation function. The outputs of the two convolutional paths are multiplied to obtain the spatial feature map. ; The low-frequency features output by the frequency domain feature extraction module or high frequency characteristics As input features, the input features are processed through two parallel convolutional paths, and then transformed into the frequency domain by fast Fourier transform to obtain frequency domain attention weights. The frequency domain attention weights are combined with the learnable parameter matrix. After multiplication, the feature maps of the two parallel paths are obtained by inverse Fourier transform. and feature map ; For the feature map After applying the GeLU nonlinear activation function and the feature map Multiply to obtain frequency domain features ; The spatial feature map With the frequency domain features After addition, the output of the spatial frequency domain feature fusion module is obtained through convolution operation. .

9. A thick cloud removal device for SAR-optical remote sensing image fusion, characterized in that, include: The image acquisition module is used to acquire SAR images and corresponding optical remote sensing images; The remote sensing image cloud removal module is used to perform multimodal feature fusion based on image data; The image output module is used to output cloud-free remote sensing images.

10. A computer system, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the thick cloud removal method for SAR-optical remote sensing image fusion as described in any one of claims 1 to 8.

Images