Cloud removal method based on hybrid neural network fusing optical and sar images

Abstract

This invention discloses a cloud removal method based on hybrid neural networks fusing optical and SAR images, belonging to the field of image processing technology. The method involves acquiring a dataset and preprocessing it, then dividing it into training and validation sets according to a predetermined ratio. A cloud removal model based on hybrid neural networks fusing optical and SAR images is constructed, mainly composed of a lightweight fusion residual block, an asymmetric feature pyramid module, a parallel gated attention module, and a cloud mask-guided attention fusion module. The preprocessed training and validation datasets are input into the cloud removal model for training and validation. The loss function is calculated, backpropagation is performed, network parameters are updated, and the optimal parameter model is obtained. The preprocessed test set is input into the trained optimal parameter model, and the cloud-removed image is output. This invention achieves high-quality restoration of cloud-covered areas and preservation of ground feature characteristics by jointly utilizing optical image information and SAR image characteristics.

Description

Technical Field

[0001] This invention relates to a cloud removal method, specifically a cloud removal method based on hybrid neural networks fusing optical and SAR images, belonging to the field of image processing technology. Background Technology

[0002] With the advancement of satellite technology, the remote sensing images that can be acquired have the characteristics of wide coverage, fast revisit, and multiple scales; however, the opacity of clouds to spectral signals severely restricts effective observation, and the loss of surface information in areas covered by thick clouds directly affects the accuracy of land cover classification and change monitoring; therefore, effectively removing cloud interference has become a key step in remote sensing preprocessing.

[0003] Existing cloud removal methods for optical remote sensing images can be divided into two categories: physics-prior-driven and data feature-driven. Early research relied on expert knowledge and physical modeling, developing physics-prior-based cloud removal methods. These include cloud removal methods based on dark channel priors, cloud removal methods based on fog line / physical scattering models, and cloud removal methods based on optimization and variation. These methods have poor generalization ability due to their reliance on specific assumptions, and their performance degrades in complex scenes. Therefore, synthetic aperture radar (SAR), which can penetrate cloud layers, is often used as an effective auxiliary data source.

[0004] In recent years, data feature-driven methods have developed rapidly, with convolutional neural networks (CNNs) playing a crucial role. Typical works include: two-stream networks for optical image reconstruction, but the generated images lack sufficient sharpness; deep residual networks that fuse multi-scale features through cross-layer connections to improve cloud extraction and alleviate gradient vanishing; residual multi-scale dilatation networks achieving good cloud removal results, but the models are complex and lack sufficient local feature extraction; and the use of channel attention residual blocks to enhance geometric structure extraction and suppress cloud artifacts. While these methods have made progress, they have not fundamentally solved the problem of insufficient long-range modeling capabilities of CNNs, resulting in poor coordination between the restored region and its surrounding environment.

[0005] To overcome the limitations of CNNs, Transformers are introduced to establish long-range dependencies using self-attention mechanisms, improving the consistency of restored images in terms of color and structure. Related work includes: self-attention cloud removal algorithms based on global-local fusion, achieving high-quality reconstruction but with high computational cost; introducing convolution into Transformers to perform self-attention in non-overlapping local windows, reducing computational cost and improving global consistency and color restoration; introducing Transformer encoder structures to enhance the model's ability to learn global information and alleviate image blurring; and heterogeneous parallel cloud removal frameworks that introduce global multi-scale spatial correlation models to extract optical features, but their utilization of SAR information is mostly limited to deep feature fusion, making it difficult to accurately compensate for surface information lost due to cloud cover. Although Transformer variants improve efficiency through local design, their self-attention mechanisms still struggle to capture fine structures when processing SAR images.

[0006] Recently, the state-space model Mamba has begun to be applied to cloud removal tasks, but the exploration of multi-source information fusion is still insufficient. Some models are limited to single optical image input and are suitable for thin cloud removal; other works have integrated Mamba into SAR-assisted cloud removal to model global dependencies, but the fusion strategies are mostly simple convolution followed by stitching, which fails to achieve deep feature complementarity and synergy, thus limiting the improvement of cloud removal performance.

[0007] In practical remote sensing image applications, optical images are easily affected by cloud cover. The shape and size of clouds lack qualitative standards and are mostly irregular in shape. How to accurately recover cloudless surface information is a problem that declouding models need to consider. Secondly, poor lighting conditions will affect the spectral consistency of declouded images, which is also a challenge for declouding models.

[0008] To address the aforementioned issues, a cloud removal method based on hybrid neural networks that fuses optical and SAR images is proposed. Summary of the Invention

[0009] The purpose of this invention is to provide a cloud removal method based on hybrid neural networks that fuses optical and SAR images. By using a multi-stage dual-stream coding architecture to jointly utilize the rich spectral information of optical images and the penetration characteristics of SAR images, and by introducing a cloud mask-guided attention fusion module to perform targeted learning on cloud-covered areas, this method solves the problem of accurate recovery of surface information under irregular cloud cover interference and poor lighting conditions, and achieves high-quality restoration of cloud areas and preservation of ground features.

[0010] To achieve the above-mentioned objectives, this invention provides a cloud removal method based on hybrid neural networks fusing optical and SAR images, comprising the following steps:

[0011] Step 1: Obtain the SMILE-CR and SEN12MS-CR datasets, which include cloudless images, SAR images, and clouded images. After preprocessing the datasets, divide them into training and validation sets according to the set ratio.

[0012] Step 2: Construct a Transformer-CNN hybrid neural network that fuses optical and SAR images. This mainly consists of a Lightweight Fusion Residual (LFR), an Asymmetric Feature Pyramid (AFP), a Parallel Gated Attention Transformer (PGAT), and a Cloud Guided Attention Fusion (CGAF). The Lightweight Fusion Residual processes the SAR image, the Asymmetric Feature Pyramid processes the optical image, the Parallel Gated Attention Transformer processes the fused features, and the Cloud Guided Attention Fusion Transformer performs reinforcement learning on the cloud region using a cloud mask.

[0013] Step 3: Input the preprocessed clouded images and SAR images from the training and validation sets into the hybrid neural network from Step 2 for training and validation, calculate the loss function and perform backpropagation, update the network parameters, and obtain the optimal parameter model.

[0014] Step 4: Input the preprocessed test set into the optimal parameter model trained in Step 3, and output the cloud-removed image of the clouded image.

[0015] The steps in step one, namely, obtaining the dataset, preprocessing the dataset, and partitioning the dataset, are as follows:

[0016] The acquisition of the dataset specifically involves acquiring the SMILE-CR dataset and the SEN12MS-CR dataset, which include cloud-covered images, cloudless images, and SAR images.

[0017] The dataset preprocessing specifically involves preprocessing the original images from the SMILE-CR and SEN12MS-CR datasets. This preprocessing includes cropping the VV polarization of the synthetic aperture radar (SAR) images to [−25, 0] and the VH polarization to [−35, 0]. The cropped images are then rescaled to the range [0, 1]. The 13 channels of the optical images are cropped to [0, 10000] and then normalized to the range [0, 1].

[0018] The specific method for dividing the dataset is as follows: the enhanced SMILE-CR dataset and SEN12MS-CR dataset are each divided into two training sets and two validation sets, with the training set accounting for 90% and the validation set accounting for 10%.

[0019] The steps of fusing optical and SAR images into a Transformer-CNN hybrid neural network in step two are as follows:

[0020] Step 2.1: The preprocessed SAR image is input into a lightweight fusion residual block, wherein the lightweight fusion residual block is as follows: Figure 3 As shown, it mainly consists of a Dynamic Ghost Module, an Efficient Channel Attention Module, a Convolution-BN-GELU Module, Max Pooling, Average Pooling, a Sigmoid function, and a ReLU activation function.

[0021] The input is first processed by a lightweight feature extraction module called Dynamic Phantom Convolution. This module first uses 1×1 convolutions to reduce the channel dimensionality of the input features, significantly reducing computational complexity. Then, a batch normalization layer and the SiLU activation function are introduced to improve training stability and accelerate convergence. Based on this, an efficient channel selection mechanism is used to filter important features by calculating the global mean of each channel. Combine it with learnable weights Multiply to obtain the channel importance score The topK operation is used to select the K most important channels. This design can significantly reduce the number of parameters and computational overhead while maintaining representational power.

[0022] The output of the dynamic phantom convolution module is batch normalized and then fed into the efficient channel attention module. This module learns and assigns weights to each channel through adaptive one-dimensional convolution, thereby enhancing the response of discriminative feature channels while suppressing secondary or redundant channels, thus improving the quality and focusing ability of feature representation.

[0023] Subsequently, the features are further refined and enhanced through two consecutive feature activation units.

[0024] The processed feature stream will be split into two branches: one through max pooling and the other through average pooling, to extract and aggregate different spatial context information; the pooling results of the two branches will be activated by the Sigmoid function to generate a spatial attention map, which will then be multiplied with the original features to achieve weighting.

[0025] Finally, the weighted output of the two branches is residually connected to the original input of the module and output through the ReLU activation function to ensure gradient flow and fuse multi-level information.

[0026] Step 2.2: The preprocessed optical image is input into the asymmetric feature pyramid module, as shown in the following figure. Figure 4 As shown, it mainly consists of asymmetric convolution, decomposed convolution, 1×1 convolution, ReLU activation function, feature activation unit, multi-scale feature aggregation (MSFA) module, dilated convolution, average pooling, batch normalization and GELU activation function.

[0027] The input features are first fed into an asymmetric convolution layer. Two sets of complementary decomposed convolutions are used to extract features in the horizontal and vertical directions, respectively, to enhance the network’s ability to perceive the geometric structure of clouds and ground features in different directions.

[0028] Next, the output features from these two directions are fused and then channel compression and nonlinear activation are achieved through 1×1 convolution and ReLU activation function.

[0029] Subsequently, the features are further refined and enhanced through two cascaded feature activation units to improve the network's expressive power. The outputs of the feature activation units are fed into the Multi-Scale Feature Aggregation (MSFA) module. This module extracts multi-scale edge detail information through a parallel multi-branch structure to address the significant multi-scale variations exhibited by clouds in images: three branches use dilated convolutions with different dilation rates to expand the receptive field; another branch employs an average pooling-convolution structure to fuse local context. The outputs of all branches are processed by batch normalization and the GELU activation function before feature concatenation.

[0030] Finally, a 1×1 convolutional layer is used to achieve dimensionality reduction and feature fusion, outputting enhanced multi-scale features. The enhanced features output by the MSFA module are then connected to the module's original input via residual connections. The sum is then processed by a ReLU activation function, ultimately outputting a feature representation that integrates directional details and multi-scale contextual information, forming a complete pyramid structure.

[0031] Step 2.3: The feature images output from steps 2.1 and 2.2 are passed through a gated self-attention module, wherein the gated self-attention module is as follows: Figure 5 As shown, it mainly consists of convolutional layers, window attention mechanism, gating mechanism, sigmoid activation function, convolutional integration layer, ReLU activation function, and dropout layer.

[0032] The gated self-attention module processes two information streams in parallel: one is a multi-scale feature aggregation module, which is used to fuse multi-scale information from SAR and optical images to overcome cloud cover and extract robust features; the other is a gated self-attention module, whose core objective is to model the low-level orientational relationship between clouded and cloudless areas to enhance effective features and suppress interference.

[0033] In the gated self-attention branch, the input features are first preprocessed by convolutional layers and then fed into the window attention mechanism to obtain the attention features. Subsequently, Compared with the original input features The features are then concatenated. The concatenated features are processed through a gating mechanism, which generates a pair of complementary attention weight maps through convolution and a sigmoid activation function. and These two weight maps are respectively related to the original input. and attention characteristics The algorithm performs element-wise weighted multiplication and summation, and finally integrates the results through a convolutional layer to output the enhanced feature representation.

[0034] The output of the gated self-attention module and the original input Residual connections are performed. The summed result is then passed through a convolutional layer, a ReLU activation function, and a random dropout layer for nonlinear transformation and regularization to enhance generalization ability. This processed result is then passed through another convolutional layer and added a second time to the result of the aforementioned residual connections, ultimately outputting features that integrate attention enhancement and multi-scale context.

[0035] Step 2.4: The feature images output from Steps 2.1, 2.2, and 2.3 are processed through a cloud mask-guided attention fusion module, as described in the following steps: Figure 6 As shown, it mainly consists of 1×1 convolution, GELU activation function, ECA channel attention module, Softmax function, and Sigmoid gating mechanism.

[0036] First, the three input features are processed uniformly. Each feature is transformed by 1×1 convolution and GELU activation function to achieve channel-dimensional alignment and non-linear feature enhancement.

[0037] The enhanced features are then fed into the ECA channel attention module. This module learns and assigns a weight to each channel of the feature map through adaptive one-dimensional convolution, thereby amplifying important features and suppressing minor or interfering features in the channel dimension to improve the quality of feature representation.

[0038] Meanwhile, the input cloud mask undergoes deep feature extraction and dimensionality adjustment via a separate convolutional module, followed by normalization using the Softmax function to generate a corresponding spatial weight map. This map accurately reflects the probability distribution of cloud-covered areas in the image.

[0039] In the fusion phase, the aforementioned spatial weight map is multiplied element-wise with each path feature source after ECA weighting to achieve spatial guidance based on cloud masks. Then, a Sigmoid gating mechanism is used to adaptively weight the multiplication results to obtain the final attention output for each feature source.

[0040] The outputs of all feature sources are concatenated, integrated through a convolutional layer, and activated again by the GELU function to finally generate a high-quality cloud-reconstructed image.

[0041] The lightweight fusion residual block structure in step 2.1 is as follows:

[0042] The lightweight fusion residual block mainly consists of a dynamic phantom convolution module, an efficient channel attention module, a feature activation unit, max pooling, average pooling, a sigmoid function, and a ReLU activation function;

[0043] The dynamic phantom convolution module is used to perform lightweight feature extraction and channel selection;

[0044] The high-efficiency channel attention module is used to enhance the feature response of key channels;

[0045] The feature activation unit is used to further deepen and enhance the feature representation;

[0046] The max pooling and average pooling are used to extract and aggregate different spatial context information;

[0047] The Sigmoid function is used to generate spatial attention weights;

[0048] The preprocessed SAR image is input into a lightweight fusion residual block. The input features are first extracted using a dynamic phantom convolution module: first, dimensionality reduction is achieved through 1×1 convolution, followed by batch normalization and SiLU activation function for stable training; then, the global mean of each channel is calculated and multiplied by the learnable weights, and the most important channels are selected using the topK operation to reduce the number of parameters while maintaining representational power.

[0049] The output of this module is batch normalized and then fed into the efficient channel attention module. This module assigns weights to each channel through adaptive one-dimensional convolution to enhance key channels and suppress redundant information.

[0050] Subsequently, the features are further enhanced by two consecutive feature activation units. The processed feature stream is divided into two branches, which aggregate different spatial context information through max pooling and average pooling, respectively. The results of the two branches are activated by the Sigmoid function to generate a spatial attention map, which is then weighted and fused with the original features.

[0051] Finally, the weighted output is residually connected to the original input of the module, and then output through the ReLU activation function to fuse multi-level information and ensure effective gradient flow.

[0052] The asymmetric feature pyramid module structure in step 2.2 is as follows:

[0053] The asymmetric feature pyramid module mainly consists of asymmetric convolution, decomposed convolution, 1×1 convolution, ReLU activation function, feature activation unit, multi-scale feature aggregation module, dilated convolution, average pooling, batch normalization and GELU activation function.

[0054] The asymmetric convolution and deconvolution mentioned above are used to extract features in different directions to perceive geometric structures;

[0055] The 1×1 convolution and ReLU activation function are used to achieve channel compression and nonlinear activation;

[0056] The feature activation unit is used to deepen and enhance feature representation;

[0057] The multi-scale feature aggregation module captures multi-scale contextual information through dilated convolution and "average pooling-convolution" structures;

[0058] The batch normalization and GELU activation function are used to stabilize training and handle multi-branch outputs;

[0059] The residual connections are used to fuse the original input and enhanced features to form a pyramid structure.

[0060] The preprocessed optical images are input into the asymmetric feature pyramid module. First, features in the horizontal and vertical directions are extracted through two sets of decomposition convolutions in the asymmetric convolution to enhance the perception of the geometric structure of clouds and ground features in different directions.

[0061] Next, the features from these two directions are fused, and 1×1 convolution and ReLU function are used for channel compression and nonlinear activation.

[0062] Subsequently, the features are enhanced by two cascaded feature activation units and then fed into a multi-scale feature aggregation module. This module employs a multi-branch parallel structure: three branches use dilated convolutions with different dilation rates to expand the receptive field, while the other branch fuses local context through an "average pooling-convolution" structure. The outputs of each branch are batch normalized and processed by the GELU function before being concatenated, and then dimensionality reduction and fusion are achieved through 1×1 convolutions to obtain enhanced multi-scale features.

[0063] Finally, the obtained multi-scale features are added to the original input of the module through residual connections, and then activated by the ReLU function to output the final feature representation that integrates directional details and multi-scale context.

[0064] The structure of the gated self-attention module in step 2.3 is as follows:

[0065] The gated self-attention module mainly consists of convolutional layers, window attention mechanism, gating mechanism, sigmoid activation function, convolutional integration layer, ReLU activation function, and random dropout layer;

[0066] The convolutional layer is used to preprocess the input features;

[0067] The aforementioned window attention mechanism is used to model the directional correlation between cloudy and cloudless areas;

[0068] The aforementioned gating mechanism, in conjunction with the Sigmoid activation function, generates a complementary attention weight map;

[0069] The convolutional integration layer is used to fuse the weighted features;

[0070] The ReLU activation function and random dropout layer are used for nonlinear transformation and regularization to enhance generalization ability.

[0071] The feature images output from steps 2.1 and 2.2 are input into a gated self-attention module for parallel processing. One branch, via a multi-scale feature aggregation module, fuses multi-scale information from SAR and optical images to extract robust features; the other branch, via a gated self-attention branch, models inter-regional correlations to enhance effective features.

[0072] In this branch, the input features are first preprocessed by convolutional layers, and then attention features are obtained through a window attention mechanism. Subsequently, Compared with the original input features The data is concatenated and processed by a gating mechanism: this mechanism generates a pair of complementary attention weight maps through convolution and a sigmoid function. and These two weighted graphs are respectively related to... and The elements are weighted and summed one by one, and finally integrated through a convolutional layer to output the enhanced feature representation.

[0073] This output is the same as the original input. After residual connections, the data is transformed and regularized sequentially through convolutional layers, ReLU activation functions, and random dropout layers. This result is then processed by another convolutional layer and added a second time to the result of the aforementioned residual connections, ultimately outputting features that integrate attention guidance and multi-scale context.

[0074] The structure of the cloud mask-guided attention fusion module in step 2.4 is as follows:

[0075] The cloud mask-guided attention fusion module mainly consists of 1×1 convolution, GELU activation function, ECA channel attention module, independent convolution module, softmax function, sigmoid gating mechanism and splicing fusion layer;

[0076] The 1×1 convolution and GELU activation function are used to complete the channel alignment and nonlinear enhancement of multi-path input features;

[0077] The ECA channel attention module is used to adaptively enhance important features along the channel dimension;

[0078] The independent convolutional module and Softmax function are used to generate an accurate spatial weight map from the cloud mask;

[0079] The Sigmoid gating mechanism is used for adaptive weighted fusion of spatially guided features;

[0080] The stitching and fusion layer is used to integrate all feature sources and output the final reconstructed image.

[0081] Input the feature images and cloud masks output from steps 2.1 to 2.3 into this module.

[0082] First, each feature is channel-aligned and enhanced using a 1×1 convolution and the GELU activation function. The enhanced features are then fed into the ECA channel attention module, where weights are assigned to each channel using one-dimensional convolution to improve feature quality.

[0083] Meanwhile, the input cloud mask is processed by an independent convolutional module to extract deep features, and then a spatial weight map reflecting the probability distribution of cloud coverage is generated by the Softmax function.

[0084] During the fusion phase, this spatial weight map is multiplied element-wise with the ECA-weighted features of each path to achieve spatial guidance.

[0085] Subsequently, the multiplication result is adaptively weighted using a Sigmoid gating mechanism to obtain the final attention output for each feature source.

[0086] The outputs of all feature sources are concatenated, integrated through a convolutional layer, and activated again by the GELU function to finally generate a high-quality cloud-reconstructed image.

[0087] Step three specifically includes:

[0088] Step 3.1: Randomly initialize the parameters of the cloud removal network that fuses optical and SAR images. Input the preprocessed training and validation data from Step 1 into the cloud removal network based on hybrid neural network that fuses optical and SAR images in Step 2 to generate a cloud removal effect image and calculate the loss.

[0089] Step 3.2: Backpropagate the loss and update the network parameters. Use minimizing the loss function as the optimization objective to obtain and save the optimal parameter model.

[0090] The loss function in step 3.2 combines Smooth L1 loss, multi-scale structural similarity index (SSIM) loss, and spectral angle matching (SAM) loss. In actual detection, Smooth L1 loss provides an optimization objective that is insensitive to outliers and has a stable gradient in regression tasks. SSIM loss can maintain the structural similarity between de-clouded and cloudless images, while SAM loss can effectively maintain and optimize the spectral feature fidelity of the image. The loss function is expressed as follows.

[0091] ;

[0092] ;

[0093] ;

[0094] In the formula It is the normalization coefficient, which averages the absolute errors of all pixels to obtain the mean of the element-wise absolute errors, where B, C, H, and W are the batch, the number of image channels, the number of vertical pixels, and the number of horizontal pixels, respectively. , and These are brightness similarity, contrast similarity, and structural similarity. and These are the highest-scale and scale-traversal variables, respectively. The weighting coefficients for the highest-scale brightness component. The weighting coefficients for the contrast component at the j-th scale are... denoted as the weighting coefficient of the structural component at scale j. It is the dot product operation of vectors, where It is a true optical image without clouds. It outputs a cloudless image, superscript. It is the transpose operator. are the Euclidean norms of T and P, respectively, used to normalize the vectors and ensure that the dot product falls within the interval [−1,1].

[0095] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0096] The present invention also provides a storage medium storing a computer program or instructions that, when the computer program or instructions are run on a computer, execute the steps of the method described.

[0097] Compared with the prior art, the present invention has the following beneficial effects:

[0098] This invention proposes a cloud removal method for remote sensing images based on heterogeneous parallel neural networks. It achieves accurate reconstruction of surface information by deeply fusing optical and SAR images. The method constructs a dual-stream coding architecture: a lightweight fusion residual block extracts geometric textures and edge details from the SAR image; an asymmetric feature pyramid module extracts orientation-aware multi-scale contextual features from the optical image; subsequently, a parallel gated attention module models and fuses the heterogeneous features, using a gating mechanism to establish long-range dependencies between cloud-covered and cloudless regions, achieving cross-modal feature complementarity; finally, a cloud mask-guided attention fusion module introduces cloud mask priors, providing dual guidance and refinement of features in both spatial and channel dimensions, reconstructing obscured information while suppressing cloud interference, thus achieving high-quality image restoration. Attached Figure Description

[0099] Figure 1A flowchart illustrating the cloud removal method based on hybrid neural network fusion of optical and SAR images provided by this invention;

[0100] Figure 2 The structure diagram of the cloud removal model based on hybrid neural network fusion of optical and SAR images provided by this invention;

[0101] Figure 3 This is a structural diagram of the lightweight fusion residual block provided by the present invention;

[0102] Figure 4 A structural diagram of the asymmetric feature pyramid module provided by this invention;

[0103] Figure 5 This is a structural diagram of the parallel gating attention module provided by the present invention;

[0104] Figure 6 This is a structural diagram of the cloud mask-guided attention fusion module provided by the present invention;

[0105] Figure 7 This is a schematic diagram of the cloud removal result of the cloud removal method based on hybrid neural network fusion of optical and SAR images provided by the present invention. Detailed Implementation

[0106] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0107] The cloud removal method based on hybrid neural networks fusing optical and SAR images provided in the embodiments, such as Figure 1 As shown, it includes the following steps:

[0108] Step 1: Obtain the SMILE-CR and SEN12MS-CR datasets. After preprocessing the datasets, divide them into training and validation sets according to the set ratio.

[0109] The steps in step one—obtaining the dataset, preprocessing the dataset, and partitioning the dataset—are as follows:

[0110] The datasets to be acquired are: the SMILE-CR dataset and the SEN12MS-CR dataset, which include SAR images, clouded images and cloudless images;

[0111] The dataset preprocessing is as follows: The original images in the SMILE-CR and SEN12MS-CR datasets are preprocessed. First, SAR images, cloudless optical images, and cloudy optical images are read, and the null values ​​in the data are replaced with the mean of each image. Then, the SAR images are cropped based on a preset threshold (-25dB to 0dB) and linearly normalized to the [0,1] interval. The optical images are cropped based on the reflectance range (0-10000) and normalized by dividing by 10000. Finally, a binary cloud mask for the cloudy images is generated.

[0112] The dataset was divided as follows: the enhanced SMILE-CR dataset and SEN12MS-CR dataset were randomly divided into two parts, with the training set accounting for 90% and the validation set accounting for 10%.

[0113] Step 2: Construct a cloud removal model based on a hybrid neural network that fuses optical and SAR images. The structure of this model is as follows: Figure 2 As shown, the input is a synthetic aperture radar image with cloud images and cloud masks, and the output is a de-clouded image. It mainly comprises four core components: a lightweight fusion residual block (LFR) responsible for processing SAR images, an asymmetric feature pyramid module (AFP) for extracting optical image features, a parallel gated attention module (PGAT) for deep fusion of heterogeneous features, and a cloud mask-guided attention fusion module (CGAF) that uses cloud mask information to specifically learn cloud-covered areas. The entire architecture achieves a de-clouding process from feature extraction and cross-modal fusion to accurate reconstruction through multi-stage processing. Represents SAR image features at different stages, where , The cloud image features represent different stages, among which , This represents the fusion characteristics at different stages, where... .

[0114] Step two involves building a cloud removal model based on a hybrid neural network that fuses optical and SAR images, employing a hierarchical encoding and decoding architecture. The model first extracts multi-scale geometric features from the SAR image and hierarchical spectral features from the clouded optical image through parallel branches. Then, at each level, a gated self-attention mechanism is used to achieve deep fusion of cross-modal features, and cross-level aggregation combines low-level details with high-level semantics. During the reconstruction phase, a cloud mask is introduced for multi-level spatial guidance, and a progressive decoder gradually restores the resolution and outputs a cloudless image. This process, through multi-scale supervision and composite loss function optimization, fully utilizes the cloud penetration capability of SAR to achieve accurate reconstruction of surface information in cloud areas while maintaining spectral consistency and spatial detail integrity.

[0115] Step two specifically includes:

[0116] Step 2.1: The preprocessed SAR image is input into the lightweight fusion residual block, by... Figure 3 As shown, the input features first enter the dynamic phantom convolution module. This module first compresses the channel dimension through 1×1 convolutional layers to reduce computational burden, followed by batch normalization and SiLU activation function for stable training. Then, 3×3 grouped convolution is used to group the feature channels for independent convolution to further reduce the number of parameters and computational cost, and batch normalization and SiLU activation function are applied again. The core of the data processing is a statistical channel selection mechanism: by calculating the global mean of each channel feature and multiplying it by a set of learnable weight parameters, a channel importance score is obtained; based on this score, the Top-K selection strategy is used to retain the most important feature channels, thereby achieving a significant reduction in the number of parameters while maintaining representational capability. The above process is shown in the following formula.

[0117] ;

[0118] ;

[0119] ;

[0120] ;

[0121] ;

[0122] In the formula This represents the preprocessed SAR image. and These represent the processed SAR image features. This represents the output of the dynamic phantom convolution module. express Activation function Indicates batch normalization, Represents a 1×1 convolution. This represents a 3×3 grouped convolution. Indicates the first The global mean of each channel, where B, H, and W represent the batch size, the number of vertical pixels in the feature map, and the number of horizontal pixels, respectively. Represents the input image. Channel importance score vector. From learnable weight parameters Channel statistical characteristics Multiplying them together, we get It is based on the calculated channel importance score. Sort all channels and select the K most important ones.

[0123] The features, processed by the above unit and batch-normalized again, are fed into an efficient channel attention unit. This unit uses adaptive one-dimensional convolution to generate independent weight coefficients for each channel, thereby amplifying the signal response of key channels and suppressing interference from unimportant or redundant channels, thus improving the overall discriminative power of the features. Subsequently, the feature stream is further deepened by two consecutive feature activation units, which consist of 3×3 convolutions, batch normalization, and the Gelu activation function. These units can extract spatial features, stabilize the training process, and introduce nonlinearity, thereby enhancing the model's expressive power. Afterward, the feature stream is processed in parallel by two branches: one branch captures significant local features through max pooling, and the other integrates the global context through average pooling. Both branches then sequentially pass through 1×1 convolutions for compressing the channel dimension, ReLU activation for introducing nonlinearity, and 1×1 convolutions for restoring the channel dimension. The outputs of the two branches are normalized by the Sigmoid function to generate two spatial attention weight maps, which are then multiplied element-wise with the original feature map to achieve adaptive weighting in the spatial domain.

[0124] Finally, the two weighted features are fused, and the fused features are added to the initial input of the module to form a residual structure. This result is finally output through the ReLU activation function. This design not only ensures the effective propagation of gradients in deep networks, but also completes cross-layer fusion of information from the bottom layer to the top layer, as shown in the following equation.

[0125] ;

[0126] ;

[0127] ;

[0128] ;

[0129] In the formula This represents the features of the SAR image after further processing. and These represent the output characteristics of the two pooling methods, This represents the output of the lightweight fused residual block. This represents the GELU activation function. This represents a high-efficiency channel attention unit. This represents a 3×3 convolution. express Activation function express Activation function and These represent max pooling and average pooling, respectively.

[0130] Step 2.2: The cloud-covered image after data preprocessing is input into the asymmetric feature pyramid module, where... Figure 4 As shown, the input features are first processed by an asymmetric convolutional layer. This layer utilizes two sets of structurally complementary decomposition convolutional kernels to extract features along the horizontal and vertical directions of the image, thereby enhancing the model's perception of directional structures (such as cloud boundaries and ground feature outlines). Specifically, through two complementary asymmetric convolutional branches, branch 1 first captures horizontal spatial features through a 1×3 convolution, followed by batch normalization and ReLU activation, then captures vertical spatial features through a 3×1 convolution, and finally performs another batch normalization. Branch 2 adopts the reverse order: first, it captures vertical spatial features through a 3×1 convolution, then performs batch normalization and ReLU activation, then captures horizontal spatial features through a 1×3 convolution, and finally performs another batch normalization. The two branches extract horizontal-vertical and vertical-horizontal cross-directional joint features, respectively, comprehensively covering spatial information in various directions.

[0131] The resulting two directional feature maps are concatenated by channels and then compressed by a 1×1 convolutional layer. After activation by the ReLU function, a preliminary nonlinear transformation and dimensionality reduction are achieved. The above process is shown in the following equation.

[0132] ;

[0133] ;

[0134] ;

[0135] In the formula This represents a preprocessed image with clouds. , and All of these indicate cloud image features after processing by different components. and This represents 1×3 convolution and 3×1 convolution. This indicates channel splicing.

[0136] Subsequently, the feature stream undergoes in-depth processing through two cascaded feature activation units to enhance its representational capabilities. Its output is fed into a multi-scale feature aggregation module, which employs a parallel architecture to capture multi-scale information: three branches use dilated convolutions with different dilation coefficients to obtain differentiated receptive fields while maintaining resolution, adapting to changes in cloud size; an additional branch aggregates local neighborhood information through average pooling followed by convolution. The outputs of each branch are processed by batch normalization and the GELU activation function, and then concatenated along the channel dimension. The concatenated multi-scale features are then processed through a 1×1 convolutional layer to achieve cross-scale information fusion and channel number regularization.

[0137] Finally, to preserve low-frequency information and promote training, the fused multi-scale features are added to the original input of the module through a residual connection. The result is then passed through the ReLU activation function to output an enhanced feature that simultaneously contains directional details and rich contextual information, forming a complete feature pyramid output. The above process is shown in the following equation.

[0138] ;

[0139] ;

[0140] ;

[0141] Concat , , , ;

[0142] ;

[0143] In the formula This indicates the cloud-bearing image features output by the component. This represents the output after multi-scale feature aggregation. This represents the output of the asymmetric feature pyramid module. It is a 3×3 dilated convolution. It is the expansion rate. The values ​​are 1, 2, and 3. These are the outputs of different branches.

[0144] Step 2.3: The feature images output from Steps 2.1 and 2.2 are passed through a gated self-attention module. Figure 5 As shown, this module first extracts local features using 3×3 convolutions and the ReLU activation function, introducing a nonlinear transformation. Then, it processes two information streams in parallel. One stream aggregates multi-scale features to capture multi-scale contextual information, while the other stream performs fine-grained attention modulation through gated self-attention. In the gated attention branch, the input features are preprocessed by 3×3 convolutions and then the attention features are calculated using a window attention mechanism. This feature is concatenated with the original input and fed into a gated unit consisting of 3×3 convolutions, ReLU, and the Sigmoid activation function, generating a pair of complementary spatial weight maps. and These two weight maps perform element-wise weighting on the original input and attention features, respectively. The weighted results are summed and then fused through a 3×3 convolution to obtain the attention-enhanced feature representation. This output is added to the output of the multi-scale feature aggregation to obtain a dual-branch output, which is then passed through a 3×3 convolutional layer, a ReLU activation function, and a random dropout layer for nonlinear transformation and regularization. This result is then processed by another 3×3 convolutional layer and added to the dual-branch output. The final output integrates deep features that combine attention guidance and multi-scale context, as shown in the following equation.

[0145] ;

[0146] ;

[0147] ;

[0148] ;

[0149] ;

[0150] In the formula This represents a multi-scale feature aggregation module. This represents the input to the parallel gated attention module. This represents the output after the input has undergone multi-scale feature aggregation. This indicates the output after window attention. This represents the output after gated self-attention. This represents the output of parallel gated attention. Indicates window attention. This indicates a splitting operation. This indicates a randomly discarded layer.

[0151] Step 2.4: The cloud mask-guided attention fusion module constructed in this invention aims to integrate multi-source features and accurately reconstruct the cloud-covered area, such as... Figure 6 As shown, this module first preprocesses the three output features from steps 2.1, 2.2, and 2.3, achieving channel alignment and non-linear enhancement through 1×1 convolution and the GELU activation function. The enhanced features are then processed by an efficient channel attention unit, which assigns weights to each channel through adaptive one-dimensional convolution to strengthen key feature channels and suppress redundant information.

[0152] Meanwhile, the input cloud mask first passes through a 3×3 convolutional layer to extract features, then a non-linearity is introduced through the GELU activation function, followed by another 3×3 convolution for further mapping, and finally a spatial weight map reflecting the probability distribution of the cloud region is generated using the Softmax function. During the feature fusion stage, this spatial weight map is multiplied element-wise with each path feature that has undergone channel weighting, thereby guiding the features in the spatial domain based on the prior information of the cloud mask.

[0153] Subsequently, a gating mechanism using a sigmoid function adaptively weights the product to further refine the contribution of each feature source. Finally, all weighted feature sources are concatenated, integrated through a 3×3 convolution, and activated by the GELU function to output a high-quality cloudless reconstructed image. The process is illustrated in the following equation.

[0154] ;

[0155] ;

[0156] ;

[0157] In the formula , and These are SAR features, fused features, and optical features. It is a spatial weight map generated based on the cloud mask. This indicates the final cloud-free image.

[0158] Step 3: Input the preprocessed training and validation dataset images into the cloud removal model of the fused optical and SAR images in Step 2 for training and validation, calculate the loss function and perform backpropagation, update the network parameters, and obtain the optimal parameter model.

[0159] Step three specifically includes:

[0160] Step 3.1: Randomly initialize the parameters of the cloud removal model that fuses optical and SAR images. Input the preprocessed training and validation data from Step 1 into the cloud removal model based on hybrid neural network fusion of optical and SAR images in Step 2 to generate a cloud removal effect image and calculate the loss.

[0161] Step 3.2: Backpropagate the loss and update the network parameters, such as the weights, biases, and parameters of the normalized layers. Minimize the loss function as the optimization objective to obtain and save the optimal parameter model.

[0162] The loss function in step 3.2 combines the absolute value loss (Smooth L1), the multi-scale structural similarity index (SSIM) loss, and the spectral angle matching (SAM) loss. In actual detection, the Smooth L1 loss provides an optimization objective that is insensitive to outliers and has a stable gradient in regression tasks. The multi-scale structural similarity index loss can maintain the structural similarity between cloud-free and cloud-removed images, while the spectral angle matching (SAM) loss can effectively maintain and optimize the spectral feature fidelity of the image. The loss function formula is as follows.

[0163] ;

[0164] ;

[0165] ;

[0166] In the formula This is the normalization coefficient, which averages the absolute errors of all pixels to obtain the mean of the element-wise absolute errors. Here, B, C, H, and W represent the batch size, the number of image channels, the number of vertical pixels, and the number of horizontal pixels, respectively. , and These are brightness similarity, contrast similarity, and structural similarity, among which... and These are the highest-scale and scale-traversal variables, respectively. The weighting coefficients for the highest-scale brightness component. Let be the weighting coefficient of the contrast component at the j-th scale. Here, represents the weighting coefficient of the j-th scale structural component. These weighting coefficients are set using the default values ​​from classic multi-scale structural similarity indices.

[0167] Specifically, regarding the total number of scales =5, and the weighting coefficients are set sequentially as follows: , , . It is the dot product operation of vectors, where It is a true optical image without clouds. It outputs a cloudless image, superscript. It is the transpose operator. are the Euclidean norms of T and P, respectively, used to normalize the vectors and ensure that the dot product falls within the interval [−1,1].

[0168] Step 4: Input the preprocessed test set into the optimal parameter model trained in Step 3, and output the cloud-removed image.

[0169] To verify the effectiveness of the proposed cloud removal method based on hybrid neural network fusion of optical and SAR images, this embodiment conducted experiments on two standard datasets, SEN12MS-CR and SMILE-CR. An NVIDIA RTX3090 GPU and PyTorch 1.12 framework were used, and the Adam optimizer was used for 200 training epochs. Peak signal-to-noise ratio (PSNR), structural similarity (SSIM), spectral angle map (SAM), and mean absolute error (MAE) were selected as quantitative evaluation metrics. A fair comparison was made with five current mainstream cloud removal models (SPA GAN, SAR2OPT, Dsen2-cr, GLF-CR, HPN-CR). In the table,  indicates the use of this data source,  indicates the absence of this data source, bold indicates the optimal metric, MSI represents optical images, and SAR represents synthetic aperture radar images.

[0170] Table 1 Comparison of cloud removal results of different models on the SEN12MS-CR dataset

[0171]

[0172] Table 2 Comparison of cloud removal results for different models on the SMILE_CR dataset

[0173]

[0174] The quantitative results are shown in Tables 1 and 2. Our method achieved optimal values ​​across all metrics on both test sets. This advantage stems from the collaborative innovation of our core modules: the LFR module extracts enhanced stable structural features from SAR images, providing crucial spatial priors for cloud area reconstruction and significantly improving spectral fidelity; the AFP module accurately captures the multi-scale context of clouds and ground features through asymmetric and multi-scale convolution, effectively balancing reconstruction accuracy and structural similarity; the PGAT module utilizes gated attention to establish global color associations, ensuring tonal consistency and suppressing artifacts; and the final CGAF module performs dynamic feature fusion based on cloud masks, achieving adaptive and refined reconstruction of cloud areas and clear regions. In summary, this model addresses key issues such as cross-modal feature extraction, multi-scale fusion, and global consistency, demonstrating its superiority in quantitative metrics.

[0175] The embodiments also provide the cloud removal results of this method and the comparison method, such as Figure 7 As shown, and analyzed, as follows:

[0176] SpA GAN relies solely on a single optical mode for cloud removal, lacking supplementary SAR information. This results in images with blurred details and difficulty in identifying land cover types. Under thick cloud cover, the lack of SAR structural information leads to inaccurate estimation of obscured areas, causing widespread blurring in the reconstructed image. SAR2OPT, lacking cloud-containing optical input, relies excessively on SAR imagery, causing spectral distortion. Furthermore, DSen2-CR improves performance by capturing SAR structural information using residual blocks, but its reconstructed images are prone to artifacts and cluttered details. GLF-CR, drawing inspiration from the Transformer, alleviates spectral distortion and cluttered details by balancing global and local fusion, but the generated image still exhibits some blurring. HPN-CR generates images with significant spectral differences, making it difficult to restore surface colors in thinly clouded areas. In contrast, the cloud-removed images from this invention retain superior texture details in cloudless areas, exhibit fewer false details and blur artifacts in cloud-covered scenes, and generate more realistic image structures, recovering more details and spectral information.

[0177] This invention provides a cloud removal method based on hybrid neural networks fusing optical and SAR images. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A cloud removal method based on hybrid neural network fusion of optical and SAR images, characterized in that, Includes the following steps: Step 1: Acquire SAR images and cloud-covered images of the corresponding areas, and preprocess them to divide them into training, validation, and test sets; Step 2: Construct a hybrid neural network that fuses cloud-covered images and SAR images; the hybrid neural network includes a SAR branch, a fusion branch, and an optical branch; The hybrid neural network adopts a multi-level progressive fusion mechanism, wherein the first parallel gated attention module of the fusion branch receives the preprocessed SAR image and the cloud image and performs initial cross-modal fusion. The subsequent parallel gated attention modules at each level receive the geometric texture features extracted from the SAR branch by the lightweight fusion residual block of the previous level, the fusion features output by the parallel gated attention module, and the multi-scale contextual features extracted from the optical branch by the asymmetric feature pyramid module, thereby achieving cross-modal feature complementarity. After the highest level of the network, the cloud mask-guided attention fusion module receives the outputs of the highest level parallel gated attention module, lightweight fusion residual block and asymmetric feature pyramid module, and introduces cloud mask prior to guide the features of each path in both spatial and channel dimensions, thereby outputting a cloud-free image. Step 3: Input the training set and validation set from Step 1 into the hybrid neural network in Step 2 for training and validation, calculate the loss function and perform backpropagation, update the network parameters, and obtain the parameters to update the model; Step 4: Input the test set from Step 1 into the parameter update model trained in Step 3, and output the cloud-removed image.

2. The cloud removal method based on hybrid neural network fusion of optical and SAR images according to claim 1, characterized in that, The lightweight fusion residual block in step 2 includes: a dynamic phantom convolution module, an efficient channel attention module, a feature activation unit, a dual-channel pooling layer, a spatial weighting function, and a ReLU activation function; The specific implementation process of the lightweight fusion residual block is as follows: The input features are first processed by the dynamic phantom convolution module, and lightweight feature selection and compression are achieved through 1×1 convolution channel dimensionality reduction, batch normalization, SiLU activation function and sorting and filtering mechanism based on global mean and learnable parameters. Then, the output features of the dynamic phantom convolution module are input into the efficient channel attention module, which assigns weights to each channel through adaptive one-dimensional convolution to enhance the signal response of key channels and suppress redundant information. Next, the output features of the high-efficiency channel attention module are further processed through successive feature activation units; The processed features are processed by dual pooling layers to extract different spatial context information through max pooling and average pooling respectively. An attention map is generated by a spatial weighting function and fused and weighted with the input features of the dual pooling layers. Finally, the geometric texture features are output through residual connections and ReLU activation function.

3. The cloud removal method based on hybrid neural network fusion of optical and SAR images according to claim 1, characterized in that, The asymmetric feature pyramid module in step 2 includes: asymmetric convolution, decomposed convolution, 1×1 convolution, ReLU activation function, feature activation unit, multi-scale feature aggregation module, dilated convolution, average pooling, batch normalization, and GELU activation function; The specific implementation process of the asymmetric feature pyramid module is as follows: First, features in the horizontal and vertical directions are extracted by two sets of decomposition convolutions in asymmetric convolution to enhance the perception of the geometric structure of clouds and ground features in different directions. Next, the horizontal and vertical features are concatenated, and channel compression and nonlinear activation are performed using 1×1 convolution and the ReLU function. Subsequently, the features are enhanced by two cascaded feature activation units and then sent to the multi-scale feature aggregation module for processing. The multi-scale feature aggregation module adopts a multi-branch parallel structure, in which three branches use dilated convolutions with different dilation rates to expand the receptive field, and another branch fuses local contextual information through an average pooling-convolution structure. The outputs of each branch are concatenated after batch normalization and the GELU function, and then dimensionality reduction and fusion are achieved through 1×1 convolution to obtain the enhanced multi-scale features. Finally, the enhanced multi-scale features are connected via residual connections and the ReLU activation function to obtain multi-scale contextual features.

4. The cloud removal method based on hybrid neural network fusion of optical and SAR images according to claim 1, characterized in that, The parallel gated attention module in step 2 includes: convolutional layers, window attention mechanism, gating mechanism, sigmoid activation function, ReLU activation function, and random dropout layer; The specific implementation process of the parallel gated attention module is as follows: First, the multi-source features are merged and then preliminary feature processing is performed using convolution and ReLU activation functions; Then, the parallel gated attention module performs two-branch processing on the processed features. One branch uses the multi-scale feature aggregation module to extract multi-scale features; the other branch uses a gated self-attention branch to model the correlation between regions to enhance feature representation. In the gated self-attention branch, the input features are first preprocessed by a convolutional layer, and then attention features are obtained through a window attention mechanism. Subsequently, the attention features are concatenated with the input features, and the concatenated features are processed by a gating mechanism. The gating mechanism generates a pair of complementary attention weight maps through convolution and the sigmoid activation function, and then the weights are weighted element-wise with the input features and the attention features respectively and summed. Finally, the weights are integrated by a convolutional layer to output the enhanced feature representation. Finally, the summation results of the two branches are successively transformed and regularized through convolutional layers, ReLU activation functions, and random dropout layers. After further processing by convolutional layers, they are added to the summation results to output the fused features.

5. The cloud removal method based on hybrid neural network fusion of optical and SAR images according to claim 1, characterized in that, The cloud mask-guided attention fusion module in step 2 includes: 1×1 convolution, GELU activation function, efficient channel attention module, independent convolution module, softmax function, sigmoid gating mechanism, and splicing fusion layer; The specific implementation process of the cloud mask-guided attention fusion module is as follows: First, the highest-level fusion features, geometric texture features, and multi-scale context features are each channel aligned and enhanced by 1×1 convolution and GELU activation function. The enhanced features are then fed into the efficient channel attention module, where weights are assigned to each channel by one-dimensional convolution to obtain the weighted features. Then, the input cloud mask is processed by an independent convolutional module to extract deep features, and then a spatial weight map reflecting the probability distribution of cloud coverage is generated by the Softmax function. Next, in the fusion stage, the spatial weight map is multiplied element-wise with each weighted feature to achieve spatial guidance; then, the multiplication result is adaptively weighted through the Sigmoid gating mechanism to obtain the final attention output of each feature source. Finally, the final attention outputs of all feature sources are concatenated, integrated through a convolutional layer, and processed again by the GELU activation function to generate a cloud-reconstructed image.

6. The cloud removal method based on hybrid neural network fusion of optical and SAR images according to claim 1, characterized in that, The loss function in step 3 combines the absolute value loss function, the multi-scale structural similarity exponential loss function, and the spectral angle matching loss function to optimize pixel-level error, structural similarity, and spectral fidelity. Among them, the absolute value loss function Multi-scale structural similarity index loss function Spectral angle matching loss function They are represented as follows: ， ， ， Where B, C, H, and W represent the batch size, number of channels, vertical pixel count, and horizontal pixel count of the image, respectively; M and j are the highest scale and scale traversal variables, respectively. , and These are brightness similarity, contrast similarity, and structural similarity. The weighting coefficients for the highest-scale brightness component. The weighting coefficients for the contrast component at the j-th scale are... The weighting coefficients for the j-th scale structural components; superscript It is the transpose operator; and These are real optical cloudless images. Output cloud-removed image The Euclidean norm, It is an inverse cosine function.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the cloud removal method based on hybrid neural network fusion of optical and SAR images as described in any one of claims 1 to 6.

8. A storage medium, characterized in that, The system contains a computer program or instructions that, when executed on a computer, perform the steps of the cloud removal method based on hybrid neural network fusion of optical and SAR images as described in any one of claims 1 to 6.

Images