SAR image-assisted optical remote sensing image cloud removal method and device based on Mama model

Abstract

The invention discloses an SAR image auxiliary optical remote sensing image cloud removal method and device based on a Mama model. The method comprises the following steps: acquiring an optical remote sensing image polluted by cloud and a corresponding auxiliary SAR image; respectively extracting shallow layer features through a shallow layer feature extraction module; the shallow layer features are input into a Modal-Traversing-SS2D fusion module, and deep interaction and fusion of the features of the optical image and the SAR image are realized through cross scanning and modal separation; inputting the fusion features into N multi-scale multi-branch Mama modules which are connected in series to generate advanced features; and mapping the advanced features to the size of the optical remote sensing image through an image reconstruction module, and outputting a cloudless optical remote sensing image in combination with long jump connection. According to the method, the Mama model is introduced into a remote sensing image thick cloud removal task, and the global dependency relationship of the image is effectively modeled; and the global and local geographic proximity in the remote sensing image is fully utilized. And cross-modal feature cross scanning and interaction are realized, and an efficient and accurate solution is provided for optical remote sensing image cloud removal.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and more particularly to a SAR image-assisted optical remote sensing image cloud removal method and device based on a Mamba model. Background Art

[0002] Optical remote sensing images, due to their high spatial resolution and rich spectral information, are widely used in change detection, disaster monitoring, and hyperspectral image classification. However, optical remote sensing images use a passive imaging mode, capturing light reflected from ground objects by the sensor. These images are easily obscured by thick clouds and their shadows, resulting in information degradation or loss, greatly complicating subsequent application tasks. According to global cloud cover data provided by the International Satellite Cloud Climate Program (ISCCP), 63% of the Earth's surface is covered by clouds. The widespread presence of large-scale cloud cover in optical remote sensing images has a significant negative impact on their subsequent processing and application. Therefore, cloud removal methods are essential for reconstructing cloud-free remote sensing images, especially for researchers and projects that require continuous observation of optical remote sensing imagery time series. Because thick clouds severely block the propagation of light reflected from ground objects, ground object information in optical remote sensing images is almost completely lost in areas covered by thick clouds. However, microwaves have the ability to penetrate clouds, allowing SAR (SAR) images to provide accurate ground object information even under thick cloud obstruction. Therefore, using SAR images as auxiliary data to remove thick clouds from optical remote sensing images has been a hot topic and a challenge.

[0003] Early SAR image-assisted optical remote sensing image declouding methods achieved thick cloud removal through methods such as pixel replacement, dictionary learning, and sparse representation. With the development of deep learning technology, some researchers have begun to use convolutional neural networks to remove thick clouds from optical remote sensing images based on SAR images, and have achieved even better results. However, convolutional neural networks are limited by the size of the convolution kernel and lack global modeling capabilities. When the cloud coverage in optical remote sensing images is high and the scene is complex, it is difficult to achieve excellent results. The rapid development of visual transformers has provided a new solution to this problem. Specifically, the visual transformer can effectively model the relationship between any region and the entire image through the self-attention mechanism, thereby effectively solving the problem that CNN networks have difficulty modeling global dependencies. Currently, an increasing number of methods are introducing transformer models in the task of thick cloud removal.

[0004] Although the aforementioned methods have achieved excellent performance in declouding optical remote sensing images, they still have the following three shortcomings.

[0005] First, thick cloud cover can cause partial loss of ground object information, making it difficult to effectively repair occluded areas solely based on local information. A global receptive field (GRF) can leverage contextual information from the entire image to fill in the missing information, helping to maintain overall image consistency. However, existing methods struggle to effectively achieve a GRF.

[0006] Second, remote sensing images contain objects of various sizes, and areas covered by thick clouds also vary in size. Therefore, it is important to obtain features of different scales and use them for image declouding. Current methods lack the full utilization of multi-scale features.

[0007] Third, due to the significant differences between optical remote sensing images and SAR images, deep fusion of the image features of the two modalities is necessary to fully leverage the complementarity of the two data types and improve cloud removal. Current methods typically concatenate the two modal images at the input and feed them into the network, or simply perform feature fusion by adding, subtracting, or concatenating the features of the two modalities. However, there is a lack of effective methods for deep fusion of the two modalities. Summary of the Invention

[0008] In view of this, the present invention provides a SAR image-assisted optical remote sensing image declouding method and device based on the Mamba model, which can at least partially solve the problems that the above-mentioned declouding methods are difficult to effectively achieve a global receptive field, fail to fully utilize multi-scale features, and lack means to deeply fuse the features of the two modalities.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] In a first aspect, an embodiment of the present invention provides a SAR image-assisted optical remote sensing image declouding method based on a Mamba model, comprising the following steps:

[0011] S10, obtaining an optical remote sensing image contaminated by clouds and its corresponding auxiliary SAR image;

[0012] S20, respectively extracting shallow features of the optical remote sensing image and the SAR image through a shallow feature extraction module;

[0013] S30, inputting the shallow features into the Modal-Traversing-SS2D fusion module, achieving deep interaction and fusion of optical image and SAR image features through cross scanning and modal separation, and generating fusion features;

[0014] S40, inputting the fused features into N serially connected multi-scale multi-branch Mamba modules to generate high-level features;

[0015] S50 , mapping the high-level features to the size of the optical remote sensing image through an image reconstruction module, and outputting a cloud-free optical remote sensing image in combination with a long skip connection.

[0016] Furthermore, in step S30, the Modal-Traversing-SS2D fusion module includes the following operations:

[0017] (a) The shallow features of a set of input optical remote sensing images and SAR images are expanded into a one-dimensional feature sequence along four directions respectively, and rearranged in a modal crossover manner to generate a mixed feature sequence;

[0018] (b) processing the generated mixed feature sequence through four parallel S6Blocks, and outputting the processed feature sequence;

[0019] (c) The processed feature sequence is separated by modality and then merged into two-dimensional features according to the expansion direction through the Cross Merge module; the merged features are subjected to channel splicing and 1×1 convolution to complete cross-modal feature aggregation.

[0020] Furthermore, the Modal-Traversing-SS2D fusion module processing formula is expressed as follows:

[0021]

[0022] Among them, the input features and are optical image features and SAR image features, respectively; t represents the modality type, which can be either opt or sar; LN(·) represents the layer normalization function, Linear(·) represents the linear layer, DWC(·) represents the depthwise convolution, MTSS2D(·) represents the Modal-Traversing SS2D, and AGG(·) represents feature aggregation, which consists of a Concat layer for feature concatenation on the channel and a 1×1 convolution layer for feature interaction; the superscripts 1 / 2 / 3 / 4 / 5 correspond to the output results of the respective stages.

[0023] Furthermore, in step S40, the multi-scale multi-branch Mamba module includes a multi-scale image encoder and a multi-branch RSSG module;

[0024] The multi-scale image encoder extracts multi-scale features of input features by setting up depth-wise separable convolutions with different dilation rates in parallel;

[0025] The multi-branch RSSG module performs global dependency modeling on the multi-scale features and combines convolution branches to extract local features to generate high-level features.

[0026] Furthermore, the depthwise separable convolution with different dilation rates includes a 3×3 depthwise dilated convolution and a 1×1 pointwise convolution; the formula is expressed as follows:

[0027] F r =DSDC r (F in ),r=1,2,3(2)

[0028] Among them, DSDC r represents a depth-wise separable convolution with a dilation rate of r, F r Represents the image features of the corresponding convolution output.

[0029] Furthermore, the convolution branch combined in the multi-branch RSSG module consists of a 1×1 convolution, a 3×3 depthwise convolution and a residual connection; the formula is expressed as follows:

[0030] G0=ConvLF(F1)

[0031] G r =RSSG(F r ),r=1,2,3(3)

[0032] F o =AGG(G0,G1,G2,G3)

[0033] Among them, ConvLF(·) represents the convolution branch, represents the output of each branch when the expansion rate r = 1, 2, 3, RSSG(·) represents the multi-branch RSSG module, AGG(·) represents feature aggregation, It represents the final output feature and serves as the input of the multi-branch RSSG module in the next stage.

[0034] In a second aspect, an embodiment of the present invention further provides a SAR image-assisted optical remote sensing image declouding device based on a Mamba model, using the SAR image-assisted optical remote sensing image declouding method based on a Mamba model as described in any one of the first aspects, the device comprising:

[0035] Input module, used to obtain cloud-contaminated optical remote sensing images and their corresponding auxiliary SAR images;

[0036] A shallow feature extraction module, used to extract shallow features of the optical remote sensing image and the SAR image respectively;

[0037] Modal-Traversing-SS2D fusion module, used to achieve deep interaction and fusion of optical image and SAR image features through cross-scanning and modal separation of the shallow features to generate fused features;

[0038] A multi-scale multi-branch Mamba module, configured to input the fused features into N multi-scale multi-branch Mamba modules connected in series to generate high-level features;

[0039] An image reconstruction module is used to map the high-level features to the size of the optical remote sensing image and output a cloud-free optical remote sensing image in combination with a long skip connection.

[0040] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0041] Memory for storing computer programs;

[0042] The processor is configured to implement the Mamba model-based SAR image-assisted optical remote sensing image declouding method as described in any one of the first aspects when executing the program stored in the memory.

[0043] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the SAR image-assisted optical remote sensing image declouding method based on the Mamba model as described in any one of the first aspects.

[0044] The description of the second to third aspects of the present invention can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second to third aspects can refer to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0045] It can be seen from the above technical solutions that compared with the prior art, the present invention has the following technical advantages:

[0046] 1. The Mamba model is introduced into the field of thick cloud removal in remote sensing images, so that the global information of the image can be extracted efficiently.

[0047] 2. A multi-scale multi-branch Mamba model is proposed. The multi-scale features of the image are extracted through a multi-scale image encoder and combined with the multi-branch Mamba to simultaneously represent fine and coarse features. At the same time, it is combined with the convolution branch to fully utilize the global and local geographic proximity that is widely present in remote sensing images.

[0048] 3. A Modal-Traversing-SS2D FusionModule is proposed. By cross-scanning the optical image features and SAR image features, the deep fusion of the two modal features is achieved, thereby making full use of the complementarity of the two modal data and improving the declouding effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0050] Figure 1 This is a flow chart of the SAR image-assisted optical remote sensing image declouding method based on the Mamba model provided by the present invention.

[0051] Figure 2 This is a diagram showing the overall principle structure of the SAR image-assisted optical remote sensing image declouding method based on the Mamba model provided by the present invention.

[0052] Figure 3 This is the Modal-Traversing SS2D structure diagram provided by the present invention.

[0053] Figure 4 This is a visualization diagram of the results of different methods provided by the present invention on the SMILE-CR dataset.

[0054] Figure 5 This is a visualization result diagram of different methods provided by the present invention on the SEN12MS-CR dataset.

[0055] Figure 6 This is a block diagram of the SAR image-assisted optical remote sensing image declouding device based on the Mamba model provided by the present invention.

[0056] Figure 7 This is a structural block diagram of the computer device provided by the present invention. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] Example 1:

[0059] The embodiment of the present invention discloses a SAR image-assisted optical remote sensing image declouding method based on the Mamba model, referring to Figure 1 As shown, the following steps are included:

[0060] S10, obtaining an optical remote sensing image contaminated by clouds and its corresponding auxiliary SAR image;

[0061] S20, respectively extracting shallow features of the optical remote sensing image and the SAR image through a shallow feature extraction module;

[0062] S30, inputting the shallow features into the Modal-Traversing-SS2D fusion module, achieving deep interaction and fusion of optical image and SAR image features through cross scanning and modal separation, and generating fusion features;

[0063] S40, inputting the fused features into N serially connected multi-scale multi-branch Mamba modules to generate high-level features;

[0064] S50. Mapping the high-level features to the size of the optical remote sensing image through an image reconstruction module, and combining long skip connections to output a cloud-free optical remote sensing image. The high-level features are gradually restored to the original optical image size through upsampling or deconvolution layers. Combined with long skip connections, the original input optical image (or shallow features) are directly added to the reconstruction result. This preserves the original pixel information in cloud-free areas, preventing network errors and improving color consistency and visual quality (e.g., vegetation areas not obscured by clouds retain their true tones).

[0065] like Figure 1 As shown, first, in step S10, an optical remote sensing image contaminated by clouds is received as input. It includes areas obscured by thick clouds and receives auxiliary SAR images R represents a set of real numbers, the subscript opt represents an optical remote sensing image, and sar represents a SAR image. C represents the number of channels, H represents the height, and W represents the width. The SAR image is SAR data of the same area as the optical remote sensing image and provides ground feature information that penetrates the clouds.

[0066] Secondly, in step S20, the shallow feature extraction module (SFE) contains two branches, each of which is composed of a connected 3×3 convolution; shallow feature extraction is performed on the optical remote sensing image and the SAR image respectively, and the shallow features of the two modalities are extracted by SFE. and Among them C s Indicates the number of channels of extracted shallow features; by preliminarily extracting basic features such as edges and textures, the computational complexity of subsequent modules is reduced.

[0067] Then, in step S30, and It is sent to the Modal-Traversing-SS2D fusion module to achieve full interaction and fusion of the two modal features and obtain the fusion feature

[0068] Next, in step S40, the fusion feature F fusion Input N multi-scale multi-branch Mamba modules to perform advanced features Extraction.

[0069] Finally, in step S50, the reconstruction module maps the high-level features to the size of the input optical remote sensing image, completing thick cloud removal. Furthermore, to effectively convey information about cloud-free areas, the input optical image is added to the network output via a long-skip connection to produce the final cloud-free optical image.

[0070] The present invention solves the three major challenges of cloud removal in optical remote sensing images (global modeling, multi-scale features, and modal fusion), and provides an efficient, accurate, and scalable technical solution for thick cloud removal.

[0071] Next, the present invention will introduce step S30 and step S40 in detail:

[0072] In step S30, the Modal-Traversing-SS2D fusion module is introduced to achieve deep fusion of the two modal features by cross-scanning the optical image features and the SAR image features, thereby making full use of the complementarity of the two modal data and improving the declouding effect of the model.

[0073] (1) Modal-Traversing-SS2D fusion module

[0074] Due to the different imaging mechanisms, there are large differences between SAR images and visible light images. Therefore, it is necessary to fully interact and fuse the images of the two modalities in order to fully utilize the complementarity of the two data and improve the cloud removal effect. Most of the current methods use simple addition and subtraction or channel splicing to perform fusion, and lack effective modal fusion methods. In order to solve the above problems, the present invention proposes a Modal-Traversing-SS2D fusion module based on the design concept of the visual state space model. Through Modal Traversing Cross-scan and S6Block, while achieving the global receptive field, the deep interactive fusion of the two modal features is realized. The specific structure is as follows: Figure 2 shown.

[0075] Assumptions and are optical image features and SAR image features respectively, then the process can be:

[0076]

[0077] Where t represents the modality type, which can be either opt or sar. LN(·) represents the layer normalization function, Linear(·) represents the linear layer, DWC(·) represents the depthwise convolution, MTSS2D(·) represents the Modal-Traversing SS2D, and AGG(·) represents the feature aggregation module, which consists of a Concat layer for feature concatenation on the channel and a 1×1 convolution layer for feature interaction. The superscripts 1 / 2 / 3 / 4 / 5 represent the output results of the corresponding stage.

[0078] like Figure 3 As shown in the figure, specifically, the Modal-Traversing SS2D module consists of three parts: Modal Traversing Cross-scan, S6Block, and Cross Merge.

[0079] For the input optical image feature F opt and SAR image features F sar First, the two-dimensional image features are expanded into four one-dimensional feature sequences along the four directions from upper left to lower right, lower right to upper left, upper right to lower left, and lower left to upper right, and rearranged in a modal crossover manner, such as alternating arrangement of "light-SAR-light-SAR".

[0080] Example: Optical feature row sequence → SAR feature row sequence → Optical feature column sequence

[0081] →Column sequence of SAR features.

[0082] Next, these one-dimensional feature sequences are processed by four parallel S6Blocks. Four sets of parallel S6Blocks (based on the state space model) perform global dependency modeling on the mixed sequence to capture cross-modal long-range correlations. The output feature sequences are then separated by modality and reshaped back to the original size (two-dimensional features) according to the expansion direction through the Cross Merge module. The outputs in the four directions are then summed and merged. Finally, the four output results are merged by summing to obtain the final output. Assume x(t)∈R L Represents the input sequence, y(t)∈R L Represents the output sequence, h(t)∈R L represents the hidden state, A∈R L×L Represents the state transfer matrix, B∈R L×1 Denotes the input projection matrix, C∈R 1×L Represents the observation matrix, then S6Block can be mathematically described by the following equation:

[0083] h'(t)=Az(t)+Bx(t)

[0084] y(t)=Ch(t)

[0085] In order to apply the above model to the field of deep learning, it needs to be discretized. This process usually adopts the Zero-Order Hold (ZOH) rule, assuming that Δ represents the time scale sampling parameter, and Represents the discretized matrix, then the discretized S6Block can be expressed as:

[0086]

[0087] y t =Hh t

[0088] Through Modal-Traversing SS2D, the features at each position in each modality can not only integrate the information of the features at other positions of the modality, but also integrate the feature information of all positions of the other modality from different directions, thus achieving full interaction and fusion of the features of the two modalities.

[0089] The Modal-Traversing SS2D module breaks through the limitations of traditional stitching or addition and subtraction fusion through cross-scanning and modal separation mechanisms.

[0090] In step S40, a multi-scale multi-branch Mamba model is proposed, which extracts multi-scale features of the image through a multi-scale image encoder and combines it with the multi-branch Mamba to simultaneously represent fine features and coarse features. At the same time, it is combined with the convolution branch to fully utilize the global and local geographic proximity that is widely present in remote sensing images.

[0091] (2) Multi-scale and multi-branch Mamba module

[0092] Thick cloud cover will cause some ground object information to be missing, and it is difficult to effectively repair the occluded area by relying solely on local information. The global receptive field can use the contextual information of the entire image to fill in the missing parts, which helps to maintain the overall consistency of the image. In addition, since the sizes of ground objects in remote sensing images vary, the sizes of thick cloud coverage areas are also different. Therefore, extracting and utilizing the multi-scale features of the image is crucial for thick cloud removal in optical remote sensing images. However, existing methods are generally difficult to achieve a global receptive field and lack full utilization of the multi-scale features of the image. In order to solve these problems, the present invention combines the residual state space group (RSSG) module in MambaIR to propose a multi-scale multi-branch Mamba module, which extracts and utilizes the multi-scale features of the image while achieving a global receptive field. The module consists of two parts: a multi-scale image coding module and a multi-branch RSSG module. The specific structure is as follows Figure 2Next, we will introduce these two modules in detail.

[0093] 2.1. Multi-scale image coding module: Through parallel setting of depth-wise separable convolutions with different dilation rates, multi-scale features of the input features are extracted.

[0094] In this invention, a multi-scale image coding module is introduced to extract fine-grained and coarse-grained image features at the same feature level. In addition, the proposed method adopts a multi-branch structure, which will introduce more image coding modules. In order to reduce the parameters and computational burden of the model, the present invention uses multiple parallel depth-separable convolutions with different dilation rates, each of which contains a 3×3 depth-dilated convolution and a 1×1 point convolution. Assume that the input image feature is The process can be expressed by the following formula:

[0095] F r =DSDC r (F in ),r=1,2,3(2)

[0096] Among them, DSDC r represents a depth-wise separable convolution with a dilation rate of r, F r Represents the image features of the corresponding convolution output. In this embodiment, r takes values of 1, 2, or 3. For example: Fine-grained features (r = 1) capture local details (such as road texture). Medium-scale features (r = 2) extract medium-range structures (such as building outlines). Coarse-grained features (r = 3) model global context (such as mountain distribution).

[0097] Since convolutions with different dilation rates have different effective receptive fields, the outputs F1, F2, and F3 of this layer simultaneously extract features of different scales of the input image. Finally, these image features are sent to different multi-branch RSSG modules.

[0098] 2.2. Multi-branch RSSG module: It performs global dependency modeling on multi-scale features and combines convolution branches to extract local features to generate high-level features.

[0099] Most current cloud removal methods concatenate multi-scale features in the channel dimension, failing to fully utilize these features. Considering the excellent performance of RSSG in image restoration tasks, this example uses RSSG as a foundational module to construct a multi-branch Mamba layer. Features at different scales are fed into RSSG for global relationship modeling. The generated features are then fused through a feature aggregation module, thereby simultaneously representing both coarse-grained and fine-grained image features at the same feature level.

[0100] Although Mamba can model long-range dependencies and effectively capture the global context of an image, it often ignores local information of the image. Convolutional networks, due to their weight sharing mechanism, can effectively capture local information. Therefore, in order to make up for the shortcomings of RSSG in extracting local information, the present invention introduces an additional convolution branch in the multi-branch RSSG to extract local information. The convolution branch consists of two 1×1 convolutions, a 3×3 depthwise convolution, and a residual connection. The specific structure is as follows: Figure 2 Assume r = 1, 2, 3 represents the image features of different scales output by the multi-scale image coding module, and the multi-branch RSSG module can be expressed as:

[0101] G0=ConvLF(F1)

[0102] G r =RSSG(F r ),r=1,2,3(3)

[0103] F o =AGG(G0,G1,G2,G3)

[0104] Among them, ConvLF(·) represents the convolution branch, Represents the output results of each branch corresponding to the expansion rate r = 1, 2, 3, The final output features are used as input to the next stage multi-branch RSSG module. The multi-branch RSSG module takes into account both global context and local details to adapt to the restoration needs of different cloud coverage scales.

[0105] The present invention uses two representative remote sensing image declouding datasets, SMILE-CR and SEN12MS-CR, for experiments. Mean Absolute Error (MAE), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measurement (SSIM), and Spectral Angle Mapper (SAM) are used as evaluation indicators. The technical effects of the present invention are as follows:

[0106] Quantitative experiments:

[0107] Table 1 shows the experimental results of all methods on the simulated dataset SMILE-CR and the real dataset SEN12MS-CR. The best results are indicated in bold. The table shows that Transformer-based or Mamba-based methods outperform CNN-based methods, largely due to their stronger ability to model long-range dependencies. Due to the linear complexity of Mamba, when the number of parameters is similar, the computational cost of Mamba-based methods is significantly lower than that of Transformer-based methods. For example, the number of parameters of the MambaIR method is 73.3% of that of CRFormer (3.52 divided by 4.8), but its computational cost is only 24.9% of that of CRFomer (43.45 divided by 174.1).

[0108] Furthermore, our method achieved the best experimental results on both datasets. Specifically, on the SMILE-CR dataset, our method achieved the best results in terms of MAE, PSNR, and SSIM. While our method slightly outperformed MambaIR in terms of SAM, it outperformed MambaIR in other metrics, such as MAE and PSNR. This demonstrates that our method reconstructs cloud-free optical images with higher accuracy. On the SEN12MS-CR dataset, our method achieved the best results in terms of MAE, SAM, and SSIM. While slightly inferior to CRFomer in terms of PSNR, it outperformed CRFomer in metrics such as SAM and SSIM, demonstrating that our method restores superior cloud-free optical images. Finally, the table also shows that despite our method having more parameters and computational overhead than SpGAN, it significantly outperforms SpGAN. Compared to other methods, our method not only outperforms them overall but also requires fewer parameters and computational overhead, fully demonstrating its high efficiency.

[0109] Table 1 Comparison results of different methods on two datasets.

[0110]

[0111]

[0112] Note: The calculation amount and parameter quantities in the table are calculated based on the input image resolution of 128×128.

[0113] Qualitative experiments:

[0114] Figure 4 and Figure 5The visualization results of different methods on the Smile-CR and SEN12MS-CR datasets are shown for a more intuitive comparison. Among them, (a) is a remote sensing image with clouds, (b) a Sar image, (c) the de-clouding effect of SpGAN, (d) the de-clouding effect of DSen2-CR, (e) the de-clouding effect of USSRN-CR, (f) the de-clouding effect of CRFormer, (g) the de-clouding effect of MambaIR, (h) the de-clouding effect of MSMBMa-CR of the present invention, and (g) the de-clouding effect of GT.

[0115] As can be seen from the figure, the SpGAN method performs significantly worse than other methods. This may be due to the weak constraints on SpGAN's model output, resulting in a significant color deviation between the declouded image and the true image. Furthermore, when cloud cover is low, declouding is relatively easy. Methods like CRFormer and MambaIR achieve good color consistency with the true value and provide superior visual quality. However, as cloud cover increases, removing thick clouds becomes increasingly difficult, and the color consistency and visual quality of these methods decline.

[0116] In comparison, our proposed method (MSMBMa-CR) achieved the lowest MAE (0.011), the highest PSNR (41.04), and the best SSIM (0.9277) on the SMILE-CR dataset, demonstrating its optimal cloud removal performance. Its parameter count (1.14M) and computational complexity (15.83G) were significantly lower than those of other methods, demonstrating its high efficiency. Our proposed method achieved optimal cloud removal results, achieving the best PSNR and SSIM metrics, as well as visual quality, under varying cloud cover conditions, fully demonstrating its effectiveness.

[0117] The Mamba model-based SAR image-assisted optical remote sensing image declouding method proposed in this paper introduces the Mamba model into the task of thick cloud removal from remote sensing images, effectively modeling the global dependencies of the image. Furthermore, a multi-scale, multi-branch structure is proposed that can simultaneously characterize both coarse and fine features of the image. Combined with convolutional branches, it fully utilizes the global and local geographic proximity in remote sensing images. Furthermore, a Modal-Traversing-SS2D fusion module is designed to achieve deep interaction and fusion of the two modal features through cross-scanning. Experimental results demonstrate that the proposed method outperforms the most advanced declouding methods, while requiring fewer parameters and requiring less computation.

[0118] Example 2:

[0119] Based on the same inventive concept, the present invention also provides a SAR image-assisted optical remote sensing image declouding device based on the Mamba model, using the SAR image-assisted optical remote sensing image declouding method based on the Mamba model as described in Example 1 above, referring to Figure 6 As shown, the device includes:

[0120] Input module, used to obtain cloud-contaminated optical remote sensing images and their corresponding auxiliary SAR images;

[0121] A shallow feature extraction module, used to extract shallow features of the optical remote sensing image and the SAR image respectively;

[0122] Modal-Traversing-SS2D fusion module, used to achieve deep interaction and fusion of optical image and SAR image features through cross-scanning and modal separation of the shallow features to generate fused features;

[0123] A multi-scale multi-branch Mamba module, configured to input the fused features into N multi-scale multi-branch Mamba modules connected in series to generate high-level features;

[0124] An image reconstruction module is used to map the high-level features to the size of the optical remote sensing image and output a cloud-free optical remote sensing image in combination with a long skip connection.

[0125] The device achieves high-precision cloud removal of optical remote sensing images through multimodal deep fusion, multi-scale feature extraction and efficient global modeling, combining low computing cost with high practicality, providing reliable technical support for the efficient use of remote sensing data.

[0126] Example 3:

[0127] Based on the same inventive concept, the present invention further provides a computer device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0128] Memory for storing computer programs;

[0129] The processor is configured to implement the SAR image-assisted optical remote sensing image declouding method based on the Mamba model as described in Example 1 when executing the program stored in the memory.

[0130] like Figure 7 As shown, the electronic device may include: a processor 71, a communication interface 72, a memory 73, and a communication bus 74, wherein the processor 71, the communication interface 72, and the memory 73 communicate with each other via the communication bus 44. The processor 71 may call the logic instructions in the memory 73 to execute a SAR image-assisted optical remote sensing image declouding method based on the Mamba model, the method comprising:

[0131] S10, obtaining an optical remote sensing image contaminated by clouds and its corresponding auxiliary SAR image;

[0132] S20, respectively extracting shallow features of the optical remote sensing image and the SAR image through a shallow feature extraction module;

[0133] S30, inputting the shallow features into the Modal-Traversing-SS2D fusion module, achieving deep interaction and fusion of optical image and SAR image features through cross scanning and modal separation, and generating fusion features;

[0134] S40, inputting the fused features into N serially connected multi-scale multi-branch Mamba modules to generate high-level features;

[0135] S50 , mapping the high-level features to the size of the optical remote sensing image through an image reconstruction module, and outputting a cloud-free optical remote sensing image in combination with a long skip connection.

[0136] Example 4:

[0137] An embodiment of the present invention also provides a computer-readable storage medium, in which a program stored is used to execute the SAR image-assisted optical remote sensing image declouding method based on the Mamba model of the above-mentioned embodiment 1. The program can be executed on a processor. The storage medium can be a hard disk, a solid-state drive (SSD), a USB flash drive, an optical disc, a cloud storage, or any other medium that can be used to store and read computer programs. The program stored in the medium is loaded into the processor memory for execution to complete various functions. The storage medium is connected to a hardware device, so that the computer can execute the above-mentioned steps of optical / SAR input → SFE shallow feature extraction → modal fusion → multi-scale processing to generate high-level features → image reconstruction and long-skip addition → cloud-free output.

[0138] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0141] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0142] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A SAR image-assisted optical remote sensing image declouding method based on the Mamba model, characterized in that: The following steps are involved: S10, obtaining an optical remote sensing image contaminated by clouds and its corresponding auxiliary SAR image; S20, respectively extracting shallow features of the optical remote sensing image and the SAR image through a shallow feature extraction module; S30, inputting the shallow features into the Modal-Traversing-SS2D fusion module, achieving deep interaction and fusion of optical image and SAR image features through cross scanning and modal separation, and generating fusion features; S40, inputting the fused features into N serially connected multi-scale multi-branch Mamba modules to generate high-level features; S50 , mapping the high-level features to the size of the optical remote sensing image through an image reconstruction module, and outputting a cloud-free optical remote sensing image in combination with a long skip connection.

2. The SAR image-assisted optical remote sensing image declouding method based on the Mamba model according to claim 1, characterized in that: In step S30, the Modal-Traversing-SS2D fusion module includes the following operations: (a) The shallow features of a set of input optical remote sensing images and SAR images are expanded into a one-dimensional feature sequence along four directions respectively, and rearranged in a modal crossover manner to generate a mixed feature sequence; (b) processing the generated mixed feature sequence through four parallel S6 blocks, and outputting the processed feature sequence; (c) The processed feature sequences are separated by modality and then merged into two-dimensional features according to the expansion direction through the Cross Merge module; The merged features are subjected to channel concatenation and 1×1 convolution to complete cross-modal feature aggregation.

3. The SAR image-assisted optical remote sensing image declouding method based on the Mamba model according to claim 2, characterized in that: The Modal-Traversing-SS2D fusion module processing formula is expressed as follows: Among them, the input features and are optical image features and SAR image features, respectively; t represents the modality type, which can be either opt or sar; LN(·) represents the layer normalization function, Linear(·) represents the linear layer, DWC(·) represents the depthwise convolution, MTSS2D(·) represents the Modal-Traversing SS2D, and AGG(·) represents feature aggregation, which consists of a Concat layer for feature concatenation on the channel and a 1×1 convolution layer for feature interaction; the superscripts 1 / 2 / 3 / 4 / 5 correspond to the output results of the respective stages.

4. The SAR image-assisted optical remote sensing image declouding method based on the Mamba model according to claim 1, characterized in that: In step S40, the multi-scale multi-branch Mamba module includes a multi-scale image encoder and a multi-branch RSSG module; The multi-scale image encoder extracts multi-scale features of input features by setting up depth-wise separable convolutions with different dilation rates in parallel; The multi-branch RSSG module performs global dependency modeling on the multi-scale features and combines convolution branches to extract local features to generate high-level features.

5. The SAR image-assisted optical remote sensing image declouding method based on the Mamba model according to claim 4, characterized in that: The depth-wise separable convolution with different dilation rates includes a 3×3 depth-wise dilated convolution and a 1×1 point-wise convolution; the formula is expressed as follows: F r =DSDC r (F in ),r=1,2,3 (2) Among them, DSDC r represents a depth-wise separable convolution with a dilation rate of r, F r Represents the image features of the corresponding convolution output.

6. The SAR image-assisted optical remote sensing image declouding method based on the Mamba model according to claim 4, characterized in that: The convolution branch combined in the multi-branch RSSG module consists of a 1×1 convolution, a 3×3 depthwise convolution and a residual connection; the formula is as follows: Among them, ConvLF(·) represents the convolution branch, represents the output of each branch when the expansion rate r = 1, 2, 3, RSSG(·) represents the multi-branch RSSG module, AGG(·) represents feature aggregation, It represents the final output feature and serves as the input of the multi-branch RSSG module in the next stage.

7. A SAR image-assisted optical remote sensing image declouding device based on the Mamba model, characterized in that: A SAR image-assisted optical remote sensing image declouding method based on the Mamba model according to any one of claims 1 to 6 is used, wherein the device comprises: Input module, used to obtain cloud-contaminated optical remote sensing images and their corresponding auxiliary SAR images; A shallow feature extraction module, used to extract shallow features of the optical remote sensing image and the SAR image respectively; Modal-Traversing-SS2D fusion module, used to achieve deep interaction and fusion of optical image and SAR image features through cross-scanning and modal separation of the shallow features to generate fused features; A multi-scale multi-branch Mamba module, configured to input the fused features into N multi-scale multi-branch Mamba modules connected in series to generate high-level features; An image reconstruction module is used to map the high-level features to the size of the optical remote sensing image and output a cloud-free optical remote sensing image in combination with a long skip connection.

8. An electronic device, characterized in that: The processor, the communication interface, the memory and the communication bus are connected to each other via the communication bus. Memory for storing computer programs; The processor is configured to implement the Mamba model-based SAR image-assisted optical remote sensing image declouding method according to any one of claims 1 to 6 when executing the program stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the SAR image-assisted optical remote sensing image declouding method based on the Mamba model as described in any one of claims 1 to 6 is implemented.

Images