Optical remote sensing cloud and cloud shadow removal method, system, device and storage medium

Abstract

The present invention relates to the field of image enhancement technology, specifically to a method, system, device and storage medium for removing clouds and cloud shadows from optical remote sensing. To solve the technical problem of poor image declouding effect in the prior art, the present invention first preprocesses optical images and SAR images separately to solve the speckle noise problem in the SAR image and enhances the high-frequency information in the optical image. Then, the penetration ability of the SAR enhanced image is fused with the spectral information of the optical enhanced image, and the fusion result is processed by a channel and space coupled attention mechanism to obtain a semantic complementary feature image. Finally, the semantic complementary feature image is subjected to several structural reconstruction processes to retain high-frequency details and low-frequency semantics, thereby improving the image reconstruction quality. The obtained structural reconstruction feature image is subjected to convolution processing to obtain a cloud noise removed reconstructed image. The present invention is applied to the field of image cloud and cloud shadow removal, and has good removal effect and high image restoration quality.

Description

Technical Field

[0001] The present invention relates to the field of image enhancement technology, and in particular to a method, system, device and storage medium for removing clouds and cloud shadows from optical remote sensing. Background Art

[0002] In the field of cloud and shadow removal from remote sensing images, existing technologies use layered designs and stacked residual groups to reduce clouds and cloud shadows, thereby improving the spectral and structural fidelity of the reconstructed image. Furthermore, residual blocks with channel attention mechanisms can adaptively emphasize useful features while suppressing speckle noise in synthetic aperture radar (SAR) images or cloud distortion in optical data, thereby improving the quality of multimodal fusion. However, fluctuations between different spectral bands result in insufficient spectral restoration and preservation of structural edges when recovering areas with extreme cloud cover and complex ground structures, especially in dense clouds or clouds with complex textures. This results in low cloud removal quality and poor image restoration. Summary of the Invention

[0003] The object of the present invention is to provide a method, system, device and storage medium for removing clouds and cloud shadows from optical remote sensing.

[0004] The technical solutions of the present invention are as follows:

[0005] A method for removing clouds and cloud shadows from optical remote sensing, comprising the following operations:

[0006] S1. Acquire an optical image and a SAR image of the area to be processed; perform dynamic logarithmic compression on the SAR image to obtain a dynamic compressed SAR image; perform adaptive non-local mean filtering on the dynamic compressed SAR image to obtain a filtered SAR image; perform depth denoising and structure restoration on the filtered SAR image to obtain an enhanced SAR image; perform fast Fourier transform on the optical image to map the optical image spatial domain to the frequency domain, perform wavelet enhancement in the frequency domain to obtain an optical texture edge enhanced image; perform inverse Fourier transform on the optical texture edge enhanced image and perform element-by-element addition on the optical image to obtain an optical enhanced image;

[0007] S2. Perform feature fusion processing on the SAR enhanced image and the optical enhanced image based on the mutual attention mechanism to obtain a feature fusion image; the feature fusion image is processed by the channel and spatial coupled attention mechanism to obtain a semantic complementary feature image; the channel and spatial coupled attention mechanism processing operation is as follows: obtain the pixel gradient feature map of the feature fusion image, perform spatial attention processing and channel attention processing respectively after convolution processing, and obtain the attention mechanism feature image by element-by-element multiplication; extract the context information of the attention mechanism feature image, and obtain the semantic complementary feature image by pyramid processing of the multimodal dynamic mechanism;

[0008] S3. The semantic complementary feature image is subjected to several structural reconstruction processes to obtain a structural reconstructed feature image; the structural reconstructed feature image is subjected to convolution processing to obtain a cloud noise removed reconstructed image.

[0009] The operations of deep denoising and structure restoration processing in S1 are as follows: the filtered SAR image is subjected to convolution processing to obtain a filtered SAR convolution image; the filtered SAR convolution image is subjected to several residual connection processing based on depthwise separable convolution and residual connection processing based on nonlinear processing to obtain a filtered SAR residual connection image; the filtered SAR residual connection image is subjected to convolution processing to obtain a SAR enhanced image.

[0010] The feature fusion processing operation based on the mutual attention mechanism in S2 is as follows: the SAR enhanced image is convolved and multiplied element-wise with the optical enhanced image, and then added element-by-element with the optical enhanced image to obtain the initial fused image; the query feature of the initial fused image, the value feature and the key feature of the SAR enhanced image are convolved and processed by the attention mechanism to obtain the attention fused image; the attention fused image is globally pooled, fully connected and feature reshaped, and then multiplied element-by-element with the attention fused image to obtain the attention fused enhanced image; the SAR enhanced image is globally pooled, fully connected and feature reshaped, and then multiplied element-by-element with the SAR enhanced image to obtain the SAR enhanced feature image; the SAR enhanced feature image and the attention fusion enhanced image are element-by-element added to obtain the feature fused image.

[0011] The pyramid processing operation of the multimodal dynamic mechanism in S2 is as follows: the context information of the attention mechanism feature image is convolved to obtain the context convolution feature; the context convolution feature is upsampled and added element-by-element with the context convolution feature to obtain the initial semantic complementary feature; the initial semantic complementary feature is upsampled and added element-by-element with the context convolution feature to obtain the semantic complementary feature image.

[0012] The operations of structural reconstruction processing of the semantic complementary feature image in S3 are as follows: the semantic complementary feature image is processed by convolution, ReLU activation function and convolution to obtain a semantic complementary convolution feature image; the semantic complementary feature image is processed by channel and spatial coupling attention mechanism to obtain a semantic complementary attention feature image; the semantic complementary feature image is processed by a multi-layer perceptron to obtain a semantic complementary perception feature image; the semantic complementary perception feature image and the semantic complementary feature image are multiplied element by element to obtain a first semantic complementary fusion image; the semantic complementary convolution feature image and the semantic complementary attention feature image are added element by element to obtain a second semantic complementary fusion image; the first semantic complementary fusion image and the second semantic complementary fusion image are added element by element to obtain an initial structural reconstruction image; the structural gradient information of the initial structural reconstruction image is fused with the optical reference image, and after convolution processing, it is added element by element with the initial structural reconstruction image to obtain a first structural reconstruction feature image, which is used to perform the second structural reconstruction processing operation.

[0013] The dynamic compression SAR image in S1 is obtained based on the maximum brightness, standard deviation and image information entropy of the SAR normalized image; the SAR normalized image is obtained by normalizing the SAR image.

[0014] The smoothing parameters in the adaptive non-local means filtering process in S1 are obtained based on the pixel information of each position in the dynamic compression SAR image.

[0015] An optical remote sensing cloud and cloud shadow removal system, used to implement the above-mentioned optical remote sensing cloud and cloud shadow removal method, comprising:

[0016] The image preprocessing module is used to obtain optical images and SAR images of the area to be processed; the SAR image is subjected to dynamic logarithmic compression to obtain a dynamic compressed SAR image; the dynamic compressed SAR image is subjected to adaptive non-local mean filtering to obtain a filtered SAR image; the filtered SAR image is subjected to depth denoising and structure restoration to obtain a SAR enhanced image; the optical image is subjected to fast Fourier transform to map the optical image spatial domain to the frequency domain, and wavelet enhancement is performed in the frequency domain to obtain an optical texture edge enhanced image; the optical texture edge enhanced image is subjected to inverse Fourier transform and then element-by-element addition with the optical image to obtain an optical enhanced image;

[0017] The semantic complementary feature image generation module is used to perform feature fusion processing on the SAR enhanced image and the optical enhanced image based on the mutual attention mechanism to obtain a feature fusion image; the feature fusion image is processed by the channel and spatial coupled attention mechanism to obtain a semantic complementary feature image; the channel and spatial coupled attention mechanism processing operation is as follows: the pixel gradient feature map of the feature fusion image is obtained, and after convolution processing, spatial attention processing and channel attention processing are performed respectively, and the attention mechanism feature image is obtained by element-by-element multiplication; the context information of the attention mechanism feature image is extracted and the semantic complementary feature image is obtained by pyramid processing of the multimodal dynamic mechanism;

[0018] The cloud noise removal and reconstruction image generation module is used to obtain a structural reconstruction feature image by subjecting the semantic complementary feature image to several structural reconstruction processes; the structural reconstruction feature image is subjected to convolution processing to obtain a cloud noise removal and reconstruction image.

[0019] An optical remote sensing cloud and cloud shadow removal device comprises a processor and a memory, wherein the processor implements the above-mentioned optical remote sensing cloud and cloud shadow removal method when executing a computer program stored in the memory.

[0020] A computer-readable storage medium is used to store a computer program, wherein when the computer program is executed by a processor, the above-mentioned optical remote sensing cloud and cloud shadow removal method is implemented.

[0021] The beneficial effects of the present invention are:

[0022] The present invention provides a method for removing clouds and cloud shadows from optical remote sensing. First, the optical image and SAR image of the area to be processed are preprocessed separately to solve the problems of high dynamic range and multiplicative speckle noise in SAR images and enhance the high-frequency information in the optical image. Then, the penetration ability of the SAR enhanced image and the spectral information of the optical enhanced image are subjected to feature fusion processing based on the mutual attention mechanism, which solves the image blur and information loss problems of the optical remote sensing image under cloud cover, obtains a feature fused image, and processes the feature fused image with channel and spatial coupled attention mechanism, which enhances the complementarity between the SAR image and the optical image, thereby accurately fusing multi-source information and effectively improving the structural consistency and clarity, and obtain a semantic complementary feature image; finally, the semantic complementary feature image is subjected to several structural reconstruction processes, and in the structural reconstruction process, the semantic complementary convolution features reflecting the edge information of the underlying texture of the image, the semantic complementary attention features reflecting the key area information of the image channel and spatial dimensions, and the semantic complementary perceptual features reflecting the high-level semantic abstract information of the image are fused to enhance the structural details, and the gradient information is used to provide a priori structural constraints for the reconstruction, which can retain both high-frequency details and low-frequency semantics during reconstruction, improve the image reconstruction quality, and obtain a structural reconstruction feature image; the structural reconstruction feature image is subjected to convolution processing to obtain a cloud noise removed reconstructed image; it is used in the field of image cloud and cloud shadow removal, with good removal effect and high image restoration quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] By reading the detailed description of the preferred embodiment below, the solutions and advantages of the present application will become clear to those skilled in the art. The accompanying drawings are only for illustrating the preferred embodiment and are not to be considered as limiting the present invention.

[0024] In the attached figure:

[0025] Figure 1 In the embodiment, a schematic diagram of the process of the method of this embodiment;

[0026] Figure 2 : This is a qualitative comparison of the visual processing capabilities of the method of this embodiment and three existing methods in three types of cloud cover scenes;

[0027] Figure 3 : This is a quantitative comparison chart of the visual processing capabilities of the method of this embodiment and three existing methods in three types of cloud cover scenes;

[0028] Figure 4 : is a comparison diagram of the determination coefficient between the predicted NDVI and the ground truth, between the method of this embodiment and three existing methods;

[0029] Figure 52 is a diagram showing the effect of removing clouds and cloud shadows in different large-scale scenes using the method of this embodiment. DETAILED DESCRIPTION

[0030] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings.

[0031] This embodiment provides a method for removing clouds and cloud shadows from optical remote sensing. Figure 1 , including the following operations:

[0032] S1. Acquire an optical image and a SAR image of the area to be processed; perform dynamic logarithmic compression on the SAR image to obtain a dynamic compressed SAR image; perform adaptive non-local mean filtering on the dynamic compressed SAR image to obtain a filtered SAR image; perform depth denoising and structure restoration on the filtered SAR image to obtain an enhanced SAR image; perform fast Fourier transform on the optical image to map the optical image spatial domain to the frequency domain, perform wavelet enhancement in the frequency domain to obtain an optical texture edge enhanced image; perform inverse Fourier transform on the optical texture edge enhanced image and perform element-by-element addition on the optical image to obtain an optical enhanced image;

[0033] S2. Perform feature fusion processing on the SAR enhanced image and the optical enhanced image based on the mutual attention mechanism to obtain a feature fused image; the feature fused image is processed by the channel and spatial coupled attention mechanism to obtain a semantic complementary feature image;

[0034] The channel and spatial coupled attention mechanism processing operation is as follows: obtain the pixel gradient feature map of the feature fusion image, perform spatial attention processing and channel attention processing respectively after convolution processing, and obtain the attention mechanism feature image by element-by-element multiplication; extract the context information of the attention mechanism feature image, and obtain the semantic complementary feature image by pyramid processing of the multimodal dynamic mechanism;

[0035] S3. The semantic complementary feature image is subjected to several structural reconstruction processes to obtain a structurally reconstructed feature image. The structurally reconstructed feature image is subjected to convolution processing to obtain a cloud noise removed reconstructed image. The specific operation steps are detailed as follows.

[0036] S1. Obtain an optical image and a SAR image of the area to be processed; subject the SAR image to dynamic logarithmic compression to obtain a dynamically compressed SAR image; subject the dynamically compressed SAR image to adaptive non-local mean filtering to obtain a filtered SAR image; subject the filtered SAR image to depth denoising and structure restoration to obtain a SAR enhanced image; subject the optical image to fast Fourier transform, mapping the optical image spatial domain to the frequency domain, and performing wavelet enhancement in the frequency domain to obtain an optical texture edge enhanced image; subject the optical texture edge enhanced image to inverse Fourier transform processing and then perform element-by-element addition with the optical image to obtain an optically enhanced image.

[0037] The optical image and SAR image of the area to be processed are preprocessed separately to solve the high dynamic range and multiplicative speckle noise problems in the SAR image, and to enhance the high-frequency information in the optical image to improve the quality of the processed image, thereby improving the subsequent cloud removal and reconstruction effect.

[0038] First, optical images and SAR images of the area to be processed containing cloud noise are obtained as cloud and cloud shadow processing objects, and image reconstruction is performed.

[0039] Then, the optical image and SAR image are preprocessed respectively to solve the high dynamic range and multiplicative speckle noise problems in SAR images and enhance the high-frequency information in the optical image.

[0040] The process of preprocessing optical images is as follows: the SAR image is subjected to dynamic logarithmic compression to map its original exponential distribution into a linear domain, effectively reducing the dynamic range of the image and improving the stability of subsequent processing to obtain a dynamically compressed SAR image; the dynamically compressed SAR image is subjected to adaptive non-local mean filtering to remove part of the multiplicative noise while keeping the key structure and edge information intact to obtain a filtered SAR image; the filtered SAR image is subjected to deep denoising and structure restoration to enhance the image's long-range dependency and ability to maintain detailed texture, thereby improving the usability and expressiveness of the SAR image to obtain a SAR enhanced image.

[0041] The dynamic compression SAR image in S1 is obtained based on the maximum brightness, standard deviation and image information entropy of the SAR normalized image; the SAR normalized image is obtained by normalizing the SAR image; that is, the above-mentioned dynamic logarithmic compression processing operation is achieved through the following formula:

[0042] ,

[0043] ,

[0044] For dynamic compression of SAR images, C is the contrast adjustment constant (based on the dynamic changes of the image pixel characteristics of the SAR image to be processed), is the SAR normalized image (obtained by normalizing the SAR image), 、 、 are the maximum brightness, standard deviation and image information entropy of the SAR normalized image, respectively. 、 are the first empirical coefficient and the second empirical coefficient respectively, is the denominator compensation amount. Compared with the existing logarithmic compression processing method, the dynamic logarithmic compression processing method of this embodiment can adaptively adjust the logarithmic compression strength according to the brightness distribution of the image, thereby improving the image compression accuracy.

[0045] Compared to the fixed smoothing parameters in conventional non-local mean filtering, the smoothing parameters in the adaptive non-local mean filtering process are dynamically variable and are obtained based on the pixel information at each position in the dynamically compressed SAR image. This enhances sensitivity to similarity differences and protects structural information at image edges or in high-texture areas, while also enhancing the smoothing effect in flat image areas. The smoothing parameters in this embodiment are calculated using the following formula:

[0046] ,

[0047] is the smoothing parameter at the position (x, y) of the dynamic compression SAR image, is the initial smoothing coefficient, is a hyperparameter, 、 is the neighborhood pixel variance in the x and y directions at the position (x, y) of the dynamically compressed SAR image.

[0048] The above-mentioned deep denoising and structure restoration processing operations can be implemented through the Swin Transformer network. However, in order to further improve the long-range dependency of the image and the ability to preserve detailed textures, thereby enhancing the usability and expressiveness of the SAR image, this embodiment designs an efficient deep denoising and structure restoration processing method. The operation process is as follows: the filtered SAR image is subjected to convolution processing to obtain a filtered SAR convolution image; the filtered SAR convolution image is subjected to several residual connection processing based on depthwise separable convolution and residual connection processing based on nonlinear processing to obtain a filtered SAR residual connection image; and the filtered SAR residual connection image is subjected to convolution processing to obtain a SAR enhanced image.

[0049] The residual connection processing based on depthwise separable convolution is as follows: after the input undergoes convolution, depthwise separable convolution, and convolution processing, it is element-wise added to the input to obtain an output for performing the residual connection processing based on nonlinear processing. The residual connection processing based on nonlinear processing is specifically as follows: after the input (the output of the residual connection processing based on depthwise separable convolution is repeated a number of times) is convolved to obtain a convolution input; after the convolution input undergoes nonlinear processing (which can be achieved through a sigmoid function), it is element-wise multiplied with the convolution input, and then element-wise added to the input (the output of the residual connection processing based on depthwise separable convolution is repeated a number of times) to obtain an output for performing the next residual connection processing based on depthwise separable convolution, or as a filtered SAR residual connection image. The number of times the residual connection processing based on depthwise separable convolution and the residual connection processing based on nonlinear processing is preferably four.

[0050] The operation of preprocessing the optical image is as follows: the optical image is processed by fast Fourier transform, and the optical image spatial domain is mapped to the frequency domain in order to separate different frequency components. Then, wavelet enhancement processing is performed in the frequency domain to specifically enhance the medium and high frequency components, thereby highlighting the texture details and edge structure in the image, and obtaining an optical texture edge enhanced image; the optical texture edge enhanced image is processed by inverse Fourier transform, and the enhanced frequency feature map is restored to the spatial domain to obtain an enhanced image with richer detail information. After that, it is added element by element with the optical image to achieve complementarity between the frequency domain enhancement information and the original optical image, and an optically enhanced image is obtained, which effectively alleviates the edge blur and detail loss problems caused by cloud occlusion, and further improves the quality of subsequent cloud removal and image reconstruction.

[0051] S2. The SAR enhanced image and the optical enhanced image are subjected to feature fusion processing based on the mutual attention mechanism to obtain a feature fusion image; the feature fusion image is processed by the channel and spatial coupled attention mechanism to obtain a semantic complementary feature image.

[0052] The penetration ability of SAR enhanced images and the spectral information of optical enhanced images are subjected to feature fusion processing based on the mutual attention mechanism, which solves the image blur and information loss problems of optical remote sensing images under cloud cover, and obtains feature fused images. The feature fused images are then processed with the channel and spatial coupled attention mechanism, which enhances the complementarity between SAR images and optical images, thereby accurately fusing multi-source information, effectively improving the structural consistency and clarity of the image after cloud removal, and obtaining semantically complementary feature images.

[0053] Firstly, the SAR enhanced image and the optical enhanced image are subjected to feature fusion processing based on the mutual attention mechanism. At the same time, the prior information of the optical image and the SAR image is introduced to enhance the features in the cloud contaminated image and obtain the feature fused image.

[0054] The operation of feature fusion processing based on the mutual attention mechanism is as follows: after convolution processing, the SAR enhanced image is multiplied element-wise with the optical enhanced image, and then element-wise added with the optical enhanced image to obtain the initial fused image; the query feature of the initial fused image, the value feature and the key feature of the SAR enhanced image are respectively convolved and processed by the attention mechanism to obtain the attention fused image; the attention fused image is globally pooled, fully connected and feature reshaped (which can be achieved through the reshape function), and then element-wise multiplied with the attention fused image to obtain the attention fused enhanced image; the SAR enhanced image is globally pooled, fully connected and feature reshaped (which can be achieved through the reshape function), and then element-wise multiplied with the SAR enhanced image to obtain the SAR enhanced feature image; the SAR enhanced feature image and the attention fusion enhanced image are element-wise added to obtain the feature fused image.

[0055] The feature-fused image is then processed through a channel- and spatial-coupled attention mechanism to produce a semantically complementary feature image. The channel- and spatial-coupled attention mechanism operates as follows: A pixel gradient feature map of the feature-fused image is obtained (achievable through edge detection), followed by (two-dimensional) convolution, followed by spatial attention and channel attention processing, respectively. Element-wise multiplication enhances the ability to recognize object edges and cloud structures in the image, improving the ability to perceive object edges and structural details in complex scenes, resulting in an attention mechanism feature image. Contextual information is extracted from the attention mechanism feature image to enhance understanding of object boundaries and cloud morphology. Pyramid processing using a multimodal dynamic mechanism enhances context and detail representation, enhancing understanding of object boundaries and cloud morphology, resulting in a semantically complementary feature image.

[0056] The above-mentioned extraction of context information of the attention mechanism feature image can be achieved by performing convolution processing on the attention mechanism feature image at different scales and then fusing them in the channel dimension.

[0057] The pyramid processing operation of the above-mentioned multimodal dynamic mechanism is as follows: the context information of the attention mechanism feature image is convolved to obtain the context convolution feature; the context convolution feature is upsampled and then added element-by-element with the context convolution feature to obtain the initial semantic complementary feature; the initial semantic complementary feature is upsampled and then added element-by-element with the context convolution feature to obtain the semantic complementary feature image.

[0058] S3. The semantic complementary feature image is subjected to several structural reconstruction processes to obtain a structural reconstructed feature image; the structural reconstructed feature image is subjected to convolution processing to obtain a cloud noise removed reconstructed image.

[0059] The semantic complementary feature image is subjected to several structural reconstruction processes. During the structural reconstruction process, the semantic complementary convolution features reflecting the edge information of the underlying texture of the image, the semantic complementary attention features reflecting the key area information of the image channel and spatial dimensions, and the semantic complementary perceptual features reflecting the high-level semantic abstract information of the image are fused to enhance the structural details. The gradient information is used to provide a priori structural constraints for the reconstruction, which can retain both high-frequency details and low-frequency semantics during reconstruction, avoid "over-smoothing" or "semantic confusion" problems, improve the image reconstruction quality, and obtain a structural reconstruction feature image. The structural reconstruction feature image is subjected to convolution processing to obtain a cloud noise removed reconstructed image.

[0060] Taking the first structural reconstruction process as an example, the operating steps of the structural reconstruction process are as follows.

[0061] Step 1: The semantic complementary feature image is processed by convolution, ReLU activation function and convolution to obtain a semantic complementary convolution feature image; the semantic complementary feature image is processed by the channel and spatial coupled attention mechanism to obtain a semantic complementary attention feature image; the semantic complementary feature image is processed by the multi-layer perceptron to obtain a semantic complementary perception feature image.

[0062] Step 2: The semantically complementary perception feature image and the semantically complementary feature image are multiplied element by element to obtain a first semantically complementary fusion image; the semantically complementary convolution feature image and the semantically complementary attention feature image are added element by element to obtain a second semantically complementary fusion image; the first semantically complementary fusion image and the second semantically complementary fusion image are added element by element to obtain an initial structure reconstructed image.

[0063] Step 3: The structural gradient information of the initial structural reconstruction image (which can be obtained by edge detection processing of the initial structural reconstruction image) is fused with the optical reference image. After convolution processing, it is added element by element with the initial structural reconstruction image to obtain the first structural reconstruction feature image; which is used to perform the second structural reconstruction processing.

[0064] The number of times of the above-mentioned structure reconstruction process is preferably 4 times.

[0065] In order to verify the removal effect of the method of this embodiment, the following experiments were conducted.

[0066] In the experiment, the method of this embodiment (hereinafter referred to as MSGRF-CR method) was compared with three most advanced methods in the prior art (including Dsen2-CR method, GLF-CR method, HS 2P method) for comparative evaluation. The comparison results are shown in Table 1. As can be clearly seen from Table 1, the Dsen2-CR method has the lowest peak signal-to-noise ratio (PSNR) and mean square error (MSE), while the structural similarity index (SSIM) and spectral angle mapper (SAM) are only slightly higher than the GLF-CR model, which has the lowest SSIM and SAM. In contrast, GLF-CR has the best MSE performance and its PSNR value is second only to the Dsen2-CR model. HS 2 The P model ranked second in terms of PSNR, SSIM, and SAM, but third in terms of MSE. In contrast, the method in this embodiment (MSGRF-CR method) consistently outperformed the three existing state-of-the-art methods in all four metrics, demonstrating its superior cloud removal performance.

[0067] Table 1 Summary of reconstruction effects of different reconstruction methods

[0068] .

[0069] At the same time, the qualitative visual processing capabilities of the method in this embodiment (MSGRF-CR method) and three existing methods under three types of cloud cover (large range, medium range and small range) are compared in the experiment. Figure 2 . Figure 2 The middle image from top to bottom is a cloudy optical image, a SAR image, a cloudless optical image, and the result image processed by the DSen2-CR method, the result image processed by the GLF-CR method, and the HS 2 The result graph processed by the P method and the result graph processed by the method of this embodiment (MSGRF-CR method) are shown in Figure 1. The size of each image is 128×128. Figure 2 Counting from left to right in the middle image, the first and second columns show different scenes with heavy cloud cover (large-scale cloud cover), with cloud coverage exceeding 80%. The third and fourth columns show different scenes with medium cloud cover (medium-scale cloud cover), with coverage between 40% and 80%. The third and fourth columns show different scenes with low cloud cover (small-scale cloud cover), with cloud coverage less than 40%. Figure 2 As can be seen in Figure 2, the GLF-CR method produces the worst visual results, with the generated images being too blurry and unable to recover clear structures, accurate colors, or textures, especially in the presence of large cloud cover. In contrast, the Dsen2-CR method improves on the GLF-CR model in terms of color, texture, and boundary recovery. However, it still exhibits significant color and texture distortion, noticeable artifacts, and blurred boundaries, and the image quality degrades severely when the cloud cover increases from minimal to medium and large areas. 2The P method restores the structure, details and texture more faithfully, and outperforms the GLF-CR method and the Dsen2-CR method; however, the HS 2 The P method only restores structure locally and is accompanied by considerable noise. In contrast, the method of this embodiment (MSGRF-CR method) produces results closest to cloud-free images, effectively outlining clear object boundaries and preserving high-fidelity details. Under moderate cloud cover, it accurately reconstructs cloud-obscured areas with rich detail. Under minimal cloud cover, it preserves texture and edge information, resulting in higher clarity. Even under extensive cloud cover, the method of this embodiment (MSGRF-CR method) produces images closest to cloud-free conditions, outperforming other models. However, further improvement in structural detail and boundary refinement is still necessary.

[0070] Next, to evaluate the performance of the method in this example (MSGRF-CR method) and three existing methods under various cloud cover conditions, the experiment also used datasets with five cloud coverage intervals: 0%-20%, 20%-40%, 40%-60%, 60%-80%, and 80%-100%. These cloud coverage intervals range from light cloud cover to heavy cloud cover, which helps to systematically evaluate the robustness and accuracy of each method. The experimental evaluation uses quantitative indicators - MSE, PSNR, SSIM and SAM - to measure the effectiveness of each method in handling different cloud densities. The experimental results are shown in [1]. Figure 3 .exist Figure 3 However, when the cloud coverage is lower than 60%, the method of this embodiment (MSGRF-CR method) outperforms all other models. At the same time, in terms of the SAM index, the method of this embodiment (MSGRF-CR method) outperforms the three existing methods under all conditions. In other words, the overall performance of the method of this embodiment (MSGRF-CR method) is the best.

[0071] In addition, in order to further evaluate the effectiveness of the method of this embodiment (MSGRF-CR method) in reconstructing vegetation information under cloud cover, 720 representative cloudy remote sensing images and their corresponding cloud-free predicted images were randomly selected in the experiment, and the normalized vegetation index (NDVI) was calculated using four methods. The coefficient of determination (R²) was calculated to quantify the consistency of the normalized difference vegetation index between the original image and the predicted image, and the R² value distribution of the four models was statistically analyzed and compared. In addition, in order to intuitively demonstrate the vegetation reconstruction performance of the method, the experiment randomly selected three representative test images with the lowest, median, and highest R² values, and generated an NDVI scatter plot with a fitted regression line to intuitively illustrate the correlation between the ground truth and predicted NDVI of the four methods, as shown in Figure 2. Figure 4 shown. Figure 4 The blue area represents the interquartile range, the middle red line represents the median, and the black line represents the range. Figure 4 It can be clearly seen that the method of this embodiment (MSGRF-CR method) has the highest median R², a narrow interquartile range, and the best performance, reflecting its excellent accuracy and stability.

[0072] Finally, in order to evaluate the robustness and generalization ability of the method of this embodiment (MSGRF-CR method), the experiment used the dataset provided by Ebel et al. (2022c), in which non-overlapping 256 × 256 pixel image fragments were reassembled into panoramic images. On this basis, the experiment designed an experiment with large-scale scene images (4096 × 3072 pixels), and adopted a sliding window preprocessing method to divide the original image into 1024 × 1024 pixel sub-image blocks. There is an overlap of 128 pixels between adjacent image blocks, and each scene produces approximately 425 sub-images. Each method processes each sub-image block independently, and then reconstructs the complete scene output by seamlessly merging the processed image blocks. The experimental results of large-scale cloud reconstruction are shown in Figure 2. Figure 5 As shown, Figure 5 From top to bottom in the figure are the cloudy optical image, SAR image, cloud-free optical image, MSGRF-CR method processing result diagram, locally enlarged area diagram of the cloudy optical image (the enlarged area comes from the cloudy optical image, and the image name is simplified to cloudy optical image (enlarged version)), locally enlarged area diagram of the SAR image (the enlarged area comes from the SAR image, and the image name is simplified to SAR image (enlarged version)), locally enlarged area diagram of the cloud-free optical image (the enlarged area comes from the cloud-free optical image, and the image name is simplified to cloud-free optical image (enlarged version)), and locally enlarged area diagram of the MSGRF-CR method processing result (the enlarged area comes from the MSGRF-CR method processing result diagram, and the image name is simplified to MSGRF-CR method (enlarged version)). Among them, the local magnified area map of cloudy optical image, the local magnified area map of SAR image, the local magnified area map of cloudless optical image, and the local magnified area map of MSGRF-CR method processing result all contain 8 small pictures. Two adjacent small pictures form a group of scene pictures. The scenes are four kinds of scenes: mountains, plains, forests, and towns. The location of the terrain magnified area corresponds to the yellow frame area and red frame area of each scene in the MSGRF-CR method processing result map. Figure 5 It can be seen that the method of this embodiment (MSGRF-CR method) has excellent overall cloud removal performance in large-scale cloud cover images, which is comparable to the original cloud pollution image ( Figure 5 Compared with the medium cloudy optical image), the method in this embodiment (MSGRF-CR method) can effectively remove the clouds and restore the surface with clear ground structure and boundary details ( Figure 5In addition, Figure 5 The enlarged details of the areas marked by red and yellow frames in the MSGRF-CR processing results are shown in Figure 5 Figure 1 shows a zoomed-in image of the MSGRF-CR processing results (see the MSGRF-CR method (zoomed version)). These zoomed-in details demonstrate that the final reassembled image, with a size of 4096 × 3072, exhibits high global consistency. Although the image was processed using 1024 × 1024 patches, no noticeable stitching artifacts or boundary discontinuities were observed. Furthermore, in areas with dense cloud cover, the method (MSGRF-CR method) in this example accurately restores surface details such as ground texture, vegetation cover, and urban boundaries.

[0073] This embodiment further provides an optical remote sensing cloud and cloud shadow removal system, which is used to implement the above-mentioned optical remote sensing cloud and cloud shadow removal method, including:

[0074] The image preprocessing module is used to obtain optical images and SAR images of the area to be processed; the SAR image is subjected to dynamic logarithmic compression to obtain a dynamic compressed SAR image; the dynamic compressed SAR image is subjected to adaptive non-local mean filtering to obtain a filtered SAR image; the filtered SAR image is subjected to depth denoising and structure restoration to obtain a SAR enhanced image; the optical image is subjected to fast Fourier transform to map the optical image spatial domain to the frequency domain, and wavelet enhancement is performed in the frequency domain to obtain an optical texture edge enhanced image; the optical texture edge enhanced image is subjected to inverse Fourier transform and then element-by-element addition with the optical image to obtain an optical enhanced image;

[0075] The semantic complementary feature image generation module is used to perform feature fusion processing on the SAR enhanced image and the optical enhanced image based on the mutual attention mechanism to obtain a feature fusion image; the feature fusion image is processed by the channel and spatial coupled attention mechanism to obtain a semantic complementary feature image; the channel and spatial coupled attention mechanism processing operation is as follows: the pixel gradient feature map of the feature fusion image is obtained, and after convolution processing, spatial attention processing and channel attention processing are performed respectively, and the attention mechanism feature image is obtained by element-by-element multiplication; the context information of the attention mechanism feature image is extracted and the semantic complementary feature image is obtained by pyramid processing of the multimodal dynamic mechanism;

[0076] The cloud noise removal and reconstruction image generation module is used to obtain a structural reconstruction feature image by subjecting the semantic complementary feature image to several structural reconstruction processes; the structural reconstruction feature image is subjected to convolution processing to obtain a cloud noise removal and reconstruction image.

[0077] This embodiment also provides an optical remote sensing cloud and cloud shadow removal device, including a processor and a memory, wherein the processor implements the above-mentioned optical remote sensing cloud and cloud shadow removal method when executing a computer program stored in the memory.

[0078] This embodiment further provides a computer-readable storage medium for storing a computer program, wherein when the computer program is executed by a processor, the above-mentioned optical remote sensing cloud and cloud shadow removal method is implemented.

[0079] This embodiment provides a method for removing clouds and cloud shadows from optical remote sensing. First, the optical image and SAR image of the area to be processed are preprocessed separately to solve the problems of high dynamic range and multiplicative speckle noise in SAR images and enhance the high-frequency information in the optical image. Then, the penetration ability of the SAR enhanced image and the spectral information of the optical enhanced image are subjected to feature fusion processing based on the mutual attention mechanism, which solves the image blur and information loss problems of optical remote sensing images under cloud cover, obtains a feature fused image, and processes the feature fused image with channel and spatial coupled attention mechanism, which enhances the complementarity between the SAR image and the optical image, thereby accurately fusing multi-source information and effectively improving the structural consistency and clarity, and obtain a semantic complementary feature image; finally, the semantic complementary feature image is subjected to several structural reconstruction processes, and in the structural reconstruction process, the semantic complementary convolution features reflecting the edge information of the underlying texture of the image, the semantic complementary attention features reflecting the key area information of the image channel and spatial dimensions, and the semantic complementary perceptual features reflecting the high-level semantic abstract information of the image are fused to enhance the structural details, and the gradient information is used to provide a priori structural constraints for the reconstruction, which can retain both high-frequency details and low-frequency semantics during reconstruction, improve the image reconstruction quality, and obtain a structural reconstruction feature image; the structural reconstruction feature image is subjected to convolution processing to obtain a cloud noise removed reconstructed image; it is used in the field of image cloud and cloud shadow removal, with good removal effect and high image restoration quality.

Claims

1. A method for removing clouds and cloud shadows from optical remote sensing, characterized in that: The following operations are included: S1. Acquire an optical image and a SAR image of the area to be processed; subject the SAR image to dynamic logarithmic compression to obtain a dynamic compressed SAR image; subject the dynamic compressed SAR image to adaptive non-local mean filtering to obtain a filtered SAR image; subject the filtered SAR image to depth denoising and structure restoration to obtain an enhanced SAR image; The optical image is processed by fast Fourier transform, and the optical image spatial domain is mapped to the frequency domain. Wavelet enhancement is performed in the frequency domain to obtain an optical texture edge enhanced image. The optical texture edge enhanced image is processed by inverse Fourier transform and then element-by-element added to the optical image to obtain an optically enhanced image. S2, performing feature fusion processing on the SAR enhanced image and the optical enhanced image based on the mutual attention mechanism to obtain a feature fused image; The feature fusion image is processed by the channel and spatial coupled attention mechanism to obtain a semantically complementary feature image; The channel and spatial coupled attention mechanism processing operation is as follows: obtain the pixel gradient feature map of the feature fusion image, perform spatial attention processing and channel attention processing respectively after convolution processing, and obtain the attention mechanism feature image by element-by-element multiplication; extract the context information of the attention mechanism feature image, and obtain the semantic complementary feature image by pyramid processing of the multimodal dynamic mechanism; S3. The semantic complementary feature image is subjected to several structural reconstruction processes to obtain a structural reconstructed feature image; the operations of the structural reconstruction process of the semantic complementary feature image are as follows: the semantic complementary feature image is subjected to convolution, ReLU activation function and convolution process to obtain a semantic complementary convolution feature image; the semantic complementary feature image is processed by the channel and spatial coupling attention mechanism to obtain a semantic complementary attention feature image; the semantic complementary feature image is processed by the multi-layer perception mechanism to obtain a semantic complementary perception feature image; the semantic complementary perception feature image and the semantic complementary feature image are multiplied element by element to obtain a first semantic complementary fusion image; the semantic complementary convolution feature image and the semantic complementary attention feature image are added element by element to obtain a second semantic complementary fusion image; the first semantic complementary fusion image and the second semantic complementary fusion image are added element by element to obtain an initial structural reconstruction image; the structural gradient information of the initial structural reconstruction image is fused with the optical reference image, and after convolution process, the image is added element by element to the initial structural reconstruction image to obtain a first structural reconstruction feature image for performing the second structural reconstruction process; The structural reconstruction feature image is processed by convolution to obtain the cloud noise removed reconstructed image.

2. The optical remote sensing cloud and cloud shadow removal method according to claim 1, characterized in that: In S1, the operations of deep denoising and structure restoration are: The filtered SAR image is subjected to convolution processing to obtain a filtered SAR convolution image; the filtered SAR convolution image is subjected to several residual connection processing based on depthwise separable convolution and residual connection processing based on nonlinear processing to obtain a filtered SAR residual connection image; the filtered SAR residual connection image is subjected to convolution processing to obtain a SAR enhanced image.

3. The optical remote sensing cloud and cloud shadow removal method according to claim 1, characterized in that: In S2, the feature fusion processing based on the mutual attention mechanism is as follows: After convolution processing, the SAR enhanced image is multiplied element-by-element with the optical enhanced image, and then added element-by-element with the optical enhanced image to obtain the initial fused image; the query feature of the initial fused image, the value feature and the key feature of the SAR enhanced image are respectively convolved and processed by the attention mechanism to obtain the attention fused image; the attention fused image is globally pooled, fully connected and feature reshaped, and then multiplied element-by-element with the attention fused image to obtain the attention fused enhanced image; the SAR enhanced image is globally pooled, fully connected and feature reshaped, and then multiplied element-by-element with the SAR enhanced image to obtain the SAR enhanced feature image; the SAR enhanced feature image and the attention fusion enhanced image are element-by-element added to obtain the feature fused image.

4. The optical remote sensing cloud and cloud shadow removal method according to claim 1, characterized in that: In S2, the pyramid processing operations of the multimodal dynamic mechanism are: The context information of the attention mechanism feature image is convolved to obtain the context convolution feature; the context convolution feature is upsampled and added element-by-element with the context convolution feature to obtain the initial semantic complementary feature; the initial semantic complementary feature is upsampled and added element-by-element with the context convolution feature to obtain the semantic complementary feature image.

5. The optical remote sensing cloud and cloud shadow removal method according to claim 1, characterized in that: The dynamic compression SAR image in S1 is obtained based on the maximum brightness, standard deviation and image information entropy of the SAR normalized image; the SAR normalized image is obtained by normalizing the SAR image.

6. The optical remote sensing cloud and cloud shadow removal method according to claim 1, characterized in that: The smoothing parameters in the adaptive non-local means filtering process in S1 are obtained based on the pixel information of each position in the dynamic compression SAR image.

7. An optical remote sensing cloud and cloud shadow removal system, used to implement the optical remote sensing cloud and cloud shadow removal method according to claim 1, characterized in that: include: Image preprocessing module, used to obtain optical images and SAR images of the area to be processed; The SAR image is processed by dynamic logarithmic compression to obtain a dynamic compressed SAR image; the dynamic compressed SAR image is processed by adaptive non-local mean filtering to obtain a filtered SAR image; the filtered SAR image is processed by depth denoising and structure restoration to obtain a SAR enhanced image; the optical image is processed by fast Fourier transform to map the optical image spatial domain to the frequency domain, and wavelet enhancement is performed in the frequency domain to obtain an optical texture edge enhanced image; the optical texture edge enhanced image is processed by inverse Fourier transform and then element-by-element addition with the optical image to obtain an optical enhanced image; The semantic complementary feature image generation module is used to perform feature fusion processing on the SAR enhanced image and the optical enhanced image based on the mutual attention mechanism to obtain a feature fused image; The feature fusion image is processed by the channel and spatial coupled attention mechanism to obtain a semantically complementary feature image; The channel and spatial coupled attention mechanism processing operation is as follows: obtain the pixel gradient feature map of the feature fusion image, perform spatial attention processing and channel attention processing respectively after convolution processing, and obtain the attention mechanism feature image by element-by-element multiplication; extract the context information of the attention mechanism feature image, and obtain the semantic complementary feature image by pyramid processing of the multimodal dynamic mechanism; The cloud noise removal and reconstruction image generation module is used to obtain a structural reconstruction feature image by subjecting the semantic complementary feature image to several structural reconstruction processes; the structural reconstruction feature image is subjected to convolution processing to obtain a cloud noise removal and reconstruction image.

8. An optical remote sensing cloud and cloud shadow removal device, characterized in that: The method comprises a processor and a memory, wherein when the processor executes the computer program stored in the memory, the method for removing cloud and cloud shadows from optical remote sensing according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the optical remote sensing cloud and cloud shadow removal method according to any one of claims 1 to 6 is implemented.

Images