SAR (Synthetic Aperture Radar) and multispectral image fusion cloud removal method and system
Through the dual-U-type end-to-end neural network designed by ConvNeXt backbone network, the conversion and image fusion of SAR images to MSI are optimized, and the problem of insufficient feature information after SAR images is solved in the prior art, and efficient cloud removal and information are achieved Recovery improves the quality of remote sensing images.
Patent Information
- Application Number
- CN202411152351.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-08
AI Technical Summary
In the field of remote sensing image decloud, especially in the two-stage indirect fusion method, the SAR image cannot effectively provide feature information of multiple spectral bands after being converted into simulated multispectral images, resulting in unsatisfactory cloud removal effect and network design Complexity and accuracy are difficult to balance.
Using a dual U-type end-to-end neural network based on ConvNeXt, through two-stage feature learning and overall supervision, the image conversion from SAR image to MSI is optimized in the first stage, and the weighted three-source image fusion is carried out in the second stage, and the cloud mask weighted image is provided. Lose weights appropriately, optimize network training to generate decloud MSI results.
It realizes the integration of SAR images and MSI in cloud removal, which improves the accuracy of cloud removal and information utilization, reduces the complexity of network design, and enhances the stability and accuracy of cloud removal effects.
Smart Images

Figure CN118918427A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision and deep learning, and relates to a method and system for cloud removal by synthetic aperture radar and multispectral image fusion based on an end-to-end two-stage conversion-fusion network, which is suitable for multispectral image cloud removal scenarios with high fidelity and low computational requirements. Background Art
[0002] Different satellites have different imaging principles when acquiring remote sensing data. The spatial resolution, measurement scale, imaging angle and other characteristics of the images are also different, thus generating data of different modes, such as Synthetic Aperture Radar (SAR) images, LiDAR (Light Detection And Ranging,
[0003] The acquisition of optical remote sensing images often cannot avoid the interference of clouds, and the cloud cover in the data becomes a major pain point and difficulty in subsequent downstream tasks such as target classification and ground monitoring. Due to the thin and thick cloud cover in optical remote sensing images, a large amount of important and available ground information in the cloud area is damaged or missing, such as water bodies, vegetation, land, building information, etc., which seriously affects the quality and availability of the image. Thanks to the construction of remote sensing big data sets, data-driven deep learning algorithms have developed rapidly in the field of remote sensing image declouding, and image fusion is one of the major research hotspots. Compared with single-modality remote sensing image declouding, multimodal fusion, which uses auxiliary modalities to provide missing information for the target modality, provides a broader and more effective cloud processing idea and method, improves the quality of remote sensing data, and ensures the realization of various downstream tasks based on remote sensing data.
[0004] Cloud removal in remote sensing images, especially thick cloud removal, is an extremely important and indispensable processing task in remote sensing image processing. The goal of this task is triggered by the cloud problem in optical remote sensing images. In essence, the ultimate goal is to reconstruct the surface information and ground features of the cloud area, such as texture and color. Multimodal image fusion uses multiple images of two or more different modalities to fuse according to a certain method, and finally generates a single image. The generated result contains the effective target information in the fused data, and integrates the features of different aspects and levels to improve the information completeness and accuracy. In the multimodal fusion method, the current research work can be divided into two categories according to the number of stages in the fusion process: one is direct fusion, which uses SAR images and multispectral images with clouds to directly extract features and complete the fusion; the other is two-stage indirect fusion, which first processes the SAR image input, converts it into an optical image, and then fuses it with the cloud image. Due to the obvious difference between SAR images and MSI, the information of a single spatial dimension of SAR images in the direct fusion method is insufficient compared with multispectral images with multiple spectral bands. After SAR images are converted into simulated multispectral images, they can provide more information for image fusion. At the same time, staged learning provides greater control and optimization for network training. However, due to the following problems, the two-stage indirect fusion method still has certain limitations and challenges: (1) At present, most models in the field of image fusion reconstruction are based on generative adversarial networks (GANs). However, the core framework and adversarial learning mechanism of GANs in their design lead to the inability of GAN methods to meet the fidelity requirements, that is, the predicted image only needs to deceive the discriminator with a certain probability without completely restoring the accurate image content. At the same time, there are also networks that are not based on GANs, but have completed more accurate detail reconstruction, such as methods that use the Transformer architecture or ideas. Its global attention design can provide global guidance for local detail recovery of cloud removal, but the design complexity is quadratically related to the input, which will cause problems when reconstructing large quantities of high-precision images. (2) Converting SAR images to optical images before fusion does alleviate the information gap to a certain extent, but most of the existing methods for converting SAR image modalities to optical modalities focus on converting SAR images into RGB images containing only three spectral bands. For MSI with multiple spectral bands, it contains more feature information than SAR images with only a single spatial dimension. Therefore, although existing methods can generate ideal simulated RGB images, it is still a difficult problem to effectively generate simulated MSI, making the converted MSI unable to provide effective feature information for the subsequent fusion stage.
[0005] In recent years, the research on cloud removal by direct fusion of SAR images and optical images has developed rapidly. Variants of deep learning networks such as GAN and Residual Neural Network (ResNet) have made great contributions in reconstructing missing information by direct fusion. Grohnfeldt et al. (GROHNFELDTC, SCHMITTM, ZHU XA conditional generative adversarial network to fuse SAR and multispectral optical data for cloud removal from sentinel-2images[C] / / IGARSS2018-2018IEEE International Geoscience and Remote Sensing Symposium.IEEE, 2018:1726-1729.) designed for the first time a conditional generative adversarial network (cGAN) specifically for fusing SAR images and MSI, introduced SAR images as an auxiliary, and generated images not covered by clouds from optical images containing clouds; SAR2OPT (BERMUDEZ J, HAPP P, OLIVEIRA D, et al. SAR to optical image synthesis for cloud removal with generative adversarial networks[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2018, 4: 5-11.) The cGANs network was trained using paired SAR and optical image patches, and other spectral bands in the image were reconstructed only based on SAR data during the prediction phase. Although GANs has a strong ability to generate images, its training on input data that is damaged over a large range will result in unstable predictions, and the detailed predictions of GANs are not close enough to the real situation because they only need to deceive the discriminator, resulting in unsatisfactory generation results.In order to improve the robustness in this aspect, Meraner et al. were inspired by ResNet (HE K, ZHANG X, RENS, et al. Deep residual learning for image recognition [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2016: 770-778.) and proposed a new deep residual neural network model DSen2CR (MERANER A, EBEL P, ZHU XX, et al. Cloud removal in sentinel-2imagery using a deep residual neural network and sar-optical data fusion [J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 166: 333-346.), which greatly improved the stability of prediction generation. However, since the residual cascade can only obtain incomplete information of SAR image data, the prediction effect of this method in specific areas will be reduced.
[0006] In addition to the direct fusion approach, there is also a two-stage fusion approach. In the first stage, the SAR image is first converted into a simulated optical cloud-free image. In the second stage, the simulated optical cloud-free image is fused with the cloud-containing optical image input to complete the reconstruction of the cloud area information. Gao et al. (GAO J, YUAN Q, LI J, et al. Cloud removal with fusion of high resolution optical and sar images using generative adversarial networks [J]. Remote Sensing, 2020, 12 (1): 191.) first used a simulated cloud layer in the optical image, designed a GAN-based network to remove the cloud, and finally applied the network to the optical image containing the real cloud layer. Compared with the direct fusion, the image conversion stage is the main factor causing the change in the network architecture. For this stage, there has been some extensive research work. For example, in the work based on cGAN (LI Y, FU R, MENG X, et al. A sar-to-optical image translation method based on conditional generation adversarial network (cGAN) [J]. IEEE Access, 2020, 8: 60338-60343.), the VV single-polarization SAR is converted into a three-channel optical image. As mentioned above, the current two-stage fusion strategy is mainly based on GAN implementation, so there are still common problems faced in direct fusion, namely unstable prediction and inaccurate details. Moreover, in the first image conversion stage, the existing methods only convert the optical image into a simulated optical image of three spectral bands, and the information provided is still limited, and it is impossible to achieve the optimal solution for information utilization in MSI de-clouding. Therefore, designing a network architecture that can balance the accuracy of the network model and the design complexity and can make full use of SAR information is an important issue in the field of two-stage fusion de-clouding. Summary of the invention
[0007] In view of the shortcomings of the prior art, the present invention proposes a SAR and MSI fusion declouding method based on an end-to-end two-stage conversion-fusion network. SAR images and MSI are used for image conversion and image fusion, and finally the cloud removal in MSI is completed. Considering the low fidelity of GAN and the high computational load of Transformer, the present invention selects a CNN model ConvNeXt (LIU Z, MAO H, WU CY, et al. A convnet for the 2020s [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 11976-11986.) as the network backbone. By referring to the network layer structure parameter design of Transformer to improve the key components in ResNet, ConvNeXt can be comparable to Transformer in various image processing tasks. The present invention uses the backbone network architecture of ConvNeXt to design a double U-shaped end-to-end neural network to achieve two-stage feature learning and overall supervision. Firstly, the basic units of encoder and decoder are constructed by ConvNeXt. In the first image conversion stage, the first U-type image conversion subnetwork with encoding and decoding is constructed to optimize the image conversion effect from SAR image to MSI and provide more comprehensive and useful information for the converted MSI. This conversion stage can provide deep structural information and accurate spectral information for the cloud part of the image in the subsequent image fusion stage. In the second fusion stage, the weighted three-source image fusion is completed, and the second U-type image fusion subnetwork with encoding and decoding is constructed to generate the final cloud-free MSI result. In this stage, the cloud mask weight (Cloud Mask Score, CMS) image is first calculated to provide reliable cloud area division, concentration and other information. Then, the probability weight provided by the weight image is used to weight the cloud part of the cloud MSI with the weight value provided by the weight map. The weighted superimposed image is input into the image fusion subnetwork to complete the image fusion adjustment, and finally the cloud-free MSI result is generated. Finally, the loss function is redesigned, and CMS is used to provide appropriate loss weights for the results of different stages, and the proposed two-stage fusion network is jointly optimized.By designing a loss function suitable for the network, the network training is guided and trained on the large-scale multimodal remote sensing dataset SEN12MS-CR (EBEL P, MERANER A, SCHMITT M, et al. Multisensor data fusion for cloud removal in global and all-season sentinel-2 imagery [J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59 (7): 5866-5878.).
[0008] The technical solution provided by the present invention is a SAR and multispectral image fusion declouding method, comprising the following steps:
[0009] Step 1, obtain a simulation data set and a real data set, both of which include cloud-free SAR images and cloud-covered multispectral MSI images;
[0010] Step 2, constructing a two-stage conversion-fusion de-clouding network, wherein the conversion-fusion de-clouding network includes two parts: an image conversion stage and an image fusion stage, and using the obtained simulated data set and the real data set as network input;
[0011] In the image conversion stage, the first U-shaped image conversion subnetwork with encoding and decoding is constructed. The image conversion subnetwork includes several image conversion layers, each of which includes an image conversion encoding basic unit and a decoding basic unit. Each image conversion encoding basic unit includes a downsampling operation of the image using a convolutional layer and a feature extraction and encoding block Block designed using ConvNeXt. Each decoding basic unit includes a Decoder that uses pixel rearrangement and transposed convolution to implement image upsampling and decoding operations, a channel dimension connection operation Concat and a feature extraction and decoding block Block designed using ConvNeXt. The first layer first inputs a cloud-free SAR image. The feature vector obtained after passing through the image conversion encoding basic unit of the last layer enters the decoding basic unit, and finally passes through a HardTanh activation function to obtain a simulated cloud-free MSI image.
[0012] In the image fusion stage, a second U-shaped image fusion subnetwork with encoding and decoding is constructed. The image fusion subnetwork includes several image fusion layers, each of which contains an image fusion encoding basic unit and a decoding basic unit. The structure of the image fusion encoding basic unit is the same as the image conversion encoding basic unit, and the decoding basic unit is shared with the image conversion subnetwork. The input of the image fusion subnetwork is a simulated cloud-free MSI image. Cloudy multispectral MSI image C opt The fused image after weighted superposition and merging operation with the cloud mask weight image CMS is output as the final de-clouding MSI result;
[0013] Step 3: Combine the loss function to train a two-stage conversion-fusion declouding network, and use the trained network to achieve declouding of cloudy multispectral images MSI.
[0014] Further, the processing process of the downsampling operation in the image conversion coding basic unit is as follows;
[0015] Assume that the eigenvector of the basic unit of the input image conversion coding is x i , where i represents the i-th layer; first, downsample the feature vector, using the feature that convolution can reduce the resolution, use a convolution kernel size greater than 3 and a step size greater than 1 to convolve the feature vector, assuming that the height of the feature vector is h in , with a width of w in , the convolution kernel size is k, the step size is s, then the height h of the output feature vector after convolution out , width w out Calculated by the following formula:
[0016]
[0017] in Represents the rounding down operation. Through this convolution operation, a downsampled feature vector is obtained. Where i represents the feature vector located at the i-th layer, and d represents the feature vector after downsampling operation, which is used to distinguish the feature vector from the input. Assuming the downsampling operation is Down(·), the process is expressed as:
[0018]
[0019] Furthermore, the processing process of the coding block Block in the image conversion coding basic unit is as follows;
[0020] The core of this encoding block is to use a large 7×7 convolution kernel to perform convolution. First, the input feature vector Use 1×1 convolution for channel adjustment, then convolution with 7×7 large convolution kernel, layer normalization and activation operation to get the output feature vector Where i represents the feature vector located in the i-th layer, b represents the output of the feature vector after passing through the Block, and is used for the feature vector with the same input Distinguish, for the input feature vector The process of channel adjustment is recorded as The convolution process with a large convolution kernel is recorded as The layer normalization is recorded as LN(·), and the activation uses the GELU function and is therefore recorded as GELU(·). The process is expressed as:
[0021]
[0022] The expression of the GELU activation function is:
[0023]
[0024] Where x represents the independent variable;
[0025] If the above process is recorded as Block(·), it can be simplified to:
[0026]
[0027] Furthermore, the specific processing of the decoding basic unit is as follows;
[0028] For the feature vector input into the decoding basic unit, the upsampling operation of the feature is first implemented by pixel rearrangement and transposed convolution, and then after layer normalization and GELU activation operations, the width and height of the feature vector are simultaneously enlarged by r times while keeping the total number of pixels of the feature vector unchanged by pixel rearrangement, and the number of channels becomes 1 / r of the original. 2 , after rearrangement, the feature vector is adjusted using the transposed convolution operation, the transposed convolution kernel size is 3, the step size is 1, and the padding is 1; assuming that the feature vector input by the decoding basic unit is Where i represents the feature vector located in the i-th layer, e represents the feature vector as the input of the Decoder, which is used to distinguish the symbols of other feature vectors, the pixel rearrangement operation is recorded as PS(·), and the transposed convolution operation is recorded as The output upsampled feature vector is recorded as Where i represents the feature vector at the i-th layer, and u represents the feature vector as the output of the Decoder, which is used to distinguish it from other feature vector symbols. The process is expressed as:
[0029]
[0030] After obtaining the upsampled feature vector, the feature vector is connected with the feature vector output by the previous image conversion coding basic unit in the channel dimension to further retain the underlying features. Then, the feature extraction and decoding block Block designed by ConvNeXt is used. Here, the decoding block Block has the same structure as the encoding block Block. After these operations, the output feature vector is Where i represents the feature vector located at the i-th layer, ud represents the output of the feature vector after the connection operation and the decoding block Block operation, which is used to distinguish it from other feature vector symbols. The connection operation is recorded as Concat. The above process is expressed as:
[0031]
[0032] Furthermore, the expression of the HardTanh activation function is:
[0033]
[0034] Where x represents the input.
[0035] Furthermore, the cloud mask generation method is used to calculate the probability value of each pixel in the cloudy multispectral image MSI as a cloud, which is denoted as in
[0036] Furthermore, the weighted stacking and merging operation aims to replace the cloudy pixels in the cloudy multispectral MSI image with the corresponding pixels of the simulated cloud-free multispectral MSI image output by the image conversion subnetwork, thus forming a pseudo cloud-free MSI image. As the input of the image fusion sub-network; the weighted superposition merging operation is expressed as follows:
[0037]
[0038] Among them, CMS, C opt , They are all images with the same width and height.
[0039] Furthermore, the loss function is defined by the following formula:
[0040]
[0041] in It consists of two parts, one of which is to simulate cloud-free losses. Part of it is the network prediction loss
[0042] exist In the loss, according to the smoothed loss The difference between the true value GT of the cloud-free multispectral image MSI and the simulated cloud-free multispectral image MSI is calculated, and γ1 = CMS is used as a guide for weight distribution. The weight distribution is performed by point-by-point multiplication to increase the weight of the cloudy area. The calculation is shown in the following formula:
[0043]
[0044] in, Represents the smoothing loss N ttp =H×W×C msi Represents the total number of pixels in the image, C msi is the number of channels of the simulated cloud-free multispectral image MSI, H × W is the size of the input SAR image;
[0045] and The loss is corresponding to Loss to evaluate GT and P out The difference between the two is calculated, and γ2 = 1-CMS is used as a guide for weight distribution to increase the weight of the cloud-free area. The calculation method is shown in the following formula:
[0046]
[0047] The present invention also provides a SAR and multispectral image fusion and cloud removal system, comprising a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a SAR and multispectral image fusion and cloud removal method as described in the above technical solution.
[0048] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: the present invention proposes a two-stage conversion-fusion declouding network model to address the two problems that can be improved and enhanced in the field of SAR image and MSI fusion declouding, namely, the inconsistency between the accuracy and design complexity of the network model and the insufficient utilization of SAR information. The network uses SAR information to remove the cloud layer in MSI and reconstruct the ground information of the cloudy area, while maintaining the integrity of the information of the cloud-free area during the prediction process. A large number of simulation and actual experimental results have proved the effectiveness and practicality of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 This is a two-stage conversion-convergence cloud-free network architecture diagram.
[0050] Figure 2 The following are test input examples of the datasets DatasetV1 and DatasetV2 of the embodiment.
[0051] Figure 3 This is a test output legend for the datasets DatasetV1 and DatasetV2 of the embodiment.
[0052] Figure 4 It is the visualization result of the intermediate and final output of the data set DatasetV1 of the embodiment.
[0053] Figure 5It is the visualization result of the intermediate and final output of the data set DatasetV2 of the embodiment. DETAILED DESCRIPTION
[0054] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0055] The present invention mainly aims at the demand of SAR and multispectral image fusion and de-clouding, and proposes a SAR and multispectral image fusion and de-clouding method and system based on an end-to-end two-stage conversion-fusion network. SAR information is used to remove the cloud layer in MSI, reconstruct the ground information of the cloudy area, and keep the integrity of the cloud-free area information in the prediction process. In the image conversion stage, the dual-wavelength information in the SAR image is converted to generate a simulated cloud-free MSI containing multiple spectral segments, and the spatial features of the SAR image are converted into features of an approximate optical image and generate spectral dimension information. In this stage, the present invention establishes a U-shaped image conversion subnetwork, which can extract and convert features at different scales, and the output simulated MSI has clear spatial information and reliable spectral information. Between the image fusion stage and the conversion stage, the simulated cloud-free MSI, the cloud mask weight CMS and the cloud MSI input are weighted superimposed to obtain a pseudo cloud-free MSI image, which is then further processed in the image fusion stage to generate the final predicted de-clouded image. In order to improve the de-clouding effect of the network, the present invention designs a loss function to optimize the network training, balance the learning center of gravity and learning direction of the network in the two stages, reduce detail blur, and improve reconstruction accuracy.
[0056] Step 1: Design the basic units for each layer of the encoder part in the two-stage sub-network, including the downsampling operation of the image using the convolutional layer and the feature extraction and encoding blocks designed using ConvNeXt;
[0057] Step 2: Design the basic units for each layer of the decoder part in the two-stage sub-network, including the Decoder that uses pixel rearrangement and transposed convolution to implement image upsampling and decoding operations, the channel dimension connection operation Concat, and the feature extraction and decoding block Block designed using ConvNeXt.
[0058] Step three, construct a two-stage conversion-fusion de-clouding network. The network uses ConvNeXt as the network backbone and consists of an image conversion subnetwork and an image fusion subnetwork. Both subnetworks are U-shaped and symmetrical, sharing the same decoder. The two stages are divided into an image conversion stage and an image fusion stage. A weighted superposition merging operation is used between the two stages to fully utilize the information of SAR images and MSI.
[0059] Step 4: Design a loss function, in which CMS is introduced. Combined with the loss function, the SAR and multispectral image fusion declouding network is trained, and the trained network is used to remove clouds from remote sensing images.
[0060] Step 1: The basic unit structure diagram of the encoder is as follows Figure 1 As shown, assuming that the feature vector of the basic unit of the input encoder is x i (i=1,2,3,4), where i represents the feature vector of the basic unit of the i-th layer in the feature input network. First, the feature vector is downsampled. By using the feature that convolution can reduce the resolution, the feature vector is convolved with a convolution kernel size greater than 3 and a step size greater than 1. Assume that the height of the feature vector is h in , with a width of w in , the convolution kernel size is k, the step size is s, then the height h of the output feature vector after convolution out , width w out It can be calculated by the following formula:
[0061]
[0062] in Indicates the rounding down operation. Through this convolution operation, a downsampled feature vector can be obtained. Where i represents the feature vector located at the i-th layer, and d represents the feature vector after downsampling operation, which is used to distinguish the feature vector from the input. Assuming the downsampling operation is Down(·), the process can be expressed as:
[0063]
[0064] In practice, the convolution kernel k used in the first layer downsampling operation of the present invention is 4 in size and the step size k is 4, which makes the feature vector downsampled by 4 times in this layer. The convolution kernel k used in the 2nd, 3rd, and 4th layer downsampling operations is 2 in size and the step size k is 2, which makes the feature vector downsampled by 2 times in this layer.
[0065] Furthermore, we construct a feature extraction and encoding block designed using ConvNeXt. The core of this unit block is to perform convolution using a large 7×7 convolution kernel. First, the input feature vector Use 1×1 convolution for channel adjustment, then convolution with 7×7 large convolution kernel, layer normalization and activation operation to get the output feature vector Where i represents the feature vector located in the i-th layer, b represents the output of the feature vector after passing through the Block, and is used for the feature vector with the same input Distinguish, for the input feature vector The process of channel adjustment is recorded as The convolution process with a large convolution kernel is recorded as The layer normalization is recorded as LN(·), and the activation uses the GELU function and is therefore recorded as GELU(·). The process can be expressed as:
[0066]
[0067] The expression of the GELU activation function is:
[0068]
[0069] If the above process is recorded as Block(·), the process can be simplified as follows:
[0070]
[0071] The large convolution kernel in this basic unit is convolution The convolution kernel size k is 7, the step size s is 1, and the padding p is 3.
[0072] Step 2: The basic unit structure diagram of the decoder is as follows Figure 2 As shown in the figure, we first construct a decoder that uses pixel rearrangement and transposed convolution to implement image upsampling and decoding operations. For the feature vector input to this unit, we first use pixel rearrangement and transposed convolution to implement feature upsampling operations, and then go through layer normalization and GELU activation operations. By using pixel rearrangement, while keeping the total number of pixels of the feature vector unchanged, the width and height of the feature vector can be enlarged by r times at the same time, and the number of channels becomes 1 / r of the original. 2 , after rearrangement, the feature vector needs to be adjusted using the transposed convolution operation, with a transposed convolution kernel size of 3, a step size of 1, and a padding of 1. Assume that the feature vector input to this unit is Where i represents the feature vector located in the i-th layer, e represents the feature vector as the input of the Decoder, which is used to distinguish the symbols of other feature vectors, the pixel rearrangement operation is recorded as PS(·), and the transposed convolution operation is recorded as The output upsampled feature vector is recorded as Where i represents the feature vector at the i-th layer, and u represents the feature vector as the output of the Decoder, which is used to distinguish it from other feature vector symbols. The process can be expressed as:
[0073]
[0074] In practice, the present invention sets r for the pixel rearrangement used in the first layer upsampling to 4, corresponding to the 4-fold downsampling of the first layer in the encoder, and sets r for the pixel rearrangement used in the second, third, and fourth layers upsampling to 2, also corresponding to the downsampling ratio of the corresponding number of layers in the encoder. The purpose of this is to enable the network to complete the Concat operation normally.
[0075] After obtaining the upsampled feature vector, the feature vector is connected to the feature vector output by the previous encoder unit in the channel dimension to further retain the underlying features. Then, the feature extraction and decoding block Block designed by ConvNeXt is used. The Block structure here is consistent with the Block structure in the encoder, so it will not be repeated. After these operations, the output feature vector is Here, i represents that the feature vector is located in the i-th layer, ud represents that the feature vector is the output after connection and Block, which is used to distinguish it from other feature vector symbols. The connection operation is recorded as Concat. The above process can be expressed as:
[0076]
[0077] In practice, the Block structure parameters here are the same as those in the above steps. Figure 1 The Block in will not be described in detail.
[0078] Step three: Construct an end-to-end two-stage conversion-fusion de-clouding network with ConvNeXt as the network backbone, including the modules constructed by the above two steps. The end-to-end de-clouding network consists of two stages, namely the image conversion stage and the image fusion stage. The present invention uses ConvNeXt (tiny) as the backbone of the network. The backbone network is a convolutional network that has been redesigned based on the specific layer design parameters of the Transformer. After the redesign, it not only retains the original advantages of the convolutional network in the field of image processing, that is, it can avoid the contradiction between speed and system design faced by the Transformer in large-scale input and feature extraction, but also can achieve image processing effects comparable to the Transformer, with higher accuracy, robustness and scalability.
[0079] As mentioned above, the two-stage conversion-fusion de-clouding network constructed by the present invention is composed of two sub-networks, the conversion network and the fusion network, and the two sub-networks are symmetrical in structure, both are U-shaped networks, and share the same decoding basic unit in the decoding stage, and the two sub-networks have the same number of layers. After the conversion network outputs, the feature vector is not directly input into the fusion network, but first uses the calculated CMS to weightedly superimpose the cloud MSI and the simulated cloudless MSI output by the conversion network and then input into the fusion sub-network.
[0080] First, an image conversion subnetwork is constructed. Each layer can be recorded as Layeri (i=1, 2, 3, 4). The first input of the first layer of the network is a cloud-free SAR image. In the encoding stage, the image passes through the basic units constructed in step 1 in sequence. The network has four layers in total, that is, the input passes through downsampling and Block blocks in sequence, and is executed 4 times in total. In this process, as the number of layers increases, deeper feature information will be mined and encoded, thereby effectively providing effective feature information for the subsequent decoding process.
[0081] Furthermore, after the feature vector is output by the last layer of encoding basic units, it enters the decoding stage. This stage still has a 4-layer structure, but compared with the encoding stage, the width and height of the feature vector gradually increase. The final output feature vector passes through a HardTanh activation function, which is a simulated cloud-free MSI output of the converted SAR image. The expression of the HardTanh activation function is:
[0082]
[0083] In order to facilitate the explanation of network design, the present invention uses represents the input SAR image, where H×W is the size of the image, C sar is the number of channels of the SAR image. The converted simulated cloud-free MSI ( C msi is the number of channels corresponding to the cloud MSI image) can be obtained by the following formula:
[0084]
[0085] Among them, F translation (·) represents the image conversion process, including the first 4 layers of encoding basic units, 4 layers of decoding basic units and the last HardTanh activation mentioned in the above process.
[0086] Furthermore, the input form of the sub-network is clarified before constructing the image fusion sub-network, and the simulated cloud-free MSI image is obtained after passing through the above conversion sub-network. Next, we need to use the weighted stacking operation to fuse the three-source images, that is, the converted simulated cloud-free MSI There is cloud MSI input (C opt ) and cloud mask weight (CMS) image. First, according to the cloud mask generation method adopted by Meraner et al., the probability value of each pixel in the cloud MSI being a cloud is calculated, which is denoted as Thus we get the CMS image That is, for each pixel at each coordinate, there is a probability value about whether it has clouds, which is used as the weight value for the subsequent weighted superposition merging. The purpose of the weighted superposition merging operation is to replace the cloudy pixels in the cloudy MSI with the corresponding pixels of the simulated cloud-free MSI output by the image conversion subnetwork, forming a pseudo cloud-free MSI image. The weighted overlay merging operation can be summarized as follows:
[0087]
[0088] Among them, CMS, C opt , They are all images with the same width and height. Therefore, the operation represented by the above formula represents the relevant operation of pixels at the corresponding coordinates. Therefore, the width and height of the pseudo cloud-free MSI image obtained are also consistent with the above three images.
[0089] Furthermore, we construct an image fusion subnetwork. As mentioned above, the two subnetworks have symmetric structures, that is, the network forms of the two are consistent, but the difference is that the input and output feature vectors are inconsistent. Therefore, we will not go into details here, and directly give the expression form. The input of the fusion network is the pseudo cloud-free MSI image obtained in the above steps. The final predicted cloud-free MSI output is denoted as P out , image fusion processing is F fusion (·), the above steps can be summarized as follows:
[0090]
[0091] Step 4: The loss function of the present invention is inspired by the attention mechanism and introduces CMS into network training to better control the network learning direction. The loss function is defined by the following formula:
[0092]
[0093] From the above formula, we can directly get It consists of two parts, one of which is to simulate cloud-free losses. Part of it is the network prediction loss Each part will be explained below.
[0094] First, in In the loss, according to the smoothed loss Calculate the true cloud-free MSI value (GT) and simulate cloud-free The difference is calculated, and γ1 = CMS is used as a guide for weight distribution. The weight distribution is performed by point-by-point multiplication to increase the weight of the cloudy area. The loss is calculated as shown in the following formula:
[0095]
[0096] in, Represents the smoothing loss N ttp =H×W×C msi Represents the total number of pixels in the image.
[0097] and The loss is corresponding, through Loss to evaluate GT and P out The difference between the two is calculated, and γ2=1-CMS is used as a guide for weight distribution to increase the weight of the cloud-free area. The calculation method of this loss is shown in the following formula:
[0098]
[0099] We selected SEN12MS-CR as the data source to further generate simulated and real datasets. SEN12MS-CR is based on the SEN12MS (SCHMITT M, HUGHES LH, QIU C, et al. SEN12MS-a curated dataset of georeferenced multi-spectral sentinel-1 / 2 imagery for deep learning and data fusion [J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2019, IV-2 / W7: 153-160.) dataset, and conducted a preliminary screening to manually remove the lossy images caused by image acquisition, image cutting, etc. The original SEN12MS dataset provides more than 180,000 samples from non-overlapping target regions (Regions of Interest, ROI), which are distributed in all continents with traces of inhabitants and all meteorological seasons in the world. Each sample is a triplet of dual-wavelength SAR image patches, MSI image patches acquired by Sentinel-2, and MODIS land cover maps, breaking through the limitations of spatial coverage, land diversity, or the number of available samples. The SEN12MS-CR dataset consists of 169 non-overlapping ROIs with equal probability distribution across all continents and seasons, containing more than 120,000 data samples, each of which consists of a triplet of Sentinel-2 cloudy and cloudless MSI and the corresponding Sentinel-1 SAR image. Among them, the SAR data provided by Sentinel-1 contains polarization information in the VV and VH bands, and the MSI data provided by Sentinel-2 has 13 spectral bands. The data size of each image is 256×256px2 (unit: pixel, Pixel, abbreviated as px), and each spectral band of the SAR image and MSI is rescaled to the interval [0,1]. That is, the two polarization bands VV and VH of the SAR image are clipped to [-25.0,0] and [-32.5,0] respectively, and then scaled to the range of [0,1]; and all bands of the MSI are first clipped to [0,10000] and also rescaled to the range of [0,1]. In addition, the present invention also performs median filtering on the SAR image to suppress speckle noise. Since it was found during the inspection of the data set that the SEN12MS-CR data set still contains some images with and without clouds whose contents are greatly different due to different acquisition times, the present invention screened the data set again to avoid such data.Finally, the present invention selects part of the data from the dataset to form two sub-datasets, which are respectively recorded as DatasetV1 and DatasetV2. The SEN12MS-CR dataset consists of four parts, which are divided into spring, summer, autumn and winter according to the season, each constituting an independent group, and each group contains images acquired in the corresponding season in the form of multiple sub-groups (corresponding to multiple ROIs). DatasetV1 consists of 2822 images under the first five ROIs (1, 6, 8, 9, 15) in the original spring group (ROIs1158_spring) as a training set and 100 images under two ROIs (8, 9) as a test set; DatasetV2 consists of 2000 images in 10 ROIs evenly distributed under the original spring group as a training set and 100 images as a test set.
[0100] During the network training process, the neural network model designed by the present invention is trained on a single NVIDIA RTX3090Ti GPU (24GB) graphics card, the batch size is set to 16, and the maximum number of training iterations is set to 1000. The optimizer uses the AdamW optimizer with a learning rate of 0.001. In order to enhance the data used in training, during the training process, the original input data of size 256×256px2 is randomly cropped to 128×128px2 for each iteration.
[0101] Using the SEN12MS-CR dataset as the data source, we selected some data to form two sub-datasets for training, which are denoted as DatasetV1 and DatasetV2. DatasetV1 consists of 2822 images from the first five ROIs (1, 6, 8, 9, 15) in the original spring grouping (ROIs1158_spring) as the training set and 100 images from two ROIs (8, 9) as the test set; DatasetV2 consists of 2000 images from 10 ROIs evenly distributed under the original spring grouping as the training set and 100 images as the test set. Examples of input images of the two test sets are shown in the attached figure. Figure 2 The final processing results on the two data sets are shown in the attached Figure 3 For the output results of more images of the first dataset DatasetV1 and the visualization results of its intermediate variables, see the attached Figure 4 As shown in the figure. For the output results of more images of the second dataset DatasetV2 and the visualization results of its intermediate variables are shown in the figure. Figure 5 shown.
[0102] On the other hand, an embodiment of the present invention also provides a SAR and multispectral image fusion and cloud removal system, including a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a SAR and multispectral image fusion and cloud removal method as described in the above scheme.
[0103] It should be understood that parts not elaborated in detail in this specification belong to the prior art.
[0104] It should be understood that the above description of the embodiments is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.
Claims
1. A method for cloud removal by fusion of SAR and multispectral images, characterized in that: The steps include: Step 1, obtain a simulation data set and a real data set, both of which include cloud-free SAR images and cloud-covered multispectral MSI images; Step 2, constructing a two-stage conversion-fusion de-clouding network, wherein the conversion-fusion de-clouding network includes two parts: an image conversion stage and an image fusion stage, and using the obtained simulated data set and the real data set as network input; In the image conversion stage, the first U-shaped image conversion subnetwork with encoding and decoding is constructed. The image conversion subnetwork includes several image conversion layers, each of which includes an image conversion encoding basic unit and a decoding basic unit. Each image conversion encoding basic unit includes a downsampling operation of the image using a convolutional layer and a feature extraction and encoding block Block designed using ConvNeXt. Each decoding basic unit includes a Decoder that uses pixel rearrangement and transposed convolution to implement image upsampling and decoding operations, a channel dimension connection operation Concat and a feature extraction and decoding block Block designed using ConvNeXt. The first layer first inputs a cloud-free SAR image. The feature vector obtained after passing through the image conversion encoding basic unit of the last layer enters the decoding basic unit, and finally passes through a HardTanh activation function to obtain a simulated cloud-free MSI image. In the image fusion stage, a second U-shaped image fusion subnetwork with encoding and decoding is constructed. The image fusion subnetwork includes several image fusion layers, each of which contains an image fusion encoding basic unit and a decoding basic unit. The structure of the image fusion encoding basic unit is the same as the image conversion encoding basic unit, and the decoding basic unit is shared with the image conversion subnetwork. The input of the image fusion subnetwork is a simulated cloud-free MSI image. Cloudy multispectral MSI image C opt The fused image after weighted superposition and merging operation with the cloud mask weight image CMS is output as the final de-clouding MSI result; Step 3: Combine the loss function to train a two-stage conversion-fusion declouding network, and use the trained network to achieve declouding of cloudy multispectral images MSI.
2. The cloud removal method for SAR and multispectral image fusion as claimed in claim 1, characterized in that: The processing process of the downsampling operation in the image conversion coding basic unit is as follows; Assume that the eigenvector of the basic unit of the input image conversion coding is x i , where i represents the i-th layer; first, downsample the feature vector, using the feature that convolution can reduce the resolution, use a convolution kernel size greater than 3 and a step size greater than 1 to convolve the feature vector, assuming that the height of the feature vector is h in , with a width of w in , the convolution kernel size is k, the step size is s, then the height h of the output feature vector after convolution out , width w out Calculated by the following formula: in Represents the rounding down operation. Through this convolution operation, a downsampled feature vector is obtained. Where i represents the feature vector located at the i-th layer, and d represents the feature vector after downsampling operation, which is used to distinguish the feature vector from the input. Assuming the downsampling operation is Down(·), the process is expressed as:
3. The cloud removal method for SAR and multispectral image fusion as claimed in claim 1, characterized in that: The processing process of the coding block Block in the image conversion coding basic unit is as follows; The core of this encoding block is to use a large 7×7 convolution kernel to perform convolution. First, the input feature vector Use 1×1 convolution for channel adjustment, then convolution with 7×7 large convolution kernel, layer normalization and activation operation to get the output feature vector Where i represents the feature vector located in the i-th layer, b represents the output of the feature vector after passing through the Block, and is used for the feature vector with the same input Distinguish, for the input feature vector The process of channel adjustment is recorded as The convolution process with a large convolution kernel is recorded as The layer normalization is recorded as LN(·), and the activation uses the GELU function and is therefore recorded as GELU(·). The process is expressed as: The expression of the GELU activation function is: Where x represents the independent variable; If the above process is recorded as Block(·), it can be simplified to:
4. The cloud removal method for SAR and multispectral image fusion as claimed in claim 1, characterized in that: The specific processing process of the decoding basic unit is as follows; For the feature vector input into the decoding basic unit, the upsampling operation of the feature is first implemented by pixel rearrangement and transposed convolution, and then after layer normalization and GELU activation operations, the width and height of the feature vector are simultaneously enlarged by r times while keeping the total number of pixels of the feature vector unchanged by pixel rearrangement, and the number of channels becomes 1 / r of the original. 2 , after rearrangement, the feature vector is adjusted using the transposed convolution operation, the transposed convolution kernel size is 3, the step size is 1, and the padding is 1; assuming that the feature vector input by the decoding basic unit is Where i represents the feature vector located in the i-th layer, e represents the feature vector as the input of the Decoder, which is used to distinguish the symbols of other feature vectors, the pixel rearrangement operation is recorded as PS(·), and the transposed convolution operation is recorded as The output upsampled feature vector is recorded as Where i represents the feature vector at the i-th layer, and u represents the feature vector as the output of the Decoder, which is used to distinguish it from other feature vector symbols. The process is expressed as: After obtaining the upsampled feature vector, the feature vector is connected with the feature vector output by the previous image conversion coding basic unit in the channel dimension to further retain the underlying features. Then, the feature extraction and decoding block Block designed by ConvNeXt is used. Here, the decoding block Block has the same structure as the encoding block Block. After these operations, the output feature vector is Where i represents the feature vector located at the i-th layer, ud represents the output of the feature vector after the connection operation and the decoding block Block operation, which is used to distinguish it from other feature vector symbols. The connection operation is recorded as Concat. The above process is expressed as:
5. The cloud removal method for SAR and multispectral image fusion as claimed in claim 1, characterized in that: The expression of HardTanh activation function is: Where x represents the input.
6. The cloud removal method for SAR and multispectral image fusion according to claim 1, characterized in that: The cloud mask generation method is used to calculate the probability value of each pixel in the cloudy multispectral image MSI as a cloud, which is denoted as in 7. The cloud removal method for SAR and multispectral image fusion according to claim 1, characterized in that: The purpose of the weighted stacking and merging operation is to replace the cloudy pixels in the cloudy multispectral MSI image with the corresponding position pixels of the simulated cloud-free multispectral MSI image output by the image conversion subnetwork to form a pseudo cloud-free MSI image. As the input of the image fusion sub-network; the weighted superposition merging operation is expressed as follows: Among them, CMS, C opt , They are all images with the same width and height.
8. The cloud removal method for SAR and multispectral image fusion as claimed in claim 1, characterized in that: The loss function is defined by the following formula: in It consists of two parts, one of which is to simulate cloud-free losses. Part of it is the network prediction loss exist In the loss, according to the smoothed loss The difference between the true value GT of the cloud-free multispectral image MSI and the simulated cloud-free multispectral image MSI is calculated, and γ1 = CMS is used as a guide for weight distribution. The weight distribution is performed by point-by-point multiplication to increase the weight of the cloudy area. The calculation is shown in the following formula: in, Represents the smoothing loss N ttp =H×W×C msi Represents the total number of pixels in the image, C msi is the number of channels of the simulated cloud-free multispectral image MSI, H × W is the size of the input SAR image; and The loss is corresponding, through Loss to evaluate GT and P out The difference between the two is calculated, and γ2 = 1-CMS is used as a guide for weight distribution to increase the weight of the cloud-free area. The calculation method is shown in the following formula:
9. A SAR and multispectral image fusion cloud removal system, characterized by: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a SAR and multispectral image fusion declouding method as claimed in any one of claims 1 to 8.
Citation Information
Cited By
Remote sensing thick cloud image restoration method and system based on multi-scale feature fusion
CN120782679A
