SAR and optical image fusion method and system based on frequency-space interaction

Abstract

The invention discloses an SAR and optical image fusion method and system based on frequency-space interaction, and belongs to the technical field of computer vision and deep learning, and the method comprises the steps: obtaining SAR and optical remote sensing image data which are matched in pairs; inputting the acquired SAR image data into a pre-trained SAR-to-optical image conversion network to obtain pseudo-optical image data; inputting the SAR and optical images which are matched in pairs and the pseudo-optical image into the trained fusion model to obtain an SAR and optical fusion image; and correcting the single-channel fusion image by means of a mask to obtain an RGB color image fusing the spectral information and spatial distribution characteristics of the optical image and the structure texture details of the SAR image. According to the method, the problem that an effective SAR and optical image feature extraction method and fusion strategy are lacked is solved, the defect that the fusion result is limited by the expression of the image to be fused is overcome, reasonable and effective feature relation and representation modeling are achieved, and high-quality SAR and optical image fusion is guaranteed.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and deep learning, and particularly relates to a SAR and optical image fusion method and system based on frequency-spatial interaction. Background Art

[0002] Optical remote sensing images and SAR (Synthetic Aperture Radar) images, as two main types of remote sensing satellite image data, play important roles in fields such as geospatial analysis, natural resource monitoring, and military reconnaissance. Optical remote sensing images are obtained based on passive imaging technology and can provide high-resolution spatial distribution information and spectral features for ground object observation, but are susceptible to factors such as clouds, fog, and lighting. SAR images are obtained relying on active microwave remote sensing technology, are not affected by factors such as clouds, fog, and lighting, can provide ground information at night or under adverse weather conditions, and can support continuous and stable earth monitoring. However, SAR images face problems of speckle noise interference and poor visual interpretability, and are difficult to directly apply in earth monitoring tasks. Therefore, how to design an effective SAR and optical image pixel-level fusion method, fully combine the advantages of both, support high-resolution, highly interpretable, continuous, and stable earth monitoring tasks, and provide high-quality data representation for subsequent various tasks based on geospatial analysis is worthy of further exploration.

[0003] The existing SAR and optical image pixel-level fusion research methods can be divided into two major categories: traditional methods and deep learning-based methods. Among them, traditional methods mainly draw on the idea of multi-spectral image and panchromatic image fusion, and can be specifically divided into: image fusion methods based on the spatial domain, image fusion methods based on the transform domain, and image fusion methods based on model-driven. Pixel-level fusion methods based on deep learning mainly include pixel-level fusion methods based on convolutional neural networks (CNNs) and pixel-level fusion methods based on generative adversarial networks (GANs). Although the existing methods have made significant contributions to the research of SAR and optical image fusion, they face the following problems: (1) The feature extraction of traditional methods relies on manually designed filters. In addition, the fusion of multi-source features also requires manually constructing fusion rules or fusion knowledge bases. However, how to design an ideal feature extraction method and fusion strategy is a daunting task, and the fusion effect of traditional methods mostly highly depends on the correlation between the images to be fused, making it difficult to achieve a highly generalized fusion algorithm to ensure the overall performance and robustness of remote sensing images in various applications. (2) In the existing pixel-level fusion methods based on deep learning, pixel-level fusion methods based on CNNs often rely on ground truth data with labels to supervise the training of the network model to obtain the fused image; or design a loss function to constrain the retention and expression of feature information in the fused image. However, the complexity of SAR images makes the construction of data labels time-consuming and laborious, and the fused results obtained only by constraining with the loss function are limited by the expression defects of the images to be fused themselves (for example, the missing color information in the areas covered by clouds in optical images is difficult to directly supplement with SAR images). (3) Pixel-level fusion methods based on generative adversarial networks (GANs) can effectively overcome the defect that supervised learning depends on a large amount of labeled data. At the same time, it also provides a new idea for translating SAR images into optical images to assist subsequent multi-source remote sensing image fusion, using GANs to enhance the correlation between images, and proving the feasibility of information fusion through image translation. However, the quality of the fused images finally obtained by the existing methods depends on the accuracy of image translation, lacking reasonable and effective feature relationship and representation modeling.

[0004] Therefore, it is necessary to design a SAR and optical image fusion method and system based on frequency-space interaction for the above problems. Summary of the Invention

[0005] The objective of the present invention is to address the problems existing in the prior art and provide a frequency-space interaction-based SAR and optical image fusion method and system, which realizes high-quality SAR and optical image fusion based on the interaction between the frequency domain and the spatial domain. This network adopts three types of image information in data input, including: paired SAR and optical images, and pseudo-optical images obtained by using a pre-trained SAR-to-optical image conversion network. The input images are respectively passed to the subsequent frequency-space interaction fusion module through a data preprocessing module, so as to extract and fuse the features of each modality at multiple scales and levels. At the same time, in order to make up for the lack of details caused by factors such as clouds and fog in optical images, a mask-based correction strategy is introduced into the network, and the anti-interference advantage of SAR images is used to effectively compensate for the occluded areas. Through this multi-modal fusion method, while retaining the spectral information of the optical image, the present invention fully integrates the structural and texture features of SAR images, thereby obtaining a fused image with higher fidelity and detail expressiveness.

[0006] According to one aspect of this specification, a frequency-space interaction-based SAR and optical image fusion method is provided, including: Obtain paired SAR and optical remote sensing image data; Input the obtained SAR image data into a pre-trained SAR-to-optical image conversion network to obtain pseudo-optical image data; Input the paired SAR and optical images and the pseudo-optical images into a trained fusion model to obtain a SAR and optical fused image; wherein, the fusion model includes: a data preprocessing module, which is used to preliminarily process the input paired SAR and optical images and pseudo-optical images, and generate a feature embedding space to obtain multi-source embedding features; a frequency-space interaction fusion module, which is used to realize the feature extraction and fusion of multi-source embedding features at multiple levels and scales to obtain fusion features and multi-level pseudo-optical features; a mask-based correction module, which is used to supplement and correct by using the pseudo-optical image to finally obtain a single-channel fused image; Use a mask to correct the single-channel fused image to obtain an RGB color image that integrates the spectral information, spatial distribution characteristics of the optical image, and the structural texture details of the SAR image.

[0007] Furthermore, the data preprocessing module preliminarily processes the input paired SAR and optical images and pseudo-optical images, including: Image preprocessing process: perform gray-scale conversion and normalization processing on the input optical and pseudo-optical images; perform normalization processing on the input SAR image; Feature embedding process: use a convolutional module to process the preprocessed multi-source images to obtain multi-source embedding features, and the multi-source images are the optical image, pseudo-optical image, and SAR image after normalization processing.

[0008] Further, the grayscale conversion process converts a 3-channel optical or pseudo-optical RGB image into a single-channel grayscale image, expressed as:

[0009]

[0010] Wherein, and respectively represent the single-channel grayscale images after the conversion of the optical image and the pseudo-optical image. , , and s , , are respectively the 3 RGB channels of the optical image and the pseudo-optical image; the normalization process is used to scale the data to the interval, expressed as:

[0011] Wherein, represents the original image, represents the image after the normalization process.

[0012] Further, the convolution module consists of a convolutional layer, a BatchNorm normalization layer, and a ReLU activation function layer.

[0013] Further, the construction of the frequency-space interaction fusion module includes 3 stages, and each stage includes: Spatial domain feature extraction sub-module: composed of multiple parallel ResNet convolution modules, which are respectively used to extract the spatial features of optical, SAR, and pseudo-optical images; Frequency domain fusion sub-module: For the embedded features of optical and SAR images, the discrete Fourier transform is respectively used to map them to the frequency domain, so as to extract the phase spectrum and amplitude spectrum respectively; then the phase spectrum and amplitude spectrum are respectively fused to obtain the fused phase feature and amplitude feature; through the inverse discrete Fourier transform, the fused phase and amplitude features are remapped to the spatial domain and passed through the convolution module to obtain the final frequency domain fusion feature; Optical, SAR, and pseudo-optical image features enter each stage of the spatial domain feature extraction sub-module. After calculation, deeper optical, SAR, and pseudo-optical image features are obtained and then fed into the spatial domain feature extraction sub-module of the next stage. All pseudo-optical image features are concatenated along the channel dimension to obtain multi-level pseudo-optical features. The embedded features of the optical and SAR images enter the frequency domain fusion sub-module of the first stage, and the first frequency domain fusion feature is calculated. The optical and SAR image features calculated by the spatial domain feature extraction sub-module of the first stage enter the frequency domain fusion sub-module of the second stage, and the second frequency domain fusion feature is calculated. Similarly, the optical and SAR image features calculated by the spatial domain feature extraction sub-module of the second stage enter the frequency domain fusion sub-module of the third stage, and the third frequency domain fusion feature is calculated. The three frequency domain fusion features form a frequency domain fusion feature set. The three frequency domain fusion features are concatenated along the channel dimension and processed by a convolution module to obtain the fusion feature.

[0014] Further, the construction of the mask-based correction module includes: Mask generation sub-module based on the dark channel: Calculate the dark channel prior based on the original optical image, distinguish the cloud area from the non-cloud area, and generate the corresponding grayscale mask. Pseudo-optical feature equalization process: Introduce mean-standard deviation correction to standardize the feature distribution of the pseudo-optical image so that it is consistent with the fusion feature in the numerical scale. Correction fusion sub-module: The standardized pseudo-optical image features and the fusion features are fed into the correction fusion sub-module, and fusion is achieved using the mask as the weight to obtain a single-channel fusion image.

[0015] Further, the input of the correction fusion sub-module is the multi-level pseudo-optical features after standardization , the fusion feature and the grayscale cloud mask Mask; Fusion is achieved using Mask as the weight, and the fusion result is concatenated with the fusion feature input to the correction fusion module along the feature channel dimension and then passed through a convolution module to obtain a single-channel fusion image ; The implementation process is expressed as:

[0016] where represents concatenating features along the channel dimension of the feature, represents the convolution module.

[0017] Further, correcting the single-channel fusion image with the mask includes: For each pixel of the fused RGB color image , when the corresponding grayscale cloud mask Mask satisfies the condition, the pixel value of the pseudo-optical image is used to replace the pixel value of the original optical image; otherwise, the information of the original optical image is retained, and the implementation process is expressed as:

[0018]

[0019] Among them, are the horizontal and vertical coordinates of the pixel point, , , and , , are the R, G, and B channels of the optical image and the fused image respectively, That is, the threshold in the cloud area detection-generated mask.

[0020] Furthermore, the loss function used to train the fusion model is divided into two stages: The first-stage loss function ensures that the network learns the frequency-space interaction fusion ability, which is expressed as:

[0021] Among them, represents the set of frequency-domain fusion features, and are the single-channel optical and SAR images respectively, is an important weight for balancing the loss of structural information and pixel information; is the structural information loss. Using the metric based on structural similarity, the structural similarities between the optical image and the SAR image and the frequency-domain fusion features at each level are calculated respectively; is the pixel information loss. Based on the mean square error, the pixel-level distances between the optical image and the frequency-domain fusion features at each level are calculated; The second-stage loss function ensures that the network learns the high-quality reconstruction ability of the fused image, which is expressed as:

[0022] Among them, and are the weight parameters for balancing pixel information and fusion comprehensive loss respectively; is the fusion comprehensive loss, which is expressed as:

[0023]

[0024]

[0025] Among them, is the image average intensity loss, is the image average gradient loss, and respectively represent the height and width of the fused image, i.e., the single-channel fused image, and are the SAR and the single-channel optical image; through the maximum selection strategy integrates the pixel intensity distributions of the SAR and the single-channel optical image, and uses the integral distribution to constrain the pixel intensity distribution of the fused image; is the Sobel operator, is the tuning parameter.

[0026] The present invention also provides a SAR and optical image fusion system based on frequency-space interaction, including: A data acquisition unit for acquiring paired matching SAR and optical remote sensing image data; A pseudo-optical image generation module for inputting the acquired SAR image data into a pre-trained SAR-to-optical image conversion network to obtain pseudo-optical image data; An image fusion model construction unit for inputting the paired matching SAR and optical images and the pseudo-optical image into a trained fusion model to obtain a SAR and optical fused image; wherein, the fusion model includes: a data preprocessing module for preliminarily processing the input paired matching SAR and optical images and the pseudo-optical image, and generating a feature embedding space to obtain multi-source embedding features; a frequency-space interaction fusion module for implementing feature extraction and fusion of the multi-source embedding features at multiple levels and multiple scales to obtain fusion features and multi-level pseudo-optical features; a mask-based correction module for using the pseudo-optical image for supplementary correction to finally obtain a single-channel fused image; A fused image correction unit for correcting the single-channel fused image with the help of a mask to obtain an RGB color image that fuses the spectral information, spatial distribution characteristics of the optical image, and the structural texture details of the SAR image.

[0027] According to one aspect of the present specification, there is provided an electronic device including a memory and a processor, the memory storing a computer program, characterized in that when the processor executes the computer program, the steps of the above-mentioned SAR and optical image fusion method based on frequency-space interaction are implemented.

[0028] According to one aspect of the present specification, there is provided a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned SAR and optical image fusion method based on frequency-space interaction are implemented.

[0029] According to one aspect of the specification of the present invention, there is provided a computer program product including instructions, which, when running on a computer, causes the computer to execute the steps of the SAR and optical image fusion method based on frequency-space interaction described above.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows: Aiming at the problems of lack of effective SAR and optical image feature extraction methods and fusion strategies, the present invention constructs a frequency-space interaction fusion module. In each stage, the spatial domain feature extraction module is used to extract the spatial features of optical, SAR, and pseudo-optical images, and the frequency domain fusion module is used to fuse SAR and optical images in the frequency domain, so as to effectively extract and fuse multi-source image features at multiple scales and levels. Aiming at the problem that the fusion result obtained by the loss function constraint is limited by the expression defects of the images to be fused itself, the present invention constructs a mask-based correction module, and uses the anti-interference advantage of SAR images to effectively compensate the occluded areas, making up for the lack of details caused by factors such as clouds and fog in optical images. Aiming at the problem of lack of reasonable and effective relationship and characterization modeling between features, the present invention uses a two-stage loss function to constrain the training process of the network, which is respectively used to ensure that the network learns the frequency-space interaction fusion ability and the high-quality reconstruction ability of the fused image. Description of the Drawings In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0031] Figure 1 It is the overall process framework diagram of the embodiment of the present invention; Figure 2 It is the structural schematic diagram of the frequency domain fusion module of the embodiment of the present invention; Figure 3 It is the structural schematic diagram of the spatial domain feature extraction module of the embodiment of the present invention; Figure 4 It is the structural schematic diagram of the mask-based correction module of the embodiment of the present invention; Figure 5 It is the fusion comprehensive quality comparison result diagram of the embodiment of the present invention; Figure 6 It is the fusion cloud removal comparison result diagram of the embodiment of the present invention. Detailed Embodiments

[0032] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0033] As Figure 1 shown, the embodiments of the present invention also provide an overall process framework diagram of the SAR and optical image fusion method based on frequency-space interaction, specifically including: obtaining paired and matched SAR and optical remote sensing image data; inputting the obtained SAR image data into a pre-trained SAR-to-optical image conversion network to obtain pseudo-optical image data; inputting the paired and matched SAR and optical images and the pseudo-optical image into a trained fusion model to obtain a SAR and optical fusion image; wherein, the training of the fusion model includes: constructing a SAR, optical, and pseudo-optical remote sensing image data set of the same detection target or scene; constructing a fusion model based on the frequency-space interaction SAR and optical image fusion network, including: a data preprocessing module for preliminarily processing the input paired and matched SAR and optical images and the pseudo-optical image, and generating a feature embedding space to obtain multi-source embedding features; a frequency-space interaction fusion module for realizing feature extraction and fusion of the multi-source embedding features at multiple levels and multiple scales to obtain fusion features, a frequency-domain fusion feature set, and multi-level pseudo-optical features; a mask-based correction module for using the advantage of the SAR image in anti-interference to supplement and correct the detailed information lost in the optical image affected by cloud and fog weather by using the pseudo-optical image, and finally obtaining a single-channel fusion image. Training the model based on the constructed data set to obtain a trained fusion model; correcting the single-channel fusion image with a mask to obtain an RGB color image that combines the spectral information, spatial distribution characteristics of the optical image, and the structural texture details of the SAR image.

[0034] Step 1, obtain paired and matched SAR and optical remote sensing image data.

[0035] Step 2, input the obtained SAR image data into a pre-trained SAR-to-optical image conversion network to obtain pseudo-optical image data FAKE. The pre-trained SAR-to-optical image conversion network Transfer is implemented based on a generative adversarial network.

[0036] Step 3, construct a data preprocessing module. The data preprocessing module consists of two parts: an image preprocessing process and a feature embedding process. Among them, in the image preprocessing process, the input optical and pseudo-optical images are first subjected to grayscale conversion and normalization processing. At the same time, the input SAR image is normalized to ensure the consistency of data in terms of numerical scale and format, laying a foundation for the deep feature extraction of subsequent modules. The grayscale conversion process refers to converting a 3-channel optical or pseudo-optical RGB image into a single-channel grayscale image, which can be expressed as:

[0037]

[0038] Among them, and respectively represent the single-channel grayscale images after the conversion of the optical image and the pseudo-optical image, , , and , , are respectively the 3 RGB channels of the optical image and the pseudo-optical image. The normalization process is used to scale the data to the interval, which can be expressed as:

[0039] Among them, represents the original image, represents the image after normalization processing.

[0040] The feature embedding process uses a convolutional module to process the preprocessed multi-source images. The multi-source images are the optical image, pseudo-optical image, and SAR image output by the image preprocessing process, and obtain multi-source embedded features , and . The convolutional module consists of convolutional layers, BatchNorm normalization layers, and ReLU activation function layers. Among them, the ReLU activation function is:

[0041] Among them, represents the input feature of the ReLU activation function.

[0042] Step 4, construct a frequency-space interaction fusion module, which is used to realize the feature extraction and fusion of multi-source embedded features at multiple levels and scales, and obtain fusion features, a frequency-domain fusion feature set, and multi-level pseudo-optical features. The core components of the frequency-space interaction fusion module include: a spatial-domain feature extraction sub-module and a frequency-domain fusion sub-module, as shown in Figure 2 andFigure 3 As shown in the figure. Among them, the spatial domain feature extraction sub-module consists of 3 parallel ResNet convolutional modules, which are used to extract the spatial features of optical, SAR, and pseudo-optical images respectively. The spatial domain feature extraction includes 3 stages. In each stage, the optical, SAR, and pseudo-optical image features enter the spatial domain feature extraction sub-module of each stage. After calculation, deeper optical, SAR, and pseudo-optical image features are obtained and then sent to the spatial domain feature extraction sub-module of the next stage. The implementation process can be expressed as:

[0043] Among them, , and respectively represent the optical, SAR, and pseudo-optical image features of the th stage, represents the spatial domain feature extraction sub-module of the th stage, . Among them, the optical, SAR, and pseudo-optical image features of the first stage , and are the multi-source embedded features output by the data preprocessing module. All pseudo-optical image features are concatenated along the channel dimension to obtain multi-level pseudo-optical features , that is, . represents concatenating features along the channel dimension of the features.

[0044] The frequency domain fusion sub-module performs discrete Fourier transform (DFT) on the optical and SAR image features and respectively to map them to the frequency domain, so as to extract the phase spectrum and amplitude spectrum respectively. Then, the phase spectrum and amplitude spectrum are fused respectively to obtain the fused phase feature and amplitude feature. Through the inverse discrete Fourier transform (IDFT), the fused phase and amplitude features are remapped to the spatial domain and passed through a convolutional module to obtain the final frequency domain fusion feature. The implementation process can be expressed as:

[0045]

[0046]

[0047]

[0048] Among them, and respectively represent the DFT transform and transform, and represent the optical and SAR image features input to this frequency-domain fusion sub-module, respectively represent the phase spectrum and amplitude spectrum corresponding to the optical image features in the frequency domain, respectively represent the phase spectrum and amplitude spectrum corresponding to the SAR image features in the frequency domain. is the frequency-domain fusion process, which realizes the fusion of the phase spectrum and amplitude spectrum, are the fused phase feature and amplitude feature respectively, represents the convolution module, is the final frequency-domain fusion feature obtained by this frequency-domain fusion sub-module.

[0049] Among them, the frequency-domain fusion process includes: for the fusion of the phase spectrum, first use the feature mapping module based on the attention mechanism to process the phase spectrum of the SAR image and extract its structure guidance information. This information is used as a weight to perform weighted fusion on the phase spectra of the optical and SAR images, and the fused phase feature is further extracted through the convolution module . Similarly, considering that the amplitude spectrum contains the spatial distribution information of the image, use the feature mapping module based on the attention mechanism to process the amplitude spectrum of the optical image to obtain the spatial guidance information, and fuse the amplitude spectra of the optical and SAR images accordingly. After passing through the convolution module, the fused amplitude feature is obtained. The implementation process can be expressed as:

[0050]

[0051] Among them, represents the convolution module, represents the feature mapping module, and its implementation process can be expressed as:

[0052] Among them, represents the input feature of the feature mapping module, and represent the maximum pooling calculation and average pooling calculation. represents concatenating the features in the channel dimension of the features, represents the convolution module, is the activation function. The described Sigmoid activation function is:

[0053] Among them, represents the input feature of the Sigmoid activation function.

[0054] The frequency-domain feature fusion also includes three stages. The optical and SAR image features in the multi-source embedded features output by the data preprocessing module enter the frequency-domain fusion sub-module in the first stage to calculate the first frequency-domain fusion feature; the optical and SAR image features calculated by the spatial-domain feature extraction sub-module in the first stage enter the frequency-domain fusion sub-module in the second stage to calculate the second frequency-domain fusion feature; similarly, the optical and SAR image features calculated by the spatial-domain feature extraction sub-module in the second stage enter the frequency-domain fusion sub-module in the third stage to calculate the third frequency-domain fusion feature. The three frequency-domain fusion features are concatenated along the channel dimension and processed by the convolution module to obtain the fusion feature . The implementation process can be expressed as:

[0055]

[0056] Among them, represents the th frequency-domain fusion sub-module, and respectively represent the optical and SAR image features obtained by the spatial-domain feature extraction sub-module in the th stage. is the frequency-domain fusion feature obtained in the th stage. represents concatenating the features along the channel dimension of the features, represents the convolution module. All the frequency-domain fusion features constitute the frequency-domain fusion feature set .

[0057] Step 5, construct a mask-based correction module, as Figure 4 shown. The mask-based correction module includes a mask generation sub-module based on the dark channel, a pseudo-optical feature equalization process, and a correction fusion sub-module. Among them, the mask-based correction module first calculates the dark channel prior based on the original optical image OPT, thereby quickly and accurately distinguishing the cloud area from the non-cloud area and generating the corresponding grayscale mask Mask. Among them, the calculation process of the dark channel prior can be expressed as:

[0058] Among them, represents the dark channel value at pixel , represents the color channel Under the following circumstances, the pixel brightness value is the center of the local neighborhood window. Based on the dark channel prior, using a preset threshold to perform binarization to generate the corresponding grayscale mask Mask. The implementation process can be expressed as:

[0059] Among them, represents the mask value at pixel . When is greater than the set threshold , it is considered that the pixel belongs to the cloud area and is assigned a value of 1; otherwise, the pixel belongs to the non-cloud area and is assigned a value of 0. In order to reduce noise interference, after the mask is generated, morphological operations (such as opening and closing operations) are further used to smooth the mask to obtain a more accurate cloud area detection result.

[0060] The pseudo-optical feature equalization process introduces mean-standard deviation correction to standardize the feature distribution of multi-level pseudo-optical features so that it is consistent with the fusion feature in the numerical scale. The implementation process of mean-standard deviation correction can be expressed as:

[0061] Among them, is the multi-level pseudo-optical feature, is the fusion feature, and represent the mean and standard deviation operations respectively. is the multi-level pseudo-optical feature after standardization. Through this operation, the statistical characteristics of the multi-level pseudo-optical feature are closer to the fusion feature, thus ensuring the illumination balance and overall coordination of the final fusion result.

[0062] The input of the correction fusion sub-module is the multi-level pseudo-optical feature after standardization, the fusion feature and the grayscale cloud mask Mask. Using Mask as the weight to achieve fusion, and after splicing the fusion result with the fusion feature input to the correction fusion sub-module along the feature channel dimension, a single-channel fusion image is obtained through a convolution module. The implementation process can be expressed as:

[0063] Among them, represents splicing the features along the channel dimension of the features, represents the convolution module.

[0064] Step 6: Correct the single-channel fused image with the help of a mask to obtain an RGB color image that combines the spectral information, spatial distribution characteristics of the optical image, and the structural texture details of the SAR image. For each pixel of the fused RGB color image , when the corresponding grayscale cloud mask Mask satisfies the condition of, use the pixel value of the pseudo-optical image to replace the pixel value of the original optical image; otherwise, retain the information of the original optical image. The implementation process can be expressed as:

[0065]

[0066] where , , and , , are the three RGB channels of the optical image and the fused image respectively, i.e., the threshold in the cloud area detection-generated mask, which is set to the empirical value of 0.3.

[0067] Step 7: Construct a frequency-space interaction-based SAR and optical image fusion network for SAR and optical remote sensing image fusion, as shown in Figure 1 . The specific steps are as follows: Use the data preprocessing module, frequency-space interaction fusion module, and mask-based correction module described in Steps 3, 4, and 5 to obtain the SAR and optical remote sensing image fusion result. And correct the single-channel fused image with the help of a mask to obtain an RGB color image that combines the spectral information, spatial distribution characteristics of the optical image, and the structural texture details of the SAR image.

[0068] Specifically, input the obtained SAR image data into the pre-trained SAR-to-optical image conversion network to obtain the pseudo-optical image data, i.e., Figure 1 FAKE in.

[0069] Specifically, input the paired SAR and optical images and the pseudo-optical image into the data preprocessing module to obtain the multi-source embedded features, i.e., Figure 1 in , and .

[0070] Specifically, input the multi-source embedded features into the frequency-space interaction fusion module to obtain the fused features, the frequency-domain fused feature set, and the multi-level pseudo-optical features, i.e., Figure 1 in , and 。

[0071] Specifically, the fused features, the frequency-domain fused feature set, and the multi-level pseudo-optical features are input into the mask-based correction module to obtain the final single-channel fused image, that is, Figure 4 in 。

[0072] Specifically, the single-channel fused image is corrected by means of a mask to obtain an RGB color image that combines the spectral information, spatial distribution characteristics of the optical image, and the structural texture details of the SAR image, that is, Figure 1 Result in

[0073] Step 8, design a loss function suitable for this network. It includes two stages: The loss function in the first stage ensures that the network learns the frequency-space interaction fusion ability, which can be expressed as:

[0074] Among them, represents the frequency-domain fused feature set, and are the single-channel optical and SAR images respectively, is an important weight for balancing the structural information loss and the pixel information loss. is the structural information loss. Using a metric based on Structural Similarity (SSIM), the structural similarities between the optical image and the SAR image and the frequency-domain fused features at each level are calculated respectively. is the pixel information loss. Based on the Mean-Square Error (MSE), the pixel-level distances between the optical image and the frequency-domain fused features at each level are calculated. Through such a design, the network can not only take into account the preservation of global structural information during the learning process, but also pay attention to the accurate restoration of local details, thus providing high-quality fused features for the subsequent reconstruction stage.

[0075] The loss function in the second stage ensures that the network learns the high-quality reconstruction ability of the fused image, which can be expressed as:

[0076] Among them, and are the weight parameters for balancing the pixel information and the fused comprehensive loss respectively. By reasonably setting these two weights, the network can further optimize the overall fusion effect while maintaining the structure and details during the training process, so that the finally generated image reaches a higher level in terms of visual perception and quantitative indicators. is the fused comprehensive loss, which can be expressed as:

[0077]

[0078]

[0079] Among them, is the loss of the average intensity of the image, is the loss of the average gradient of the image. and respectively represent the height and width of the fused image, i.e., the single-channel fused image, and are the SAR and the single-channel optical image. Through the maximum selection strategy the pixel intensity distributions of the SAR and the single-channel optical image are integrated, and the pixel intensity distribution of the fused image is constrained by the integral distribution. is the Sobel operator, is the tuning parameter.

[0080] Specifically, it is trained on the WHU-OPT-SAR dataset composed of paired and matched optical and SAR image data, and the SAR and optical remote sensing images are fused according to the obtained SAR and optical image fusion network based on frequency-space interaction.

[0081] Specifically, based on Steps 1 - 8, the fusion results of the SAR and optical remote sensing images on the WHU-OPT-SAR dataset composed of paired and matched optical and SAR image data are obtained. In order to compare with other methods, we use the traditional spatial-domain image fusion methods GS, Brovey, SFIM, and VSM-WLS, the transform-domain image fusion methods DWT, NSCT, and the hybrid method Hybrid-MSD. In addition, we also compare the method of the present invention with 11 fusion frameworks including the classical general fusion methods based on deep learning, Densefuse, IFCNN, DIF-Net, and U2Fusion on the above dataset.

[0082] Specifically, in order to quantitatively evaluate the fusion result, the present invention also introduces a no-reference image evaluation index, including: an index based on information entropy: Peak Signal-to-Noise Ratio (PSNR) and MSE, an index based on image features: Spatial Frequency (SF) and Standard Deviation (SD), and an index based on correlation: Correlation Coefficients (CC) and Sum of Correlation Differences (SCD), which are used to evaluate the fusion result. Among them, the larger the values of the PSNR, SF, SD, and SCD indexes, the higher the quality of the fused image; the closer the value of the MSE index is to 0, the higher the quality of the fused image; the closer the value of the CC index is to 1, the higher the quality of the fused image. The quantitative comparison results are shown in Table 1. The bold results indicate the optimal results, and the underlined results indicate the sub-optimal results.

[0083] Table 1 Comparison results of quantitative evaluation indexes of different methods on the WHU-OPT-SAR dataset

[0084] Specifically, the embodiment of the present invention also provides a representation of the quantitative index results. Although the PSNR and MSE indexes of the method proposed by the present invention are slightly inferior to those of U2Fusion (PSNR is 64.15 dB and MSE is 0.0204), they still reach a relatively high level and can better retain the original spectral information. In terms of SF and SD, the method proposed by the present invention obtains the best values of 27.38 and 27.80 respectively, which indicates that the method has obvious advantages in restoring the detailed structure and can effectively enhance the local detail performance of the image. The method proposed by the present invention obtains 0.6290 and 1.1681 in CC and SCD respectively, indicating that the fusion result not only maintains the original spectral characteristics but also better takes into account the reconstruction of the structural information. Generally speaking, the SAR and optical image fusion method based on frequency-space interaction proposed by the present invention obtains three best indexes and two sub-optimal indexes, and overall performs the best in objective data. While retaining the spectral information, it significantly enhances the ability to restore the structure and texture details and can meet the effective pixel-level fusion requirements of SAR and optical images.

[0085] Specifically, in order to qualitatively evaluate the fusion result, the embodiment of the present invention also provides a qualitative visual result comparison, as shown in Appendix Figure 5 and Appendix Figure 6As shown in the figure. In comparison, the method proposed by the present invention can not only retain the spectral information of the optical image, but also supplement the texture information of the SAR image, while reducing the generation of speckle noise. In addition, the method proposed by the present invention can better retain the spectral information of the optical image, effectively supplement the texture details of the SAR image, and compensate the spectral information by using the pseudo-optical image in scenes with local thin clouds, large-scale thick clouds or completely blocked by clouds and fog. Finally, a high-quality cloud removal effect is achieved, which fully proves the advantages and practicability of the method proposed by the present invention in remote sensing image cloud removal processing, especially in dealing with complex clouds and retaining ground object details.

[0086] The implementation basis of each embodiment of the present invention is achieved through programmed processing by a device with processor functions. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this actual situation, on the basis of the above embodiments, an embodiment of the present invention provides a SAR and optical image fusion system based on frequency-space interaction, which is used to execute the SAR and optical image fusion method based on frequency-space interaction in the above method embodiments.

[0087] The system includes: an acquisition data unit for acquiring paired matching SAR and optical remote sensing image data; A pseudo-optical image generation module for inputting the acquired SAR image data into a pre-trained SAR-to-optical image conversion network to obtain pseudo-optical image data; An image fusion model construction unit for inputting the paired matching SAR and optical images and the pseudo-optical image into a trained fusion model to obtain a SAR and optical fusion image; wherein, the fusion model includes: a data preprocessing module for preliminarily processing the input paired matching SAR and optical images and the pseudo-optical image, and generating a feature embedding space to obtain multi-source embedding features; a frequency-space interaction fusion module for realizing feature extraction and fusion of the multi-source embedding features at multiple levels and multiple scales to obtain fusion features, a frequency-domain fusion feature set and multi-level pseudo-optical features; a mask-based correction module for using the advantage of the SAR image to resist interference, and using the pseudo-optical image to supplement and correct the detailed information lost in the optical image due to cloudy weather, and finally obtaining a single-channel fusion image; A fusion image correction unit for correcting the single-channel fusion image by means of a mask to obtain an RGB color image that combines the spectral information, spatial distribution characteristics of the optical image and the structural texture details of the SAR image.

[0088] The specific implementation manners of each unit are the same as each step, and the present invention will not describe them.

[0089] The SAR and optical image fusion system based on frequency-space interaction provided by the embodiments of the present invention aims at the problem of lacking effective SAR and optical image feature extraction methods and fusion strategies. By using several modules, through a fusion model, and using a two-stage loss function to constrain the training process of the network, it ensures that the network learns the frequency-space interaction fusion ability and the high-quality reconstruction ability of the fused image, and realizes high-quality SAR and optical image fusion.

[0090] Based on the same inventive concept as the foregoing embodiments, the embodiments of the present invention further provide an electronic device, including a memory and a processor. The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement a SAR and optical image fusion method based on frequency-space interaction as proposed in the above embodiments.

[0091] The embodiments of the present invention also provide a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it overcomes the problem of lacking effective SAR and optical image feature extraction methods and fusion strategies, avoids the fusion result being limited by the expression defects of the images to be fused, realizes reasonable and effective relationship and representation modeling between features, and ensures high-quality SAR and optical image fusion. The storage medium can be any non-volatile storage device such as a hard disk, a solid-state drive, a flash drive, an optical disc, etc., used to store computer program codes and necessary data files. The stored computer program includes: a data acquisition module, a ground object classification module.

[0092] The embodiments of the present invention also provide a computer program product containing instructions, which, when running on a computer, wholly or partially generates a SAR and optical image fusion method based on frequency-space interaction as proposed in the above embodiments. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0093] Finally, it should be noted that the above specific embodiments are only relatively representative examples of the present invention. Obviously, the present invention is not limited to the above specific embodiments and there can be many variations. Any simple modification, equivalent change, and modification made to the above specific embodiments based on the technical essence of the present invention shall be considered to fall within the protection scope of the present invention.

Claims

1. A SAR and optical image fusion method based on frequency-space interaction, characterized in that Including: Obtain paired and matched SAR and optical remote sensing image data; Input the obtained SAR image data into a pre-trained SAR-to-optical image conversion network to obtain pseudo-optical image data; Input the paired and matched SAR and optical images and the pseudo-optical images into a trained fusion model to obtain a SAR and optical fusion image; wherein, the fusion model includes: a data preprocessing module for preliminarily processing the input paired and matched SAR and optical images and the pseudo-optical images, and generating a feature embedding space to obtain multi-source embedding features; a frequency-space interaction fusion module for implementing feature extraction and fusion of the multi-source embedding features at multiple levels and scales to obtain fusion features and multi-level pseudo-optical features; a mask-based correction module for using the pseudo-optical image for supplementary correction to finally obtain a single-channel fusion image; Correct the single-channel fusion image with the aid of a mask to obtain an RGB color image that combines the spectral information, spatial distribution characteristics of the optical image, and the structural texture details of the SAR image.

2. The method for fusing SAR and optical images based on frequency-space interaction according to claim 1, characterized in that: The data preprocessing module preliminarily processes the input paired and matched SAR and optical images and the pseudo-optical images, including: Image preprocessing process: perform grayscale conversion and normalization processing on the input optical and pseudo-optical images; perform normalization processing on the input SAR image; Feature embedding process: use a convolution module to process the preprocessed multi-source images to obtain multi-source embedding features, and the multi-source images are the normalized optical image, pseudo-optical image, and SAR image.

3. A method for fusing SAR and optical images based on frequency-space interaction according to claim 2, characterized in that: The grayscale conversion process is to convert a 3-channel optical or pseudo-optical RGB image into a single-channel grayscale image, expressed as: Among them, and respectively represent the single-channel grayscale images after the conversion of the optical image and the pseudo-optical image, , , and , , are respectively the three RGB channels of the optical image and the pseudo-optical image; the normalization process is used to scale the data to the interval, expressed as: Among them, represents the original image, represents the image after normalization processing.

4. A method for fusing SAR and optical images based on frequency-space interaction according to claim 2, characterized in that: The convolution module consists of a convolution layer, a BatchNorm normalization layer, and a ReLU activation function layer.

5. A method for fusing SAR and optical images based on frequency-space interaction according to claim 1, characterized in that: The construction of the frequency-space interaction fusion module includes 3 stages, and each stage Including: Spatial domain feature extraction sub-module: composed of multiple parallel ResNet convolution modules, respectively used to extract the spatial features of the optical, SAR, and pseudo-optical images; Frequency domain fusion sub-module: For the embedding features of the optical and SAR images, respectively use the discrete Fourier transform to map them to the frequency domain, so as to extract the phase spectrum and amplitude spectrum respectively; then fuse the phase spectrum and amplitude spectrum respectively to obtain the fused phase feature and amplitude feature; Through the inverse discrete Fourier transform, remap the fused phase and amplitude features back to the spatial domain, and obtain the final frequency domain fusion features through a convolution module; The optical, SAR, and pseudo-optical image features enter the spatial domain feature extraction sub-module of each stage, and deeper optical, SAR, and pseudo-optical image features are obtained through calculation, and then sent to the spatial domain feature extraction sub-module of the next stage. All pseudo-optical image features are concatenated along the channel dimension to obtain multi-level pseudo-optical features; The embedding features of the optical and SAR images enter the frequency domain fusion sub-module of the first stage, and the first frequency domain fusion feature is obtained through calculation; The features of the optical and SAR images calculated by the spatial domain feature extraction sub-module in the first stage enter the frequency domain fusion sub-module in the second stage to calculate the second frequency domain fusion feature. Similarly, the features of the optical and SAR images calculated by the spatial domain feature extraction sub-module in the second stage enter the frequency domain fusion sub-module in the third stage to calculate the third frequency domain fusion feature. The three frequency domain fusion features form a frequency domain fusion feature set. The three frequency domain fusion features are concatenated along the channel dimension and processed by a convolutional module to obtain the fusion feature.

6. The method for fusing SAR and optical images based on frequency-space interaction according to claim 1, wherein: The construction of the mask-based correction module includes: Mask generation sub-module based on dark channel: Calculate the dark channel prior based on the original optical image, distinguish the cloud area and non-cloud area, and generate the corresponding grayscale mask; Pseudo-optical feature equalization process: Introduce mean-standard deviation correction to standardize the feature distribution of the pseudo-optical image to make it consistent with the fusion feature in the numerical scale; Correction fusion sub-module: The standardized pseudo-optical image features and fusion features are fed into the correction fusion sub-module and fused with the mask as the weight to obtain a single-channel fusion image.

7. A method for fusing SAR and optical images based on frequency-space interaction according to claim 6, characterized in that: The input of the calibration fusion sub-module is the multi-level pseudo-optical features after standardization , the fused features and the grayscale cloud mask Mask; use Mask as the weight to achieve fusion, and splice the fusion result with the fused features input to the calibration fusion module along the feature channel dimension, and then obtain a single-channel fused image through a convolution module ; The implementation process is expressed as: Among them, indicates that the features are concatenated in the channel dimension of the features, represents a convolutional module.

8. A method for fusing SAR and optical images based on frequency-space interaction according to claim 1, characterized in that: Correcting the single-channel fusion image with the mask includes: For each pixel of the fused RGB color image , when the corresponding grayscale cloud mask Mask satisfies the condition, the pixel value of the pseudo-optical image is used to replace the pixel value of the original optical image; otherwise, the information of the original optical image is retained, and the implementation process is expressed as: Among them, are the horizontal and vertical coordinates of the pixel point, , , and , , are the R, G, and B channels of the optical image and the fused image respectively, i.e., the threshold in the mask generated by cloud and fog region detection.

9. A method for fusing SAR and optical images based on frequency-space interaction according to claim 1, characterized in that: The loss function used to train the fusion model is divided into two stages: The first-stage loss function ensures that the network learns the frequency-space interaction fusion ability, expressed as: Among them, represents the frequency-domain fusion feature set, and are the single-channel optical and SAR images respectively, is an important weight for balancing the loss of structural information and the loss of pixel information; is the loss of structural information. By using a metric based on structural similarity, the structural similarities between the optical image and the SAR image and the frequency-domain fusion features at each level are calculated respectively; is the loss of pixel information. Based on the mean square error, the pixel-level distances between the optical image and the frequency-domain fusion features at each level are calculated; The second-stage loss function ensures that the network learns the high-quality reconstruction ability of the fusion image, expressed as: Among them, and are the weight parameters for balancing pixel information and the fusion comprehensive loss respectively; is the fusion comprehensive loss, expressed as: Among them, is the loss of the average intensity of the image, is the loss of the average gradient of the image, and respectively represent the height and width of the fused image, i.e., the single-channel fused image, and are the SAR and the single-channel optical image; through the maximum selection strategy the pixel intensity distributions of the SAR and the single-channel optical image are integrated, and the integral distribution is used to constrain the pixel intensity distribution of the fused image; is the Sobel operator, is the tuning parameter.

10. A SAR and optical image fusion system based on frequency-space interaction, characterized in that, Including: Data acquisition unit, used to acquire paired SAR and optical remote sensing image data; Pseudo-optical image generation module, used to input the acquired SAR image data into a pre-trained SAR-to-optical image conversion network to obtain pseudo-optical image data; Image fusion model construction unit, used to input the paired SAR and optical images and pseudo-optical images into the trained fusion model to obtain the SAR and optical fusion image; wherein, the fusion model includes: a data preprocessing module, used to preliminarily process the input paired SAR and optical images and pseudo-optical images, and generate a feature embedding space to obtain multi-source embedding features; a frequency-space interaction fusion module, used to implement feature extraction and fusion of the multi-source embedding features at multiple levels and scales to obtain fusion features and multi-level pseudo-optical features; a mask-based correction module, used to supplement and correct with the pseudo-optical image to finally obtain a single-channel fusion image; Fusion image correction unit, used to correct the single-channel fusion image with the mask to obtain an RGB color image that fuses the spectral information, spatial distribution characteristics of the optical image, and the structural texture details of the SAR image.

Images