A SAR-Visible Light Image Fusion Method Based on Dual Discriminators

Abstract

A dual-discriminator-based SAR and visible light image fusion method includes constructing a dataset, generating a pre-fused image using the GFCE algorithm, constructing a GAN network with dual discriminators (DD-GAN), and training the DD-GAN network to fuse SAR and visible light images. This invention utilizes the generative adversarial mechanism of GAN networks and introduces the pre-fused image as a guide for the generator, offering the following advantages: it avoids manually designing complex fusion rules; it can output a contrast-enhanced fused image from an unenhanced source image, and the fused image retains the spectral information of the visible light image while containing a large amount of detailed information from both the SAR and visible light images; it obtains a high-quality fused image in a shorter time, effectively resolving the contradiction between high-quality fused images and low generation time.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a multi-source image fusion method. Background Technology

[0002] The fusion of SAR (Synthetic Aperture Radar) and visible light images has wide applications in target detection, disaster prediction, and land resource statistics, and has significant research value. It can support subsequent image analysis and information extraction, and lay the foundation for better and more accurate target identification and detection. However, SAR images suffer from low contrast, severe speckle noise, and unclear details, resulting in fused images with indistinct details and low contrast, which does not conform to human visual perception. Therefore, how to make the fused image retain the spectral information of the visible light image while containing more detailed information from both SAR and visible light images, and having good contrast that conforms to human visual perception, is currently a research hotspot in the fusion of SAR and visible light images.

[0003] Deep learning is extremely popular in the field of computer vision, excelling in object detection and recognition. Due to its superior learning capabilities, researchers have applied deep learning to image fusion, resulting in many excellent methods. Feng et al. proposed a branched CNN network to extract features separately from SAR and multispectral images, achieving better results than a single-branch network. (Feng Q, Zhu D, Yang J, et al. Multisource hyperspectral and lidar data fusion for urban land-use mapping based on a modified two-branch convolutional neural network[J].ISPRS International Journal of Geo-Information,2019,8(1):28.) Ye et al. proposed a novel SOSTF fusion framework for SAR and visible light image fusion. In order to preserve the structural and texture information of the input image, multi-scale features are extracted from the image using an encoder; then, DenseNet is used for feature fusion; finally, the decoder is used to reconstruct the fused image. This algorithm can effectively fuse complementary features of SAR and visible light images. However, currently, image fusion using convolutional neural networks still requires the design of fusion strategies. The quality of the fusion strategy design directly affects the fusion result, and manually designed fusion strategies often fail to achieve the best fusion effect. Therefore, some researchers have also tried to use generative adversarial networks for image fusion.

[0004] Kong et al. addressed the issues of significant speckle noise and severe spectral distortion in the fusion of visible light and SAR images. They proposed a fusion method for optical and SAR images based on Dense-UGAN and Gram–Schmidt transform. The generator uses dense connections with a U-shaped network (Dense-UGAN) to deepen the network structure and obtain deeper information from the source images (Kong Y, Hong F, Leung H, et al. A fusion method of optical image and SAR image based on dense-UGAN and Gram–Schmidt transform[J]. Remote Sensing, 2021, 13(21):4274.). Bao Deng et al. proposed modality classification and ground feature classification tasks for extracting differential features from SAR optical images. They designed a fine-tuned VGG convolutional neural network model, achieving better classification results with less training time. For SAR optical image mapping, a space-frequency consistent generative adversarial network (SFGAN) model framework is proposed, which makes the texture details of the generated images more realistic and the contour features of roads and rivers clearer (B. Deng and H. Lv, "Research on Image Fusion Method of SAR and Visible Image Based on CNN," 2022 IEEE 4th International Conference on Civil Aviation Safety and Information Technology (ICCASIT), Dali, China, 2022, pp. 1400-1403, doi:10.1109 / ICCASIT55263.2022.9987074.). However, current fused images cannot well contain the detailed information of SAR images, and the contrast is low, which is not conducive to human visual perception. Summary of the Invention

[0005] The purpose of this invention is to propose a SAR and visible light image fusion method based on a dual discriminator. This method utilizes the generative adversarial mechanism of Generative Adversarial Networks (GANs) and, for the first time, uses a pre-fused image as a guide for the generator in the field of SAR and visible light image fusion. This avoids the need for manually designing complex fusion rules, achieving the output of a contrast-enhanced fused image from an input source image that has not undergone contrast enhancement. Furthermore, the fused image retains the spectral information of the visible light image while containing a large amount of detailed information from both the SAR and visible light images. It also achieves high-quality fused images in a shorter time, effectively resolving the contradiction between high-quality fused images and short generation time.

[0006] This invention is achieved through the following technical solution.

[0007] The present invention discloses a SAR and visible light image fusion method based on dual discriminators, comprising the following steps:

[0008] Step 1: Constructing the Dataset: First, visible light images and SAR images are acquired from Landsat 8 and Sentinel-1 satellites, respectively. Then, the acquired images are registered using ENVI 5.2 software. After registration, the images are cropped using ENVI 5.2 software to obtain SAR and visible light image pairs. Finally, the BM3D algorithm is used to denoise the cropped SAR images. After denoising, the original dataset is obtained. Next, the original dataset is divided into training and test sets: First, a portion of image pairs are selected from the original dataset and further cropped to form part of the training set; finally, the remaining original dataset is used as the test set.

[0009] Step 2: Generate pre-fused images using the GFCE algorithm: The original GFCE algorithm only enhances the contrast of visible light images during fusion. Considering the low contrast and unclear details of SAR images, the contrast of SAR images is also enhanced during the generation of pre-fused images, resulting in pre-fused images with good spectral information and clear details. The same pre-fused image pairs as the training set are selected and cropped, and combined with the part of the training set obtained in Step 1 to form a complete training set. At this point, both the training set and the test set are completed.

[0010] Step 3: Construct a GAN network with dual discriminators (DD-GAN): DD-GAN consists of a generator and two discriminators. The generator produces the fused image; the two discriminators have identical structures and are used to evaluate the quality of the fused image. To comprehensively extract and efficiently utilize image features, densely connected blocks and an attention mechanism module are added to the generator. Furthermore, to address issues such as unclear details in the fused image and its poor performance for human visual perception, the pre-fused image obtained in Step 2 is used to guide the generator's generation.

[0011] Step 4: Train the DD-GAN network using the training set from Step 1.

[0012] Step 5: Fusion of SAR and visible light images: Input the test set from Step 1 into the DD-GAN network trained in Step 4 to obtain a fused image that retains the spectral information of the visible light image and the detailed information of the SAR and visible light image. Then, evaluate the quality of the fused image using subjective and objective evaluation metrics.

[0013] Furthermore, the DD-GAN network described in step 3 comprises two parts: a generator and a discriminator.

[0014] The generator's network structure comprises three convolutional layers (Conv1, Conv2, Conv3), a Denseblock module, and a Resblock_CBAM module. The input consists of channel-connected SAR and visible light images. After passing through Conv1, the output feature map is fed into the Denseblock module for thorough feature extraction. It is then input into the Resblock_CBAM module for weighted feature extraction. The weighted feature map is then fed into Conv2 for further feature extraction, and finally into Conv3 to obtain the fused image I. g .

[0015] Conv1 and Conv2 each consist of three parts: a convolutional layer, a batch normalization layer, and an activation function layer. Since downsampling is not performed, no pooling layer is included. The kernel size for both Conv1 and Conv2 is set to 3×3, which effectively optimizes the feature maps. Batch normalization is added to prevent the vanishing gradient problem; Batch Normalization (BN) is chosen for the batch normalization layer, making the gradient more stable and helping it propagate effectively to each layer. For the activation function, LeakyReLU is chosen to increase the non-linearity of the network model. Conv3 consists of two parts: a convolutional layer and an activation function layer. The kernel size is set to 1×1 to reduce the channel dimension and obtain an output image that meets the target number of channels. The activation function chosen is Tanh.

[0016] The Denseblock module contains three convolutional blocks: Dense1, Dense2, and Dense3. All convolutional kernels are set to 3×3, and batch normalization and activation functions are Batch Normalization (BN) and LeakyReLU. The output of each convolutional block is concatenated with the output of the previous convolutional block along the channel dimension, fully utilizing the features extracted from each convolutional layer.

[0017] The Resblock_CBAM is an improved residual block consisting of three parts: Bottleneck, CBAM, and shortcut. Bottleneck contains three convolutional layers. The first convolutional layer uses a 1×1 kernel to compress the number of channels, reducing the dimensionality of the input feature map and thus reducing the computational cost, shrinking the number of channels to half of the original number. This layer is followed by a batch normalization (BN) layer and a ReLU activation function. The second convolutional layer uses a 3×3 kernel to extract features, capturing local feature information in the input feature map. This layer is also followed by a batch normalization (BN) layer and a ReLU activation function. The third convolutional layer uses a 1×1 kernel to restore the number of channels. The Bottleneck output feature map F of size (120, 120, 64) is input into the CBAM, where (120, 120) represents the height and width of the feature map, and 64 is the number of channels. First, it enters the channel attention module, where the height and width of the feature map are max-pooled and average-pooled to obtain two feature maps of size (1, 1, 64). These are then fed into the multilayer perceptron to obtain channel weights. Finally, the obtained weights are summed and fed into the sigmoid activation function, outputting a channel attention feature map M of size (1, 1, 64). C Then M C Multiplying the input feature map F by the input feature map yields a feature map F' of size (120, 120, 64), which serves as the input to the spatial attention module. After inputting into the spatial attention module, max pooling and average pooling are first performed on each channel of the input feature map, resulting in two feature maps of size (120, 120, 1). These two feature maps are then concatenated along the channel dimension to obtain a feature map of size (120, 120, 2). A convolutional layer is then used to reduce the dimensionality of the feature map, resulting in a feature map of size (120, 120, 1). Finally, the obtained feature map is fed into the sigmoid activation function to obtain the spatial attention feature map M. S Then M S Multiplying by F' yields the final output of CBAM, which consists of channel-oriented and spatial attention feature maps. Finally, the input of Bottleneck and the output of CBAM are shortened to obtain the final feature map.

[0018] After the generator produces the fused image, it is fed into the discriminator for discrimination. The two discriminators have identical structures, each consisting of four convolutional modules and one linear layer. The first convolutional module differs from the other three in that it consists of a convolutional layer and a LeakyReLU activation layer, excluding a batch normalization layer. The other three convolutional modules have the same structure, consisting of a convolutional layer, a batch normalization layer, and a LeakyReLU activation layer. The kernel size of all convolutional layers is set to 3×3, and the stride is set to 2, rapidly reducing the width and height of the feature map. Batch normalization (BN) is chosen, and to prevent noise introduction, padding is only performed on the input image of the first layer. The last linear layer transforms the flattened feature map into an output representing the relative distance between the generated and real images. To reduce the number of parameters, the weights of the third and fourth convolutional blocks and the linear layer are shared.

[0019] Further, the process of training the DD-GAN network in step 4 includes: inputting the SAR and visible light image training set samples obtained in step 2 into the DD-GAN network for training; setting the learning rate of the generator to 0.0001 and the learning rate of the discriminator to 0.0001 to balance the learning rates of the generator and discriminator, and selecting Adam as the optimizer. The training ratio of the discriminator to the generator is 1:1, the batch size is 32, and a total of 10 epochs are trained. By continuously iterating and updating the network hyperparameters, the network training process is completed when the set number of iterations is reached.

[0020] The SAR and visible light image fusion algorithm based on a dual discriminator provided by this invention avoids the need for manually designing complex fusion rules. It achieves the output of a contrast-enhanced fused image from an input source image that has not undergone contrast enhancement. Furthermore, the fused image retains the spectral information of the visible light image while containing a large amount of detailed information from both the SAR and visible light images. It also achieves high-quality fused images in a shorter time, effectively resolving the contradiction between high-quality fused images and time-efficient generation. This invention fuses SAR and visible light images to obtain a fused image that retains the spectral information of the visible light image, the detailed information of both the SAR and visible light images, and conforms to human visual perception. Attached Figure Description

[0021] Figure 1 This is an overall framework diagram of the Dual Discriminator Network (DD-GAN) proposed in this invention. g For the fused image generated by the generator, I gV (I g -I VIS ) is the discrimination value used by the first discriminator to determine whether the fused image is a visible light image, I gS (I g -ISAR ) is the second discriminator's discrimination value for determining whether the fused image is a SAR image.

[0022] Figure 2 It is the network structure of the generator.

[0023] Figure 3 It is the network structure of the Denseblock module in the generator.

[0024] Figure 4 It is the network structure of the Resblock_CBAM module in the generator.

[0025] Figure 5 It is the network structure of Bottleneck in the Resblock_CBAM module.

[0026] Figure 6 It is the network structure of CBAM in the Resblock_CBAM module.

[0027] Figure 7 It is the network structure of the channel attention module in the CBAM module.

[0028] Figure 8 It is the network structure of the spatial attention module in the CBAM module.

[0029] Figure 9 It is the network structure of the discriminator.

[0030] In the diagram, ⊕ represents the sum of the obtained weights. It is the Sigmoid activation function. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below through embodiments. The described embodiments are some embodiments of the present invention, but not all embodiments; based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] As a feasible embodiment of the present invention, images from three regions were selected as experimental data. These regions include features such as water bodies, soil, vegetation, mountains, and buildings, which allows for better verification of the fusion effect and makes the experimental results more reliable. The image data from the three regions are: images of the Bohai Sea area in Northeast China, images of the Tianjin Binhai New Area, and images of the Yujiang River area in Nanning, Guangxi. The visible light images were obtained from the Geospatial Data Cloud website (…). https: / / www.gscloud.cn / searchData was obtained from the Landsat 8 satellite, and a true-color image was created by sequentially selecting the red, green, and blue bands, with a resolution of 5m. SAR images were obtained from the website operated by the European Space Agency. https: / / scihub.copernicus.eu / dhus / # / home To acquire data, select Sentinel-1B GRD level data, choose VV polarization mode, and set the resolution to 30m. Then, use the registration tool in ENVI 5.2 software to register and resample the SAR and visible light images. The main steps are as follows:

[0033] 1. First, click File->open…, select the folder containing the images to be registered, and load the SAR and visible light images to be registered respectively.

[0034] 2. Then, in the toolbox on the right, select Geometric Correction->Registration->ImageRegistration Workflow, use the SAR image as the reference image and the visible light image as the image to be registered, and click Next.

[0035] 3. Begin registration. First, click "Main," select the matching algorithm. SAR and visible light images are different image types, so mutual information is selected as the matching algorithm. Set the Tie point precision value to discard registration points that do not meet the preset precision; leave other options at their default values. Next, click "Seed Tie Points" to manually select several reference points in the center and around the edges of the image. Finally, click "Advanced," set the registration bands of the visible light image to match those of the SAR image, leave other options at their default values, and click Next to generate registration points. Check the scores of the registration points, delete points with low scores, then click "Warping," select the resampling method (cubic convolution interpolation is selected in this invention), and click Next to obtain the registered image.

[0036] After image registration, the three sets of images were cropped using ENVI 5.2 software to create SAR and visible light image pairs with a width of 408 and a height of 555. To expand the dataset, each selected cropped region was rotated by 45° each time, resulting in 8 image pairs, for a total of 215 image pairs. After cropping, the SAR images were denoised using the BM3D algorithm. After denoising, 36 image pairs were selected and cropped to 120×120 pixels, resulting in 22,488 image pairs. These 36 image pairs were then fused using the GFCE algorithm to obtain pre-fused images, which were also cropped to 120×120 pixels, resulting in another 22,488 image pairs. The 22,488 SAR, visible light, and pre-fused image pairs were then cut and pasted into three folders: SAR, VIS, and PF, to form the training set. The remaining 179 image pairs (555×408 pixels) were used as the test set.

[0037] Step 1: Constructing the dataset: First, visible light images and SAR images were acquired from Landsat 8 and Sentinel-1 satellites, respectively. Then, the acquired images were registered using ENVI 5.2 software, and the images were cropped into 215 pairs with a pixel size of 555×408. Finally, the BM3D algorithm was used to denoise the SAR images. After denoising, 36 pairs of images were selected and cropped into 22,488 pairs of SAR and visible light image pairs with a pixel size of 120×120.

[0038] Step 2: Generate pre-fused images using the GFCE algorithm: The original GFCE algorithm only enhances the contrast of the visible light image during fusion. Considering the low contrast and unclear details of the SAR image, the contrast of the SAR image is also enhanced during the generation of the pre-fused image, resulting in a pre-fused image with good spectral information and clear details. The same 36 pairs of pre-fused images as in Step 1 are selected and cropped to obtain 22,488 pairs of SAR and visible light image pairs with a pixel size of 120×120. These 22,488 pairs of images obtained in Step 1 form the training set, and the remaining 179 pairs of images with a pixel size of 555×408 are used as the test set.

[0039] Step 3: Construct a GAN network with dual discriminators (DD-GAN): DD-GAN consists of a generator and two discriminators. The generator produces the fused image; the two discriminators have identical structures and are used to evaluate the quality of the fused image. To comprehensively extract and efficiently utilize image features, densely connected blocks and an attention mechanism module are added to the generator. Furthermore, to address issues such as unclear details in the fused image and its poor performance for human visual perception, the pre-fused image obtained in Step 2 is used to guide the generator's generation.

[0040] Step 4: Train the DD-GAN network using the training set from Step 1.

[0041] Step 5: Fusion of SAR and visible light images: Input the test set from Step 1 into the DD-GAN network trained in Step 4 to obtain a fused image that retains the spectral information of the visible light image and the detailed information of the SAR and visible light image. Then, evaluate the quality of the fused image using subjective and objective evaluation metrics.

[0042] The DD-GAN network described in step 3 of this embodiment includes two parts: a generator and a discriminator.

[0043] The generator's network structure consists of three convolutional layers (Conv1, Conv2, Conv3), a Denseblock module, and a Resblock_CBAM module. The input is a channel-connected SAR and visible light image. After passing through Conv1, the output feature map is input to the Denseblock module for thorough feature extraction. Then, it is input to the Resblock_CBAM module for weighted feature extraction. The weighted feature map is then input to Conv2 for further feature extraction, and finally input to Conv3 to obtain the fused image I. g .

[0044] Conv1 and Conv2 consist of three parts: convolutional layers, batch normalization layers, and activation function layers. Since downsampling is not performed, pooling layers are not included. The kernel size for both Conv1 and Conv2 is set to 3×3, which effectively optimizes the feature maps. Batch normalization is added to prevent the vanishing gradient problem; Batch Normalization (BN) is chosen for the batch normalization layer, making the gradient more stable and helping it propagate effectively to each layer. For the activation function, LeakyReLU is chosen to increase the non-linearity of the network model. Conv3 consists of two parts: convolutional layers and activation function layers. The kernel size for the convolutional layers is set to 1×1 to reduce the channel dimension and obtain an output image that meets the target number of channels. Tanh is chosen as the activation function.

[0045] The Denseblock module contains three convolutional blocks: Dense1, Dense2, and Dense3. All convolutional kernels are set to 3×3, and batch normalization (BN) and LeakyReLU are chosen as the activation functions. The output of each convolutional block is concatenated with the output of the previous convolutional block along the channel dimension, fully utilizing the features extracted from each convolutional layer.

[0046] Resblock_CBAM is an improved residual block consisting of three parts: Bottleneck, CBAM, and shortcut. Bottleneck contains three convolutional layers. The first convolutional layer uses a 1×1 kernel to compress the number of channels, reducing the dimensionality of the input feature map and thus reducing the computational cost, shrinking the number of channels to half of the original number. This layer is followed by a batch normalization (BN) layer and a ReLU activation function. The second convolutional layer uses a 3×3 kernel to extract features, capturing local feature information from the input feature map. This layer is also followed by a batch normalization (BN) layer and a ReLU activation function. The third convolutional layer uses a 1×1 kernel to restore the number of channels. The Bottleneck output feature map F of size (120, 120, 64) is input into the CBAM, where (120, 120) represents the height and width of the feature map, and 64 is the number of channels. First, it enters the channel attention module, where the height and width of the feature map are max-pooled and average-pooled to obtain two feature maps of size (1, 1, 64). These are then fed into the multilayer perceptron to obtain channel weights. Finally, the obtained weights are summed and fed into the sigmoid activation function, outputting a channel attention feature map M of size (1, 1, 64). C Then M C Multiplying the input feature map F by the input feature map yields a feature map F' of size (120, 120, 64), which serves as the input to the spatial attention module. After inputting into the spatial attention module, max pooling and average pooling are first performed on each channel of the input feature map, resulting in two feature maps of size (120, 120, 1). These two feature maps are then concatenated along the channel dimension to obtain a feature map of size (120, 120, 2). A convolutional layer is then used to reduce the dimensionality of the feature map, resulting in a feature map of size (120, 120, 1). Finally, the obtained feature map is fed into the sigmoid activation function to obtain the spatial attention feature map M. S Then M S Multiplying by F' yields the final output of CBAM, which consists of channel-oriented and spatial attention feature maps. Finally, the input of Bottleneck and the output of CBAM are shortened to obtain the final feature map.

[0047] After the generator produces the fused image, it is fed into the discriminator for discrimination. The two discriminators have identical structures, each consisting of four convolutional layers and one linear layer. Unlike the other three convolutional modules, the first convolutional module consists of a convolutional layer and a LeakyReLU activation layer, excluding a batch normalization layer. The other three convolutional modules have the same structure, consisting of a convolutional layer, a batch normalization layer, and a LeakyReLU activation layer. The kernel size of all convolutional layers is set to 3×3, and the stride is set to 2, rapidly reducing the width and height of the feature map. Batch normalization (BN) is chosen, and to prevent the introduction of noise, padding is only performed on the input image of the first layer. The last linear layer transforms the flattened feature map into an output representing the relative distance between the generated and real images. To reduce the number of parameters, the weights of the third and fourth convolutional blocks and the linear layer are shared.

[0048] Step 4, the process of training the DD-GAN network, includes: inputting the SAR and visible light image training set samples obtained in Step 2 into the DD-GAN network for training; setting the learning rate of the generator to 0.0001 and the learning rate of the discriminator to 0.0001 to balance the learning rates of the generator and discriminator; selecting Adam as the optimizer. The training ratio of the discriminator to the generator is 1:1, the batch size is 32, and the training lasts for 10 epochs. By continuously iterating and updating the network hyperparameters, the network training process is completed when the set number of iterations is reached.

[0049] The final results of the fused image quality and the time required to generate one fused image obtained in this embodiment are as follows. To further demonstrate the advantages of this invention, the method of this invention was compared with traditional image fusion algorithms. The fusion quality of the fused images obtained by each algorithm under the same experimental conditions and the average time required to generate one image were compared. Tables 1 and 2 show that the SAR and visible light image fusion algorithm based on dual discriminators obtains high-quality fused images while taking less time to generate one image, achieving the optimal balance between fused image quality and time consumption.

[0050] Table 1 Evaluation results of fused image quality under different algorithms

[0051]

[0052]

[0053] Table 2. Time required to generate a fused image under different algorithms.

[0054]

[0055]

[0056] The above description merely illustrates preferred embodiments of the present invention, and while the description is relatively specific and detailed, it should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications, improvements, and substitutions without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A method for fusing SAR and visible light images based on a dual discriminator, characterized by: Includes the following steps: Step 1: Constructing the dataset: First, acquire visible light images and SAR images from Landsat 8 and Sentinel-1 satellites, respectively; then, register the acquired images using ENVI 5.2 software. After registration, continue to crop the images using ENVI 5.2 software to obtain SAR and visible light image pairs; finally, use the BM3D algorithm to denoise the cropped SAR images. After denoising, the original dataset is obtained; Next, the original dataset is divided into a training set and a test set: First, a portion of image pairs are selected from the original dataset and further cropped to form part of the training set; finally, the remaining original dataset is used as the test set. Step 2: Generate pre-fused images using the GFCE algorithm: During the generation of pre-fused images, the contrast of the SAR images is simultaneously enhanced to obtain pre-fused images with good spectral information and clear details; and select the same pre-fused image pairs as the training set for cropping, and combine them with the part of the training set obtained in Step 1 to form a complete training set. At this point, both the training set and the test set have been completed. Step 3: Construct a GAN network with dual discriminators, DD-GAN: This includes a generator and two discriminators. The generator is used to generate the fused image; the two discriminators have the same structure and are used to evaluate the quality of the fused image; and dense connection blocks and attention mechanism modules are added to the generator. The pre-fused image obtained in step 2 is then used to guide the generation of the generator. The generator's network structure includes three convolutional layers (Conv1, Conv2, and Conv3); a Denseblock module; and a Resblock_CBAM module. The input consists of channel-connected SAR and visible light images. After passing through Conv1, the output feature map is input to the Denseblock module for full feature extraction. Then, it is input to Resblock_CBAM for weighted feature extraction. The weighted feature map is then input to Conv2 for further feature extraction, and finally input to Conv3 to obtain the fused image I. g ; Step 4: Train the DD-GAN network using the training set from Step 1; Step 5: Fusion of SAR and visible light images: Input the test set from Step 1 into the DD-GAN network trained in Step 4 to obtain a fused image that retains the spectral information of the visible light image and the detailed information of the SAR and visible light image. Then, evaluate the quality of the fused image using subjective and objective evaluation metrics.

2. The SAR and visible light image fusion method based on a dual discriminator according to claim 1, Its characteristic is that the DD-GAN network described in step 3 includes two parts: a generator and a discriminator. Conv1 and Conv2 each consist of three parts: a convolutional layer, a batch normalization layer, and an activation function layer. The kernel size for both Conv1 and Conv2 is set to 3×3. Batch normalization (BN) is selected for the batch normalization layer, and LeakyReLU is chosen as the activation function. Conv3 consists of two parts: a convolutional layer and an activation function layer. The kernel size for the convolutional layer is set to 1×1, and Tanh is chosen as the activation function. The Denseblock module contains three convolutional blocks, namely Dense1, Dense2, and Dense3, with each convolutional kernel set to 3×3. Batch normalization and activation functions are BN and LeakyReLU. The output of each convolutional block is concatenated with the output of the previous convolutional block in the channel dimension to make full use of the features extracted by each convolutional layer. The Resblock_CBAM consists of three parts: Bottleneck, CBAM, and shortcut. Bottleneck contains three convolutional layers. The first convolutional layer uses a 1×1 kernel for channel compression, reducing the dimensionality of the input feature map. This layer is followed by a batch normalization (BN) layer and a ReLU activation function. The second convolutional layer uses a 3×3 kernel for feature extraction, capturing local feature information from the input feature map. This layer is also followed by a batch normalization (BN) layer and a ReLU activation function. The third convolutional layer... A 1×1 convolutional kernel is used to recover the number of channels. A feature map F of size (120, 120, 64) output from the Bottleneck is input into the CBAM, where (120, 120) represents the height and width of the feature map, and 64 represents the number of channels. First, it enters the channel attention module, where the height and width of the feature map are max-pooled and average-pooled to obtain two feature maps of size (1, 1, 64). These are then fed into the multilayer perceptron to obtain channel weights. Finally, the obtained weights are summed and fed into the sigmoid activation function, outputting a channel attention feature map of size (1, 1, 64). Then Multiplying it by the input feature map F yields a feature map of size (120, 120, 64). The input is used as the input to the spatial attention module. After inputting into the spatial attention module, firstly, max pooling and average pooling are performed on each channel of the input feature map to obtain two feature maps of size (120, 120, 1). Then, the two feature maps are concatenated along the channel dimension to obtain a feature map of size (120, 120, 2). Next, a convolutional layer is used to reduce the dimensionality of the feature map to obtain a feature map of size (120, 120, 1). Finally, the obtained feature map is fed into the sigmoid activation function to obtain the spatial attention feature map. Then and Multiplying them together yields the final output of CBAM, namely the attention feature maps in the channel and spatial directions; finally, the input of Bottleneck and the output of CBAM are shortened to obtain the final feature map. After the generator generates the fused image, it is fed into the discriminator for discrimination. The two discriminators have identical structures, each consisting of four convolutional modules and one linear layer. The first convolutional module consists of a convolutional layer and a LeakyReLU activation layer; the other three convolutional modules have the same structure, consisting of a convolutional layer, a batch normalization layer, and a LeakyReLU activation layer. The kernel size of all convolutional layers is set to 3×3, and the stride is set to 2, which rapidly reduces the width and height of the feature map. Batch normalization is selected, and to prevent the introduction of noise, padding is only performed on the input image of the first layer. The last linear layer transforms the flattened feature map into an output that represents the relative distance between the generated image and the real image. To reduce the number of parameters, the weights of the third and fourth convolutional blocks and the linear layer are shared.

3. The SAR and visible light image fusion method based on a dual discriminator according to claim 1, The characteristic of step 4 is that the process of training the DD-GAN network includes: inputting the SAR and visible light image training set samples obtained in step 2 into the DD-GAN network for training; setting the learning rate of the generator to 0.0001 and the learning rate of the discriminator to 0.0001 to balance the learning rates of the generator and the discriminator, and selecting Adam as the optimizer; the ratio of the number of training iterations of the discriminator to the generator is 1:1, the batch size is 32, and a total of 10 epochs are trained; and the network hyperparameters are continuously updated iteratively, and the training process of the network is completed when the number of iterations reaches the set number of iterations.

Images