An image fusion method and system based on dual discriminator generative adversarial network

Abstract

The present invention discloses an image fusion method and system based on a dual-discriminator generative adversarial network, belonging to the field of computer vision. The method includes constructing and training a dual-discriminator generative adversarial image fusion model; wherein the generator includes a dense feature extraction and fusion module, a feature enhancement module, and a decoding and reconstruction module; the dense feature extraction and fusion module extracts and fuses the features of the two frames of images to be fused to obtain a fused feature map; the feature enhancement module performs a global average pooling operation, two fully connected layers, and a Sigmoid activation function on the fused feature map to obtain a feature enhancement coefficient, and then multiplies the feature enhancement coefficient with the fused feature map to obtain an enhanced feature map; and the decoding and reconstruction module decodes and reconstructs the enhanced feature map to obtain a fused image. The present invention effectively retains the information in different source images, while improving the generator's ability to fit and model the fusion of different source images, thereby improving the quality of the fused image.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and more specifically, relates to an image fusion method and system based on a dual-discriminator generative adversarial network. Background Art

[0002] With the rapid development of sensor technology, different types of sensors, such as visible light, infrared, and lidar, are widely used in target detection, tracking, monitoring, and early warning systems. Different sensor types acquire different information, and a single sensor cannot provide all the information required for certain applications. Using image fusion technology to fuse sensor information from different sources to improve the performance of subsequent processing tasks is an important research direction.

[0003] Traditional image fusion methods use manually designed algorithms to extract image features for fusion, but this struggles to obtain effective feature representations for images from different sensor types. The resulting feature fusion strategies are crude, making it difficult to effectively preserve the specific information in the images from different sensors, resulting in limited image fusion performance. Deep learning-based image fusion methods leverage the hierarchical distributed feature representation capabilities of deep learning networks to adaptively extract image features to obtain activity level measurements and fusion weights, effectively improving the quality of fused images. However, image fusion lacks ground truth, making it difficult to train deep learning networks using supervised learning methods and effectively preserve the specific information from different source images. Generative adversarial networks (GANs) leverage the fitting capabilities of deep learning networks to implicitly model the generative process and employ a discriminator for supervised training, making them an effective approach for generating images. However, in image fusion applications, designing the generator network model, loss function, and training method to both fit the model for fusion of different source images and effectively preserve the specific information from these images, thereby improving the quality of the fused image, remains an unresolved challenge. Summary of the Invention

[0004] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides an image fusion method and system based on a dual-discriminator generative adversarial network, which aims to solve the technical problem that the generator network model in the image fusion method based on the generative adversarial network is difficult to effectively fit the model of fusion of different source images while retaining specific information in different source images.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for image fusion based on a dual-discriminator generative adversarial network is provided, comprising:

[0006] S1. Construct and train a dual-discriminator generative adversarial image fusion model; the dual-discriminator generative adversarial image fusion model includes a generator and two discriminators, wherein the generator includes a dense feature extraction and fusion module, a feature enhancement module, and a decoding and reconstruction module; the dense feature extraction and fusion module extracts and fuses the features of the two frames of images to be fused to obtain a fused feature map; the feature enhancement module performs a global average pooling operation, two fully connected layers, and a Sigmoid activation function on the fused feature map to obtain a feature enhancement coefficient, and then multiplies the feature enhancement coefficient with the fused feature map to obtain an enhanced feature map; the decoding and reconstruction module decodes and reconstructs the enhanced feature map to obtain a fused image;

[0007] S2. Input the two frames of images to be fused into the dual discriminator to generate an adversarial image fusion model, and the generator outputs the image fusion result.

[0008] Furthermore, the generator loss function is:

[0009]

[0010] Among them, L adv1 is the adversarial loss between the generator and the discriminator 1, L adv2 is the adversarial loss between the generator and the discriminator 2, SSIM1 and SSIM2 are the structural similarity coefficients between the generator's generated image and image 1 and image 2, and L content is the content loss, α is the balance coefficient;

[0011]

[0012] v and i represent the two frames of images to be fused, G(v,i) represents the fused image output by the generator, D1(·) represents the output of discriminator 1, D2(·) represents the output of discriminator 2, and N represents the batch size.

[0013] Furthermore, the decoding and reconstruction module consists of multiple convolutional layers, each layer uses batch normalization, the last convolutional layer uses the tanh activation function, and the remaining convolutional layers use the ReLU activation function.

[0014] Furthermore, each discriminator includes multiple convolutional layers and one linear layer connected in sequence.

[0015] Furthermore, the two discriminator loss functions are defined as follows:

[0016]

[0017] Here, v and i represent the two frames of images to be fused, G(v,i) represents the fused image output by the generator, D1(·) represents the output of discriminator 1, D2(·) represents the output of discriminator 2, and N represents the batch size.

[0018] Furthermore, the dense feature extraction fusion module consists of multiple convolutional layers; each layer connects the output features of all previous layers on the channel as input.

[0019] The present invention also provides an image fusion system based on a dual-discriminator generative adversarial network, comprising:

[0020] A model construction and training module is provided for constructing and training a dual-discriminator generative adversarial image fusion model; the dual-discriminator generative adversarial image fusion model includes a generator and two discriminators, wherein the generator includes a dense feature extraction and fusion module, a feature enhancement module, and a decoding and reconstruction module; the dense feature extraction and fusion module extracts and fuses the features of the two frames of images to be fused to obtain a fused feature map; the feature enhancement module performs a global average pooling operation, two fully connected layers, and a Sigmoid activation function on the fused feature map to obtain a feature enhancement coefficient, and then multiplies the feature enhancement coefficient with the fused feature map to obtain an enhanced feature map; the decoding and reconstruction module decodes and reconstructs the enhanced feature map to obtain a fused image;

[0021] In the online fusion module, the two frames of images to be fused are input into the dual discriminator to generate an adversarial image fusion model, and the generator outputs the image fusion result.

[0022] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art.

[0023] (1) The present invention designs a network structure of the generator, adopts a dense feature extraction and fusion module to extract and fuse multi-layer features, uses a feature enhancement module to further enhance the features, and inputs the enhanced features into a decoding and reconstruction module composed of multiple convolutional layers in series to obtain a fused image. The three modules work together to effectively retain the information in different source images, while improving the generator's ability to fit and model the fusion of different source images and improve the quality of the fused image.

[0024] (2) The present invention designs an image fusion method based on a dual-discriminator generative adversarial network, which adopts a dual discriminator combined with a comprehensive adversarial content loss function. By introducing gradient information loss and pixel intensity loss, and adopting a balance coefficient based on the structural similarity coefficient to dynamically adjust the balance between the two adversarial losses, while solving the instability phenomenon that occurs during the training process of the generative adversarial network, it also more effectively retains the specific information contained in the image to be fused, thereby improving the performance of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of an image fusion method based on a dual-discriminator generative adversarial network provided by an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of the architecture of a dual-discriminator generative adversarial image fusion network model provided in an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of the generator structure of a dual-discriminator generative adversarial image fusion network provided by an embodiment of the present invention;

[0028] Figure 4 Schematic diagram of the discriminator structure of the dual discriminator generative adversarial image fusion network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0030] Figure 1 For a flowchart of the image fusion method based on a dual discriminator generative adversarial network provided by an embodiment of the present invention, see Figure 1 , combined with Figure 2-Figure 4 , the image fusion method based on the dual discriminator generative adversarial network in this embodiment is described in detail, and the method includes the following steps.

[0031] S1, build and train a dual discriminator to generate an adversarial image fusion network model, see Figure 2 , the dual discriminator generation adversarial image fusion network model includes a generator and two discriminators, see Figure 3 The generator includes a dense feature extraction and fusion module, a feature enhancement module and a decoding and reconstruction module.

[0032] The dense feature extraction and fusion module extracts and fuses the features of the two images to be fused to produce a fused feature map. This module consists of six convolutional layers, each with a 3×3 kernel size, a stride of 1, and 48 kernels. The output features of the previous convolutional layers are concatenated across channels and serve as the input to the current convolutional layer. After the two images to be fused are merged, the dense feature extraction and fusion module performs feature extraction and fusion, outputting a fused feature map.

[0033] The feature enhancement module performs channel feature enhancement on the fused feature map to obtain an enhanced feature map. Figure 3The feature enhancement module performs a global average pooling operation, two fully connected layers, and a Sigmoid activation function on the fused feature map to obtain a feature enhancement coefficient. The feature enhancement coefficient is then multiplied by the fused feature map to obtain an enhanced feature map. The calculation process is as follows:

[0034] F′=Sigmoid(W fc2 (W fc1 (AveragePool(F))))⊙F

[0035] Among them, F′ represents the enhanced feature map, F represents the fused feature map, AveragePool(*) represents the global average pooling layer processing, W fc1 (*) and W fc2 (*) represents the first and second linear layer processing respectively, Sigmoid (*) represents the Sigmoid function, and ⊙ represents the Hadamard product operation.

[0036] The decoding and reconstruction module decodes and reconstructs the enhanced feature map to produce a fused image. The decoding and reconstruction module consists of five convolutional layers. Each convolutional layer has a kernel size of 3×3, a stride of 1, and the number of kernels is 240, 128, 64, 32, and 1, respectively. Batch normalization is applied to each layer. The last convolutional layer uses the tanh activation function, and the remaining convolutional layers use the ReLU activation function.

[0037] See Figure 4 In this embodiment, each discriminator consists of three convolutional layers and one linear layer. The convolution kernel size of each convolutional layer is 3×3, the stride is 2, and the number of convolution kernels is 16, 32, and 64, respectively. The output of each layer is batch normalized, and the activation function is ReLU. The input of each discriminator is the output image of the generator and one of the two input frames of the generator. The output of the discriminator is the JS divergence.

[0038] The loss function of the generator includes the generator loss function and two discriminator loss functions, and the dual discriminator generative adversarial image fusion network model is trained in a two-stage alternating manner.

[0039] The generator loss function is a comprehensive adversarial content loss function, which is composed of the adversarial loss L adv1 、L adv2 and content loss L content It consists of three parts, which are defined as follows:

[0040]

[0041] Among them, L adv1 is the adversarial loss between the generator and the discriminator 1, L adv2is the adversarial loss between the generator and the discriminator 2, SSIM1 and SSIM2 are the structural similarity coefficients between the generator's generated image and image 1 and image 2, and L content is the content loss, α is the balance coefficient, and preferably α is 0.6.

[0042] Adversarial loss L adv1 and L adv2 is defined as follows:

[0043]

[0044] Here, v and i represent the two frames of images to be fused, G(v,i) represents the fused image output by the generator, D1(·) represents the output of discriminator 1, D2(·) represents the output of discriminator 2, and N represents the batch size.

[0045] The calculation formula of the structural similarity coefficient SSIM is as follows:

[0046]

[0047] SSIM=SSIM A,F +SSI M B ,F

[0048] Among them, A and B represent the images to be fused, F represents the fusion result image, X represents A or B, x and f represent the sliding window image blocks of image X and image F respectively, u x and u f represents the grayscale mean of image blocks x and f, σ x and σ f represents the grayscale variance of image blocks x and f, σ xf represents the covariance between image block x and image block f, and C1, C2 and C3 are constants.

[0049] Content loss L content is defined as follows:

[0050]

[0051] Among them, v and i represent the two frames of images to be fused, H and W are the height and width of the image respectively, G(v,i) represents the fused image output by the generator, ‖·‖ F represents the F-norm, ‖·‖ TV represents the TV norm, β is the balance coefficient, and preferably β is 2.8.

[0052] The two discriminator loss functions are defined as follows:

[0053]

[0054] Here, v and i represent the two frames of images to be fused, G(v,i) represents the fused image output by the generator, D1(·) represents the output of discriminator 1, D2(·) represents the output of discriminator 2, and N represents the batch size.

[0055] The two-stage alternating training includes a discriminator training phase and a generator training phase. In the discriminator training phase, the generator is kept unchanged and the two discriminators are trained with the goal of maximizing the two discriminator loss functions. In the generator training phase, the discriminator is kept unchanged and the generator is trained with the goal of minimizing the comprehensive adversarial content loss function.

[0056] S2. Input the two frames of images to be fused into the dual discriminator to generate an adversarial image fusion model, and the generator outputs the image fusion result.

[0057] To verify the image fusion results of the dual-discriminator generative adversarial network-based image fusion method in this embodiment of the present invention, a dataset for image fusion was constructed based on the TNO image fusion dataset. Forty pairs of infrared and visible light images were selected from the TNO dataset and cropped with overlapping layers using a step size of 14 to serve as the training set. Seven pairs of infrared and visible light images were also selected from the TNO dataset as the test set.

[0058] After the data set is prepared, the network model is trained. The batch size is set to 12 during training, the initial learning rate is 0.002, and the exponential decay is reduced to 0.8 of the original value after each training batch.

[0059] The trained dual-discriminator-based generative adversarial image fusion network is used to test the test set. At the same time, several other classic image fusion methods are used for testing. The average values ​​of eight performance indicators, including information entropy (EN), correlation coefficient (CC), mutual information (MI), standard deviation (SD), average gradient (AG), mean square error (MSE), difference correlation sum (SCD) and peak signal-to-noise ratio (PSNR), are used for comparison. The experimental results are shown in Table 1.

[0060] Table 1 Comparison of average performance indicators obtained using various image fusion methods

[0061] method EN CC MI SD AG MSE SCD PSNR CBF 6.7343 0.4799 13.4686 0.1268 0.0193 0.0256 1.2951 63.6399 JSR 6.7490 0.5553 13.4980 0.1500 0.0181 0.0615 1.6047 60.3482 DCHWT 6.7654 0.5335 13.5209 0.1242 0.0143 0.0249 1.3141 63.9527 GTF 6.7755 0.4784 13.5510 0.1358 0.0137 0.0300 0.9289 63.6970 MEFGAN 6.4260 0.5382 12.8521 0.1113 0.0088 0.0347 1.0914 62.9152 FusionGAN 6.4260 0.5382 12.8521 0.1113 0.0088 0.0347 1.0914 62.9152 Sia-Fusion 6.7510 0.6399 13.7469 0.1321 0.0155 0.0189 1.6413 65.9007 This embodiment 7.2440 0.5775 14.4880 0.1868 0.0637 0.0580 1.6519 61.9325

[0062] The experimental results in Table 1 show that, among the eight performance indicators, this embodiment achieves the best performance in five indicators: information entropy (EN), mutual information (MI), standard deviation (SD), average gradient (AG), and correlation sum of differences (SCD). It also achieves the highest number of optimal indicators among all methods, and achieves the second-best result among all fusion methods in terms of correlation coefficient (CC). This embodiment has certain advantages over other image fusion methods.

[0063] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image fusion method based on a dual-discriminator generative adversarial network, characterized in that: include: S1. Construct and train a dual-discriminator generative adversarial image fusion model; the dual-discriminator generative adversarial image fusion model includes a generator and two discriminators, wherein the generator includes a dense feature extraction and fusion module, a feature enhancement module, and a decoding and reconstruction module; the dense feature extraction and fusion module extracts and fuses the features of the two frames of images to be fused to obtain a fused feature map; the feature enhancement module performs a global average pooling operation, two fully connected layers, and a Sigmoid activation function on the fused feature map to obtain a feature enhancement coefficient, and then multiplies the feature enhancement coefficient with the fused feature map to obtain an enhanced feature map; the decoding and reconstruction module decodes and reconstructs the enhanced feature map to obtain a fused image; S2. Input the two frames of images to be fused into the dual discriminator to generate an adversarial image fusion model, and the generator outputs the image fusion result; The generator loss function is: Among them, L adv1 is the adversarial loss between the generator and the discriminator 1, L adv2 is the adversarial loss between the generator and the discriminator 2, SSIM1 and SSIM2 are the structural similarity coefficients between the generator's generated image and image 1 and image 2, and L content is the content loss, α is the balance coefficient; v and i represent the two frames of images to be fused, G(v,i) represents the fused image output by the generator, D1(·) represents the output of discriminator 1, D2(·) represents the output of discriminator 2, and N represents the batch size.

2. The image fusion method based on a dual discriminator generative adversarial network according to claim 1, characterized in that: The decoding and reconstruction module consists of multiple convolutional layers, each layer uses batch normalization, the last convolutional layer uses the tanh activation function, and the remaining convolutional layers use the ReLU activation function.

3. The image fusion method based on a dual discriminator generative adversarial network according to claim 1, characterized in that: Each discriminator consists of multiple convolutional layers and one linear layer connected in sequence.

4. The image fusion method based on a dual discriminator generative adversarial network according to claim 3, characterized in that: The two discriminator loss functions are defined as follows: Here, v and i represent the two frames of images to be fused, G(v,i) represents the fused image output by the generator, D1(·) represents the output of discriminator 1, D2(·) represents the output of discriminator 2, and N represents the batch size.

5. The image fusion method based on a dual discriminator generative adversarial network according to any one of claims 1 to 4, characterized in that: The dense feature extraction fusion module consists of multiple convolutional layers; each layer connects the output features of all previous layers on the channel as input.

6. An image fusion system based on a dual-discriminator generative adversarial network, characterized in that: include: Model building and training module, building and training a dual discriminator to generate an adversarial image fusion model; The dual-discriminator generative adversarial image fusion model includes a generator and two discriminators, wherein the generator includes a dense feature extraction and fusion module, a feature enhancement module and a decoding and reconstruction module; the dense feature extraction and fusion module extracts and fuses the features of the two frames of images to be fused to obtain a fused feature map; the feature enhancement module performs a global average pooling operation, two fully connected layers, and a Sigmoid activation function on the fused feature map to obtain a feature enhancement coefficient, and then multiplies the feature enhancement coefficient with the fused feature map to obtain an enhanced feature map; the decoding and reconstruction module decodes and reconstructs the enhanced feature map to obtain a fused image; The online fusion module inputs the two frames of images to be fused into the dual discriminator to generate an adversarial image fusion model, and the generator outputs the image fusion result; Among them, the generator loss function is: Among them, L adv1 is the adversarial loss between the generator and the discriminator 1, L adv2 is the adversarial loss between the generator and the discriminator 2, SSIM1 and SSIM2 are the structural similarity coefficients between the generator's generated image and image 1 and image 2, and L comtent is the content loss, α is the balance coefficient; v and i represent the two frames of images to be fused, G(v,i) represents the fused image output by the generator, D1(·) represents the output of discriminator 1, D2(·) represents the output of discriminator 2, and N represents the batch size.

7. A computer-readable medium storing a computer program, wherein the computer-readable medium implements the steps of the method according to any one of claims 1 to 5 when the computer program is executed by a processor.

Images