Cross-domain image fusion algorithm based on saliency weighting and generative adversarial network

Abstract

The invention discloses a cross-domain image fusion algorithm based on saliency weighting and a generative adversarial network, and the algorithm improves the feature extraction and reconstruction capabilities of an encoder and a decoder on a high-resolution image through introducing a Restormer module, and generates a saliency mask based on an infrared image through employing a saliency injection module. The cross-modal features are dynamically weighted to focus high-value regions, and computing resources are saved. Besides, a generative adversarial network (GAN) and a double-discriminator mechanism are introduced, authenticity, visual quality and information fidelity of a fused image are improved through adversarial training, and distribution consistency of a fusion result and a source mode is constrained through double discriminators. Therefore, the problem of artifacts possibly existing in a traditional method is solved, and the naturalness of fusion is improved. According to the method, an excellent fusion effect is shown on a plurality of data sets, and compared with an existing method, the training efficiency and the model performance are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a cross-domain image fusion algorithm based on saliency weighting and generative adversarial networks, for fusing images with different modal features. Background Technology

[0002] With the rapid development of sensor technology and artificial intelligence, the demand for human perception of complex environments is increasing, making multimodal image fusion technology a key research direction. In particular, infrared and visible light images play an important role in many application scenarios due to their complementary imaging mechanisms. Visible light images can capture rich texture details and color information, but their image quality is easily affected by lighting conditions (such as nighttime, strong light, and shadows), leading to loss of detail or decreased contrast. In contrast, infrared images can penetrate harsh environments and capture the thermal radiation information of targets, but they are relatively blurry in terms of resolution and texture representation. Therefore, effectively fusing these two modalities to integrate their complementary features is of great significance for improving target recognition capabilities, scene understanding accuracy, and meeting all-weather operation requirements.

[0003] Existing image fusion techniques face multiple challenges in handling complex environment perception. Particularly in high-resolution image processing, the performance limitations of some deep learning models in feature extraction and reconstruction affect the accurate preservation of detail information. Furthermore, in extreme imaging environments, existing methods struggle to reliably preserve key target and scene details and are prone to artifacts, impacting the realism and naturalness of the fusion results. To achieve a fusion effect that highlights the thermal radiation characteristics of infrared targets while fully preserving the rich details and natural colors of visible light, avoiding information bias, and balancing algorithm efficiency and accuracy to meet practical deployment needs, developing an image fusion method that effectively improves detail preservation, optimizes fusion results in extreme environments, enhances the realism and naturalness of the fused image, and maintains efficiency is crucial. Summary of the Invention

[0004] Purpose of the invention: This invention aims to provide a cross-domain image fusion algorithm based on saliency weighting and generative adversarial networks to address the shortcomings of existing technologies in high-resolution image detail processing, extreme environment adaptability, realism and naturalness of fusion results, and training efficiency.

[0005] To achieve the above objectives, the technical solution provided by this invention is: a cross-domain image fusion algorithm based on saliency-weighted summative generative adversarial networks, the core technical solution of which includes:

[0006] Integrated Restormer Block: The Restormer module is embedded in the generator (encoder and decoder). This module significantly enhances the model's feature extraction and reconstruction capabilities for high-resolution images by cascading a Multi-Depth Convolutional Head Transposed Attention Module (MDTA) and a Gated Deep Convolutional Feedforward Network (GDFN).

[0007] Saliency Injection Module: Design a saliency injection module that generates a saliency mask by calculating the saliency map of the infrared image, and uses the mask to dynamically weight and fuse the extracted infrared and visible light features to focus on high-value target areas;

[0008] Dual discriminator network and adversarial training mechanism: The dual discriminators constrain the distribution consistency between the fused image and the visible light and infrared images respectively, thereby precisely ensuring the realism, naturalness, and modal consistency of the fusion result. Through adversarial training between the generator and the dual discriminators, the model is guided to generate high-quality fused images that are detail-preserving, target-prominent, and visually natural.

[0009] Loss function design: includes fusion loss (such as intensity loss, structural similarity loss, gradient loss) to ensure the fidelity of the fused image; includes generator adversarial loss and discriminator loss to guide the adversarial training process of the model.

[0010] This invention provides a cross-domain image fusion algorithm based on saliency weighting and generative adversarial networks, the technical solution of which includes the following steps:

[0011] S1: Input a pair of infrared and visible light images;

[0012] S2: Feature extraction and fusion of generator networks;

[0013] S3: Image reconstruction using a generator network;

[0014] S4: Dual Discriminator Evaluation and Adversarial Training;

[0015] S5: Repeat steps S2 to S4, continuously optimizing the generator through adversarial training between the generator and the dual discriminator until a high-quality fused image is output.

[0016] Furthermore, the specific method of step S2 is as follows:

[0017] S2.1: The input infrared and visible light images are processed by the encoder of the generator network and integrated with the Restormer module to extract high-resolution features. The Restormer module achieves effective extraction and preliminary representation of multi-scale features through its internal MDTA and GDFN structures;

[0018] S2.2: Weighted fusion using the saliency injection module: Calculate the saliency map S(k) of the infrared image = ∑ j∈I d(k,j), and through the formula Generate a saliency mask S Mask Using a saliency mask, the infrared and visible light features extracted by the encoder are weighted and modulated. The specific formula is as follows:

[0019] F' IR =Concat(F IR ,S Mask F IR ) and F' VIS =Concat(F VIS ,(1-S Mask )F VIS The weighted and modulated features are then fused to obtain the fused feature F. f =F' IR +F' VIS ;

[0020] Furthermore, the specific method for step S3 is as follows:

[0021] S3.1: Generating a fused image using a decoder: The fused features F obtained in S2 are... f The input is fed into the decoder of the generator network, where it undergoes layer-by-layer processing and reconstruction, ultimately outputting a fused image I. f , where I f =D(F1) f F2 f F3 f F4 f ), Fi f =S(Fi IR ,Fi VIS ), i = 1, 2, 3, 4;

[0022] Furthermore, the specific method for step S4 is as follows:

[0023] S4.1: Discriminator 1 evaluates the consistency with the visible light image. The generated fused image I... f And the original visible light image I VIS The input is fed into the first discriminator D1(·), which is used to evaluate the distribution consistency between the two and output the discrimination result;

[0024] S4.2: Discriminator 2 evaluates the consistency with the infrared image. The generated fused image I... f And the original visible light image I VIS The input is fed into the second discriminator D2(·), which is used to evaluate the distribution consistency between the two and output the discrimination result;

[0025] S4.3: Calculate the loss function. The loss function includes the original image reconstruction loss L. recon Fusion loss L fuse Generator-versus-loss L G_adv and discriminator loss L D The original image reconstruction loss consists of intensity loss. and structural similarity L SSIM The loss consists of, for example, infrared images, reconstruction loss L. IR The specific formula is as follows:

[0026] in:

[0027] L SSIM (I IR , I′ IR ) = 1 - SSIM(I IR , I' IR );

[0028] Generate adversarial loss L G_adv =E[log(1-D1(G(I)] IR ,I VIS )))]+E[log(1-D2(G(I IR ,I VIS )))];

[0029] Discriminator loss in:

[0030]

[0031] S4.4: Based on the calculated total loss function The parameters of the generator and discriminator are updated through backpropagation.

[0032] Further explanation regarding key technology modules:

[0033] The Restormer module is integrated into the encoder and decoder of the generator. Its structure includes a Multi-Depth Convolutional Head Transposed Attention (MDTA) module and a Gated Deep Convolutional Feedforward Network (GDFN). The MDTA part contains normalization, attention computation, and residual connections, while the GDFN part contains parallel branching and feature integration. Its specific computation process is as follows:

[0034] X Norm = Norm(X) Q,K,V=DConv 3×3 (Conv 1×1 (X)) X Atten=Attention(Q,K,V) X MDTA =Conv 1×1 (X Atten )+X

[0035] Saliency Injection Module: This module quantifies the perceptual importance of image regions. The saliency value is calculated using the formula: S(k) = ∑ j∈I d(k,j), the significance plot is obtained through the formula It is binarized into a saliency mask. The saliency mask is used to weight infrared and visible light features, achieving dynamic feature fusion. The specific calculation is as follows:

[0036] F' IR =Concat(F IR ,S Mask F IR ) and F' VIS =Concat(F VIS ,(1-S Mask )F VIS ), eventually merging into F f =F' IR +F' VIS ;

[0037] Dual discriminator structure: Each discriminator adopts a four-level cascaded structure, including a basic feature extraction layer, a normalized feature processing layer, and a fully connected layer, which are used to extract discriminative features from the input image and output normalized discrimination results;

[0038] Two-stage training strategy: to ensure that the model can learn gradually, first establish a good basic feature extraction capability (stage one), and then optimize the fusion effect through adversarial mechanism (stage two). Attached Figure Description

[0039] Figure 1 This is the overall network framework in this embodiment of the invention;

[0040] Figure 2 This is the generator structure in the embodiments of the present invention;

[0041] Figure 3 This is the Restormer block in this embodiment of the invention;

[0042] Figure 4 This is the saliency injection module in the embodiments of the present invention;

[0043] Figure 5 This is the discriminator structure in the embodiments of the present invention. Detailed Implementation

[0044] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and examples, in order to help understand the technical solution of the present invention, but should not be construed as limiting the scope of protection of the present invention.

[0045] This invention mainly comprises three core modules: a generator, a saliency injection module, and a dual discriminator adversarial mechanism.

[0046] Generator Structure: The generator adopts a U-Net-like architecture with an encoder-decoder. To enhance the model's feature extraction and reconstruction capabilities on high-resolution images, this invention introduces Restormer blocks in several key layers of the encoder and decoder;

[0047] The encoder section comprises multiple feature extraction stages. Each stage further processes the previously extracted features and incorporates a Restormer block to enhance the expressive power of the features. At the end of each stage, downsampling operations (such as max pooling) are typically performed to reduce the size of the feature map and increase the receptive field.

[0048] Decoder section: The decoder structure is mirror-symmetric to the encoder, gradually restoring the resolution of the feature maps through upsampling operations. At each layer of the decoder, its upsampled features are added point-by-point to the output features of the corresponding stage of the encoder (enhanced by the Restormer block). This skip connection mechanism helps preserve high-frequency details and texture information and alleviates the gradient vanishing problem.

[0049] Integration of the Restormer Block: The Restormer block is an efficient module that integrates Multi-Head Transposed Attention (MDTA) and Gated Deep Convolutional Feedforward Network (GDFN). MDTA captures long-range dependencies through an attention mechanism, while GDFN provides fine-grained processing of local features. This design enables the generator to better capture and fuse complementary information from different modalities.

[0050] Saliency Injection Module: The core of this module is to use saliency detection to guide feature fusion, focusing more computational resources and attention on target regions with higher perceptual value;

[0051] Saliency Calculation and Mask Generation: Saliency values ​​are calculated based on the Euclidean distance between pixels. A saliency map is obtained through calculation and then binarized to generate a saliency mask S. Mask This mask highlights salient regions in the image. Furthermore, to simplify computation and focus on the most valuable information, saliency detection and mask generation are performed solely based on the infrared image.

[0052] Feature injection method: A saliency mask is used to dynamically weight the infrared and visible light features extracted by the encoder. Specifically, infrared features are injected with a saliency mask S. Mask Multiplication enhances the signal, while visible light features are enhanced by interaction with (1-S) Mask The features are multiplied and weighted. The weighted features are then concatenated to highlight key objectives and effectively integrate cross-modal information.

[0053] Dual-Discriminator Adversarial Mechanism: To generate more realistic, detailed, and visually natural fused images, this invention introduces a dual-discriminator adversarial mechanism. This mechanism consists of a generator and two discriminators, which together form an adversarial training framework.

[0054] Discriminator 1: This discriminator focuses on evaluating the similarity in detail and texture between the fused image and the visible light image. It is designed to train the generator to produce fused results that preserve the rich texture and edge details of the visible light image.

[0055] Discriminator 2: This discriminator focuses on evaluating the degree of integration of thermal radiation information between the fused image and the infrared image. It aims to ensure that the fused image can effectively capture and retain the key thermal target information provided by the infrared image;

[0056] Adversarial training process: The generator learns to generate more realistic, detailed fused images that can take into account both infrared and visible light modal information by continuously trying to generate images that can deceive the two discriminators.

[0057] The loss function consists of the following parts: Intensity loss: constrains pixel-level fidelity; Structural similarity loss: maintains the overall structural integrity of the fused image; Gradient loss: preserves the detailed texture and contour features of the image; Generator adversarial loss: used to drive the generator to generate more realistic and natural fused images, optimized by adversarial judgment of the discriminator; Discriminator loss: used to train the discriminator to accurately distinguish between real and generated images.

[0058] This invention is applicable to scenarios such as target detection in harsh environments, visual perception for autonomous driving, and security monitoring. It can provide clear, high-contrast images and enhance target recognition and environmental perception capabilities.

Claims

1. A cross-domain image fusion algorithm based on saliency-weighted summative generative adversarial networks, characterized in that, Includes the following steps: S1: Input a pair of infrared and visible light images; S2: Feature extraction and fusion of generator networks; S3: Image reconstruction using a generator network; S4: Dual Discriminator Evaluation and Adversarial Training; S5: Repeat steps S2 to S4, continuously optimizing the generator through adversarial training between the generator and the dual discriminator until a high-quality fused image is output.

2. The cross-domain image fusion algorithm based on saliency-weighted sum and generative adversarial network according to claim 1, characterized in that, Step S2 includes: S2.1: The input infrared and visible light images are processed by the encoder of the generator network and integrated with the Restormer module to extract high-resolution features. The Restormer module achieves effective extraction and preliminary representation of multi-scale features through its internal MDTA and GDFN structures; S2.2: Weighted fusion using the saliency injection module: Calculate the saliency map S(k) of the infrared image = ∑ j∈I d(k,j), and through the formula A saliency mask is generated, and then the infrared and visible light features extracted by the encoder are weighted and modulated using the saliency mask. The specific formula is as follows: F' IR =Concat(F IR ,S Mask F IR );F' VIS =Concat(F VIS ,(1-S Mask )F VIS ), The weighted and modulated features are fused to obtain the fused feature F. f =F' IR +F' VIS .

3. The cross-domain image fusion algorithm based on saliency-weighted sum and generative adversarial network according to claim 1, characterized in that, Step S3 includes: S3.1: Generating a fused image using a decoder: The fused features F obtained in S2 are... f The input is fed into the decoder of the generator network, where it undergoes layer-by-layer processing and reconstruction, ultimately outputting a fused image I. f ,in: I f D(F1) f F2 f F3 f F4 f );Fi f S(Fi IR Fi VIS ),i=1,2,3,4。 4. The cross-domain image fusion algorithm based on saliency-weighted sum and generative adversarial network according to claim 1, characterized in that, Step S4 includes: S4.1: Discriminator 1 evaluates the consistency with the visible light image. The generated fused image I... f And the original visible light image I VIS The input is fed into the first discriminator D1(·), which is used to evaluate the distribution consistency between the two and output the discrimination result; S4.2: Discriminator 2 evaluates the consistency with the infrared image. The generated fused image I... f And the original visible light image I VIS The input is fed into the second discriminator D2(·), which is used to evaluate the distribution consistency between the two and output the discrimination result; S4.3: Calculate the loss function. The loss function includes the original image reconstruction loss L. recon Fusion loss L fuse Generator-versus-loss L G_adv and discriminator loss L D . Generator Adversarial Loss L G_adv The specific formula is as follows: L G_adv =E[log(1-D1(G(I IR ,I VIS )))]+E[log(1-D2(G(I IR ,I VIS )))]; Discriminator loss L D The specific formula is as follows: in: S4.4: Based on the calculated total loss function L G =L Gadv +λ4L fuse The parameters of the generator and discriminator are updated through backpropagation.

Images