An infrared and visible light image fusion system and method based on a multi-branch autoencoder

Abstract

The present application belongs to the technical field of image processing, aiming at the technical problem that the method combining CNN and attention mechanism has made progress in the collaborative extraction of local and global features, but still needs further optimization in fully extracting and retaining complementary features, the present application proposes an infrared and visible light image fusion system and method based on a multi-branch autoencoder, which extracts the complementary features of the fused infrared and visible light images, the extracted low-frequency global features and high-frequency local features, and finally generates a fusion image based on the Restormer module of the multi-head attention mechanism. Since the present application fully combines the advantages of convolutional neural network (CNN) and attention mechanism in feature extraction, the quality of the fusion image is significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an infrared and visible light image fusion system and method based on a multi-branch autoencoder. Background Technology

[0002] Image fusion is a technique designed to enhance images of the same scene acquired simultaneously by different sensors, generating richer, more detailed images with improved visual quality. This provides more reliable foundational data for subsequent image processing tasks. Infrared and visible light image fusion is an important subfield of multimodal image fusion with broad application prospects. Specifically, infrared images, by capturing the thermal radiation information of objects, can effectively reveal the contours and features of targets under adverse weather conditions. Visible light images, by capturing reflected light from the scene, better reveal the texture details of targets that conform to human visual perception. Leveraging the advantages of both, significant complementary features can be extracted from infrared and visible light images, overcoming the limitations of high noise and low resolution in infrared images, and the sensitivity of visible light images to changes in illumination. These fused image features can provide richer foundational information for subsequent image processing tasks, such as object detection, object tracking, and semantic segmentation.

[0003] In recent years, infrared and visible light image fusion methods have been mainly divided into traditional methods and deep learning-based methods. Traditional methods encompass spatial domain, transform domain, statistical optimization, and filtering techniques. In the spatial domain, fusion methods achieve fusion through pixel-level weighting or by selecting appropriate pixel values ​​based on criteria such as brightness and contrast. For example, the method proposed by Wang et al., based on high-pass filtering and mean coefficient ratio, exhibits excellent detail preservation capabilities in low-light and complex scenes; Aparna et al.'s entropy-based multi-exposure image fusion method further enhances image contrast and detail. In the transform domain, Nencini et al. used curvelet transform to capture multi-scale edge and texture information, while Duan et al. combined wavelet transform and saliency detection to achieve good results in detail preservation and salient feature enhancement.

[0004] Statistical and optimization methods integrate infrared and visible light features by analyzing the statistical characteristics of images and combining them with optimization algorithms. For example, Ma et al. significantly improved the detail representation of the fused image through gradient propagation and minimization of total variation. Filtering-based methods utilize the spatial frequency feature decomposition of images to organically combine edge, detail, and low-frequency information from infrared and visible light. For instance, Zhang et al. achieved effective fusion of detail information and improved visual effects through guided filtering and sparse representation. These methods each have advantages in different application scenarios, providing a rich theoretical foundation and practical support for infrared and visible light image fusion, and promoting the continuous development of this field.

[0005] While traditional methods demonstrate effectiveness in specific scenarios, they often struggle to fully integrate multimodal features in complex environments and are susceptible to noise interference and resolution limitations. With the rapid development of deep learning technology in computer vision, methods such as Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANs), and attention mechanisms have been applied to image fusion tasks. CNN-based fusion methods typically design appropriate CNN structures to extract different features (such as edges, textures, and contrast) from infrared and visible light images, and select fusion strategies (such as weighted averaging, feature concatenation, or additive fusion) according to task requirements, ultimately generating the fused result through image reconstruction. For example, Li et al. designed the DenseBlock deep convolutional network, focusing on extracting detailed information from infrared and visible light images and combining it with convolutional features for effective fusion. This method has significant advantages in detail preservation, significantly improving the clarity of the fused image. Zhang et al., on the other hand, extracted multi-scale features from images using CNNs and employed an adaptive weighting strategy for fusion, successfully achieving effective integration of different modal information and improving the clarity and contrast of the fused image. GAN-based fusion methods utilize the generator and discriminator in a generative adversarial network (GAN) to generate more realistic fused images through adversarial training, and further improve the fusion effect through adversarial loss and content loss. For example, Ma et al. first applied GAN to the fusion of infrared and visible light images. The generator learned the common features of the two images, and the discriminator helped the generator produce a more realistic fused image, thus significantly improving the fusion effect. Li et al. introduced multiple constraints to optimize GAN training, significantly enhancing the detail preservation and visual effect of infrared and visible light images, especially showing excellent performance in low-light and complex scenes. However, CNN-based methods still have limitations in global feature extraction and long-distance dependency modeling, resulting in the loss of texture details and low-contrast information in the fused image. Attention-based methods have improved the global feature extraction capability to some extent, but still have shortcomings in the extraction of low-contrast information with small modal differences. Although methods combining CNN and attention mechanisms have made progress in the collaborative extraction of local and global features, further optimization is needed in fully extracting and preserving complementary features.

[0006] In recent years, with the continuous development of deep feature extraction and adaptive reconstruction mechanisms, autoencoder structures have been widely used in the fusion of infrared and visible light images. These methods extract features from infrared and visible light images through an encoder and reconstruct the fused image using a decoder, automatically learning the common features between the two to improve detail preservation and image quality. Existing methods mainly include the following three feature extraction approaches: 1) concatenating the infrared and visible light image channels and inputting them into a single encoder; 2) inputting them into separate independent encoder networks; 3) inputting them into specially designed high-frequency and low-frequency feature extraction networks. In the single encoder method, DenseFuse proposed by Li et al. and FusionDN by Xu et al. extract multi-level features through densely connected convolutional modules and combine them with adaptive weighting strategies to improve the detail preservation and visual effect of the fused image. However, these methods still have limitations in fully exploiting complementary features. In the separate encoder approach, U2Fusion by Xu et al. uses independent encoders to extract features and combines them with adaptive weighting strategies to achieve efficient fusion of multimodal features, but the utilization of complementary features is still insufficient. In addition, Prabhakar et al. achieved multimodal fusion through high-frequency and low-frequency feature extraction networks, but there is still room for further optimization in mining low-frequency global features and high-frequency local features from infrared and visible light images.

[0007] In recent years, fusion methods have gradually incorporated emerging technologies such as CNNs, Transformers, and attention mechanisms to enhance the depth and adaptability of feature extraction. Li et al. introduced attention mechanisms into image fusion, constructing a deep learning-based fusion framework that effectively generated high-quality fused images. Zhang et al., on the other hand, applied Transformers to image fusion, leveraging their powerful long-range dependency modeling capabilities to significantly improve the performance of fused images in terms of global consistency and detail preservation. These studies demonstrate that constructing fusion methods that fully exploit the complementary characteristics of multimodal images is key to improving fusion results. Although methods combining CNNs and attention mechanisms have made progress in the collaborative extraction of local and global features, further optimization is still needed to fully extract and preserve complementary features. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing technologies and provide an infrared and visible light image fusion system and method based on a multi-branch autoencoder, which can efficiently extract and fuse low-frequency global features, high-frequency local features, and complementary features of infrared and visible light images. Through a multi-branch structure design, full fusion of multimodal information is achieved.

[0009] This invention integrates complementary features extracted from infrared and visible light images, incorporates extracted low-frequency global features and high-frequency local features, and finally generates a fused image based on a Restormer module with a multi-head attention mechanism. Because this invention fully combines the advantages of convolutional neural networks (CNNs) and attention mechanisms in feature extraction, it significantly improves the quality of the fused image.

[0010] This invention provides a method for fusing infrared and visible light images, comprising the following steps:

[0011] Features of infrared and visible light images are extracted and fused to obtain infrared image fusion features and visible light image fusion features. The infrared image fusion features and visible light image fusion features are then reconstructed to obtain reconstructed infrared and visible light images.

[0012] Features of the reconstructed infrared and visible light images are extracted and fused to obtain fused features. The fused features are then reconstructed to obtain the final fused image.

[0013] Preferably, the infrared image fusion features and the visible light image fusion features are obtained through the following methods:

[0014] The unique features of infrared and visible light images are extracted separately;

[0015] Shared features are extracted from infrared and visible light images to obtain shared features;

[0016] Based on the extracted unique features of infrared and visible light images, channel splicing and fusion are performed with shared features to obtain infrared image fusion features and visible light image fusion features.

[0017] Preferably, the specific method for channel stitching and fusion based on the unique features of the extracted infrared and visible light images and the shared features is as follows:

[0018] High-frequency local features of the unique features of the extracted infrared and visible light images are extracted separately to obtain the detailed features of the infrared and visible light images;

[0019] Low-frequency global features of the unique features of the extracted infrared and visible light images are extracted separately to obtain the basic features of the infrared and visible light images.

[0020] The extracted infrared and visible light images are respectively stitched and fused by channel splicing to obtain infrared image fusion features and visible light image fusion features.

[0021] Preferably, the features of the reconstructed infrared image and the visible light image are extracted and fused to obtain fused features, which is specifically achieved through the following methods:

[0022] The unique features of the reconstructed infrared and visible light images were extracted separately.

[0023] Shared features between the reconstructed infrared and visible light images are extracted to obtain the reconstructed shared features;

[0024] The unique features extracted from the reconstructed infrared and visible light images are combined with the shared features from the reconstruction to obtain the fused features.

[0025] Preferably, the specific method for channel stitching and fusion based on the extracted unique features of the reconstructed infrared and visible light images and the shared features of the reconstruction is as follows:

[0026] High-frequency local features of the unique features of the reconstructed infrared and visible light images are extracted separately to obtain the detailed features of the reconstructed infrared and visible light images.

[0027] The low-frequency global features of the unique features of the reconstructed infrared and visible light images are extracted separately to obtain the basic features of the reconstructed infrared and visible light images.

[0028] The extracted detailed features, basic features, and shared features of the reconstructed infrared image and the visible light image are channel-seam-fused to obtain the fused features.

[0029] Preferably, the process of reconstructing the infrared image fusion features and the visible light image fusion features separately is constrained by the first-stage multimodal consistency loss function, the expression of which is:

[0030] L total I =L ir +L vi +μ1L decomp

[0031] Among them, L total I L is the first-stage multimodal consistency loss function. ir L represents the loss function between the reconstructed infrared image and the original infrared image. vi L represents the loss function for reconstructing the visible light image and comparing it to the original visible light image. decomp Let μ1 be the correlation loss function, and μ1 be the weighting factor that controls the contribution of correlation loss.

[0032] Preferably, the process of reconstructing the fused features is constrained by a second-stage multimodal consistency loss function, the expression of which is:

[0033] L totalII =α3L SSIM {I,V} +β3L MSE {I,V} +γ3L Grad {I,V} +μ2L decomp

[0034] Among them, L total II This is the second-stage multimodal consistency loss function, where α3, β3, γ3, and μ2 are all weighting factors, and L... SSIM {I,V} For the structural loss function, L MSE {I,V} Let L be the mean squared error loss function. Grad {I,V} This is the gradient loss function.

[0035] A second aspect of the present invention provides an infrared and visible light image fusion system based on a multi-branch autoencoder, implemented using the above-described fusion method, characterized in that it includes:

[0036] The first training unit is used to extract and fuse the features of infrared and visible light images respectively to obtain infrared image fusion features and visible light image fusion features. The infrared image fusion features and visible light image fusion features are then reconstructed to obtain reconstructed infrared and visible light images.

[0037] The second training unit is used to extract and fuse the features of the reconstructed infrared image and the visible light image respectively to obtain fused features, and then reconstruct the fused features to obtain the final fused image.

[0038] The first training unit and the second training unit have the same structure.

[0039] Preferably, the first training unit includes:

[0040] The encoder is used to extract features from infrared and visible light images respectively;

[0041] The fusion layer is used to fuse the features extracted from the infrared and visible light images to obtain infrared image fusion features and visible light image fusion features.

[0042] The decoder is used to reconstruct the infrared image fusion features and the visible light image fusion features respectively, so as to obtain the reconstructed infrared image and the visible light image.

[0043] Preferably, the encoder includes:

[0044] A unique feature extraction module is used to extract unique features from infrared and visible light images;

[0045] The detail feature extraction module is used to extract the high-frequency local features of the unique features of the extracted infrared and visible light images respectively, so as to obtain the detail features of the infrared and visible light images.

[0046] The basic feature extraction module is used to extract the low-frequency global features of the unique features of the extracted infrared and visible light images respectively, so as to obtain the basic features of the infrared and visible light images.

[0047] The shared feature extraction module is used to extract shared features from infrared and visible light images to obtain shared features.

[0048] Compared with the prior art, the advantages of this invention are as follows:

[0049] (1) Through the joint design of the detail feature extraction module, the basic feature extraction module and the shared feature extraction module, the present invention enables the autoencoder to have a multi-branch structure, which can simultaneously extract the low-frequency global features and high-frequency local features of infrared images and visible light images as well as their complementary features, thereby achieving full fusion of multimodal information, effectively combining the advantages of CNN and attention mechanism in feature extraction, and significantly improving the quality of the fused image.

[0050] (2) This invention combines structural similarity loss, mean square error loss, gradient loss and correlation driving loss function, and optimizes the fusion process through a two-stage training strategy to design a multimodal consistency loss function; this loss function effectively improves the performance of the fused image in preserving complementary and salient features, and avoids the need to manually design fusion strategies;

[0051] (3) Extensive experimental results on public datasets such as TNO, MSRS, and RoadScene demonstrate that MBAFuse exhibits significant advantages in both subjective visual effects and objective evaluation. Compared with the seven classic infrared and visible light image fusion methods, MBAFuse can simultaneously preserve key target details and texture information in the generated fused image, fully demonstrating the effectiveness and innovation of the method. Furthermore, extensive experimental results on public datasets such as TNO, MSRS, and RoadScene also show that MBAFuse significantly outperforms the seven classic infrared and visible light image fusion methods in most subjective visual effects and objective evaluation metrics, especially in EN, SD, SF, and VIF metrics, where the generated fused image can simultaneously preserve target details and texture information.

[0052] (4) To verify the generalization ability of MBAFuse, this invention was applied to medical image fusion and RGB color image fusion tasks. The results showed that excellent results could be obtained without fine-tuning, which proved the good adaptability and robustness of the method in various image fusion tasks. Attached Figure Description

[0053] Figure 1 This is an architecture diagram of the infrared and visible light image fusion system based on a multi-branch autoencoder proposed in Embodiment 2 of the present invention; wherein, (a) is the first training unit (corresponding to training stage I); and (b) is the second training unit (corresponding to training stage II).

[0054] Figure 2 This is a framework design diagram of the basic feature extraction module, the detailed feature extraction module, and the shared feature extraction module in Embodiment 2 of the present invention; wherein, (a) is the basic feature extraction module; (b) is the detailed feature extraction module; and (c) is the shared feature extraction module.

[0055] Figure 3 The diagram shows the comparative analysis results of the MBAFuse architecture proposed in this invention and the existing autoencoder architecture IVIF process; where (a) is the existing single encoder method; (b) is the existing independent encoder method; (c) is the existing shared encoder method; and (d) is the MBAFuse architecture proposed in this invention.

[0056] Figure 4 This image shows a comparison of the fusion results of the proposed MBAFuse method and three deep learning-based fusion methods on images.

[0057] Figure 5 The subjective comparison results of the proposed MBAFuse method with seven other state-of-the-art fusion methods on the TNO dataset are shown.

[0058] Figure 6 The results of a subjective comparison between the MBAFuse method proposed in this invention and seven other state-of-the-art fusion methods on the MSRS dataset;

[0059] Figure 7 The results show the subjective comparison between the proposed MBAFuse method and seven other state-of-the-art fusion methods on the RoadScene dataset.

[0060] Figure 8 Subjective comparison results of ablation experiments on the TNO, MSRS, and RoadScene datasets;

[0061] Figure 9 This is a subjective comparison of the generalization ability of the proposed MBAFuse method with seven other state-of-the-art fusion methods on the medical dataset MRI-CT.

[0062] Figure 10 This is the RGB color image fusion result of the MBAFuse method proposed in this invention on the RoadScene dataset. Detailed Implementation

[0063] The following will be described in conjunction with embodiments of the present invention. Figures 1 to 10 The technical solutions in the embodiments of the present invention are clearly and completely described herein. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0064] Example 1

[0065] This invention provides a method for fusing infrared and visible light images, specifically including the following steps:

[0066] Features of infrared and visible light images are extracted and fused to obtain infrared image fusion features and visible light image fusion features. The infrared image fusion features and visible light image fusion features are then reconstructed to obtain reconstructed infrared and visible light images.

[0067] Features of the reconstructed infrared and visible light images are extracted and fused to obtain fused features. The fused features are then reconstructed to obtain the final fused image.

[0068] In this embodiment of the invention, the infrared image fusion features and the visible light image fusion features are obtained through the following methods:

[0069] The unique features of infrared and visible light images are extracted separately;

[0070] Shared features are extracted from infrared and visible light images to obtain shared features;

[0071] Based on the extracted unique features of infrared and visible light images, channel splicing and fusion are performed with shared features to obtain infrared image fusion features and visible light image fusion features.

[0072] In this embodiment of the invention, the specific method for channel stitching and fusion based on the unique features of the extracted infrared image and visible light image and the shared features is as follows:

[0073] High-frequency local features of the unique features of the extracted infrared and visible light images are extracted separately to obtain the detailed features of the infrared and visible light images;

[0074] Low-frequency global features of the unique features of the extracted infrared and visible light images are extracted separately to obtain the basic features of the infrared and visible light images.

[0075] The extracted infrared and visible light images are respectively stitched and fused by channel splicing to obtain infrared image fusion features and visible light image fusion features.

[0076] In this embodiment of the invention, features of the reconstructed infrared image and visible light image are extracted and fused to obtain fused features, specifically achieved through the following method:

[0077] The unique features of the reconstructed infrared and visible light images were extracted separately.

[0078] Shared features between the reconstructed infrared and visible light images are extracted to obtain the reconstructed shared features;

[0079] The unique features extracted from the reconstructed infrared and visible light images are combined with the shared features from the reconstruction to obtain the fused features.

[0080] In this embodiment of the invention, the specific method for channel stitching and fusion based on the extracted unique features and reconstructed shared features of the reconstructed infrared and visible light images is as follows:

[0081] High-frequency local features of the unique features of the reconstructed infrared and visible light images are extracted separately to obtain the detailed features of the reconstructed infrared and visible light images.

[0082] The low-frequency global features of the unique features of the reconstructed infrared and visible light images are extracted separately to obtain the basic features of the reconstructed infrared and visible light images.

[0083] The extracted detailed features, basic features, and shared features of the reconstructed infrared image and the visible light image are channel-seam-fused to obtain the fused features.

[0084] In this embodiment of the invention, the process of reconstructing infrared image fusion features and visible light image fusion features is constrained by a first-stage multimodal consistency loss function, the expression of which is:

[0085] L total I =L ir +L vi +μ1L decomp

[0086] Among them, L total I L is the first-stage multimodal consistency loss function. ir L represents the loss function between the reconstructed infrared image and the original infrared image. vi L represents the loss function for reconstructing the visible light image and comparing it to the original visible light image. decompLet μ1 be the correlation loss function, and μ1 be the weighting factor that controls the contribution of correlation loss.

[0087] In this embodiment of the invention, the process of reconstructing the fused features is constrained by a second-stage multimodal consistency loss function, the expression of which is:

[0088] L total II =α3L SSIM {I,V} +β3L MSE {I,V} +γ3L Grad {I,V} +μ2L decomp

[0089] Among them, L total II This is the second-stage multimodal consistency loss function, where α3, β3, γ3, and μ2 are all weighting factors, and L... SSIM {I,V} For the structural loss function, L MSE {I,V} Let L be the mean squared error loss function. Grad {I,V} This is the gradient loss function.

[0090] Example 2

[0091] This invention provides an infrared and visible light image fusion system based on a multi-branch autoencoder, implemented using the fusion method of Embodiment 1 above, including:

[0092] The first training unit is used to extract and fuse the features of infrared and visible light images respectively to obtain infrared image fusion features and visible light image fusion features. The infrared image fusion features and visible light image fusion features are then reconstructed to obtain reconstructed infrared and visible light images.

[0093] The second training unit is used to extract and fuse the features of the reconstructed infrared image and the visible light image respectively to obtain fused features, and then reconstruct the fused features to obtain the final fused image.

[0094] The first training unit and the second training unit have the same structure.

[0095] In this embodiment of the invention, the first training unit includes:

[0096] The encoder is used to extract features from infrared and visible light images respectively;

[0097] The fusion layer is used to fuse the features extracted from the infrared and visible light images to obtain infrared image fusion features and visible light image fusion features.

[0098] The decoder is used to reconstruct the infrared image fusion features and the visible light image fusion features respectively, so as to obtain the reconstructed infrared image and the visible light image.

[0099] In this embodiment of the invention, the encoder includes:

[0100] A unique feature extraction module is used to extract unique features from infrared and visible light images;

[0101] The detail feature extraction module is used to extract the high-frequency local features of the unique features of the extracted infrared and visible light images respectively, so as to obtain the detail features of the infrared and visible light images.

[0102] The basic feature extraction module is used to extract the low-frequency global features of the unique features of the extracted infrared and visible light images respectively, so as to obtain the basic features of the infrared and visible light images.

[0103] The shared feature extraction module is used to extract shared features from infrared and visible light images to obtain shared features.

[0104] This invention proposes an infrared and visible light image fusion system (MBAFuse) based on a multi-branch autoencoder. The system consists of an encoder, a fusion layer, and a decoder. The encoder includes a shared feature extraction module (SFEM) and independent modules for extracting detail and basic features from infrared and visible light images, respectively. The SFEM is a complementary feature extraction network constructed based on the existing DenseBlock module combined with an inter-layer matching mechanism, such as... Figure 2 As shown in (c) above, the SFEM is designed to efficiently extract complementary features from infrared and visible light images. The independent modules consist of a Basic Feature Extraction Module (BFEM) and a Detail Feature Extraction Module (DFEM).

[0105] like Figure 2 As shown in (b), DFEM is constructed based on existing invertible neural network (INN) modules. DFEM is designed to extract finer texture and edge information. In this invention, DFEM is used to extract high-frequency local features from infrared and visible light images. The specific working principle is as follows:

[0106] First, the input features Channel segmentation is performed, and the upper branch extracts the [1:C / 2] channel features, while the lower branch extracts the [C / 2+1:C] channel features.

[0107] Secondly, the upper and lower branches undergo feature extraction and interaction through a convolutional neural network (CNN) and an inter-block multi-layer interaction (IBB) module, respectively, where [1:C / 2] represents the interaction operations at different stages. After multiple interactions and fusions, the two branches achieve information fusion through feature concatenation and element-wise addition, ultimately obtaining detailed feature outputs.

[0108] like Figure 2 As shown in (a), the BFEM is designed specifically for visual tasks, capable of capturing global contextual information and significantly improving the extraction of basic features from infrared and visible light images. In this invention, BFEM is used to extract low-frequency global features from infrared and visible light images. Its structure is built based on the existing Outlook Attention module, and the specific working process is as follows:

[0109] First, input features The shape is transformed from [B,C,H,W] to [B,H,W,C]; subsequently, a local region feature [B,H,W,C,k] is generated through a Linear+Unfold operation. 2 The other path generates an attention weight vector [B,H,W,k] via Linear. 2 ×k 2 The normalized Attention Map is obtained through Softmax; then, the local region features are weighted and fused using the Attention Map, and the features are restored to the spatial dimension [B,H,W,C] through FLD+Linear operation to obtain low-frequency global features.

[0110] The decoder includes the existing multi-head attention-based Restormer module, which is used for high-quality restoration and reconstruction of high-resolution images to ensure the sharpness and detail of the fused image.

[0111] The structure and workflow of the DenseBlock module in this embodiment of the invention can be found in the reference "Li H, Wu XJ. DenseFuse: A fusion approach to infrared and visible images[J].IEEE Transactions on Image Processing,2018,28(5):2614-2623.";

[0112] The structure and workflow of the Outlook Attention module in this embodiment of the invention can be found in the reference "Yuan L, Hou Q, Jiang Z, et al. Volo: Vision outlooker for visual recognition[J]. IEEE transactions on pattern analysis and machine intelligence, 2022, 45(5): 6575-6586.";

[0113] The structure and workflow of the reversible neural network (INN) module in this embodiment of the invention can be found in the reference "Dinh L, Sohl-Dickstein J, Bengio S. Density estimation using real nvp[J].arXivpreprint arXiv:1605.08803,2016".

[0114] I. The following is a detailed introduction to the MBAFuse proposed in this invention.

[0115] 1. Encoder

[0116] The encoder consists of four components: a specific feature extraction module, a detail feature extraction module (DFEM) based on an INN module, a basic feature extraction module (BFEM) based on an Outlook Attention module, and a shared feature extraction module (SFEM) based on a DenseBlock module. The design of these three modules (DFEM, BFEM, and SFEM) enables the encoder to have a multi-branch structure, facilitating the extraction of multimodal features. For ease of description, this invention defines several symbols:

[0117] Infrared and visible light images are represented as I∈R, respectively. H×W and V∈R H×W The detail feature extraction module, the basic feature extraction module, and the shared feature extraction module are denoted as D(·), B(·), and S(·), respectively.

[0118] In the dual-branch architecture of the encoder, a unique feature extraction module was designed to extract specific features from infrared and visible light images. This module employs convolution, batch normalization (BN), and ReLU activation function (Re) operations to support early visual processing and improve optimization stability. The formula for the initial extraction of unique features from infrared and visible light images is expressed as follows:

[0119]

[0120] in, Conv1 represents the features of infrared and visible light images after convolution filtering. 1,64 (·) represents a 1×1 convolution operation with 1 input and 64 output channels, Conv3 64,64 (·) represents a 3×3 convolution operation with 64 input and 64 output channels, and BN(·) and Re(·) represent batch normalization and ReLU activation layers, respectively. To fully extract detailed features from infrared and visible light images, this invention designs and uses a DFEM module. Due to its bidirectional feature extraction and lossless reconstruction characteristics, this module can effectively extract detailed features from different source images in image processing. Its formula is as follows:

[0121] Φ I D =D(Φ I CNN ),Φ V D =D{Φ V CNN} (2)

[0122] Where, Φ I D and Φ V D These represent the detailed features of infrared and visible light images, respectively. Furthermore, to extract the fundamental features of infrared and visible light images, this invention uses a BFEM constructed from Outlook Attention modules. Outlook Attention is an attention mechanism specifically designed for visual tasks, capable of effectively capturing contextual information, thereby better extracting the fundamental features of different source images. The BFEM flowchart is as follows:

[0123] Φ I B =B(Φ I CNN ),Φ V B =B(Φ V CNN (3)

[0124] Where, Φ I B and Φ V B These represent the basic features of infrared and visible light images, respectively. To obtain shared features between infrared and visible light images, this invention designs and employs a DenseBlock module (SFEM), specifically designed for image data with two input channels. This module is used to extract shared features from infrared and visible light images, and the process formula is as follows:

[0125] Φ {I,V} S=S({I,V}) (4)

[0126] Where, Φ {I,V} S These represent complementary features of infrared and visible light images, also referred to as shared features in this invention.

[0127] 2. Fusion Layer

[0128] like Figure 1 As shown, the MBAFuse provided in this embodiment of the invention employs a two-stage training strategy. In this strategy, the first stage primarily focuses on the reconstruction of the original image, emphasizing the extraction of detailed features to enhance its detail performance under the correlation loss function. Specifically, the shared features extracted from infrared and visible light images via SFEM are combined with the detailed and basic features extracted via DFEM and BFEM, respectively, to form the channel-seam fusion layer of the first stage. Figure 2 As shown in (a) above. The second stage further concatenates the shared features of the infrared and visible light images with their respective detail and fundamental features to construct a second-stage channel-stitching fusion layer for reconstructing the fused image, as shown in (a). Figure 2 As shown in (b) above. The channel splicing and fusion layer of these two stages can be represented by the following formula:

[0129] Stage I:

[0130] Stage II:F {I,V} C(Φ I,V S ,Φ I B ,Φ V B ,Φ I D ,Φ V D (5)

[0131] Among them, F I F V and F {I,V} These represent the fusion layers of the first-stage infrared image, the first-stage visible light image, and the second-stage reconstructed fusion image, respectively, with C(.) being the correlation operator.

[0132] 3. Decoder

[0133] In the decoder, this invention designs and uses a Restormer module based on multi-head attention. Features obtained from the channel splicing and fusion layer are input into the decoder DC(.) for image reconstruction. In the first stage, the obtained features F... I and F VInput into DC(.) to obtain the reconstructed image. and In the second stage, since the first stage has effectively obtained the shared features and individual details and basic features of infrared and visible light images, this invention directly uses the weight parameters obtained from the first stage training for further training (using only the encoder network parameters). The feature F obtained by passing the shared features and individual details and basic features of infrared and visible light images through a channel stitching and fusion layer is... {I,V} The input is fed into DC(.) to reconstruct the final fused image F. The decoder stage can be represented by the following formula:

[0134] Stage I:

[0135] Stage II: F = DC(F) {I,V} (6)4. Design of two-stage loss function

[0136] The proposed MBAFuse employs a two-stage training strategy. In the first stage, to better reconstruct the original image, this invention specifically considers the differences in imaging mechanisms between infrared and visible light images. These two modalities not only have significant differences but also contain some shared features. Therefore, this invention proposes a multimodal consistency loss function to combine the shared features between images with their respective unique details and fundamental features, thereby achieving more accurate original image reconstruction. The expression for the multimodal consistency loss function in the first stage is as follows:

[0137] L total I =L ir +L vi +μ1L decomp (7)

[0138] Where, L ir L represents the loss function between the reconstructed infrared image and the original infrared image. vi This represents the loss function used to reconstruct the visible light image compared to the original visible light image. Furthermore, to better achieve mode decomposition of infrared and visible light images in terms of detail and fundamental features, this invention introduces a correlation loss function L. decomp This enhances the correlation of detailed features between infrared and visible light images while reducing the correlation of basic features. Furthermore, μ1 represents a weighting factor controlling the contribution of correlation loss; in the MBAFuse of this invention, μ1 is set to 2. Its loss function L... ir L vi and L decomp The formulas are defined as follows:

[0139]

[0140] The weighting factors α1, β1, γ1, α2, β2, and γ2 control the contributions of structural loss, mean squared error loss, and gradient loss during infrared and visible light image reconstruction, respectively, and are set to 8, 2, 20, 8, 2, and 20. Taking the reconstructed infrared image as an example... The structural loss is represented by the structural similarity index SSIM(.). Where H and W represent the height and width of the image, respectively, and ||||1 represents the L1 norm operator. Gradient loss is used to evaluate the gradient difference between the reconstructed image and the infrared image. ▽ is the gradient operator. CC(.) is the correlation operator. To avoid a denominator of 0 leading to a NaN total loss function, and to make the training process focus more on the detailed features between modality decomposition processes and reduce the influence of basic features, σ is set to 1.01 in the correlation loss function calculation.

[0141] After completing the first stage of training, this invention can reconstruct the original image relatively accurately. Based on this, a second stage of training is directly performed to achieve the reconstruction of the fused image. To better suit the generation of fused images, a general multimodal consistency loss function for reconstructing fused images from infrared and visible light images is specifically designed. This loss function integrates structural similarity loss, mean squared error loss, gradient loss, and correlation decomposition loss. The multimodal consistency loss function for the second stage is defined as follows:

[0142] L total II =α3L SSIM {I,V} +β3L MSE {I,V} +γ3L Grad {I,V} +μ2L decomp (11)

[0143] Among them, L total II For the second-stage multimodal consistency loss function, α3, β3, γ3, and μ2 are weighting factors used to control the weights of structural loss, mean squared error loss, gradient loss, and correlation decomposition loss for infrared images, visible light images, and fused images during training. These weights are set to 8, 2, 20, and 2, respectively. SSIM {I,V} For the structural loss function, L MSE {I,V} Let L be the mean squared error loss function. Grad {I,V} Let L be the gradient loss function. Each loss function in the second stage differs slightly from the loss function in the first stage (only L...). decomp(The losses are the same). The structural loss, mean squared error loss, and gradient loss functions are defined as follows:

[0144] L SSIM {I,V} =1-SSIM(F,max{I,V}) (12)

[0145]

[0146] Where H and W represent the height and width of the image, respectively, and max{,} represents the maximum selection. ∠ denotes the sign for the F-norm, and ▽ denotes the sign for the gradient.

[0147] 5. Structural Comparison and Analysis of the MBAFuse Proposed in This Invention and Mainstream Infrared and Visible Light Image Fusion Methods

[0148] Currently, the architecture of mainstream infrared and visible light image fusion methods mainly includes: Figure 3 The three types of patterns are shown in (a)-(c). Figure 3 The architecture in (a) can effectively extract complementary features by stitching the infrared and visible light image channels together and inputting them into the same encoder and decoder network, but it is insufficient in capturing the global and local features unique to each modality. Figure 3 In (b), the features of the two modalities are extracted using independent encoders, and then the image is reconstructed through a fusion strategy. Although there is an improvement in the extraction of features of each modality, the fusion of complementary information is relatively limited. Figure 3 (c) The introduction of a shared encoder in the design alleviates the shortcomings of the two architectures mentioned above to some extent, but it still fails to fully extract the complementary features between the two modalities. Compared with the architectures mentioned above, such as Figure 3 As shown in (d), the MBAFuse proposed in this invention integrates a DenseBlock module to extract complementary features and introduces an Invertible Neural Network (INN) module and an Outlook Attention module to extract low-frequency global features and high-frequency local features, respectively. This method fully combines the advantages of Convolutional Neural Networks (CNNs) and attention mechanisms in feature extraction, significantly improving the quality of the fused image.

[0149] To verify the superiority of the MBAFuse proposed in this invention, Figure 4 A comparative example of infrared and visible light images is presented, comparing the present invention's MBAFuse with three deep learning-based fusion methods (DIDFuse, U2Fusion, and DATFuse). To more intuitively demonstrate the advantages of MBAFuse in fusing the unique global, local, and complementary features of infrared and visible light images, Figure 4Two areas were selected for magnified analysis: the infrared image highlighted the thermal radiation characteristics of the target person, while the visible light image highlighted texture details. Figure 4 It can be clearly observed that although DIDFuse, U2Fusion, and DATFuse all exhibit good fusion results, U2Fusion and DATFuse fail to fully capture the thermal radiation features in the infrared image on the left side of the magnified area. While DIDFuse is close to MBAFuse in extracting thermal radiation information, its fusion performance in the magnified area of ​​the lower right corner of the visible light image is significantly insufficient in terms of texture details. Overall, MBAFuse performs better in preserving feature complementarity and exhibits a superior fusion effect in terms of human visual perception.

[0150] II. Simulation Experiment

[0151] This section of the invention first describes the training and testing datasets and implementation details, then introduces the evaluation metrics, and compares and analyzes seven advanced methods from recent years at both subjective and objective levels. The effectiveness of the proposed MBAFuse is verified through ablation experiments, and it is further applied to medical image fusion and RGB color image fusion tasks to comprehensively evaluate its generalization performance.

[0152] 1. Dataset and Implementation Details

[0153] Regarding dataset selection, this invention utilizes three mainstream publicly available infrared and visible light image fusion datasets: TNO, MSRS, and RoadScene. In the MBAFuse study, 1444 pairs of infrared and visible light images were selected from the MSRS dataset as the training set. To obtain sufficient data and utilize GPU memory resources, the images were cropped with a stride of 200 and a size of 128×128, resulting in 8649 pairs of infrared and visible light images as the training set (TNO and RoadScene datasets were not used for training). During the testing phase, 25 pairs of images were selected from the TNO dataset, 361 pairs from the MSRS dataset, and 50 pairs from the RoadScene dataset for subjective and objective comparison experiments with seven other state-of-the-art methods.

[0154] The training data for the proposed MBAFuse dataset contains 8649 pairs of infrared and visible light images, with the visible light images in the MSRS dataset in RGB format. This invention converts the RGB images into YUV channels, obtaining three components: Y, U, and V. Y V U and V VThe Y-channel image was used as the visible light image for training. Both infrared and visible light images were normalized to the [0,1] interval. During training, the learning rate was set to 0.0001, and the Adam optimizer was used to optimize the deep model. The training epochs were set to 15, with the first training stage lasting 10 epochs and the second training stage lasting 5 epochs. The weights of the loss function, α1, β1, γ1, α2, β2, γ2, μ1, α3, β3, γ3, and μ2, were set to 8, 2, 20, 8, 2, 20, 2, 8, 20, 2. This experiment was based on the PyTorch framework and ran on an NVIDIA V100 GPU.

[0155] 2. Comparison methods and objective evaluation indicators

[0156] To verify the superiority of MBAFuse in the field of infrared and visible light image fusion, this invention provides a comprehensive comparison with seven state-of-the-art deep learning-based infrared and visible light image fusion algorithms. These methods include DenseFuse, NestFuse, CSFusion, U2Fusion, DIDFuse, SDNet, and DATFuse. The source code for all these methods is publicly available or provided by their original authors.

[0157] Regarding objective evaluation metrics, this invention employs six common infrared and visible light image fusion evaluation standards, including entropy (EN), standard deviation (SD), spatial frequency (SF), mutual information (MI), visual information fidelity (VIF), and Qabf. These metrics are important tools for measuring image fusion effectiveness, with higher values ​​generally indicating better fusion results and better reflecting the detail richness, information retention, and visual consistency of the fused image. Furthermore, these metrics help to comprehensively evaluate the fusion performance of different methods from multiple dimensions, ensuring that the obtained fusion results have high effectiveness and robustness in practical applications.

[0158] 3. Comparison with state-of-the-art methods

[0159] In the TNO dataset test, this invention selected 5 typical images from the fusion results of 25 infrared and visible light images, and conducted a subjective comparative analysis of MBAFuse with 7 other methods, such as... Figure 5 As shown. To visually demonstrate the fusion effect of each method, key areas of each image have been magnified. Through... Figure 5Subjective comparisons show that while other methods achieved fusion to some extent, they still have shortcomings. For example, in the magnified right-hand image of the second column, NestFuse, CSFusion, U2Fusion, DIDFuse, and SDNet failed to effectively preserve the details of tree branches in the visible light image. While DenseFuse and DATFuse partially preserved details, they were still inferior to MBAFuse in terms of detail representation and contrast. Similarly, in the magnified left-hand image of the second column, the results of DenseFuse, NestFuse, CSFusion, U2Fusion, SDNet, and DATFuse were darker, especially in terms of contrast, which was inferior to MBAFuse. Furthermore, in the magnified left-hand image of the fifth column, the fusion of tree branch details by NestFuse, CSFusion, DIDFuse, and SDNet was not clear enough, while MBAFuse preserved these details better. In the magnified left-hand areas of the first and third columns, MBAFuse also significantly outperformed other methods in terms of saliency feature fusion. Overall, MBAFuse performed exceptionally well in subjective evaluations, particularly in preserving thermal radiation information in infrared images and capturing details in visible light images, significantly outperforming the other seven methods.

[0160] In terms of objective metrics, the performance advantage of MBAFuse was further verified. This invention calculated six objective evaluation metrics for each image and took the average for comprehensive comparison. The results are shown in Table 1 below, where bold and underlined values ​​represent the best and second-best values, respectively.

[0161] Table 1. Objective comparison of various methods on the TNO dataset.

[0162]

[0163] The results in Table 1 demonstrate that MBAFuse achieves the best performance in entropy (EN), standard deviation (SD), spatial frequency (SF), visual information fidelity (VIF), and Qabf metrics, reflecting its efficient preservation of image details and global information. Although it does not reach the best value in mutual information (MI) metrics, its performance is still superior to NestFuse, CSFusion, U2Fusion, DIDFuse, and SDNet, and only slightly different from DenseFuse and DATFuse. Overall, the MBAFuse method performs excellently in both subjective and objective evaluations, fully demonstrating its superior performance in infrared and visible light image fusion tasks.

[0164] In the MSRS dataset test, this invention selected 5 typical images from the fusion results of 361 infrared and visible light images, and conducted a subjective comparative analysis of MBAFuse with 7 other methods. To more intuitively demonstrate the fusion effect of each method, key areas of each image were locally magnified. The results are as follows: Figure 6 As shown. (Through) Figure 6 As can be seen, in the magnified area on the right side of the first column, the NestFuse, CSFusion, U2Fusion, DIDFuse, and SDNet methods failed to effectively preserve details in the visible light image, resulting in the loss of some key scene information (such as zebra crossing details). Although the DenseFuse and DATFuse methods improved in detail representation, they were still inferior to the MBAFuse method. In the magnified area on the right side of the second column, the DenseFuse, NestFuse, CSFusion, U2Fusion, DIDFuse, and SDNet methods were relatively blurry in preserving the sign font, while the DATFuse and MBAFuse methods were able to clearly preserve the font information. In the magnified area on the right side of the fifth column, the MBAFuse method significantly outperformed the CSFusion, U2Fusion, and SDNet methods in terms of license plate detail and contrast. Overall, MBAFuse showed a significant advantage over other methods in preserving the thermal radiation information and detail feature fusion of infrared images.

[0165] Regarding objective indicators, this invention adopts the same calculation method as the TNO dataset, and the results are shown in Table 2 below. For the six evaluation indicators, the best value is marked in bold, and the second best value is indicated by an underline.

[0166] Table 2. Objective comparison of various methods on the MSRS dataset.

[0167]

[0168] The results in Table 2 show that the MBAFuse method achieves the best performance in all five metrics: entropy (EN), standard deviation (SD), spatial frequency (SF), visual information fidelity (VIF), and Qabf, fully demonstrating its superiority in detail preservation and contrast enhancement. Although it does not surpass the best value of the DATFuse method in mutual information (MI), MBAFuse still achieves the second-best result. Overall, the MBAFuse method performs excellently in all six objective evaluation metrics, further validating its outstanding performance on the MSRS dataset.

[0169] In the RoadScene dataset test, this invention selected five typical images from the fusion results of 50 pairs of infrared and visible light images, and conducted a subjective comparative analysis of MBAFuse with seven other methods. The subjective evaluation method was consistent with that of the TNO and MSRS datasets. To more intuitively demonstrate the fusion effect of each method, this invention performed local magnification processing on key areas of the images. The results are as follows: Figure 7 As shown. (Through) Figure 7 As can be seen, in the magnified area on the right side of the first column, the NestFuse, U2Fusion, and SDNet methods perform poorly in fusing tree branches and leaves, with insufficient preservation of thermal radiation features and detail information, while the MBAFuse method performs better in these aspects. In the magnified area on the right side of the second column, the MBAFuse method more accurately preserves important scene attributes and generates clear environmental information compared to the DenseFuse, NestFuse, U2Fusion, DIDFuse, SDNet, and DATFuse methods, while other methods fail to effectively restore these details. In the magnified area on the left side of the fourth column, the DenseFuse, NestFuse, and U2Fusion methods perform poorly in preserving infrared features, showing a significant gap compared to the MBAFuse method's fusing effect. Overall, the MBAFuse method outperforms other methods in preserving infrared thermal radiation information and fusing detail features, demonstrating a superior fusing effect.

[0170] In the comparison of objective indicators, this invention uses the same calculation method as the TNO and MSRS datasets, and the results are shown in Table 3 below. For the six evaluation indicators, the best value is marked in bold, and the second best value is indicated by an underline.

[0171] Table 3 shows an objective comparison of the methods on the RoadScene dataset.

[0172]

[0173]

[0174] As shown in Table 3, although the MBAFuse method did not achieve the best results on the MI and Qabf metrics, it reached the second-best value on the MI metric and its performance on the Qabf metric was very close to the best. Furthermore, the MBAFuse method achieved the best performance on the EN, SD, SF, and VIF metrics. These results further validate the superior performance of the MBAFuse method on the RoadScene dataset, fully demonstrating its good generalization ability and robustness.

[0175] 4. Ablation test

[0176] To further verify the design rationality of the proposed MBAFuse method, further ablation experiments were conducted on its loss function. In Experiments 1 and 2, the gradient loss function weights γ1, γ2, and γ3 of the first and second stages were modified from 20 to 5 and 35, respectively, to reduce or increase their role in preserving image edges and detail information, thereby verifying the optimal values ​​of the set gradient loss function weights. In Experiments 3 and 4, the correlation loss function weights μ1 and μ2 of the first and second stages were adjusted from 2 to 1 and 4, respectively, to reduce or enhance the correlation and dependency of detail features. In Experiment 5, the first stage used to reconstruct the original image was removed, and only the second stage used to generate the fused image was used for training. For each ablation experiment, the fusion effect on the TNO, MSRS, and RoadScene datasets was quantitatively evaluated through subjective evaluation and multiple objective metrics (including EN, SD, SF, MI, VIF, and Qabf). The subjective evaluation results are as follows: Figure 8 As shown, the experimental results for the objective indicators are listed in Tables 4, 5 and 6.

[0177] In the subjective comparison of ablation experiments, this invention has performed local magnification processing on the results for easier observation and comparison. For example... Figure 8 As shown, columns one, two, and three present the fusion results on the TNO, MSRS, and RoadScene datasets, respectively. In the right-hand magnified areas of the third and fourth images in column one, MBAFuse significantly outperforms Experiments 1 and 2 in terms of detail and edge preservation. Furthermore, in the left-hand magnified areas of Experiments 3 and 4, MBAFuse demonstrates more prominent and optimized results in contrast and complementary feature fusion, regardless of whether the weights of the correlation loss function are increased or decreased. In Experiment 5, despite using only the second stage of training, the fusion results are still excellent, especially in the left-hand magnified area, where MBAFuse shows more prominent human targets and clearer scene representation. In column two, compared to the magnified areas of human heads in Experiments 2, 3, 4, and 5, MBAFuse's fusion results better preserve detailed contours and edge details; compared to Experiment 5, MBAFuse performs better in preserving contrast information. Finally, on the RoadScene dataset, compared to results obtained by modifying the loss weights, MBAFuse outperforms other experiments in both target representation and scene clarity.

[0178] In terms of objective comparison, the ablation experiment results on the TNO dataset are shown in Table 4. Table 4 shows that, compared with other methods, the proposed MBAFuse method, although not reaching optimal values ​​for SD and MI, is close to the optimal values, and achieves optimal values ​​for EN, SD, VIF, and Qabf. The ablation experiment results on the MSRS dataset are shown in Table 5. Table 5 results show that although the MBAFuse method did not reach optimal values ​​for EN, SD, and SF, it exhibited the best performance for MI, VIF, and Qabf. The ablation experiment results on the RoadScene dataset are shown in Table 6. Table 6 results show that MBAFuse achieved optimal values ​​for EN, SD, and SF. Based on the above objective comparisons, it can be concluded that the hyperparameters designed for the MBAFuse method almost reached optimal values ​​throughout the ablation experiments, further verifying the rationality and effectiveness of the MBAFuse method.

[0179] Table 4 shows the ablation experimental results on the TNO dataset.

[0180]

[0181] Table 5 Ablation experimental results on the MSRS dataset.

[0182]

[0183] Table 6 Ablation experimental results on the RoadScene dataset

[0184]

[0185] 5. Method scalability

[0186] To further investigate the generalization ability of the proposed MBAFuse method, 21 images were directly selected from the medical image MRI-CT dataset for fusion without fine-tuning. Subjective and objective comparisons were then performed with seven other methods: DenseFuse, NestFuse, CSFusion, U2Fusion, DIDFuse, SDNet, and DATFuse. The subjective comparison results are as follows: Figure 9 As shown, the objective comparison results are listed in Table 7.

[0187] In subjective comparison, from Figure 9The magnified area on the left side of the first column clearly shows that the fused images obtained by the DenseFuse, NestFuse, CSFusion, U2Fusion, DIDFuse, and SDNet methods have significant limitations in capturing texture details of visible light images. The fused images produced by these methods are relatively blurry and lack sufficient preservation of detail features, similar to the results of the DIDFuse method. In contrast, in Figure 9 In the magnified area on the right side of the third column, the MBAFuse method demonstrates a significantly better fusion of thermal radiation features from infrared images and texture details from visible light than the other seven methods. Figure 9 All the magnified local areas shown clearly demonstrate that, compared to the other seven methods, the MBAFuse method has a significant advantage in appropriately preserving complementary information from the source image.

[0188] Table 7 Comparison of the generalization ability of the MBAFuse method on the MRI-CT medical image dataset.

[0189]

[0190] In the objective comparison, as shown in Table 7, based on the six indicators EN, SD, SF, MI, VIF, and Qabf, the MBAFuse method achieved optimal values ​​for EN, SD, VIF, and Qabf on the medical image MRI-CT dataset, while the differences from the optimal values ​​for SF and MI were relatively small. These results further verify the good generalization ability of the proposed MBAFuse method on the medical image MRI-CT dataset.

[0191] To further verify the generalization performance of the MBAFuse method, this invention uses visible light images used in the training phase. The RGB channels of these images have been converted to the YCbCr color space, and the Y channel is used as the visible light image for training. Therefore, without fine-tuning, this invention directly uses the fused image and converts the Cb and Cr channels of the original infrared image to RGB color images to generate the final RGB color image. To further verify the generalization performance of the proposed MBAFuse method, this invention selects five fusion results from the RoadScene dataset, such as... Figure 10 As shown. (Through) Figure 10 As can be seen, the MBAFuse method exhibits good generalization performance in fusing scenes from different RoadScene datasets. The fused image combines the thermal target information of infrared images with the texture and color details of visible light images, resulting in realistic and clear scene reproduction, effectively overcoming the limitations of a single modality. Color reproduction is natural and distortion-free, providing a high-quality image foundation for subsequent applications and fully validating the effectiveness and stability of the proposed method.

[0192] In summary, this invention, through the joint design of a detail feature extraction module, a basic feature extraction module, and a shared feature extraction module, enables the autoencoder to have a multi-branch structure, which can simultaneously extract low-frequency global features and high-frequency local features of infrared and visible light images, as well as their complementary features, thereby achieving full fusion of multimodal information. It effectively combines the advantages of CNN and attention mechanisms in feature extraction, significantly improving the quality of the fused image.

[0193] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for fusing infrared and visible light images, characterized in that, Includes the following steps: Features of infrared and visible light images are extracted and fused to obtain infrared image fusion features and visible light image fusion features. The infrared image fusion features and visible light image fusion features are then reconstructed to obtain reconstructed infrared and visible light images. Features of the reconstructed infrared and visible light images are extracted and fused to obtain fused features. The fused features are then reconstructed to obtain the final fused image.

2. The infrared and visible light image fusion method according to claim 1, characterized in that, Infrared image fusion features and visible light image fusion features were obtained through the following methods: The unique features of infrared and visible light images are extracted separately; Shared features are extracted from infrared and visible light images to obtain shared features; Based on the extracted unique features of infrared and visible light images, channel splicing and fusion are performed with shared features to obtain infrared image fusion features and visible light image fusion features.

3. The infrared and visible light image fusion method according to claim 2, characterized in that, The specific method for channel stitching and fusion based on the unique features extracted from infrared and visible light images and shared features is as follows: High-frequency local features of the unique features of the extracted infrared and visible light images are extracted separately to obtain the detailed features of the infrared and visible light images; Low-frequency global features of the unique features of the extracted infrared and visible light images are extracted separately to obtain the basic features of the infrared and visible light images. The extracted infrared and visible light images are spliced ​​and fused through channels to obtain infrared image fusion features and visible light image fusion features.

4. The infrared and visible light image fusion method according to claim 1, characterized in that, The features of the reconstructed infrared and visible light images are extracted and fused to obtain the fused features, which is achieved through the following method: The unique features of the reconstructed infrared and visible light images were extracted separately; Shared features between the reconstructed infrared and visible light images are extracted to obtain the reconstructed shared features; The unique features extracted from the reconstructed infrared and visible light images are combined with the shared features from the reconstruction to obtain the fused features.

5. The infrared and visible light image fusion method according to claim 4, characterized in that, The specific method for channel stitching and fusion based on the unique features extracted from the reconstructed infrared and visible light images and the shared features of the reconstruction is as follows: High-frequency local features of the unique features of the reconstructed infrared and visible light images are extracted separately to obtain the detailed features of the reconstructed infrared and visible light images. The low-frequency global features of the unique features of the reconstructed infrared and visible light images are extracted separately to obtain the basic features of the reconstructed infrared and visible light images. The extracted detailed features, basic features, and shared features of the reconstructed infrared image and the visible light image are channel-seam-fused to obtain the fused features.

6. The infrared and visible light image fusion method according to claim 1, characterized in that, The process of reconstructing infrared image fusion features and visible light image fusion features separately is constrained by the first-stage multimodal consistency loss function, the expression of which is: L total I =L ir +L vi +μ1L decomp Among them, L total I L is the first-stage multimodal consistency loss function. ir L represents the loss function between the reconstructed infrared image and the original infrared image. vi L represents the loss function for reconstructing the visible light image and comparing it to the original visible light image. decomp Let μ1 be the correlation loss function, and μ1 be the weighting factor that controls the contribution of correlation loss.

7. The infrared and visible light image fusion method according to claim 1, characterized in that, The process of reconstructing the fused features is constrained by the second-stage multimodal consistency loss function, the expression of which is: L total II =α3L SSIM {I,V} +β3L MSE {I,V} +γ3L Grad {I,V} +μ2L decomp Among them, L total II This is the second-stage multimodal consistency loss function, where α3, β3, γ3, and μ2 are all weighting factors, and L... SSIM {I,V} For the structural loss function, L MSE {I,V} Let L be the mean squared error loss function. Grad {I,V} This is the gradient loss function.

8. An infrared and visible light image fusion system based on a multi-branch autoencoder, characterized in that, The fusion method described in any one of claims 1-7 is characterized by comprising: The first training unit is used to extract and fuse the features of infrared and visible light images respectively to obtain infrared image fusion features and visible light image fusion features. The infrared image fusion features and visible light image fusion features are then reconstructed to obtain reconstructed infrared and visible light images. The second training unit is used to extract and fuse the features of the reconstructed infrared image and the visible light image respectively to obtain fused features, and then reconstruct the fused features to obtain the final fused image. The first training unit and the second training unit have the same structure.

9. The system according to claim 8, characterized in that, The first training unit includes: The encoder is used to extract features from infrared and visible light images respectively; The fusion layer is used to fuse the features extracted from the infrared and visible light images to obtain infrared image fusion features and visible light image fusion features. The decoder is used to reconstruct the infrared image fusion features and the visible light image fusion features respectively, so as to obtain the reconstructed infrared image and the visible light image.

10. The system according to claim 9, characterized in that the encoder include: A unique feature extraction module is used to extract unique features from infrared and visible light images; The detail feature extraction module is used to extract the high-frequency local features of the unique features of the extracted infrared and visible light images respectively, so as to obtain the detail features of the infrared and visible light images. The basic feature extraction module is used to extract the low-frequency global features of the unique features of the extracted infrared and visible light images respectively, so as to obtain the basic features of the infrared and visible light images. The shared feature extraction module is used to extract shared features from infrared and visible light images to obtain shared features.

Images