Infrared and visible light image fusion network system and application method thereof
Patent Information
- Application Number
- CN202411225766.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2026-10-30
- Estimated Expiration
- 2044-09-03
AI Technical Summary
CNN摧毁远程图像的相关性和固有的小感受野不能有效地提取全局信息，进而影响图像融合的结果
[0071]本发明优先通过构建特征提取块，在特征提取块中设置结合多尺度可分离卷积模块和可变形卷积，从而提高对融合后的特征图中环境信息以及突出信息的提取；
Smart Images
Figure CN119130826B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image fusion technology, specifically to an infrared and visible light image fusion network system and its application method. Background Technology
[0002] Image fusion technology combines information from multiple image sources to generate a new image, demonstrating its unique value across various fields. Image fusion tasks generally include infrared and visible light image fusion, medical image fusion, multi-exposure image fusion, and multi-focus image fusion. Multimodal image fusion typically involves feature extraction, feature fusion, and feature reconstruction. The resulting fused image not only contains more information and richer details but also exhibits greater robustness, providing a more comprehensive scene description and laying a solid foundation for subsequent image processing and analysis tasks. This technology is particularly suitable for advanced vision systems such as military reconnaissance, target recognition, target tracking, and semantic segmentation. It effectively reduces data redundancy and significantly improves the efficiency and accuracy of data processing. Therefore, image fusion plays a crucial role in the preprocessing stage of modern vision tasks, providing strong support for various applications and driving the development of related technologies and the optimization of decision-making.
[0003] Infrared and visible light image fusion is an important topic in the field of image processing, with wide applications in military and security. Due to the practicality of infrared and visible light images, existing image fusion methods can be broadly categorized into two types: traditional methods and deep learning-based methods. Traditional image fusion methods mainly focus on feature extraction from the original images. These methods include sparse representation-based, subspace-based, saliency-based, and total variation-based methods. While traditional methods have achieved good fusion results after a long development period, they have many shortcomings. Due to the deficiencies of traditional methods, researchers have developed new deep learning-based fusion methods to address some inherent problems of traditional image fusion methods. Deep learning-based fusion methods can be broadly classified into three categories: methods based on convolutional neural networks (CNNs), methods based on autoencoders (AEs), and methods based on generative adversarial networks (GANs).
[0004] Existing deep learning models are mostly built upon CNNs or GANs. Deep learning-based image fusion methods use CNNs for specific operations during the feature extraction stage. However, due to the small receptive field of CNNs, it's difficult to model long-range dependencies in images. This limitation directly impacts the quality of the fused image; therefore, we need to model global dependencies. In 2023, Chen et al. proposed an end-to-end image fusion method combining CNNs and Vision Transformers to address this problem. This method retains the advantages of CNNs in extracting local features while improving the global dependencies of the images, resulting in a better fused image.
[0005] To improve the fusion of infrared and visible light images, spatial and channel attention mechanisms can amplify useful information in the image and suppress harmful interference by assigning different weights. This significantly enhances the model's ability to capture long-range dependencies by assigning differentiated weights to different regions and feature channels. This method optimizes the fusion process, generating fused images with rich detail and high consistency. SENet is a model that improves the performance of convolutional neural networks by introducing a Squeeze-and-Excitation module. "Squeeze" (compression) and "Excitation" (excitation) enable explicit modeling and adaptive calibration of channel-level features. The "Squeeze" operation compresses the information of each channel into a single value through global average pooling, while the "Excitation" operation uses this compressed information to calculate the weights of each channel, thereby recalibrating the feature map, strengthening useful features, and suppressing irrelevant features. CBAM (Convolutional Block Attention Module) is an attention mechanism used to enhance the feature representation capabilities of convolutional neural networks. CBAM's design is inspired by SENet's SE blocks, but CBAM further expands this attention mechanism, enabling it to capture feature relationships in both spatial and channel dimensions more precisely.
[0006] Objective drawbacks of existing technologies:
[0007] 1) Traditional image fusion techniques are limited by the limitations of manual feature extraction, making it difficult to adapt to complex image fusion needs. While this method is acceptable for simple scenes, its performance improvement is limited in variable and complex environments, and it cannot effectively capture subtle features in images, thus affecting the fusion effect.
[0008] 2) In infrared and visible light image fusion tasks, a key step is extracting crucial features, such as texture and edges, from both types of images. However, existing network architectures have limitations in extracting these features; they often struggle to capture subtle details in the images and are susceptible to interference from factors such as thermal radiation, which degrades the quality of the final fused image. To improve fusion performance, further research and development are needed to create fusion techniques that can more effectively integrate texture features at different scales and reduce interference from irrelevant information.
[0009] 3) In the field of image fusion, CNNs and their variants are widely used due to their powerful feature extraction and generalization capabilities. Moreover, their network structures are mature, and corresponding computing devices can accelerate their computation. However, CNNs also have their own drawbacks. CNNs destroy the correlation of distant images, and their inherently small receptive field prevents them from effectively extracting global information, thus affecting the image fusion results. Unfortunately, almost all existing image fusion frameworks use CNN networks as feature extractors without establishing long-term correlations in the images.
[0010] Therefore, a new solution is needed to address the above problems. Summary of the Invention
[0011] The purpose of this invention is to provide an infrared and visible light image fusion network system and its application method. By combining spatial and channel attention mechanisms and multi-scale feature extraction technology, it is specifically designed for processing infrared and visible light image fusion tasks. The core design of this network is to improve the quality of the fused image through fine feature integration while maintaining operational efficiency, thereby solving the technical problems mentioned in the background art.
[0012] To achieve the above objectives, the present invention provides the following technical solution: an infrared and visible light image fusion network system, comprising at least an input part, a pixel-wise loss module, a feature extraction module, a spatial channel cooperative attention module, an image reconstruction module, and a perceptual loss function;
[0013] The input section is used to input infrared images and visible light images;
[0014] The pixel-by-pixel loss module is used to calculate the difference between each pixel of the reconstructed image and the real image, and is used to evaluate image quality;
[0015] The feature extraction module is used to stitch and fuse the input infrared image and visible light image, thereby enhancing the recognition and integration of key features in the image and generating a high-quality fused image with rich details and significant object information.
[0016] The spatial-channel collaborative attention module combines the advantages of spatial attention and channel attention, capturing the interdependence of features in the spatial and channel dimensions for comprehensive reconstruction of input features, providing a comprehensive feature representation for subsequent processing steps;
[0017] The image reconstruction module is used to fuse or restore the processed information into a final image, integrating information from different sources or levels into a clear and complete image;
[0018] The perceptual loss function is based on the VGG19 network and is a loss function used to evaluate the perceptual quality of an image.
[0019] Furthermore, the feature extraction module adopts a dual-branch strategy, which includes a first branch and a second branch;
[0020] The first branch is equipped with a multi-scale separable convolution module that combines blueprint separable convolution with multi-scale convolution structure. The multi-scale separable convolution module is used to enhance the representation ability of feature maps and image resolution through multi-scale convolution, and effectively extract multi-scale features of the image.
[0021] The second branch includes a module consisting of four deformable convolutional layers connected by dense connections. This module is specifically designed to extract salient object features from the fused image. The second branch introduces dense connection technology to ensure that the output of each deformable convolutional layer can be cumulatively contributed to subsequent layers, achieving efficient information flow and feature integration in the network.
[0022] Furthermore, the multi-scale separable convolution module employs three different sizes of convolution kernels, and the multi-scale separable convolution module performs depthwise separable convolution through the Conv2d layer.
[0023] Furthermore, the overall architecture of the spatial channel collaborative attention module includes a first branch, a second branch, and a third branch;
[0024] The first branch is the top branch, and the first branch is the horizontal spatial dimension W of the processed image, which is the height branch;
[0025] The second branch is the middle branch, which processes the vertical spatial dimension H, i.e., the width branch;
[0026] The third branch is the bottom branch, and the third branch focuses on the interaction of the channel dimension of the feature, that is, the channel branch;
[0027] The top and middle branches are used to capture the interdependencies of features in the spatial dimension, and the bottom branch is used to capture the interactions between channels.
[0028] An application method for an infrared and visible light image fusion network system includes at least the following steps:
[0029] S1: Output infrared and visible light images to the feature extraction block through the input part;
[0030] S2: In the feature extraction block, the input infrared image and visible light image are stitched and fused together. After passing through a multi-scale separable convolutional module and a module consisting of four deformable convolutional layers connected by dense connections, the network can capture both local details and global context of the image through multi-scale feature extraction technology.
[0031] S3: The spatial channel collaborative attention module inputting the fused image is refined to further enhance the recognition and integration of key features in the image;
[0032] S4: Utilize per-pixel loss and perceptual loss to ensure that important visual information of the image is preserved during the fusion process.
[0033] Furthermore, the application of the feature extraction block includes at least the following steps:
[0034] The first branch integrates four multi-scale separable convolutions and 1×1 convolutions, which are called multi-scale separable convolution blocks, and the second branch sets up a module composed of four deformable convolutional layers connected by dense connections.
[0035] By introducing dense connection technology in the second branch;
[0036] This allows the operations of the two branches to be performed simultaneously;
[0037] Finally, by applying the Concat operation on the channel dimension, the feature maps obtained from the two branches are concatenated to obtain a more refined feature representation.
[0038] Furthermore, the application of the multi-scale separable convolution module includes at least the following steps:
[0039] First, MSSConv is used to perform depthwise convolution on the input feature map, capturing spatial features at different scales through convolution kernels of different sizes;
[0040] Then, by cascading the results of MSSConv and the original input feature map multiple times, multi-scale and multi-depth level fusion of the feature map was achieved.
[0041] Next, the feature map is further processed through a 1x1 convolutional layer to adjust the number of channels and integrate information;
[0042] Ultimately, this provides richer and more distinctive feature representations for the next layer of the network.
[0043] Furthermore, the application of the spatial channel collaborative attention module includes at least the following steps:
[0044] First, after the input data enters the network, it goes through some preliminary convolutional layers, which are responsible for extracting preliminary features of the image.
[0045] Next, the network uses residual connections to help gradients propagate in deep networks, preventing gradient vanishing or exploding problems during training.
[0046] By employing highly branched architecture, the fluidity and adaptability of data within the model are ensured, and the model is given flexibility to optimize the feature extraction process for specific visual tasks.
[0047] The operation process is similar to that of spatial branching through width branching, but the arrangement operation puts the width first, and only focuses on the transformation and extraction of feature width in the later stages.
[0048] Features of different scales are adjusted and fused through the channel branches;
[0049] Finally, the features obtained from the height branch, width branch, and width branch are fused using the spatial channel collaborative attention module.
[0050] If spatial interaction is enabled, the results of channel interaction will be combined with the features extracted in the spatial dimension and fused by weighted averaging.
[0051] If spatial interaction is not enabled, only the features in height and width are averaged and fused.
[0052] In this way, the spatial channel collaborative attention module can output a comprehensive feature representation that integrates spatial and channel dimension information, providing rich feature maps for subsequent network layers.
[0053] Furthermore, the high degree of branching ensures the fluidity and adaptability of data within the model, and also provides the model with flexibility, enabling it to optimize the feature extraction process for specific visual tasks. This process includes at least the following steps:
[0054] Input data first enters the model and undergoes a preliminary feature extraction stage;
[0055] Next, residual connections are used to help the network maintain effective gradient flow during deep training;
[0056] In this stage, the data undergoes a permute operation to adjust the dimensional order of the feature mapping to suit the requirements of subsequent operations. Then, pooling operations are performed on the features, including average pooling and standard deviation pooling, to extract statistical information from the features, enhance the expressive power of the features, and help reduce overfitting.
[0057] Next, layer normalization is performed to stabilize the learning process and improve the model's generalization performance.
[0058] In the core part of feature extraction, the model uses a convolutional kernel of size K to capture the features of the image, which is used to further refine the feature extraction in the height direction;
[0059] Here, K may represent different values, which can be set by the user.
[0060] After the convolution operation is completed, the feature map is again subjected to a permutation operation, and then multiplied with the value obtained from the first permutation operation to integrate the features.
[0061] Finally, the feature maps are permuted again in preparation for the final output or further processing;
[0062] The entire height branching process is a step-by-step process of extracting and integrating features along the height direction. Multiple permutation operations play a key role in the model, allowing the model to dynamically adjust the dimensional order of the data to adapt to the processing needs of different layers.
[0063] Furthermore, the adjustment and fusion of features at different scales through the channel branching includes at least the following steps:
[0064] Divide the channel branch into the first branch and the second branch;
[0065] The first branch of the channel branch is similar to the height branch and the width branch, except that it lacks an arrangement operation. This branch focuses on the transformation and extraction of feature channels.
[0066] To enhance the representation of channel features, the second branch of the channel branch utilizes multi-scale cross attention to extract channel features;
[0067] Capture multi-scale features by using convolutional kernels of different sizes and corresponding padding;
[0068] Multi-scale cross-attention first applies a deep convolutional layer, and then further refines the features through multiple independent convolutional kernels;
[0069] Finally, channel blending is performed using convolutional layers with a kernel size of 1 to adjust and fuse features at different scales.
[0070] Compared with the prior art, the beneficial effects of the present invention are:
[0071] This invention prioritizes the construction of feature extraction blocks, which incorporate multi-scale separable convolutional modules and deformable convolutions to improve the extraction of environmental and salient information from the fused feature maps.
[0072] Then, by introducing a spatial channel collaborative attention module, the channel information and spatial information of the feature map are refined, while the channel information and spatial information interact to aggregate more information and improve the ability to capture information from the feature map.
[0073] Furthermore, experimental results demonstrate that the proposed method outperforms state-of-the-art methods on the MSRS dataset, highlighting potential future research directions to improve the accuracy of fused images and promote the development of advanced vision tasks. Attached Figure Description
[0074] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0075] Figure 1 This is the overall network framework diagram of the present invention;
[0076] Figure 2 This is a schematic diagram of the feature extraction block of the present invention;
[0077] Figure 3 This is a schematic diagram of the multi-scale separable convolution module of the present invention;
[0078] Figure 4 This is a schematic diagram of the spatial channel collaborative attention module of the present invention;
[0079] Figure 5 This is a schematic diagram illustrating a qualitative comparison between the MSSCFusion method of the present invention and six state-of-the-art methods on the MSRS dataset. Detailed Implementation
[0080] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0081] Example 1:
[0082] See Figure 1 An infrared and visible light image fusion network system includes at least an input part, a per-pixel loss module, a feature extraction module (FEBlock), a spatial channel cooperative attention module (SCCAM), an image reconstruction module (Image Reconstructo), and a perceptual loss function;
[0083] The input section is used to input infrared and visible light images;
[0084] The pixel-by-pixel loss module is used to calculate the difference between each pixel in the reconstructed image and the real image, and is used to evaluate image quality;
[0085] The feature extraction module is used to stitch and fuse the input infrared and visible light images, thereby enhancing the identification and integration of key features in the images and generating a high-quality fused image with rich details and significant object information.
[0086] The spatial-channel collaborative attention module combines the advantages of spatial attention and channel attention, capturing the interdependencies of features in the spatial and channel dimensions for comprehensive reconstruction of input features, providing a comprehensive feature representation for subsequent processing steps;
[0087] The image reconstruction module is used to fuse or restore the processed information into the final image, integrating information from different sources or levels into a clear and complete image;
[0088] The perceptual loss function is based on the VGG19 network and is used to evaluate the perceptual quality of images.
[0089] See Figure 2 The feature extraction module adopts a two-branch strategy, which includes a first branch and a second branch.
[0090] The first branch is equipped with a multi-scale separable convolutional module (MSSCB) that combines blueprint-separated convolution with multi-scale convolutional structures. The multi-scale separable convolutional module is used to enhance the representation ability of feature maps and image resolution through multi-scale convolution, and effectively extract multi-scale features of the image.
[0091] The second branch includes a module consisting of four deformable convolutional layers connected by dense connections. This module is specifically designed to extract salient object features from the fused image. The second branch introduces dense connection technology to ensure that the output of each deformable convolutional layer can be cumulatively contributed to subsequent layers, achieving efficient information flow and feature integration in the network.
[0092] See Figure 3 The multi-scale separable convolution module uses three different sizes of convolution kernels, with kernel sizes of 1, 3 and 5 respectively. The multi-scale separable convolution module performs depthwise separable convolution through the Conv2d layer.
[0093] See Figure 4 The overall architecture of the spatial channel collaborative attention module includes a first branch, a second branch, and a third branch;
[0094] The first branch is the top branch, which is the horizontal spatial dimension W of the image being processed, i.e., the height branch;
[0095] The second branch is the middle branch, which processes the vertical spatial dimension H, i.e., the width branch;
[0096] The third branch is the bottom branch. The third branch focuses on the interaction of the channel dimension of the feature, which is the channel branch.
[0097] The top and middle branches are used to capture the interdependencies of features in the spatial dimension, while the bottom branch is used to capture the interactions between channels.
[0098] The perceptual loss function is structured based on the VGG19 network, decomposing it into four feature levels of varying depths, each capturing different levels of semantic information. Considering the requirements of image fusion tasks, we compute the perceptual loss between the output image and the visible light image at a shallower feature level to emphasize the importance of structural and detailed information. For infrared images, the perceptual loss is computed at a deeper feature level to preserve more salient features. Specifically, the perceptual loss between the output image and the visible light image is computed at the first feature level, while the perceptual loss with the infrared image is computed at the fourth feature level. The perceptual loss is calculated using the mean absolute error (MAE) loss function to measure the similarity between features.
[0099] Example 2:
[0100] See Figure 1 Based on the above embodiment one, an application method for an infrared and visible light image fusion network system is proposed, which includes at least the following steps:
[0101] S1: Output infrared and visible light images to the feature extraction block through the input part;
[0102] S2: In the feature extraction block, the input infrared image and visible light image are stitched and fused together. After passing through a multi-scale separable convolutional module and a module consisting of four deformable convolutional layers connected by dense connections, the network can capture both local details and global context of the image through multi-scale feature extraction technology.
[0103] S3: The spatial channel collaborative attention module inputting the fused image is refined to further enhance the recognition and integration of key features in the image;
[0104] S4: Utilize per-pixel loss and perceptual loss to ensure that important visual information of the image is preserved during the fusion process.
[0105] See Figure 2 The application of feature extraction blocks includes at least the following steps:
[0106] In the first branch, Multi-Scale Separable Convolution (MSSConv) is a key technique. It employs three different kernel sizes (1, 3, 5) to perform depthwise separable convolution through Conv2d layers. This design allows the network to capture both local details and global information of the image simultaneously. By applying MSSConv and 1x1 convolutions multiple times, the network learns more complex feature representations. Depthwise separable convolution also reduces the number of model parameters and computational cost. In this convolution, the `groups` parameter of each depthwise convolutional layer is set to the number of input channels, ensuring that each input channel is convolved independently, and its output channel count matches the input channel count. Pointwise convolutional layers then perform 1x1 convolutions on the outputs of the depthwise convolutional layers, mixing features from different channels and adjusting the output channel count according to the final requirements. Through residual connections, we add the concatenated output to the original input `x`, which helps gradient flow and may improve training stability.
[0107] In the second branch, we leverage the ability of deformable convolutions to effectively extract local information, highlighting key details in the fused image. To further enhance feature transfer and fusion, we employ dense connections to ensure that the output of each deformable convolutional layer cumulatively contributes to subsequent layers, achieving efficient information flow and feature integration within the network. Finally, the information from the two branches is further fused using a concat operation.
[0108] See Figure 4 The application of the spatial channel collaborative attention module includes at least the following steps:
[0109] First, after the input data enters the network, it goes through some preliminary convolutional layers, which are responsible for extracting preliminary features of the image.
[0110] Next, the network uses residual connections to help gradients propagate in deep networks, preventing gradient vanishing or exploding problems during training.
[0111] By employing highly branched architecture, the fluidity and adaptability of data within the model are ensured, and the model is given flexibility to optimize the feature extraction process for specific visual tasks.
[0112] The operation process is similar to that of spatial branching through width branching, but the arrangement operation puts the width first, and only focuses on the transformation and extraction of feature width in the later stages.
[0113] Features at different scales are adjusted and fused through channel branching;
[0114] Finally, the features obtained from the height branch, width branch, and width branch are fused using the spatial channel collaborative attention module.
[0115] If spatial interaction is enabled, the results of channel interaction will be combined with the features extracted in the spatial dimension and fused by weighted averaging.
[0116] If spatial interaction is not enabled, only the features in height and width are averaged and fused.
[0117] In this way, the spatial-channel collaborative attention module can output a comprehensive feature representation that integrates spatial and channel dimension information, providing rich feature maps for subsequent network layers. Through this design, the spatial-channel collaborative attention module not only enhances the expressive power of features, but also enhances the training stability and gradient flow of the model through residual connections, thereby improving the model's ability to understand complex data structures.
[0118] By employing highly branched architecture, the fluidity and adaptability of data within the model are ensured, and the model is given flexibility to optimize the feature extraction process for specific visual tasks. This process includes at least the following steps:
[0119] Input data first enters the model and undergoes a preliminary feature extraction stage;
[0120] Next, residual connections are used to help the network maintain effective gradient flow during deep training;
[0121] In this stage, the data undergoes a permute operation to adjust the dimensional order of the feature mapping to suit the requirements of subsequent operations. Then, pooling operations are performed on the features, including average pooling and standard deviation pooling, to extract statistical information from the features, enhance the expressive power of the features, and help reduce overfitting.
[0122] Next, layer normalization is performed to stabilize the learning process and improve the model's generalization performance.
[0123] In the core part of feature extraction, the model uses a convolutional kernel of size K to capture the features of the image, which is used to further refine the feature extraction in the height direction;
[0124] Here, K may represent different values, which can be set by the user.
[0125] After the convolution operation is completed, the feature map is again subjected to a permutation operation, and then multiplied with the value obtained from the first permutation operation to integrate the features.
[0126] Finally, the feature maps are permuted again in preparation for the final output or further processing;
[0127] The entire height branching process is a step-by-step process of extracting and integrating features along the height direction. Multiple permutation operations play a key role in the model, allowing the model to dynamically adjust the dimensional order of the data to adapt to the processing needs of different layers.
[0128] Adjusting and fusing features at different scales through channel branching includes at least the following steps:
[0129] Divide the channel branch into the first branch and the second branch;
[0130] The first branch of the channel branch is similar to the height and width branches, except that it lacks a permutation operation. This branch focuses on the transformation and extraction of feature channels.
[0131] To enhance the representation of channel features, the second branch of the channel branch utilizes multi-scale cross attention to extract channel features;
[0132] Multi-scale features are captured by using convolutional kernels of different sizes (5,[1,7],[1,11],[1,21]) and corresponding padding (2,[0,3],[0,5],[0,10]);
[0133] Multi-scale cross-attention first applies a deep convolutional layer conv0 (5×5 convolutional layer), and then further refines the features through multiple independent convolutional kernels (conv{i}_1,conv{i}_2);
[0134] Finally, channel blending is performed using convolutional layers with a kernel size of 1 to adjust and fuse features at different scales.
[0135] Based on the above embodiments one and two, the following technical evaluation is proposed:
[0136] To comprehensively evaluate the proposed algorithm, this invention performs extensive quantitative and qualitative evaluations on the MSRS dataset. Furthermore, it is compared with six state-of-the-art algorithms—six CNN-based methods: SDNet, U2Fusion, SeAFusion, DIVFusion, PSFusion, and SCGRFuse. The implementations of these six algorithms are publicly available, and the parameters set in this invention are consistent with those in the original paper.
[0137] In terms of quantitative evaluation, this invention selected six indicators—PSNR, MI, VIF, SSIM, FMI_pixel, FMI_dct, and Qabf—to objectively evaluate the fusion effect. Higher values for these indicators indicate better fusion performance. The quantitative results of these six statistical indicators for 361 pairs of images are shown in Table 1. Bold values represent the best performance, and underlined values represent the second-best performance.
[0138] Table 1 provides a quantitative comparison of 361 pairs of images from the MSRS dataset based on six metrics.
[0139]
[0140] As shown in Table 1, this invention demonstrates significant advantages in PSNR, FMI, and Qabf. PSNR characterizes the ratio of peak power to noise power in the fused image, reflecting the distortion during the fusion process at the pixel level. A higher PSNR value indicates less distortion during fusion, meaning the fused image is closer to the source image. FMI, based on MI and feature information, represents the amount of feature information transferred from the source image to the fused image. A higher FMI value indicates more features transferred to the fused image. Qabf measures the edge information transferred from the source image to the fused image. The Sobel edge operator is typically used to calculate edge strength and direction in Qabf measurement. A higher Qabf value indicates more edge information transferred to the fused image.
[0141] Furthermore, this invention exhibits the best PSNR, indicating that the fused image has minimal distortion and is closer to the source image. The method also achieves the best Qabf, meaning the fused image retains more edge information. Additionally, this invention demonstrates the highest FMI, indicating the fused image contains the most feature information. The SSIM metric of this invention is only slightly different from SCGRFuse. This invention showcases fusion results for both daytime and nighttime scenes, using a red box to magnify an area to illustrate varying degrees of spectral contamination of texture details. In daytime scenes, U2Fusion and SDNet fail to preserve texture details well in visible light images, and other methods are inevitably affected by unwanted information. For nighttime scenes, all algorithms fuse complementary information from infrared and visible light images to some extent, but most introduce unwanted information into the fused image, manifested in the weakening of prominent targets and contamination of texture background details. Only this invention, PSFusion, and SCGRFuse retain rich texture details and prominent targets, but compared to SCGRFuse, the fused image of this invention has higher contrast.
[0142] See Figure 5Image fusion aims to preserve the advantages of different images in the fused image. However, existing fusion algorithms are complex in design and neglect the interaction between spatial channels in the attention mechanism and the impact of multi-scale feature extraction on depth features. To address these issues, this invention proposes an infrared and visible light image fusion method called MSSCFusion. First, we construct a feature extraction block (FEBlock), combining the advantages of multi-scale separable convolution (MSSC) and deformable convolution (DeformerConv2d) to improve the extraction of environmental and salient information in the fused feature map, respectively. Then, we introduce a spatial and channel collaborative attention mechanism module (SCCAM) to refine the channel and spatial information of the feature map while allowing the channel and spatial information to interact to aggregate more information and improve the information capture capability of the feature map. Experimental results show that the proposed method outperforms state-of-the-art methods on the MSRS dataset, highlighting potential future research directions to improve the accuracy of fused images and promote the development of advanced vision tasks.
[0143] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
Claims
1. An infrared and visible light image fusion network system, characterized in that: It includes at least an input part, a pixel-wise loss module, a feature extraction module, a spatial channel cooperative attention module, an image reconstruction module, and a perceptual loss function; The input section is used to input infrared images and visible light images; The pixel-by-pixel loss module is used to calculate the difference between each pixel of the reconstructed image and the real image, and is used to evaluate image quality; The feature extraction module is used to stitch and fuse the input infrared image and visible light image, thereby enhancing the recognition and integration of key features in the image and generating a high-quality fused image with rich details and significant object information. The feature extraction module adopts a dual-branch strategy, which includes a first branch and a second branch. The first branch is equipped with a multi-scale separable convolution module that combines blueprint separable convolution with multi-scale convolution structure. The multi-scale separable convolution module is used to enhance the representation ability of feature maps and image resolution through multi-scale convolution, and effectively extract multi-scale features of the image. The second branch includes a module consisting of four deformable convolutional layers connected by dense connections. This module is specifically designed to extract salient object features from the fused image. The dense connection technique introduced in the second branch ensures that the output of each deformable convolutional layer cumulatively contributes to subsequent layers, achieving efficient information flow and feature integration within the network. The spatial-channel collaborative attention module combines the advantages of spatial attention and channel attention, capturing the interdependence of features in the spatial and channel dimensions for comprehensive reconstruction of input features, providing a comprehensive feature representation for subsequent processing steps; The overall architecture of the spatial channel collaborative attention module includes a top branch, a middle branch, and a bottom branch; The top branch is the horizontal spatial dimension W of the processed image, which is the height branch; The intermediate branch is for processing the vertical spatial dimension H, which is the width branch; The bottom branch refers to the interaction of the channel dimension of the feature, which is the channel branch; The top and middle branches are used to capture the interdependencies of features in the spatial dimension, and the bottom branch is used to capture the interactions between channels; The image reconstruction module is used to fuse or restore the processed information into a final image, integrating information from different sources or levels into a clear and complete image; The perceptual loss function is based on the VGG19 network and is a loss function used to evaluate the perceptual quality of an image.
2. The infrared and visible light image fusion network system according to claim 1, characterized in that: The multi-scale separable convolution module employs three different sizes of convolution kernels, and performs depthwise separable convolution through the Conv2d layer.
3. An application method for an infrared and visible light image fusion network system, based on the infrared and visible light image fusion network system according to any one of claims 1-2, characterized in that: At least the following steps are included: S1: Output infrared and visible light images to the feature extraction module through the input part; S2: In the feature extraction module, the input infrared image and visible light image are stitched and fused. After passing through a multi-scale separable convolution module and a module consisting of four deformable convolutional layers connected by dense connections, the network can capture both local details and global context of the image through multi-scale feature extraction technology. S3: The spatial channel collaborative attention module inputting the fused image is refined to further enhance the recognition and integration of key features in the image; S4: Utilize per-pixel loss and perceptual loss to ensure that important visual information of the image is preserved during the fusion process.
4. The application method of the infrared and visible light image fusion network system according to claim 3, characterized in that: The application of the feature extraction module includes at least the following steps: The first branch integrates four multi-scale separable convolutions and 1×1 convolutions, which are called multi-scale separable convolution blocks, and the second branch sets up a module composed of four deformable convolutional layers connected by dense connections. By introducing dense connection technology in the second branch; This allows the operations of the two branches to be performed simultaneously; Finally, by applying the Concat operation on the channel dimension, the feature maps obtained from the two branches are concatenated to obtain a more refined feature representation.
5. The application method of the infrared and visible light image fusion network system according to claim 4, characterized in that: The application of the multi-scale separable convolutional module includes at least the following steps: First, MSSConv is used to perform depthwise convolution on the input feature map, capturing spatial features at different scales through convolution kernels of different sizes; Then, by concatenating the results of MSSConv multiple times with the original input feature map, multi-scale and multi-depth level fusion of feature maps was achieved; Next, the feature map is further processed through a 1x1 convolutional layer to adjust the number of channels and integrate information; Ultimately, this provides richer and more distinctive feature representations for the next layer of the network.
6. The application method of the infrared and visible light image fusion network system according to claim 3, characterized in that: The application of the spatial channel collaborative attention module includes at least the following steps: First, after the input data enters the network, it goes through some preliminary convolutional layers, which are responsible for extracting preliminary features of the image. Next, the network uses residual connections to help gradients propagate in deep networks, preventing gradient vanishing or exploding problems during training. By employing highly branched architecture, the fluidity and adaptability of data within the model are ensured, and the model is given flexibility to optimize the feature extraction process for specific visual tasks. The operation process is similar to that of spatial branching through width branching, but the arrangement operation puts the width first, and only focuses on the transformation and extraction of feature width in the later stages. Features of different scales are adjusted and fused through the channel branches; Finally, the features obtained from the height branch, width branch, and channel branch are fused using a spatial channel collaborative attention module. If spatial interaction is enabled, the results of channel interaction will be combined with the features extracted in the spatial dimension and fused by weighted averaging. If spatial interaction is not enabled, only the features in height and width are averaged and fused. In this way, the spatial channel collaborative attention module can output a comprehensive feature representation that integrates spatial and channel dimension information, providing rich feature maps for subsequent network layers.
7. The application method of the infrared and visible light image fusion network system according to claim 6, characterized in that: The high branching ensures the fluidity and adaptability of data within the model, and also provides flexibility to optimize the feature extraction process for specific visual tasks. This process includes at least the following steps: Input data first enters the model and undergoes a preliminary feature extraction stage; Next, residual connections are used to help the network maintain effective gradient flow during deep training; In this stage, the data undergoes a sorting operation to adjust the dimensional order of the feature mapping to meet the requirements of subsequent operations. Then, pooling operations are performed on the features, including average pooling and standard deviation pooling, to extract statistical information from the features, enhance the expressive power of the features, and help reduce overfitting. Next, layer normalization is performed to stabilize the learning process and improve the model's generalization performance. In the core part of feature extraction, the model uses a convolutional kernel of size K to capture the features of the image, which is used to further refine the feature extraction in the height direction; Here, K may represent different values, which can be set by the user. After the convolution operation is completed, the feature map is again subjected to a permutation operation, and then multiplied with the value obtained from the first permutation operation to integrate the features. Finally, the feature maps are permuted again in preparation for the final output or further processing; The entire height branching process is a step-by-step process of extracting and integrating features along the height direction. Multiple permutation operations play a key role in the model, allowing the model to dynamically adjust the dimensional order of the data to adapt to the processing needs of different layers.
8. The application method of the infrared and visible light image fusion network system according to claim 6, characterized in that: Adjusting and fusing features at different scales through the channel branches includes at least the following steps: Divide the channel branch into two branches: the first branch of the channel branch and the second branch of the channel branch. Compared to the height and width branches, one branch of the channel branch has one less arrangement operation, and the channel branch focuses on transformation and extraction on the feature channels. To enhance the representation of channel features, the two branches of the channel branch utilize multi-scale cross attention to extract channel features; Capture multi-scale features by using convolutional kernels of different sizes and corresponding padding; Multi-scale cross-attention first applies a deep convolutional layer, and then further refines the features through multiple independent convolutional kernels; Finally, channel blending is performed using convolutional layers with a kernel size of 1 to adjust and fuse features at different scales.