Infrared and visible light image fusion network system and application method thereof

Abstract

The invention discloses an infrared and visible light image fusion network system and an application method thereof, and relates to the technical field of image fusion. According to the method, a feature extraction block is constructed preferentially, and a multi-scale separable convolution module and deformable convolution are combined in the feature extraction block, so that extraction of environment information and highlighted information in a fused feature map is improved; then, by introducing a space channel cooperation attention module, channel information and space information of the feature map are refined, meanwhile, the channel information and the space information are made to interact to aggregate more information, and the information capturing capacity of the feature map is improved; experimental results show that the performance of the proposed method on an MSRS data set is superior to that of the most advanced method, and the future potential research direction is highlighted, so that the precision of the fused image is improved, and meanwhile, the development of advanced visual tasks is promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to an infrared and visible light image fusion network system and an application method thereof. Background Art

[0002] Image fusion technology generates a new image by combining the information of multi-source images. This method has shown its unique value in many fields. Image fusion tasks generally include infrared and visible light image fusion, medical image fusion, multi-exposure image fusion and multi-focus image fusion. Multimodal image fusion is generally divided into feature extraction, feature fusion and feature reconstruction. The generated fused image not only has more information and richer details, but also has stronger robustness. It can provide a more comprehensive scene description, thus laying a solid foundation for subsequent image processing and analysis tasks. This technology is particularly suitable for advanced visual systems such as military reconnaissance, target recognition, target tracking and semantic segmentation. It can not only effectively reduce the duplication of data, but also significantly improve the efficiency and accuracy of data processing. Therefore, image fusion plays a vital role in the preprocessing stage of modern visual tasks, provides strong support for various applications, and promotes the development of related technologies and the optimization of decision-making.

[0003] The fusion of infrared and visible light images is an important topic in the field of image processing and has a wide range of applications in the military and security fields. Due to the practicality of infrared images and visible light images, the existing image fusion methods can be roughly divided into two categories, namely traditional methods and deep learning-based methods. Traditional image fusion methods mainly focus on feature extraction of original images. Traditional image fusion methods also include sparse representation-based methods, subspace-based methods, saliency-based methods, and total variation-based methods. Although traditional methods can also achieve good fusion effects after a long development cycle, they have many defects. Due to the shortcomings of traditional methods, researchers have developed new fusion methods based on deep learning to solve some inherent problems of traditional image fusion methods. Fusion methods based on deep learning can be roughly divided into three categories: methods based on convolutional neural networks (CNNs), methods based on autoencoders (AEs), and methods based on generative adversarial networks (GANs).

[0004] Existing deep learning models are basically built on the basis of CNN or GAN networks. The image fusion method based on deep learning uses CNN for specific operations in the feature extraction stage. Since the receptive field of CNN is very small, it is difficult to model the long-range dependency of the image. The limitation of the receptive field directly affects the quality of the fused image, so we need to model the global dependency. In 2023, Chen et al. proposed an end-to-end image fusion method combining CNN and Vision Transformer to solve the above problems. While retaining the advantages of CNN in extracting local features, the above method improves the global dependency of the image to obtain a better fused image.

[0005] In order to improve the effect of infrared and visible light image fusion, the application of spatial and channel attention mechanisms can amplify useful information in the image and suppress the interference of harmful information by assigning different weights to different regions and feature channels, and significantly improve the model's ability to capture long-distance dependent information. This method can optimize the fusion process to generate fused images with rich details and high consistency. SENet is a model that improves the performance of convolutional neural networks by introducing Squeeze-and-Excitation modules, "Squeeze" (compression) and "Excitation" (excitation) - to achieve explicit modeling and adaptive calibration of channel-level features. The "Squeeze" operation compresses the information of each channel into a single value through global average pooling, while the "Excitation" operation uses this compressed information to calculate the weight of each channel, thereby recalibrating the feature map, strengthening useful features and suppressing irrelevant features. CBAM (Convolutional Block Attention Module) is an attention mechanism used to enhance the feature representation ability of convolutional neural networks. The design of CBAM is inspired by the SE block of SENet, but CBAM further extends this attention mechanism to enable it to capture feature relationships in both spatial and channel dimensions in more detail.

[0006] Objective disadvantages of existing technology:

[0007] 1) Traditional image fusion technology is limited by the limitations of artificial feature extraction and is difficult to adapt to complex image fusion requirements. This method is still acceptable for processing simple scenes, but its performance improvement is limited in changing environments and complex scenes, and it cannot effectively capture subtle features in the image, thus affecting the fusion effect.

[0008] 2) In the task of fusion of infrared and visible light images, the key step is to extract key features such as texture and edges from both images. However, existing network structures have limitations in extracting these features. They often have difficulty capturing subtle details in the image and are easily affected by interference factors such as thermal radiation, which will reduce the quality of the final fused image. In order to improve the fusion effect, further research and development is needed to integrate texture features of different scales more effectively and reduce the interference of irrelevant information.

[0009] 3) In the field of image fusion, CNN and its variants are widely used due to their powerful feature extraction and generalization capabilities. Moreover, their network structure is mature and the corresponding computing devices can accelerate their calculations. However, CNN also has its own defects. CNN destroys the correlation of long-range images and its inherent small receptive field cannot effectively extract global information, which in turn affects the results of image fusion. Unfortunately, almost all existing image fusion frameworks use CNN networks as feature extractors without establishing long-range correlations in images.

[0010] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0011] The purpose of the present invention is to provide an infrared and visible light image fusion network system and its application method, which is specially used to process the fusion task of infrared and visible light images by combining spatial and channel attention mechanisms and multi-scale feature extraction technology. The design core of the network is to improve the quality of the fused image through fine feature integration while maintaining the high efficiency of the operation to solve the technical problems raised in the background technology.

[0012] To achieve the above object, the present invention provides the following technical solutions: an infrared and visible light image fusion network system, comprising at least an input part, a pixel-by-pixel loss module, a feature extraction module, a spatial channel collaborative attention module, an image reconstruction module and a perceptual loss function;

[0013] The input part is used to input infrared images and visible light images;

[0014] The pixel-by-pixel loss module is used to calculate the difference between each pixel of the reconstructed image and the real image, so as to evaluate the image quality;

[0015] The feature extraction module is used to splice and fuse the input infrared image and visible light image, thereby enhancing the recognition and integration of key features in the image and generating a high-quality fused image with rich details and significant object information;

[0016] The spatial channel collaborative attention module combines the advantages of spatial attention and channel attention, captures the feature interdependence in spatial and channel dimensions, and is used to fully reconstruct the input features, providing a comprehensive feature representation for subsequent processing steps;

[0017] The image reconstruction module is used to fuse or restore the processed information into a final image, integrating information from different sources or levels into a clear and complete image;

[0018] The perceptual loss function is based on the VGG19 network, and the perceptual loss function is a loss function used to evaluate the perceptual quality of an image.

[0019] Furthermore, the feature extraction module adopts a dual-branch strategy, and the dual-branch strategy includes a first branch and a second branch;

[0020] The first branch is equipped with a multi-scale separable convolution module that combines blueprint separation convolution with a multi-scale convolution structure, and the multi-scale separable convolution module is used to enhance the representation ability and image resolution of the feature map through multi-scale convolution, and effectively extract the multi-scale features of the image;

[0021] The second branch includes a module composed of four deformable convolutional layers connected densely. The module in the second branch is used to specifically extract the features of salient objects in the fused image. The second branch introduces dense connection technology to ensure that the output of each layer of deformable convolution can be cumulatively contributed to the subsequent layers, thereby achieving efficient information flow and feature integration in the network.

[0022] Furthermore, the multi-scale separable convolution module adopts convolution kernels of three different sizes, and the multi-scale separable convolution module performs depth-wise separable convolution through a Conv2d layer.

[0023] Furthermore, the overall architecture of the spatial channel collaborative attention module includes a first branch, a second branch, and a third branch;

[0024] The first branch is a top branch, and the first branch processes the horizontal spatial dimension W of the image, that is, a height branch;

[0025] The second branch is a middle branch, and the second branch processes the vertical space dimension H, that is, the width branch;

[0026] The third branch is a bottom branch, and the third branch focuses on the channel dimension interaction of the features, that is, the channel branch;

[0027] The top branch and the middle branch are used to capture feature interdependencies in the spatial dimension, and the bottom branch is used to capture inter-channel interactions.

[0028] An application method of an infrared and visible light image fusion network system comprises at least the following steps:

[0029] S1: Outputs infrared image and visible light image to feature extraction block through input part;

[0030] S2: In the feature extraction block, the input infrared image and visible light image are spliced ​​and fused, and then the multi-scale separable convolution module and four deformable convolution layers are densely connected to form a module. The multi-scale feature extraction technology enables the network to capture both the local details and the global context of the image.

[0031] S3: The spatial channel collaborative attention module inputs the fused image for refinement to further enhance the recognition and integration of key features in the image;

[0032] S4: Per-pixel loss and perceptual loss are used to ensure that important visual information of the image is preserved during the fusion process.

[0033] Furthermore, the application of the feature extraction block comprises at least the following steps:

[0034] The first branch integrates four multi-scale separable convolutions and 1×1 convolutions, which is a multi-scale separable convolution block, and the second branch sets a module consisting of four deformable convolution layers through dense connections;

[0035] By introducing dense connection technology in the second branch;

[0036] Allows the operations of the two branches to be performed simultaneously;

[0037] Finally, by applying the Concat operation on the channel dimension, the feature maps obtained from the two branches are concatenated to obtain a more refined feature representation.

[0038] Furthermore, the application of the multi-scale separable convolution module includes at least the following steps:

[0039] First, we use MSSConv to perform deep convolution on the input feature map, and use convolution kernels of different sizes to capture spatial features of different scales.

[0040] Then, by concatenating the results of multiple applications of MSSConv and the original input feature map, multi-scale and multi-depth fusion of feature maps is achieved;

[0041] Next, the feature map is further processed by a 1x1 convolutional layer to adjust the number of channels and integrate information;

[0042] Ultimately, a richer and more discriminative feature representation is provided to the next layer of the network.

[0043] Furthermore, the application of the spatial channel collaborative attention module includes at least the following steps:

[0044] First, after the input data enters the network, it is processed by some preliminary convolutional layers, which are responsible for extracting preliminary features of the image;

[0045] Next, the network uses residual connections to help the gradient propagate in the deep network to prevent the gradient from disappearing or exploding during training;

[0046] The high degree of branching ensures the fluidity and adaptability of data in the model, and also provides flexibility for the model to optimize the feature extraction process for specific visual tasks;

[0047] Through the width branch, the operation process is similar to the spatial branch, but the arrangement operation puts the width in the front, and the latter only focuses on the transformation and extraction of feature width;

[0048] Adjusting and fusing features of different scales through the channel branches;

[0049] Finally, the spatial channel collaborative attention module is used to fuse the features obtained from the height branch, width branch, and width branch;

[0050] If spatial interaction is enabled, the results of channel interaction will be combined with the features extracted in the spatial dimension and fused by weighted averaging;

[0051] If spatial interaction is not enabled, only the features in height and width are averaged and fused;

[0052] In this way, the spatial-channel collaborative attention module can output a comprehensive feature representation that integrates spatial and channel dimension information, providing rich feature maps for subsequent network layers.

[0053] Furthermore, the high degree of branching ensures the fluidity and adaptability of data in the model and provides flexibility for the model to optimize the feature extraction process for specific visual tasks. The process includes at least the following steps:

[0054] The input data first enters the model and undergoes a preliminary feature extraction phase;

[0055] Next, residual connections are used to help the network maintain effective gradient flow during deep training;

[0056] In this stage, the data is permuted to adjust the dimensional order of the feature map to meet the requirements of subsequent operations. Subsequently, the features are pooled, including average pooling and standard deviation pooling, which are used to extract the statistical information of the features, enhance the expressiveness of the features, and help reduce overfitting.

[0057] Next, layer normalization is performed to stabilize the learning process and improve the generalization performance of the model;

[0058] In the core part of feature extraction, the model uses a convolution kernel of size K to capture the features of the image, which is used to further refine the feature extraction in the height direction;

[0059] Here K may represent different values, which are set by the user;

[0060] After the convolution operation is completed, the feature map is permuted again and multiplied with the value obtained by the first permutation operation to integrate the features;

[0061] Finally, the feature maps are permuted again to prepare for final output or further processing;

[0062] The entire highly branched process is a process of gradually extracting and integrating features in the height direction. Multiple permutation operations play a key role in the model, allowing the model to dynamically adjust the dimensional order of the data to adapt to the processing requirements of different layers.

[0063] Furthermore, the adjusting and fusing features of different scales through the channel branches at least comprises the following steps:

[0064] Divide the channel branch into a first branch and a second branch;

[0065] The first branch of the channel branch is similar to the height branch and the width branch, except that it lacks an arrangement operation. This branch focuses on the transformation and extraction of feature channels.

[0066] In order to enhance the expression of channel features, the second branch of the channel branch uses multi-scale cross attention to extract channel features;

[0067] Capture multi-scale features through convolution kernels of different sizes and corresponding padding;

[0068] Multi-scale criss-cross attention first applies a deep convolutional layer and then further refines the features through multiple independent convolutional kernels;

[0069] Finally, a convolution layer with a convolution kernel size of 1 is used to perform channel mixing to adjust and fuse features of different scales.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] The present invention preferably constructs a feature extraction block, and sets a multi-scale separable convolution module and a deformable convolution in the feature extraction block, thereby improving the extraction of environmental information and salient information in the fused feature map;

[0072] Then, by introducing the spatial channel collaborative attention module, the channel information and spatial information of the feature map are refined, and the channel information and spatial information are allowed to interact to aggregate more information, thereby improving the information capture capability of the feature map.

[0073] Experimental results show that the proposed method outperforms the state-of-the-art methods on the MSRS dataset, highlighting potential future research directions to improve the accuracy of fused images while promoting the development of advanced vision tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0075] Figure 1 It is the overall network framework diagram of the present invention as a whole;

[0076] Figure 2 It is a schematic diagram of a feature extraction block of the present invention;

[0077] Figure 3 Schematic diagram of the multi-scale separable convolution module of the present invention;

[0078] Figure 4 Schematic diagram of the spatial channel collaborative attention module of the present invention;

[0079] Figure 5 Schematic diagram of qualitative comparison between MSSCFusion of the present invention and six most advanced methods on the MSRS dataset. DETAILED DESCRIPTION

[0080] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0081] Embodiment 1:

[0082] See also Figure 1 , an infrared and visible light image fusion network system, comprising at least an input part, a pixel-by-pixel loss module (Per-pixel Loss), a feature extraction module (FEBlock), a spatial channel collaborative attention module (SCCAM), an image reconstruction module (Image Reconstructo) and a perceptual loss function (Perceptual loss);

[0083] The input part is used to input infrared images and visible light images;

[0084] The pixel-by-pixel loss module is used to calculate the difference between each pixel of the reconstructed image and the real image to evaluate the image quality;

[0085] The feature extraction module is used to stitch and fuse the input infrared image and visible light image, thereby enhancing the recognition and integration of key features in the image and generating a high-quality fused image with rich details and significant object information;

[0086] The spatial-channel collaborative attention module combines the advantages of spatial attention and channel attention, captures the feature interdependencies in spatial and channel dimensions, and is used to fully reconstruct the input features, providing a comprehensive feature representation for subsequent processing steps;

[0087] The image reconstruction module is used to fuse or restore the processed information into the final image, integrating information from different sources or levels into a clear and complete image;

[0088] The perceptual loss function is based on the VGG19 network, and the perceptual loss function is a loss function used to evaluate the perceptual quality of an image.

[0089] See also Figure 2 ,The feature extraction module adopts a dual branch strategy, which includes a first branch and a second branch;

[0090] The first branch is equipped with a multi-scale separable convolution module (MSSCB) that combines blueprint separation convolution with a multi-scale convolution structure. The multi-scale separable convolution module is used to enhance the representation ability of feature maps and image resolution through multi-scale convolution, and effectively extract multi-scale features of images;

[0091] The second branch includes a module composed of four deformable convolutional layers connected densely. The module in the second branch is used to specifically extract the features of salient objects in the fused image. The dense connection technology is introduced in the second branch to ensure that the output of each layer of deformable convolution can be cumulatively contributed to the subsequent layers, thereby realizing efficient information flow and feature integration in the network.

[0092] See also Figure 3 The multi-scale separable convolution module uses three different sizes of convolution kernels, the sizes of which are 1, 3, and 5 respectively. The multi-scale separable convolution module performs depth-wise separable convolution through the Conv2d layer.

[0093] See also Figure 4 ,The overall architecture of the spatial channel collaborative attention module includes the first branch, the second branch, and the third branch;

[0094] The first branch is the top branch, and the first branch processes the horizontal spatial dimension W of the image, that is, the height branch;

[0095] The second branch is the middle branch, and the second branch processes the vertical spatial dimension H, that is, the width branch;

[0096] The third branch is the bottom branch. The third branch focuses on the channel dimension interaction of the features, which is the channel branch.

[0097] The top and middle branches are used to capture feature interdependencies in the spatial dimension, and the bottom branch is used to capture inter-channel interactions.

[0098] The structure of the perceptual loss function is based on the VGG19 network, which is decomposed into four feature levels of different depths, each of which captures a different level of semantic information. Considering the needs of the image fusion task, the perceptual loss between the output image and the visible light image is calculated at a shallower feature level to emphasize the importance of structure and detail information. For infrared images, the perceptual loss is calculated at a deeper feature level, with the aim of retaining more significant features in infrared images. Specifically, the perceptual loss between the output image and the visible light image is calculated at the first feature level, while the perceptual loss with the infrared image is calculated at the fourth feature level. The perceptual loss is calculated using the mean absolute error (MAE) loss function to measure the similarity between features.

[0099] Embodiment 2:

[0100] See also Figure 1 Based on the above embodiment 1, an application method of an infrared and visible light image fusion network system is proposed, which at least includes the following steps:

[0101] S1: Outputs infrared image and visible light image to feature extraction block through input part;

[0102] S2: In the feature extraction block, the input infrared image and visible light image are spliced ​​and fused, and then the multi-scale separable convolution module and four deformable convolution layers are densely connected to form a module. The multi-scale feature extraction technology enables the network to capture both the local details and the global context of the image.

[0103] S3: The spatial channel collaborative attention module inputs the fused image for refinement to further enhance the recognition and integration of key features in the image;

[0104] S4: Per-pixel loss and perceptual loss are used to ensure that important visual information of the image is preserved during the fusion process.

[0105] See also Figure 2 , the application of the feature extraction block includes at least the following steps:

[0106] In the first branch, multi-scale separable convolution (MSSConv) is the key technology, which adopts three different sizes of convolution kernels (1, 3, 5) to perform depth-wise separable convolution through Conv2d layer. This design enables the network to capture local details and global information of the image at the same time. By applying MSSConv and 1x1 convolution multiple times, the feature representation learned by the network is more complex. Depth-wise separable convolution also reduces the number of parameters and computation of the model. In this convolution, the groups parameter of each depth-wise convolution layer is set to the number of input channels, ensuring that each input channel is convolved independently and its output channel number matches the number of input channels. The point-wise convolution layer then performs 1x1 convolution on the output of the depth-wise convolution layer, mixes the features of different channels, and adjusts the number of output channels according to the final demand. Through the residual connection, we add the output after the splicing operation to the original input x, which helps the gradient flow and may improve the stability of training.

[0107] In the second branch, we use the deformable convolution to effectively extract local information to highlight the key information in the fused image. To further enhance feature transfer and fusion, we use a dense connection method to ensure that the output of each layer of deformable convolution can contribute cumulatively to the subsequent layers, achieving efficient information flow and feature integration in the network. Finally, the information obtained from the two branches is further fused through the Concat operation.

[0108] See also Figure 4 , the application of the spatial channel collaborative attention module includes at least the following steps:

[0109] First, after the input data enters the network, it is processed by some preliminary convolutional layers, which are responsible for extracting preliminary features of the image;

[0110] Next, the network uses residual connections to help the gradient propagate in the deep network to prevent the gradient from disappearing or exploding during training.

[0111] The high degree of branching ensures the fluidity and adaptability of data in the model, and also provides flexibility for the model to optimize the feature extraction process for specific visual tasks;

[0112] Through the width branch, the operation process is similar to the spatial branch, but the arrangement operation puts the width in the front, and the latter only focuses on the transformation and extraction of feature width;

[0113] Adjust and fuse features of different scales through channel branches;

[0114] Finally, the spatial channel collaborative attention module is used to fuse the features obtained from the height branch, width branch, and width branch;

[0115] If spatial interaction is enabled, the results of channel interaction will be combined with the features extracted in the spatial dimension and fused by weighted averaging;

[0116] If spatial interaction is not enabled, only the features in height and width are averaged and fused;

[0117] In this way, the spatial channel collaborative attention module can output a comprehensive feature representation that integrates spatial and channel dimension information, providing rich feature maps for subsequent network layers; through this design, the spatial channel collaborative attention module not only enhances the expressiveness of features, but also enhances the training stability and gradient flow of the model through residual connections, improving the model's ability to understand complex data structures.

[0118] The high degree of branching ensures the fluidity and adaptability of data in the model, and also provides the model with flexibility to optimize the feature extraction process for specific visual tasks. The process includes at least the following steps:

[0119] The input data first enters the model and undergoes a preliminary feature extraction phase;

[0120] Next, residual connections are used to help the network maintain effective gradient flow during deep training;

[0121] In this stage, the data is permuted to adjust the dimensional order of the feature map to meet the requirements of subsequent operations. Subsequently, the features are pooled. The pooling operations include average pooling and standard deviation pooling, which are used to extract the statistical information of the features, enhance the expressiveness of the features, and help reduce overfitting.

[0122] Next, layer normalization is performed to stabilize the learning process and improve the generalization performance of the model;

[0123] In the core part of feature extraction, the model uses a convolution kernel of size K to capture the features of the image, which is used to further refine the feature extraction in the height direction;

[0124] Here K may represent different values, which are set by the user;

[0125] After the convolution operation is completed, the feature map is permuted again and multiplied with the value obtained by the first permutation operation to integrate the features;

[0126] Finally, the feature maps are permuted again to prepare for final output or further processing;

[0127] The entire highly branched process is a process of gradually extracting and integrating features in the height direction. Multiple permutation operations play a key role in the model, allowing the model to dynamically adjust the dimensional order of the data to adapt to the processing requirements of different layers.

[0128] Adjusting and fusing features of different scales through channel branches includes at least the following steps:

[0129] Divide the channel branch into a first branch and a second branch;

[0130] The first branch of the channel branch is similar to the height branch and the width branch, except that it lacks a permutation operation. This branch focuses on the transformation and extraction of feature channels.

[0131] In order to enhance the expression of channel features, the second branch of the channel branch uses multi-scale cross attention to extract channel features;

[0132] Capture multi-scale features through convolution kernels of different sizes (5, [1, 7], [1, 11], [1, 21]) and corresponding padding (2, [0, 3], [0, 5], [0, 10]);

[0133] Multi-scale cross attention first applies a deep convolutional layer conv0 (5×5 convolutional layer), and then further refines the features through multiple independent convolutional kernels (conv{i}_1,conv{i}_2);

[0134] Finally, a convolution layer with a convolution kernel size of 1 is used to perform channel mixing to adjust and fuse features of different scales.

[0135] Based on the above-mentioned Embodiment 1 and Embodiment 2, the following technical evaluation is proposed:

[0136] In order to comprehensively evaluate the proposed algorithm, the present invention conducts extensive quantitative and qualitative evaluations on the MSRS dataset. In addition, the present invention is compared with six state-of-the-art algorithms, namely six CNN-based methods, namely SDNet, U2Fusion, SeAFusion, DIVFusion, PSFusion, SCGRFuse. The implementations of these six algorithms are all public, and the parameters set by the present invention are also consistent with the original paper.

[0137] In terms of quantitative evaluation, the present invention selects six indicators, PSNR, MI, VIF, SSIM, FMI_pixel, FMI_dct, and Qabf, to objectively evaluate the fusion effect. The larger the value of these indicators, the better the fusion performance. The quantitative results of the six statistical indicators for 361 pairs of images are shown in Table 1. The bold values ​​are the best values, and the underlined values ​​are the second best values.

[0138] Table 1 compares the 361 pairs of images in the MSRS dataset using six indicators.

[0139]

[0140] As shown in Table 1, the present invention shows significant advantages in PSNR, FMI, and Qabf. PSNR characterizes the ratio of peak power to noise power in the fused image, and can reflect the distortion in the fusion process from the pixel level. The higher the PSNR value, the smaller the distortion in the fusion process, that is, the closer the fused image is to the source image. FMI is based on MI and feature information, and represents the amount of feature information transferred from the source image to the fused image. The higher the FMI value, the more features are transferred to the fused image. Qabf is used to measure the edge information transferred from the source image to the fused image. In the Qabf metric, the Sobel edge operator is usually used to calculate the edge strength and direction. The higher the Qabf value, the more edge information is transferred to the fused image.

[0141] In addition, the present invention presents the best PSNR, which means that the fused image of the present invention has little distortion and the fused image is closer to the source image. The method of the present invention also obtains the best Qabf, which means that the fused image retains more edge information. In addition, the present invention also presents the highest FMI, which means that the fused image contains the most feature information. The SSIM index of the present invention is slightly different from that of SCGRFuse. The present invention shows the fusion results of scenes mainly during the day and at night, and uses a red frame to enlarge an area to illustrate the phenomenon that the texture details are polluted to different degrees by the spectrum. In addition, in daytime scenes, U2Fusion and SDNet cannot well preserve the texture details of visible light images, and other methods are inevitably interfered by useless information. For nighttime scenes, it can be seen that all algorithms have integrated the complementary information in infrared and visible light images to a certain extent, but most algorithms have introduced useless information in the fused image, which is manifested in the weakening of significant targets and the pollution of texture background details. Only the present invention, PSFusion and SCGRFuse can retain rich texture details and prominent targets, but compared with SCGRFuse, the fused image of the present invention has a higher contrast.

[0142] See also Figure 5The purpose of image fusion is to make the fused image maintain the advantages of different images. However, the existing fusion algorithm is complex in design and ignores the interaction between spatial channels in the attention mechanism and the influence of multi-scale feature extraction on deep features. In order to solve these problems, the present invention proposes a method for fusion of infrared and visible light images called MSSCFusion. First, we construct a feature extraction block (FEBlock) to combine the advantages of multi-scale separable convolution (MSSC) and deformable convolution (DeformerConv2d) to improve the extraction of environmental information and salient information in the fused feature map respectively. Then, we introduce the spatial and channel collaborative attention mechanism module (SCCAM) to refine the channel information and spatial information of the feature map while allowing the channel information to interact with the spatial information to aggregate more information and improve the information capture ability of the feature map; the experimental results of the present invention show that the proposed method performs better than the most advanced methods on the MSRS dataset, highlighting the potential research direction in the future to improve the accuracy of the fused image and promote the development of advanced visual tasks.

[0143] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. An infrared and visible light image fusion network system, characterized by: At least includes an input part, a pixel-by-pixel loss module, a feature extraction module, a spatial channel collaborative attention module, an image reconstruction module and a perceptual loss function; The input part is used to input infrared images and visible light images; The pixel-by-pixel loss module is used to calculate the difference between each pixel of the reconstructed image and the real image, so as to evaluate the image quality; The feature extraction module is used to splice and fuse the input infrared image and visible light image, thereby enhancing the recognition and integration of key features in the image and generating a high-quality fused image with rich details and significant object information; The spatial channel collaborative attention module combines the advantages of spatial attention and channel attention, captures the feature interdependence in spatial and channel dimensions, and is used to fully reconstruct the input features, providing a comprehensive feature representation for subsequent processing steps; The image reconstruction module is used to fuse or restore the processed information into a final image, integrating information from different sources or levels into a clear and complete image; The perceptual loss function is based on the VGG19 network, and the perceptual loss function is a loss function used to evaluate the perceptual quality of an image.

2. The infrared and visible light image fusion network system according to claim 1, characterized in that: The feature extraction module adopts a dual-branch strategy, which includes a first branch and a second branch; The first branch is equipped with a multi-scale separable convolution module that combines blueprint separation convolution with a multi-scale convolution structure, and the multi-scale separable convolution module is used to enhance the representation ability and image resolution of the feature map through multi-scale convolution, and effectively extract the multi-scale features of the image; The second branch includes a module composed of four deformable convolutional layers connected densely. The module in the second branch is used to specifically extract the features of salient objects in the fused image. The second branch introduces dense connection technology to ensure that the output of each layer of deformable convolution can be cumulatively contributed to the subsequent layers, thereby achieving efficient information flow and feature integration in the network.

3. The infrared and visible light image fusion network system according to claim 2, characterized in that: The multi-scale separable convolution module uses convolution kernels of three different sizes, and the multi-scale separable convolution module performs depth-wise separable convolution through the Conv2d layer.

4. The infrared and visible light image fusion network system according to claim 1, characterized in that: The overall architecture of the spatial channel collaborative attention module includes a first branch, a second branch and a third branch; The first branch is a top branch, and the first branch processes the horizontal spatial dimension W of the image, that is, a height branch; The second branch is a middle branch, and the second branch processes the vertical space dimension H, that is, the width branch; The third branch is a bottom branch, and the third branch focuses on the channel dimension interaction of the features, that is, the channel branch; The top branch and the middle branch are used to capture feature interdependencies in the spatial dimension, and the bottom branch is used to capture inter-channel interactions.

5. An application method of an infrared and visible light image fusion network system, based on the infrared and visible light image fusion network system according to any one of claims 1 to 4, characterized in that: At least the following steps are included: S1: Outputs infrared image and visible light image to feature extraction block through input part; S2: In the feature extraction block, the input infrared image and visible light image are spliced ​​and fused, and then the multi-scale separable convolution module and four deformable convolution layers are densely connected to form a module. The multi-scale feature extraction technology enables the network to capture both the local details and the global context of the image. S3: The spatial channel collaborative attention module inputs the fused image for refinement to further enhance the recognition and integration of key features in the image; S4: Per-pixel loss and perceptual loss are used to ensure that important visual information of the image is preserved during the fusion process.

6. The application method of the infrared and visible light image fusion network system according to claim 4 is characterized by: The application of the feature extraction block comprises at least the following steps: The first branch integrates four multi-scale separable convolutions and 1×1 convolutions, which is a multi-scale separable convolution block, and the second branch sets a module consisting of four deformable convolution layers through dense connections; By introducing dense connection technology in the second branch; Allows the operations of the two branches to be performed simultaneously; Finally, by applying the Concat operation on the channel dimension, the feature maps obtained from the two branches are concatenated to obtain a more refined feature representation.

7. The application method of the infrared and visible light image fusion network system according to claim 6 is characterized by: The application of the multi-scale separable convolution module comprises at least the following steps: First, we use MSSConv to perform deep convolution on the input feature map, and use convolution kernels of different sizes to capture spatial features of different scales. Then, by concatenating the results of multiple applications of MSSConv and the original input feature map, multi-scale and multi-depth fusion of feature maps is achieved; Next, the feature map is further processed by a 1x1 convolutional layer to adjust the number of channels and integrate information; Ultimately, a richer and more discriminative feature representation is provided to the next layer of the network.

8. The application method of the infrared and visible light image fusion network system according to claim 4 is characterized by: The application of the spatial channel collaborative attention module comprises at least the following steps: First, after the input data enters the network, it is processed by some preliminary convolutional layers, which are responsible for extracting preliminary features of the image; Next, the network uses residual connections to help the gradient propagate in the deep network to prevent the gradient from disappearing or exploding during training; The high degree of branching ensures the fluidity and adaptability of data in the model, and also provides flexibility for the model to optimize the feature extraction process for specific visual tasks; Through the width branch, the operation process is similar to the spatial branch, but the arrangement operation puts the width in the front, and the latter only focuses on the transformation and extraction of feature width; Adjusting and fusing features of different scales through the channel branches; Finally, the spatial channel collaborative attention module is used to fuse the features obtained from the height branch, width branch, and width branch; If spatial interaction is enabled, the results of channel interaction will be combined with the features extracted in the spatial dimension and fused by weighted averaging; If spatial interaction is not enabled, only the features in height and width are averaged and fused; In this way, the spatial-channel collaborative attention module can output a comprehensive feature representation that integrates spatial and channel dimension information, providing rich feature maps for subsequent network layers.

9. The application method of the infrared and visible light image fusion network system according to claim 8 is characterized by: The high degree of branching ensures the fluidity and adaptability of data in the model and provides flexibility for the model to optimize the feature extraction process for specific visual tasks. The process includes at least the following steps: The input data first enters the model and undergoes a preliminary feature extraction phase; Next, residual connections are used to help the network maintain effective gradient flow during deep training; In this stage, the data is permuted to adjust the dimensional order of the feature map to meet the requirements of subsequent operations. Subsequently, the features are pooled, including average pooling and standard deviation pooling, which are used to extract the statistical information of the features, enhance the expressiveness of the features, and help reduce overfitting. Next, layer normalization is performed to stabilize the learning process and improve the generalization performance of the model; In the core part of feature extraction, the model uses a convolution kernel of size K to capture the features of the image, which is used to further refine the feature extraction in the height direction; Here K may represent different values, which are set by the user; After the convolution operation is completed, the feature map is permuted again and multiplied with the value obtained by the first permutation operation to integrate the features; Finally, the feature maps are permuted again to prepare for final output or further processing; The entire highly branched process is a process of gradually extracting and integrating features in the height direction. Multiple permutation operations play a key role in the model, allowing the model to dynamically adjust the dimensional order of the data to adapt to the processing requirements of different layers.

10. The application method of the infrared and visible light image fusion network system according to claim 8, characterized in that: The adjusting and fusing of features of different scales through the channel branches at least comprises the following steps: Divide the channel branch into a first branch and a second branch; The first branch of the channel branch is similar to the height branch and the width branch, except that it lacks an arrangement operation. This branch focuses on the transformation and extraction of feature channels. In order to enhance the expression of channel features, the second branch of the channel branch uses multi-scale cross attention to extract channel features; Capture multi-scale features through convolution kernels of different sizes and corresponding padding; Multi-scale criss-cross attention first applies a deep convolutional layer and then further refines the features through multiple independent convolutional kernels; Finally, a convolution layer with a convolution kernel size of 1 is used to perform channel mixing to adjust and fuse features of different scales.

Images