A method for fusion of infrared light and visible light images

Abstract

The invention discloses a method for fusing infrared and visible light images, and relates to the technical field of image fusion. The invention is designed based on a spatial channel attention mechanism and a gradient aggregation residual dense block, combines the advantages of ResNeXt and DenseNet, and integrates Sobel and Laplacian operators to retain the strong texture and weak texture of the feature, and introduces a spatial and channel attention mechanism to refine the channel information and spatial information of the feature map, thereby improving the information capture capability of the feature map, and uses a pooling fusion block to fuse the refined spatial and channel feature maps to obtain high-quality fusion features.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a method for fusing infrared light and visible light images. Background Art

[0002] Image fusion is an important image enhancement technology. It can extract meaningful information from different source images and fuse them into new images. The fused images usually have the advantages of strong robustness, rich information, and the ability to show more complex and detailed scene expressions. It can not only reduce data redundancy, but also promote the development and decision-making of subsequent applications. Therefore, image fusion has been widely used in the preprocessing module of advanced visual tasks, and has good application prospects in military surveillance, target detection, target tracking, semantic segmentation, etc.

[0003] Due to the practicality of infrared images and visible light images, many image fusion technologies have been born in recent years, which can be roughly divided into two categories: traditional methods and deep learning-based methods. According to different mathematical transformations, traditional methods can be further divided into methods based on multi-scale transformations, such as discrete wavelet transform (DWT), methods based on representation learning, such as sparse representation (SR) and joint sparse representation (JSR), subspace-based methods, saliency-based methods, and hybrid models. Deep learning-based methods can be divided into three categories according to the network architecture: models based on autoencoders (AE), models based on convolutional neural networks (CNN), and models based on generative adversarial networks (GAN).

[0004] Existing deep learning models are basically built on the basis of CNN or GAN networks. Although these methods have achieved good results in the field of image fusion, they all emphasize the quality of image fusion and ignore the needs of advanced visual tasks. In 2022, Tang et al. first proposed the image fusion framework SeAFusion combined with advanced visual tasks. They did not make great innovations in network architecture or learning paradigm, but they looked at the image fusion task from a new perspective, that is, using advanced visual tasks to drive image fusion. The emergence of SeAFusion has proposed new possibilities for image fusion.

[0005] In order to better promote the performance of infrared and visible light image fusion, the use of spatial and channel attention mechanisms can amplify useful information in the image and suppress the interference of harmful information by assigning different weights, while enriching the semantic information and spatial information of the image to promote the application of subsequent high-level visual tasks. The NestFuse network is a pioneering work in the field of image fusion. For multi-scale deep feature fusion, they proposed a fusion strategy based on spatial and channel attention models. In the spatial attention model, the deep features are calculated by l1 norm and soft-max operator to obtain enhanced deep features. In the channel attention model, the channel fusion features are obtained by global pooling and soft-max operator, and finally the spatial features and channel features are added to obtain the final fusion feature reference:

[0006] Deficiencies of existing technology:

[0007] 1) Traditional methods use the same transformation to extract features from different source images. However, this operation does not take into account the feature differences of the source images, which may result in poor expressiveness of the extracted features. On the other hand, the choice of fusion strategies and the complexity of human design in traditional methods limit the improvement of performance. These artificially designed fusion strategies are not only unable to learn, but also bring certain artifacts to the fusion results.

[0008] 2) Before the infrared and visible light images are fused, useful feature information needs to be extracted from them, including texture, edges, etc. However, the existing network architecture cannot effectively extract coarse and fine granularity detail features, and is easily disturbed by thermal radiation, thus affecting the quality of the fused image. How to effectively integrate coarse and fine texture features and suppress the interference of useless information still requires further breakthroughs.

[0009] 3) Existing fusion methods tend to pursue better visual quality and higher evaluation indicators, and rarely systematically consider whether the fused image can promote high-level visual tasks. Although some studies have introduced perceptual losses at the feature level to constrain the fused image and the source image, the perceptual loss cannot effectively enhance the semantic information in the fused image. In addition, other researchers guide the image fusion process by segmentation masks, but the masks only segment some prominent objects, which is limited in enhancing semantic information.

[0010] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0011] The purpose of the present invention is to provide a method for fusing infrared light and visible light images to solve the technical problems raised in the background technology.

[0012] To achieve the above object, the present invention provides the following technical solution: a method for fusing infrared light and visible light images, characterized in that it comprises at least the following steps:

[0013] S1: Build a fusion network, using CNN's end-to-end fusion network as the basic framework. The end-to-end CNN network has powerful feature extraction and learning capabilities;

[0014] S2: Build a gradient aggregation residual dense block, which combines the advantages of ResNext and DenseNet networks, retains shallow and deep features through split-conversion-merge and dense connection, and enhances the feature extraction ability of the network;

[0015] S3: Build a spatial channel attention module, which is used to amplify useful information, suppress the interference of useless information, and promote the development of semantic segmentation tasks to process the deep features of the input;

[0016] S4: Build a separation network to fully enhance the semantic information of the fused image.

[0017] Preferably, the application of the fusion network in S1 at least includes the following steps:

[0018] By sending the infrared light and visible light images to the feature extraction module respectively, the respective depth features are extracted;

[0019] The extracted deep features are then concatenated and sent to the spatial channel attention mechanism to further extract features and suppress the interference of useless information;

[0020] Finally, the fused image is generated through the feature reconstruction module.

[0021] Preferably, the branches in the gradient aggregated residual dense block in S2 perform a set of transformations, each transformation on a low-dimensional embedding, whose outputs are aggregated by summing.

[0022] Preferably, the gradient aggregated residual dense block in S2 sets the cardinality of the aggregated transformation to 2, and is composed of at least one residual block and one residual dense block, and each gradient aggregated residual dense block contains three branches to improve the diversity of extracted features so that it can make full use of the deep features extracted by each convolutional layer in the block;

[0023] Since the source image contains rich texture details, the gradient aggregation residual dense block also integrates the Laplacian operator and the Sobel operator to retain more coarse textures and fine textures in the image.

[0024] Preferably, the spatial channel attention module in S3 at least includes multi-scale feature extraction, spatial channel attention mechanism and pooling fusion block.

[0025] Preferably, the application of the spatial channel attention module in S3 at least comprises the following steps:

[0026] Four convolution blocks with different kernel sizes are used to capture multi-scale and multi-receptive field deep features, which are then fed into the spatial and channel attention mechanisms respectively;

[0027] Generate spatial and channel attention masks through self-attention functions to improve the network's information capture ability, thereby obtaining more accurate spatial and semantic information;

[0028] After the refinement of the attention branch, the channel feature map is upsampled to restore it to its original size. In addition, the present invention uses a convolution operation to control the number of channels of the spatial feature map;

[0029] The generated spatial and channel attention feature maps are fused using the pooling fusion block to generate a fused feature map that satisfies both the intra-class similarity and inter-class difference requirements.

[0030] Preferably, the application of the S4 separation network comprises at least the following steps:

[0031] A real-time segmentation model Bilateral attention decoder is introduced to segment the fused image, and the segmentation network outputs the segmentation result and auxiliary segmentation result;

[0032] The gap between the segmentation result and the semantic label reflects the richness of the semantic information contained in the fused image;

[0033] This gap is used to construct semantic loss;

[0034] Semantic loss is used to guide the training of the fusion network through back-propagation, forcing the fused image to contain more semantic information.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention is designed based on the spatial channel attention mechanism and the gradient aggregation residual dense block, combining the advantages of ResNeXt and DenseNet while integrating Sobel and Laplacian operators to retain the strong texture and weak texture of the features, and by introducing the spatial and channel attention mechanism, the channel information and spatial information of the feature map are refined to improve the information capture ability of the feature map, and a pooling fusion block is used to fuse the refined spatial and channel feature maps to obtain high-quality fused features; the experimental results of the present invention show that the proposed method performs better than the most advanced methods on the MSRS dataset, highlighting the potential research direction in the future to improve the accuracy of the fused image and promote the development of advanced visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0038] Figure 1 It is the spatial attention model under the existing technology;

[0039] Figure 2 It is the channel attention model under the existing technology;

[0040] Figure 3 It is a schematic diagram of the overall network framework of SCGRFuse of the present invention;

[0041] Figure 4 is a schematic diagram of a fusion network of the present invention;

[0042] Figure 5 is a schematic diagram of the GRXDB module of the present invention;

[0043] Figure 6 Schematic diagram of a multi-scale spatial attention module of the present invention;

[0044] Figure 7 It is a schematic diagram of qualitative comparison between SCGRFuse of the present invention and nine most advanced methods on the MSRS dataset;

[0045] Figure 8 A schematic diagram of the process of generating channel attention and spatial attention masks in the present invention;

[0046] Fig. 9 Schematic diagram of the overall structure of the pooling fusion block of the present invention. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0048] See also Figure 1 and Figure 2 It can be seen that there are spatial attention models and channel attention models under the existing technology.

[0049] Embodiment 1:

[0050] This embodiment is used to disclose a method for fusing infrared and visible light images based on a spatial channel attention mechanism and a gradient aggregation residual dense block;

[0051] The design of SCGRFuse is a combination of image fusion and semantic segmentation tasks. It first uses the fusion network to obtain the fusion result of the input infrared and visible light images, and uses the content loss to guide the training of the entire fusion network. The fused image is then sent to the semantic segmentation network, and more semantic information is integrated into the fused image through the semantic loss generated by the segmentation network, so as to optimize the entire fused image. The overall architecture of the network is as follows: Figure 3 shown.

[0052] 1) Fusion Network

[0053] The present invention uses an end-to-end fusion network based on CNN as the basic framework. The end-to-end CNN network has powerful feature extraction and learning capabilities. The infrared and visible light images are respectively sent to the feature extraction module to extract their respective deep features, and then the extracted deep features are spliced ​​and sent to the spatial channel attention mechanism to further extract features and suppress the interference of useless information. Finally, the fused image is generated through the feature reconstruction module. Its network structure is as follows Figure 4 shown.

[0054] 2) Gradient Aggregation Residual Dense Block (GRXDB)

[0055] Gradient Aggregation Residual DenseBlocks (GRXDB) combines the advantages of ResNext and DenseNet networks by splitting

[0056] -Conversion-merging and dense connection methods are used to retain shallow features and deep features. The branches in the GRXDB module perform a set of transformations, each of which is on a low-dimensional embedding, and its output is aggregated by summing. The present invention sets the cardinality of the aggregation transformation to 2, wherein the module of the present invention consists of a residual block and a residual dense block. More precisely, each GRXDB contains three branches to improve the diversity of extracted features so that it can make full use of the deep features extracted by each convolutional layer in the block. Since the source image contains rich texture details, the Laplacian operator and the Sobel operator are also integrated in GRXDB to retain more coarse and fine textures in the image. GRXDB greatly enhances the feature extraction capability of the network, and its structure is as follows: Figure 5 shown.

[0057] 3) Spatial Channel Attention Module (SCAM)

[0058] In order to amplify useful information, suppress the interference of useless information, and promote the development of semantic segmentation tasks, the present invention proposes to use a multi-scale spatial channel attention mechanism to process the input deep features. SCAM mainly consists of three parts: multi-scale feature extraction, spatial channel attention mechanism, and pooling fusion block. Since the same-scale convolution will hinder the extraction of multi-scale information in different features, before calculating the attention weight, in order to obtain more comprehensive source image representation information, the present invention uses four convolution blocks with different kernel sizes to capture multi-scale and multi-receptive field deep features, and then sends the captured deep features to the spatial and channel attention mechanisms respectively. In order to improve the information capture ability of the network and obtain more accurate spatial information and semantic information, the present invention generates spatial and channel attention masks through self-attention functions. After the refinement of the attention branch, the present invention uses an upsampling operation on the channel feature map to restore it to its original size. In addition, the present invention uses a convolution operation to control the number of channels of the spatial feature map. In order to generate a feature map that meets the requirements of both intra-class similarity and inter-class difference, the present invention uses a pooling fusion block (PFB) to fuse the generated spatial and channel attention feature maps, thereby generating a fused feature map that meets the requirements. The structure of the spatial channel attention module is as follows Figure 6 shown.

[0059] 4) Segmentation Network

[0060] In order to fully enhance the semantic information of the fused image, the present invention introduces a real-time segmentation model, BilateralAttention Decoder, to segment the fused image. The segmentation network outputs the segmentation result and the auxiliary segmentation result. The gap between the segmentation result and the semantic label can reflect the richness of the semantic information contained in the fused image. Therefore, this gap can be used to construct the semantic loss. Finally, the semantic loss is used to guide the training of the fusion network through back propagation, forcing the fused image to contain more semantic information.

[0061] Embodiment 2:

[0062] This embodiment is used to further disclose an application of performance evaluation based on the above embodiment 1;

[0063] In order to comprehensively evaluate the proposed method, the present invention conducts extensive quantitative and qualitative evaluations on the MSRS dataset. In addition, the present invention is compared with nine state-of-the-art methods, including a traditional method, namely GTF, an ae-based method, namely DenseFuse, two GAN-based methods, namely FusionGAN, GANMcC, four CNN-based methods, namely IFCNN, SDNet, U2Fusion, SeAFusion, and an image decomposition-based method, namely DeFusion. The implementations of these nine methods are all public, and the parameters set by the present invention are also consistent with the original paper. Among them, DenseFuse and IFCNN use element-wise addition and element-wise maximum fusion strategies to fuse deep features, respectively.

[0064] In terms of quantitative evaluation, the present invention selects seven indicators, EN, MI, VIF, SF, SD, SCD, and Qabf, to objectively evaluate the fusion effect. In addition, the present invention also uses IoU to quantify the segmentation performance. The larger the value of these indicators, the better the fusion performance. The quantitative results of 7 statistical indicators for 361 pairs of images are shown in Table 1. The segmentation results are shown in Table 2.

[0065] Table 1 Quantitative comparison of 7 indicators on 361 pairs of images from the MSRS dataset

[0066]

[0067] As shown in Table 1. The present invention shows significant advantages in SD, SF, and MI. The higher the SD, the higher the contrast of the fused image. The higher the SF value, the clearer the fused image and the better the quality of the fused image. The higher the MI value, the more information the fused image conveys. In addition, the present invention presents the best VIF, which means that the fused image of the present invention is more in line with the human visual system. The method of the present invention also obtains the best Qabf, which means that the fused image retains more edge information. In addition, the present invention also presents the highest EN, which means that the fused image contains the most information. The present invention follows SeAFusion only slightly in the SCD indicator. The qualitative results of SCGRFuse on the MSRS dataset are shown in the following example. Figure 7 As shown. The present invention respectively shows the fusion results of two scenes mainly during the day and at night, and uses a red frame to enlarge an area to illustrate the phenomenon that the texture details are subject to different degrees of spectral pollution. In addition, the present invention uses a green frame to highlight the problem of useless information weakening the prominent target. In daytime scenes, neither GTF nor FusionGAN can retain the texture details of visible light images well, and other methods are inevitably interfered by useless information. For night scenes, it can be seen that all methods have integrated the complementary information in infrared and visible light images to a certain extent, but most methods have introduced useless information in the fused image, which is manifested in the weakening of significant targets and the pollution of texture background details. Only the present invention and SeAFusion can retain rich texture details and prominent targets, but compared with SeAFusion, the fused image of the present invention has a higher contrast.

[0068] Table 2 Segmentation performance of visible light, infrared and fused images on the MSRS dataset (miou).

[0069]

[0070]

[0071] The segmentation results are shown in Table 2. It can be seen that the present invention is basically in a leading position in various types of IoU and ranks first in MIoU. This is due to the two advantages of the present invention. On the one hand, the present invention effectively integrates the complementary information of infrared images and visible light images, which helps the segmentation model to fully understand the imaging scene. On the other hand, SCGRFuse improves the information capture capability under the action of spatial and channel attention mechanisms, and enhances the spatial information and semantic information under the guidance of semantic loss, so that the segmentation network can more accurately describe the imaging scene.

[0072] Embodiment three:

[0073] This embodiment is used to disclose the specific operation process under the above embodiment.

[0074] The purpose of image fusion is to make the fused image maintain the advantages of different images. However, the existing fusion methods are complex in design and ignore the impact of attention mechanism on deep features. To solve these problems, benfam proposed a method for infrared and visible light image fusion called SCGRFuse. First, we constructed a gradient aggregation residual dense block (GRXDB), which combines the advantages of ResNeXt and DenseNet while integrating Sobel and Laplacian operators to preserve the strong and weak textures of features. Then, we introduced spatial and channel attention mechanisms to refine the channel information and spatial information of feature maps to improve the information capture ability of feature maps, and used a pooling fusion block to fuse the refined spatial and channel feature maps to obtain high-quality fused features; the experimental results of this paper show that the proposed method performs better than the most advanced methods on the MSRS dataset, highlighting the potential future research direction to improve the accuracy of fused images while promoting the development of advanced visual tasks.

[0075] Among them, the gradient aggregation residual dense block (GRXDB) and the spatial channel attention module (SCAM) are the key technologies of the present invention.

[0076] 1) Gradient Aggregation Residual Dense Block

[0077] The overall operation flow of the gradient aggregation residual dense block (such as Figure 5 As shown): The GRXDB of the present invention includes three branches. The main branch deploys three 3x3 convolution blocks and two 1x1 convolution blocks. In order to make fuller use of various convolution layers to extract features, the present invention introduces dense connections into the main branch, and uses 1x1 convolution blocks to deal with the differences between channel dimensions. In addition, the main branch also introduces the Laplacian operator to further extract the weak texture of the feature. In addition, in addition to two 3x3 convolution blocks and one 1x1 convolution block, the residual branch also integrates the Sobel operator to retain the strong texture of the feature. The third branch does not perform any processing, which we call the residual branch to retain the information of the input features. Finally, the outputs of the main branch, the residual branch, and the remaining branch are added by element addition to integrate the deep features. It is worth noting that the Relu activation function is equivalent to directly discarding negative activations. This approach may be effective for classification tasks, but it is not suitable for image fusion tasks. In order to better meet the needs of image fusion, the present invention sets the activation function of GRXDB to Leaky ReLU that can retain negative activation information.

[0078] 2) Spatial channel attention module

[0079] The operation flow of the spatial channel attention module (such as Figure 6 As shown): The present invention uses four convolution blocks with different kernel sizes to capture multi-scale and multi-receptive field depth features. The sizes of the convolution blocks are 1x1, 3x3, 5X5, and 7x7 respectively. The activation function of the convolution block of the present invention still selects Leaky ReLU, and then splices the obtained deep features in the channel direction. Finally, a 1x1 convolution block is used to change the number of channels and send them to the spatial and channel attention layer. In the attention mechanism, generating attention masks Ms and Mc is the most important process, in order to improve the information capture ability of the network, thereby obtaining more accurate spatial information and semantic information. The present invention generates spatial and channel attention masks through a self-attention function, and its main process is as follows Figure 8 As shown. In order to generate a channel attention mask, first, the present invention uses global average pooling to average all pixels of each channel map, and then obtains a new 1*1 channel map. Then, the Tanh activation function and the 1x1 convolution block are used to adjust the value of the feature map. It is worth noting that the present invention divides the calculated value of the tanh activation function by 2 and adds 0.5 so that it can be mapped to the range of [0,1]. Finally, the present invention multiplies the calculated channel map with the input feature map to generate the output of the channel attention branch. Similarly, in order to obtain a spatial attention mask, the present invention first uses a 1x1 convolution block to reduce the number of channels of the input feature, and then uses maximum pooling and average pooling to obtain two feature maps. Both feature maps have only one channel and the size is consistent with the input feature. Next, the two feature maps are spliced ​​together, and the number of channels is changed to 1 by using a 1x1 convolution block. The activation function of the convolution block is still tanh, which is consistent with the channel attention mechanism. Finally, the generated spatial attention mask is multiplied by the input feature map to obtain the output of the final spatial attention branch. After the refinement of the attention branch, the present invention uses an upsampling operation on the channel feature map to restore it to its original size, and uses a convolution operation to control the number of channels of the spatial feature map. In order to generate a feature map that meets the requirements of both intra-class similarity and inter-class difference, the present invention uses a Pooling fusion block (PFB) to fuse the generated spatial and channel attention feature maps. The structure of PFB is as follows: Fig. 9 As shown in the figure. The present invention uses 3x3 average pooling to smooth the upsampled channel feature map, and uses reflection filling to construct the true boundary to reduce boundary artifacts. Finally, it is spliced ​​with the spatial feature map, and the boundary is corrected using a 1x1 convolution block and a Leaky ReLU activation function to generate a fused feature map that meets the requirements.

[0080] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for fusing infrared light and visible light images, characterized in that: At least the following steps are included: S1: Build a fusion network, using CNN's end-to-end fusion network as the basic framework. The end-to-end CNN network has powerful feature extraction and learning capabilities; S2: Build a gradient aggregation residual dense block, which combines the advantages of ResNext and DenseNet networks, retains shallow and deep features through split-conversion-merge and dense connection, and enhances the feature extraction ability of the network; S3: Build a spatial channel attention module, which is used to amplify useful information, suppress the interference of useless information, and promote the development of semantic segmentation tasks to process the deep features of the input; S4: Build a separation network to fully enhance the semantic information of the fused image; The application of the fusion network in S1 at least includes the following steps: By sending the infrared light and visible light images to the feature extraction module respectively, the respective depth features are extracted; The extracted deep features are then concatenated and sent to the spatial channel attention mechanism to further extract features and suppress the interference of useless information; Finally, the fused image is generated through the feature reconstruction module; The branches in the gradient aggregation residual dense block in S2 perform a set of transformations, each on a low-dimensional embedding, whose outputs are aggregated by summing; The gradient aggregated residual dense block in S2 sets the cardinality of the aggregated transformation to 2, and is composed of at least one residual block and one residual dense block. Each gradient aggregated residual dense block contains three branches to improve the diversity of extracted features, so that it can make full use of the deep features extracted by each convolutional layer in the block; Since the source image contains rich texture details, the gradient aggregation residual dense block also integrates the Laplacian operator and the Sobel operator to retain more coarse textures and fine textures in the image; The spatial channel attention module in S3 at least includes multi-scale feature extraction, spatial channel attention mechanism and pooling fusion block; The application of the spatial channel attention module in S3 at least comprises the following steps: Four convolution blocks with different kernel sizes are used to capture multi-scale and multi-receptive field deep features, which are then fed into the spatial and channel attention mechanisms respectively; Generate spatial and channel attention masks through self-attention functions to improve the network's information capture ability, thereby obtaining more accurate spatial and semantic information; After the refinement of the attention branch, the channel feature map is upsampled to restore it to its original size. In addition, the present invention uses a convolution operation to control the number of channels of the spatial feature map; Use the pooling fusion block to fuse the generated spatial and channel attention feature maps to generate a fused feature map that satisfies both the intra-class similarity and inter-class difference requirements; The application of the S4 separation network includes at least the following steps: A real-time segmentation model Bilateral attention decoder is introduced to segment the fused image, and the segmentation network outputs the segmentation result and auxiliary segmentation result; The gap between the segmentation result and the semantic label reflects the richness of the semantic information contained in the fused image; This gap is exploited to construct semantic loss; Semantic loss is used to guide the training of the fusion network through back-propagation, forcing the fused image to contain more semantic information.

Images