An infrared light and visible light image fusion system based on shunt attention transformer

Abstract

The application discloses an infrared light and visible light image fusion system based on a shunt attention Transformer, and relates to the technical field of image fusion; the system comprises a shunt attention Transformer network model, is used for infrared light and visible light image fusion, and the overall network model comprises a shunt attention feature extraction unit, a cross attention fusion unit and a feature reconstruction unit. In order to generate a fusion image with rich scene details and good visual effects, infrared light and visible light images are respectively sent into the network model to extract shallow local features. Then, the feature extraction unit is used to acquire coarse-grained and fine-grained details in a single attention layer. In the feature fusion unit, a cross attention mechanism is introduced to fuse cross-domain complementary features. In addition, in the reconstruction stage, the feature reconstruction unit adopts dense skip connection, maximally utilizes deep and shallow features of different scales, and constructs a fusion image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image fusion, and particularly relates to an infrared light and visible light image fusion system based on a shunt attention Transformer. BACKGROUND

[0002] Due to the limitations of physical characteristics, imaging mechanisms and observation angles of sensors, a single visual sensor cannot extract enough information from a scene. Based on the theory of thermal radiation, infrared sensors highlight the heat source area without environmental interference. However, due to the low resolution of the images generated by infrared sensors, they usually lack structural features and detailed information. In contrast, visible sensors can generate friendly visual images with high spatial resolution. In order to inherit the advantages of the two sensors, it is an effective method to preserve the thermal radiation and texture information of infrared and visible light images through image fusion technology. The fusion result has superior visual perception and scene representation ability, and can be widely used in image enhancement, semantic segmentation, target detection and other fields.

[0003] The key to infrared light and visible light image fusion is how to effectively integrate heat source features and detailed texture information. In the past few decades, a large number of traditional fusion methods and deep learning-based fusion methods have been proposed.

[0004] Traditional fusion methods mainly consist of spatial domain and multi-scale transform domain methods. Spatial domain methods usually calculate the weighted average of the pixel-level saliency of input images to obtain the fused image. Multi-scale transform domain methods include wavelet transform and curvelet transform, etc. The input is transformed to the transform domain through mathematical transformation, and fusion rules are designed to fuse the images. Since the above traditional methods do not consider the feature differences between the source images, they often bring negative effects to the fused images. In addition, the fusion rules and activity level measurements of traditional methods cannot adapt to complex scenes, which also brings challenges to their wide application.

[0005] In recent years, deep learning has gradually become the mainstream in the field of image fusion due to its outstanding deep feature extraction capability. These methods not only can automatically extract deep features from data, but also overcome the problems faced by traditional methods when adapting to complex scenes. However, these methods can only utilize local information for image fusion and cannot utilize long-range dependencies to further improve the fusion results. Some fusion methods based on Transformer benefit from the complementary aggregation of global contextual features and have shown excellent performance, but they still have certain limitations. First, compared with convolutional neural network-based methods, Transformer-based methods are less efficient in generating test images. In addition, most Transformer-based methods usually process a series of tokens directly after segmenting the image, which will result in high memory consumption. Second, since existing Transformer fusion networks ignore the mixed-granularity features within layers, the ability to preserve fine-grained details and coarse-grained objects is limited. Finally, the feature fusion process only fuses information within a single domain, lacking contextual information, which can affect the visual performance of the fusion results. SUMMARY

[0006] The present application aims to provide an infrared and visible light image fusion system based on a split attention Transformer, which is used to extract and fuse global granularity information and local features, and can significantly reduce the computational cost by utilizing remote learning and reducing the number of input tokens.

[0007] To achieve the above-mentioned purpose, the present application provides an infrared and visible light image fusion system based on a split attention Transformer, which comprises a split attention Transformer network model for fusing infrared and visible light images. The input infrared and visible light images first enter a shallow convolutional block composed of multiple convolutional layers to extract shallow local features, and then a multi-scale technique is used to decompose the shallow local features into different scales to capture shallow information at different scales.

[0008] Further, the split attention Transformer network model comprises a split attention feature extraction unit, a cross attention fusion unit, and a feature reconstruction unit.

[0009] Further, the split attention feature extraction unit comprises three stage blocks, each stage block comprising six split Transformer sub-blocks, each sub-block being driven by multi-granularity learning and injecting tokens into heterogeneous receptive fields to capture multi-granularity information while reducing the number of tokens; the split Transformer sub-block models global features to extract multi-granularity features within a global range.

[0010] Further, the cross-attention fusion unit includes two cross-attention residual blocks, each of which is provided with a self-attention-based intra-domain fusion block to effectively integrate global interaction information in the same domain, and a cross-attention-based inter-domain fusion block to further integrate global interaction information between different domains.

[0011] Further, the inter-domain fusion block utilizes a cross-attention mechanism to realize interaction of global feature information, integrates cross-domain information, and uses a skip connection to retain information of different domains, thereby realizing alternative integration of global inter-domain and cross-domain interaction.

[0012] Further, the feature reconstruction unit includes a deep reconstruction block and a shallow reconstruction block for mapping the aggregated deep features back to the image space; the deep reconstruction block includes four self-attention blocks, and the shallow reconstruction block includes two convolutional layers with a kernel of 3x3 and a step of 1, each followed by a ReLU activation function.

[0013] Further, the deep reconstruction block is used to refine the fused deep features and realize multi-scale feature reconstruction from a global perspective; the shallow reconstruction block and the convolutional layer are used to further restore the image size, and then a skip connection is used to strengthen feature transmission, thereby maximizing the reuse of features of different layers to construct a fused image.

[0014] Further, the shunt attention Transformer network model refinement process is quantified as:

[0015]

[0016]

[0017]

[0018]

[0019] I Fu =FRU(F FUS )

[0020] wherein S(·) represents a shallow convolutional block, I n and V i represent input infrared and visible light images respectively; SAFEU(·) is a shunt attention feature extraction unit, and represent infrared and visible light deep granularity features respectively; CAFU(·) represents a cross-attention fusion unit, and respectively represent the aggregated infrared light and visible light output characteristics after the intra-domain and inter-domain interaction; Concat(·) represents concatenation in the channel dimension; FRU(·) represents a feature reconstruction unit; F FUS represents the fused deep feature; I Fu is the generated fusion image after reconstruction and up-sampling.

[0021] As a further further, the shunt attention Transformer network model is constrained by using a granularity loss function during training, and the granularity loss function includes a structural similarity loss, a fine-grained loss, and a coarse-grained loss; the granularity loss L G is represented as:

[0022] L G = αL S + β(L FG + L CG )

[0023] Wherein, L S , L FG and L CG represent the structural similarity loss, the fine-grained loss and the coarse-grained loss respectively; α, β are loss function hyperparameters.

[0024] As a further further, the structural similarity loss L S is:

[0025] L S = 1-SSIM(I n , V i , I Fu )

[0026] Wherein, I Fu is the fusion result, and the symbol SSIM(·) represents a structural similarity function, which is defined as:

[0027]

[0028] Wherein, I * represents the source image V i or I n ; μ and σ respectively represent the mean and standard deviation; C1, C2 and C3 are constants to maintain stability;

[0029] The fine-grained loss L FG and the coarse-grained loss L CG are respectively:

[0030]

[0031]

[0032] where ||·||1 denotes L1-norm, max{·} denotes element-wise maximum selection, denotes Sobel gradient operator, |·| represents absolute value operation; H, W are the height and width of the image, and gamma is a hyperparameter.

[0033] Compared with the prior art, the above technical scheme of the present application has the advantages that the shunt attention Transformer network model establishes long-range dependency between images, extracts and integrates granular features, and reduces the number of tokens to reduce the computational cost, and the time efficiency of generating test images is not only higher than that of the fusion method based on Transformer, but also has an advantage over the fusion method based on convolutional neural network. The shunt attention feature extraction unit is used to realize the joint extraction of coarse-grained and fine-grained features in each attention layer. The cross-attention fusion unit fully realizes the interaction of intra-domain and inter-domain deep features and cross-domain information fusion, and the feature reconstruction unit combines feature mapping and reconstructed images, so that the network model can restore fusion images of different resolutions. In addition, the granular loss function composed of structural similarity loss, fine-grained loss and coarse-grained loss is used to drive the network, and the granular information control and structure maintenance are used to realize feature extraction and fusion. The fusion image generated by the shunt attention Transformer network model has better visual perception, including sufficient significant features and texture detail information, and higher time efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 It is an infrared and visible light image fusion system principle block diagram;

[0035] Figure 2 It is a principle diagram of cross-attention fusion unit and reconstruction unit;

[0036] Figure 3 It is a qualitative comparison diagram between the method and other advanced fusion methods on the MSRS dataset;

[0037] Figure 4 It is a qualitative comparison diagram between the method and other advanced fusion methods on the M 3 FD dataset;

[0038] Figure 5 It is a quantitative comparison diagram between the method and other advanced fusion methods on the M 3 FD dataset;

[0039] Figure 6 It is a detection qualitative comparison diagram between the method and other advanced fusion methods on the MSRS dataset. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and are not used to limit the present application, that is, the described examples are only a part of the examples of the present application, but not all the examples.

[0041] Embodiment 1

[0042] As shown in the left upper corner, the split attention feature extraction unit is designed for better exploration of multi-granularity deep features, which includes three stage blocks, and each stage block includes six split Transformer sub-blocks, and each sub-block is driven by multi-granularity learning. Figure 1 In order to generate a fusion image with rich scene details and good visual effect, the infrared light and visible light images are respectively sent into the network model to obtain significant information and texture details. The input infrared light and visible light images first enter a shallow convolutional block composed of four convolutional layers and extract shallow local features. Then, the multi-scale technology is used to decompose the shallow local features to different scales to capture the shallow information at different scales, and the extracted different scale shallow information helps to better understand the image details.

[0043] In order to solve the problems of expensive memory consumption and only providing a single scale token in the self-attention mechanism, the present application introduces multi-granularity joint learning in the feature extraction stage to learn the granularity information within a single attention layer. As shown in the left upper corner, Figure 1 As shown in the left upper corner, the split attention feature extraction unit is designed for better exploration of multi-granularity deep features, which includes three stage blocks, and each stage block includes six split Transformer sub-blocks, and each sub-block is driven by multi-granularity learning.

[0044] The core of multi-granularity joint learning is the split attention mechanism, which models different scale objects within the same attention layer and learns multi-granularity information in parallel. The K and V values on the attention head within the same attention layer are down-sampled to different sizes to reduce the number of tokens, so as to realize the capture of coarse and fine-grained information and the merging of features. Then, the mixed granularity features are aggregated through the skip connection. This mechanism has good computational efficiency and fine-grained detail information retention capability.

[0045] After extracting deep features, the present application designs a cross-attention fusion unit to further mine and aggregate intra-domain and inter-domain context information, as shown in the left lower corner. Figure 2As shown. It includes two cross-attention residual blocks, and in each residual block, an intra-domain fusion block based on self-attention is designed to effectively integrate the global interaction information in the same domain, and an inter-domain fusion block based on cross-attention is designed to further integrate the global interaction information between different domains. The inter-domain fusion block utilizes the cross-attention mechanism to realize the exchange of global feature information; by merging the cross-domain information, the skip connection is used to retain the information of different domains, realizing the alternative integration of global inter-domain and cross-domain interaction.

[0046] After fully aggregating the complementary information in different domains, the feature reconstruction unit is used to map the aggregated deep features back to the image space. First, deploy a deep reconstruction block to refine the fused deep features, and realize the reconstruction of multi-scale features from a global perspective. Then, two shallow reconstruction blocks based on convolutional neural network and a convolutional layer are used to further restore the image size; the skip connection is used to strengthen the feature transmission, and the features of different layers are reused to the maximum extent to construct the fusion image. The deep reconstruction block includes four self-attention blocks, and the shallow reconstruction block includes two convolutional layers with a kernel size of 3x3 and a stride of 1, followed by a ReLU activation function.

[0047] The refinement process of the shunt attention Transformer network model can be quantified as:

[0048]

[0049]

[0050]

[0051]

[0052] I Fu =FRU(F FIS )

[0053] Where S(·) represents a shallow convolutional block, I n and V i represent the input infrared and visible light images respectively. SAFEU(·) is a shunt attention feature extraction unit, and represent the infrared and visible light deep granularity features respectively. CAFU(·) represents a cross-attention fusion unit, and represent the aggregated infrared and visible light output features after intra-domain and inter-domain interaction respectively. Concat(·) represents concatenation in the channel dimension; FRU(·) represents a feature reconstruction unit; F FUS represents the fused deep features. I Fu is the fusion image generated after reconstruction and up-sampling.

[0054] The purpose of image fusion is to integrate the detailed information of source images into a single fused image, and to generate a fused image with significant objects according to the intensity information of the source images. In order to pursue better feature learning ability, the present application adopts a granularity loss function to extend feature extraction and fusion to the control of coarse and fine-grained information and the maintenance of structural similarity. The granularity loss function is used to constrain the network model during training, and the function includes structural similarity loss, fine-grained loss and coarse-grained loss to realize that the fused image has similar structure and granularity information as the input image. The granularity loss function L G is expressed as:

[0055] L G = αL S + β(L FG + L CG )

[0056] wherein L S , L FG and L CG represent structural similarity loss, fine-grained loss and coarse-grained loss respectively.

[0057] The structural similarity loss L S combines three parts of brightness, structure and contrast, and it is an effective method for measuring the structural similarity between two different images, and it is defined as:

[0058] L S = 1-SSIM(I n , V i , I Fu )

[0059] wherein I Fu is the fusion result, and the symbol SSIM(·) represents the structural similarity function, which is defined as:

[0060]

[0061] wherein I * represents the source image V i or I n ; μ and σ represent the mean and standard deviation respectively; C1, C2 and C3 are constants to maintain stability.

[0062] The fine-grained loss L FG is used to guide the network to retain as much detailed features as possible, and the coarse-grained loss L CG aims to guide the network to capture appropriate target information; the fine-grained and coarse-grained losses are defined as:

[0063]

[0064]

[0065] where ||·||1 denotes the L1-norm, max{·} denotes the element-wise maximum selection, denotes the Sobel gradient operator, which can measure the texture details of the image; |·| represents the absolute value operation.

[0066] To demonstrate the superiority of the proposed system, first, the test image pairs are selected on the MSRS dataset, and compared with ten state-of-the-art infrared and visible image fusion methods, and then the M 3 FD and TNO datasets are selected for further verification; in the qualitative evaluation, the images are evaluated by the human visual system, such as image details, brightness and target integrity. As shown in Fig. Figures 3-4 compared with other methods, the proposed split attention Transformer network model obtains better visual perception effect in maintaining visible details and infrared targets. At the same time, it has better fusion performance in maintaining visible detail texture and infrared salient target distribution. The generated fusion image is more consistent with human visual perception, and the subjective visual effect is clear and natural.

[0067] In order to avoid the interference of human factors and comprehensively measure the fusion ability, the objective quantitative indicators MI, Q abf , VIF, AG, SF and SSIM are also used to quantitatively evaluate the performance of the fusion results. The test image pairs in the M 3 FD dataset are used as the test set to complete different infrared and visible image fusion tasks, and the quantitative results are shown in Fig. Figure 5 It is obvious that the quantitative experimental results obtained by the present application obtain the highest value of all image VIF indicators except the SSIM indicator. Since the VIF indicator is consistent with the human visual system, it proves that the network has better human visual effect. At the same time, the fusion image retains a large amount of information from the infrared and visible light source images. SF and AG can reflect the details and texture of the fusion image, respectively. Although the SSIM indicator of the proposed method is not the best, but the comparable results still mean that the fusion image obtained by the proposed method includes enough structure and gradient information.

[0068] In order to explore the influence of infrared and visible image fusion on multi-modal target detection, the network generated fusion image is used to evaluate the performance of target detection. YOLOv7 is used as the benchmark model for target detection. Figure 6The qualitative results of object detection are shown. The results show that the network generated fusion image has the best detection performance, especially in the person and car classes. By fusing the salient region features and texture information, a more comprehensive scene description is provided, and the detection accuracy is improved. In addition, the present application uses DeeplabV3+ as a benchmark model for training, and compares the effectiveness of the model by using the intersection over union. The results of semantic segmentation effectively integrate the global context granularity information, and the intra-domain and inter-domain complementary information also enhances the semantic features of the fusion image, improves the model perception ability and segmentation accuracy.

[0069] The model proposed in the present application has significant advantages in visual performance and objective evaluation. The generated fusion image has better visual perception, including sufficient salient features and texture detail information, and higher time efficiency. In target detection and semantic segmentation, it also shows the potential in high-level visual tasks. Therefore, the system proposed in the present application is helpful for the development of infrared and visible light image fusion.

[0070] The foregoing description of specific exemplary embodiments of the application is intended to be illustrative only and is not intended to limit the application to the precise forms described. Many modifications and variations are possible in light of the above teachings without departing from the spirit or essential characteristics of the present application. The exemplary embodiments are chosen and described in order to explain the principles of the application and its practical application to thereby enable others skilled in the art to best utilize the application and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the application be defined by the claims and their equivalents.

Claims

1. An infrared and visible light image fusion system based on a shunting attention Transformer, characterized in that, The application relates to a shunt attention Transformer network model for fusing infrared light and visible light images. The input infrared light and visible light images first enter a shallow convolution block composed of multiple convolution layers to extract shallow local features, and then the shallow local features are decomposed into different scales by using a multi-scale technology to capture shallow information at different scales. wherein, represents a shallow convolutional block, and represent inputted infrared and visible light images, respectively; is a split attention feature extraction unit, and represent infrared and visible light deep-level granularity features, respectively; represents a cross-attention fusion unit, and represent infrared and visible light output features aggregated after intra-domain and inter-domain interaction, respectively; represents concatenation in the channel dimension; represents a feature reconstruction unit; represents a fused deep feature; is a fused image generated after reconstruction and up-sampling; The shunt attention Transformer network model comprises a shunt attention feature extraction unit, a cross attention fusion unit and a feature reconstruction unit. The shunt attention feature extraction unit comprises three stage blocks, each stage block comprises six shunt Transformer sub-blocks, each sub-block is driven by multi-granularity learning, tokens are injected into a heterogeneous receptive field to capture multi-granularity information while reducing the number of tokens; the shunt Transformer sub-block models global features to extract multi-granularity features in a global range. The cross attention fusion unit comprises two cross attention residual blocks, each residual block is provided with an intra-domain fusion block based on self-attention to effectively integrate global interaction information in the same domain, and an inter-domain fusion block based on cross attention to further integrate global interaction information between different domains; The feature reconstruction unit comprises a deep reconstruction block and a shallow reconstruction block, which are used for mapping the aggregated deep features back to the image space; the deep reconstruction block comprises four self-attention blocks, and the shallow reconstruction block comprises two convolutional layers with a kernel of 1 and a step of 1, each followed by a ReLU activation function. and a step of 1, each followed by a ReLU activation function.

2. The infrared and visible image fusion system based on the shunt attention Transformer according to claim 1, wherein, The inter-domain fusion block utilizes a cross attention mechanism to realize the interaction of global feature information, merges cross-domain information, uses a skip connection to retain information of different domains, and realizes the alternative integration of global inter-domain and cross-domain interaction.

3. The infrared and visible image fusion system based on the shunt attention Transformer of claim 1, wherein, The deep reconstruction block is used for refining the fused deep features and realizing multi-scale feature reconstruction from a global perspective; the image size is further restored through the shallow reconstruction block and the convolution layer, then the skip connection is used to strengthen feature transmission, and the features of different layers are maximally reused to construct a fused image.

4. The infrared and visible image fusion system based on the shunt attention Transformer of claim 1, wherein, The shunt attention Transformer network model is constrained by using a granularity loss function during training, and the granularity loss function includes structural similarity loss, fine-grained loss and coarse-grained loss. is represented as: wherein, , and denote structural similarity loss, fine-grained loss and coarse-grained loss, respectively; , are loss function hyperparameters.

5. The infrared and visible image fusion system based on the shunt attention Transformer of claim 4, wherein, Structural similarity loss is: wherein is the fusion result, and denotes the structural similarity function, which is defined as: wherein representative source image or ; and denote the mean and standard deviation, respectively; , and are constants that maintain stability; Fine-grained loss Coarse-grained loss Respectively: wherein denotes -norm, denotes element-wise max selection, denotes Sobel gradient operator, denotes absolute value operation; H, W are height and width of the image, γ is a hyperparameter.

Images