Infrared and visible image fusion method based on second-order attention mixed features

Abstract

The application provides an infrared and visible light image fusion method based on second-order attention mixed features, first-order and second-order statistics of an infrared image and a visible light image are input, global features and local features of the infrared image and the visible light image are adaptively integrated, so that the fusion image obtains stronger feature expression, meanwhile, pixel intensity and texture details are retained, comparable contrast is maintained while more realistic edges are realized, the output result retains rich edge and texture information transmitted by the input image, has fewer artifacts, and better meets human visual perception; compared with other infrared and visible light image fusion algorithms based on deep learning, the application has better fusion effect.

Description

Technical Field

[0001] This invention belongs to the field of image fusion technology in information processing, and particularly relates to an infrared and visible light image fusion method based on second-order attention fusion features. Background Technology

[0002] Due to the limitations of human vision, single-modal images have inherent shortcomings in information representation: while visible light images can accurately capture detailed features such as texture and color of objects, they are easily affected by environmental factors such as lighting conditions and obstructions; while infrared images can penetrate darkness and smoke and clearly present the outline of targets through thermal radiation information, they lack effective representation of surface details. Image fusion technology, by collaboratively utilizing image information from multiple sensors in the same scene to generate a fused image, not only overcomes the limitations of a single image but also obtains a more accurate, reliable, and comprehensive scene description, facilitating further processing and analysis by human vision or computers. This demonstrates significant application value in fields such as security monitoring, intelligent transportation, and military reconnaissance.

[0003] Traditional infrared and visible light image fusion methods mainly include algorithms such as saliency detection, spatial transformation, multi-scale decomposition, and sparse representation. These methods typically achieve image information fusion based on the spatial or transform domain. Traditional methods suffer from limitations such as high complexity in fusion rule design and stringent application constraints, which to some extent hinder further technological breakthroughs. With the application of artificial intelligence technology, image fusion research is gradually developing towards deep learning-based fusion algorithms.

[0004] Current deep learning-based fusion algorithms mainly encompass three major systems: autoencoder-based frameworks, convolutional neural network-based feature extraction models, and generative adversarial network-based adversarial learning methods. Early autoencoder (AE) algorithms used pre-trained encoders and decoders on natural image datasets, fusing features using methods such as addition, maximum selection, or L1 norm. However, their downsampling operation easily leads to detail loss, and training on natural image datasets is difficult to effectively capture infrared image features. Although introducing short connections between the encoder and decoder improved the detail preservation problem, there is still room for optimization. Convolutional neural network (CNN)-based fusion methods achieve an end-to-end feature extraction, fusion, and reconstruction process. Carefully designed loss functions guide the fusion module to effectively integrate multimodal information. However, during feature extraction, the internal correlations of features are often ignored, leading to differences in structure between the fused image and the original image. In the field of Generative Adversarial Networks (GANs), adversarial game theory is introduced into image fusion. Through adversarial training between the generator and discriminator, feature extraction, fusion, and reconstruction are implicitly integrated, simultaneously preserving thermal radiation information and texture details. However, the single-discriminator architecture suffers from insufficient training stability. Although subsequent optimizations have employed a single-generator, dual-discriminator architecture and incorporated attention mechanisms to focus on key target regions, the fusion results remain unsatisfactory for high-resolution images or complex scenes. Overall, the following problems persist: First, it struggles to adapt to changes in complementary information between infrared and visible light images in complex scenes such as heavy fog or nighttime, exhibiting insufficient ability to collaboratively preserve thermal radiation information and texture details; second, during feature extraction, the internal correlations of features are ignored, leading to differences in structure between the fused image and the original image; and third, the deep relationships between features at different scales are not effectively utilized, resulting in distortion and artifacts in the fused image. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides an infrared and visible light image fusion method based on second-order attention fusion features, which can be used for infrared and visible light image fusion in complex environments such as low light, heavy fog, and nighttime.

[0006] An infrared and visible light image fusion method based on second-order attention fusion features includes the following steps:

[0007] The visible light image and the infrared image are respectively input into the feature extraction module for dual-scale feature extraction, resulting in high-resolution scale feature maps of the visible light image, low-resolution scale feature maps of the visible light image, high-resolution scale feature maps of the infrared image, and low-resolution scale feature maps of the infrared image.

[0008] The feature maps of the two scales corresponding to the visible light image and the two scales corresponding to the infrared image are combined in pairs, and then the four sets of feature maps are input into a two-stream second-order attention fusion module for feature fusion to obtain four fused feature maps.

[0009] The four fused feature maps are each input into a dynamic feature upsampling module for adaptive feature reconstruction, resulting in four sets of reconstructed images of the same scale.

[0010] The four reconstructed images are stitched together and then input into the convolution module to obtain the final fused image.

[0011] Furthermore, the feature extraction module includes two high-resolution scale dynamic feature downsampling units (AFEM) and two low-resolution scale dynamic feature downsampling units (AFEM).

[0012] The visible light image is divided into two branches. The first visible light image branch is processed by one of the high-resolution scale dynamic feature downsampling units (AFEM) to obtain a high-resolution scale feature map of visible light. The second visible light image branch is processed by another low-resolution scale dynamic feature downsampling unit (AFEM) to obtain a low-resolution scale feature map of visible light.

[0013] The infrared image is divided into two branches. The first infrared image branch is processed by another high-resolution scale dynamic feature downsampling unit (AFEM) to obtain a high-resolution scale feature map of the infrared image. The second infrared image branch is processed by another low-resolution scale dynamic feature downsampling unit (AFEM) to obtain a low-resolution scale feature map of the infrared image.

[0014] Furthermore, any dynamic feature downsampling unit AFEM includes two 1×1 convolutional layers, an activation function, a depthwise separable convolution DSConv, and a max pooling layer;

[0015] The input branch image of the current dynamic feature downsampling unit is divided into two sub-paths. The first sub-path is processed sequentially through two 1×1 convolutional layers and an activation function to obtain the first sub-path processed image. Simultaneously, the first sub-path processed image is multiplied element-wise with the feature map weights output by the first 1×1 convolutional layer to obtain the first sub-path product image. The other sub-path is processed sequentially through a depthwise separable convolution DSConv and a max pooling layer to obtain the second sub-path processed image. The first sub-path product image and the second sub-path processed image are summed element-wise to obtain the final output of the current dynamic feature downsampling unit.

[0016] Furthermore, the depthwise separable convolution DSConv includes sequentially cascaded 3×3 depthwise convolutional layers, 1×1 pointwise convolutional layers, and a nonlinear activation function; wherein, the depthwise separable convolution DSConv processes the input feature F as follows:

[0017]

[0018] in, This represents a 3×3 depth convolutional layer. represents a 1×1 pointwise convolutional layer, ReLU(·) represents a non-linear activation function, and DSConv(F) is the final output of the depthwise separable convolution DSConv.

[0019] Furthermore, any two-stream second-order attention fusion module includes a first-order channel attention unit, a second-order channel attention unit (SCAM), a first-order spatial attention unit, a second-order spatial attention unit (SSAM), two upsampling units, and two PReLU activation functions, wherein an upsampling unit and a PReLU activation function are combined to form a sampling activation unit.

[0020] The combined feature map input to the current two-stream second-order attention fusion module is re-divided into a visible light feature map and an infrared feature map; the visible light feature map is divided into two branches and enters a sampling activation unit respectively, and the infrared feature map is also divided into two branches and enters a sampling activation unit respectively;

[0021] The infrared feature map branch and the visible light feature map branch that have passed through one of the sampling activation units are respectively subjected to the first branch weight acquisition operation to obtain the infrared image channel attention branch weight and the visible light image channel attention branch weight. The first branch weight acquisition operation is specifically as follows: the current feature map branch is divided into two sub-paths. The first sub-path is input into the first-order channel attention unit for feature extraction to obtain the first channel sub-path features. The second sub-path is input into the second-order channel attention unit SCAM for feature extraction to obtain the second channel sub-path features. The first channel sub-path features and the second channel sub-path features are multiplied element-wise to obtain the channel attention branch weight.

[0022] The infrared feature map branch and the visible light feature map branch, which have passed through another sampling activation unit, are respectively subjected to the second branch weight acquisition operation to obtain the infrared image spatial attention branch weight and the visible light image spatial attention branch weight. The second branch weight acquisition operation is specifically as follows: the current feature map branch is divided into two sub-paths. The first sub-path is input into the first-order spatial attention unit for feature extraction to obtain the first spatial sub-path features. The second sub-path is input into the second-order spatial attention unit SSAM for feature extraction to obtain the second spatial sub-path features. The first spatial sub-path features and the second spatial sub-path features are multiplied element-wise to obtain the spatial attention branch weight.

[0023] The infrared image features are obtained by multiplying the infrared image channel attention branch weights, the infrared image spatial attention branch weights, and the infrared feature map; the visible light image features are obtained by multiplying the visible light image channel attention branch weights, the visible light image spatial attention branch weights, and the visible light feature map.

[0024] The infrared image features and the visible light image features are added element by element, and the result is the final output of the two-stream second-order attention fusion module.

[0025] Furthermore, the process of feature extraction by the first-order channel attention unit is as follows:

[0026]

[0027] The process of feature extraction by the second-order channel attention unit SCAM is as follows:

[0028]

[0029] The process of obtaining the channel attention branch weights is as follows:

[0030] φ=φ (1) ☉φ (2)

[0031] Where F represents the sub-path features of the input first-order channel attention unit and the second-order channel attention unit SCAM, and AvgPool(·) represents global average pooling. Let PReLU be the dimension reduction matrix, and PReLU(·) be the activation function. Let σ be an upgraded matrix, φ be the sigmoid activation function, and φ be the sigmoid activation function. (1) Let Cov(F) be the covariance matrix of the first channel subpath, and μ be the characteristic of the first channel subpath. φ =AvgPool(F), where H and W are the height and width of the sum sub-path, respectively, F:ij is the pixel value in the i-th row and j-th column of F, T represents the transpose, vec(·) represents the vectorization matrix, and φ (2) φ represents the feature of the second channel sub-path, φ represents the channel attention branch weight, and ⊙ represents element-wise multiplication.

[0032] Furthermore, the process of feature extraction by the first-order spatial attention unit is as follows:

[0033]

[0034] The process of feature extraction by the second-order spatial attention unit (SSAM) is as follows:

[0035]

[0036] The process of obtaining the spatial attention branch weights is as follows:

[0037]

[0038] Where F represents the subpath features of the input first-order spatial attention unit and second-order spatial attention unit SSAM, and Conv 3×3 (·) represents a 3×3 convolution, σ is the sigmoid activation function, and Conv′ 3×3 (·) represents another 3×3 convolution, and ⊙ represents element-wise multiplication. Features of the first spatial sub-path Features of the second spatial sub-path For spatial attention branch weights.

[0039] Furthermore, any dynamic feature upsampling unit AIRM includes three depthwise separable convolutions DSConv, two nearest neighbor upsampling units, a 1×1 convolutional layer, and an activation function;

[0040] The fused feature map of the current dynamic feature upsampling unit AIRM is divided into two sub-paths. The first sub-path is processed sequentially through the first depthwise separable convolution DSConv, the first nearest neighbor upsampling unit, a 1×1 convolutional layer, and an activation function to obtain the first sub-path processed image. Simultaneously, the first sub-path processed image is multiplied element-wise with the feature image output by the first nearest neighbor upsampling unit to obtain the first sub-path product image. The other sub-path is processed sequentially through the second depthwise separable convolution DSConv, the second nearest neighbor upsampling unit, and the third depthwise separable convolution DSConv to obtain the second sub-path processed image. The first sub-path product image and the second sub-path processed image are summed element-wise to obtain the final output of the current dynamic feature upsampling unit AIRM.

[0041] Furthermore, the loss function used to train the fusion model, which consists of a feature extraction module, four two-stream second-order attention fusion modules, four dynamic feature upsampling modules, and a convolutional module, is as follows:

[0042]

[0043] in, For the total loss, For pixel loss, For feature loss, For style loss, The loss is denoted as α, β, and γ, which are balance coefficients.

[0044] Beneficial effects:

[0045] 1. This invention provides an infrared and visible light image fusion method based on second-order attention-based hybrid features. By inputting first-order and second-order statistics of infrared and visible light images, it adaptively integrates global and local features of the infrared and visible light images, resulting in a fused image with more powerful feature representation. It simultaneously preserves pixel intensity and texture details, achieving more realistic edges while maintaining comparable contrast. The output retains the rich edge and texture information transmitted by the input image, with fewer artifacts, better satisfying human visual perception. Compared with other deep learning-based infrared and visible light image fusion algorithms, this invention has better fusion performance.

[0046] 2. This invention provides an infrared and visible light image fusion method based on second-order attention hybrid features. It decomposes standard convolution into depthwise convolution and pointwise convolution, reducing the number of parameters and model complexity while maintaining feature extraction capability, laying a solid foundation for the development of subsequent dynamic feature processing modules. Based on depthwise separable convolution, a dynamic feature downsampling unit is constructed to capture image features at different scales. By introducing an adaptive adjustment mechanism, the fusion ratio of features at each scale is dynamically adjusted to achieve efficient adaptive feature extraction.

[0047] 3. This invention provides an infrared and visible light image fusion method based on second-order attention fusion features. First, a second-order channel attention unit is constructed. This unit focuses on mining the second-order dependencies between channels. By analyzing the intrinsic connections between different channel features, it identifies channels that are more important for expressing key image information and assigns them higher weights, highlighting key channel information and preparing for the construction of a two-stream second-order attention fusion module. Second, a second-order spatial attention unit is constructed. This unit focuses on modeling the second-order relationships between spatial locations in the image. By calculating the similarity between different locations in space, it captures spatial context information of the image, clarifies the importance of each region in the image, and provides spatial dimension feature processing capabilities for the construction of the two-stream second-order attention fusion module. Finally, the constructed second-order channel attention unit and second-order spatial attention unit are combined to form a two-stream second-order attention fusion module, strengthening the feature interaction between channel feature maps and spatial feature maps, and adaptively performing feature fusion.

[0048] 4. This invention provides an infrared and visible light image fusion method based on second-order attention fusion features. It uses depthwise separable convolution to construct a dynamic feature upsampling module, and develops a dynamic feature upsampling module to adaptively reconstruct low-resolution feature maps into high-resolution images, thereby achieving high-quality conversion of fused features into the final image.

[0049] 5. This invention provides an infrared and visible light image fusion method based on second-order attention fusion features. It designs a loss function composed of a weighted combination of pixel loss, feature loss, style loss and block loss, which comprehensively considers the preservation of thermal radiation of infrared images and detailed texture information of visible light images, and has a better fusion effect. Attached Figure Description

[0050] Figure 1 The overall network framework provided by this invention;

[0051] Figure 2 A schematic diagram of depth-separable convolution provided by the present invention;

[0052] Figure 3 A schematic diagram of the dynamic feature downsampling unit provided by the present invention;

[0053] Figure 4 A schematic diagram of the channel attention branch weight acquisition process provided by the present invention;

[0054] Figure 5 A schematic diagram illustrating the spatial attention branch weight acquisition process provided by this invention;

[0055] Figure 6 This is a schematic diagram of the two-stream second-order attention fusion module provided by the present invention;

[0056] Figure 7 A schematic diagram of the dynamic feature upsampling unit provided by the present invention;

[0057] Figure 8 In the first set of source images of the present invention, (a) and (b) are infrared and visible light images, respectively; (c) and (d) are infrared and visible light images, respectively; and (e) and (f) are infrared and visible light images, respectively.

[0058] Figure 9 In the figures (a)-(d), the fused images generated by fusing the first set of source images using the present invention, Densefuse (Densefuse: A fusion approach to infrared and visible images), DRF (DRF: Disentangled representation for visible and infrared image fusion), and BTSFusion (BTSFusion: Fusion of infrared and visible image via a mechanism of balancing texture and salience) are respectively.

[0059] Figure 10 In the figures (a)-(d), the fused images generated by fusing the second set of source images using the present invention, Densefuse (Densefuse: A fusion approach to infrared and visible images), DRF (DRF: Disentangled representation for visible and infrared image fusion), and BTSFusion (BTSFusion: Fusion of infrared and visible image via a mechanism of balancing texture and salience) are respectively.

[0060] Figure 11 In the figures (a)-(d), the images generated by fusing the third set of source images are respectively the images generated by the present invention, Densefuse (Densefuse: A fusion approach to infrared and visible images), DRF (DRF: Disentangled representation for visible and infrared image fusion), and BTSFusion (BTSFusion: Fusion of infrared and visible image via a mechanism of balancing texture and salience). Detailed Implementation

[0061] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0062] Hardware environment: Computer

[0063] Software environment: Linux system; Python language environment software.

[0064] This invention addresses the problem of infrared and visible light image fusion by proposing a method based on second-order attention fusion features. The method comprises three main parts: an adaptive residual downsampling module that uses learnable residuals to extract detailed and semantically rich dual-layer features; a two-stream attention fusion module based on second-order spatial and channel-based features that aggregates deep features from the same and adjacent scales; and an adaptive residual reconstruction module that utilizes multi-scale fusion features for image reconstruction. The overall network framework of this method is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0065] The visible light image and the infrared image are respectively input into the feature extraction module for dual-scale feature extraction, resulting in high-resolution scale feature maps of the visible light image, low-resolution scale feature maps of the visible light image, high-resolution scale feature maps of the infrared image, and low-resolution scale feature maps of the infrared image.

[0066] The feature maps of the two scales corresponding to the visible light image and the two scales corresponding to the infrared image are combined in pairs, and then the four sets of feature maps are input into a two-stream second-order attention fusion module for feature fusion to obtain four fused feature maps.

[0067] The four fused feature maps are each input into a dynamic feature upsampling module for adaptive feature reconstruction, resulting in four sets of reconstructed images of the same scale.

[0068] The four reconstructed images are stitched together and then input into the convolution module to obtain the final fused image.

[0069] Furthermore, the feature extraction module includes two high-resolution scale dynamic feature downsampling units (AFEM) and two low-resolution scale dynamic feature downsampling units (AFEM).

[0070] The visible light image is divided into two branches. The first visible light image branch is processed by one of the high-resolution scale dynamic feature downsampling units (AFEM) to obtain a high-resolution scale feature map of visible light. The second visible light image branch is processed by another low-resolution scale dynamic feature downsampling unit (AFEM) to obtain a low-resolution scale feature map of visible light.

[0071] The infrared image is divided into two branches. The first infrared image branch is processed by another high-resolution scale dynamic feature downsampling unit (AFEM) to obtain a high-resolution scale feature map of the infrared image. The second infrared image branch is processed by another low-resolution scale dynamic feature downsampling unit (AFEM) to obtain a low-resolution scale feature map of the infrared image.

[0072] like Figure 3As shown, any dynamic feature downsampling unit AFEM includes two 1×1 convolutional layers, an activation function, a depthwise separable convolution DSConv, and a max pooling layer;

[0073] The input branch image of the current dynamic feature downsampling unit is divided into two sub-paths. The first sub-path is processed sequentially through two 1×1 convolutional layers and an activation function to obtain the first sub-path processed image. Simultaneously, the first sub-path processed image is multiplied element-wise with the feature map weights output by the first 1×1 convolutional layer to obtain the first sub-path product image. The other sub-path is processed sequentially through a depthwise separable convolution DSConv and a max pooling layer to obtain the second sub-path processed image. The first sub-path product image and the second sub-path processed image are summed element-wise to obtain the final output of the current dynamic feature downsampling unit.

[0074] Therefore, the Dynamic Feature Subsampling Unit (AFEM) achieves hierarchical feature extraction and reconstruction through depthwise separable convolutions and adaptive residual connections. AFEM is used for feature subsampling and consists of a main path and a learnable residual path. The main path first extracts features through DSConv, then reduces the spatial resolution through a max pooling layer. The residual path generates residuals by adjusting the spatial dimensions of the input features through convolution, and then generates learnable weights through backpropagation using convolutions and a sigmoid activation function, dynamically adjusting the residuals. The main path features and the learnable residuals are added together to adaptively obtain the sampled features. The calculation process is as follows:

[0075] X doum =MaxPool(DSConv(F))

[0076] R d =Conv 1×1 (F)

[0077] w d =σ(Conv 1×1 (R d ))

[0078]

[0079] In the formula, F represents the input features, DSConv(·) is a depthwise separable convolution, MaxPool(·) is a max pooling layer, and Conv... 1×1 (·) represents a 1×1 convolution, σ is the sigmoid activation function, and ⊙ represents element-wise multiplication.

[0080] like Figure 2 As shown, the depthwise separable convolution DSConv includes sequentially cascaded 3×3 depthwise convolutional layers, 1×1 pointwise convolutional layers, and a nonlinear activation function; the processing procedure of the depthwise separable convolution DSConv on the input feature F is as follows:

[0081]

[0082] in, This represents a 3×3 depth convolutional layer. represents a 1×1 pointwise convolutional layer, ReLU(·) represents a non-linear activation function, and DSConv(F) is the final output of the depthwise separable convolution DSConv.

[0083] Therefore, Depthwise Separable Convolution (DSConv) decomposes standard convolution into two steps: depthwise convolution and pointwise convolution. It independently applies 3×3 convolution to each channel of the input feature to capture spatial patterns, and fuses spatial information across channels through 1×1 convolution. This design reduces computational complexity and improves computational efficiency while maintaining feature expressiveness.

[0084] It should be noted that the core of the framework of this invention is a dual-stream second-order attention fusion module. It mines the correlation between infrared and visible light image features through a second-order channel attention mechanism and a second-order spatial attention mechanism. The first-order attention focuses on the importance of different positions, while the second-order attention further captures the correlation between each position, thereby strengthening the feature interaction between the channel feature map and the spatial feature map and adaptively carrying out feature fusion.

[0085] Specifically, such as Figure 6 As shown, any two-stream second-order attention fusion module includes a first-order channel attention unit, a second-order channel attention unit (SCAM), a first-order spatial attention unit, a second-order spatial attention unit (SSAM), two upsampling units, and two PReLU activation functions. Here, an upsampling unit and a PReLU activation function are combined to form a sampling activation unit.

[0086] The combined feature map input to the current two-stream second-order attention fusion module is re-divided into a visible light feature map and an infrared feature map; the visible light feature map is divided into two branches and enters a sampling activation unit respectively, and the infrared feature map is also divided into two branches and enters a sampling activation unit respectively;

[0087] The infrared feature map branch and the visible light feature map branch that have passed through one of the sampling activation units are respectively subjected to the first branch weight acquisition operation to obtain the infrared image channel attention branch weight and the visible light image channel attention branch weight. The first branch weight acquisition operation is specifically as follows: the current feature map branch is divided into two sub-paths. The first sub-path is input into the first-order channel attention unit for feature extraction to obtain the first channel sub-path features. The second sub-path is input into the second-order channel attention unit SCAM for feature extraction to obtain the second channel sub-path features. The first channel sub-path features and the second channel sub-path features are multiplied element-wise to obtain the channel attention branch weight.

[0088] The infrared feature map branch and the visible light feature map branch, which have passed through another sampling activation unit, are respectively subjected to the second branch weight acquisition operation to obtain the infrared image spatial attention branch weight and the visible light image spatial attention branch weight. The second branch weight acquisition operation is specifically as follows: the current feature map branch is divided into two sub-paths. The first sub-path is input into the first-order spatial attention unit for feature extraction to obtain the first spatial sub-path features. The second sub-path is input into the second-order spatial attention unit SSAM for feature extraction to obtain the second spatial sub-path features. The first spatial sub-path features and the second spatial sub-path features are multiplied element-wise to obtain the spatial attention branch weight.

[0089] The infrared image features are obtained by multiplying the infrared image channel attention branch weights, the infrared image spatial attention branch weights, and the infrared feature map; the visible light image features are obtained by multiplying the visible light image channel attention branch weights, the visible light image spatial attention branch weights, and the visible light feature map.

[0090] The infrared image features and the visible light image features are added element by element, and the result is the final output of the two-stream second-order attention fusion module.

[0091] Therefore, it can be seen that the dual-stream second-order attention fusion module uses second-order channels and spatial attention to weight infrared and visible light features, and the contribution of learnable scalar weighted balanced modes is expressed by the following formula:

[0092]

[0093] Where ψ m =σ(θ) m ), m∈{ir,vis} are learnable scalars, These are used as model parameters to ensure that the contributions of infrared and visible light can be dynamically adjusted.

[0094] Specifically, such as Figure 4 As shown, the process of feature extraction by the first-order channel attention unit is as follows:

[0095]

[0096] The process of feature extraction by the second-order channel attention unit SCAM is as follows:

[0097]

[0098] The process of obtaining the channel attention branch weights is as follows:

[0099] φ=φ (1) ☉φ (2)

[0100] Where F represents the sub-path features of the input first-order channel attention unit and the second-order channel attention unit SCAM, and AvgPool(·) represents global average pooling. Let PReLU be the dimension reduction matrix, and PReLU(·) be the activation function. Let σ be an upgraded matrix, φ be the sigmoid activation function, and φ be the sigmoid activation function. (1) Let Cov(F) be the covariance matrix of the first channel subpath, and μ be the characteristic of the first channel subpath. φ =AvgPool(F), where H and W are the height and width of the sum sub-path, respectively, F:ij is the pixel value in the i-th row and j-th column of F, T represents the transpose, vec(·) represents the vectorization matrix, and φ (2) φ represents the feature of the second channel sub-path, φ represents the channel attention branch weight, and ⊙ represents element-wise multiplication.

[0101] Therefore, this invention captures the inter-channel dependencies through the statistical information of first-order and second-order channel attention units. The first-order channel attention unit performs global average pooling on the feature F, compresses the spatial dimension, activates it with PReLU, and then generates the first channel sub-path feature φ through dimensionality increase and sigmoid activation function. (1) The second-order channel attention unit reflects the correlation of channels by calculating the covariance matrix of sub-path F. The quantized covariance matrix is ​​then reduced in dimensionality to compress the spatial dimension, followed by PReLU activation and dimensionality-increasing convolution to ensure spatial alignment. Finally, the sigmoid activation function generates the second-channel sub-path feature φ. (2) Fusion attention is to integrate the features φ of the first channel subpath. (1) With the second channel sub-path characteristic φ (2) Element-wise multiplication enhances the robustness of channel selection.

[0102] Specifically, such as Figure 5 As shown, the process of feature extraction by a first-order spatial attention unit is as follows:

[0103]

[0104] The process of feature extraction by the second-order spatial attention unit (SSAM) is as follows:

[0105]

[0106] The process of obtaining the spatial attention branch weights is as follows:

[0107]

[0108] Where F represents the subpath features of the input first-order spatial attention unit and second-order spatial attention unit SSAM, and Conv 3×3 (·) represents a 3×3 convolution, σ is the sigmoid activation function, and Conv′ 3×3 (·) represents another 3×3 convolution, and ⊙ represents element-wise multiplication. Features of the first spatial sub-path Features of the second spatial sub-path For spatial attention branch weights.

[0109] It should be noted that the process of obtaining the channel attention branch weights focuses on the "salient channels" by modeling the feature covariance matrix, and its computational complexity is O(Ci). 2 If the process of obtaining the spatial attention branch weights is also implemented using a similar method, its computational complexity is O(C(HW)). 2 The computational cost is enormous. Here, we consider using local convolution to directly model the importance of spatial location. First-order spatial attention focuses on "salient regions" such as the target outline, while second-order spatial attention uses local convolution to model the complex relationships of the spatial neighborhood and capture "structural features" such as texture patterns and geometric shapes.

[0110] like Figure 5 As shown, the first-order spatial attention unit directly weights spatial location, normalizing the weights through convolution and the sigmoid activation function to enhance the response of the target region and suppress background noise. The second-order spatial attention unit is similar to the first-order unit, used to refine spatial structure and distinguish similar regions that are difficult for the first-order attention unit to discern, such as different textures of similar objects. Element-wise multiplication combines the "regional saliency" of the first-order attention unit with the "structural details" of the second-order attention unit. Although the first-order and second-order spatial attention units have the same structure, they belong to two independent branches of the neural network. Their convolutional kernel weights and biases are not shared, thus learning different features through different training processes. This allows them to capture regional saliency and structural details respectively, ultimately achieving accurate spatial modeling of complex scenes through fusion.

[0111] Furthermore, such as Figure 7As shown, any dynamic feature upsampling unit AIRM includes three depthwise separable convolutions DSConv, two nearest neighbor upsampling units, a 1×1 convolutional layer, and an activation function;

[0112] The fused feature map of the current dynamic feature upsampling unit AIRM is divided into two sub-paths. The first sub-path is processed sequentially through the first depthwise separable convolution DSConv, the first nearest neighbor upsampling unit, a 1×1 convolutional layer, and an activation function to obtain the first sub-path processed image. Simultaneously, the first sub-path processed image is multiplied element-wise with the feature image output by the first nearest neighbor upsampling unit to obtain the first sub-path product image. The other sub-path is processed sequentially through the second depthwise separable convolution DSConv, the second nearest neighbor upsampling unit, and the third depthwise separable convolution DSConv to obtain the second sub-path processed image. The first sub-path product image and the second sub-path processed image are summed element-wise to obtain the final output of the current dynamic feature upsampling unit AIRM.

[0113] It should be noted that the Dynamic Feature Upsampling Unit (AIRM) achieves hierarchical feature extraction and reconstruction through depthwise separable convolutions and adaptive residual connections. DFUM is used for feature upsampling in feature reconstruction. Its structure is symmetrical to AFEM. The main path extracts the input features via DSConv, restores the spatial resolution through nearest-neighbor upsampling, and then restores the number of output channels via DSConv again. The residual path adjusts the number of channels of the input features via DSConv, generates residuals through upsampling, and then generates learnable weights through convolution and sigmoid activation functions via the backpropagation path, which dynamically adjust the residuals. The main path features and the learnable residuals are added together to adaptively obtain the sampled features. The calculation process is as follows:

[0114] X up =DSConv(Upsample(DSConv(F)))

[0115] R u =Upsample(DSConv(F))

[0116] w u =σ(Conv 1×1 (R u ))

[0117] F u =X up +w u ☉R u

[0118] Here, Upsample(·) represents upsampling, and this mechanism uses adaptive weights w d w uBalance the sampling characteristics of the main path with the detailed information of the residual path to avoid high-frequency loss caused by sampling.

[0119] Furthermore, in order to simultaneously preserve the thermal radiation of the infrared image and the detailed texture information of the visible light image, this invention designs a multi-scale adaptive loss function, which is based on pixel loss. Feature loss Style loss Block loss It is a weighted combination. The total loss function is defined as:

[0120]

[0121] in, For the total loss, For pixel loss, For feature loss, For style loss, The loss is denoted as α, β, and γ, which are balance coefficients.

[0122] Pixel loss adaptively balances pixel similarity between infrared and visible light images through intensity weights and gradient weights, and includes image-level MS-SSIM loss and MSE loss. Its loss function is defined as follows: In the formula, ξ represents the weight coefficient of the MSE loss. MS-SSIM is used to measure the multi-scale structural similarity between the fused image and the source image. To highlight the importance of detailed regions, gradient-based adaptive weights are used, defined as:

[0123]

[0124] In the formula I f To fuse images, I m The input images are infrared and visible light; where m∈{ir,vis}, the same below; MS-SSIM(·) is a multi-scale structural similarity measurement function; The gradient weights are obtained by normalizing the gradient norm of the first two VGG-16 features.

[0125]

[0126] In the formula Φ i (·) represents the i-th layer feature of VGG-16, c detail `\u` is the normalization constant, and `softmax(·)` is the softmax function. The MSE loss is used to constrain the pixel intensity consistency between the fused image and the source image, employing intensity weights based on entropy and standard deviation, defined as:

[0127]

[0128] in, The intensity weights are obtained by normalizing the entropy en(·) and standard deviation std(·) of the features from the first two VGG-16 layers, i.e.

[0129]

[0130] In the formula, δ is the balance coefficient between entropy and standard deviation, and c int This is the normalization constant.

[0131] Feature loss, defined by constraining the fusion of images through deep feature extraction to ensure that the gradient is consistent with visible light features and the intensity is consistent with infrared features, is as follows:

[0132]

[0133] in: These represent the feature gradients of the fused, visible light, and infrared images at layer t, respectively. In this experiment, there are four feature layers, therefore T = 4, and λ... t η is the scale weight, and η is the strength constraint weight.

[0134] Style loss is achieved by matching the Gram matrix of shallow features from the VGG-16 network to preserve the pixel distribution characteristics of the infrared image, and is defined as follows:

[0135]

[0136] in is the Gram matrix; KL(·) is the KL divergence.

[0137] Block loss enhances the constraints of the detail region through local window statistics, and includes local MS-SSIM loss and MSE loss, defined as:

[0138]

[0139] in, and These are the SSIM loss and MSE loss within window W, respectively, and the local weights. It is calculated in the same way as the global weight, but based on local statistics within the window.

[0140] The present invention simulates the entire process in a Python environment on a Linux operating system. 37 sets of images from the TNO dataset are randomly selected to test the effectiveness of the method of the present invention, and three sets of images from the dataset are displayed.

[0141] The first set of images is Figure 8Images (a)-(b) are a set of images composed of people, houses, cars, and the sky. The infrared image contains the positions and outlines of people, houses, and vehicles, while the visible light image effectively represents the gradations in the sky and the outlines of house windows. The fused image obtained by the method of this invention is shown in 8(a). The fusion result shows that the details of people, houses, and cars are well preserved, and the sky retains more rich details than in the visible light image.

[0142] The second set of images is Figure 8 (c)-(d) are a set of helicopter images. In the infrared image, the overall outline of the helicopter is clear, while in the light-bearing image, the helicopter has rich detailed texture information, showing the texture layers of the fixed boarding footboard and tail. The fused image obtained by the method of this invention is shown in 9(a). The fusion result shows that the overall shape and details of the helicopter in the image are well preserved, with both a clear outline of the helicopter and details such as the fixed boarding footboard, tail, and windows.

[0143] The third set of images is Figure 8 Images (e)-(f) are a set of images showing pedestrians walking in a forest. The source images show significant differences in imaging, primarily in that the infrared image contains clear information about the pedestrians' positions and outlines, while the visible light image displays the forest background, road outlines, and other features. The fused image obtained by the method of this invention is shown in 10(a). The fused image demonstrates rich detail, prominent pedestrian targets, and combines the characteristics of both infrared and visible light images.

[0144] This invention is compared with other technologies, namely Densefuse (Densefuse: Afusion approach to infrared and visible images), DRF (DRF: Disentangled representation for visible and infrared image fusion), and BTSFusion (BTSFusion: Fusion of infrared and visible image via a mechanism of balancing texture and salience). The result obtained by fusing the first set of images using each method is as follows: Figure 8 (b)-(d), the result obtained by fusing the second set of images is Figure 9 (b)-(d), the result obtained by fusing the third set of images is Figure 10 (b)-(d).

[0145] Qualitatively, the fusion results of this invention outperform other techniques in terms of image saliency, detail, and texture. Quantitatively, five evaluation metrics were used to assess the algorithm's performance: entropy (EN), standard deviation (SD), correlation coefficient (CC), sum of differential correlations (SCD), and multi-scale structural similarity (MS-SSIM). EN measures the amount of information contained in the image based on information theory; SD reflects the contrast of the fused image; SCD and CC calculate the correlation between the source and fusion results; and MS-SSIM evaluates the structural consistency between the fused and source images from a multi-scale perspective. Higher values ​​for these five metrics indicate a higher amount of information contained in the fused image, a higher similarity to the original image, and a better fusion effect. The quantitative comparison results of the fused images obtained by different methods are shown in Table 1.

[0146] As shown in Table 1, this method achieved the highest average values ​​across five evaluation metrics—EN, SD, CC, SCD, and MS-SSIM—on the TNO dataset. These five statistical values ​​indicate that the method simultaneously preserves pixel intensity in infrared images and texture details in visible light images, thus better meeting human visual perception requirements.

[0147] Table 1. Mean values ​​of objective evaluation indicators for the TNO dataset comparison experiment.

[0148]

[0149] As can be seen, the infrared and visible light image fusion method of the present invention can better fuse two source images. Compared with the other three methods, it has a better fusion effect, more prominent image targets, fewer artifacts, and is more conducive to subsequent image applications.

[0150] In summary, infrared and visible light image fusion aims to generate a fused image containing intensity and detail information from the source images. A key challenge is effectively integrating the complementary information from multimodal images of the same scene. With the development of artificial intelligence, deep learning-based infrared and visible light image fusion algorithms have become a research hotspot in this field. However, most existing intelligent methods tend to design deeper or wider networks, neglecting the internal correlations of feature maps. To address this issue, this paper proposes a hybrid feature and two-stream second-order attention fusion method. As can be seen from the detailed steps of this invention, the proposed infrared and visible light image fusion method based on hybrid feature detail compensation using two-stream second-order attention adaptively integrates global and local features of infrared and visible light images through first- and second-order statistics of features, resulting in a more powerful feature representation in the fused image. This method uses a learnable downsampling module to decompose the source image into features at different scales; then, based on the two-stream second-order attention fusion module, it integrates channel and spatial feature information, applying second-order spatial and channel attention maps for feature fusion on features at the same and cross-scale; finally, it reconstructs the output image through adaptive residual reconstruction blocks. The loss function consists of image loss, feature loss, style loss, and block loss. Qualitative and quantitative experimental results on the TNO dataset show that the proposed fusion method can preserve the detailed information of both modalities and achieves better results compared with other state-of-the-art methods.

[0151] Meanwhile, the comparative experiments of this invention also show that, in qualitative evaluation, the algorithm proposed in this invention preserves both pixel intensity and texture details. The result achieves more realistic edges while maintaining comparable contrast. The output results retain the rich edge and texture information transmitted by the input image, with fewer artifacts, and better meet human visual perception. In quantitative evaluation, the average values ​​of the five comparative indicators are better.

[0152] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A method for fusing infrared and visible light images based on second-order attention fusion features, characterized in that, Includes the following steps: The visible light image and the infrared image are respectively input into the feature extraction module for dual-scale feature extraction, resulting in high-resolution scale feature maps of the visible light image, low-resolution scale feature maps of the visible light image, high-resolution scale feature maps of the infrared image, and low-resolution scale feature maps of the infrared image. The feature maps of the two scales corresponding to the visible light image and the two scales corresponding to the infrared image are combined in pairs, and then the four sets of feature maps are input into a two-stream second-order attention fusion module for feature fusion to obtain four fused feature maps. The four fused feature maps are each input into a dynamic feature upsampling module for adaptive feature reconstruction, resulting in four sets of reconstructed images of the same scale. The four reconstructed images are stitched together and then input into the convolution module to obtain the final fused image; Any two-stream second-order attention fusion module includes a first-order channel attention unit, a second-order channel attention unit (SCAM), a first-order spatial attention unit, a second-order spatial attention unit (SSAM), two upsampling units, and two PReLU activation functions. Here, an upsampling unit and a PReLU activation function are combined to form a sampling activation unit. The combined feature map input to the current two-stream second-order attention fusion module is re-divided into a visible light feature map and an infrared feature map; the visible light feature map is divided into two branches and enters a sampling activation unit respectively, and the infrared feature map is also divided into two branches and enters a sampling activation unit respectively; The infrared feature map branch and the visible light feature map branch that have passed through one of the sampling activation units are respectively subjected to the first branch weight acquisition operation to obtain the infrared image channel attention branch weight and the visible light image channel attention branch weight. The first branch weight acquisition operation is specifically as follows: the current feature map branch is divided into two sub-paths. The first sub-path is input into the first-order channel attention unit for feature extraction to obtain the first channel sub-path features. The second sub-path is input into the second-order channel attention unit SCAM for feature extraction to obtain the second channel sub-path features. The first channel sub-path features and the second channel sub-path features are multiplied element-wise to obtain the channel attention branch weight. The infrared feature map branch and the visible light feature map branch, which have passed through another sampling activation unit, are respectively subjected to the second branch weight acquisition operation to obtain the infrared image spatial attention branch weight and the visible light image spatial attention branch weight. The second branch weight acquisition operation is specifically as follows: the current feature map branch is divided into two sub-paths. The first sub-path is input into the first-order spatial attention unit for feature extraction to obtain the first spatial sub-path features. The second sub-path is input into the second-order spatial attention unit SSAM for feature extraction to obtain the second spatial sub-path features. The first spatial sub-path features and the second spatial sub-path features are multiplied element-wise to obtain the spatial attention branch weight. The infrared image features are obtained by multiplying the infrared image channel attention branch weights, the infrared image spatial attention branch weights, and the infrared feature map; the visible light image features are obtained by multiplying the visible light image channel attention branch weights, the visible light image spatial attention branch weights, and the visible light feature map. The infrared image features and the visible light image features are added element by element, and the result is the final output of the two-stream second-order attention fusion module.

2. The infrared and visible light image fusion method based on second-order attention fusion features as described in claim 1, characterized in that, The feature extraction module includes two high-resolution scale dynamic feature downsampling units (AFEM) and two low-resolution scale dynamic feature downsampling units (AFEM). The visible light image is divided into two branches. The first visible light image branch is processed by one of the high-resolution scale dynamic feature downsampling units (AFEM) to obtain a high-resolution scale feature map of visible light. The second visible light image branch is processed by another low-resolution scale dynamic feature downsampling unit (AFEM) to obtain a low-resolution scale feature map of visible light. The infrared image is divided into two branches. The first infrared image branch is processed by another high-resolution scale dynamic feature downsampling unit (AFEM) to obtain a high-resolution scale feature map of the infrared image. The second infrared image branch is processed by another low-resolution scale dynamic feature downsampling unit (AFEM) to obtain a low-resolution scale feature map of the infrared image.

3. The infrared and visible light image fusion method based on second-order attention fusion features as described in claim 2, characterized in that, Any dynamic feature downsampling unit (AFEM) consists of two 1×1 convolutional layers, an activation function, a depthwise separable convolution DSConv, and a max pooling layer; The input branch image of the current dynamic feature downsampling unit is divided into two sub-paths. The first sub-path is processed sequentially through two 1×1 convolutional layers and an activation function to obtain the first sub-path processed image. Simultaneously, the first sub-path processed image is multiplied element-wise with the feature map weights output by the first 1×1 convolutional layer to obtain the first sub-path product image. The other sub-path is processed sequentially through a depthwise separable convolution DSConv and a max pooling layer to obtain the second sub-path processed image. The first sub-path product image and the second sub-path processed image are summed element-wise to obtain the final output of the current dynamic feature downsampling unit.

4. The infrared and visible light image fusion method based on second-order attention fusion features as described in claim 3, characterized in that, The depthwise separable convolution DSConv comprises sequentially cascaded 3×3 depthwise convolutional layers, 1×1 pointwise convolutional layers, and a non-linear activation function; the processing procedure of the depthwise separable convolution DSConv on the input feature F is as follows: in, This represents a 3×3 depth convolutional layer. Represents a 1×1 pointwise convolutional layer. Represents a non-linear activation function. This is the final output of the depthwise separable convolution DSConv.

5. The infrared and visible light image fusion method based on second-order attention fusion features as described in claim 1, characterized in that, The process of feature extraction by the first-order channel attention unit is as follows: The process of feature extraction by the second-order channel attention unit SCAM is as follows: The process of obtaining the channel attention branch weights is as follows: in, F To input the subpath features of the first-order channel attention unit and the second-order channel attention unit SCAM, For global average pooling, For a dimension reduction matrix, For activation function, For an upgraded matrix, It is the sigmoid activation function. This is the characteristic of the first channel sub-path. for F The covariance matrix, , H and W These represent the height and width of the sum sub-path, respectively. for F The i Line number j The pixel values ​​in the column, where T represents transpose. Represents a vectorized matrix. This is a characteristic of the second channel sub-path. For channel attention branch weights, This indicates element-wise multiplication.

6. The infrared and visible light image fusion method based on second-order attention fusion features as described in claim 1, characterized in that, The process of feature extraction by a first-order spatial attention unit is as follows: The process of feature extraction by the second-order spatial attention unit (SSAM) is as follows: The process of obtaining the spatial attention branch weights is as follows: in, F To input the subpath features of the first-order spatial attention unit and the second-order spatial attention unit SSAM, for convolution, It is the sigmoid activation function. For another convolution, This indicates element-wise multiplication. Features of the first spatial sub-path Features of the second spatial sub-path For spatial attention branch weights.

7. The infrared and visible light image fusion method based on second-order attention fusion features as described in claim 1, characterized in that, Any dynamic feature upsampling unit AIRM consists of three depthwise separable convolutions DSConv, two nearest neighbor upsampling units, a 1×1 convolutional layer, and an activation function; The fused feature map of the current dynamic feature upsampling unit AIRM is divided into two sub-paths. The first sub-path is processed sequentially through the first depthwise separable convolution DSConv, the first nearest neighbor upsampling unit, a 1×1 convolutional layer, and an activation function to obtain the first sub-path processed image. Simultaneously, the first sub-path processed image is multiplied element-wise with the feature image output by the first nearest neighbor upsampling unit to obtain the first sub-path product image. The other sub-path is processed sequentially through the second depthwise separable convolution DSConv, the second nearest neighbor upsampling unit, and the third depthwise separable convolution DSConv to obtain the second sub-path processed image. The first sub-path product image and the second sub-path processed image are summed element-wise to obtain the final output of the current dynamic feature upsampling unit AIRM.

8. The infrared and visible light image fusion method based on second-order attention fusion features as described in claim 1, characterized in that, The loss function used to train the fusion model, which consists of a feature extraction module, four two-stream second-order attention fusion modules, four dynamic feature upsampling modules, and a convolution module, is as follows: in, For the total loss, For pixel loss, For feature loss, For style loss, For block loss, , , This is the balance coefficient.

Images