Cross-domain image fusion algorithm based on multi-level hybrid attention feature extraction

Abstract

The application discloses a cross-domain image fusion method based on multi-level mixed attention feature extraction, which aims at the complementarity of infrared and visible light images in information dimension, realizes effective representation and adaptive fusion of local details and global semantic information by constructing a two-stage training structure, combining multi-level feature extraction, mixed attention mechanism and cross-domain fusion module. The encoder extracts different level features through a hierarchical progressive structure, the fusion module uses cross attention to strengthen the information complementarity between modalities, and the decoder realizes high-quality reconstruction of the fused image. The method performs well on multiple standard datasets and is superior to existing methods in edge definition, hot target retention and texture detail reconstruction.

Description

Technical Field

[0001] This invention relates to the fields of image processing, artificial intelligence and multimodal computer vision, and in particular to a cross-domain image fusion method that integrates infrared and visible light images. It is based on multi-level feature extraction and a hybrid attention mechanism and belongs to the field of image fusion and intelligent perception technology. Background Technology

[0002] In tasks involving intelligent security, autonomous driving, and disaster monitoring, infrared and visible light images offer complementary advantages in thermal target detection and texture detail representation, respectively. Infrared images excel in anti-interference performance and effectively capture thermal targets, but lack detailed information; visible light images offer rich textures and clear scene details, but are susceptible to lighting conditions. In practical applications, effective fusion of these two types of images is crucial for improving target detection accuracy and enhancing scene perception capabilities. High-quality fused images provide more comprehensive and reliable information support for subsequent decision-making and processing.

[0003] Traditional infrared and visible light image fusion methods can accomplish basic fusion tasks, but they rely on manually designed features and fusion rules, resulting in weak adaptability and limited fusion performance in complex scenes. While deep learning methods have made some progress in this field in recent years, existing CNN models struggle to model long-range dependencies, and Transformer structures suffer from high computational costs. Furthermore, existing models still have room for improvement in multi-scale feature extraction, cross-modal information interaction, and the quality of fused images. Therefore, it is essential to overcome the limitations of existing methods, improve the fusion model's ability to model long-range dependencies, reduce computational costs, and optimize multi-scale feature extraction, cross-modal information interaction, and the quality of fused images. Summary of the Invention

[0004] Purpose of the invention: To overcome the shortcomings of existing technologies, a cross-domain image fusion algorithm based on multi-level hybrid attention feature extraction is proposed to improve the performance of fused images in terms of hot target preservation, edge sharpness, and texture detail reconstruction, while ensuring the computational efficiency and deployability of the model.

[0005] To achieve the above objectives, the technical solution provided by this invention is: a cross-domain image fusion algorithm based on multi-level hybrid attention feature extraction, comprising the following steps and structural modules:

[0006] S1: Overall framework of the model based on a two-stage network structure design;

[0007] S2: A multi-level feature extraction module that uses hierarchical hybrid attention for four-layer feature extraction;

[0008] S3: A cross-domain fusion module that utilizes a cross-attention mechanism to perform bidirectional interaction between infrared and visible light features;

[0009] S4: A network architecture loss function designed by combining strength loss, similarity loss, and gradient loss.

[0010] Furthermore, the specific method of step S1 is as follows:

[0011] S1.1: Phase 1: Reconstruct the original image from the input image based on the designed encoder and decoder;

[0012] S1.2: Stage 2 adds a cross-domain fusion module to the network structure of Stage 1 to fuse the features of images from different modalities, and finally generates a fused image containing information from both modalities.

[0013] S1.3: Let I be the original input infrared image. IR ∈R H×W×1 The visible light image is I VIS ∈R H×W×3 The encoder is E(·), the decoder is D(·), and the cross-domain fusion module is f(·).

[0014] S1.4: Original Image Reconstruction. In Stage One, paired infrared and visible light images {I IR ,I VIS The data is input into the network, where it is processed through embedding layers and an encoder to extract features at different levels {F1, F2, F3, F4}.

[0015] F1 IR F2 IR F3 IR F4 IR =E(Conv 3×3 (I IR ));

[0016] F1 VIS F2 VIS F3 VIS F4 VIS =E(Conv 3×3 (I VIS ));

[0017] The extracted features are then fed into the decoder, and finally, a 3×3 convolution is performed to obtain the reconstructed original image {I′}. IR ,I′ VIS}:

[0018] I' IR =d(F1) IR F2 IR F3 IRF4 IR );

[0019] I' VIS =D(F1) VIS F2 VIS F3 VIS F4 VIS );

[0020] S1.5: Image Fusion. In stage two, paired infrared and visible light images {I IR ,I VIS The features are input into the encoder trained in stage one to obtain features of different modalities. Then, the infrared image features and visible light image features of each layer are fed together into the cross-domain fusion module to obtain the fused feature F of that layer. f :

[0021] Fi f =F(Fi) IR ,Fi VIS ), i = 1, 2, 3, 4;

[0022] Finally, all fusion features {F1 f F2 f F3 f F4 f The fused image I is generated through a decoder and a 3×3 convolution. f :

[0023] I f =DConv 3×3 (D(F1 f F2 f F3 f F4 f )).

[0024] Furthermore, the specific method of step S2 is as follows:

[0025] S2.1: Encoder Structure. The encoder consists of four layers of hybrid attention modules, with a downsampling operation performed after each layer to extract features at different depths layer by layer;

[0026] S2.2: Encoder operation procedure. (Note F) embed F is the feature extracted from the input image after passing through the embedding layer. decoder_IR These are the features after decoding. HA(·) represents the hybrid attention module, Down(·) represents downsampling, and Up(·) represents downsampling. The encoder flow is then represented as follows:

[0027] F1 IR =HA(F embed_IR );

[0028] Fi IR =HA(Down(F(i-1)) IR ), i = 2, 3, 4;

[0029] S2.3: Decoder Structure. The decoder adopts a mirror-symmetric structure with the encoder, consisting of four layers of hybrid attention and three upsampling operations;

[0030] S2.4: Decoder operation procedure:

[0031] F decoder_IR =HA(F1) IR +UP(HA(F2 IR +UP(HA(F3 IR +UP(HA(F4 IR )))))));

[0032] S2.5: Hybrid Attention Structure. The hybrid attention module integrates channel attention (enhancing discriminative channels), spatial attention (strengthening key regions), and self-attention (building long-range dependencies) to form an attention structure with strong feature representation capabilities;

[0033] S2.6: Let the input feature map be X∈R H′×W′×C′ The features X obtained through channel attention mentioned above C The calculation method is as follows:

[0034] X C =Sigmoid(Conv 1×1 (ReLU(Conv 1×1 (GAP(X)))))X;

[0035] S2.7: The feature X obtained above through spatial attention C&S The calculation method is as follows:

[0036] X C&S =Sigmoid(Conv 1×1 (GN(X c )))X C ;

[0037] S2.8: In the calculation of self-attention, a concatenation of 1×1 convolutions and depthwise convolutions is used to obtain Q, K, and V:

[0038] Q,K,V=DConv 3×3 (Conv 1×1 (X));

[0039] Among them, DConv 3×3 (·) indicates a depthwise convolution with a kernel size of 3×3;

[0040] Then, reshape the shapes of Q, K, and V:

[0041] Q′,K′,V′=Reshape(Q,K,V);

[0042] Features X obtained after self-attention SA Represented as:

[0043]

[0044] S2.9: Finally, the features extracted through hybrid attention are concatenated to obtain feature X. C&S&SA :

[0045] X C&S&SA =Conv 1×1 (Concat(X C&S V SA )).

[0046] Furthermore, the specific method for step S3 is as follows:

[0047] S3: The cross-domain fusion module employs a multi-path cross-attention mechanism to enable bidirectional interaction between infrared and visible light features. Multiple attention paths are constructed (IR→VIS, VIS→IR, fusion→IR / VIS) to fully model the semantic relationships between modalities. Finally, a unified fusion feature is achieved through 1×1 convolution. The detailed calculation process is as follows:

[0048] Q VIS ,K VIS V VIS =DConv 3×3 (Conv 1×1 (F VIS ));

[0049] Q IR ,K IR V IR =DConv 3×3 (Conv 1×1 (F IR ));

[0050] Q f =DConv 3×3 (Conv 1×1 (Conv 1×1 (Concat(F VIS +F IR ))));

[0051] Atten1 = Attention(Q) VIS ,K IR V IR );

[0052] Atten2 = Attention(Q) IR ,K VIS V VIS );

[0053] Atten3 = Attention(Q) f ,K VIS V VIS );

[0054] Atten4 = Attention(Q) f ,K IR V IR );

[0055] F f =Conv 1×1 (Concat(Atten1,Atten2,Atten3,Atten4);

[0056] In the above formula, Attention(·) represents the calculation of attention, which is the same as the calculation method of self-attention in mixed attention, F f This indicates the characteristics after fusion.

[0057] Furthermore, the specific method for step S4 is as follows:

[0058] S4.1: The loss function consists of three parts: intensity loss for pixel fidelity constraints, structural similarity loss for constraining the overall structural consistency of the image, and gradient loss for preserving image texture details and edge features.

[0059] S4.2: Phase 1 is the reconstruction phase, where the encoder and decoder are trained using intensity loss and structural similarity loss. The loss function is constructed as follows:

[0060]

[0061] L SSIM (I IR ,I′ IR ) = 1 - SSIM(I IR ,I′ IR );

[0062]

[0063]

[0064] In the above formula, L represents the intensity loss in phase one. SSIM L represents the structural similarity loss. IR and L VISThese represent the reconstruction losses for infrared and visible light images, respectively. λ1 and λ2 represent the overall reconstruction loss in stage one, and are adjustable hyperparameters.

[0065] S4.3: Stage two is the fusion stage, which introduces gradient loss to combine the differences between the original image and the fused image, guiding the network to generate high-quality fusion results. The loss function construction process is as follows:

[0066]

[0067]

[0068]

[0069] In the above formula, L represents the intensity loss in stage two. grad Represents gradient loss, This represents the overall loss in Phase Two. It is the Sobel gradient operator, and λ3 is an adjustable hyperparameter. Attached Figure Description

[0070] Figure 1 This is a block diagram of the overall network architecture in an embodiment of the present invention;

[0071] Figure 2 This is a structural diagram of the hybrid attention module in an embodiment of the present invention;

[0072] Figure 3 This is a schematic diagram of the cross-domain fusion module in an embodiment of the present invention;

[0073] Figure 4 This is a comparison chart of the fusion effect of the designed algorithm and the baseline algorithm in this embodiment of the invention on the MSRS dataset;

[0074] Figure 5 This is a comparison chart of the fusion effect of the designed algorithm and the baseline algorithm in this embodiment of the invention on the TNO dataset;

[0075] Figure 6 This is a comparison chart of the fusion effect of the designed algorithm and the baseline algorithm in the embodiments of the present invention on the RoadScene dataset. Detailed Implementation

[0076] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0077] like Figure 1As shown in the embodiments of the present invention, the cross-domain image fusion algorithm based on multi-level hybrid attention feature extraction includes the following key modules: encoder, decoder, hybrid attention module, cross-domain fusion module, two-stage network structure, and loss function design.

[0078] S1: The encoder module based on a hierarchical progressive structure extracts image features;

[0079] S1.1: Encoder Structure. The encoder consists of four layers of hybrid attention modules, with a downsampling operation performed after each layer to extract features at different depths layer by layer;

[0080] S1.2: Encoder operation procedure. (Note F) embed F is the feature extracted from the input image after passing through the embedding layer. decoder_IR These are the features after decoding. HA(·) represents the hybrid attention module, Down(·) represents downsampling, and Up(·) represents downsampling. The encoder flow is then represented as follows:

[0081] F1 IR =HA(F embed_IR );

[0082] Fi IR =HA(Down(F(i-1)) IR ), i = 2, 3, 4.

[0083] S2: Reconstructing image features using a decoder with a mirror-symmetric structure to the encoder;

[0084] S2.1: Decoder Structure. The decoder adopts a mirror-symmetric structure with the encoder, consisting of four layers of hybrid attention and three upsampling operations;

[0085] S2.2: Decoder operation procedure:

[0086] F decoder_IR =HA(F1) IR +UP(HA(F2 IR +UP(HA(F3 IR +UP(HA(F4 IR )))))))

[0087] S3: A hybrid attention module that uses hierarchical hybrid attention for four-layer feature extraction. Its block diagram is as follows: Figure 2 As shown;

[0088] S3.1: Hybrid Attention Structure. The hybrid attention module integrates channel attention (enhancing discriminative channels), spatial attention (strengthening key regions), and self-attention (building long-range dependencies) to form an attention structure with strong feature representation capabilities. The structural diagram of the hybrid attention module is shown below. Figure 2 As shown;

[0089] S3.2: Let the input feature map be X∈R H′×W′×C′ The features X obtained through channel attention mentioned above C The calculation method is as follows:

[0090] X C =Sigmoid(Conv 1×1 (ReLU(Conv 1×1 (GAP(X)))))X;

[0091] S3.3: The feature X obtained above through spatial attention C&S The calculation method is as follows:

[0092] X C&S =Sigmoid(Conv 1×1 (GN(X c )))X C ;

[0093] S3.4: In the calculation of self-attention, a concatenation of 1×1 convolutions and depthwise convolutions is used to obtain Q, K, and V:

[0094] Q,K,V=DConv 3×3 (Conv 1×1 (X));

[0095] Among them, DConv 3×3 (·) indicates a depthwise convolution with a kernel size of 3×3;

[0096] Then, reshape the shapes of Q, K, and V:

[0097] Q′,K′,V′=Reshape(Q,K,V);

[0098] Features X obtained after self-attention SA Represented as:

[0099]

[0100] S3.5: Finally, the features extracted through hybrid attention are concatenated to obtain feature X. C&S&SA :

[0101] X C&S&SA =Conv 1×1 (Concat(X C&S ,X SA )).

[0102] S4: A cross-domain fusion module that utilizes a cross-attention mechanism to perform bidirectional interaction between infrared and visible light features. Its structural block diagram is as follows: Figure 3 As shown;

[0103] The cross-domain fusion module employs a multi-path cross-attention mechanism to enable bidirectional interaction between infrared and visible light features. Multiple attention paths are constructed (IR→VIS, VIS→IR, fusion→IR / VIS) to fully model the semantic relationships between modalities. Finally, a unified fusion feature is achieved through 1×1 convolution. The detailed calculation process is as follows:

[0104] Q VIS ,K VIS V VIS =DConv 3×3 (Conv 1×1 (F VIS ));

[0105] Q IR ,K IR V IR =DConv 3×3 (Conv 1×1 (F IR ));

[0106] Q f =DConv 3×3 (Conv 1×1 (Conv 1×1 (Concat(F VIS +F IR ))));

[0107] Atten1 = Attention(Q) VIS ,K IR V IR );

[0108] Atten2 = Attention(Q) IR ,K VIS V VIS );

[0109] Atten3 = Attention(Q) f ,K VIS V VIS );

[0110] Atten4 = Attention(Q) f ,K IR V IR );

[0111] F f =Conv 1×1 (Concat(Atten1,Atten2,Atten3,Atten4);

[0112] In the above formula, Attention(·) represents the calculation of attention, which is the same as the calculation method of self-attention in mixed attention, F f This indicates the characteristics after fusion.

[0113] S5: Overall algorithm flow based on a two-stage network architecture design;

[0114] S5.1: Phase 1: Reconstruct the original image from the input image based on the designed encoder and decoder;

[0115] S5.2: Stage 2 adds a cross-domain fusion module to the network structure of Stage 1 to fuse the features of images from different modalities, and finally generates a fused image containing information from both modalities.

[0116] S5.3: Let the original input infrared image be I. IR ∈R H×W×1 The visible light image is I VIS ∈R H×W×3 The encoder is E(·), the decoder is D(·), and the cross-domain fusion module is F(·).

[0117] S5.4: Original Image Reconstruction. In Stage 1, paired infrared and visible light images {I IR ,I VIS The data is input into the network, where it is processed through embedding layers and an encoder to extract features at different levels {F1, F2, F3, F4}.

[0118] F1 IR F2 IR F3 IR F4 IR =E(Conv 3×3 (I IR ));

[0119] F1 VIS F2 VIS F3 VIS F4 VIS =E(Conv 3×3 (I VIS ));

[0120] The extracted features are then fed into the decoder, and finally, a 3×3 convolution is performed to obtain the reconstructed original image {I′}. IR ,I′ VIS}:

[0121] I' IR =D(F1) IR F2 IR F3 IR F4IR );

[0122] I' VIS =D(F1) VIS F2 VIS F3 VIS F4 VIS );

[0123] S5.5: Image Fusion. In Phase Two, paired infrared and visible light images {I IR ,I VIS The features are input into the encoder trained in stage one to obtain features of different modalities. Then, the infrared image features and visible light image features of each layer are fed together into the cross-domain fusion module to obtain the fused feature F of that layer. f :

[0124] Fi f =F(Fi) IR ,Fi VIS ), i = 1, 2, 3, 4;

[0125] Finally, all fusion features {F1 f F2 f F3 f F4 f The fused image I is generated through a decoder and a 3×3 convolution. f The image fusion performance comparison between the designed algorithm and the baseline algorithm on the MSRS, TNO, and RoadScene datasets is shown in the following figures. Figure 4 , Figure 5 , Figure 6 As shown;

[0126] I f =Conv 3×3 (D(F1 f F2 f F3 f F4 f )).

[0127] S6: A network architecture loss function designed by combining intensity loss, similarity loss, and gradient loss;

[0128] S6.1: The loss function consists of three parts: intensity loss for pixel fidelity constraints, structural similarity loss for constraining the overall structural consistency of the image, and gradient loss for preserving image texture details and edge features.

[0129] S6.2: Phase 1 is the reconstruction phase, where the encoder and decoder are trained using intensity loss and structural similarity loss. The loss function construction process is as follows:

[0130]

[0131] L SSIM (I IR ,I′ IR ) = 1 - SSIM(I IR ,I′ IR );

[0132]

[0133]

[0134] In the above formula, L represents the intensity loss in phase one. SSIM L represents the structural similarity loss. IR and L VIS These represent the reconstruction losses for infrared and visible light images, respectively. λ1 and λ2 represent the overall reconstruction loss in stage one, and are adjustable hyperparameters.

[0135] S6.3: Stage two is the fusion stage, which introduces gradient loss to combine the differences between the original image and the fused image, guiding the network to generate high-quality fusion results. The loss function construction process is as follows:

[0136]

[0137]

[0138]

[0139] In the above formula, L represents the intensity loss in stage two. grad Represents gradient loss, This represents the overall loss in Phase Two. It is the Sobel gradient operator, and λ3 is an adjustable hyperparameter.

Claims

1. A cross-domain image fusion algorithm based on multi-level hybrid attention feature extraction, characterized in that, The method comprises the following steps: S1: a model overall framework based on a two-stage network structure design; S2: a multi-level feature extraction module using hierarchical mixed attention for four-layer feature extraction; S3: a cross-domain fusion module using a cross-attention mechanism to bidirectionally interact infrared and visible light features; S4: a network architecture loss function designed by combining strength loss, similarity loss and gradient loss; The step S1 comprises: S1.1: in the first stage, the designed encoder and decoder are used to reconstruct the original image based on the input image; S1.2: in the second stage, a cross-domain fusion module is added based on the network structure in the first stage, and the features of different modal images are fused to finally generate a fusion image containing two modal information; S1.3: record the original input infrared image as , the visible light image is , the encoder is , the decoder is , and the cross-domain fusion module is ; S1.4: original image reconstruction, in phase one, pairs of infrared images and visible light images are input into the network, different levels of features are extracted by embedding layers and encoders : ； ； The extracted features are then sent into the decoder, and finally a 3x3 convolution is used to get the reconstructed original image : ； ； S1.5: image fusion, in phase two, the pair of infrared image and visible light image are input into the trained encoder in phase one to obtain features of different modalities, then, the infrared image features and visible light image features of each layer are sent into the cross-domain fusion module together to obtain the fusion features of the layer : ； Finally, all the fused features The fused image is generated by a decoder and a 3x3 convolution : ； The step S2 comprises: S2.1: the encoder is composed of four mixed attention modules, and a downsampling operation is performed once after each layer to extract features of different depths layer by layer; S2.2: record is the feature extracted from the input image after the embedding layer, is the feature after the decoder, denotes the hybrid attention module, denotes down-sampling, denotes down-sampling, the encoder flow is denoted as: ； ； S2.3: the decoder adopts a structure symmetric to the encoder, which is composed of four mixed attention and three upsampling operations; S2.4: the operation process of the decoder: ； S2.5: the mixed attention module fuses channel attention to enhance discriminability, spatial attention to strengthen key areas, and self-attention to construct long-range dependencies, forming an attention structure with strong feature expression ability; S2.6: record the input feature map as the feature obtained by the channel attention The calculation method is: ； S2.7: the spatial attention derived features described above The calculation method is: ； S2.8: In the computation of self-attention, a concatenation of 1 x 1 convolution and depthwise convolution is used to get , , : ； wherein denotes a depth convolution with a kernel size of 3x3; Then, the shape of , , is reshaped: ； features obtained after self-attention is represented as: ； S2.9: Finally, the features extracted by the mixed attention are concatenated to obtain the feature : ； The step S3 comprises: S3: the cross-domain fusion module adopts a multi-path cross-attention mechanism to bidirectionally interact infrared and visible light features, constructs multiple attention paths, fully models the semantic relationship between modalities, and finally fuses them into unified fusion features through 1×1 convolution, and the detailed calculation process is as follows: ； ； ； ； ； ； ； ； In the above formula, represents attention calculation, which is the same as the calculation method of self-attention in hybrid attention, represents the fused feature; The step S4 comprises: S4.1: the loss function comprises three parts, namely, the strength loss for pixel fidelity constraint, the structure similarity loss for constraining the overall consistency of the image structure, and the gradient loss for preserving image texture details and edge features; S4.2: in the first stage, the encoder and the decoder are trained using the strength loss and the structure similarity loss, and the loss function construction process is as follows: ； ； ； ； In the above formula, denotes the intensity loss of stage one, denotes the structural similarity loss, and denote the reconstruction loss of the infrared image and the visible light image, respectively, denotes the overall reconstruction loss of stage one, , is an adjustable hyperparameter; S4.3: in the second stage, the gradient loss is introduced, and the difference between the original image and the fusion image is used to guide the network to generate high-quality fusion results, and the loss function construction process is as follows: ； ； ； In the above formula, denotes the loss of strength in phase two, denotes the loss of gradient, denotes the overall loss in phase two, is the gradient operator, is an adjustable hyperparameter.

Images