An infrared and visible light image fusion method based on a multi-scale hybrid converter

Abstract

This invention discloses an infrared and visible light image fusion method based on a multi-scale hybrid transformer, comprising the following steps: constructing an infrared and visible light image fusion model framework based on a multi-scale hybrid transformer; the model framework includes an auxiliary encoder, a main encoder, and a decoder; the auxiliary encoder extracts global information of infrared and visible light, the main encoder extracts local contextual information of infrared and visible light, and the auxiliary encoder guides the main encoder to fuse global and local information, so that the fused image contains multi-scale source image features; the multi-scale source image features are input into the decoder to complete image reconstruction. This invention guides a convolutional neural network to reconstruct fused image features through a multi-scale hybrid transformer, effectively capturing low-level spatial features and high-level semantic context, modeling from both local and global perspectives, and resulting in a stronger ability to represent thermal radiation information and texture details in the fused image.

Description

Technical Field

[0001] This invention relates to a method for fusing infrared and visible light images based on a multi-scale hybrid converter, belonging to the field of image processing technology. Background Technology

[0002] Currently, fusing visible light and infrared images can produce images with rich texture details and high-brightness thermal radiation targets, thus solving the problem that single-type images cannot meet engineering requirements. Image fusion methods can be used to obtain high-quality, information-rich images. This method already has broad application prospects in target recognition, military applications, and medical diagnosis.

[0003] Existing infrared and visible light image fusion techniques are mainly divided into traditional fusion methods and deep learning fusion methods. Traditional fusion methods are mainly divided into multi-scale transformation and other methods. In 2020, CHEN et al. proposed an infrared and visible light image fusion method based on multi-scale low-rank decomposition. This method effectively compensates for image information through multi-scale and multi-level fusion and improves the image fusion effect by designing specific fusion rules. (Chen Chaoqi, Meng Xiangchao, Shao Feng, Fu Randi. An infrared and visible light image fusion method based on multi-scale low-rank decomposition[J]. Acta Optica Sinica, 2020, 40(11).) However, these traditional methods have limited ability to mine complex image features, low computational efficiency, and poor visual effect of fused images.

[0004] Furthermore, deep learning-based methods have achieved excellent results in image fusion, mainly divided into convolutional neural networks and generative adversarial networks, among other methods. In 2021, LI et al. proposed an end-to-end residual fusion network for infrared and visible image fusion, which trains the encoder and intermediate fusion layer separately through a two-stage method. However, due to the use of only convolution to extract features, it lacks efficiency in capturing global contextual information and has limited ability to extract fine-grained information from images. (An end-to-end residual fusion network for infrared and visible images[J].Information Fusion,2021,73:72-86.) To address these issues, some existing image fusion methods use only self-attention-based transformers to model global information. However, most methods weaken the global modeling ability of the transformer while using self-attention. At the same time, the single multilayer perceptron structure has limited ability to extract and aggregate information, does not model multi-scale information in potential image information, and lacks spatial inductive bias when modeling local information. Summary of the Invention

[0005] The problem to be solved by the present invention is to overcome the defects of the prior art and provide an infrared and visible light image fusion method based on a multi-scale hybrid converter.

[0006] To address the above problems, this invention provides an infrared and visible light image fusion method based on a multi-scale hybrid converter, comprising:

[0007] Construct an end-to-end image fusion network model, which includes a master encoder, an auxiliary encoder, and a decoder.

[0008] The auxiliary encoder models global long-range dependency features using a Transformer to extract global information features from the input infrared and visible light images.

[0009] The main encoder extracts contextual features within a specific receptive field through convolution, which are then used to extract local information from the input infrared and visible light images.

[0010] The decoder maps the learned fusion features to the original resolution and obtains the fused image by gradually fusing image features from the bottom layer to the top layer.

[0011] Furthermore, the auxiliary encoder includes an infrared path and a visible light path;

[0012] Both the infrared path and the visible light path include stage 1, stage 2, stage 3 and stage 4, and each stage is connected sequentially.

[0013] Furthermore, the number of input channels and output channels in each stage are 16, 32, 64, 128 and 32, 64, 128, 256, respectively;

[0014] Each of the four stages includes a downsampling operation (patch embedding) and a multi-scale hybrid attention block (MSHA). Infrared and visible light images are input into the auxiliary encoder, respectively, and pass through each stage to obtain global infrared information features and global visible light information features.

[0015] Furthermore, stages 1, 2, and 4 stack downsampling and multi-scale hybrid attention modules twice, and stage 3 stacks downsampling and multi-scale hybrid attention modules four times.

[0016] The downsampling operation doubles the number of channels in the input feature and halves the height and width of the input feature.

[0017] The multi-scale hybrid attention module includes a window multi-head self-attention module (W-MSA), a transform window multi-head self-attention module (SW-MSA), a channel and height spatial dimension interactive attention module, a channel and width spatial dimension interactive attention module, a multi-scale feedforward network, and a layer normalization operation (Layernorm).

[0018] The windowed multi-head self-attention module (W-MSA), the channel-and-high spatial dimension interactive attention module, and the channel-and-wide spatial dimension interactive attention module are connected in parallel. The input features F are stacked for the first time. l-1 After layer normalization, standardized features are obtained. Then, after triple attention, these features are added to the input features to obtain a hybrid attention feature map F. l The hybrid attention feature map is then processed through layer normalization and a multi-scale feedforward network to obtain features, which are then added to the hybrid attention feature map to obtain the multi-scale hybrid attention feature H. l The calculation formula can be expressed as:

[0019] F l =W-MSA(L(F l-1 ))+scale×(CW(L(F l-1 ))+CH(L(F l-1 )))+F l-1 ,

[0020] H l =MSFFN(LN(F l ))+F l l 1,

[0021] Second stack, input features F1 l-1 After layer normalization, standardized features are obtained. Then, these features are added to the input features through triple attention to obtain the hybrid attention feature map F1. l The hybrid attention feature map is then processed through layer normalization and a multi-scale feedforward network to obtain features, which are then added to the hybrid attention feature map to obtain the multi-scale hybrid attention feature H1. l The calculation formula can be expressed as:

[0022] F1 l =SW-MSA(L(F1) l-1 ))+scale×(CW(L(F1 l-1 ))+CH(L(F1 l-1 )))+F1 l-1 ,

[0023] H1 l =MSFFN(LN(F1) l ))+F1 l ,

[0024] W-MSA is a multi-head self-attention window, SW-MSA is a multi-head self-attention window transformed window, both designed to capture long-distance dependencies and obtain high-level semantic features. L is the LayerNorm operation, designed to improve gradient propagation and enhance generalization ability. CW is channel-wide spatial dimension interaction attention, CH is channel-high spatial dimension interaction attention. Both emphasize multi-dimensional interaction without reducing the importance of dimensions, eliminating the indirect correspondence between channels and weights. MSFFN is a multi-scale feedforward network, designed to obtain features at different scales of the image and obtain richer image information. scale is a hyperparameter, here scale is 0.5, where stage 3 is 4 stacks, that is, repeating the above operation twice.

[0025] Furthermore, the channel and high spatial dimension interactive attention module includes Permute operation, Z-Pool operation, 7×7 convolution kernel, batch normalization, and sigmoid activation function operation. The input feature F1 is transformed into H×C×W dimension features through the Permute operation, then Z-Pool is performed on the H dimension, followed by 7×7 convolution and batch normalization, and then further processed by the sigmoid activation function to obtain the attention map. The attention map is multiplied with the transformed features and then transformed again by the Permute operation to obtain the C×H×W dimension spatial interactive attention feature F2, which can be expressed as:

[0026] F2=P(P(F1)·Sig(Conv(Z(P(F1))))),

[0027] Where P is the Permute operation, Z is the Z-Pool operation, Sig is the Sigmoid activation function operation, and Conv is the convolution operation, this module enhances the ability to exchange spatial information by capturing the cross-channel interaction between channels and spatial dimension H, while encoding more accurate spatial information.

[0028] The channel-wide spatial dimension interactive attention module includes Permute operation, Z-Pool operation, 7×7 convolution, batch normalization, and Sigmoid activation function operation. Input feature F3 is transformed into W×H×C dimension features through the Permute operation, then Z-Pool is performed on the W dimension, followed by 7×7 convolution and batch normalization, and finally Sigmoid activation function to obtain the attention map. The attention map is multiplied by the transformed features, and then transformed again through the Permute operation to obtain the C×H×W dimension spatial interactive attention feature F4. The calculation can be expressed as:

[0029] F4=P(P(F3)·Sig(Conv(Z(P(F3))))),

[0030] Where P is the Permute operation, Z is the Z-Pool operation, Sig is the Sigmoid activation function operation, and Conv is the convolution operation. This module enhances the ability to exchange spatial information by capturing the cross-channel interaction between channels and spatial dimension W, while encoding more accurate spatial information.

[0031] Furthermore, the multi-scale feedforward network includes a 1×1 convolution, a 3×3 depthwise separable convolution, a 5×5 depthwise separable convolution, a ReLU activation function, and a concatenation operation. The input features are processed through two parallel branches of different scales. In one branch, a 1×1 convolution, a 3×3 depthwise separable convolution, and a ReLU are sequentially connected. In the other branch, a 1×1 convolution, a 5×5 depthwise separable convolution, and a ReLU are sequentially connected. The input feature F first passes through a 1×1 convolution kernel, a 3×3 depthwise separable convolution kernel, a 5×5 depthwise separable convolution kernel, and a ReLU activation function. The input features are obtained by separating convolution and ReLU activation function to obtain feature M1. The input features are then processed by a 1×1 convolution kernel, a 5×5 depthwise separable convolution, and a ReLU activation function to obtain feature S1. These two features are then concatenated and processed by 3×3 and 5×5 depthwise separable convolutions and ReLU activation functions respectively to obtain features M2 and S2. Features M2 and S2 are then concatenated and processed by a 1×1 convolution, and finally added element-wise with the input features to obtain the multi-scale feature MS. The calculation can be expressed as follows:

[0032] M1 = R(DW3(Conv1(F)))

[0033] S1 = R(DW5(Conv1(F))),

[0034]

[0035] Where R is the ReLU activation function, DW is a depthwise separable convolution (with 3×3 and 5×5 kernels respectively) to obtain a wider receptive field and reduce computation, Conv is a 1×1 convolution, and Concat is a channel concatenation operation. This is an element-wise addition operation;

[0036] The main encoder includes an infrared path and a visible light path;

[0037] Furthermore, both the infrared and visible light paths include 1×1 convolution kernels, downsampling operations, four shared gradient residual invertible neural network modules, and four channel spatial fusion modules. In the infrared path, the 1×1 convolution kernel, downsampling, the first gradient residual invertible neural network module, and the first channel spatial fusion module are connected in sequence; the second gradient residual invertible neural network module and the second channel spatial fusion module are connected in sequence; the third gradient residual invertible neural network module and the third channel spatial fusion module are connected in sequence; and the fourth gradient residual invertible neural network module and the fourth channel spatial fusion module are connected in sequence. The visible light path is similarly connected.

[0038] The input infrared and visible light images are respectively processed by 1×1 convolution and downsampling, and then sequentially passed through the first gradient residual invertible neural network module, the first channel spatial fusion module, the second gradient residual invertible neural network module, the second channel spatial fusion module, the third gradient residual invertible neural network module, the third channel spatial fusion module, and the fourth gradient residual invertible neural network module and the fourth channel spatial fusion module to obtain the infrared image features and visible light image features fused with the global information features in the auxiliary encoding.

[0039] Furthermore, the number of input and output channels of the gradient residual invertible neural network module is 16, 32, 64, 128 and 32, 64, 128, 256, respectively. Among them, the fourth gradient residual invertible neural network module is stacked twice, and the rest are stacked once.

[0040] The gradient residual invertible neural network module includes a fine-grained information extraction path and an edge information extraction path. The fine-grained information extraction path includes a 1×1 convolution, BN, a 4×4 convolution, a LeakyReLU activation function, and an invertible neural network. The edge information path includes a depthwise separable convolution (DWConv) and a Scharr gradient operator. The input infrared and visible light features are first processed by a 1×1 convolution, BN, and LeakyReLU, and then processed by an invertible neural network to obtain fine-grained features. Next, the input features are processed by a depthwise separable convolution and a Scharr gradient operator to obtain edge information features. The two features are added together and then processed by a 4×4 convolution and a LeakyReLU activation function to finally output infrared local information features and visible light local information features, respectively.

[0041] Furthermore, the reversible neural network includes three reversible residual modules. Each reversible residual module includes a 1×1 convolution, a 3×3 depthwise separable convolution, an activation function (ReLU6), and an element-wise addition operation. The input features are divided into two equal parts, F1 and F2, according to the channel dimension. One part undergoes a 1×1 convolution, a ReLU6 operation, a 3×3 depthwise separable convolution, a ReLU6 operation, a 1×1 convolution, and a ReLU6 operation, and finally, it is added to the original input features F1 element-wise to obtain the output feature Y1. Then, the first part of the output feature Y1 is multiplied by the feature exponent obtained from the second reversible residual block and the second part F2 to obtain the feature Y. 12 The output feature Y2 is obtained by adding the feature obtained from the second part through the third residual module. The calculation can be expressed as:

[0042] Y1 = IR(F1), Y2=IR(F2)+Y 12 ,

[0043] Among them, IR is the reversible residual module, which extracts detailed information through the reversible residual network and improves the model's ability to extract high-frequency information;

[0044] The channel spatial fusion module includes a channel attention path, a spatial attention path, and an information aggregation path. Input local information is enhanced with spatial attention to obtain enhanced local detail features, input global information is enhanced with channel attention to obtain global information, and local and global information are processed by Hadamard to obtain interactive information, effectively capturing global and local contextual information.

[0045] The channel attention includes a 1×1 convolution kernel, a ReLU activation function, and a Sigmoid activation function. The input features are sequentially processed by a 1×1 convolution, a ReLU activation function, a 1×1 convolution, and a Sigmoid activation function to obtain a channel attention map. The channel attention map is then multiplied element-wise with the input features to obtain the global features.

[0046] The spatial attention includes a 7×7 convolution kernel, average pooling, max pooling, concatenation, and a sigmoid activation function. The input features are processed by average pooling and max pooling, then concatenated along the channel dimension, and then processed by a 7×7 convolution and a sigmoid activation function to obtain a spatial attention map. The spatial attention map is then multiplied element-wise with the input features to obtain local detail features.

[0047] After concatenating local detail features, global features, and interactive features, local-global features are obtained. The main encoder can extract a total of 4 sets of infrared and visible light local-global features. After concatenating the local-global features of each set of infrared and visible light, 4 sets of pre-fused features are finally obtained.

[0048] Furthermore, the decoder includes four stages, each of which operates exactly the same and is connected sequentially. Each stage includes a concatenation operation, a deconvolution with a 4×4 kernel, a stride of 2, and padding of 1, a BatchNorm operation, and a ReLU activation function. In each stage, the input features are upsampled after passing through deconvolution, BatchNorm, and the ReLU activation function.

[0049] First, the fourth group of pre-fused features is input into the first stage and concatenated with the third group of fused features in the channel dimension to obtain the fused features. The third, second, and first groups of pre-fused and fused features are then processed through the second, third, and fourth stages to obtain the final fused image. In the fourth stage, the ReLU activation function is replaced with Tanh.

[0050] Furthermore, the infrared and visible light image fusion method based on a multi-scale hybrid converter is as follows:

[0051] 32 pairs of images were selected from the TNO dataset as the dataset. The gray values ​​of the images were converted to [-1,1]. The images were cropped using a 128×128 window with a stride of 32, resulting in 6184 pairs of image patches as the training set.

[0052] Set the loss function L total , represented as:

[0053] L total =λ1L SSIM +λ2L patchNCE

[0054] Where λ1, λ2, and λ3 are all hyperparameters, L SSIM For structural similarity loss, L patchNCE To compare the losses;

[0055] Based on the training set and loss function L total The initial infrared and visible light image fusion model based on the multi-scale hybrid converter was trained. The Adam optimizer was used to update the network model parameters during the training process until the training was completed, resulting in a trained infrared and visible light image fusion model based on the multi-scale hybrid converter.

[0056] The beneficial effects achieved by this invention are as follows:

[0057] 1. This invention overcomes the lack of modeling for long-range dependent features in convolutional neural network models. This invention uses Transformer as an auxiliary encoder and convolutional neural network as the main encoder. While maintaining sensitivity to local context information, it supplements the learning of global information, guides the main encoder to learn richer features, and the model has better generalization ability.

[0058] 2. This invention develops a multi-scale hybrid attention module (MSHA) and constructs a novel multi-scale hybrid transformer framework (MHformer). It also introduces channel and wide spatial dimension interaction attention modules and channel and high spatial dimension interaction attention modules to capture cross-channel interaction between channel dimension and spatial dimension, thereby eliminating the boundary ambiguity caused by self-attention, enhancing information exchange, and encoding more accurate spatial information.

[0059] 3. This invention develops a multi-scale feedforward network to replace the traditional multilayer perceptron structure. It aggregates and fuses richer features of the image based on multi-scale information, improves the model's anti-interference ability, and overcomes the problem of insufficient multi-scale information extraction by general Transformers.

[0060] 4. This invention develops a gradient residual invertible neural network module, which gradually extracts local contextual information and fine-grained features of an image through an invertible neural network, supplements the learning of gradient magnitude information through gradient operators, and integrates the two features in a residual connection manner to improve the ability to represent edge contours and details.

[0061] 5. This invention introduces a channel space fusion module. During the feature extraction process, by simultaneously utilizing the different characteristics of convolutional neural networks and transformers, the fusion capability is improved and made more compact. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the overall fusion network of the method described in this invention;

[0063] Figure 2 This is a schematic diagram of a multi-scale hybrid converter structure;

[0064] Figure 3 A schematic diagram of the spatial interaction attention module structure for channels and heights;

[0065] Figure 4 A schematic diagram of the spatial interaction attention module structure for channels and widths;

[0066] Figure 5 This is a schematic diagram of the channel space fusion module structure;

[0067] Figure 6 This is a schematic diagram of a gradient residual invertible neural network structure;

[0068] Figure 7 This is a schematic diagram of a multi-scale feedforward network structure.

[0069] Figure 8 This is a comparison diagram of the first set of fused images and the source image (2_men_in_front_of_house);

[0070] Figure 9 This is a schematic diagram comparing the second set of fused images with the source image (soldier_in_trench_1). Detailed Implementation

[0071] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0072] An infrared and visible light image fusion method based on a multi-scale hybrid transformer includes the following stages:

[0073] S1, as Figure 1 As shown, an infrared and visible light image fusion model framework based on a multi-scale hybrid converter is constructed. The model framework mainly consists of a master encoder, an auxiliary encoder, and a decoder. Both the master encoder and the auxiliary encoder contain infrared and visible light paths. The master encoder extracts local contextual information of infrared and visible light, while the auxiliary encoder extracts global information of infrared and visible light. The auxiliary encoder guides the master encoder to effectively fuse global and local information, so that the fused image contains richer source image features.

[0074] S11, such as Figure 1 As shown, an auxiliary encoder is constructed. The auxiliary encoder includes an infrared path and a visible light path;

[0075] Both the infrared path and the visible light path include stage 1, stage 2, stage 3 and stage 4, and each stage is connected sequentially.

[0076] Furthermore, the number of input channels and output channels in each stage are 16, 32, 64, 128 and 32, 64, 128, 256, respectively;

[0077] Each of the four stages includes a downsampling operation (patch embedding) and a multi-scale hybrid attention block (MSHA). Infrared and visible light images are input into the auxiliary encoder, respectively, and pass through each stage to obtain global infrared information features and global visible light information features.

[0078] Furthermore, stages 1, 2, and 4 stack downsampling and multi-scale hybrid attention modules twice, and stage 3 stacks downsampling and multi-scale hybrid attention modules four times.

[0079] The downsampling operation doubles the number of channels in the input feature and halves the height and width of the input feature.

[0080] S12, as Figure 2 As shown, a multi-scale hybrid transformer is constructed. The multi-scale hybrid attention module includes a window multi-head self-attention module (W-MSA), a transform window multi-head self-attention module (SW-MSA), a channel and high spatial dimension interactive attention module, a channel and wide spatial dimension interactive attention module, a multi-scale feedforward network, and a layer normalization operation (Layernorm).

[0081] The windowed multi-head self-attention module (W-MSA), the channel-and-high spatial dimension interactive attention module, and the channel-and-wide spatial dimension interactive attention module are connected in parallel. The input features F are stacked for the first time. l-1 After layer normalization, standardized features are obtained. Then, after triple attention, these features are added to the input features to obtain a hybrid attention feature map F. l The hybrid attention feature map is then processed through layer normalization and a multi-scale feedforward network to obtain features, which are then added to the hybrid attention feature map to obtain the multi-scale hybrid attention feature H. l The calculation formula can be expressed as:

[0082] F l =W-MSA(L(F l-1 ))+scale×(CW(L(F l-1 ))+CH(L(F l-1 )))+F l-1 H l =MSFFN(LN(F l ))+F l ,

[0083] Second stack, input features F1 l-1 After layer normalization, standardized features are obtained. Then, these features are added to the input features through triple attention to obtain the hybrid attention feature map F1. l The hybrid attention feature map is then processed through layer normalization and a multi-scale feedforward network to obtain features, which are then added to the hybrid attention feature map to obtain the multi-scale hybrid attention feature H1. l The calculation formula can be expressed as:

[0084] F1 l =SW-MSA(L(F1) l-1 ))+scale×(CW(L(F1l-1 ))+CH(L(F1 l-1 )))+F1 l-1 H1 l =MSFFN(LN(F1) l ))+F1 l ,

[0085] W-MSA is a multi-head self-attention window, SW-MSA is a multi-head self-attention window transformed window, both designed to capture long-distance dependencies and obtain high-level semantic features. L is the LayerNorm operation, designed to improve gradient propagation and enhance generalization ability. CW is channel-wide spatial dimension interaction attention, CH is channel-high spatial dimension interaction attention. Both emphasize multi-dimensional interaction without reducing the importance of dimensions, eliminating the indirect correspondence between channels and weights. MSFFN is a multi-scale feedforward network, designed to obtain features at different scales of the image and obtain richer image information. scale is a hyperparameter, here scale is 0.5, where stage 3 is 4 stacks, that is, repeating the above operation twice.

[0086] S13, as Figure 3 As shown, a spatial dimension interaction attention module is constructed, encompassing both channel and height dimensions. This module includes a Permute operation, a Z-Pool operation, a 7×7 convolution, batch normalization, and a Sigmoid activation function. Input feature F1 is transformed into H×C×W dimension features through the Permute operation, followed by a Z-Pool operation in the H dimension, then a 7×7 convolution, batch normalization, and finally a Sigmoid activation function to obtain an attention map. This attention map is multiplied by the transformed features, and then transformed again through the Permute operation to obtain the C×H×W dimension spatial interaction attention feature F2. The calculation can be expressed as:

[0087] F2=P(P(F1)·Sig(Conv(Z(P(F1))))),

[0088] Where P is the Permute operation, Z is the Z-Pool operation, Sig is the Sigmoid activation function operation, and Conv is the convolution operation, this module enhances the ability to exchange spatial information by capturing the cross-channel interaction between channels and spatial dimension H, while encoding more accurate spatial information.

[0089] S14, as Figure 4As shown, a spatial dimension interaction attention module is constructed, encompassing both channel and width dimensions. This module includes a Permute operation, a Z-Pool operation, a 7×7 convolution, batch normalization, and a Sigmoid activation function. Input feature F3 is transformed into W×H×C dimensions using the Permute operation, followed by a Z-Pool operation on the W dimension, then a 7×7 convolution, batch normalization, and finally a Sigmoid activation function to obtain an attention map. This attention map is multiplied by the transformed features and then transformed again using the Permute operation to obtain a C×H×W dimension spatial interaction attention feature F4. The calculation can be expressed as follows:

[0090] F4=P(P(F3)·Sig(Conv(Z(P(F3))))),

[0091] Where P is the Permute operation, Z is the Z-Pool operation, Sig is the Sigmoid activation function operation, and Conv is the convolution operation. This module enhances the ability to exchange spatial information by capturing the cross-channel interaction between channels and spatial dimension W, while encoding more accurate spatial information.

[0092] S15, such as Figure 7 As shown, a multi-scale feedforward network is constructed. This multi-scale feedforward network includes 1×1 convolutions, 3×3 depthwise separable convolutions, 5×5 depthwise separable convolutions, a ReLU activation function, and a concatenation operation. The input features pass through two parallel branches of different scales. In one branch, a 1×1 convolution, a 3×3 depthwise separable convolution, and a ReLU are sequentially connected. In the other branch, a 1×1 convolution, a 5×5 depthwise separable convolution, and a ReLU are sequentially connected. The input feature F first passes through a 1×1 convolution kernel, a 3×3 depthwise separable convolution kernel, and a ReLU. Convolution and ReLU activation function yield feature M1. The input feature is passed through a 1×1 convolution kernel, a 5×5 depthwise separable convolution, and a ReLU activation function to obtain feature S1. These two features are then concatenated, and then passed through a 3×3 depthwise separable convolution and a ReLU activation function, and a 5×5 depthwise separable convolution and a ReLU activation function, respectively, to obtain features M2 and S2. Features M2 and S2 are then concatenated, passed through a 1×1 convolution, and then element-wise added to the input feature to obtain the multi-scale feature MS. The calculation can be expressed as:

[0093] M1 = R(DW3(Conv1(F)))

[0094] S1 = R(DW5(Conv1(F))),

[0095]

[0096] Where R is the ReLU activation function, DW is a depthwise separable convolution (with 3×3 and 5×5 kernels respectively) to obtain a wider receptive field and reduce computation, Conv is a 1×1 convolution, and Concat is a channel concatenation operation. This is an element-wise addition operation;

[0097] S2, as Figure 1 As shown, a main encoder is constructed, and the global information from the auxiliary encoder in S1 is guided into the main encoder for effective fusion. The main encoder includes an infrared path and a visible light path;

[0098] Furthermore, both the infrared and visible light paths include 1×1 convolution kernels, downsampling operations, four shared gradient residual invertible neural network modules, and four channel spatial fusion modules. In the infrared path, the 1×1 convolution kernel, downsampling, the first gradient residual invertible neural network module, and the first channel spatial fusion module are connected in sequence; the second gradient residual invertible neural network module and the second channel spatial fusion module are connected in sequence; the third gradient residual invertible neural network module and the third channel spatial fusion module are connected in sequence; and the fourth gradient residual invertible neural network module and the fourth channel spatial fusion module are connected in sequence. The visible light path is similarly connected.

[0099] The input infrared and visible light images are respectively processed by 1×1 convolution and downsampling, and then sequentially passed through the first gradient residual invertible neural network module, the first channel spatial fusion module, the second gradient residual invertible neural network module, the second channel spatial fusion module, the third gradient residual invertible neural network module, the third channel spatial fusion module, and the fourth gradient residual invertible neural network module and the fourth channel spatial fusion module to obtain the infrared image features and visible light image features fused with the global information features in the auxiliary encoding.

[0100] Furthermore, the number of input and output channels of the gradient residual invertible neural network module is 16, 32, 64, 128 and 32, 64, 128, 256, respectively. Among them, the fourth gradient residual invertible neural network module is stacked twice, and the rest are stacked once.

[0101] S21, as Figure 6As shown, a gradient residual invertible neural network module is constructed. This module includes a fine-grained information extraction path and an edge information extraction path. The fine-grained information extraction path includes a 1×1 convolution, Batch Normalization (BN), a 4×4 convolution, a LeakyReLU activation function, and an invertible neural network. The edge information path includes a depthwise separable convolution (DWConv) and a Scharr gradient operator. The input infrared and visible light features are first processed by a 1×1 convolution, BN, and LeakyReLU, then by the invertible neural network to obtain fine-grained features. Next, the input features are processed by the depthwise separable convolution and the Scharr gradient operator to obtain edge information features. The two features are added together and then processed by a 4×4 convolution and a LeakyReLU activation function, finally outputting infrared local information features and visible light local information features, respectively.

[0102] Furthermore, the reversible neural network includes three reversible residual modules. Each reversible residual module includes a 1×1 convolution, a 3×3 depthwise separable convolution, an activation function (ReLU6), and an element-wise addition operation. The input features are divided into two equal parts, F1 and F2, according to the channel dimension. One part undergoes a 1×1 convolution, a ReLU6 operation, a 3×3 depthwise separable convolution, a ReLU6 operation, a 1×1 convolution, and a ReLU6 operation, and finally, it is added to the original input features F1 element-wise to obtain the output feature Y1. Then, the first part of the output feature Y1 is multiplied by the feature exponent obtained from the second reversible residual block and the second part F2 to obtain the feature Y. 12 The output feature Y2 is obtained by adding the feature obtained from the second part through the third residual module. The calculation can be expressed as:

[0103] Y1 = IR(F1), Y2=IR(F2)+Y 12 ,

[0104] Among them, IR is the reversible residual module, which extracts detailed information through the reversible residual network and improves the model's ability to extract high-frequency information;

[0105] S22, as Figure 5 As shown, a channel spatial fusion module is constructed. This module includes a channel attention path, a spatial attention path, and an information aggregation path. Input local information is enhanced with spatial attention to obtain enhanced local detail features, while input global information is enhanced with channel attention to obtain global information. The local and global information are processed by Hadamard to obtain interactive information, effectively capturing both global and local contextual information.

[0106] The channel attention includes a 1×1 convolution kernel, a ReLU activation function, and a Sigmoid activation function. The input features are sequentially processed by a 1×1 convolution, a ReLU activation function, a 1×1 convolution, and a Sigmoid activation function to obtain a channel attention map. The channel attention map is then multiplied element-wise with the input features to obtain the global features.

[0107] The spatial attention includes a 7×7 convolution kernel, average pooling, max pooling, concatenation, and a sigmoid activation function. The input features are processed by average pooling and max pooling, then concatenated along the channel dimension, and then processed by a 7×7 convolution and a sigmoid activation function to obtain a spatial attention map. The spatial attention map is then multiplied element-wise with the input features to obtain local detail features.

[0108] After concatenating local detail features, global features, and interactive features, local-global features are obtained. The main encoder can extract a total of 4 sets of infrared and visible light local-global features. After concatenating the local-global features of each set of infrared and visible light, 4 sets of pre-fused features are finally obtained.

[0109] S3, as Figure 1 As shown, a decoder is constructed, and the output of the S2 master encoder is input into the decoder to complete image reconstruction. The decoder consists of four stages, each with identical operations, and the stages are connected sequentially. Each stage includes a concatenation operation, a deconvolution with a 4×4 kernel, a stride of 2, and padding of 1, a BatchNorm operation, and the ReLU activation function. In each stage, the input features are upsampled after passing through deconvolution, BatchNorm, and the ReLU activation function.

[0110] First, the fourth group of pre-fused features is input into the first stage and concatenated with the third group of fused features in the channel dimension to obtain the fused features. The third, second, and first groups of pre-fused and fused features are then processed through the second, third, and fourth stages to obtain the final fused image. In the fourth stage, the ReLU activation function is replaced with the Tanh hyperbolic tangent function.

[0111] S4, Network Model Training. Using TNO images as the training dataset, grayscale conversion and data augmentation are performed on the source images of the two input modalities. The network model is trained using structural similarity and contrast loss to obtain the network model parameters.

[0112] S41, Dataset Preprocessing. 32 pairs of images were selected from the TNO dataset as the dataset. The grayscale values ​​of the images were converted to [-1, 1]. The images were cropped using a 128×128 window with a stride of 32, resulting in 6184 pairs of image patches as the training set.

[0113] S5, Set the loss function. The loss function formula can be expressed as:

[0114] L total =λ1L SSIM +λ2L patchNCE ,

[0115] Where λ1 and λ2 are both hyperparameters.

[0116] The above structural similarity loss L SSIM Structural similarity is represented as

[0117]

[0118] Where μ represents the pixel mean, σ xy Let C1 represent the covariance and σ represent the variance. C1 and C2 are used to avoid... The minimum value set close to zero is typically 0.01. 2 0.03 2 L SSIM Represented as:

[0119]

[0120]

[0121]

[0122] Where I x I y and I f These represent the visible light image, infrared image, and fused image, respectively. W represents the sliding window with a step size of 1, and P... i The value of pixel i is represented by m and n, which represent the size of the sliding window. In this invention, the window is set to 16×16.

[0123] The aforementioned contrast loss L patchNCE Represented as

[0124]

[0125]

[0126] Where k represents the encoded feature sample, k + Let k represent a positive sample similar to k. - Let represent a negative sample that is dissimilar to k, and τ represent the temperature coefficient, typically taken as τ = 0.07. S represents the number of sampling locations in the image feature layer, where s ∈ {1, 2, ..., S}, and D... sThis represents any feature sequence in the channel space. By calculating the similarity between a certain region in the space and positive and negative samples, the most similar parts of the fused image and the source image are effectively preserved.

[0127] The Adam optimizer used in the training process updates the network model parameters, with the learning rate set to 0.01, Epoch set to 30, and Batchsize set to 22.

[0128] Furthermore, to verify the fusion effect of infrared and visible light images based on the multi-scale hybrid converter trained by the above method, this embodiment of the invention also verifies the trained infrared and visible light image fusion model based on the multi-scale hybrid converter from both quantitative and qualitative aspects.

[0129] Specifically, during the testing phase, 20 sets of images from TNO (excluding the training set) were selected for testing and verification. Seven typical methods were chosen for comparison, including DenseFuse, U2Fusion, RFN-Nest, SEDRFuse, IFCNN, GANMcC, and CSF. Furthermore, six quantitative evaluation metrics were used: information entropy (EN), average gradient (AG), spatial frequency (SF), mutual information (MI), standard deviation (SD), and visual fidelity (VIF). The verification results included both qualitative and quantitative evaluations.

[0130] (1) Qualitative evaluation. Figure 8 and Figure 9 Two sets of TNO images are presented. By comparison, two advantages of this invention can be observed. The fused image of this invention can better capture thermal radiation information, and the brightness contrast of the person in the infrared image is largely preserved, such as... Figure 8 and Figure 9 The target is a person. Secondly, the fused image of this invention can better represent the detailed textures and background features in visible light images, for example... Figure 8 The brightness of the mid-sky and the outline of the bush texture, for example Figure 9 The color of the trench top is closer to that of light, resulting in better resistance to infrared interference and the absence of image artifacts. Finally, the fused image of this invention can acquire richer source image information, effectively overcoming the limitations of a single sensor, and producing a more natural and clear fused image.

[0131] (2) Quantitative Evaluation. Table 1 presents the objective comparison results of 20 sets of images from the TNO dataset. The best and second-best average values ​​are marked with bold and underline, respectively. It can be seen that the present invention achieves the best average values ​​of EN, MI, SD, AG, VIF, and SF. The maximum values ​​of EN and MI indicate that the channel spatial fusion module and the multi-scale feedforward network of the present invention can better fuse local and global information from the source images, making the fused image contain richer information. The maximum values ​​of AG, SF, and SD indicate that the gradient residual invertible network can extract rich detail and texture information, and the self-attention and two spatial interaction modules can encode more accurate spatial information. The maximum value of VIF indicates that the fused image of the present invention has a good visual effect, and the image is clearer and more natural. The present invention guides the convolutional neural network to reconstruct the fused image features through a multi-scale hybrid transformer, effectively capturing low-level spatial features and high-level semantic context, and modeling from local and global perspectives. The fused image has stronger thermal radiation information and texture detail representation capabilities.

[0132] Table 1. Quantitative comparison results of 20 sets of images from the TNO dataset.

[0133]

Claims

1. A method for fusion of infrared and visible light images based on a multi-scale hybrid converter, characterized in that, Includes the following steps: An infrared and visible light image fusion model framework based on a multi-scale hybrid transformer is constructed. The model framework includes an auxiliary encoder, a main encoder, and a decoder. The auxiliary encoder extracts global information of infrared and visible light, the main encoder extracts local contextual information of infrared and visible light, and the auxiliary encoder guides the main encoder to fuse global and local information so that the fused image contains multi-scale source image features. The multi-scale source image features are input into the decoder to complete image reconstruction. The auxiliary encoder includes four stages, each of which includes downsampling operations and a multi-scale hybrid attention module. Infrared and visible light images are input into the auxiliary encoder and pass through the four stages in sequence to obtain global infrared information features and global visible light information features, respectively. Stage 1, Stage 2, and Stage 4 stack downsampling and multi-scale hybrid attention modules twice, while Stage 3 stacks downsampling and multi-scale hybrid attention modules four times. The multi-scale hybrid attention module includes a window multi-head self-attention module, a transform window multi-head self-attention module, a channel and height spatial dimension interaction attention module, a channel and width spatial dimension interaction attention module, a multi-scale feedforward network, and a layer normalization operation; the window multi-head self-attention module, the channel and height spatial dimension interaction attention module, and the channel and width spatial dimension interaction attention module are connected in parallel. The stacking of phases 1, 2, and 4 twice includes: First stacking, input feature F l-1 After layer normalization, standardized features are obtained. Then, after triple attention, these features are added to the input features to obtain a hybrid attention feature map F. l Hybrid attention feature map F l The features are then processed sequentially through layer normalization and a multi-scale feedforward network, and then combined with the hybrid attention feature map F. l Adding them together yields the multi-scale hybrid attention feature H. l The formula is expressed as: F l =W-MSA(L(F l-1 ))+scale×(CW(L(F l-1 ))+CH(L(F l-1 )))+F l-1 , H l =MSFFN(LN(F l ))+F l ; Second stack, input features F1 l-1 After layer normalization, standardized features are obtained. Then, these features are added to the input features through triple attention to obtain the hybrid attention feature map F1. l Hybrid attention feature map F1 l The features are then processed sequentially through layer normalization and a multi-scale feedforward network, and then combined with the hybrid attention feature map F1. l The summation yields the multi-scale hybrid attention feature H1. l The formula is expressed as: F1 l =SW-MSA(L(F1 l-1 ))+scale×(CW(L(F1 l-1 ))+CH(L(F1 l-1 )))+F1 l-1 , H1 l =MSFFN(LN(F1 l ))+F1 l ; Where W-MSA is window multi-head self-attention, SW-MSA is transform window multi-head self-attention, L is LayerNorm operation, CW is channel and wide spatial dimension interactive attention, CH is channel and high spatial dimension interactive attention, MSFFN is multi-scale feedforward network, scale is hyperparameter, and stage 3 is four stacks, that is, repeating the above operations twice.

2. The infrared and visible light image fusion method based on a multi-scale hybrid converter according to claim 1, characterized in that, The spatial dimension interaction attention module includes Permute operation, Z-Pool operation, 7×7 convolution, batch normalization, and activation function operation. Input feature F1 is transformed into H×C×W dimension features through the Permute operation, then Z-Pool is performed on the H dimension, followed by 7×7 convolution and batch normalization, and finally a Sigmoid activation function to obtain an attention map. The attention map is multiplied by the transformed features, and then transformed again through the Permute operation to obtain the C×H×W dimension spatial interaction attention feature F2, expressed as: F2=P(P(F1)Sig(Conv(Z(P(F1))))), Where P is the Permute operation, Z is the Z-Pool operation, Sig is the Sigmoid activation function operation, and Conv is the convolution operation.

3. The infrared and visible light image fusion method based on a multi-scale hybrid converter according to claim 1, characterized in that, The channel-wide spatial dimension interactive attention module includes a Permute operation, a Z-Pool operation, a 7×7 convolution, batch normalization, and activation function operations. Input feature F3 is transformed into W×H×C dimension features through the Permute operation, then Z-Pool is performed on the W dimension, followed by a 7×7 convolution and batch normalization, and finally a Sigmoid activation function to obtain an attention map. The attention map is multiplied by the transformed features, and then transformed again through the Permute operation to obtain the C×H×W dimension spatial interactive attention feature F4, expressed as: F4=P(P(F3)Sig(Conv(Z(P(F3))))), Where P is the Permute operation, Z is the Z-Pool operation, Sig is the Sigmoid activation function operation, and Conv is the convolution operation.

4. The infrared and visible light image fusion method based on a multi-scale hybrid converter according to claim 1, characterized in that, The multi-scale feedforward network includes a 1×1 convolution, a 3×3 depthwise separable convolution, a 5×5 depthwise separable convolution, a ReLU activation function, and a concatenation operation. The input features are processed through two parallel branches of different scales. In one branch, a 1×1 convolution, a 3×3 depthwise separable convolution, and a ReLU activation function are sequentially connected. In the other branch, a 1×1 convolution, a 5×5 depthwise separable convolution, and a ReLU activation function are sequentially connected. The input feature F first passes through a 1×1 convolution kernel, a 3×3 depthwise separable convolution, and a ReLU activation function to obtain feature M1. The input feature then passes through a 1×1 convolution kernel, a 5×5 depthwise separable convolution, and a ReLU activation function to obtain feature S1. The two features are then concatenated and then passed through a 3×3 depthwise separable convolution and a ReLU activation function, and a 5×5 depthwise separable convolution and a ReLU activation function, respectively, to obtain features M2 and S2. Features M2 and S2 are then concatenated and passed through a 1×1 convolution, and then added element-wise with the input feature to obtain the multi-scale feature MS. The formula is expressed as: M1 = R(DW3(Conv1(F))) S1 = R(DW5(Conv1(F))) Where R is the ReLU activation function, DW is the depthwise separable convolution, Conv is a 1×1 convolution, and Concat is the channel concatenation operation. This is an element-wise addition operation.

5. The infrared and visible light image fusion method based on a multi-scale hybrid converter according to claim 1, characterized in that, The main encoder includes an infrared path and a visible light path; Both the infrared path and the visible light path include a 1×1 convolution, a downsampling operation, four shared gradient residual invertible neural network modules, and four channel spatial fusion modules; The input infrared and visible light images are respectively processed by 1×1 convolution and downsampling, and then sequentially processed by the first gradient residual invertible neural network module, the first channel spatial fusion module, the second gradient residual invertible neural network module, the second channel spatial fusion module, the third gradient residual invertible neural network module, the third channel spatial fusion module, the fourth gradient residual invertible neural network module, and the fourth channel spatial fusion module to obtain the infrared image features and visible light image features fused with the global information features in the auxiliary encoder.

6. The infrared and visible light image fusion method based on a multi-scale hybrid converter according to claim 5, characterized in that, The gradient residual invertible neural network module includes a fine-grained information extraction path and an edge information extraction path. The fine-grained information extraction path includes a 1×1 convolution, Batch Normalization (BN), a 4×4 convolution, a LeakyReLU activation function, and an invertible neural network. The edge information path includes a depthwise separable convolution and a Scharr gradient operator. The input infrared and visible light features are first processed by a 1×1 convolution, BN, and LeakyReLU, and then by an invertible neural network to obtain fine-grained features. Next, the input features are processed by a depthwise separable convolution and a Scharr gradient operator to obtain edge information features. The two features are added together and then processed by a 4×4 convolution and a LeakyReLU activation function to finally output infrared local information features and visible light local information features, respectively.

7. The infrared and visible light image fusion method based on a multi-scale hybrid converter according to claim 5, characterized in that, The channel spatial fusion module includes a channel attention path, a spatial attention path, and an information aggregation path. Input local information is processed through the spatial attention path to obtain enhanced local detail features, and input global information is processed through the channel attention path to obtain enhanced global information. Local and global information are processed by Hadamard to obtain interaction features. The local detail features, global features, and interaction features are concatenated to obtain local-global features. The main encoder extracts local-global features of infrared and visible light. The local-global features of each group of infrared and visible light are concatenated to obtain pre-fused features.

8. A computer storage medium, characterized in that, The computer storage medium stores a computer program that is executed by a processor to perform the method described in any one of claims 1-7.

Images