Infrared and visible light fusion method based on multi-scale feature interaction enhancement

Abstract

The invention provides an infrared and visible light image fusion method based on multi-scale feature interaction enhancement, belongs to the field of image processing, and aims to improve the image fusion quality in a complex environment. The method comprises the following steps: firstly, performing feature extraction on infrared and visible light images through a double-flow multi-scale encoder to generate multi-level features of different scales; and then, cross-modal information interaction is carried out by adopting a feature interaction enhancement module, fusion features are optimized through a space and channel attention mechanism, and clear expression of key areas such as target heat radiation and image edges is ensured. In the decoding stage, multi-scale jump connection and layer-by-layer up-sampling are utilized to reconstruct a high-resolution fusion image. And finally, training the fusion network through a content-gradient joint loss function to realize balance between detail reservation and global information. The method is suitable for tasks such as monitoring and target identification in a complex scene, and can effectively improve the image fusion effect especially under the conditions of low illumination, smoke shielding and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of image information processing, and in particular relates to an infrared and visible light image fusion method based on multi-scale feature interactive enhancement. Background Art

[0002] Infrared (IR) images and visible light (VIS) images each exhibit unique advantages due to their different imaging mechanisms. Infrared images can capture the thermal radiation information of objects, so they can still clearly display key targets in the scene in complex environments such as low light and smoke. However, infrared images are insufficient in expressing the texture and detail information of objects. In contrast, visible light images have rich details and texture information, but their imaging quality will be significantly reduced under harsh environmental conditions (such as low light or complex background). Therefore, single-modality images are usually difficult to fully reflect all key information in complex scenes, and it is difficult to meet the requirements of practical applications for image quality and information integrity.

[0003] Researchers have developed a variety of image fusion techniques to combine the advantages of infrared (IR) and visible light (VIS) images to generate comprehensive images that contain rich radiation information as well as details and textures. The main fusion methods include pixel-level, feature-level, and deep learning-based fusion methods. Pixel-level fusion methods generate fused images by performing weighted average or arithmetic operations on infrared and visible light images at the pixel level, but when dealing with complex backgrounds and noise, it is easy to cause information loss or blurring, and it is difficult to retain the key features of the two modalities. Feature-level fusion methods generate images with more useful information by extracting and fusing the features of infrared and visible light images. Although this method improves the shortcomings of pixel-level methods to a certain extent, due to the lack of modeling of global information, the correlation between modalities is often ignored, resulting in poor fusion effects. In recent years, deep learning technology, especially fusion methods based on convolutional neural networks (CNNs), has made significant progress in the field of image fusion. These methods improve the quality of fused images by training neural networks to automatically learn fusion strategies. However, traditional CNN models are limited by the size of the receptive field and are difficult to effectively capture long-range dependencies in images. In addition, existing deep learning methods often fail to fully consider the characteristic differences of different modal images and the impact of the environment on image quality, so the fusion effect is still limited in complex environments. This shows that a more advanced image fusion method needs to be developed to overcome the limitations of existing technologies. Summary of the invention

[0004] The purpose of this invention is to propose an infrared and visible light image fusion method based on multi-scale feature interactive screening, aiming to solve the shortcomings of existing image fusion methods when facing low-quality images (such as overexposure, low illumination, smoke occlusion and similar backgrounds), and to achieve high-quality fusion effects by effectively integrating the features of infrared and visible light images.

[0005] A method for fusion of infrared and visible light images based on multi-scale feature interactive enhancement includes the following steps:

[0006] Step S1: Extract multi-scale features. Use a two-stream multi-scale encoder to extract features from infrared images and visible light images respectively, and obtain four sets of infrared and visible light image features of different scales.

[0007] Step S2: Multi-scale feature fusion. The four groups of features of different scales extracted from the infrared image and the visible light image are respectively input into four feature interaction enhancement modules. After the cross-modal feature interaction and attention enhancement mechanism of each module, four groups of fused features of different scales are generated.

[0008] Step S3: Reconstruct the fused image through the decoder. The decoder consists of multiple decoding blocks, each of which shares the same structure as the encoding block. Each decoding block takes the fused features as input, and uses a combination of layer-by-layer upsampling and skip connections to gradually restore the spatial resolution of the image, thereby obtaining a reconstructed fused image.

[0009] Step S4: Train the fusion network based on the visible light image, the infrared image, the fused image and the content-gradient guided loss function. When the loss function converges, a trained fusion network is obtained, and the visible light image and the infrared image are fused based on the network.

[0010] Furthermore, in step S1, a dual-stream encoder is used to extract multi-scale features of infrared and visible light images. Specifically, the following steps are included:

[0011] Step S11: Shallow feature extraction: The infrared image and the visible light image are respectively subjected to preliminary feature extraction through a 1×1 convolutional layer and a ReLU activation layer to obtain their respective shallow features.

[0012] Step S12: Multi-scale feature extraction. After the shallow feature extraction is completed, deep features of different scales are further extracted through a series of encoding modules and their corresponding downsampling operations. The dual-stream encoder design in the present invention includes two independent and parallel processing paths, one specifically for processing infrared image data and the other for processing visible light image data. Each path contains four encoding modules with consistent structures, which are designed to process data of their respective modalities.

[0013] In the infrared path, the processing starts with the first encoding module, which receives the shallow features provided by the previous stage as its input. Subsequently, the output features undergo a downsampling operation, which uses a 3×3 convolution kernel and sets the stride to 2 to reduce the spatial dimension of the feature map to half of the original size and double the number of channels of the feature map. The processed feature map is then passed to the next encoding module, where the obtained output features will continue to undergo a similar process, gradually reducing the spatial resolution to one-quarter and one-eighth of the original size until the fourth encoding module is completed. This design ensures that feature information at different scales can be encoded step by step and fully extract multi-level feature information from it.

[0014] The visible light path and the infrared path use exactly the same processing steps, that is, the visible light image data is processed through the same encoding module structure and downsampling technology. After the above encoding process, the multi-scale features generated by the infrared image and the visible light image are obtained. and (where i∈{1,2,3,4}), represents the multi-level feature information extracted at different scales.

[0015] Step S13: The specific operation process of the encoding module is as follows:

[0016] The encoding module adopts a dual-branch structure, including a CNN branch and a Transformer branch, which are responsible for extracting local details and global context information of the image respectively. in , the CNN branch is used to capture local details, and its calculation formula is as follows:

[0017] F cn =ReLU(Conv(ReLU(Conv 3×3 (X in )))).

[0018] Among them, F cn ReLU(g) represents the output feature, ReLU(g) represents the ReLU activation function, and Conv(g) represents a 3×3 convolution operation with a stride of 1. The Transformer branch obtains global feature information through the self-attention mechanism, and its calculation process is as follows:

[0019] {Q,K,V}={X in W Q ,X in W K ,X in W V}.

[0020] Among them, X inis the input feature of the Transformer branch, Q, K and V represent the query, key and value matrices respectively; W Q , W K , W V represents the learnable weight matrix; d k represents the dimension of the key. The attention mechanism is calculated by the following formula:

[0021]

[0022] Among them, Head i Represents the calculation process of the i-th attention head. This process is extended to multi-head self-attention MSA(·) to consider various attention distributions:

[0023] MSA(Q,K,V)=Concat(Head 1 ,Head 2 ,...,Head h )W O

[0024] Among them, Concat(·) represents the concatenation of channel dimensions, h is the number of attention heads, and W O is the output weight matrix of the multi-head self-attention. The output of the multi-head attention mechanism is processed by layer normalization, and the calculation formula is:

[0025] Z=LN(MSA(Q,K,V))+Q

[0026] Among them, Z represents the output of the intermediate process, and LN(.) represents the normalization operation. Then the final output features are generated through the feedforward neural network and residual connection:

[0027] F tr =LN(FFN(Z))+Z

[0028] Among them, F tr is the output feature of the Transformer branch, and FFN(·) represents a feed-forward neural network.

[0029] Finally, the outputs of the CNN branch and the Transformer branch are concatenated in the channel dimension, and the concatenated features are processed through 1×1 convolution and ReLU activation layers to generate the final output features of the encoding module.

[0030] Furthermore, in step S2, the specific process of multi-scale feature fusion includes:

[0031] Step S21: Cross-modal feature interaction. For each set of infrared features F ir and visible light feature F vis, and are input into the corresponding feature interaction enhancement modules. Each module uses two Transformer blocks to achieve deep interaction of cross-modal information. The specific process is as follows:

[0032] Each group of infrared features F ir and visible light feature F vis Processing is done by computing query, key, and value matrices:

[0033]

[0034] in, and are respectively learnable weight matrices. Between the two Transformer blocks, in each feature interaction enhancement module, the key and value matrices of infrared and visible light are exchanged respectively to achieve information exchange between the modalities, thereby making full use of their respective feature advantages. Specifically: for infrared features:

[0035]

[0036] For visible light features:

[0037]

[0038] Among them, Head i,ir and Head i,vis Represents the calculation results of the i-th attention head in infrared features and visible light features respectively. Use multi-head self-attention to aggregate the outputs of multiple attention heads:

[0039]

[0040] in, and is the multi-head attention output weight matrix. Then it is processed through the feedforward network layer and the normalization layer to generate the final interactive enhancement feature:

[0041] Z ir =LN(MSA ir )+Q ir

[0042] Z vis =LN(MSA vis )+Q vis

[0043] F tr,ir =LN(FFN(Z ir ))+Z ir

[0044] F tr,vis =LN(FFN(Z vis ))+Zvis

[0045] Among them, F tr,ir and F tr,vis They are the outputs after information interaction.

[0046] Through this step, four sets of interaction features of different scales are generated.

[0047] Step S22: Each group of interaction features F tr,ir and F tr,vis First, concatenate in the channel dimension to obtain the integrated feature F mix , and then enter the spatial attention and channel attention modules at the same time to further optimize the detailed expression of the fused features. The spatial attention mechanism analyzes the spatial distribution of the input features and highlights the areas that are critical to the fused image. Specifically, spatial attention can effectively capture the target thermal radiation area and edge information in the image.

[0048] The calculation formula of the spatial attention mechanism is:

[0049] F sa =σ(Conv(Concat(AvgPool(F in ),MaxPool(F mix ))))

[0050] Among them, AvgPool(·) and MaxPool(·) represent the global average pooling and maximum pooling operations respectively, σ is the Sigmoid activation function, and F out is the output spatial attention feature.

[0051] The channel attention mechanism analyzes the relative importance of different channels and dynamically adjusts the weight of each channel to enhance the expression of high-value channels and suppress redundant information. The calculation formula of channel attention is:

[0052] F ca =σ(MLP(AvgPool(F mix )))

[0053] Among them, σ is the Sigmoid activation function, MLP represents the multi-layer perceptron, and F ca is the output feature.

[0054] Step S23: Fusion and output. Feature F after spatial and channel attention optimization sa and F ca Concatenate and generate enhanced fusion feature F through a layer of 3×3 convolution operation reThis fusion feature combines important information of space and channel, which helps to improve the detail performance of the fused image. This process is carried out in each feature interaction enhancement module, and finally four fusion features of different scales are obtained. Realize effective fusion results of infrared and visible light information.

[0055] Furthermore, the process of reconstructing the fused image in step S3 includes:

[0056] Step S31: In the decoding stage, the fused features of the fourth layer are first input into the decoding module for processing. Then, after the upsampling operation, the feature is spliced ​​with the fused features of the third layer, and the dimension is reduced through a 3×3 convolution operation (with a stride of 1). The upsampling operation uses a transposed convolution to expand the size of the feature map so that the height and width are both twice the original. Subsequently, the processed features are input into the next decoding module, upsampled after the decoding operation, and spliced ​​with the fused features of the second layer, and another dimensionality reduction is achieved through another 3×3 convolution operation (with a stride of 1). Then, the obtained features are input into the next decoding module, upsampled after the decoding operation, and spliced ​​with the fused features of the first layer, and finally the final dimensionality reduction is completed through a 3×3 convolution operation (with a stride of 1). Finally, the obtained features are input into the last decoding module, and the final fused features are obtained after processing.

[0057] Step S32: The fused features output by the final decoding module undergo a 1×1 convolution operation (with a stride of 1) to generate the final fused image.

[0058] Through multi-scale jump connections, features of different scales are gradually synthesized to reconstruct the details and completeness of the fused image.

[0059] Furthermore, in step S4, the training loss function adopts content-gradient joint loss, and the specific calculation process of the loss function is as follows:

[0060] Step S41: The calculation formula of content loss is:

[0061]

[0062] Among them, I f represents the fused image, I ir and I vis represents infrared and visible light images, H and W represent the height and width of the image respectively.

[0063] Step S42: The gradient loss formula is:

[0064]

[0065] Among them, L gradrepresents the gradient loss function, ▽ represents the Sobel gradient operator, and |·| represents the absolute value operation. Step S43: The overall loss function is a weighted combination of the content loss and the gradient loss:

[0066] L total =L int +α·L grad

[0067] Among them, L total is the overall loss function, and α is a hyperparameter that balances content loss and gradient loss.

[0068] The present invention has the following beneficial effects:

[0069] The present invention significantly improves the fusion effect of infrared and visible light images through multi-scale feature extraction and cross-modal feature interaction enhancement, especially in complex environments such as low light and smoke occlusion. A dual-stream multi-scale encoder is used to extract multi-level features of infrared and visible light respectively, ensuring that the global information and local details of the scene are fully captured at different scales. The cross-modal feature interaction module enhances the information complementarity between modalities through feature exchange and attention mechanism, so that the fused image can simultaneously retain the thermal radiation characteristics of the infrared image and the detailed texture expression of the visible light image. In addition, the spatial and channel attention mechanism further optimizes the feature expression of key areas, such as the thermal characteristics of the target object and the image edge details, thereby improving the overall clarity and information integrity of the image. The decoding stage uses multi-scale jump connections and layer-by-layer upsampling to effectively reconstruct high-resolution fused images, ensuring that the details are restored while maintaining the overall structure of the image. The overall design makes the present invention particularly suitable for monitoring, target recognition and image processing tasks in complex scenes, and can still generate high-quality, information-rich fused images under harsh conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flow chart of the method of the present invention;

[0071] Figure 2 It is the overall network block diagram of the present invention; DETAILED DESCRIPTION

[0072] In order to make the content of the present invention clearer and easier to understand, the present invention is further described in detail below with reference to the accompanying drawings.

[0073] In the implementation of the present invention, Figure 2 As shown, a method for fusion of infrared and visible light images based on interactive screening of multi-scale features comprises the following steps:

[0074] Step S1: extracting multi-scale features, using a dual-stream multi-scale encoder to extract multi-scale features from infrared images and visible light images respectively, to obtain four sets of infrared image and visible light image features of different scales;

[0075] Step S2: Multi-scale feature fusion: The extracted four sets of infrared image and visible light image features of different scales are input into the feature interaction enhancement module to generate fused features through cross-modal feature interaction and attention mechanism;

[0076] Step S3: reconstructing the fused image through a decoder, wherein the decoder generates a high-resolution fused image according to the fused features;

[0077] Step S4: Based on the infrared image, the visible light image, the fused image and the content-gradient loss function, the fusion network is trained to obtain a model for the fusion of infrared and visible light images.

[0078] Specifically, the Figure 2 The overall framework of the proposed infrared and visible light image fusion method is mainly composed of three parts: multi-scale encoder, multi-scale fusion module and multi-scale decoder. In addition, the specific process of the proposed fusion method is as follows:

[0079] In step S1, a dual-stream encoder is used to extract multi-scale features of infrared and visible light images. Specifically, the following steps are included:

[0080] First, multi-scale features are extracted from infrared images and visible light images. Specifically, a dual-stream encoder structure is used to extract features from infrared images and visible light images respectively. The dual-stream encoder contains four encoding modules, each of which is responsible for feature extraction at different scales.

[0081] Shallow feature extraction: First, the infrared image and visible light image are respectively subjected to preliminary feature extraction through a 1×1 convolutional layer to generate shallow features.

[0082] Multi-scale feature extraction: After obtaining shallow features, deep features are further extracted through multiple encoding modules. The output feature map of each encoding module is downsampled to reduce its spatial resolution step by step, while increasing the number of channels, thereby obtaining features at different scales.

[0083] In this step, the multi-scale features of infrared images and visible light images are expressed as and Where i∈{1,2,3,4} represents different scales.

[0084] Furthermore, in step S2, the specific process of multi-scale feature fusion includes:

[0085] The features extracted from infrared images and visible light images are cross-modally fused using a feature interaction enhancement module.

[0086] Cross-modal feature interaction: infrared image features and visible light image features The feature interaction enhancement modules are input separately. Each module contains two Transformer blocks. The infrared features and visible light features realize information interaction between modalities by exchanging key and value matrices. In this way, the advantages of the two modal images can be fully utilized.

[0087] Attention mechanism enhancement: After feature interaction, the fused features are further optimized through spatial attention mechanism and channel attention mechanism. Spatial attention is used to identify key areas in the image (such as thermal radiation targets and edges), while channel attention enhances high-value features and suppresses redundant information by adjusting the weights of each channel.

[0088] Each set of fused features is processed through a 3×3 convolutional layer to generate the final interactive enhancement features.

[0089] Furthermore, the process of reconstructing the fused image in step S3 includes:

[0090] The decoder is used to reconstruct the fused features and restore the spatial resolution of the image.

[0091] Layer-by-layer decoding: The decoder structure is symmetrical to the encoder structure. Each decoding module expands the size of the feature map through upsampling. Upsampling uses transposed convolution to increase the height and width of the image to twice the original, and jumps to the encoded features of the previous layer. This operation enables the reconstruction process to retain feature information at all scales.

[0092] Final reconstruction: Finally, after layer-by-layer decoding and upsampling operations, the generated fused image is converted into the final output image through 1×1 convolution. This image combines the thermal radiation information of the infrared image and the details and texture information of the visible light image, achieving high-quality image fusion.

[0093] Furthermore, in step S4, the training loss function adopts content-gradient joint loss, and the specific calculation process of the loss function is as follows:

[0094] In order to improve the quality of the fused image, a weighted combination of content loss and gradient loss is used as the loss function for training.

[0095] Content loss: Evaluate the pixel-level content difference of the image by comparing the fused image with the input infrared and visible light images.

[0096] Gradient loss: By calculating the gradient information of the image, the fused image is ensured to retain clear edge and structural information. Specifically, the Sobel operator is used to calculate the gradient of the image, and the gradient difference is considered in the loss function.

[0097] The overall loss function is a weighted combination of content loss and gradient loss, which is used to train the network until the loss function converges to obtain a trained fusion network model.

[0098] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.

Claims

1. A method for fusion of infrared and visible light based on interactive screening of multi-scale features, characterized by: The steps include: Step S1: extracting multi-scale features, using a dual-stream multi-scale encoder to extract multi-scale features from infrared images and visible light images respectively, to obtain four sets of infrared image and visible light image features of different scales; Step S2: Multi-scale feature fusion: the extracted four sets of image features of different scales are input into the feature interaction enhancement module to generate fusion features through cross-modal feature interaction and attention mechanism; Step S3: reconstructing the fused image through a decoder, wherein the decoder generates a high-resolution fused image according to the fused features; Step S4: Based on the infrared image, the visible light image, the fused image and the content-gradient loss function, the fusion network is trained to obtain a model for the fusion of infrared and visible light images.

2. The infrared and visible light image fusion method based on multi-scale feature interactive enhancement according to claim 1 is characterized in that: In step S1, a dual-stream encoder is used to extract multi-scale features of infrared and visible light images, specifically including: Step S11: shallow feature extraction, the infrared image and the visible light image are respectively subjected to feature extraction through the convolution layer and the activation layer to obtain their respective shallow features. Step S12: Multi-scale feature extraction. Shallow features are processed through a series of encoding modules and their corresponding downsampling operations to extract deep features of multiple scales. The dual-stream encoder includes two independent parallel paths for processing infrared images and visible light images respectively, and each path contains multiple encoding modules with the same structure. In the infrared image path, shallow features enter the first encoding module for processing, and the output features are processed through downsampling operations to gradually reduce the spatial resolution and deepen the number of channels of the feature map until all encoding modules are processed. The visible light image path uses the same processing steps as the infrared image path to generate multi-scale features of infrared and visible light images. and (where i∈{1,2,3,4}).

3. The infrared and visible light image fusion method based on multi-scale feature interactive enhancement according to claim 2 is characterized in that: The dual-stream encoder includes a CNN branch and a Transformer branch. The CNN branch is used to capture local details, and the Transformer branch is used to extract global context information. The specific implementation steps are as follows: Given the input X of the encoding block in , the CNN branch is used to capture local details, and the calculation formula is as follows: F cn =ReLU(Conv(ReLU(Conv(X in )))). Among them, F cn ReLU(g) represents the output feature of the CNN branch, ReLU(g) represents the ReLU activation function, and Conv(g) represents the convolution operation. The Transformer branch obtains global feature information through the self-attention mechanism, and its calculation process is as follows: {Q,K,V}={X in W Q ,X in W K ,X in W V }. Where Q, K, and V represent query, key, and value matrices, respectively; W Q , W K , W V Represents a learnable weight matrix. The attention mechanism is calculated by the following formula: Among them, Head i represents the output result of the i-th attention head, d k represents the dimension of the key. The process is expanded to multi-head self-attention MSA(·): MSA(Q,K,V)=Concat(Head1,Head2,...,Head h )W O Among them, Concat(·) represents the concatenation operation, h is the number of attention heads, and W O is the output weight matrix of the multi-head self-attention. The output of the multi-head attention mechanism is processed by layer normalization, and the calculation formula is: Z=LN(MSA(Q,K,V))+X in Among them, Z is the normalized output feature, and LN(.) represents the normalization operation. Then the final output feature of the Transformer branch is generated through feedforward neural network processing: F tr =LN(FFN(Z))+Z Among them, FFN(·) represents a feed-forward neural network. Finally, the outputs of the CNN branch and the Transformer branch are concatenated in the channel dimension, and the concatenated features are processed by the convolution layer and the ReLU activation layer to generate the output features of the encoding module.

4. The infrared and visible light image fusion method based on multi-scale feature interactive enhancement according to claim 1 is characterized in that: In step S2, the feature interaction enhancement module uses two Transformer blocks to perform cross-modal information interaction by exchanging the key and value matrices of infrared and visible light. The specific process is as follows: Each group of infrared features F ir and visible light feature F vis Processing is done by computing query, key, and value matrices: in, and are respectively learnable weight matrices. Between the two Transformer blocks, the key and value matrices of infrared and visible light are exchanged respectively to realize information exchange between the modalities, thereby making full use of their respective feature advantages: Among them, Head i,ir and Head i,vis Represents the calculation results of the i-th attention head in infrared features and visible light features respectively. Multi-head self-attention is used to aggregate the outputs of multiple attention heads: in, and is the multi-head attention output weight matrix. Then it is processed through the feedforward network layer and the normalization layer to generate the final interaction features: Z ir =LN(MSA ir )+Q ir Z vis =LN(MSA vis )+Q vis F tr,ir =LN(FFN(Z ir ))+Z ir F tr,vis =LN(FFN(Z vis ))+Z vis Among them, F tr,ir and F tr,vis They are the outputs after information interaction.

5. The infrared and visible light image fusion method based on multi-scale feature interactive enhancement according to claim 5 is characterized in that: The feature interaction enhancement module further includes spatial attention and channel attention mechanisms. The spatial attention is used to highlight key target areas, and the channel attention is used to dynamically adjust the feature channel weights. The calculation formula of the spatial attention mechanism is: F sa =σ(Conv(Concat(AvgPool(F in ),MaxPool(F mix )))) Among them, AvgPool(·) and MaxPool(·) represent the global average pooling and maximum pooling operations respectively, σ is the Sigmoid activation function, and F sa is the spatial attention feature of the output. The channel attention mechanism dynamically adjusts the weight of each channel by analyzing the relative importance of different channels to enhance the expression of high-value channels and suppress redundant information. The calculation formula of channel attention is: F ca =σ(MLP(AvgPool(F mix ))) Among them, MLP represents multi-layer perceptron, F ca is the output feature. The feature F after spatial and channel attention optimization sa and F ca Generate enhanced fusion feature F through concatenation and 3×3 convolution operation re .

6. The infrared and visible light image fusion method based on multi-scale feature interactive enhancement according to claim 1 is characterized in that: In step S3, the decoder restores the spatial resolution of the image through upsampling operation and multi-scale skip connection to generate a fused image. The specific process is as follows: First, the fused features of the fourth layer are input to the decoding module for processing. Then, after the upsampling operation, the feature is spliced ​​with the fused features of the third layer, and the dimension is reduced through the convolution operation. The upsampling operation uses transposed convolution to expand the size of the feature map, so that the height and width become twice the original. Subsequently, the processed features are input to the next decoding module, upsampled after the decoding operation, and spliced ​​with the fused features of the second layer, and the dimension is reduced again through another convolution operation. Then, the obtained features are input to the next decoding module, upsampled after the decoding operation, and spliced ​​with the fused features of the first layer, and finally the final dimension reduction is completed through the convolution operation. Finally, the obtained features are input to the last decoding module, and the final fused features are obtained after processing. The fused features output by the last decoding module are subjected to a layer of 1×1 convolution operation to generate the final fused image.

7. The infrared and visible light image fusion method based on multi-scale feature interactive enhancement according to claim 1 is characterized in that: The training loss function in step S4 is a weighted combination of content loss and gradient loss, specifically: L total =L int +α·L grad Among them, L total is the overall loss function, L int is the content loss function, L grad is the gradient loss function, α is a hyperparameter for balancing content loss and gradient loss, H and W represent the height and width of the image, respectively, ||g||1 represents the l1-norm regularization operation, ▽ represents the Sobel gradient operator, |g represents the absolute value operation, I f represents the fused image, I ir and I vis Represents infrared and visible light images.

Images