A method for fusing infrared and visible light images under low illumination conditions
By employing a dual-attention mechanism involving multi-layer convolution and pooling, combined with multi-head self-attention and cross-attention, the feature extraction challenge of infrared and visible light image fusion under low illumination is solved, achieving a more comprehensive image fusion effect suitable for special scenarios such as underground coal mines.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING TECH & BUSINESS UNIV
- Filing Date
- 2022-12-08
- Publication Date
- 2026-05-12
AI Technical Summary
In low-light environments, the fusion of infrared and visible light images is difficult to effectively extract features, resulting in loss of local details and decreased contrast. Existing methods are difficult to adapt to different fusion strategies.
A dual attention mechanism is adopted, which obtains multi-level feature maps through multi-layer convolution and max pooling layers. Multi-head self-attention and multi-head cross-attention processing are combined to enhance the global interaction and local correlation of feature maps. Feature map fusion is achieved by using decoding convolution blocks and upsampling operations.
Under low-light conditions, it effectively reduces the loss of image details, improves contrast, and achieves a more comprehensive image fusion effect.
Smart Images
Figure CN115861754B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fusion of infrared and visible light, specifically to a fusion method under low-light conditions based on a dual attention mechanism, belonging to the field of cross-modal image fusion. Background Technology
[0002] In the era of information explosion, sensor technology has developed rapidly. The types of sensors are constantly increasing, and the information they capture is becoming increasingly diversified. Due to different imaging mechanisms, different modalities of the same scene contain different information. Infrared sensors image based on the thermal radiation characteristics of objects, but are insensitive to changes in brightness and often lack scene details. Visible light cameras describe object features based on light reflection, offering high resolution and capturing rich texture details, but suffer from significant information loss in low-light conditions. To address the problem of incomplete representation of temporal, spatial, and spectral information in single-sensor imaging systems, image fusion technology has emerged. This technology fully utilizes the complementarity of different sensor imaging systems to generate a more comprehensive fused image of the scene. Therefore, this technology is widely used in military, remote sensing, medical, and security monitoring fields.
[0003] Current infrared and visible light image fusion algorithms can be divided into two categories: traditional fusion algorithms and deep learning-based fusion methods. Among traditional infrared and visible light image fusion methods, multi-scale transformation methods are considered classic due to their focus on multi-scale features across modalities. However, these methods rely excessively on manually extracted features, making it difficult to find universal features to adapt to different fusion strategies. Therefore, in recent years, deep learning-based infrared and visible light image fusion methods have received widespread attention. For low-light environments such as underground coal mines, even using deep learning methods to automatically extract features from infrared and visible light images and then fuse them significantly increases the difficulty. Summary of the Invention
[0004] The purpose of this invention is to realize a method for fusing infrared and visible light images in low-light environments. It uses a dual attention mechanism to achieve global interaction and fusion of infrared and visible light image features, reduce the loss of local details, and achieve spatial restoration of low contrast.
[0005] Specifically, the present invention provides a method for fusing infrared and visible light images under low-light conditions, the steps of which include:
[0006] 1) Perform convolution processing on infrared and visible light heterogeneous source images under low illumination conditions, and then use the cascaded connection of multiple convolution and max pooling layers to obtain multi-level heterogeneous dual-path feature maps.
[0007] 2) Perform Multi-Head Self-Attention (MHSA) on the deepest feature map to extract remote feature information from the feature map; perform Multi-Head Cross Attention (MHCA) on the remaining shallow feature maps to enhance relevant regions in the network connection.
[0008] 3) Standardize the feature maps at different levels and perform MHSA processing to obtain the attention matrix. Then, sum the results pixel by pixel and fuse the two feature maps by concatenation.
[0009] 4) By using multiple decoding convolutional blocks and upsampling operations, feature maps at different levels are fused to obtain a low-light infrared and visible light fused image.
[0010] Furthermore, in step 1), in order to address the problems of severe loss of detail information and significant decrease in contrast of infrared and visible light images under low illumination background, the infrared and visible light heterogeneous source images are processed by one layer of 1×1 convolution and two layers of 3×3 convolution, respectively. Then, one layer of max pooling and two layers of 3×3 convolution are used as a unit, and several groups of such units are cascaded. The feature maps extracted by each group of units constitute multi-level heterogeneous dual-path feature maps.
[0011] Further, in step 2), MHSA processing is performed on the deepest-level feature map to extract long-range feature information. First, the input feature map is positionally encoded, representing it as a two-dimensional matrix in tensor form, while capturing both absolute and relative information between objects in the feature map. The current feature matrix is then processed with three embedding matrices to obtain the query matrix Q, the key matrix K, and the value matrix V. The similarity between all elements in K and Q is calculated and input into the softmax function. The result is then processed by a weighted average of the value matrix V to obtain the attention matrix, which considers all interaction information between K and Q. Finally, the attention matrix is restored to tensor form to obtain the MHSA output.
[0012] Further, in step 2), MHCA processing is performed on the feature maps of the remaining shallow layers to enhance relevant regions in the network connections; the key matrix K and query matrix Q of this layer are calculated from the feature map Y of the previous layer; the value matrix V is calculated from the feature map S input to this layer; the attention matrix A is calculated, and a tensor Z is obtained through 1×1 convolution, batch normalization (BN) processing, sigmoid activation, and upsampling, which has the same size as the feature map S input to this layer; then Z and S are dot-product processed, and the result obtained after position encoding of the feature map Y of the previous layer is convolved and upsampled, and concatenated with the dot-product result to obtain the MHCA output.
[0013] Further, step 3) is implemented as follows: 1×1 convolution is performed on the current input feature map to integrate multi-channel features, and the value range of each element in the feature map is specified through standardization; MHSA processing, 1×1 convolution, and standardization are performed in sequence, and the integrated attention matrix is used as the output of this processing; it is used to extract long-range dependencies and spatial dependencies in the two feature maps; the corresponding pixels of the input feature map and the MHSA-processed feature map are summed, and the two feature maps are fused by concatenation, which retains the dependencies and spatial features contained in the two input feature maps respectively.
[0014] Furthermore, the fusion processing of feature maps at different levels in step 4) is implemented as follows: two-layer concatenated convolution operations are performed on feature maps at different levels to integrate multi-channel feature information; the deep-level feature maps are gradually downgraded to low-level feature maps through upsampling operations, and then spliced with the actual low-level features to finally output a fused image, ensuring that its size is consistent with the infrared and visible light images input to the entire network.
[0015] The advantage of this invention lies in its focus on the fusion of infrared and visible light images under the special condition of low illumination, and it provides feasible solutions for fields such as underground coal mine scene reconstruction. Attached Figure Description
[0016] Figure 1 Flowchart of a low-light infrared and visible light image fusion method based on dual attention mechanism;
[0017] Figure 2 Flowchart for multi-level heterogeneous dual-path feature map extraction;
[0018] Figure 3 MHSA processing flowchart;
[0019] Figure 4 MHCA processing flowchart;
[0020] Figure 5 Flowchart of the same-level heterogeneous dual-path feature map fusion processing;
[0021] Figure 6 Flowchart of feature map fusion processing at different levels. Detailed Implementation
[0022] The technical solutions of the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Figure 1 As shown in the figure, the method for fusing low-light infrared and visible light images based on a dual attention mechanism disclosed in this embodiment includes the following steps:
[0023] S101, Obtain multi-level heterogeneous dual-path feature maps; Perform MHSA processing on the deepest level feature map to extract remote feature information from the feature map; Perform MHCA processing on the remaining shallow feature maps to enhance relevant regions in network connectivity.
[0024] S102, perform standardization and new MHSA processing to obtain the attention matrix, then sum the pixels one by one, and fuse the two feature maps by concatenation.
[0025] S103 utilizes multiple decoding convolutional blocks and upsampling operations to fuse feature maps at different levels, ultimately obtaining a fused image of low-light infrared and visible light.
[0026] In S101, convolutional layers, max pooling layers, MHCA processing modules, and MHSA processing modules are used to obtain feature maps of different levels through convolution and pooling. MHSA processing is performed on the deepest feature maps, while MHCA processing is performed on the non-deepest feature maps. The specific processing flow is as follows: Figure 2 As shown.
[0027] S201: The input image is processed by 1×1 convolution kernel and two layers of 3×3 convolution kernel to extract the features of the image.
[0028] S202: Max pooling is performed on the output feature map of the previous layer, and two layers of 3×3 convolution kernels are used for convolution to extract deeper features and reduce the size of the feature map. This process is repeated 3 times to obtain three sets of feature maps at different levels.
[0029] S203: Perform MHSA processing on the deepest level feature map to extract remote feature information from the feature map;
[0030] S204: Perform MHCA processing on non-deepest level feature maps to enhance relevant regions in network connectivity.
[0031] In S203, the deepest level feature map needs to be processed by the MHSA module. The processing flow is as follows: Figure 3 As shown,
[0032] S301: Encode the position of the feature map to be processed, represent the tensor as a two-dimensional matrix, and capture the absolute and relative information between objects in the image.
[0033] S302: The tensor-form feature map is expanded into a two-dimensional matrix by summation, which facilitates subsequent processing;
[0034] S303: Embedding matrix W obtained through training q W k W vCalculate the query matrix Q, the key matrix K, and the value matrix V;
[0035] S304: Calculate the similarity between the query matrix Q and the key matrix K;
[0036] S305: Perform softmax non-linear processing on the similarity results;
[0037] S306: The weighted average of the results is achieved using the value matrix V. The processes S304~306 above can be represented by the following formula.
[0038] (1)
[0039] Among them, the query matrix Key matrix Value matrix , It is the dimension of the feature map after it is expanded into a column vector;
[0040] S307: Convert the obtained feature matrix into tensor form to obtain the MHSA output.
[0041] In S204, MHCA processing needs to be performed on the non-deepest level feature maps separately. The processing flow is as follows: Figure 4 As shown,
[0042] S401: Encode the position of a deeper feature map, represent the tensor as a two-dimensional matrix, and capture the absolute and relative information between objects in the feature map at the same time.
[0043] S402: Similarly, position encoding is performed on the input feature map of this layer, and the tensor is represented as a two-dimensional matrix, while capturing the absolute and relative information between objects in the feature map.
[0044] S403, S404: Summation is performed on the matrix-transformed results respectively;
[0045] S405: Obtain the query matrix Q and key value matrix K through the embedding matrix obtained by network training and the feature map of the next deeper level, and obtain the value matrix V from the embedding matrix and the current level feature map. Perform the processing on the query matrix Q, key value matrix K and value matrix V as shown in formula (1).
[0046] S406: Channel integration of feature maps is achieved through 1×1 convolution, and batch normalization is used to minimize overfitting. The feature maps are then subjected to Sigmoid nonlinear transformation and upsampling.
[0047] S407: In parallel, upsampling and convolution are performed on the feature map of the deeper layer to ensure that the feature maps of the two layers are of the same size at this time;
[0048] S408: Perform a dot product operation on the current layer feature map and the feature map before attention mechanism processing;
[0049] S409: The transformed feature map from the previous layer is concatenated with the feature map of the current layer to form the output of MHCA.
[0050] In S102, feature map fusion processing needs to be completed, such as... Figure 5 As shown:
[0051] S501: Performs 1×1 convolution and normalization on the infrared and visible light channels respectively, which not only integrates multiple feature maps, but also ensures that the specific values in the feature maps are fixed within a certain range.
[0052] S502: Perform MHSA processing on the current feature map respectively. The process is as follows: Figure 3 As shown;
[0053] S503: Perform 1×1 convolution and normalization on the feature maps of the two channels respectively;
[0054] S504: Then, the current feature map and the initial input feature map of the path are summed pixel by pixel, which means that features at different levels are fused.
[0055] S505: Directly stitches together the heterogeneous feature maps of the two channels to achieve feature map fusion.
[0056] In S103, the fusion of feature maps from different levels is required, such as... Figure 6 As shown:
[0057] S601: Perform multi-layer convolution processing on feature maps at different levels to further extract their features;
[0058] S602: In order to merge the deep-level feature map with the shallow-level feature map, the deep-level feature map is upsampled separately to ensure that the size of the processed feature map is consistent with that of the previous layer feature map.
[0059] S603: By stitching feature maps of the same size together, this method makes comprehensive use of multi-scale feature maps;
[0060] S604: Perform convolution processing on the stitched feature maps to facilitate the recovery of the fused image.
[0061] Furthermore, to demonstrate that the invented method has a stronger advantage over existing methods in fusing infrared and visible light images under low-light conditions, the fusion performance was compared using publicly available LLVIP dataset data under low-light conditions. The comparison results are shown in Table 1. In Table 1, DCHWT is the Discrete Cosine Harmonic Wavelet Transform method, ConvSR is the Convolutional Sparse Representation method, VggML is the Multi-Layer Deep Feature Fusion method, DenseFuse is the Sparse Fusion method, IFCNN is the Global Image Fusion method based on Convolutional Neural Networks, NestFuse is the Fusion method based on Network Connections and Spatial Attention, FusionGan is the Fusion method based on Generative Adversarial Networks, U2Fusion is the Unified Unsupervised Image Fusion Network, and RFN-Nest is the End-to-End Residual Fusion Network.
[0062] Table 1 Comparison of Fusion Effects for LLVIP Data
[0063] method En SD MI <![CDATA[N abf ]]> SCD MS-SSIM DCHWT 6.36723 64.97891 15.5137 0.12295 1.60993 0.81226 ConvSR 6.63433 67.54361 13.26865 0.07951 1.00488 0.80844 VggML 6.18260 48.15779 12.36521 0.08206 1.63522 0.87478 DenseFuse 6.67158 67.57282 13.34317 0.09214 1.83502 0.89896 IFCNN 6.59545 66.87578 13.19090 0.17959 1.71375 0.90527 NestFuse 6.91971 82.75242 13.23942 0.13405 1.73353 0.86248 FusionGan 6.36285 54.35752 12.72570 0.06706 1.45685 0.73182 U2Fusion 6.75708 64.91158 13.51416 0.29088 1.79837 0.92533 RFN-Nest 6.84134 71.90131 13.68269 0.07288 1.83676 0.91456 Method of the present invention 6.87391 71.54806 13.86783 0.13525 1.78242 0.94934
[0064] Finally, it should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the claims.
Claims
1. A method for fusing infrared and visible light images under low-light conditions, comprising the following steps: 1) Perform convolution processing on infrared and visible light heterogeneous source images under low illumination conditions, and then use the cascaded connection of multiple convolution and max pooling layers to obtain multi-level heterogeneous dual-path feature maps. 2) Perform multi-head self-attention (MHSA) processing on the deepest heterogeneous dual-path feature maps simultaneously to extract remote feature information from the feature maps; perform multi-head cross-attention (MHCA) processing on the remaining heterogeneous dual-path shallow feature maps simultaneously to enhance relevant regions in network connectivity. 3) Simultaneously standardize the feature maps of different levels from the heterogeneous dual-path after processing in step 2), and perform multi-head self-attention (MHSA) processing to obtain the attention matrix. Then, sum the attention matrix pixel-by-pixel and fuse the dual-path feature maps by concatenation. Specifically, this includes: 3-1) Perform a 1×1 convolution on the current input feature map to integrate multi-channel features, and use standardization to specify the value range of each element in the feature map; 3-2) Perform multi-head self-attention (MHSA) processing, 1×1 convolution, and standardization sequentially. The integrated attention matrix is used as the output to extract long-range and spatial dependencies in the dual-path feature maps. 3-3) Summing the corresponding pixels of the input feature map and the processed feature map, and then fusing the two feature maps by concatenation. This fusion retains the dependency and spatial features contained in the two input feature maps respectively. 4) Using the fused dual-path feature map obtained in step 3), multiple decoding convolutional blocks and upsampling operations are used to fuse feature maps at different levels, and finally obtain a low-light infrared and visible light fused image.
2. The method for fusing infrared and visible light images under low illumination conditions as described in claim 1, characterized in that, In step 1), the infrared and visible light heterogeneous images are processed by one layer of 1×1 convolution and two layers of 3×3 convolution, respectively. Then, one layer of max pooling and two layers of 3×3 convolution are used as a unit, and several groups of this unit are cascaded. The feature maps extracted by each group of units constitute multi-level heterogeneous dual-path feature maps.
3. The method for fusing infrared and visible light images under low illumination conditions as described in claim 1, characterized in that, Step 2) involves simultaneously performing multi-head self-attention (MHSA) processing on the deepest-level heterogeneous dual-path feature maps. The specific steps include: a) Encode the position of the input feature map and represent the tensor as a two-dimensional matrix, while capturing the absolute and relative information between objects in the feature map; b) Perform operations on the current feature matrix and the three embedding matrices to obtain the query matrix Q, the key matrix K, and the value matrix V, respectively; c) Calculate the similarity between all elements in K and Q and input it into the softmax function. The result is then processed by weighted averaging of the value matrix V to obtain the attention matrix, which takes into account all the interaction information between K and Q. d) Restore the attention matrix to tensor form to obtain the multi-head self-attention (MHSA) output.
4. The method for fusing infrared and visible light images under low illumination conditions as described in claim 1, characterized in that, Step 2) involves simultaneously performing Multi-Head Cross-Attention (MHCA) processing on the remaining heterogeneous dual-path shallow feature maps. The specific steps include: e) Calculate the key matrix K and query matrix Q of this layer from the feature map Y of the previous layer; calculate the value matrix V from the feature map S input to this layer; complete the calculation of the attention matrix A. f) Tensor Z is obtained through 1×1 convolution, batch normalization, sigmoid activation, and upsampling, and its size is consistent with the feature map S input to this layer. g) Perform dot product on Z and S, and then convolve and upsample the result obtained by position encoding the previous layer feature map Y. Finally, concatenate the result with the dot product to obtain the multi-head cross attention (MHCA) output.
5. The method for fusing infrared and visible light images under low illumination conditions as described in claim 1, characterized in that, Step 4) specifically includes: k) Perform two-layer concatenated convolutional operations on feature maps of different levels respectively to integrate multi-channel feature information; l) By upsampling, deep features are gradually fused into low-level features, achieving the fusion of multiple feature layers; m) Output the fused image, the size of which is consistent with the infrared and visible light images input to the entire network.