An infrared and visible light image fusion method and system based on spatial perception adaptive fusion

Abstract

The application discloses an infrared and visible light image fusion method and system based on space perception adaptive fusion. The scheme is used for obtaining an original infrared image and an original visible image; a dual-branch encoder based on a densely connected block and channel attention is used for intelligently screening and dynamically weighting fusion of dual-mode features of the preprocessed image in a space perception adaptive fusion module; a multi-scale decoder based on an amplitude perception linear attention module is used for decoding the fused features and reconstructing a fused image; and a joint training method based on a segmentation network is used for injecting semantic perception capability into the fusion network. The space perception adaptive fusion module is designed between the encoder and the decoder, screening and dynamic weighting fusion of dual-mode information are realized, the quality of the fused features is significantly improved, and multi-modal complementary information is fully mined; the amplitude perception linear attention module is introduced in the decoder part, the receptive field limitation of the traditional convolution is overcome, and global context integration is realized.

Description

Technical Field

[0001] This invention belongs to the field of image processing and computer vision technology, and in particular relates to an infrared and visible light image fusion method and system based on spatial perception adaptive fusion. Background Technology

[0002] In imaging tasks, combining images from multiple sensors is necessary for better scene perception. Visible light and infrared image fusion aims to integrate the advantages of two different imaging modalities into a single image. Visible light images (VIS) offer rich texture, color, and detail, conforming to human visual habits, but they become ineffective in harsh environments such as nighttime, smoke, and strong light. Infrared images (IR) image, which sense the thermal radiation of objects, are unaffected by lighting conditions and can highlight heat sources (such as people, vehicles, and equipment) in all weather conditions, but lack scene detail. Fusion of infrared and visible light images generates a composite image that simultaneously contains the rich texture of visible light and the significant thermal target information of infrared images, achieving complementary advantages and providing a more reliable data foundation for perception applications in complex real-world environments.

[0003] Existing infrared and visible light image fusion technologies still face multiple technical bottlenecks, making it difficult to meet the growing application demands.

[0004] First, in terms of fusion strategies, most methods rely on simple feature stitching or fixed weighting strategies, which cannot dynamically adjust the fusion rules according to the image content. This results in an unintelligent fusion process and limits the full exploitation of the deep complementary potential of multimodal information.

[0005] Secondly, in terms of decoding and computational efficiency, existing methods face a dilemma between "effectiveness" and "speed." Many advanced decoders employ computationally complex attention mechanisms (such as standard self-attention, with a complexity of O(N²)) to integrate global contextual information, resulting in slow model inference speeds and difficulty in meeting the stringent real-time requirements of scenarios such as autonomous driving and real-time monitoring. On the other hand, some methods, in pursuit of speed, use traditional convolutions with limited receptive fields, which cannot effectively capture long-distance dependencies, affecting the global consistency and quality of the fused images.

[0006] Finally, in terms of training objectives and application value, existing methods generally lack "semantic awareness" capabilities. Most fusion networks focus only on the visual quality of the images themselves during training, lacking a mechanism to incorporate semantic information from downstream high-level visual tasks (such as semantic segmentation) into the training process. This results in fused images that, while visually appealing, lack a deep semantic understanding of the target and scene, making it difficult to effectively support downstream high-level visual tasks and thus limiting the practical application value of fusion technology.

[0007] Therefore, designing a fusion method that can simultaneously achieve intelligent dynamic fusion, efficient real-time decoding, and deep semantic perception capabilities is a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for fusion of infrared and visible light images based on spatial perception adaptive fusion, which realizes refined feature extraction, intelligent dynamic fusion, efficient real-time decoding, and semantic perception capabilities.

[0009] To achieve the objectives of this invention, on one hand, this invention provides a method for fusing infrared and visible light images based on spatially perceptual adaptive fusion, comprising the following steps:

[0010] Step 1: Input the original infrared image and visible light image. Convert the visible light image to the YCrCb color space and extract its Y luminance channel. At the same time, align the infrared image and the Y luminance channel by space size to obtain single-channel infrared image data and single-channel visible light luminance data with uniform size.

[0011] Step 2: The single-channel infrared image data and the single-channel visible light brightness data are progressively feature extracted using a dual-branch encoder to obtain a dual-branch 48-channel high-level feature map.

[0012] Step 3: Process the high-level feature map of the dual-branch 48-channel structure through the spatially aware adaptive fusion module to obtain a fused feature map of 48 channels.

[0013] Step 4: Process the fusion feature map of the 48 channels through the amplitude-aware linear attention module to obtain a fusion feature map of 64 channels;

[0014] Step 5: Input the 64-channel fused feature map into the multi-scale decoder to restore the image resolution and obtain a single-channel fused image;

[0015] Step 6: The single-channel fused image is used as the Y channel and spliced ​​with the CrCb channel of the original visible light image to form a brand-new YCrCb three-channel image. Then, the YCrCb three-channel image is converted back to an RGB three-channel image, and the three-channel fused image is output.

[0016] Step 7: The parameters of the dual-branch encoder, spatially aware adaptive fusion module, amplitude-aware linear attention module and multi-scale decoder from steps 2 to 5 are jointly trained in stages with semantic guidance so that the output three-channel fused image simultaneously meets visual quality requirements and high-level visual task adaptability.

[0017] Step 2 specifically includes the following steps:

[0018] Step 2-1: Input the single-channel infrared image data and the single-channel visible light brightness data into the two parallel branches of the dual-branch encoder respectively. Both branches process the corresponding input data through a 3×3 convolutional layer and the LeakyReLU activation function to obtain 16-channel infrared primary feature maps and 16-channel visible light primary feature maps respectively.

[0019] Step 2-2: Enhance the edge features of the 16-channel primary feature maps of each of the two branches using horizontal and vertical gradient convolution kernels: First, define Sobel convolution kernels in the horizontal and vertical directions; apply the Sobel convolution kernel to each channel of the primary feature map to obtain horizontal and vertical gradient maps, and finally fuse the gradient maps to enhance the edge features, obtaining a 48-channel mid-layer feature map of the two branches.

[0020] Steps 2-3: Pass the dual-branch 48-channel mid-layer feature map through a densely connected block containing four 3×3 convolutional layers and the LeakyReLU activation function to obtain 48-channel features.

[0021] Steps 2-4: Concatenate the mid-layer feature map and the 48-channel feature map of the dual-branch 48-channel array to obtain the high-layer feature map of the 48-channel array.

[0022] Steps 2-5: Obtain global feature vectors from the 48-channel high-level feature map through global average pooling and global max pooling, generate channel weights through a multilayer perceptron and a sigmoid activation function, and multiply them with the original feature map channel by channel to obtain the channel weights s.

[0023] Steps 2-6: Multiply the channel weights s with the 48-channel high-level feature map channel by channel to obtain a dual-branch 48-channel high-level feature map, including an infrared image feature enhancement map and a visible light image feature enhancement map of 48 channels.

[0024] Step 3, the spatial perception adaptive fusion module processing includes the following steps:

[0025] Step 3-1: The dual-branch 48-channel high-level feature map is processed by channel-dimensional average pooling and 3×3 convolution respectively, and the spatial attention map is inferred by the Sigmoid activation function.

[0026] Step 3-2: Selectively enhance or suppress the spatial attention map and the dual-branch 48-channel high-level feature map pixel by pixel to obtain the modulated dual-modal features;

[0027] Step 3-3: Aggregate the modulated dual-modal features and dynamically generate spatial perception fusion weights through global average pooling and multilayer perceptron, and then normalize them with Softmax to obtain adaptive fusion weights;

[0028] Steps 3-4: The modulated dual-modal features and the adaptive fusion weights are weighted and fused to obtain a 48-channel fused feature map.

[0029] The amplitude-aware linear attention module processing in step 4 includes the following steps:

[0030] Step 4-1: Process the fused feature map of the 48 channels through a 3×3 convolutional layer and the LeakyReLU activation function to obtain 64-channel features;

[0031] Step 4-2: The 64-channel features are split into four branches, Q, K, V, and O, using a 1×1 convolutional feature splitting unit. The ELU activation function and non-negativity constraints are then applied to the Q and K branches respectively.

[0032] Step 4-3: Determine the amplitude-aware attention score matrix based on the processed Q branch. Simultaneously, add rotation position encoding to the Q and K branches respectively and calculate the KV key-value interaction matrix. Perform local position feature enhancement on the V branch through 5×5 grouped convolution.

[0033] Step 4-4: The amplitude-aware attention score matrix, the KV key-value interaction matrix, and the V branch after local position feature enhancement are fused and modulated, and then combined with the O branch to obtain 64-channel features with fused multi-dimensional perception.

[0034] The multi-scale decoder processing in step 5 includes the following steps:

[0035] Step 5-1: Decode the 64-channel features fused with multi-dimensional perception through 3×3 convolution and LeakyReLU activation function to obtain 32-channel features;

[0036] Step 5-2: Decode the 32-channel features using a 3×3 convolution and LeakyReLU activation function to obtain 16-channel features;

[0037] Step 5-3: Process the 16-channel features through 1×1 convolution and Tanh activation function to obtain a single-channel fused image.

[0038] Step 7 includes the following steps:

[0039] Step 7-1: Define the joint loss function, which includes fusion loss and segmentation loss;

[0040] Step 7-2: Connect the fusion network consisting of the dual-branch encoder, spatially aware adaptive fusion module, amplitude-aware linear attention module, and multi-scale decoder from steps 2 to 5 in series with the semantic segmentation network. Input the three-channel fused image into the semantic segmentation network to obtain the segmentation result. Calculate the error using the joint loss function and backpropagate iteratively to adjust the parameters of each module in the fusion network. During training, an adaptive weight strategy is adopted to gradually increase the weight ratio of semantic loss, balancing visual quality and semantic adaptability.

[0041] Step 7-3: When the joint loss described in step 7-2 converges, terminate the training, save the optimal parameters of the fusion network, and generate the final three-channel fused image that simultaneously meets the visual quality requirements and the adaptability to high-level visual tasks.

[0042] On the other hand, the present invention also provides a system for an infrared and visible light image fusion method based on spatially perceptual adaptive fusion, comprising the following modules:

[0043] The image preprocessing module is used to preprocess the original visible light image and infrared image, including color space conversion, channel extraction and size unification, to generate single-channel input data;

[0044] The dual-branch encoder module is used to perform progressive feature extraction on the preprocessed single-channel input data. It generates a high-level feature map with 48 channels through convolutional layers, densely connected blocks, Sobel gradient operators, and channel attention mechanisms.

[0045] The spatially aware adaptive fusion module is used to fuse the feature maps output by the dual-branch encoder. It achieves high-quality and refined dynamic feature fusion through a chain mechanism of feature selection, representation optimization, and weight adaptation, generating 48-channel fused features.

[0046] The amplitude-aware linear attention module is used to enhance the amplitude and position of features during the decoding process. It outputs enhanced features through linear attention mechanism and gating fusion.

[0047] The multi-level decoder module is used to fuse enhanced features, gradually reducing the number of channels and reconstructing spatial details through multiple decoding layers, and outputting a single-channel fused image.

[0048] The image post-processing module is used to combine the single-channel fused image with the chroma channel of the visible light image, and generate the final three-channel RGB fused image through color space conversion;

[0049] The phased training control module is used to dynamically adjust the weights of fusion loss and segmentation loss during model training to achieve phased optimization.

[0050] The spatial perception adaptive fusion module specifically includes the following modules:

[0051] The spatial attention generation module receives the infrared and visible light dual-branch 48-channel high-level feature maps output by the dual-branch encoder module, and realizes the filtering of effective information and accurate positioning of key information regions in the dual-branch features, and outputs spatial attention weights.

[0052] The representation optimization module is used to modulate the original 48-channel high-level feature maps of infrared and visible light using spatial attention weights. By strengthening key information regions and suppressing redundant information such as background or noise, the dual-modal feature representation is optimized and the modulated dual-modal features are output.

[0053] The adaptive weight generation module is used to aggregate the optimized bimodal features and mine global context information to dynamically generate infrared and visible light modal adaptive fusion weights that are adapted to the current features. At the same time, based on these weights, complementary fusion of the optimized bimodal features is realized, and a high-quality fusion feature map with 48 channels is output.

[0054] Compared with existing technologies, the significant advancements of this invention are as follows: This invention designs a spatially aware adaptive fusion module between the decoder and encoder for dynamic weighted fusion. Through a chain mechanism of "feature selection → representation optimization → weight adaptation," it achieves refined and intelligent feature fusion, ensuring high computational efficiency while tapping into the potential of multimodal complementarity. An amplitude-aware linear attention module is introduced into the decoder to overcome the receptive field limitations of traditional convolution, achieving global context integration. Simultaneously, compared to traditional attention mechanisms, the computational complexity is reduced to O(N), efficiently integrating global context while meeting the requirements of real-time applications. Furthermore, a staged joint training strategy is adopted to integrate semantic segmentation constraints into the network, enabling the fused image to maintain high quality while gaining stronger semantic awareness, thus effectively supporting downstream advanced vision tasks. Through the above collaborative optimizations, this invention unifies and realizes a fusion method with refined feature extraction, intelligent dynamic fusion, efficient real-time decoding, and semantic awareness capabilities.

[0055] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description

[0056] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0057] Figure 1 This is a flowchart of the steps of the present invention;

[0058] Figure 2This is the network architecture diagram of the present invention;

[0059] Figure 3 This is a structural diagram of the spatial perception adaptive fusion module of the present invention;

[0060] Figure 4 This is a structural diagram of the multi-scale decoder of the present invention;

[0061] Figure 5 This is a structural diagram of the amplitude-sensing linear attention module of the present invention;

[0062] Figure 6 Figure 1 is a comparative diagram of the subjective fusion results of this invention. Figure 2 shows an infrared image (IR), a visible light image (VIS), a fused image generated by DenseFuse (based on dense block-convolutional layer feature extraction), a fused image generated by SDNet (based on semantically driven fusion network method), a fused image generated by U2Fusion (based on unified unsupervised fusion method), a fused image generated by SegMiF (based on segmentation-driven image fusion method), a fused image generated by SeAFusion (based on semantically perceptive fusion method), a fused image generated by MRFS (based on multi-scale residual feature fusion method), and a fused image generated by ours (based on the method of this invention).

[0063] Figure 7 This is the cumulative distribution function curve of the objective fusion evaluation index of this invention;

[0064] Figure 8 Figure 1 shows the semantic segmentation performance of this invention. Figure 2 shows the semantic segmentation results of IR (infrared image), Figure 3 shows the semantic segmentation results of VI (visible light image), Figure 4 shows the semantic segmentation results of DenseFuse (based on dense block-convolutional layer feature extraction method), Figure 5 shows the semantic segmentation results of SDNet (based on semantically driven fusion network method), Figure 6 shows the semantic segmentation results of U2Fusion (based on unified unsupervised fusion method), Figure 7 shows the semantic segmentation results of SegMiF (based on segmentation-driven image fusion method), Figure 8 shows the semantic segmentation results of SeAFusion (based on semantically perceptive fusion method), Figure 9 shows the semantic segmentation results of MRFS (based on multi-scale residual feature fusion method), Figure 10 shows the semantic segmentation results of Ours (based on the method of this invention), and Figure 11 shows the semantic segmentation results of Ground Truth. Detailed Implementation

[0065] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] This invention provides a spatially perceptive adaptive fusion method for infrared and visible light image fusion, combining... Figure 1 and Figure 2 This includes the following steps:

[0067] Step 1: Input the original infrared image and visible light image. Convert the visible light image to the YCrCb (brightness-red difference-blue difference) color space and extract its Y brightness channel. At the same time, align the infrared image and the Y brightness channel by space size to obtain single-channel infrared image data and single-channel visible light brightness data with uniform size.

[0068] The specific formula is as follows:

[0069] Y = 0.299R + 0.587G + 0.114B;

[0070] Cr = 0.5(RY) + 128;

[0071] Cb = 0.5(BY) + 128;

[0072] Where R, G, and B correspond to the red, green, and blue components of the RGB color space, respectively; Y is the luminance component; Cr is the red difference component; and Cb is the blue difference component.

[0073] Step 2: The single-channel infrared image data and the single-channel visible light brightness data are progressively feature extracted using a dual-branch encoder to obtain a dual-branch 48-channel high-level feature map.

[0074] Step 2 specifically includes the following steps:

[0075] The dual-branch encoder includes two parallel feature extraction branches with completely identical structures, corresponding to the infrared image branch and the visible light image branch respectively. Each branch consists of a feature extraction module composed of a primary convolutional layer connected in series, two sets of densely connected blocks and Sobel gradient operators connected in parallel, and a channel attention feature enhancement layer.

[0076] Step 2-1: Input the single-channel infrared image data and the single-channel visible light brightness data into the two parallel branches of the dual-branch encoder respectively. Both branches process the corresponding input data through a 3×3 convolutional layer and a LeakyReLU (with leakage correction linear unit) activation function to obtain 16-channel infrared primary feature maps and 16-channel visible light primary feature maps respectively.

[0077] The specific formula is as follows:

[0078] ;

[0079] The input image is The output feature map is ;

[0080] Step 2-2: Enhance the edge features of the 16-channel primary feature maps X of the two branches using horizontal and vertical gradient convolution kernels, and define Sobel convolution kernels in the horizontal and vertical directions. and Applying these two operators to each channel of the primary feature map X yields the horizontal and vertical gradient maps. and Finally, the gradient maps are fused to enhance edge features, resulting in a dual-branch 48-channel mid-layer feature map.

[0081] The specific formula is as follows:

[0082] ;

[0083] in, and This indicates that a fixed Sobel kernel is used for the convolution operation;

[0084] Steps 2-3: Pass the dual-branch 48-channel mid-layer feature map through a densely connected block containing four 3×3 convolutional layers and the LeakyReLU activation function to obtain 48-channel features.

[0085] The specific formula is as follows:

[0086] =[LeakyReLU LeakyReLU ( LeakyReLU ( LeakyReLU ( ))];

[0087] Where X is a 16-channel primary feature map, The first layer output is concatenated with X. For the second layer output and splicing For the third layer output and splicing;

[0088] Steps 2-4: The middle layer feature map and the 48-channel feature map of the dual-branch 48-channel are spliced ​​together to obtain the high layer feature map F of the 48-channel.

[0089] Steps 2-5: The high-level feature map F of the dual-branch 48 channels is processed by global average pooling and global max pooling to obtain a global feature vector. Channel weights are then generated using a multilayer perceptron and a sigmoid activation function, and multiplied channel by channel with the original feature map to obtain the channel weights s, as shown in the following formula:

[0090] s=σ( (LeakyReLU( ( ))));

[0091] Where, F∈ This is a high-level feature map of a dual-branch 48-channel structure. and These represent global average pooling and global max pooling, respectively. It is the first fully connected layer, which reduces the dimension from 48 to r (r is the reduction rate, set to 16). It is the second fully connected layer, which restores the dimension from r to 48, s∈ These are the learned channel weights;

[0092] Steps 2-6: Multiply the channel weights s with the 48-channel high-level feature map F channel by channel to obtain a dual-branch 48-channel high-level feature map, including an infrared image feature enhancement map of 48 channels. Visible light image feature enhancement map .

[0093] Step 3: Process the high-level feature map of the dual-branch 48-channel structure through the spatially aware adaptive fusion module to obtain a fused feature map of 48 channels.

[0094] Combination Figure 3 The spatial perception adaptive fusion module in step 3 includes the following steps:

[0095] The spatial perception adaptive fusion module includes a spatial attention generation module, a representation optimization module, and an adaptive weight generation module.

[0096] Step 3-1: The dual-branch 48-channel high-level feature map is processed by channel-dimensional average pooling and 3×3 convolution respectively. The modulation weights that can act on the entire spatial dimension are derived from the channel information to identify key information regions and achieve feature filtering.

[0097] The channel information is then fed into a miniature network consisting of 1×1 convolutions and a sigmoid activation function. The 1×1 convolutions have 48 input and 48 output channels. The 1×1 convolutions learn the non-linear relationships between channels, reweighting the features of the 48 channels in a fully connected manner, enhancing important channels, suppressing secondary channels, and inferring their respective spatial attention weights, as shown in the following equation:

[0098] ;

[0099] ;

[0100] in, and These are the spatial attention weights for infrared images and the spatial attention weights for visible light images, respectively, and their shapes remain the same. , This represents the Sigmoid activation function.

[0101] Step 3-2: Use the spatial attention weights as modulation weights and the 48-channel feature map. , Selective enhancement or suppression is performed pixel-by-pixel to optimize the feature representation of each mode, resulting in a modulated infrared image feature map. and visible light image feature map All shapes ;

[0102] After this step, the key information regions (indicated by high weight values) in each modality feature map are enhanced, while irrelevant information such as background or noise (indicated by low weight values) are effectively suppressed, resulting in modulated bimodal features, as shown in the following formula:

[0103] ;

[0104] ;

[0105] Step 3-3: Aggregate the modulated bimodal features, and based on the aggregated global context information, obtain global statistical information L through global average pooling, as shown in the following formula;

[0106] ;

[0107] Where L represents global statistics. For global average pooling, For splicing operations;

[0108] Steps 3-4: Input the global statistical information L into a multilayer perceptron (MLP); this MLP adopts a bottleneck structure of "dimensionality reduction-activation-dimensionality increase", specifically 96→ →96 (r is the reduction ratio, set to 16). The MLP outputs a 96-dimensional vector, which is then reshaped into two 48-dimensional vectors, corresponding to the fusion weights for infrared and visible light, respectively. The two reshaped weight vectors are normalized using the Softmax function (normalization exponential function) to obtain the final infrared image fusion weights. Visible light image fusion weights The sum of the weights of the two modes is 1, forming a competitive and complementary relationship, as shown in the following formula;

[0109] ;

[0110] Steps 3-5: Perform weighted fusion of the dynamically generated fusion weights and the modulated dual-modal features to output the final 48-channel fused feature map. The specific formula is shown below;

[0111] .

[0112] Step 4: Process the fusion feature map of the 48 channels through the amplitude-aware linear attention module to obtain a fusion feature map of 64 channels;

[0113] Step 4 involves the processing of amplitude-aware linear attention modules. Figure 5 Includes the following steps:

[0114] The amplitude-aware linear attention module includes a 1×1 convolutional feature splitting unit, an ELU (exponential linear unit) activation function, an amplitude-aware attention score calculation unit, a rotation position encoding unit, a KV key-value interaction calculation unit, a local position feature enhancement unit, and a feature fusion modulation unit. Each unit works together in a coordinated manner according to the process.

[0115] Step 4-1: Process the fused feature map of the 48 channels through a 3×3 convolutional layer and the LeakyReLU activation function to obtain 64-channel features;

[0116] Step 4-2: The 64-channel features are split into four branches, Q (query vector), K (key vector), V (value vector), and O (output vector), using a 1×1 convolutional feature splitting unit. The ELU activation function and non-negativity constraints are then applied to the Q and K branches respectively.

[0117] Step 4-3: Determine the amplitude-aware attention score matrix based on the processed Q branch. Simultaneously, add rotation position encoding to the Q and K branches respectively and calculate the KV key-value interaction matrix. Perform local position feature enhancement on the V branch through 5×5 grouped convolution.

[0118] Step 4-4: The amplitude-aware attention score matrix, the KV key-value interaction matrix, and the V branch after local position feature enhancement are fused and modulated, and then combined with the O branch to obtain 64-channel features with fused multi-dimensional perception.

[0119] Step 5: Input the 64-channel fused feature map into the multi-scale decoder to gradually restore the image resolution and obtain a single-channel image.

[0120] Combination Figure 4 The multi-scale decoder processing in step 5 includes the following steps:

[0121] The multi-scale decoder includes a multi-level convolutional decoding layer and a single-channel feature mapping layer, with each layer connected in series and working together.

[0122] Step 5-1: Decode the 64-channel features fused with multi-dimensional perception through 3×3 convolution and LeakyReLU activation function to obtain 32-channel features;

[0123] The specific formula is as follows:

[0124] ;

[0125] in, It features 32 channels.

[0126] Step 5-2: Decode the 32-channel features using a 3×3 convolution and LeakyReLU activation function to obtain 16-channel features;

[0127] The specific formula is as follows:

[0128] ;

[0129] in, It features 16 channels.

[0130] Step 5-3: Process the 16-channel features using a 1×1 convolution and the Tanh (hyperbolic tangent) activation function, constraining the output value range to [0,1], and outputting a single-channel fused image. .

[0131] ;

[0132] in, ;

[0133] Step 6: Combine the single-channel fused image as the Y channel with the CrCb channel of the visible light image input in Step 1 to form a new YCrCb three-channel image. Then convert the YCrCb three-channel image back to an RGB three-channel image and output the three-channel fused image. ;

[0134] Step 7: Optimize the parameters of the fusion network composed of the dual-branch encoder, spatially aware adaptive fusion module, amplitude-aware linear attention module and multi-scale decoder in steps 2 to 5 through a semantically guided phased joint training method, so that the output three-channel fused image simultaneously meets visual quality requirements and high-level visual task adaptability.

[0135] Step 7-1: Define the joint loss function, which includes fusion loss and segmentation loss;

[0136] ;

[0137] in, (epoch) is a weight coefficient that varies with the training epoch and is used to control the influence of the segmentation loss on the total loss.

[0138] The fusion loss is composed of a weighted sum of three sub-losses, and its mathematical definition is:

[0139]

[0140] Among them, strength loss ,in To merge images Grayscale version The mean absolute error loss function;

[0141] Gradient loss ).

[0142] SSIM loss , Operators for calculating structural similarity

[0143] Segmentation loss The segmentation prediction map S(⋅) is a segmentation network. For real segmentation labels, To mine cross-entropy loss function for online hard examples. When the segmentation loss weights At that time, the loss is activated.

[0144] Step 7-2: Connect the fusion network consisting of the dual-branch encoder, spatially aware adaptive fusion module, amplitude-aware linear attention module, and multi-scale decoder from steps 2 to 5 in series with the semantic segmentation network. Input the three-channel fused image output from step 6 into the semantic segmentation network to obtain the segmentation result. Calculate the error using the joint loss function and backpropagate iteratively to adjust the parameters of each module in the fusion network. During training, an adaptive weight strategy is adopted to gradually increase the weight ratio of semantic loss, balancing visual quality and semantic adaptability.

[0145] In the first training round, Using only the fusion loss function, the segmentation loss weights are set to 0. ;

[0146] In the second training epoch, the segmentation loss weight is set to 1, and semantic constraints are initially introduced. ;

[0147] In the third training epoch, the segmentation loss weight is increased to 2 to strengthen semantic constraints. ;

[0148] In the fourth training epoch, the segmentation loss weight is increased to 3 to achieve the final balance between semantic accuracy and fusion quality. .

[0149] Step 7-3, the joint loss described in step 7-2 Training terminates upon convergence, and the optimal parameters of the fusion network are saved for inference in steps 1-6, generating the final three-channel fused image that simultaneously meets visual quality requirements and high-level visual task adaptability.

[0150] The present invention provides a system for an infrared and visible light image fusion method based on spatially perceptual adaptive fusion, comprising the following modules:

[0151] The image preprocessing module is used to preprocess the original visible light image and infrared image, including color space conversion, channel extraction and size unification, to generate single-channel input data;

[0152] The dual-branch encoder module is used to perform progressive feature extraction on the preprocessed single-channel input data. It generates a high-level feature map with 48 channels through convolutional layers, densely connected blocks, Sobel gradient operators, and channel attention mechanisms.

[0153] The spatially aware adaptive fusion module is used to fuse the feature maps output by the dual-branch encoder. It achieves high-quality and refined dynamic feature fusion through a chain mechanism of feature selection, representation optimization, and weight adaptation, generating 48-channel fused features.

[0154] The amplitude-aware linear attention module is used to enhance the amplitude and position of features during the decoding process. It outputs enhanced features through linear attention mechanism and gating fusion.

[0155] The multi-level decoder module is used to fuse enhanced features, gradually reducing the number of channels and reconstructing spatial details through multiple decoding layers, and outputting a single-channel fused image.

[0156] The image post-processing module is used to combine the single-channel fused image with the chroma channel of the visible light image, and generate the final three-channel RGB fused image through color space conversion;

[0157] The phased training control module is used to dynamically adjust the weights of fusion loss and segmentation loss during model training to achieve phased optimization.

[0158] The spatial perception adaptive fusion module specifically includes the following modules:

[0159] The spatial attention generation module receives the infrared and visible light dual-branch 48-channel high-level feature maps output by the dual-branch encoder module, and realizes the filtering of effective information and accurate positioning of key information regions in the dual-branch features, and outputs spatial attention weights.

[0160] The representation optimization module is used to modulate the original 48-channel high-level feature maps of infrared and visible light using spatial attention weights. By strengthening key information regions and suppressing redundant information such as background or noise, the dual-modal feature representation is optimized and the modulated dual-modal features are output.

[0161] The adaptive weight generation module is used to aggregate the optimized bimodal features and mine global context information to dynamically generate infrared and visible light modal adaptive fusion weights that are adapted to the current features. At the same time, based on these weights, complementary fusion of the optimized bimodal features is realized, and a high-quality fusion feature map with 48 channels is output.

[0162] like Figure 6As shown, the present invention is compared with six existing mainstream image fusion methods on the MSRS dataset. Figure (a) is an IR (infrared image), Figure (b) is a VI (visible light image), Figure (c) is a fused image generated by DenseFuse (based on dense block-convolutional layer feature extraction), Figure (d) is a fused image generated by SDNet (based on semantically driven fusion network method), Figure (e) is a fused image generated by U2Fusion (based on unified unsupervised fusion method), Figure (f) is a fused image generated by SegMiF (based on segmentation-driven image fusion method), Figure (g) is a fused image generated by SeAFusion (based on semantically perceptive fusion method), Figure (h) is a fused image generated by MRFS (based on multi-scale residual feature fusion method), and Figure (i) is a fused image generated by ours (based on the method of the present invention). As can be seen from the comparison, the fused image generated by the method of the present invention not only retains the rich texture details of the visible light image (such as building textures and roads), but also significantly highlights the thermal targets (such as pedestrians and vehicles) in the infrared image. Compared with the existing methods, the results of the present invention are excellent in terms of target clarity, background detail preservation and overall visual naturalness, which effectively proves the advantages of the present method in multimodal information complementary fusion.

[0163] like Figure 7 As shown in the figure, the vertical axis represents the numerical value of the evaluation index, and the horizontal axis represents the cumulative probability. This figure compares the performance of the method of this invention with several existing mainstream fusion methods on multiple key indicators (mutual information (MI), visual information fidelity (VIF), edge preservation quality (QAB / F), and information entropy (EN). It can be seen from the figure that the curve corresponding to the method of this invention is at the top of all the curves of the comparison methods in terms of mutual information (MI), visual information fidelity (VIF), and edge preservation quality (QAB / F). In terms of information entropy (EN), the curve of this invention is also in the second-best position, following the best method. This indicates that the fused image generated by this invention achieves optimal levels in terms of multimodal information complementarity (MI), visual perception quality (VIF), and edge detail preservation (QAB / F), while also achieving a highly competitive second-best performance in terms of information content (EN), thus objectively verifying the effectiveness and superiority of this method.

[0164] like Figure 8As shown, all comparison methods and the method used in this invention are input into the same semantic segmentation network to fairly test the performance of the fused images generated by each method on downstream high-level vision tasks. A comparison with Figure (j) Ground_Truth shows that the semantic segmentation results of Figure (a) ir (infrared image) and Figure (c) DenseFuse (based on dense block-convolutional layer feature extraction method) failed to correctly segment and identify the car target; the semantic segmentation results of Figure (b) vis (visible light image) and Figure (e) U2Fusion (based on unified unsupervised fusion method) failed to correctly segment and identify the human target; Figure (d) SDNet (based on semantically driven fusion network method), Figure (f) SegMiF (based on segmentation-driven image fusion method), Figure (g) SeAFusion (based on semantically perceptive fusion method), and Figure (h)... While MRFS (Multi-Scale Residual Feature Fusion Method) correctly segmented and identified human and vehicle targets, the segmentation was incomplete. Only our method segmented and identified human and vehicle targets relatively completely. This shows that the fused image generated by our method maintains high quality while gaining stronger semantic perception capabilities, which can effectively support downstream advanced vision tasks.

[0165] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0166] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for fusing infrared and visible light images based on spatially perceptive adaptive fusion, characterized in that, Includes the following steps: Step 1: Input the original infrared image and visible light image. Convert the visible light image to the YCrCb color space and extract its Y luminance channel. At the same time, align the infrared image and the Y luminance channel by space size to obtain single-channel infrared image data and single-channel visible light luminance data with uniform size. Step 2: The single-channel infrared image data and the single-channel visible light brightness data are progressively feature extracted using a dual-branch encoder to obtain a dual-branch 48-channel high-level feature map. Step 3: Process the high-level feature map of the dual-branch 48-channel structure through the spatially aware adaptive fusion module to obtain a fused feature map of 48 channels. Step 4: Process the fusion feature map of the 48 channels through the amplitude-aware linear attention module to obtain a fusion feature map of 64 channels; Step 5: Input the 64-channel fused feature map into the multi-scale decoder to restore the image resolution and obtain a single-channel fused image; Step 6: The single-channel fused image is used as the Y channel and spliced ​​with the CrCb channel of the original visible light image to form a brand-new YCrCb three-channel image. Then, the YCrCb three-channel image is converted back to an RGB three-channel image, and the three-channel fused image is output. Step 7: The parameters of the dual-branch encoder, spatially aware adaptive fusion module, amplitude-aware linear attention module and multi-scale decoder from steps 2 to 5 are jointly trained in stages with semantic guidance so that the output three-channel fused image simultaneously meets visual quality requirements and high-level visual task adaptability.

2. The infrared and visible light image fusion method based on spatial perception adaptive fusion according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2-1: Input the single-channel infrared image data and the single-channel visible light brightness data into the two parallel branches of the dual-branch encoder respectively. Both branches process the corresponding input data through a 3×3 convolutional layer and the LeakyReLU activation function to obtain 16-channel infrared primary feature maps and 16-channel visible light primary feature maps respectively. Step 2-2: Enhance the edge features of the 16-channel primary feature maps of each of the two branches using horizontal and vertical gradient convolution kernels: First, define Sobel convolution kernels in the horizontal and vertical directions; apply the Sobel convolution kernel to each channel of the primary feature map to obtain horizontal and vertical gradient maps, and finally fuse the gradient maps to enhance the edge features, obtaining a 48-channel mid-layer feature map of the two branches. Steps 2-3: Pass the dual-branch 48-channel mid-layer feature map through a densely connected block containing four 3×3 convolutional layers and the LeakyReLU activation function to obtain 48-channel features. Steps 2-4: Concatenate the mid-layer feature map and the 48-channel feature map of the dual-branch 48-channel array to obtain the high-layer feature map of the 48-channel array. Steps 2-5: Obtain global feature vectors from the 48-channel high-level feature map through global average pooling and global max pooling, generate channel weights through a multilayer perceptron and a sigmoid activation function, and multiply them with the original feature map channel by channel to obtain the channel weights s. Steps 2-6: Multiply the channel weights s with the 48-channel high-level feature map channel by channel to obtain a dual-branch 48-channel high-level feature map, including an infrared image feature enhancement map and a visible light image feature enhancement map of 48 channels.

3. The infrared and visible light image fusion method based on spatial perception adaptive fusion according to claim 2, characterized in that, Step 3, the spatial perception adaptive fusion module processing includes the following steps: Step 3-1: The dual-branch 48-channel high-level feature map is processed by channel-dimensional average pooling and 3×3 convolution respectively, and the spatial attention map is inferred by the Sigmoid activation function. Step 3-2: Selectively enhance or suppress the spatial attention map and the dual-branch 48-channel high-level feature map pixel by pixel to obtain the modulated dual-modal features; Step 3-3: Aggregate the modulated dual-modal features and dynamically generate spatial perception fusion weights through global average pooling and multilayer perceptron, and then normalize them with Softmax to obtain adaptive fusion weights; Steps 3-4: The modulated dual-modal features and the adaptive fusion weights are weighted and fused to obtain a 48-channel fused feature map.

4. The infrared and visible light image fusion method based on spatial perception adaptive fusion according to claim 3, characterized in that, The amplitude-aware linear attention module processing in step 4 includes the following steps: Step 4-1: Process the fused feature map of the 48 channels through a 3×3 convolutional layer and the LeakyReLU activation function to obtain 64-channel features; Step 4-2: The 64-channel features are split into four branches, Q, K, V, and O, using a 1×1 convolutional feature splitting unit. The ELU activation function and non-negativity constraints are then applied to the Q and K branches respectively. Step 4-3: Determine the amplitude-aware attention score matrix based on the processed Q branch. Simultaneously, add rotation position encoding to the Q and K branches respectively and calculate the KV key-value interaction matrix. Perform local position feature enhancement on the V branch through 5×5 grouped convolution. Step 4-4: The amplitude-aware attention score matrix, the KV key-value interaction matrix, and the V branch after local position feature enhancement are fused and modulated, and then combined with the O branch to obtain 64-channel features with fused multi-dimensional perception.

5. The infrared and visible light image fusion method based on spatial perception adaptive fusion according to claim 4, characterized in that, The multi-scale decoder processing in step 5 includes the following steps: Step 5-1: Decode the 64-channel features fused with multi-dimensional perception through 3×3 convolution and LeakyReLU activation function to obtain 32-channel features; Step 5-2: Decode the 32-channel features using a 3×3 convolution and LeakyReLU activation function to obtain 16-channel features; Step 5-3: Process the 16-channel features through 1×1 convolution and Tanh activation function to obtain a single-channel fused image.

6. The infrared and visible light image fusion method based on spatial perception adaptive fusion according to claim 5, characterized in that, Step 7 includes the following steps: Step 7-1: Define the joint loss function, which includes fusion loss and segmentation loss; Step 7-2: Connect the fusion network consisting of the dual-branch encoder, spatially aware adaptive fusion module, amplitude-aware linear attention module, and multi-scale decoder from steps 2 to 5 in series with the semantic segmentation network. Input the three-channel fused image into the semantic segmentation network to obtain the segmentation result. Calculate the error using the joint loss function and backpropagate iteratively to adjust the parameters of each module in the fusion network. During training, an adaptive weight strategy is adopted to gradually increase the weight ratio of semantic loss, balancing visual quality and semantic adaptability. Step 7-3: When the joint loss described in step 7-2 converges, terminate the training, save the optimal parameters of the fusion network, and generate the final three-channel fused image that simultaneously meets the visual quality requirements and the adaptability to high-level visual tasks.

7. A system for infrared and visible light image fusion based on spatially perceptive adaptive fusion according to claims 1-6, characterized in that, Includes the following modules: The image preprocessing module is used to preprocess the original visible light image and infrared image, including color space conversion, channel extraction and size unification, to generate single-channel input data; The dual-branch encoder module is used to perform progressive feature extraction on the preprocessed single-channel input data. It generates a high-level feature map with 48 channels through convolutional layers, densely connected blocks, Sobel gradient operators, and channel attention mechanisms. The spatially aware adaptive fusion module is used to fuse the feature maps output by the dual-branch encoder. It achieves high-quality and refined dynamic feature fusion through a chain mechanism of feature selection, representation optimization, and weight adaptation, generating 48-channel fused features. The amplitude-aware linear attention module is used to enhance the amplitude and position of features during the decoding process. It outputs enhanced features through linear attention mechanism and gating fusion. The multi-level decoder module is used to fuse enhanced features, gradually reducing the number of channels and reconstructing spatial details through multiple decoding layers, and outputting a single-channel fused image. The image post-processing module is used to combine the single-channel fused image with the chroma channel of the visible light image, and generate the final three-channel RGB fused image through color space conversion; The phased training control module is used to dynamically adjust the weights of fusion loss and segmentation loss during model training to achieve phased optimization.

8. The system of an infrared and visible light image fusion method based on spatial perception adaptive fusion according to claim 7, characterized in that, The spatial perception adaptive fusion module specifically includes the following modules: The spatial attention generation module receives the infrared and visible light dual-branch 48-channel high-level feature maps output by the dual-branch encoder module, and realizes the filtering of effective information and accurate positioning of key information regions in the dual-branch features, and outputs spatial attention weights. The representation optimization module is used to modulate the original 48-channel high-level feature maps of infrared and visible light using spatial attention weights. By strengthening key information regions and suppressing redundant information such as background or noise, the dual-modal feature representation is optimized and the modulated dual-modal features are output. The adaptive weight generation module is used to aggregate the optimized bimodal features and mine global context information to dynamically generate infrared and visible light modal adaptive fusion weights that are adapted to the current features. At the same time, based on these weights, complementary fusion of the optimized bimodal features is realized, and a high-quality fusion feature map with 48 channels is output.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method as described in any one of claims 1 to 6.

10. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of claims 1 to 6.

Images