Infrared and visible light image fusion method and system, and storage medium

Abstract

This application relates to a method, system, and storage medium for infrared and visible light image fusion, belonging to the field of image processing. The infrared and visible light image fusion method includes: extracting multi-scale features from infrared and visible light images separately using a dual-stream encoder; unifying local window attention and global non-causal state space modeling through a global semantic coding block to capture long-range dependencies; introducing a semantic reordering module to proximate spatially discontinuous but semantically related feature regions in the computational graph, enhancing cross-modal semantic alignment; and utilizing a lightweight image reconstruction module to progressively fuse multi-scale features and reconstruct a high-quality fused image. This application effectively addresses the shortcomings of existing methods in global context modeling, computational efficiency, and cross-modal semantic alignment, significantly improving the thermal target saliency and texture detail preservation capabilities of the fused image, making it suitable for fields such as nighttime surveillance and autonomous driving perception.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an infrared and visible light image fusion method, system, and medium based on global context modeling and semantic reordering. Background Technology

[0002] Infrared and visible light image fusion technology plays a crucial role in vision-related applications such as nighttime surveillance, autonomous navigation, and remote sensing. Infrared images can capture the thermal radiation of objects, enabling stable perception under low light or inclement weather conditions; visible light images, on the other hand, provide fine texture and structural details under normal lighting conditions. Effectively fusing these two modalities can generate images that possess both thermal saliency and structural integrity, supporting scene understanding in complex environments.

[0003] However, infrared and visible light image fusion still faces many challenges. The two modalities have inherent domain differences in spectral response, spatial texture, and semantic focus, making feature alignment difficult. Traditional fusion methods, such as multi-scale transformations and sparse representations, rely on manually designed rules and lack the ability to model high-level semantic relationships. While convolutional neural network-based methods improve fusion results through data-driven approaches, they are limited by local receptive fields and struggle to capture global long-range dependencies. The Transformer architecture models global information through self-attention mechanisms, but its computational complexity is quadratic with the input resolution, making it unsuitable for high-resolution image fusion. State-space model-based methods can achieve global modeling with linear complexity, but most employ causal structures, limiting the acquisition of bidirectional spatial information and ignoring the correlation between spatially separated but semantically related regions, resulting in poor global consistency of the fused image. Summary of the Invention

[0004] The purpose of this application is to provide an infrared and visible light image fusion method, system and medium. Through global semantic coding, non-causal state modeling and semantic reordering mechanism, it achieves efficient fusion of infrared and visible light images and improves the visual quality and information integrity of the fused image.

[0005] To achieve the above objectives, this application provides the following technical solution: In a first aspect, embodiments of this application provide a method for fusing infrared and visible light images, comprising the following steps: Step 1: Use infrared and visible light images as the dataset and perform preprocessing; Step 2: Patch embedding and multi-scale feature extraction are performed on the infrared and visible light images respectively using a dual-stream encoder to construct a spatial pyramid feature representation; Step 3: Use global semantic coding blocks to process features at various scales, capture local details through window self-attention, and combine non-causal state space modules to model global long-range dependencies; Step 4: Based on the output of the dynamic cue selector of the non-causal state space module, perform semantic reordering and feature alignment on the feature map to obtain aligned multi-scale features. Step 5: Using the image reconstruction module, hollow spatial pyramid pooling and cross-scale feature stitching are employed to aggregate and align the multi-scale features to generate a high-resolution fused image. Step 6: Optimize the fusion process using a joint loss function and output the final fused image.

[0006] Furthermore, in step 2, the dual-stream encoder designs independent branches for infrared and visible light images respectively. The processing of each branch is as follows: first, the input image is converted into patch embedding features through a convolutional layer, and then hierarchical feature extraction is performed at multiple resolution levels. A spatial pyramid structure is constructed between each level through downsampling operations. The features of each resolution level are input into the global semantic coding block for processing.

[0007] Furthermore, the global semantic encoding block in step 3 comprises four consecutive stages: window self-attention, feedforward transformation, non-causal semantic modeling, and lightweight refinement. Each stage employs layer normalization and residual connections. Among them, window self-attention divides features into non-overlapping windows, computes multi-head self-attention to capture fine-grained texture and edge features, and feedforward transformation and lightweight refinement are both implemented using multilayer perceptron (MLP).

[0008] Furthermore, in step 3, the non-causal state space module embeds a learnable cue matrix, and generates instance-specific cues through a dynamic cue selector and a low-rank cue pool, as shown in the following formula:

[0009] Where P is the cue matrix, and S is the dynamic cue selector, which is generated using the Gumbel-Softmax function. For the low-rank cue pool, Proj is the projection operation, and F is the input feature map; the non-causal state space update formula is as follows:

[0010] Where t is the position index in the sequence. In hidden state, To enter a token, Let A, B, C, and D be the t-th element in the cue matrix P, and let A, B, C, and D be the learnable projection matrices. To output the token.

[0011] Furthermore, step 4, which performs semantic reordering and feature alignment on the 2D feature map, specifically includes: first, flattening the feature map into a 1D token sequence; calculating the similarity between each token feature vector and each prototype vector in the cue matrix; using the prototype with the highest similarity as the semantic label of the token; grouping and sorting the token sequence based on the semantic labels, so that tokens with the same or similar semantic labels are adjacent in the sequence; inputting the reordered tokens into the non-causal state space module for global modeling; after modeling, restoring the original spatial structure of the feature map through an inverse reordering operation to ensure feature alignment.

[0012] Furthermore, the image reconstruction module employs a coarse-to-fine fusion strategy, including: The coarsest-scale features are input into the dilated spatial pyramid pooling module, and processed in parallel by multiple 3×3 convolutions with different dilation rates and global average pooling. The outputs of each branch are concatenated along the channel dimension and then adjusted for the number of channels by a 1×1 convolution. The processed features are then upsampled once and convolved... Normalization After the ReLU activation module is refined by the CBR module, it is fused with the intermediate resolution level features and then refined by the CBR module. The output features are then upsampled and refined by the CBR module again before being fused with the fine-scale features. Finally, the CBR module refines the features again to generate the final fused image.

[0013] Furthermore, the joint loss function in step 6 includes intensity loss and gradient loss. The intensity loss ensures that the fused image retains significant information from both modalities by minimizing the L1 distance between the element-wise maximum values ​​of the fused image and the source image. The gradient loss encourages the fused image to retain edge and texture details by comparing the gradient responses of the source image and the fused image. The two losses are balanced by a weight factor λ to achieve an optimal balance between contrast and detail integrity in the fused image.

[0014] Furthermore, the formula for the joint loss function is as follows:

[0015]

[0016]

[0017] in, For strength loss, The gradient loss is λ, where λ is the weighting factor. , , These are infrared, visible light, and fused images, respectively. Here, H and W are the Sobel gradient operator, and H and W are the image height and width, respectively.

[0018] Secondly, the present invention provides an infrared and visible light image fusion system, the system comprising: a memory and a processor, the memory including a program for an infrared and visible light image fusion method, wherein when the program for the infrared and visible light image fusion method is executed by the processor, the steps of the infrared and visible light image fusion method described above are implemented.

[0019] Thirdly, the present invention provides a computer-readable storage medium storing program code, which, when executed by a processor, implements the steps of the infrared and visible light image fusion method as described above.

[0020] Compared with existing technologies, the advantages of this invention are as follows: It employs a dual-stream encoder design to process infrared and visible light images separately, effectively preserving the unique features of each modality. The global semantic coding block, combined with window self-attention and a non-causal state space module, captures local details while modeling global long-range dependencies with linear complexity, overcoming the performance bottleneck of traditional methods. The semantic reordering mechanism groups semantically related regions, strengthening cross-modal feature alignment and improving the global consistency of the fused image. The image reconstruction module controls computational overhead while ensuring fusion quality through hollow spatial pyramid pooling and cross-scale feature aggregation. The joint loss function balances hot saliency with structural detail preservation, resulting in excellent performance of the fused image in both objective metrics and visual quality. Experiments on public datasets such as TNO, MSRS, and M3FD show that this method outperforms existing CNN, Transformer, and state space model-based methods in terms of standard deviation, average gradient, and entropy, making it suitable for practical scenarios such as nighttime surveillance and autonomous perception. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 is the overall network architecture diagram of this application; Figure 2 is a diagram of the global semantic coding block structure of this application; Figure 3 is a structural diagram of the non-causal state space module of this application; Figure 4 is a flowchart of the image reconstruction module of this application; Figure 5 is a qualitative analysis diagram of the experimental results of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0024] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0025] The terms “first,” “second,” etc., are used only to distinguish one entity or operation from another, and should not be construed as indicating or implying relative importance, nor as requiring or implying any such actual relationship or order between these entities or operations.

[0026] As shown in Figure 1, the present invention provides an infrared and visible light image fusion method, which includes the following steps: Step 1: Data Preparation and Preprocessing: Infrared and visible light images are used as the dataset, which is divided into training, validation, and test sets. Infrared and visible light image pairs are collected from different scenes (such as nighttime streets, urban buildings, and severe weather conditions) to ensure that the dataset covers various complex situations such as changes in lighting and weather interference. The images undergo uniform preprocessing, including resizing and normalization. The dataset is then divided into training, validation, and test sets according to a preset ratio (e.g., 7:1.5:1.5) for model training, parameter validation, and performance testing.

[0027] In this embodiment, infrared and visible light image pairs from publicly available datasets such as TNO, MSRS, and M3FD were collected, covering various scenes including nighttime streets, urban buildings, and rainy / smoky conditions. The images underwent uniform preprocessing, with the image size adjusted to 256×256 and pixel values ​​normalized to the [0,1] range. The dataset was divided into a 70% training set, 15% validation set, and 15% test set. During training, data augmentation strategies such as random cropping and flipping were employed to improve the model's generalization ability.

[0028] Step 2: Dual-stream encoder feature extraction: A spatial pyramid feature representation is constructed by performing patch embedding and multi-scale feature extraction on infrared and visible light images using a dual-stream encoder. The dual-stream encoder designs independent branches for infrared and visible light images, preserving the unique features of each modality. First, the input image is converted into patch embedding features through a convolutional layer. Then, hierarchical feature extraction is performed at three resolution levels, with downsampling operations between levels to construct a spatial pyramid structure. Features at each resolution level contain semantic information at different scales; low-level features focus on detailed textures, while high-level features emphasize global semantics, providing a rich feature foundation for subsequent fusion.

[0029] In this embodiment, the dual-stream encoder designs independent feature extraction branches for infrared and visible light images, with a completely symmetrical structure. Each branch first converts the input image into patch embedding features through a 3×3 convolutional layer, with a stride of 4 and 64 output channels. Subsequently, a spatial pyramid is constructed through two downsampling modules, each consisting of a convolutional layer and a batch normalization layer, with a stride of 2 and the number of channels doubling sequentially to 128, 256, and 512. The output features of each resolution level are input into the global semantic coding block for further processing.

[0030] Step 3: Global Semantic Encoding Block Processing: This step captures local details through window self-attention and models global long-range dependencies using a non-causal state space module. The global semantic encoding block is the core module of the backbone network, unifying local detail modeling with global contextual reasoning. This module comprises four consecutive stages: window self-attention, feedforward transformation, non-causal semantic modeling, and lightweight refinement. Window self-attention divides features into non-overlapping windows, calculating multi-head self-attention within each window to effectively capture local texture and edge features. Feedforward transformation enhances channel interaction capabilities through two MLP layers. The non-causal state space module models global long-range dependencies with linear complexity, overcoming the limitations of local receptive fields. The lightweight refinement module normalizes and optimizes features, improving feature separability. Each stage employs pre-normalization and residual connections to ensure stable model optimization and consistent feature scale.

[0031] Specifically, such as Figure 2 As shown, the input to the global semantic coding block is the feature map of each resolution level, and the processing procedure is as follows: 1) Window Self-Attention: The feature map is divided into 8×8 non-overlapping windows. Multi-head self-attention (8 heads) is computed within each window to capture local texture and edge features. Relative positional bias is learned through training to enhance the ability to model positional information. The formula is as follows:

[0032] Where Q, K, and V represent the query, key, and value features, respectively. denoted as head dimension, and b as the learnable relative position offset.

[0033] 2) Feedforward transform: An MLP structure is adopted, with the number of intermediate layer channels being 4 times the number of input channels. The activation function is GELU, which enhances channel interaction and feature representation capabilities.

[0034] 3) Non-causal semantic modeling: A learnable cue matrix is ​​embedded, and the dynamic cue selector is generated using Gumbel-Softmax. The rank of the low-rank cue pool is set to 64. The hidden state dimension of the non-causal state space module is set to 128, and global long-range dependency modeling is achieved through parallel updates.

[0035] A non-causal state-space module is embedded with a learnable cue matrix, and instance-specific cues are generated through a dynamic cue selector and a low-rank cue pool, as shown in the following formula:

[0036] Where P is the cue matrix and S is the dynamic cue selector, which is generated by the Gumbel-Softmax function. This function applies noise to the class probability distribution and performs continuous relaxation, so that the discrete cue selection process remains differentiable during the training phase, thereby supporting end-to-end gradient optimization. For the low-rank cue pool, Proj is the projection operation, and F is the input feature map; the non-causal state space update formula is as follows:

[0037] Where t is the position index in the sequence. In hidden state, To enter a token, Let A be the cue matrix obtained in the previous step, and let A, B, C, and D be the learnable projection matrices. To output the token.

[0038] 4) Lightweight Refinement: Layer normalization and multilayer perceptron are used to further refine the features to enhance their nonlinear expressive power. This step, like the previous modules, has residual connections to maintain the stability of gradient flow.

[0039] Step 4: Semantic Reordering and Feature Alignment: Based on the output of the dynamic cue selector from the non-causal state space module, the 2D feature map is semantically reordered. First, the feature map is flattened into a 1D token sequence. The similarity between each token's feature vector and each prototype vector in the semantic cue matrix is ​​calculated, and the prototype with the highest similarity is used as the semantic label for that token. The token sequence is then grouped and sorted based on these semantic labels, ensuring that tokens with the same or similar semantic labels are adjacent in the sequence. The reordered tokens are input into the non-causal state space module for global modeling. After modeling, the original spatial structure of the feature map is restored through an inverse reordering operation, ensuring feature alignment.

[0040] Among them, such as Figure 3 As shown, the semantic reordering mechanism transforms the 2D feature map into a 1D token sequence based on the dynamic cue selector results in the non-causal state space module. This mechanism groups spatially separated but semantically related regions into the same computational neighborhood, enabling the non-causal state modeling process to effectively capture the correlations between these regions and avoid directional biases in traditional sequence processing. After non-causal state modeling, the token sequence is restored to its original spatial configuration through an inverse reordering operation, ensuring feature alignment.

[0041] Step 5: Using the image reconstruction module, hollow spatial pyramid pooling and cross-scale feature stitching are employed to aggregate multi-scale features and generate a high-resolution fused image; for example... Figure 4 As shown, the image reconstruction module employs a coarse-to-fine feature aggregation strategy to generate a high-quality fused image with minimal overhead: 1) Hollow Spatial Pyramid Pooling: Four 3×3 convolutions with different dilation rates (1, 3, 6, 12) are applied in parallel with global average pooling to the coarsest-scale feature (512 channels). This captures multi-scale contextual information without increasing resolution. The outputs of each branch are concatenated along the channel dimension and then adjusted to 256 channels using a 1×1 convolution. The formula is as follows: ) in, This is a characteristic of the hollow space pyramid pooling process. - For convolution operations with dilation rates r1-r4, Indicates global average pooling; 2) Cross-scale feature fusion: The processed coarse-scale features are amplified by 2 times through bilinear upsampling and then fused by convolution. Normalization After the ReLU activation module is refined into a CBR module, it is concatenated with the features of the intermediate resolution level (256 channels). After being refined by the convolution-normalization-ReLU activation module, the output features are then upsampled and refined again by the CBR module and concatenated with the fine-scale features (128 channels). Finally, the fused image is output through a 3×3 convolution.

[0042] Step 6: Optimize the fusion process using a joint loss function to balance thermal saliency with the preservation of structural details. The joint loss function consists of intensity loss and gradient loss. Intensity loss ensures the fused image retains saliency information from both modalities by minimizing the L1 distance between the element-wise maximum values ​​of the fused image and the source image; gradient loss encourages the fused image to retain edge and texture details by comparing the gradient responses of the source and fused images. The two losses are balanced by a weighting factor λ to achieve an optimal balance between contrast and detail integrity in the fused image. The formula is as follows:

[0043]

[0044]

[0045] in, For strength loss, The gradient loss is λ, where λ is the weighting factor. , , These are infrared, visible light, and fused images, respectively. Here, H and W are the Sobel gradient operator, and H and W are the image height and width, respectively. This embodiment uses the Adam optimizer for model training, with an initial learning rate of 1e-4 and a weight decay coefficient of 1e-5. The weight factor λ of the joint loss function is set to 10 to balance the intensity loss and gradient loss. An early stopping strategy is employed during training: training stops when the validation set loss shows no decrease for 10 consecutive epochs, and the optimal model parameters are saved. Model training is performed on an NVIDIA RTX A6000 GPU, with a batch size of 8 and 100 training epochs.

[0046] Step 7, Experimental Verification and Performance Comparison: To verify the beneficial effects of this invention, quantitative and qualitative experiments were conducted on the public datasets TNO, MSRS, and M3FD, and compared with existing typical methods (including DenseFuse, DIDFuse, RFN-Nest, SDNet, U2Fusion, UMF-CMGR, LRRNet, SFINet, and HAIAFusion). The quantitative comparison results are shown in Tables 1-3, with the best results highlighted in bold. The following objective evaluation metrics were used to assess the fusion quality: Standard Deviation (SD) reflects image contrast, Average Gradient (AG) measures the richness of texture details, Entropy (EN) assesses the amount of information, Edge Information Preservation (Qabf) measures the ability to preserve edge structures, and Spatial Frequency (SF) reflects the overall activity of the image. Higher values ​​for each metric indicate better performance. Experimental results show that the method of this invention achieves the best overall performance on all three datasets. On the TNO dataset, the standard deviation (SD) of this invention reached 9.18720, the average gradient (AG) reached 6.29744, the entropy (EN) reached 6.94628, and the spatial frequency (SF) reached 0.06447, all ranking first. This indicates that the fused image generated by this invention has higher contrast, richer texture details, and stronger edge structure preservation ability. On the MSRS dataset, the five indicators of SD, AG, EN, Qabf, and SF of this invention all achieved first place or very close to first place, especially significantly leading other methods in the AG and SF indicators, which reflect image sharpness. On the M3FD dataset, this invention also achieved first place in the five indicators of SD, AG, EN, Qabf, and SF, verifying the robustness and generalization ability of the method under different scenarios and sensor conditions. Figure 5 As shown in the figure, the red and yellow boxes mark two key local areas, used to compare and demonstrate the differences between the present invention and existing methods in terms of thermal target saliency and texture detail preservation. The fused image generated by the present invention has a clear and complete thermal target outline, sufficient preservation of visible light texture details (such as building edges, road signs, etc.), and a natural overall brightness distribution without obvious artifacts or blurring. The subjective visual quality is significantly better than the comparison method.

[0047] Table 1 Quantitative Comparison Results of TNO Datasets

[0048] Table 2 Quantitative Comparison Results of MSRS Datasets

[0049] Table 3 Quantitative Comparison Results of the M3FD Dataset

[0050] This application provides an infrared and visible light image fusion system, which includes a memory and a processor. The memory includes a program for an infrared and visible light image fusion method. When the program for the infrared and visible light image fusion method is executed by the processor, it implements the steps of the infrared and visible light image fusion method.

[0051] This application provides a computer-readable storage medium storing program code. When the program code is executed by a processor, it implements the steps of the infrared and visible light image fusion method described above.

[0052] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0053] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0054] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0055] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0056] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0057] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0058] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0059] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for fusing infrared and visible light images, characterized in that, Includes the following steps: Step 1: Use infrared and visible light images as the dataset and perform preprocessing; Step 2: Patch embedding and multi-scale feature extraction are performed on the infrared and visible light images respectively using a dual-stream encoder to construct a spatial pyramid feature representation; Step 3: Use global semantic coding blocks to process features at various scales, capture local details through window self-attention, and combine non-causal state space modules to model global long-range dependencies; Step 4: Based on the output of the dynamic cue selector of the non-causal state space module, perform semantic reordering and feature alignment on the feature map to obtain aligned multi-scale features. Step 5: Using the image reconstruction module, hollow spatial pyramid pooling and cross-scale feature stitching are employed to aggregate and align the multi-scale features to generate a high-resolution fused image. Step 6: Optimize the fusion process using a joint loss function and output the final fused image.

2. The infrared and visible light image fusion method as described in claim 1, characterized in that: In step 2, the dual-stream encoder designs independent branches for infrared and visible light images respectively. The processing of each branch is as follows: first, the input image is converted into patch embedding features through a convolutional layer, and then hierarchical feature extraction is performed at multiple resolution levels. A spatial pyramid structure is constructed between each level through downsampling operations. The features of each resolution level are input into the global semantic coding block for processing.

3. The infrared and visible light image fusion method as described in claim 1, characterized in that: Step 3, the global semantic encoding block, consists of four consecutive stages: window self-attention, feedforward transformation, non-causal semantic modeling, and lightweight refinement. Each stage employs layer normalization and residual connections. Among them, window self-attention divides features into non-overlapping windows, calculates multi-head self-attention to capture fine-grained texture and edge features, and feedforward transformation and lightweight refinement are both implemented using multilayer perceptron (MLP).

4. The infrared and visible light image fusion method as described in claim 1 or 3, characterized in that: In step 3, a learnable cue matrix is ​​embedded into the non-causal state space module. Instance-specific cues are generated through a dynamic cue selector and a low-rank cue pool, as shown in the following formula: Where P is the cue matrix, and S is the dynamic cue selector, which is generated using the Gumbel-Softmax function. For the low-rank cue pool, Proj is the projection operation, and F is the input feature map; the non-causal state space update formula is as follows: Where t is the position index in the sequence. In hidden state, To enter a token, Let A, B, C, and D be the t-th element in the cue matrix P, and let A, B, C, and D be the learnable projection matrices. To output the token.

5. The infrared and visible light image fusion method as described in claim 4, characterized in that: Step 4, which performs semantic reordering and feature alignment on the 2D feature map, specifically includes: first, flattening the feature map into a 1D token sequence; calculating the similarity between each token feature vector and each prototype vector in the cue matrix; using the prototype with the highest similarity as the semantic label of the token; grouping and sorting the token sequence based on the semantic labels, so that tokens with the same or similar semantic labels are adjacent in the sequence; inputting the reordered tokens into the non-causal state space module for global modeling; after modeling, restoring the original spatial structure of the feature map through an inverse reordering operation to ensure feature alignment.

6. The infrared and visible light image fusion method as described in claim 1, characterized in that: The image reconstruction module employs a coarse-to-fine fusion strategy, including: The coarsest-scale features are input into the dilated spatial pyramid pooling module, and processed in parallel by multiple 3×3 convolutions with different dilation rates and global average pooling. The outputs of each branch are concatenated along the channel dimension and then adjusted for the number of channels by a 1×1 convolution. The processed features are then upsampled once and convolved... Normalization After the ReLU activation module is refined by the CBR module, it is fused with the intermediate resolution level features and then refined by the CBR module. The output features are then upsampled and refined by the CBR module again before being fused with the fine-scale features. Finally, the CBR module refines the features again to generate the final fused image.

7. The infrared and visible light image fusion method as described in claim 1, characterized in that: In step 6, the joint loss function includes intensity loss and gradient loss. Intensity loss ensures that the fused image retains significant information from both modalities by minimizing the L1 distance between the element-wise maximum values ​​of the fused image and the source image. Gradient loss encourages the fused image to retain edge and texture details by comparing the gradient responses of the source image and the fused image. The two losses are balanced by a weight factor λ to achieve an optimal balance between contrast and detail integrity in the fused image.

8. The infrared and visible light image fusion method as described in claim 7, characterized in that: The formula for the joint loss function is as follows: in, For strength loss, The gradient loss is λ, where λ is the weighting factor. , , These are infrared, visible light, and fused images, respectively. Here, H and W are the Sobel gradient operator, and H and W are the image height and width, respectively.

9. An infrared and visible light image fusion system, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the program instructions in the memory to execute the infrared and visible light image fusion method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It includes a readable storage medium on which a computer program is stored, and when the computer program is executed, it implements the infrared and visible light image fusion method as described in any one of claims 1-8.

Images