Infrared and visible light image fusion method based on semantic prior

Abstract

The invention discloses an infrared and visible light image fusion method based on semantic prior. The method comprises the following five steps: defining a source input image and network structure parameters, constructing a three-level optimization objective function, constructing a grid coding network by adopting a residual representation module with a two-dimensional scanning characteristic, constructing a double-branch multi-level visual priori self-prompting mechanism, and performing staged processing on the characteristics by prompting an interactive fusion network. According to the method, the problem that visual quality and downstream task adaptability are difficult to consider in an existing method is effectively solved, semantic consistency, space structure integrity and visual effect of the fused image are remarkably improved, dependence on manual annotation is reduced, generalization ability and robustness of a model in a complex scene are enhanced, and the method is suitable for popularization and application. The method is suitable for multiple fields of robots, remote sensing, automatic driving and the like.

Description

Infrared and Visible Image Fusion Method Based on Semantic Prior Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method for fusing infrared and visible light images based on semantic priors. This technology is widely used in robotics, remote sensing, and autonomous driving, aiming to address how to effectively fuse infrared and visible light images in complex environments such as low light, smoke, and occlusion, thereby improving visual quality and the adaptability and accuracy of downstream perception tasks, such as object detection and semantic segmentation. Background Technology

[0002] Given that infrared imaging is robust in low-light, smoke, and obstructed conditions, while visible light images have significant advantages in resolution, contrast, and texture detail, infrared and visible light image fusion (IVIF) plays a crucial supporting role in tasks in fields such as robotics, remote sensing, and autonomous driving.

[0003] However, balancing high visual quality with the performance requirements of downstream perceptual tasks (such as object detection and semantic segmentation) during the fusion process remains a challenging problem that urgently needs to be solved in this field. Traditional infrared-visible light fusion methods are mostly based on information theory, with the core objective of maximizing the preservation of source image information. For example, the SDNet method relies on multi-scale decomposition combined with guided filtering techniques to achieve image fusion, but these methods have limitations in handling information redundancy and adapting to scene-specific characteristics, making it difficult to continuously optimize the fusion quality in complex and ever-changing scenarios. Early deep learning fusion methods, such as the DenseFuse method which uses a densely connected block encoder-decoder architecture, mainly focus on improving visualization effects by enhancing image details and contrast through network structure, but they generally suffer from defects such as edge blurring, artifacts, and semantic inconsistencies, which seriously affect their adaptability to downstream tasks. While task-driven fusion methods introduce task supervision such as segmentation and detection during training to align the fusion results with specific application scenarios, such as the DIDFuse method, these methods are highly dependent on manually labeled data. They have weak generalization ability for unseen tasks and domain shifts, and there is an inherent trade-off between task objectives and visual quality, making it difficult to simultaneously ensure semantic consistency and spatial structural integrity.

[0004] In recent years, the rise of large-scale pre-trained visual models has provided new directions for the development of IVIF technology. General semantic segmentation models and vision-language models can provide transferable high-level semantic priors, guiding the fusion network to focus on key image regions, thereby improving the semantic consistency and task adaptability of the fusion results. However, how to efficiently integrate such semantic priors into the fusion network, supplementing cross-modal structural details while avoiding the image structure degradation problem caused by prior introduction, still lacks high-performance solutions. Summary of the Invention

[0005] To overcome the limitations of existing methods, this invention proposes a semantic prior-based infrared and visible light image fusion method. By finely utilizing the semantic prior in the pre-trained network, it achieves synergistic optimization of visual quality and task adaptability. Specifically, firstly, an enhanced two-dimensional multi-directional scanning strategy is introduced, which simultaneously captures long-range dependencies and short-range local interactions through multi-directional sequence modeling, balancing global semantics and local details while maintaining the integrity of spatial structure. Secondly, a prior-guided dual-branch meta-network is designed, where the semantic segmentation branch provides high-level semantic information for the infrared image, and the detail enhancement branch contributes structural features to the visible light image. The two construct a complementary multimodal prior system, fully exploiting the value of pre-trained knowledge. Furthermore, a memory-enhanced attention module is introduced to achieve unidirectional knowledge transfer from the pre-trained model to the target encoder and fusion network, improving the stability and efficiency of prior utilization.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an infrared and visible light image fusion method based on semantic prior, comprising five steps: S-1: defining the source input image and network structure parameters, setting... This represents the source visible light and infrared input image. This indicates the corresponding semantic prior hints generated under the guidance of the meta-network. For parameters Meta-network, For parameters The coding network, For parameters The fusion network; S-2: Construct a three-level optimization objective function, and optimize the meta-network, encoding network and fusion network in sequence. The optimization process satisfies: ; ; ;in, The loss function of the detail enhancement network is used to ensure the generation of meaningful visual prior cues; Guide the encoder to learn effective feature representations; Supervise the high-quality image fusion process; S-3: Construct a grid-coded network using a residual representation module with two-dimensional scanning characteristics, and capture long-range spatial dependencies and local neighborhood interactions through block-level water wave information transmission and recursive acceleration network PW-WITRAN and horizontal-vertical gated selection unit HVGSU; S-4: Construct a dual-branch multi-level visual prior self-cue mechanism, extract high-level semantic features of infrared images through semantic segmentation branch, extract structural features of visible light images through detail enhancement branch, and construct a visual cue modulator by combining memory attention mechanism MAM; S-5: Process features in stages through a cue interaction fusion network. First, in the intramodal fusion stage, an adaptive grid network is used to fuse multi-scale features and introduce cue information. Then, in the intermodal interaction stage, cross-modal selective information exchange is achieved through MAM. Finally, the modal contribution weights are estimated and weighted summation is performed to generate the fusion result. .

[0007] Preferably, the operation process of the residual representation module described in S-3 is as follows: given input depth features First, layer-normalized LN is performed, then spatial long-range dependencies are captured through the WITRAN module, combined with learnable scale factors. Controlling the jump connection information flow satisfies: ,in To output information.

[0008] Preferably, the PW-WITRAN processing procedure in S-3 includes: S-3-1 dividing the input image into blocks to obtain... S-3-2 is processed through the HVGSU unit. Obtain the hidden states of the horizontal and vertical outputs. and ,in and , R represents the depth of the model, and R and C represent the number of rows and columns after image segmentation, respectively. For model feature dimensions; S-3-3 pairs and After concatenation and inverse block processing, the output is obtained. ,in .

[0009] Preferably, the operation of the HVGSU unit in S-3 satisfies: ; ; ;in, , 1≤r≤R, 1≤c≤C, 1≤l≤L when hour, This represents the input of the first layer; when When, it indicates the input to the subsequent layer. and These represent the horizontal and vertical hidden state inputs of the current cell, respectively. This indicates the output of the current cell.

[0010] Preferably, the gated selection unit (GSC) includes a selection gate and an output gate, and the operation process satisfies: ; ; ; ; ;in, , These represent the primary hidden state and the secondary hidden state, respectively, and x represents the input information; Let b be the weight matrix and b be the bias vector. and These represent the selection gate and the output gate, respectively; ⊙ represents element-wise multiplication; and For activation functions; and Indicates intermediate variables. This indicates that the output is hidden.

[0011] Preferably, the specific implementation of the dual-branch multi-level visual prior self-prompting mechanism described in S-4 includes: S-4-1 using a GRID feature extractor, with residual representation RP blocks as basic nodes, to obtain multi-level features. S-4-2 constructs three visual cue pools for each layer of features. , For the first Layer channel dimension; S-4-3 replaces the Long Short-Term Memory (LSTM) network in the Feature Enhancement Module (FEB) with GSC, through To achieve the integration of prior knowledge, among which To integrate prior and subsequent features.

[0012] Preferably, as described in S-2 The loss function for the detail enhancement network employs a self-supervised learning strategy and consists of a weighted average of intensity loss and structural similarity loss. ,in: ; ; and For adaptive weights, For weighted parameters, and The images show the reconstruction results for infrared and visible light images, respectively.

[0013] Preferably, the detail enhancement branch in S-4 adopts the Structural Detail Enhancement Network (SDEN), whose loss function is composed of a weighted average of the reconstruction loss and the structural tensor loss: ; ; ; This is the result of enhanced details in a visible light image. For weighted parameters, This represents the Frobenius norm.

[0014] Preferably, the loss function of the prompting interaction fusion network described in S-5 is: ;in: ; ; ; Where H and W represent the height and width of the image, respectively. Here, α is the weighting coefficient, ∇ is the weighting parameter, and the symbol ∇ represents the gradient operator. This represents absolute value operations.

[0015] Preferably, the estimation process of the modal contribution weights described in S-5 satisfies: ;in ; To contribute to the estimation results, These are feature maps at different scales. As a visual cue, the fusion result is obtained through Calculations show that ⊙ represents element-wise multiplication.

[0016] This invention introduces a semantic prior self-prompting mechanism based on a pre-trained model, which organically integrates high-level semantic information with low-level structural details. This effectively solves the technical pain point in existing infrared and visible light image fusion methods where visual quality and downstream perception tasks (such as object detection and semantic segmentation) are difficult to balance. It achieves synergistic optimization of semantic consistency, spatial structural integrity and visual perception effect, and significantly improves the practical value of fused images.

[0017] The enhanced two-dimensional multi-directional scanning strategy and gridded coding structure proposed in this invention can simultaneously and accurately capture the long-range dependencies and local neighborhood interaction features of images, completely avoiding the spatial structure destruction problem that is easily caused by traditional one-dimensional sequence modeling. Combined with a bi-branch prior meta-network and a prompting interaction fusion learning mechanism, it greatly enhances the modeling ability of complementary information between infrared and visible light modes, effectively suppressing structural degradation and detail loss during the fusion process, and ensuring the clarity and integrity of the fused image.

[0018] This invention employs a one-way prior prompting and memory-enhanced attention mechanism to achieve stable and efficient injection of pre-trained knowledge into the target fusion network, avoiding model instability and high computational overhead caused by end-to-end joint training. At the same time, the proposed joint optimization strategy of self-prompting and perceptual loss significantly reduces the dependence on large-scale manually labeled data and effectively improves the model's generalization ability and robustness in complex and variable scenarios and diverse downstream tasks. Attached Figure Description

[0019] Figure 1 is a flowchart illustrating the infrared and visible light image fusion method based on semantic prior visual self-cue learning according to an embodiment of the present invention; Figure 2 is a schematic diagram illustrating the structure of the visual cue representation learning network and the interactive fusion learning network according to an embodiment of the present invention; Figure 3 is a schematic diagram illustrating the structure of the residual representation learning module based on the 2D scanning strategy according to an embodiment of the present invention; Figure 4 is a schematic diagram illustrating the structure of the grid encoder module according to an embodiment of the present invention; Figure 5 is a schematic diagram illustrating the structure of the information interaction module according to an embodiment of the present invention; Figure 6 is a schematic diagram illustrating the qualitative results of fused images on the MSRS, M3FD, and TNO datasets; Figure 7 is a schematic diagram illustrating the qualitative results of segmentation on the MSRS dataset; Figure 8 is a schematic diagram illustrating the standardized quantitative results of parameter settings; Figure 9 is a schematic diagram illustrating the qualitative results of semantic segmentation on the MSRS dataset; Figure 10 is a schematic diagram illustrating the qualitative results of target detection on the MSRS dataset. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0021] set up This represents the source visible light and infrared input image, i.e. The corresponding semantic prior hints generated under the guidance of the meta-network are represented as follows: .set up For parameters Meta-network, For parameters The coding network, For parameters The fusion network. The optimization process is defined as follows: ,in, The loss function of the detail enhancement network is used to ensure the generation of meaningful visual prior cues; Guide the encoder to learn effective feature representations; Supervise the high-quality image fusion process. The fused output is represented as... .

[0022] It is worth noting that the prior information follows a strictly unidirectional flow throughout the network—this specific architectural feature helps stabilize the training process. During the optimization of the fusion network, the encoder's representation parameters are fine-tuned simultaneously with the fusion module, thereby achieving joint learning between the representation and fusion modules. Specifically, this invention integrates the prior information into multiple stages of the representation learning and fusion process, achieving comprehensive guidance from low-level feature extraction to high-level semantic understanding in a multi-layered manner.

[0023] Phase 1: Traditional Self-Attention Models (SSM) treat the flattened feature map as a one-dimensional (1D) token sequence. However, in a two-dimensional (2D) feature map, some spatially adjacent tiles may be far apart in the one-dimensional token sequence. This increased spatial distance leads to the loss of local pixel dependencies and relevant information. To address this issue, this invention employs a residual representation module with two-dimensional scanning characteristics to construct a mesh coding network.

[0024] 1. Residual Presentation: To adapt the WIT block to the recovery task, this invention introduces a residual representation (RP) block as shown in Figure 3(a). Given input depth features... First, layer normalization (LN) is performed. Next, a WITRAN module is implemented to capture long-range spatial dependencies. Furthermore, a learnable scaling factor is employed. This controls the flow of information through jump connections. The learning process is represented as: ,in, This indicates the output information for further processing.

[0025] PW-WITRAN Scanning: Although algorithms like Vision Mamba process image patches through multi-directional scanning, they typically flatten the inherent two-dimensional spatial structure into a one-dimensional sequence, often neglecting local neighborhood relationships and resulting in the loss of important spatial context information. To preserve this structural information, this invention introduces a block-level water wave information transmission and recursive acceleration network (PW-WITRAN). Its core component, the Horizontal-Vertical Gated Selection Unit (HVGSU), adaptively selects, transmits, and fuses information along orthogonal horizontal and vertical directions using a gating mechanism. This bidirectional interaction captures the local spatial correlation between adjacent patches and constructs global semantic associations through multi-level propagation—effectively combining local detail awareness with global contextual understanding to achieve richer feature representations. The corresponding optimization process can be expressed as: ,in, and Let c represent the hidden states of the horizontal and vertical outputs, respectively. For the depth of the model, This indicates the output information.

[0026] Horizontal-Vertical Gated Selection Unit (HVGSU): Within the PW-WITRAN scanning module, this invention incorporates an HVGSU unit. The HVGSU consists of two directional gated selection units (GSCs) used to capture long-term and short-term periodic semantic information. (1≤r≤R) and column c (1≤c≤C) in layer l (1≤ The unit operations in ≤L) are formalized as follows: in, .when hour, This represents the input of the first layer; when When, it indicates the input to the subsequent layer. and These represent the horizontal and vertical hidden state inputs of the current cell, respectively. This indicates the output of the current cell.

[0027] Gated Selective Cell (GSC): As shown in Figure 3(a), the GSC consists of a selection gate and an output gate. The selection gate controls the degree of retention of primary information and the proportion of fused information, while the output gate determines the final feature representation. In the diagram, different information flows are distinguished by color: black represents input information, red represents the primary hidden state, blue represents the secondary hidden state, and purple represents the information after primary-secondary feature fusion. Its specific operation is formally represented as follows: .in, , These represent the primary hidden state and the secondary hidden state of the input, respectively, and x represents the input information; Let b be the weight matrix and b be the bias vector. and These represent the selection gate and the output gate, respectively; ⊙ represents element-wise multiplication; and For activation functions; and Indicates intermediate variables. This indicates that the output is hidden.

[0028] in, This represents the mapping representation after processing such as two-dimensional scanning. Represents a regular linear mapping. This represents the output result after aggregation using the Hadamard product (element-by-element product).

[0029] A dual-branch, multi-layered visual prior self-prompting mechanism: Inspired by prompt learning in language models, this invention introduces a semantic prompting mechanism into visual tasks. Addressing the issues of coarse semantic representation and insufficient local detail capture in existing methods, a dual-branch self-prompting mechanism is proposed. One branch extracts intermediate layer semantic features based on BiSeNet to guide infrared image reconstruction; the other branch employs a Structure-based Detail Enhancement Network (SDEN) to extract structural features to guide detail enhancement in visible light images.

[0030] Furthermore, this invention employs a GRID feature extractor and uses RP blocks as basic nodes to obtain multi-level features. This indicates that three visual cue pools are constructed for each layer. ,in Indicates the first The layer has a channel dimension. Furthermore, a Memory Attention Mechanism (MAM) is introduced to construct a Visual Prompt (VP) modulator, adaptively integrating prior knowledge from the visual cues into the multi-channel feature representation, thereby enhancing cross-modal information interaction capabilities. Specifically, this invention replaces the original Long Short-Term Memory (LSTM) network in the Feature Enhancement Block (FEB) with a Generic SC network, formally represented as: The decoder generates the reconstruction result, and its decoding process can be represented as follows: To ensure that the reconstructed image accurately represents pixel-level intensity and structural information, this invention comprehensively considers two types of loss functions: intensity loss and structural similarity (SSIM) metric. Specifically, mean squared error (MSE) is used to measure pixel intensity differences. The weighting method for different image regions is determined by adaptive weights. and Structural similarity is represented as... Its definition is .

[0031] The two losses mentioned above are balanced using a weighting parameter φ. Therefore, the overall loss function is expressed as follows: This self-supervised learning strategy iteratively optimizes the loss function, thereby improving the overall performance of the feature extraction network.

[0032] The detail enhancement module combines vanilla convolution and differential convolution within the DEConv block to simultaneously capture intensity and gradient information. Multiple DEConv blocks are cascaded to form a detail enhancement network, which is further connected to a simple convolutional neural network (CNN) to complete the overall architecture. The network training process employs the SSIM loss function based on the structure tensor. Visible light images typically contain complex structural features, and their effective extraction is one of the core challenges in computer vision. The structure tensor is constructed by outproducts of locally smoothed gradients, forming a 2×2 symmetric positive definite matrix, which can accurately describe the local geometric structure of an image, as shown in Figure 4.

[0033] To effectively train Structural Detail Enhancement Networks (SDEN), this invention designs a loss function that is highly consistent with the network's core objective (structural detail capture). This loss function consists of two parts: reconstruction loss. and structural tensor loss The two are balanced by a weighted parameter λ, resulting in an overall SDEN loss. Defined as: in, Used to measure pixel-level differences during network reconstruction; structure tensor fidelity term Then, the structural information captured by the structure tensor is used to enhance the network's ability to preserve image details, which is defined as follows: ,in The Frobenius norm is represented. The structure tensor can describe the directionality and intensity features of local edges, providing a rich structural representation for DEConv blocks and guiding the network to more effectively focus on and preserve the detailed features of the image.

[0034] The fusion network processes features in two stages. First, in the intra-modal fusion stage, a CNN-based adaptive grid network is used to fuse multi-scale features and introduce cue information at the original resolution. Then, in the inter-modal interaction stage, a memory attention mechanism (MAM) is used to selectively exchange information between the infrared and visible light modalities. Finally, the contribution weights of each modality are estimated, and the final fusion result is generated by weighted summation of the original input images.

[0035] The fusion network for interactive prompts processes features in two stages. First, in the intra-modal fusion stage, a CNN-based adaptive mesh network fuses multi-scale features and introduces prompt information at the original resolution. Then, in the inter-modal interaction stage, a memory attention mechanism (MAM) selectively exchanges information between the infrared and visible light modalities. Finally, the contribution weights of each modality are estimated, and the final fusion result is generated by weighted summation of the original input images. In the fusion network, this invention utilizes feature maps from different scales derived from a pre-trained representation learning network (PRL). As input, the decoder employs a symmetrical multi-scale grid structure and is enhanced by a Feature Enhancement Block (FEB), which integrates the MAM and Multi-scale Attention (MSA) modules to couple perceptual information and adaptively focus on each channel at different scales. A schematic diagram of this framework is shown in Figure 2. The formal expression of PIFL can be simplified to: ,in, This represents the contribution estimation result obtained through the fusion network calculation. .

[0036] The processing procedure can be further expressed as follows: , (twenty two) ,in, This represents the contribution weight mapping. Subsequently, the fusion result can be represented as:

[0037] in, , These represent infrared images and visible light images, respectively. This represents the final fusion result. The goal of PGIF is to introduce perceptual loss. To integrate complementary information: Where H and W represent the height and width of the image, respectively, and the intensity loss of the infrared image and the visible light image are respectively expressed as... Furthermore, in order to preserve the rich texture information in the source image, this invention introduces a gradient difference-based loss function, the expression of which is: Where the symbol ∇ represents the gradient operator, This represents the absolute value operation. The overall loss function of the network is a weighted combination of the above two parts, expressed as: Through the By performing backpropagation, this invention optimizes the fusion network and fine-tunes the PRL.

[0038] In summary, the SVSFusion network effectively maintains a reasonable distribution of pixel values ​​and appropriate intensity information by introducing difference-aware loss, and enhances the ability to preserve rich texture information by combining gradient loss, thereby significantly improving the overall quality of the fusion result.

[0039] To verify the effectiveness of the proposed solution, further explanation is provided below using experimental data: 1) Dataset: To comprehensively evaluate the effectiveness of the proposed infrared and visible light image fusion (IVIF) algorithm, experiments were conducted on three widely used benchmark datasets, including the MSRS dataset, M... 3 The FD dataset and the TNO dataset are used. SDEN is trained on 1083 visible light images from the MSRS dataset; for PRL, this invention uses 1083 infrared-visible light image pairs cropped from MSRS for training. To enrich the training data distribution of PIFL, this invention uses M... 3 400 cropped image pairs were generated from the original 150 pairs of infrared-visible images in the FD dataset. The test set contains a total of 551 image pairs, including 361 pairs from the MSRS dataset, 150 pairs from the M3FD dataset (with no overlap with the training set), and 40 pairs from the TNO dataset. These datasets provide strong support for a rigorous and comprehensive evaluation of the algorithm's performance.

[0040] Implementation details: SDEN and PRL both use a batch size of 16 for 1000 training epochs; PIFL uses a batch size of 8 for 500 training epochs. All models use the AdamW optimizer with parameters set to... , The initial learning rate is 1×10⁻⁴, and it is decayed to half its original value every 20 training epochs. During the fine-tuning phase of the fusion network, the learning rate is further reduced to one-tenth of the initial value. The constants in the formula... Set to 0.6, constant Set to 5, a constant. The setting is 7. All experiments were conducted on a computer equipped with an NVIDIA GeForce RTX4090 GPU, using the PyTorch platform.

[0041] This invention analyzes the results of multimodal image fusion by evaluating the fusion quality through visualization and quantitative evaluation metrics, and compares it with nine state-of-the-art (SOTA) methods, including two traditional methods: GF and EgeFusion; five deep learning-based methods: SSL-WAEIE, LEFuse, LENFusion, CNLFusion, and SHIP; three methods based on Transformer and attention mechanisms: SwinFuse, DAFtuse, and Fusion-Mamba; and one generative model method: SDDGAN.

[0042] A qualitative comparative analysis, shown in Figure 5, presents the visual comparison results of different methods. Overall, the proposed method has two significant advantages. First, it effectively preserves multimodal information in the original images. In the day and night scenes of the MSRS dataset, details and thermal information of traffic signs in visible light images and salient targets (such as pedestrians) in infrared images are well preserved (key detail areas are magnified using pseudo-color). In the TNO dataset, door frames and ground structures are clearly visible; in the M3FD dataset, the texture details of ground stones are most fully depicted. Secondly, this method exhibits strong robustness to interference. In all test datasets, it achieves effective cross-modal information interaction, suppresses noise, and accurately reconstructs salient target details and thermal information from visible light and infrared images. These advantages are mainly attributed to the effective integration and enhancement of cue information, making this method superior to the comparative methods in overall performance and yielding more advantageous fusion results.

[0043] 2) Quantitative Comparative Analysis: This invention quantitatively compared and evaluated 12 fusion algorithms, including our method, on 551 pairs of images in the MSRS, M3FD, and TNO datasets. Table I summarizes five fusion evaluation metrics (IFC, VIFP, Q...). Y NMIN, Q AB / FThe experimental results are shown in the figure, where "↑" indicates that a larger value indicates better performance. From the various metrics, this method achieved optimal or near-optimal performance on all three datasets. Higher IFC and VIFP scores indicate that the fusion result has higher fidelity to the source image; Q Y and Q AB / F The improvement in NMIN reflects the enhancement of structural information and edge details; while the improvement in NMIN indicates a stronger information consistency between the fused image and the source image. Overall, this method shows significant advantages in information fidelity, structural consistency, and detail rendering, and is more in line with the characteristics of human visual perception.

[0044] D. Multimodal Segmentation Results 1) Qualitative Comparative Analysis Based on fused images from the MSRS dataset, the segmentation performance of this invention was evaluated on two segmentation models, SAM and SegFormer. The SAM model directly segments the fused image, and the result is shown in Figure 9; while the SegFormer model is retrained using fused images generated by different fusion methods, and the segmentation result is shown in Figure 10. Thanks to the strong visual priors and self-cueing capabilities provided by the semantic segmentation module and the detail enhancement module, the method of this invention can stably distinguish pedestrians and vehicles, foreground and background in complex scenes, and accurately segment subtle boundaries in occluded areas. Specifically, this method can accurately segment pedestrian clothing details, vehicle tires, parking space lane lines, and fine edges in occluded areas. Compared with other methods, the segmentation results of this invention have clearer edges, shadows and material changes are less likely to cause missegmentation, the fused image structure is highly consistent with the semantic mask; instance boundaries are closer to the real target shape, and texture noise in the semantic region is significantly reduced.

[0045] 2) Quantitative Comparative Analysis Table 2 presents the comparison results of IoU metrics for images generated by different fusion methods on the MSRS dataset when used for segmentation tasks. This invention retrains the segmentation network using images generated by various fusion methods. The results show that, compared with other methods, the method of this invention achieves the highest IoU values ​​across multiple categories, including background, pedestrians, bicycles, curves, vehicles, parking spaces, guardrails, cones, and speed bumps, demonstrating superior boundary delineation ability and overall recognition performance. This performance improvement is mainly attributed to two aspects: firstly, the WITRAN-based feature representation framework effectively captures local texture information and global contextual relationships; secondly, prior hints from semantic segmentation and detail enhancement branches effectively guide feature representation learning, while the information interaction module reduces multimodal differences and achieves adaptive fusion, thereby further improving segmentation performance.

[0046] E. Multimodal Target Detection Results 1) Qualitative Comparative Analysis Figure 7 shows the visualization results of target detection on the fused image using the MSRS dataset. Thanks to the prior guidance provided by the detail enhancement features, in daytime outdoor scenes, the method of this invention does not exhibit target misjudgment in the fused image, and the detection accuracy of salient targets is significantly higher; in low-light nighttime road scenes, this invention can accurately detect distant pedestrian and vehicle targets with high precision.

[0047] 2) Quantitative Comparative Analysis Table 6 presents the quantitative comparison results of different comparison methods in terms of target detection accuracy and recall on the MSRS dataset. Benefiting from the effective guidance of the prompt information, the method of this invention shows outstanding performance in terms of overall performance. Specifically, the target detection effect based on multimodal fusion image is significantly better than the detection result of single-modal image; compared with other state-of-the-art (SOTA) fusion methods, the fusion image generated by this invention shows better overall performance in all detection evaluation indicators, further verifying that the proposed fusion method has good application prospects in supporting downstream perception tasks.

[0048] F. Regularization Parameter Analysis 1) Parameter Analysis The parameters are used to balance the weighted reconstruction loss and the structural tensor loss, guiding the network to extract edge and structural information more effectively from visible light images. The reconstruction evaluation index in Figure 8(a) shows that when When the value is 0.6, the network performance reaches its optimal level.

[0049] 2) Parameter Analysis It plays a crucial role in self-supervised reconstruction networks, guiding the extraction of infrared and visible light features by balancing weighted reconstruction loss and structural loss, and achieving high-quality image reconstruction. The reconstruction index in Figure 8(b) shows that the best reconstruction performance can be obtained when 5 is taken.

[0050] 3) Parameter Analysis This method is used to adjust the trade-off between brightness loss and gradient loss, effectively preserving fine-grained texture details of the source image while maintaining significant target thermal features. According to the experimental results in Figure 8(c), when 7 is selected, this method achieves optimal performance on all fusion evaluation metrics.

[0051] Table 1. Quantitative results of target detection on the MSRS dataset

[0052]

[0053]

[0054] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for fusing infrared and visible light images based on semantic prior, characterized in that, The steps include: S-1: Define the source input image and network structure parameters, set... This represents the source visible light and infrared input image. This indicates the corresponding semantic prior hints generated under the guidance of the meta-network. For parameters Meta-network, For parameters The coding network, For parameters S-2: Construct a three-level optimization objective function to optimize the meta-network, encoding network, and fusion network sequentially. The optimization process satisfies: ； ； ;in, The loss function of the detail enhancement network is used to ensure the generation of meaningful visual prior cues; Guide the encoder to learn effective feature representations; Supervise the high-quality image fusion process; S-3: Construct a grid-coded network using a residual representation module with two-dimensional scanning characteristics, and capture long-range spatial dependencies and local neighborhood interactions through block-level water wave information transmission and recursive acceleration network PW-WITRAN and horizontal-vertical gated selection unit HVGSU; S-4: Construct a dual-branch multi-level visual prior self-cue mechanism, extract high-level semantic features of infrared images through semantic segmentation branch, extract structural features of visible light images through detail enhancement branch, and construct a visual cue modulator by combining memory attention mechanism MAM; S-5: Process features in stages through a cue interaction fusion network. First, in the intramodal fusion stage, an adaptive grid network is used to fuse multi-scale features and introduce cue information. Then, in the intermodal interaction stage, cross-modal selective information exchange is achieved through MAM. Finally, the modal contribution weights are estimated and weighted summation is performed to generate the fusion result. 。 2. The infrared and visible light image fusion method based on semantic prior as described in claim 1, characterized in that, The operation process of the residual representation module described in S-3 is as follows: given the input deep features, first perform layer normalization (LN), then capture long-range spatial dependencies through the WITRAN module, and combine with a learnable scale factor. Controlling the jump connection information flow satisfies: ,in To output information.

3. The infrared and visible light image fusion method based on semantic prior as described in claim 1, characterized in that, The PW-WITRAN processing procedure described in S-3 includes: S-3-1 dividing the input image into blocks to obtain... S-3-2 is processed through the HVGSU unit. Obtain the hidden states of the horizontal and vertical outputs. and ,in and ， R represents the depth of the model, and R and C represent the number of rows and columns after image segmentation, respectively. For model feature dimensions; S-3-3 pairs and After concatenation and inverse block processing, the output is obtained. ,in 。 4. The infrared and visible light image fusion method based on semantic prior as described in claim 3, characterized in that, The operation of the HVGSU unit described in S-3 satisfies: ； ； ;in, ,1≤r≤R,1≤c≤C,1≤ ≤L when hour, This represents the input of the first layer; when When, it indicates the input to the subsequent layer. and These represent the horizontal and vertical hidden state inputs of the current cell, respectively. This indicates the output of the current cell.

5. The infrared and visible light image fusion method based on semantic prior as described in claim 4, characterized in that, The gated selection unit (GSC) includes a selection gate and an output gate, and the operation process satisfies: ； ； ； Where, and represent the primary hidden state and the secondary hidden state, respectively, and x represents the input information; Let b be the weight matrix and b be the bias vector. and These represent the selection gate and the output gate, respectively; ⊙ represents element-wise multiplication; and For activation functions; and Indicates intermediate variables. This indicates that the output is hidden.

6. The infrared and visible light image fusion method based on semantic prior as described in claim 1, characterized in that, The specific implementation of the bi-branch multi-level visual prior self-prompting mechanism described in S-4 includes: S-4-1 using a GRID feature extractor, with residual representation RP blocks as basic nodes, to obtain multi-level features. S-4-2 constructs three visual cue pools for each layer of features. ， For the first Layer channel dimension; S-4-3 replaces the Long Short-Term Memory (LSTM) network in the Feature Enhancement Module (FEB) with GSC, through To achieve the integration of prior knowledge, among which To integrate prior and subsequent features.

7. The infrared and visible light image fusion method based on semantic prior as described in claim 1, characterized in that, As described in S-2 The loss function for the detail augmentation network employs a self-supervised learning strategy, weighted by intensity loss and structural similarity loss. composition: ,in: ； ； and For adaptive weights, For weighted parameters, and The images show the reconstruction results for infrared and visible light images, respectively.

8. The infrared and visible light image fusion method based on semantic prior as described in claim 1, characterized in that, The detail enhancement branch described in S-4 employs the Structural Detail Enhancement Network (SDEN), whose loss function is a weighted average of the reconstruction loss and the structural tensor loss. composition: ； ； ； This is the result of enhanced details in a visible light image. is the weighting parameter, representing the Frobenius norm.

9. The infrared and visible light image fusion method based on semantic prior as described in claim 1, characterized in that, The loss function of the prompting interaction fusion network described in S-5 is: ;in: ； ； ； Where H and W represent the height and width of the image, respectively. Here, α is the weighting coefficient, ∇ is the weighting parameter, and the symbol ∇ represents the gradient operator. This represents absolute value operations.

10. The infrared and visible light image fusion method based on semantic prior as described in claim 1, characterized in that, The estimation process for the modal contribution weights described in S-5 satisfies: ;in ； To contribute to the estimation results, These are feature maps at different scales, visual cues, and the fusion result is obtained through calculation. ⊙ represents element-wise multiplication.

Images