Multi-modal image fusion method based on hierarchical semantic richness

Abstract

The invention relates to a multi-modal image fusion method based on hierarchical semantic richness, and belongs to the technical field of multi-modal image fusion. According to the method, global information exchange is balanced by adopting multi-scale feature aggregation and redistribution, meanwhile, fusion and segmentation tasks are dynamically bridged, a progressive semantic dense injection strategy is introduced, and global semantics are injected into highly consistent infrared features through dense connection, so that the global semantic fusion is realized. And then the semantic-infrared mixed features are transmitted to visible light features. Secondly, two types of feature fusion modules are introduced, one is to use a cross-modal attention mechanism to carry out more comprehensive feature fusion, the other is to use semantic features as third input to enhance semantic representation of image fusion, and feature fusion is realized in a complex scene by dynamically balancing global semantic consistency and fine-grained local detail representation. According to the method, layering and enriching of semantic information are achieved through strategies such as semantic collection, distribution and injection, and the fusion visual effect and downstream perception performance are enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal image fusion, and in particular relates to a multimodal image fusion method based on layered semantic enrichment. Background Art

[0002] Limited by theoretical and hardware capabilities, single-modal sensors often have difficulty capturing images of real-world scenes effectively and comprehensively. For example, traditional cameras rely primarily on capturing light reflected from the surface of an object to generate visually rich images within the visible spectrum. However, in certain applications, such as dim lighting, haze, and smoke, they may be unable to obtain relevant target information. In contrast, infrared sensors can address problems such as target loss due to environmental factors by identifying the infrared heat signature of the target object. However, the low resolution and inherent noise of the infrared modality hinder the capture of fine texture details. The complementary nature of these two imaging devices has prompted researchers to integrate visible and infrared images and use image fusion technology to generate synthetic images that can highlight key targets and preserve texture details. This technology can enhance the clarity of targets and scene details, making it suitable for a variety of advanced vision tasks, including semantic segmentation, object detection, and saliency detection.

[0003] Deep learning-based infrared and visible image fusion (IVIF) technology is known for its powerful representation capabilities and has become a mainstream research direction. These methods are generally divided into single-task and multi-task image fusion methods. The core of single-task image fusion is to enhance pixel-level quality, particularly by emphasizing thermal signals in infrared images and structural details in visible light images. Despite significant progress in image reconstruction quality and quantitative metrics, these methods may still perform poorly in subsequent tasks. Given that IVIF can integrate complementary information from the source images to comprehensively represent the imaged scene, it has the potential to improve the performance of subsequent vision tasks. Therefore, there is growing interest in multi-task image fusion algorithms that combine IVIF with higher-level vision tasks.

[0004] For example, SeAFusion introduces a cascaded segmentation model after the image fusion network and uses its semantic loss function to guide the optimization process and enhance the training of the fusion network. TarDAL introduces a two-layer optimization formula for the joint training of fusion and object detection tasks to form a general fusion network with semantic awareness. MaeFuse is different from the joint training of upstream and downstream tasks. It uses the MAE model with rich prior knowledge as the backbone and designs a specific alignment function to generalize to the field of image fusion. SAGE and SpTFuse use the powerful segmentation model SAM as a priori guidance to promote image fusion to perceive a wider range of semantic representations. Although these methods attempt to improve the performance of downstream tasks by enhancing semantic consistency, they all have difficulty in striking a balance between highlight semantics and preserving texture details, resulting in poor detail fidelity and local detail reconstruction performance of the fused image, or poor performance of downstream tasks. MaeFuse, SAGE, and SpTFuse all show degradation of semantically submerged textures, resulting in loss of facial details. Summary of the Invention

[0005] The purpose of the present invention is to address the defects of the existing technology and provide a multimodal image fusion method based on layered semantic enrichment. The infrared and visible light fusion image obtained by the present invention can retain rich scene information in the fusion features, significantly improve the semantic information in the fusion features, and optimize the performance of downstream tasks.

[0006] To achieve the above object, the technical solution of the present invention is: a multimodal image fusion method based on layered semantic enrichment, comprising:

[0007] Step 1: Design a two-branch joint optimization framework that includes image fusion and semantic segmentation tasks. Convolutional blocks are used to extract shallow features for image fusion, and ResBlocks are further coupled to extract deep features for semantic segmentation.

[0008] Step 2: Design an attention-enhanced fusion module (AEFM) to integrate deep-level infrared and visible light features to form a fused semantic representation at multiple scales.

[0009] Step 3: A semantic collection, distribution, and injection strategy is used to efficiently integrate deep multi-scale semantic representations to form a global semantic representation, which is then hierarchically injected into shallow features and features at each scale to ensure semantic consistency of the fused features and multi-scale information interaction.

[0010] Step 4. Design a semantic-guided feature fusion module (SGFM) to integrate the shallowly injected infrared and visible light features and the upsampled global semantics to generate fused features. The fused image is obtained through the fusion decoder, and the global semantic features are used to obtain the segmentation results through the semantic decoder.

[0011] Furthermore, the feature extraction process in step 1 is expressed as:

[0012]

[0013] The subscripts ir and vi represent infrared light mode and visible light mode respectively. ir and I vi Represents the input infrared image and visible light image, and Represent the infrared and visible light features of the i-th scale respectively, i = 2, 3, 4, 5, Conv represents the convolution operation, and ResB represents the residual block.

[0014] Furthermore, the attention-enhanced fusion module AEFM in step 2 will Perform cross-context fine-grained fusion:

[0015]

[0016] in Represents the fusion feature at the i-th scale.

[0017] Furthermore, the attention-enhanced fusion module AEFM includes the following processes:

[0018] 1) Cross-attention computation, triggering attention to the global context between modalities, forming two attention maps;

[0019] 2) Attention interaction: using channel cascade and convolution interaction to obtain the local inductive bias between attentions, and convert it into attention weight using the Softmax function;

[0020] 3) Cross-value dot product operation and cascade convolution fusion are used to restore the initial dimension and generate fusion features;

[0021] The entire process of the attention-enhanced fusion module AEFM is expressed as follows:

[0022]

[0023]

[0024] Reshape represents the dimension conversion function. They represent the query, key and value at the i-th scale respectively, Conv represents the convolution operation, x represents the modality, x∈{ir,vi}, They represent the infrared light attention matrix and the visible light attention matrix at the i-th scale, respectively. Cat represents the cascade operation. Represents the cascaded attention matrix, S represents the Split dimension separation function, They represent the infrared light attention score matrix and the visible light attention score matrix at the i-th scale, respectively. It is worth noting that, considering that the second-scale features contain a lot of feature redundancy, we only use the i=3, 4, 5, and 6 scale features for subsequent processing.

[0025] Furthermore, the semantic collection in step 3 uses a feature aggregation module (FGM) to integrate multi-scale features to form a global semantic representation. Specifically, the four scale features are up-sampled and down-sampled as needed to ensure spatial consistency, and integrated features are obtained using cascade and convolution operations. Subsequently, fine-grained features are further extracted through dual-branch spatial and channel attention operations, and finally integrated to obtain a global semantic representation. The FGM process is expressed as follows:

[0026]

[0027] Among them↑ n Indicates upsampling the input by n times, ↓ n Indicates that the input is downsampled n times, F se represents global semantic representation, SA represents spatial attention, and CA represents channel attention. and Represent the semantic representation after SA and CA processing, Expressed as fine-grained global semantics.

[0028] Furthermore, the semantic distribution in step 3 distributes the global semantic representation hierarchically to features of different scales to promote multi-scale information exchange and build a bridge for the joint optimization of semantic segmentation and image fusion; specifically, Figure 3 As shown in the figure, the dense connection technology is used to gradually integrate the global semantic representation into the infrared features of the image fusion branch to generate semantically related hybrid features, which are further integrated into the visible light features. In addition, the global semantic representation is distributed to the features of each scale to promote multi-scale feature interaction:

[0029]

[0030] Among them, Sigmoid represents the activation function, and Max represents the maximum pooling operation.

[0031] Furthermore, the semantically guided feature fusion module SGFM in step 4 guides the shallow modal features through semantics to generate fused features with high semantic consistency:

[0032]

[0033] Furthermore, the semantic-guided feature fusion module SGFM includes the following processes:

[0034] Semantic cross-attention calculation and intra-modal self-attention calculation use semantic features, infrared features, and visible light features as queries, and perform cross-attention calculations with infrared and visible light features, as well as modality-specific attention calculations, to enhance attention to semantics and re-examination of the context.

[0035] Attention integration: add the intermediate semantic attention maps element by element, perform dot product operations with the modal values, and finally integrate them through convolution operations;

[0036] The overall process of the semantic-guided feature fusion module SGFM is expressed as follows:

[0037] A ir =Softmax(Q ir K ir T ),A vi =Softmax(Q vi K vi T )

[0038]

[0039] A se =Softmax(Q se K ir T +Q se K vi T )

[0040]

[0041] in and They represent the features obtained by self-attention operation of infrared modality and the features obtained by self-attention operation of visible light modality respectively. se Indicates that the semantics are used as queries and the infrared key K ir And visible light key K vi The attention matrix obtained by performing the dot product operation. and Represent the semantically enhanced infrared features and visible light features respectively.

[0042] Furthermore, the fusion decoder uses dense connection technology to reconstruct the fused image, and the semantic decoder uses stacked convolution blocks to predict the segmented image.

[0043] The loss function used by the method in the image fusion task is:

[0044]

[0045] in Indicates strength loss, Indicates texture loss.

[0046] Strength loss is defined as follows:

[0047]

[0048] where H and W are the height and width of the image, respectively, n (n=1,2) means l n -norm, Indicates I in YCbCr space f and I vi Y channel, (i, j) represents the pixel coordinates in the image, the target mask M in the above formula tar Used to selectively preserve salient objects in infrared modality.

[0049] The texture loss is defined as follows:

[0050]

[0051] The loss function used by the method in the semantic segmentation task is:

[0052]

[0053] Here, OhemCELoss represents the cross entropy loss for online difficult sample mining. seg represents the predicted segmentation result, and I label Represents the supervisory signal for segmentation.

[0054] The present invention also provides a computer-readable storage medium on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the steps of any of the above methods can be implemented.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1) A semantic collection, distribution, and injection mechanism is proposed, which aims to achieve globally balanced information exchange by seamlessly integrating multi-scale feature aggregation and hierarchical redistribution, while establishing a dynamic optimization bridge between image fusion and semantic prediction tasks.

[0057] 2) To solve the problem of semantic features dominating and causing degradation of texture clarity in local areas, the present invention introduces a progressive semantic dense injection strategy, which gradually injects the global semantic representation into the infrared features to generate hybrid features, and then incrementally injects these hybrid features into the visible light features to reduce the loss of high-frequency information.

[0058] 3) Two types of feature fusion modules are introduced, one that utilizes the cross-modal attention mechanism for more comprehensive feature fusion, and the other that utilizes semantic features as the third input to further enhance the semantic representation of image fusion.

[0059] 4) Qualitative and quantitative experimental results on image fusion tasks demonstrate that the proposed method achieves leading performance and effectively boosts the performance of downstream applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 To illustrate examples of the present invention in a visual sense;

[0061] Figure 2 This is a theoretical analysis of the present invention in extreme scenarios;

[0062] Figure 3 It is the overall framework of the present invention;

[0063] Figure 4 The feature fusion module architecture for attention enhancement of the present invention;

[0064] Figure 5 The semantically guided feature fusion module architecture of the present invention;

[0065] Figure 6 This is the qualitative result of the present invention on the MSRS dataset;

[0066] Figure 7 This is the qualitative result of the present invention on the FMB dataset;

[0067] Figure 8 This is the qualitative result of the present invention on the M3FD dataset;

[0068] Figure 9 This is the qualitative result of the present invention on the TNO dataset;

[0069] Figure 10 This is the quantitative result of the present invention on the MSRS dataset;

[0070] Figure 11 This is the quantitative result of the present invention on the FMB dataset;

[0071] Figure 12 This is the quantitative result of the present invention on the M3FD dataset;

[0072] Figure 13 This is the quantitative result of the present invention on the TNO dataset;

[0073] Figure 14 This is the qualitative result of the fusion result of the present invention on the MSRS dataset on the segmentation model;

[0074] Figure 15 This is the quantitative result of the fusion result of the present invention on the MSRS dataset on the segmentation model. DETAILED DESCRIPTION

[0075] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0076] Existing image fusion methods attempt to improve the performance of downstream tasks by enhancing semantic consistency, but they all find it difficult to strike a balance between high semantics and preserving texture details, resulting in poor detail fidelity and local detail reconstruction performance of the fused image, or poor performance in downstream tasks. Figure 1 As shown in Figure 3, MaeFuse, SAGE, and SpTFuse all suffer from the degradation of semantic submerged texture, resulting in the loss of facial details. In addition, as can be seen in the green box, all three methods present blurred distant buildings.

[0077] In multimodal image fusion tasks, the introduction of semantic information often leads to the dominance of semantic features, resulting in a significant decrease in texture clarity in local areas. This conflict between semantics and texture seriously affects the balance between global consistency and local accuracy of the fusion output. Figure 2 Four typical scenes from the MSRS and TNO datasets are presented, including glare, daylight, low-light, and smoke scenes. Visible light images present expressive scenes through larger entropy (EN) values, which are mainly contributed by background textures. Infrared images present different characteristics under different conditions. For example, in glare and low-light scenes, infrared images clearly present the edges and structural features of pedestrian targets through higher spatial frequency (SF) values, and effectively reduce irrelevant interference caused by background sparsity. In daylight and smoke scenes, although the index values ​​of infrared images are lower, they can still maintain close visual information fidelity (VIF), indicating that the transmission of saliency information is more stable. The above analysis shows that infrared images have high semantic consistency, which is particularly evident in extreme scenes.

[0078] Based on the above analysis, we propose a multimodal image fusion method based on hierarchical semantic enrichment, HSENet, which generates high-quality fused images with robust global semantic consistency and excellent local detail representation. The core innovation of HSENet lies in achieving hierarchical enrichment of semantic information through semantic collection, distribution, and injection. Specifically, the network first balances global information exchange through multi-scale feature aggregation and redistribution, while dynamically bridging the fusion and segmentation tasks. Subsequently, a progressive semantic dense injection strategy is introduced, which utilizes dense connections to first inject global semantics into highly consistent infrared features, and then transfers the semantic-infrared hybrid features into visible light features. This approach effectively enhances semantic representation while minimizing the loss of high-frequency information. Furthermore, HSENet incorporates two types of feature fusion modules: one employs a cross-modal attention mechanism for more comprehensive feature fusion, and the other uses semantic features as a third input to further enhance the semantic representation of image fusion. By dynamically balancing global semantic consistency with fine-grained local detail representation, these modules achieve robust and flexible feature fusion in complex scenes. Our method demonstrates excellent performance in visual perception tasks while fully preserving the texture characteristics of the source modalities. Comparative experiments on image fusion and semantic segmentation demonstrate the advantages of HSENet in visual quality and semantic preservation.

[0079] To validate the fusion performance of our method, we selected three widely recognized benchmark datasets: MSRS, FMB, M3FD, and TNO. The experiment involved preprocessing the training dataset by randomly partitioning it into 256×256 patches. Our HSENet network used the Adam optimizer with 2500 iterations, a learning rate of 1e-3, and a polynomial learning rate descent strategy.

[0080] Step 1: Design a two-branch joint optimization framework that includes image fusion and semantic segmentation tasks. Convolutional blocks are used to extract shallow features for image fusion, and ResBlocks are further coupled to extract deep features for semantic segmentation.

[0081] Step 2: Design an attention-enhanced fusion module (AEFM) to integrate deep-level infrared and visible light features to form a fusion semantic representation at multiple scales. The AEFM module architecture is shown in Figure 4 among;

[0082] Step 3: A semantic collection, distribution, and injection strategy is used to efficiently integrate the deep multi-scale semantic representations to form a global semantic representation, which is then hierarchically injected into the shallow features and features at each scale to ensure the semantic consistency of the fused features and the interaction of multi-scale information.

[0083] Step 4: A semantically guided feature fusion module (SGFM) is designed to integrate the shallowly injected infrared and visible light features and the upsampled global semantics to generate fused features. The SGFM module architecture is shown in Figure 5 Then the fused features are passed through the fusion decoder to obtain the fused image, while the global semantic features are passed through the semantic decoder to obtain the segmentation result;

[0084] These steps are run in Figure 3 Visible in the overall frame.

[0085] In this example, the feature extraction process in step 1 is expressed as:

[0086]

[0087] Where ir and vi represent infrared light mode and visible light mode respectively, I ir and I vi Represents the input infrared image and visible light image, and Represents the infrared and visible light features of the i-th scale, i = 2, 3, 4, 5, Conv represents the convolution operation, and ResB represents the residual block.

[0088] In this example, the attention-enhanced fusion module AEFM in step 2 will Perform cross-context fine-grained fusion:

[0089]

[0090] in Represents the fusion feature at the i-th scale.

[0091] In this example, the attention-enhanced fusion module AEFM includes the following processes:

[0092] 1) Cross-attention computation, triggering attention to the global context between modalities, forming two attention maps;

[0093] 2) Attention interaction: using channel cascade and convolution interaction to obtain the local inductive bias between attentions, and convert it into attention weight using the Softmax function;

[0094] 3) Cross-value dot product operation and cascade convolution fusion are used to restore the initial dimension and generate fusion features;

[0095] The entire process of the attention-enhanced fusion module AEFM is expressed as follows:

[0096]

[0097] Reshape represents the dimension conversion function. They represent the query, key and value at the i-th scale respectively, Conv represents the convolution operation, x represents the modality, x∈{ir,vi}, They represent the infrared light attention matrix and the visible light attention matrix at the i-th scale, respectively. Cat represents the cascade operation. Represents the cascaded attention matrix, S represents the Split dimension separation function, They represent the infrared light attention score matrix and the visible light attention score matrix at the i-th scale, respectively. It is worth noting that, considering that the second-scale features contain a lot of feature redundancy, we only use the i=3, 4, 5, and 6 scale features for subsequent processing.

[0098] In this example, the semantic collection in step 3 uses a feature aggregation module (FGM) to integrate multi-scale features to form a global semantic representation. Specifically, the four scale features are up-sampled and down-sampled as needed to ensure spatial consistency, and integrated features are obtained using cascade and convolution operations. Subsequently, fine-grained features are further extracted through dual-branch spatial and channel attention operations, and finally integrated to obtain a global semantic representation. The FGM process is expressed as follows:

[0099]

[0100] Among them↑ n Indicates upsampling the input by n times, ↓ n Indicates that the input is downsampled n times, F se represents global semantic representation, SA represents spatial attention, and CA represents channel attention. and Represent the semantic representation after SA and CA processing, Expressed as fine-grained global semantics.

[0101] In this example, the semantic distribution in step 3 distributes the global semantic representation hierarchically to features of different scales to promote multi-scale information exchange and build a bridge for the joint optimization of semantic segmentation and image fusion. Specifically, Figure 3 As shown in the figure, the dense connection technology is used to gradually integrate the global semantic representation into the infrared features of the image fusion branch to generate semantically related hybrid features, which are further integrated into the visible light features. In addition, the global semantic representation is distributed to the features of each scale to promote multi-scale feature interaction:

[0102]

[0103] Among them, Sigmoid represents the activation function, and Max represents the maximum pooling operation.

[0104] In this example, the semantically guided feature fusion module SGFM in step 4 guides the shallow modal features through semantics to produce fused features with high semantic consistency:

[0105]

[0106] In this example, the semantic-guided feature fusion module (SGFM) includes the following steps:

[0107] Semantic cross-attention calculation and intra-modal self-attention calculation use semantic features, infrared features, and visible light features as queries, and perform cross-attention calculations with infrared and visible light features, as well as modality-specific attention calculations, to enhance attention to semantics and re-examination of the context.

[0108] Attention integration: add the intermediate semantic attention maps element by element, perform dot product operations with the modal values, and finally integrate them through convolution operations;

[0109] The overall process of the semantic-guided feature fusion module SGFM is expressed as follows:

[0110] A ir =Softmax(Q ir K ir T ),A vi =Softmax(Q vi K vi T )

[0111]

[0112] A se =Softmax(Q se K ir T +Q se K vi T )

[0113]

[0114] in and They represent the features obtained by self-attention operation of infrared modality and the features obtained by self-attention operation of visible light modality respectively. se Indicates that the semantics are used as queries and the infrared key K ir And visible light key K vi The attention matrix obtained by performing the dot product operation. and Represent the semantically enhanced infrared features and visible light features respectively.

[0115] In this example, the fusion decoder uses dense connection technology to effectively reconstruct the fused image, and the semantic decoder uses stacked convolution blocks to predict the segmented image.

[0116] In this example, the loss function used by the method in the image fusion task is:

[0117]

[0118] in Indicates strength loss, Indicates texture loss.

[0119] Strength loss is defined as follows:

[0120]

[0121] where H and W are the height and width of the image, respectively, n (n=1,2) means l n -norm, Indicates I in YCbCr space f and I vi Y channel, (i, j) represents the pixel coordinates in the image, the target mask M in the above formula tar Used to selectively preserve salient objects in infrared modality.

[0122] The texture loss is defined as follows:

[0123]

[0124] In this example, the loss function used by the method in the semantic segmentation task is:

[0125]

[0126] Here, OhemCELoss represents the cross entropy loss for online difficult sample mining. seg Represents the predicted segmentation result, I label Represents the supervisory signal for segmentation.

[0127] During the specific implementation, the design scheme of multimodal image fusion based on hierarchical semantic enrichment is realized by software. In order to objectively measure the fusion performance of the method proposed in the present invention, the performance of each method is evaluated from both qualitative and quantitative aspects. Qualitative evaluation is a subjective evaluation method that depends on human visual perception. A good fusion result should contain both significant contrast of infrared images and rich texture of visible light images. Quantitative evaluation objectively evaluates the fusion performance through some statistical indicators. This paper selects mutual information (MI), visual information fidelity (VIF), quality assessment based on fuzzy and noise factors (Qabf), no-reference assessment based on fuzzy and noise factors (Nabf), structural similarity index measure (SSIM) and multi-scale structural similarity measure (MS-SSIM).

[0128] Figure 6 、 Figure 8 、 Figure 10 and Figure 12 The visual effects on three typical test sets MSRS, FMB, M3FD and TNO are shown respectively. Figure 6 The enlarged red box shows that the results of DDFM, MuFusion, SAGE, GIFNet, MaeFuse and SpTFuse fail to clearly reflect the distant buildings. In addition, the present invention amplifies the semantic information of pedestrians in the green box and finds that TarDAL has unnatural artifacts, while DDFM, SAGE and GIFNet cannot effectively highlight the details of pedestrians. SDNet and PMGI respond unnaturally to the color of street lights. The results of the present invention not only show excellent visual fidelity, but also obtain more expressive and semantically important fusion performance. Figure 8 As shown in the night scene in , DDFM suffers from severe ghost interference, while PMGI, TarDAL, MUFusion, SAGE, GIFNet, MaeFuse, and SpTFuse fail to clearly reflect the license plate letters in the green magnified area. Figure 10 Showing a daytime tunnel scene. The results of TarDAL, DDFM, MUFusion, and GIFNet lose key pedestrian information, while SAGE reflects the blurriness of the truck texture. Figure 12 The results show an outdoor scene during the day. TarDAL loses a lot of details, such as tree trunk textures and car outlines. PMGI, DDFM, MUFusion, SAGE, and GIFNet have unrealistic artifacts on the edges of leaves. SDNet and MaeFuse show blurred vehicle details in the green magnified box. Only the algorithm of the present invention and SpTFuse can restore the real scene consistent with the human visual system. Overall, the method of the present invention is significantly superior in terms of image visual quality and is more in line with the requirements of the human visual system.

[0129] Figure 7 、 Figure 9 、 Figure 11 and Figure 13 A quantitative comparison of the fusion results is shown. Figure 7 As can be seen, our results achieve the best performance on all metrics on the MSRS dataset, including MI, VIF, Qabf, Nabf, SSIM, and MS SSIM. This demonstrates that our method excels in preserving fine-grained texture details and image fidelity. This advantage is attributed to our progressive semantic dense injection, which constrains the image fusion branch to preserve the full texture details of the source modalities. Figure 9 A quantitative comparison is presented on the FMB dataset. Our method ranks first across four metrics: MI, VIF, Qabf, and Nabf, demonstrating its superiority in preserving complementary information and achieving high visual quality. Second and third place performance is achieved on SSIM and MS SSIM, respectively, likely due to differences between high-level semantic and visual features that influence structural integrity. Figure 11 Quantitative comparison results on the M3FD dataset are presented. Our method achieves the best performance on the MI, VIF, and Qabf metrics, the second best performance on the Nabf metric, and ranks third on the SSIM and MS SSIM metrics. Figure 13 Quantitative comparison results on the TNO dataset are presented. Our method achieves the best performance on the VIF, Qabf, and Nabf metrics, second-best performance on the MS SSIM metric, and ranks third on the MI and SSIM metrics. Overall, our method ranks first across the entire dataset in terms of VIF and Qabf, further demonstrating its superiority in image quality and visual fidelity.

[0130] Figure 14 The segmentation accuracy of the fused images of different algorithms in two different scenarios is demonstrated. In scenario 1, InternImage has difficulty correctly identifying the left "curve" that appears in the results of PMGI, SDNet, TarDAL, DDFM, MUFusion, SAGE, and GIFNet. The results of PMGI, MUFusion, PSFusion, and GIFNet cannot effectively predict the "person" in the middle. Except for the visible image and the results of the present invention, the results of other methods cannot identify the "curve" on the right. In scenario 2, the semantic content of the "curve" on the left of DDFM and MaeFuse is lost, while MUFusion, SAGE, GIFNet, MaeFuse, and SuperFusionSpTFuse only detect some pixels of the "car" in the middle. The results of SDNet, TarDAL, and SAGE lose the content of the "curve" on the right. Figure 15The mIoU scores of the fused image segmentation results on the MSRS test set are listed. Our method ranks in the top two for IoU across eight categories and tops the charts for both mIoU and aACC. Compared to SDNet, our method achieves a 5.39% improvement in mIoU accuracy.

[0131] The present invention also provides a computer-readable storage medium on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the steps of any of the above methods can be implemented.

[0132] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions and effects do not exceed the scope of the technical solution of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A multimodal image fusion method based on layered semantic enrichment, characterized in that: include: Step 1: Design a two-branch joint optimization framework that includes image fusion and semantic segmentation tasks. Convolutional blocks are used to extract shallow features for image fusion, and ResBlocks are further coupled to extract deep features for semantic segmentation. Step 2: Design an attention-enhanced fusion module (AEFM) to integrate deep-level infrared and visible light features to form a fused semantic representation at multiple scales. Step 3: A semantic collection, distribution, and injection strategy is used to efficiently integrate deep multi-scale semantic representations to form a global semantic representation, which is then hierarchically injected into shallow features and features at each scale to ensure semantic consistency of the fused features and multi-scale information interaction. Step 4. Design a semantic-guided feature fusion module to integrate the shallowly injected infrared and visible light features and the upsampled global semantics to generate fused features, and obtain the fused image through the fusion decoder. The global semantic features are used to obtain the segmentation results through the semantic decoder.

2. The multimodal image fusion method based on layered semantic enrichment according to claim 1, characterized in that: The feature extraction process in step 1 is expressed as: The subscripts ir and vi represent infrared light mode and visible light mode respectively. ir and I vi Represents the input infrared image and visible light image, and Represent the infrared and visible light features of the i-th scale respectively, i = 2, 3, 4, 5, 6, Conv represents the convolution operation, and ResB represents the residual block.

3. The multimodal image fusion method based on layered semantic enrichment according to claim 1, characterized in that: The attention-enhanced fusion module AEFM in step 2 will Perform cross-context fine-grained fusion: in Represents the fusion feature at the i-th scale.

4. The multimodal image fusion method based on layered semantic enrichment according to claim 3, characterized in that: The attention-enhanced fusion module AEFM includes the following processes: 1) Cross-attention computation, triggering attention to the global context between modalities, forming two attention maps; 2) Attention interaction: using channel cascade and convolution interaction to obtain the local inductive bias between attentions, and convert it into attention weight using the Softmax function; 3) Cross-value dot product operation and cascade convolution fusion are used to restore the initial dimension and generate fusion features; The entire process of the attention-enhanced fusion module AEFM is expressed as follows: Reshape represents the dimension conversion function. They represent the query, key and value at the i-th scale respectively, Conv represents the convolution operation, x represents the modality, x∈{ir,vi}, They represent the infrared light attention matrix and the visible light attention matrix at the i-th scale, respectively. Cat represents the cascade operation. Represents the cascaded attention matrix, S represents the Split dimension separation function, They represent the infrared light attention score matrix and the visible light attention score matrix at the i-th scale respectively.

5. The multimodal image fusion method based on layered semantic enrichment according to claim 1, characterized in that: In step 3, semantic collection uses a feature aggregation module (FGM) to integrate multi-scale features to form a global semantic representation. Specifically, the four scale features are up-sampled and down-sampled as needed to ensure spatial consistency, and integrated features are obtained using cascade and convolution operations. Subsequently, fine-grained features are further extracted through dual-branch spatial and channel attention operations, and finally integrated to obtain a global semantic representation. The FGM process is expressed as follows: Among them↑ n Indicates upsampling the input by n times, ↓ n Indicates that the input is downsampled n times, F se represents global semantic representation, SA represents spatial attention, CA represents channel attention, and Represent the semantic representation after SA and CA processing, Represented as fine-grained global semantics, Represents the fusion features at the i-th scale, i = 3, 4, 5, 6.

6. The multimodal image fusion method based on layered semantic enrichment according to claim 5, characterized in that: In step 3, semantic distribution distributes the global semantic representation hierarchically across features of different scales to promote multi-scale information exchange and bridge the gap between semantic segmentation and image fusion. Specifically, using dense connection technology, the global semantic representation is gradually integrated into the infrared features of the image fusion branch to generate semantically relevant hybrid features, which are then integrated into the visible light features. In addition, the global semantic representation is distributed across features of different scales to promote multi-scale feature interaction. Among them, Sigmoid represents the activation function, and Max represents the maximum pooling operation.

7. The multimodal image fusion method based on layered semantic enrichment according to claim 4, characterized in that: The semantically guided feature fusion module SGFM in step 4 guides the shallow modal features through semantics to produce fused features with high semantic consistency:

8. The multimodal image fusion method based on layered semantic enrichment according to claim 7, characterized in that: The semantic-guided feature fusion module SGFM includes the following processes: 1) Semantic cross-attention calculation and intra-modality self-attention calculation: semantic features, infrared features, and visible light features are used as queries, and cross-attention calculations are performed with infrared and visible light features, as well as attention calculation within the modality itself, to enhance attention to semantics and re-examination of the context. 2) Attention integration: add the intermediate semantic attention maps element by element, perform dot product operations with the modal values, and finally integrate them through convolution operations; The overall process of the semantic-guided feature fusion module SGFM is expressed as follows: A ir =Softmax(Q ir K ir T ),A vi =Softmax(Q vi K vi T ) A se =Softmax(Q se K ir T +Q se K vi T ) in and They represent the features of infrared modality obtained by self-attention operation and the features of visible light modality obtained by self-attention operation, respectively. se Indicates that the semantics are used as queries and the infrared key K ir And visible light key K vi The attention matrix obtained by performing dot product operation, and Represent the semantically enhanced infrared features and visible light features respectively.

9. The multimodal image fusion method based on layered semantic enrichment according to claim 1, characterized in that: The fusion decoder uses dense connection technology to reconstruct the fusion image, and the semantic decoder uses stacked convolution blocks to predict the segmented image.

10. A computer-readable storage medium storing computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, the steps of the method according to any one of claims 1 to 9 can be implemented.

Images