A low-altitude multi-modal image fusion method based on semantic prompt guidance

Abstract

This invention discloses a low-altitude multimodal image fusion method based on semantic cue guidance. This invention utilizes cue fine-tuning technology to effectively apply semantic awareness (SAM) to multimodal image fusion. This method integrates large-scale semantic knowledge into the image fusion process, thereby improving the overall performance of the fusion model. By setting multimodal cues and fine-tuning, this invention enables better perception and segmentation of different modalities, especially infrared information. This fusion forms a unified architecture for multimodal semantic segmentation and fusion, in which the segmentation and fusion processes mutually promote each other. This invention significantly improves the accuracy and robustness of multimodal image fusion, achieving superior performance compared to current state-of-the-art methods.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and particularly relates to a method and apparatus for low-altitude multimodal image fusion based on semantic prompting guidance. Background Technology

[0002] Infrared and Visible Image Fusion (IVIF) is a cutting-edge technology that leverages the advantages of multimodal imaging to generate high-information-density fused images. Its core objective is to integrate the complementary features of infrared and visible light images. Infrared imaging, based on the thermal radiation characteristics of a target, can penetrate smoke, nighttime environments, and camouflage to accurately locate concealed targets, but is limited by its low sensitivity to surface information such as texture and color. Visible light imaging, relying on reflected light information from the scene, can provide rich details and high spatial resolution consistent with human visual perception, but is susceptible to environmental interference such as insufficient lighting and extreme weather. IVIF technology, through the design of adaptive fusion strategies, spatially aligns and fuses the thermal radiation features of infrared images with the texture details of visible light images, ultimately outputting an enhanced image that combines target saliency with scene interpretability. This provides robust visual input for intelligent perception systems in complex scenarios such as security monitoring, autonomous driving, and military reconnaissance.

[0003] Early fusion techniques primarily relied on traditional image processing methods and mathematical models. The core idea was to extract features through multi-scale decomposition and then fuse them. However, due to the reliance on manually designed rules, they were poorly adaptable to complex scenes, and the fusion results were prone to artifacts or information loss. With the introduction of optimization theory, researchers began to attempt to improve fusion quality through mathematical models. Fusion rules became more complex, but computational costs were high, and it remained difficult to address modal differences, such as the conflict between thermal radiation and texture. The explosive development of deep learning completely transformed IVIF technology, especially the application of Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs). In 2017, the first end-to-end IVIF deep learning framework, CNN-Fusion, was proposed. In 2018, DenseFuse introduced dense connections to enhance feature reuse. In 2019, FusionGAN, proposed by Ma Jiayi et al., generated more natural images and preserved visible light textures. In 2020, RFN-Nest significantly improved the robustness of fusion in complex scenes using an attention mechanism. In 2021, PMGI jointly optimized fusion, object detection, and segmentation tasks, achieving task-driven adaptive fusion for the first time. In recent years, the combination of Transformer and cross-modal alignment has significantly improved the fusion quality of two-light fusion technology. SwinFusion, proposed in 2022, combines a Swin Transformer encoder and cross-modal cross-attention to achieve long-range dependency modeling and modal alignment. CDDFuse, proposed by Zhao Zixiang et al. in 2023, uses contrastive learning to guide cross-modal feature decoupling and fusion, significantly alleviating feature conflicts between infrared and visible light. However, it has a large number of parameters, making training and debugging difficult, and its controllability and interactivity still need improvement. In two-light fusion (IVIF) technology, controllability—the user's ability to intervene in the fusion process and results—is a key factor affecting the algorithm's practicality. Summary of the Invention

[0004] Purpose of the invention: In order to solve the problems existing in the prior art, the present invention provides a method and apparatus for low-altitude multimodal image fusion based on semantic prompting guidance.

[0005] Technical solution: This invention discloses a low-altitude multimodal image fusion method based on semantic prompting guidance, specifically as follows:

[0006] Step 1: Obtain the visible light image I vis and mask image Prompt m The input is fed into a visible light encoder to obtain the encoded feature F. vis Infrared image I ir and mask image Prompt m The input is fed into an infrared optical encoder to obtain the encoded feature F. ir ;

[0007] Step 2: Place F irand F vis The connection yields the initial fusion feature F ref ;

[0008] Step 3: Place F ref The reference image I is obtained by inputting it into the image decoder. ref ;

[0009] Step 4: Prompt the mask image m As a hint, the encoded features serve as supporting features, F ref As a query feature; use Prompt m F ref and F vis The input is fed into the visible light reference cue encoder to obtain the cue embedding feature P. vis ;Prompt m F ref and F ir The input is fed into the infrared light reference cue encoder to obtain the cue embedding feature P. ir ;

[0010] Step 5: Place I ref P ir and P vis Input to the image segmentation module, for P ir Segmentation is performed to obtain mask features M ir , for P vis Segmentation is performed to obtain mask features M vis ; and M ir and M vis The segmented images are then stitched together.

[0011] Step Six: Place M vis M ir P ir P vis F ir and F vis The input is fed into the prompt feature extraction module to obtain the prompt features corresponding to the infrared light image. and cue features corresponding to visible light images

[0012] Step 7: Add prompt features Hint Features and F ref Input the image into the interactive fusion module to obtain the fused image.

[0013] Furthermore, the visible light reference cue encoder and the infrared light reference cue encoder have the same structure. First, the supporting features are multiplied with the cue, and the result of the multiplication is input into the average pooling layer to obtain the feature F. t ; F tAfter being connected to the supporting features, the connection result is input into the convolutional layer to generate the resulting feature F. supp ; F t After being concatenated with the query features, the result feature F is generated. qry ; Combine a set of learnable query vectors Q with result features F supp The input is fed into the first cross-attention layer, and the output of the first cross-attention layer is fed into the first self-attention layer to obtain a controllable query vector Q for a specific category. ' Q ' and the result feature F qry The input is fed into the second cross-attention layer, and the output of the second cross-attention layer is fed into the second self-attention layer to generate a set of reference cues P. ' Finally, P ' The input is fed into the frozen SAM cue encoder to obtain the final cue embedding features.

[0014] Furthermore, the image segmentation module includes a frozen infrared SAM mask decoder, a frozen visible SAM mask decoder, and a frozen SAM image encoder; I ref Input to the frozen SAM image encoder; embed the output and cue of the frozen SAM image encoder into feature P. ir The predicted mask features M are obtained as input to the frozen infrared SAM mask decoder. ir The frozen SAM image encoder output and cueing are embedded into feature P. vis As input to the frozen visible light SAM mask decoder, the predicted mask features M are obtained. vis .

[0015] Furthermore, the cue feature extraction module includes an infrared light cue-semantic fusion module and a visible light cue-semantic fusion module; M ir P ir and F ir The input is fed into the infrared light cue-semantic fusion module to obtain the infrared light cue features. M vis P vis and F vis The input is fed into the visible light cue-semantic fusion module to obtain visible light cue features.

[0016] Furthermore, the infrared cues-semantic fusion module and the visible light cues-semantic fusion module have the same structure, downsampling the encoded features and flattening the downsampled features into a sequence format F. seq , will F seqThe embedded features of the cues are input into the cross-attention mechanism for processing. The features output by the cross-attention mechanism are subjected to a view transformation operation, and then the features after the view transformation operation are upsampled to obtain enhanced spatial features. The enhanced spatial features are then multiplied element-wise with the predicted mask features to obtain infrared light cues features or visible light cues features.

[0017] Furthermore, the interactive fusion module includes an image decoder, which will convert F... ref , as well as After element-wise addition, the results are input into the image decoder to obtain the final fused feature map.

[0018] Furthermore, both the visible light encoder and the infrared encoder employ GRDB modules.

[0019] Furthermore, the frozen SAM image encoder comprises 32 stacked transformer modules and a neck module connected to the last transformer module.

[0020] A computer device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the aforementioned low-altitude multimodal image fusion method based on semantic prompting guidance.

[0021] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned low-altitude multimodal image fusion method based on semantic prompting guidance.

[0022] Beneficial effects:

[0023] 1. This invention can achieve controllable fusion between different modal features, thereby improving the ability of the fused image to handle downstream tasks, and realizes personalized segmentation of the region of interest.

[0024] 2. This invention utilizes the powerful semantic perception capability of the SAM model itself to enrich the fusion features, thereby improving the fusion effect and enabling the fused image to perform better in downstream tasks. Furthermore, it uses fused image segmentation and fine-tunes the fusion module based on the segmentation effect.

[0025] 3. This invention effectively utilizes the rich information from multimodal data and the semantic knowledge of SAM, which helps to obtain more reliable results under different requirements. The proposed method is validated on the FMB and MSRS datasets. Extensive experiments demonstrate that the performance of this invention is superior to competing methods. Attached Figure Description

[0026] Figure 1This is an overall structural diagram of the present invention;

[0027] Figure 2 This is a flowchart of the present invention;

[0028] Figure 3 A schematic diagram of the prompt encoder of the present invention.

[0029] Figure 4 This is a schematic diagram of the prompt-semantic fusion module structure of the present invention. Detailed Implementation

[0030] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0031] This invention proposes a multimodal cue-guided segmentation and fusion framework model based on SAM (Semantic Awareness and Motion). Firstly, this invention designs a cue-guided segmentation and fusion framework, enabling targeted dual-light fusion and personalized segmentation in multimodal tasks based on different user requirements and specific scenarios. Secondly, this invention designs a fusion and segmentation facilitation framework based on SAM's semantic information perception capabilities. On the one hand, by leveraging SAM's powerful semantic perception capabilities, it extracts rich semantic feature information from dual-light images to improve the quality of dual-light fused images. On the other hand, because dual-light fused images contain rich texture and thermal information, further fine-tuning SAM enhances cue-guided segmentation capabilities. See the description below for details:

[0032] A multimodal prompting-guided segmentation and fusion framework based on SAM, the method comprising:

[0033] We design a cue-guided controllable multimodal method, using fine-tuning techniques and a mask as the cue fine-tuning model, so that the dual-light fusion results pay more attention to the user's region of interest and achieve personalized segmentation.

[0034] We design a semantic awareness feature extraction framework based on SAM, which uses SAM to extract semantic information to enrich the dual-light fusion features, and fine-tunes SAM by segmentation to make the model pay more attention to the cue region and improve the quality of dual-light fusion.

[0035] A multimodal prompting-guided segmentation and fusion collaborative promotion framework is constructed. By combining prompts with multimodal multi-tasks, model fine-tuning and multi-task collaborative promotion are carried out to achieve better fusion.

[0036] The controllable multimodal method based on prompting and guidance includes:

[0037] The user input prompt mask is used to extract preliminary correlation features from the mask and the dual-light image. Then, the features are enhanced through self-attention and cross-attention. The prompt features are obtained by the frozen SAM prompt encoder. Based on the prompt features and the dual-light features, the final fusion features containing information about the region of interest are obtained. The decoder then produces a controllable fusion image. The preliminary fusion image is embedded with dual-light prompts to obtain two segmentation masks. The dual-light masks are then added element-wise to obtain the final segmentation result.

[0038] The SAM-based semantic awareness feature extraction framework includes:

[0039] The framework introduces a frozen SAM image encoder, a cue encoder, and an image decoder, which respectively extract high-dimensional features of the image, obtain cue embeddings of the user control model’s region of interest, and output the image decoder as a fused image.

[0040] The SAM image encoder consists of 32 stacked transformer modules, followed by a neck to adjust the dimension of the output embedding; the image decoder mainly consists of two TwoWayTransformers.

[0041] The structural diagram of this implementation is as follows: Figure 1 As shown:

[0042] The basic steps of the multimodal prompting-guided segmentation and fusion collaborative facilitation framework are as follows:

[0043] Given a pair of dual-light images and a mask cue, the dual-light images are each processed by an encoder with a reference GRDB feature extraction module structure to extract preliminary features. The preliminary features of the dual-light images are concatenated to obtain preliminary fused features. The preliminary features obtained from the dual-light images are then multiplied with the mask as support features and pooled. The pooled features are then concatenated with the preliminary features of the dual-light images along the first channel. The concatenated features are then convolved to reduce the number of channels to 256. The dimensionality-reduced features are then processed with a set of learnable query vectors q (initialized to 0 and continuously updated through learning) through one cross-attention and one self-attention pass. Finally, the fused features are processed with the preliminary fused features as query features through one cross-attention and one self-attention pass to obtain a cue. This cue is then input into the SAM frozen image encoder to obtain the dual-light cue embedding. The dual-light cue embedding and the preliminary fused features after the frozen SAM image encoder are then input into the SAM mask decoder to obtain the dual-light segmentation mask. The dual-light mask is then summed element-wise to obtain the final segmentation mask. Then, the two-light images are processed through their respective branches to obtain preliminary features, and the prompt embedding and segmentation mask are input to the prompt-semantic fusion module to obtain their respective prompt features. The prompt features of the two lights are added to the preliminary fusion features to obtain the final fusion features, which are then processed by the decoder to obtain the final fusion image.

[0044] The GRDB module implemented in this paper mainly consists of convolutional layers (Conv), Leaky ReLU activation function layers, and gradient operators. 3×3 convolutional layers are used to extract local features from the data; Leaky ReLU activation function layers introduce non-linearity into the network, enhancing its expressive power; 1×1 convolutional layers can adjust the number of channels to achieve feature integration and transformation; and gradient operators are used to capture gradient information in the data, highlighting the changing parts. These components work together, fusing information through residual connections and other methods to achieve specific feature extraction and processing functions.

[0045] y=x+Conv(GradientOperator(x))+Conv(LReLU(Conv(LReLU(Conv(x))))))

[0046] The SAM image decoder includes a Transformer module that receives image embedding features, mask tokens, and IoU tokens generated by the image encoder. The multilayer perceptron (MLP) contains a mask_MLP for predicting the mask and an iou_MLP for predicting the intersection-over-union (IoU). Positional encoding is input into the Transformer in the form of image positional encoding (img_pe). Upscaling restores the low-resolution feature map to a resolution close to that of the original image, thus obtaining the final usable segmentation mask.

[0047] The multimodal cue-guided segmentation and fusion model, the device comprising:

[0048] The feature extraction module is used to extract features from multimodal input data and output the feature information of multimodal data.

[0049] The feature fusion module is used to achieve interactive fusion between multimodal features;

[0050] The prompting and guidance module is used to provide prompts for multimodal data and obtain features of regions of interest, so as to achieve controllable fusion and personalized segmentation.

[0051] The specific steps of this invention are as follows:

[0052] 101: The multimodal cue-guided segmentation and fusion model in this invention uses GRDB (Gradient Residual Dense Block) as the backbone to extract multimodal features and uses the features extracted by the gradient operator as residual connections, which can enhance the network's extraction of detailed features.

[0053] 102: The feature extraction of the part of interest guided by cueing is obtained by pooling and two sets of self-attention and cross-attention, followed by a cueing encoder of SAM to obtain a set of cue embeddings;

[0054] 103: After obtaining multimodal features and cue embeddings, the features need to be interactively fused. The transformer-based fusion module includes a self-attention module and a cross-attention module.

[0055] 104: After the initial feature fusion, dual-light fusion and segmentation of the fused image are performed. For the fusion and segmentation parts, this embodiment uses weighted intensity gradient loss, structural similarity loss and perceptual loss as fusion loss, and binary cross-entropy loss and Dice loss as segmentation loss.

[0056] The effectiveness of the proposed method was verified on two datasets: the FMB dataset and the MSRS dataset.

[0057] The Two-Light Dataset (FMB), constructed by the School of International Information Science and Technology at Dalian University of Technology, comprises 1500 pairs of precisely registered infrared and visible light images with a resolution of 800×600 pixels. Each pair has 15 pixel-level labeled categories. It features rich scene representation, covering complex environments and lighting conditions such as dense fog and strong light, with detailed annotations including common targets like roads and pedestrians, and possesses all-weather, multi-modal characteristics. Acquired using an intelligent multi-band binocular imaging system and obtained through calibration and registration, it is primarily used for research in image fusion and segmentation, particularly in scenarios such as autonomous driving and robotic operations.

[0058] The Dual-Light MSRS dataset is a multispectral road scene dataset for infrared and visible light image fusion, built upon the MFNet dataset. It removes 125 misaligned image pairs from the MFNet dataset and optimizes the contrast and signal-to-noise ratio of infrared images using an image enhancement algorithm based on dark channel priors. The final dataset contains 1444 high-quality, spatially 480×640 aligned infrared and visible light images. The dataset was acquired using an InfRec R500 camera and is the first RGB-Thermal urban scene image dataset with pixel-level annotations. It labels eight common obstacles encountered during driving and can be used to support future research on autonomous vehicles.

[0059] The multimodal cue-guided segmentation and fusion model in this invention includes a reference cue encoder, a cue-semantic fusion module, and an auxiliary network. The specific process of this invention is as follows: Figure 2 As shown: This embodiment displays a pair of infrared images and visible light images Input the infrared encoder and the visible light encoder respectively to extract feature Fir and F vi Then F ir and F vi Connect to get F ref As input to the image decoder, which consists of convolutional layers and activation layers, the reference image I is obtained. ref Use the mask as a hint along with F. ir or F vis and F ref Send them together to the reference prompt encoder. In the reference prompt encoder, use F... ir or F vis As a supporting feature, F ref As query features, the suggestion feature embedding P is obtained respectively. ir and P vis Then I ref Input a frozen SAM image encoder, the result of which is P ir Or P vis Together, they are fed into the frozen SAM mask decoder to obtain two prediction masks M. ir and M vis M ir and M vis The image is segmented after being stitched together; this is to obtain dual-light cue features. and The predicted mask, cue embedding, and encoded features are input into the cue-semantic fusion module. The initial fusion features F are then processed. ref With prompt features and Element-wise addition is performed to obtain the final fused features, which are then fed into the image decoder to generate the final fused image.

[0060] The reference encoder of the present invention is as follows Figure 3 As shown, this is used to extract cue embeddings from a controllable region of interest. Taking an infrared reference cue encoder as an example, F... ir As a supporting feature, F ref As query features, and together with the prompt m Together, they are input into the encoder. To enhance features in the region of interest, a Prompt is calculated. m With F ir The Hadamard product between them is then used to obtain F through average pooling. t This process can be described as follows:

[0061] F T =AveragePool(Prompt) m ·F ir )

[0062] To obtain a holistic feature representation that can simultaneously capture detailed local information and global context, thereby improving model performance and generalization ability, this model will use F... t With F ir The connections are then made, and the resulting features are generated through convolutional layers. F t With F ref Features of connection generation results

[0063] Q∈R N×C It is a set of learnable query vectors that they derive from F supp Extracting information from different aspects, where N represents the number of salient features the model focuses on, firstly, the query vectors Q and F... supp The input is passed through a cross-attention layer (CrossAttention 1) and then fed into a self-attention layer (SelfAttention 1) to obtain a controllable query vector Q for a specific category in the support image. ' Then, Q ' and F qry The output of this process is then fed into a self-attention layer (SelfAttention 2) to generate a set of reference cues P. ' These hints correspond to specific categories present in the query image. The process can be described as follows:

[0064] Q′=SelfAttn1(CrossAttn1(Q,F supp ))

[0065] P′=SelfAttn2(CrossAttn2(Q′,F qry ))

[0066] Then, P' is passed through the frozen SAM prompt encoder to generate the final prompt feature embedding P.

[0067] Tip - Semantic fusion module, such as Figure 4 As shown, to obtain category-specific cue features and In the infrared and visible light branches, the initial encoded features F, the cue feature embedding P, and the corresponding SAM segmentation output M are used as inputs, respectively. Considering the significant differences in how different modalities contribute to semantic information, this embodiment uses the segmentation results from each modality-specific branch (e.g., infrared and visible light) as a mask, rather than using the final fused segmentation result. Specifically, the quality of the segmentation result directly reflects the richness and quality of the high-level features provided by that modality, thus indicating its potential contribution to the overall task performance. This strategy not only helps quantify the effectiveness of each modality but also provides more accurate guidance for multimodal data fusion. First, this module processes the feature maps... Downsampling is performed to reduce the computational load on the network. Then, the downsampled features are flattened into sequence format F. seq ∈R (H*W)×C The process can be described as follows:

[0068] F seq =Flatten(Down(F))

[0069] Serialization feature F seq The previously obtained cue embedding P is processed through a cross-attention mechanism. Subsequently, a view transformation operation is performed on the features output by the attention mechanism to restore their original spatial dimensions, and an upsampling step is used to restore them to the original resolution, thus obtaining enhanced spatial features. Finally, these enhanced spatial features are element-wise multiplied with the corresponding segmentation mask to extract enhanced features associated with the specified category. This process can be described as follows:

[0070] F p =M·(Up(View(CrossAttn(F)) seq ,P))))

[0071] Evaluation metrics and agreements

[0072] For all two-light fusion methods, this invention uses AG, CC, SCD, MSE, PSNR, Qabf, Nabf, and MS_SSIM for evaluation.

[0073] AG stands for Average Gradient, a commonly used image quality evaluation metric, especially in fields such as image fusion. It reflects the degree of drastic change in grayscale within an image, i.e., the sharpness of the image.

[0074]

[0075] Where M and N are the number of rows and columns of the image, respectively, and f(i,j) is the gray value of the image at coordinates (i,j).

[0076] CC refers to the correlation coefficient, a commonly used image quality assessment metric used to measure the similarity between two images. Its calculation formula is as follows:

[0077]

[0078] Where I(i,j) and J(i,j) represent the gray values ​​of the two images at coordinates (i,j), respectively. , where M and N are the average gray values ​​of the two images, respectively, and M and N are the number of rows and columns of the image, respectively.

[0079] SCD stands for Structural Similarity Deviation, which measures the degree of difference between an image and the original image in terms of structural information. Based on the principle of Structural Similarity (SSIM), it calculates the similarity scores between the reconstructed image and the original image in terms of brightness, contrast, and structure, thus determining the deviation value between the two. A lower SCD value indicates a higher structural similarity between the reconstructed image and the original image, and thus better image quality.

[0080] MSE stands for Mean Squared Error, a commonly used image quality assessment metric. It calculates the average of the sum of squared differences between corresponding pixel values ​​in the original and processed images. Assuming both the original image I and the processed image J are of size M×N, then:

[0081]

[0082] PSNR, or Peak Signal-to-Noise Ratio, is a widely used image quality metric. It is calculated based on MSE and reflects the ratio of the maximum signal strength to the noise strength of an image.

[0083]

[0084] It is the square of the maximum value of the image pixels.

[0085] QABF is a specific image quality evaluation metric, generally used to measure the similarity and fusion effect between the fused image and the source image. It comprehensively considers multiple aspects of image information, such as brightness, contrast, and structure, and calculates the similarity measure between the fused image and the source image in different feature spaces, and then performs a weighted combination to obtain a comprehensive quality evaluation score.

[0086]

[0087] Where N is the number of image patches, w i It is the weight of the i-th image patch, μ a,i and μ b,i and are the mean values ​​of the i-th image patches in source images a and b, respectively. and These are the standard deviations of the i-th image patch in source images a and b, respectively, σ a,b,i C1 and C2 are constants, where C1 and C2 are the covariance of the i-th image patch in source images a and b.

[0088] The Nabf index is related to factors such as image noise level and detail preservation. It evaluates image quality by analyzing and quantifying the image's noise characteristics and high-frequency detail information.

[0089]

[0090] Where N is the number of image patches, w i It is the weight of the i-th image patch, σ n,i σ is the noise standard deviation of the i-th image patch. a,b,i is the covariance of the i-th image patch in source images a and b, and C is a small constant used to avoid the denominator being zero.

[0091] MS–SSIM stands for Multi-Scale Structural Similarity, an extension of the traditional SSIM metric that considers the structural similarity of images at different scales. It calculates the structural similarity of images at multiple different resolution scales and weights these similarity measures at different scales to more comprehensively evaluate image quality.

[0092]

[0093] This embodiment compares with other state-of-the-art fusion algorithms on the dual-light datasets FMB and MSRS to demonstrate the positive effects of the proposed method on fusion and segmentation tasks. Furthermore, this embodiment compares with other state-of-the-art algorithms on downstream tasks on the public datasets FMB and MSRS, thus proving the superiority of the proposed multimodal cue-guided fusion and segmentation framework among multimodal algorithms in the same field.

[0094] For the SAM model, this paper adopts the parameters of ViT-H. The model used is Adam, with a learning rate of 1×10⁻⁶. -4 The weight decays to 10. -6In the training loss part, ingrad:ssim:percep:seg = 1:1:1:1, where in the ingrad loss, the intensity ratio gradient is 1:10, in the ssim loss, the visible light and infrared image weights are 0.6 and 0.4 respectively, in the percep loss, the visible light and infrared image weights are 0.3:0.7 respectively, and in the seg loss, the bce and dice weights are 1:1.

[0095] This invention performs a horizontal comparison with three methods—DID, CDD, and SeAFusion—on multiple image metrics on the FMB and MSR datasets, and also tests them on two downstream tasks: semantic segmentation and object detection.

[0096] The experimental results used in the embodiments of the present invention are shown in Tables 1 and 2.

[0097] Table 1 Comparison of metrics for the dual-light fusion algorithm on the FMB dataset

[0098] CDD 3.794 0.599 1.623 0.038 64.309 0.689 0.023 1.023 DID 2.127 0.566 1.121 0.035 63.94 0.204 0.025 0.505 SeAFusion 3.842 0.607 1.707 0.036 64.491 0.675 0.021 1.045 MMPG-SFM 3.654 0.621 1.699 0.033 64.993 0.636 0.013 1.063

[0099] Table 2 Comparison of metrics for the dual-light fusion algorithm on the MSRS dataset

[0100] CDD 4.339 0.631 1.626 0.048 62.696 0.674 0.026 1.072 DID 4.327 0.641 1.824 0.047 61.565 0.528 0.042 0.997 SeAFusion 4.49 0.622 1.62 0.047 62.539 0.654 0.029 1.077 MMPG-SFM 4.358 0.598 1.481 0.044 63.116 0.677 0.021 1.057

[0101] As shown in Tables 1 and 2, on the FMB dataset, MMPG-SFM performs best in MSE, PSNR, Qabf, and Nabf, and second best in AG. On the MSRS dataset, MMPG-SFM performs best in CC, MSE, PSNR, Nabf, and MS_SSIM, and second best in SCD.

[0102] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the present invention will not describe the various possible combinations separately.

Claims

1. A low-altitude multimodal image fusion method based on semantic prompting, characterized in that, Specifically: Step 1: Image of visible light and mask image The input is fed into a visible light encoder to obtain the encoded features. Infrared images and mask image The input is fed into an infrared optical encoder to obtain the encoded features. ; Step Two: and Preliminary fusion characteristics were obtained through connection. ; Step 3: Put The reference image is obtained by inputting it into the image decoder. ; Step 4: Mask image As a hint, encoded features serve as supporting features. As a query feature; Will , as well as The input is fed into a visible light reference cue encoder to obtain cue embedding features. ;Will , as well as The input is fed into the infrared light reference cue encoder to obtain the cue embedding features. ; Step 5: , and The input is sent to the image segmentation module for processing. Segmentation is performed to obtain mask features. ,right Segmentation is performed to obtain mask features. ; and will and The segmented images are then stitched together. Step Six: , , , , as well as The input is fed into the prompt feature extraction module to obtain the prompt features corresponding to the infrared light image. and cue features corresponding to visible light images ; Step 7: Add prompt features Hint features as well as The image is input into the interactive fusion module to obtain the fused image; The visible light reference cue encoder and the infrared light reference cue encoder have the same structure. First, the supporting features are multiplied with the cue, and the result of the multiplication is input into an average pooling layer to obtain the features. ;Will After being connected to the supporting features, the connection result is input into the convolutional layer to generate the final features. ;Will Generate result features by concatenating with query features. ; a set of learnable query vectors Characteristics of the results The input is fed into the first cross-attention layer, and the output of the first cross-attention layer is fed into the first self-attention layer to obtain a controllable query vector for a specific category. ,Will and outcome characteristics The input is fed into the second cross-attention layer, and the output of the second cross-attention layer is fed into the second self-attention layer to generate a set of reference cues. Finally The input is fed into the frozen SAM cue encoder to obtain the final cue embedding features; The cue feature extraction module includes an infrared light cue-semantic fusion module and a visible light cue-semantic fusion module; , as well as The input is fed into the infrared light cue-semantic fusion module to obtain the infrared light cue features. ;Will , as well as The input is fed into the visible light cue-semantic fusion module to obtain visible light cue features. ; The infrared cues-semantic fusion module and the visible light cues-semantic fusion module have the same structure. They downsample the encoded features and flatten the downsampled features into a sequence format. ,Will The embedded features of the cues are input into the cross-attention mechanism for processing. The features output by the cross-attention mechanism are subjected to a view transformation operation, and then the features after the view transformation operation are upsampled to obtain enhanced spatial features. The enhanced spatial features are then multiplied element-wise with the predicted mask features to obtain infrared light cues features or visible light cues features.

2. The low-altitude multimodal image fusion method based on semantic prompting as described in claim 1, characterized in that, The image segmentation module includes a frozen infrared SAM mask decoder, a frozen visible SAM mask decoder, and a frozen SAM image encoder; Input to the frozen SAM image encoder; Embedded features of the frozen SAM image encoder output and cueing The predicted mask features are obtained as input to the frozen infrared SAM mask decoder. Embed the output and cueing of the frozen SAM image encoder into features. As input to the frozen visible light SAM mask decoder, the predicted mask features are obtained. .

3. The low-altitude multimodal image fusion method based on semantic prompting as described in claim 1, characterized in that, The interactive fusion module includes an image decoder, which will... , as well as After element-wise addition, the results are input into the image decoder to obtain the final fused feature map.

4. The low-altitude multimodal image fusion method based on semantic prompting as described in claim 1, characterized in that, Both the visible light encoder and the infrared encoder use GRDB modules.

5. The low-altitude multimodal image fusion method based on semantic prompting as described in claim 2, characterized in that, The frozen SAM image encoder consists of 32 stacked transformer modules and a neck module connected to the last transformer module.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements a low-altitude multimodal image fusion method based on semantic prompting as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a low-altitude multimodal image fusion method based on semantic prompting as described in any one of claims 1 to 5.

Images