Infrared and visible light image fusion method with enhanced scene guidance prompt characterization

Abstract

The invention belongs to the technical field of image information processing, and discloses a scene guidance prompt representation enhanced infrared and visible light image fusion method, which is divided into two stages: a first stage, constructing a scene prompt generation network, and learning global visual semantic information covering a source image through a semantic segmentation task; in order to further enhance the prompt representation capability, a visual perception context prompt module is designed, interaction is performed by using a correlation matrix between modal specific features and text features, and the text features are refined in a dynamic weighting mode, so that scene prompt representation with richer semantics is obtained. In the second stage, a cross-modal alignment fusion network guided by prompt is provided, and infrared and visible light features are mapped to a unified shared embedding space by utilizing learned scene prompt. In the process, a pixel-text similarity matrix is established through a prompt driving feature alignment module, and accurate alignment of cross-modal features is realized, so that a fusion result of semantic consistency and detail fidelity is obtained.

Description

Technical Field

[0001] This invention belongs to the field of image information processing technology, and relates to an infrared and visible light image fusion method for scene guidance and prompting representation enhancement. Background Technology

[0002] Currently, the technologies related to this invention include two aspects: the first is a cross-modal image fusion method; the second is a text-driven image fusion method.

[0003] Cross-modal image fusion methods can be broadly categorized into two types: One type involves feature decomposition methods. These methods employ pre-designed, manually decomposition strategies to decompose infrared and visible images into different types of feature representations, thus mitigating the feature gap between different modalities to some extent. For example, Li et al., in their paper "Lrrnet: A novel representation learning guided fusion network for infrared and visible images," designed a learnable low-rank representation model that uses low-rank sparse coefficients to decompose the source image into basic and salient features. Xu et al., in their paper "DRF: Disentangled representation for visible and infrared image fusion," used a decoupled representation learning method to decompose the source image into scene features and attribute features, thereby alleviating the modal feature coupling problem. However, feature decomposition methods rely on manually designed decomposition rules, making it difficult to achieve adaptive feature decomposition for complex scenes. The other type involves modality transformation methods. These methods convert the original modal image into a pseudo-modal image, making the feature distributions of different modalities more consistent, thus solving the mismatch problem in cross-modal feature spaces. Li et al., in their paper "Dstcfuse: A method based on dual-cycled cross-awareness of structure tensor for semantic segmentation via infrared and visible image fusion," proposed a dual-cycled modality transfer network that fuses pseudo-infrared and infrared images with complementary information from two modalities, thereby reducing modal differences. Wang et al., in their paper "Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration," explored a cross-modal perceptual style transfer network to achieve cross-modal conversion from visible light images to pseudo-infrared images, thus bridging significant modal differences. Although modality transfer methods perform well in mitigating feature inconsistencies, they are prone to introducing information loss and artifacts during the conversion process, thus affecting the fusion quality.

[0004] In recent years, pre-trained visual-language models have achieved rapid development by leveraging large-scale image-text pairs. For example, the CLIP model, through a contrastive learning strategy, maps image and text representations to a shared embedding space, achieving semantic alignment. Consequently, text-driven fusion learning methods have been proposed. These methods utilize manually annotated text descriptions as semantic guidance, directing fusion features to retain important information from each modality, thereby generating language-consistent fused images. For instance, Zhang et al., in their paper "Text-DiFuse: An Interactive Multi-Modal ImageFusion Framework based on Text-modulated Diffusion Model," employed a zero-shot localization model and SAM technology to recognize text instructions and locate target objects, thereby enhancing the saliency of targets in the fused image. Li et al., in their paper "From text to pixels: a context-aware semantic synergy solution for infrared and visible image fusion," utilized high-level semantic information in text descriptions to enhance semantic consistency between source images, thus improving fusion performance and target detection accuracy. However, these methods rely excessively on manually annotated text descriptions as guidance, failing to fully extract the important information from the source images.

[0005] In practical applications, the aforementioned methods struggle to fully bridge the feature gap between infrared and visible light images, resulting in poor fusion quality. To address this issue, a potential solution is to learn scene cues between infrared and visible light images using a semantic segmentation task. These scene cues can adaptively learn modality-independent feature representations from the source images, effectively bridging the feature gap between different modalities. Therefore, we propose a scene-guided cue representation-enhanced infrared and visible light image fusion method, which consists of two stages. In the first stage, a scene cue generation network learns scene cues through a semantic segmentation task, covering all visual information in the source images. Specifically, a visually perceptive contextual cue module constructs a correlation matrix between modality-specific features and text features, then dynamically weights text features to refine the scene cue representation. In the second stage, a cue-guided cross-modal alignment fusion network is designed, then maps the features of two modalities with feature gaps to a shared embedding space using the acquired scene cues. During this process, a cue-driven feature alignment module is used to establish a pixel-text similarity matrix to perform cross-modal feature alignment. Summary of the Invention

[0006] To address the feature gap problem in infrared and visible light image fusion tasks, a scene-guided cue representation enhancement method for infrared and visible light image fusion is proposed. The core idea of ​​this method is to generate scene cues using a semantic segmentation task to adaptively learn modality-independent feature representations, thereby bridging cross-modal differences and improving fusion results. Specifically, the method consists of two stages: In the first stage, a scene cue generation network is constructed, learning global visual semantic information covering the source images through a semantic segmentation task. To further enhance the cue representation capability, a visually perceptual contextual cue module is designed, utilizing the correlation matrix between modality-specific features and text features for interaction, and refining text features through dynamic weighting to obtain a semantically richer scene cue representation. In the second stage, a cue-guided cross-modal alignment fusion network is proposed, using the learned scene cues to map infrared and visible light features to a unified shared embedding space. During this process, a cue-driven feature alignment module establishes a pixel-text similarity matrix to achieve accurate alignment of cross-modal features, thereby obtaining a fusion result with semantic consistency and detail fidelity.

[0007] The technical solution of the present invention:

[0008] An infrared and visible light image fusion method with enhanced scene guidance and cue representation is provided. The infrared and visible light image fusion method consists of two stages: a scene cue generation network and a cue guidance cross-modal alignment fusion network.

[0009] The specific steps are as follows:

[0010] (1) Scene-provided network generation;

[0011] A learnable contextual cue consists of multiple learnable word vectors and a set of category names, and its formal definition is as follows:

[0012] (1)

[0013] in, This represents multiple learnable word vectors of length n. This represents a set containing k category names;

[0014] First, the infrared image and visible light images Input pre-trained CLIP image encoder Infrared modal features were extracted separately. and visible light mode characteristics At the same time, provide learningable scenario prompts. Embedded CLIP text encoder In the process of generating text features ;

[0015] Subsequently, a contextual cue module based on visual perception is introduced to model the interaction between vision and language, thereby enhancing text feature representation using visual contextual information. Specifically, modal features are calculated. Text features Similarity graph between them, where These represent infrared and visible light modal features, respectively; this similarity map is further used for weighted modal features. This generates text features with modal feature embedding. This effectively aggregates key information from the source image. This process can be formally represented as:

[0016] (2)

[0017] in, This represents the cross-attention mechanism. Query projection layer representing text features and Key and value projection layers representing visual features, respectively;

[0018] Next, the text features embedded in the modality features Modal features Matrix multiplication is performed to obtain a score map, thereby capturing the semantic relevance between scene cues and source images.

[0019] Subsequently, they were fused using element-level summation. and This results in a unified text representation. The infrared features embedded in the cue are obtained through the following methods. :

[0020] (3)

[0021] in, Indicates a feature cascade operation;

[0022] Hints at embedded visible light features It's the same operation;

[0023] Finally, the embedded infrared features will be indicated. and hints embedded visible light features Input to the infrared splitter head respectively and visible light splitter This leads to the generation of a predicted infrared segmentation mask. and visible light segmentation mask The training process uses infrared image segmentation loss. Compared with visible light image segmentation loss Optimize learnable scene prompts This allows for the efficient learning of modality-independent scene semantic representations. Infrared image segmentation loss. Compared with visible light image segmentation loss The format is as follows:

[0024] (4)

[0025] (5)

[0026] in, Indicates a separator label. Represents cross-entropy loss, Indicates Dice loss;

[0027] Among them, infrared segmentation head and visible light splitter All are composed of a fully connected structure consisting of multiple parallel MLP layers;

[0028] (2) Prompt guidance for cross-modal alignment and fusion networks;

[0029] Since learnable scene cues are a shared textual semantic prior extracted from source images, they can guide the mapping of infrared and visible light features to a shared embedding space. The cue-guided cross-modal alignment fusion network is powered by an infrared encoder. Visible light encoder Infrared prompt drive feature alignment module Visible light cue driving feature alignment module Cross-modal feature fusion module and fusion image decoder Composition; including, infrared encoder and visible light encoder The encoder consists of 3x3 convolutional layers and average pooling coding blocks, respectively; in the infrared encoder and visible light encoder Subsequently, two parallel infrared cue drive feature alignment modules are connected. Alignment module with visible light cue driving features Infrared prompt drive feature alignment module Alignment module with visible light cue driving features Both employ an attention-based alignment structure, aligning initial infrared modal features and initial visible modal features respectively through query-key-value interactions under the constraints of learned scene cues; a cross-modal feature fusion module... It consists of a fusion structure of convolutional layers and weight map modulation; fusion image decoder It mainly consists of a series of upsampling layers and 3x3 convolutional layers. Details are as follows:

[0030] like Figure 2 As shown, infrared image and visible light images Input to infrared encoder respectively and visible light encoder Extracting initial infrared modal features and initial visible light mode characteristics Frozen CLIP text encoder Receive learned scene prompts To generate text features Subsequently, the initial infrared mode characteristics and initial visible light mode characteristics Respectively with text features Combined, as an infrared cue driving feature alignment module Alignment module with visible light cue driving features Input; initial infrared features and initial visible light characteristics Unified as initial modal features First, initial modal features The query vector is obtained by projecting through a learnable linear layer. Meanwhile, text features Projected as key vectors respectively Sum value vector Subsequently, by querying the vector With key vector The interactive computation yields a text-pixel similarity matrix, used to capture the semantic relevance between language and vision. This text-pixel similarity matrix is ​​then normalized using softmax to obtain attention weights, which are used to weight the vector. To adjust the initial modal features The semantic representation of the initial modality features. Mapping to a shared embedding space; finally, reconnecting the initial modality features via residual connections. Feature refinement is performed to obtain the updated modality alignment features. The calculation formula is as follows:

[0031] (6)

[0032] in, This represents modal alignment features in a shared embedding space. It is a scaling factor. , , Through learnable linear projection layers , , get.

[0033] Then, align the modal features. and The images are stitched together, and preliminary fusion features are obtained through three 3×3 convolutional layers. Then, a 1×1 convolutional layer is used to obtain weight maps from the infrared and visible light images, respectively. and Among them, the weighted graph The calculation method is as follows:

[0034] (7)

[0035] in, This represents a convolutional layer consisting of n convolutional kernels of size k×k;

[0036] Weighted graph Calculation method and weight map The calculation method is the same;

[0037] Then, and To each and Channel-wise element-wise multiplication is performed to obtain the enhanced modal features; subsequently, the two enhanced modal features are concatenated along their channel dimensions and further fused using a 3×3 convolutional layer; the fusion process is represented as follows:

[0038] (8)

[0039] Finally, the feature input is fused into the fusion decoder. Generate fused image ;

[0040] During training, a fusion loss is introduced to constrain the fusion model to retain the content and structural information from both infrared and visible light images, as shown below:

[0041] (9)

[0042] in, This represents the absolute value error loss. This represents the structural similarity loss.

[0043] The beneficial effects of this invention are as follows: This invention adaptively learns scene cues from source images through a semantic segmentation task, effectively overcoming the shortcomings of manually annotated text descriptions. Simultaneously, the learned scene cues, as a modality-independent semantic representation, are used to guide cross-modal feature alignment, thereby eliminating feature gaps between different modalities. Based on the above mechanism, the fusion model can effectively preserve key complementary features between source images, thus generating a fused image with high structural fidelity and rich detail information. Attached Figure Description

[0044] Figure 1 Generate a network structure diagram for scene prompts.

[0045] Figure 2 This is a schematic diagram of a cross-modal alignment and fusion network structure to provide guidance.

[0046] Figure 3 This is a schematic diagram of the driver feature alignment module structure.

[0047] Figure 4 The following are visualization results of the fusion features of the prompt-driven feature alignment module, where (a1) and (a2) are infrared images, (b1) and (b2) are visible light images, (c1) and (c2) are visualization results of the fusion features of the prompt-driven fusion module without prompts, and (d1) and (d2) are visualization results of the fusion features of the prompt-driven fusion module with prompts. Detailed Implementation

[0048] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0049] In the first phase, Figure 1 Generate a network structure diagram for scene cues. Infrared image. and visible light images Input pre-trained CLIP image encoder Infrared modal features were extracted separately. and visible light mode characteristics At the same time, learnable scenario prompts will be provided. Embedded CLIP text encoder In the process of generating text features Subsequently, a contextual cue module based on visual perception is introduced to model the interaction between vision and language, thereby obtaining a text representation with enhanced visual semantics, as shown in formulas (2)-(3). Then, the infrared features embedded in the cue are... and hints embedded visible light features Input to the infrared splitter head respectively and visible light splitter And combined with infrared image segmentation loss Compared with visible light image segmentation loss Joint optimization is performed, as shown in formulas (4)-(5). In this way, the learnable scene cues can not only obtain consistent semantic representations across modalities, but also provide a unified semantic prior for subsequent cross-modal feature alignment.

[0050] In the second phase, Figure 2 This is a schematic diagram of a cross-modal alignment and fusion network structure to guide the design. First, an infrared image... and visible light images Input to infrared encoder respectively and visible light encoder Extracting initial infrared modal features and initial visible light mode characteristics Meanwhile, the CLIP text encoder is frozen. Learned scene prompts Convert to text features Then, the initial infrared mode features and initial visible light mode characteristics Text features Enter them separately Figure 3 Infrared cue driving feature alignment module Alignment module with visible light cue driving features This mapping of different modal features to a unified shared embedding space is shown in Equation (6). This design effectively eliminates inconsistencies in cross-modal feature distribution, laying a semantically unified foundation for subsequent feature fusion. Next, infrared modality alignment features... Features aligned with visible light modes Input to cross-modal feature fusion module By fully integrating the complementary information of the two modalities through a weighted aggregation mechanism, a complete fusion feature is obtained, as shown in formulas (7)-(8). Figure 4Visualization of the fused features from the cue-driven feature alignment module is presented. (a1) and (a2) are infrared images, (b1) and (b2) are visible light images, (c1) and (c2) are visualizations of the fused features without the cue-driven fusion module, and (d1) and (d2) are visualizations of the fused features with the cue-driven fusion module. In (c1) and (c2), the fused features generated by the cue-driven feature alignment module exhibit significant texture blurring and loss of structural information (e.g., buildings). In contrast, in (d1) and (d2), the fused features using the cue-driven feature alignment module retain both the thermal radiation information of the infrared image and the fine structural features of the visible light image. Experiments show that learnable scene cues can effectively map cross-modal features with feature gaps to a shared embedding space using the cue-driven feature alignment module, thereby guiding the enhancement of fused representations. Subsequently, the fused features are input into the fusion decoder. A high-quality fused image is reconstructed. Finally, the entire fusion framework employs a method... Loss and The loss function composed of the losses Optimization is shown in formula (9). Wherein, The loss ensures pixel-level consistency. The loss causes the fused image to be similar to the source image in terms of structural and perceptual quality.

Claims

1. A method for enhancing the representation of scene guidance and prompts through infrared and visible light image fusion, characterized in that, The infrared and visible light image fusion method consists of two stages: a scene cue generation network and a cue-guided cross-modal alignment fusion network. (1) Scene-provided network generation; A learnable contextual cue consists of multiple learnable word vectors and a set of category names, and its formal definition is as follows: （1） in, This represents multiple learnable word vectors of length n. This represents a set containing k category names; First, the infrared image and visible light images Input pre-trained CLIP image encoder Infrared modal features were extracted separately. and visible light mode characteristics At the same time, provide learningable scenario prompts. Embedded CLIP text encoder In the process of generating text features ; Subsequently, a contextual cueing module based on visual perception is introduced to model the interaction between vision and language: calculating modal features. Text features Similarity graph between them, where These represent infrared and visible light modal features, respectively; this similarity map is further used for weighted modal features. This generates text features with modal feature embedding. The process is formally represented as: （2） in, This represents the cross-attention mechanism. Query projection layer representing text features and Key and value projection layers representing visual features, respectively; Next, the text features embedded in the modality features Modal features Matrix multiplication is performed to obtain the score graph; Subsequently, they were fused using element-level summation. and This results in a unified text representation. The infrared features embedded in the cue are obtained through the following methods. : （3） in, Indicates a feature cascade operation; Hints at embedded visible light features It's the same operation; Finally, the embedded infrared features will be indicated. and hints embedded visible light features Input to the infrared splitter head respectively and visible light splitter This leads to the generation of a predicted infrared segmentation mask. and visible light segmentation mask The training process uses infrared image segmentation loss. Compared with visible light image segmentation loss Optimize learnable scene prompts The format is as follows: （4） （5） in, Indicates a separator label. Represents cross-entropy loss, Indicates Dice loss; Among them, infrared segmentation head and visible light splitter All are composed of a fully connected structure consisting of multiple parallel MLP layers; (2) Prompt guidance for cross-modal alignment and fusion networks; The prompt guides the cross-modal alignment and fusion network to be powered by an infrared encoder. Visible light encoder Infrared prompt drive feature alignment module Visible light cue driving feature alignment module Cross-modal feature fusion module and fusion image decoder Composition; including, infrared encoder and visible light encoder The encoder consists of 3x3 convolutional layers and average pooling coding blocks, respectively; in the infrared encoder and visible light encoder Subsequently, two parallel infrared cue drive feature alignment modules are connected. Alignment module with visible light cue driving features Infrared prompt drive feature alignment module Alignment module with visible light cue driving features Both employ an attention-based alignment structure, aligning initial infrared modal features and initial visible modal features respectively through query-key-value interactions under the constraints of learned scene cues; a cross-modal feature fusion module... It consists of a fusion structure of convolutional layers and weight map modulation; fusion image decoder It mainly consists of a series of upsampling layers and 3*3 convolutional layers.

2. The infrared and visible light image fusion method for scene guidance and prompting enhancement according to claim 1, characterized in that, The implementation process of the guided cross-modal alignment and fusion network is as follows: Infrared images and visible light images Input to infrared encoder respectively and visible light encoder Extracting initial infrared modal features and initial visible light mode characteristics Frozen CLIP text encoder Receive learned scene prompts To generate text features ; Subsequently, the initial infrared modal features and initial visible light mode characteristics Respectively with text features Combined, as an infrared cue driving feature alignment module Alignment module with visible light cue driving features Input; initial infrared features and initial visible light characteristics Unified as initial modal features First, initial modal features The query vector is obtained by projecting through a learnable linear layer. Meanwhile, text features Projected as key vectors respectively Sum value vector Subsequently, by querying the vector With key vector The interactive computation yields a text-pixel similarity matrix; this matrix is ​​then normalized using softmax to obtain attention weights, which are used to weight the vector. To adjust the initial modal features The semantic representation of the initial modality features. Mapping to a shared embedding space; finally, reconnecting the initial modality features via residual connections. Feature refinement is performed to obtain the updated modality alignment features. The calculation formula is as follows: （6） in, This represents modal alignment features in a shared embedding space. It is a scaling factor. , , Through learnable linear projection layers , , get; Then, align the infrared modes with the features. Features aligned with visible light modes The images are stitched together, and preliminary fusion features are obtained through three 3×3 convolutional layers. Then, a 1×1 convolutional layer is used to obtain weight maps from the infrared and visible light images, respectively. and Among them, the weighted graph The calculation method is as follows: （7） in, This represents a convolutional layer consisting of n convolutional kernels of size k×k; Weighted graph Calculation method and weight map The calculation method is the same; Then, and To each and Channel-wise element-wise multiplication is performed to obtain the enhanced modal features; subsequently, the two enhanced modal features are concatenated along their channel dimensions and further fused using a 3×3 convolutional layer; the fusion process is represented as follows: （8） Finally, the feature input is fused into the fusion decoder. Generate fused image ; During training, a fusion loss is introduced to constrain the fusion model to retain the content and structural information from both infrared and visible light images, as shown below: （9） in, This represents the absolute value error loss. This represents the structural similarity loss.

Images