Scene-guided prompting representation enhanced infrared and visible image fusion method

Abstract

The application belongs to the technical field of image information processing, and discloses an infrared and visible light image fusion method for scene guidance prompt representation enhancement, which is divided into two stages: in the first stage, a scene prompt generation network is constructed, and global visual semantic information covering a source image is learned through a semantic segmentation task. In order to further enhance the prompt representation capability, a visual perception context prompt module is designed, the correlation matrix between the modal specific features and the text features is utilized for interaction, and the text features are refined through a dynamic weighting mode, so that a scene prompt representation with more rich semantics is obtained. In the second stage, a prompt guided cross-modal alignment fusion network is proposed, and the learned scene prompt is utilized to map the infrared and visible light features to a unified shared embedding space. In this process, a pixel-text similarity matrix is established through a prompt driven feature alignment module, the cross-modal features are accurately aligned, and a fusion result with semantic consistency and detail fidelity is obtained.

Description

Technical Field

[0001] This invention belongs to the field of image information processing technology, and relates to an infrared and visible light image fusion method for scene guidance and prompting representation enhancement. Background Technology

[0002] Currently, the technologies related to this invention include two aspects: the first is a cross-modal image fusion method; the second is a text-driven image fusion method.

[0003] Cross-modal image fusion methods can be broadly categorized into two types: One type involves feature decomposition methods. These methods employ pre-designed, manually decomposition strategies to decompose infrared and visible images into different types of feature representations, thus mitigating the feature gap between different modalities to some extent. For example, Li et al., in their paper "Lrrnet: A novel representation learning guided fusion network for infrared and visible images," designed a learnable low-rank representation model that uses low-rank sparse coefficients to decompose the source image into basic and salient features. Xu et al., in their paper "DRF: Disentangled representation for visible and infrared image fusion," used a decoupled representation learning method to decompose the source image into scene features and attribute features, thereby alleviating the modal feature coupling problem. However, feature decomposition methods rely on manually designed decomposition rules, making it difficult to achieve adaptive feature decomposition for complex scenes. The other type involves modality transformation methods. These methods convert the original modal image into a pseudo-modal image, making the feature distributions of different modalities more consistent, thus solving the mismatch problem in cross-modal feature spaces. Li et al., in their paper "Dstcfuse: A method based on dual-cycled cross-awareness of structure tensor for semantic segmentation via infrared and visible image fusion," proposed a dual-cycled modality transfer network that fuses pseudo-infrared and infrared images with complementary information from two modalities, thereby reducing modal differences. Wang et al., in their paper "Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration," explored a cross-modal perceptual style transfer network to achieve cross-modal conversion from visible light images to pseudo-infrared images, thus bridging significant modal differences. Although modality transfer methods perform well in mitigating feature inconsistencies, they are prone to introducing information loss and artifacts during the conversion process, thus affecting the fusion quality.

[0004] In recent years, pre-trained visual-language models have achieved rapid development by leveraging large-scale image-text pairs. For example, the CLIP model, through a contrastive learning strategy, maps image and text representations to a shared embedding space, achieving semantic alignment. Consequently, text-driven fusion learning methods have been proposed. These methods utilize manually annotated text descriptions as semantic guidance, directing fusion features to retain important information from each modality, thereby generating language-consistent fused images. For instance, Zhang et al., in their paper "Text-DiFuse: An Interactive Multi-Modal ImageFusion Framework based on Text-modulated Diffusion Model," employed a zero-shot localization model and SAM technology to recognize text instructions and locate target objects, thereby enhancing the saliency of targets in the fused image. Li et al., in their paper "From text to pixels: a context-aware semantic synergy solution for infrared and visible image fusion," utilized high-level semantic information in text descriptions to enhance semantic consistency between source images, thus improving fusion performance and target detection accuracy. However, these methods rely excessively on manually annotated text descriptions as guidance, failing to fully extract the important information from the source images.

[0005] In practical applications, the aforementioned methods struggle to fully bridge the feature gap between infrared and visible light images, resulting in poor fusion quality. To address this issue, a potential solution is to learn scene cues between infrared and visible light images using a semantic segmentation task. These scene cues can adaptively learn modality-independent feature representations from the source images, effectively bridging the feature gap between different modalities. Therefore, we propose a scene-guided cue representation-enhanced infrared and visible light image fusion method, which consists of two stages. In the first stage, a scene cue generation network learns scene cues through a semantic segmentation task, covering all visual information in the source images. Specifically, a visually perceptive contextual cue module constructs a correlation matrix between modality-specific features and text features, then dynamically weights text features to refine the scene cue representation. In the second stage, a cue-guided cross-modal alignment fusion network is designed, then maps the features of two modalities with feature gaps to a shared embedding space using the acquired scene cues. During this process, a cue-driven feature alignment module is used to establish a pixel-text similarity matrix to perform cross-modal feature alignment. Summary of the Invention

[0006] To address the feature gap problem in infrared and visible light image fusion tasks, a scene-guided cue representation enhancement method for infrared and visible light image fusion is proposed. The core idea of ​​this method is to generate scene cues using a semantic segmentation task to adaptively learn modality-independent feature representations, thereby bridging cross-modal differences and improving fusion results. Specifically, the method consists of two stages: In the first stage, a scene cue generation network is constructed, learning global visual semantic information covering the source images through a semantic segmentation task. To further enhance the cue representation capability, a visually perceptual contextual cue module is designed, utilizing the correlation matrix between modality-specific features and text features for interaction, and refining text features through dynamic weighting to obtain a semantically richer scene cue representation. In the second stage, a cue-guided cross-modal alignment fusion network is proposed, using the learned scene cues to map infrared and visible light features to a unified shared embedding space. During this process, a cue-driven feature alignment module establishes a pixel-text similarity matrix to achieve accurate alignment of cross-modal features, thereby obtaining a fusion result with semantic consistency and detail fidelity.

[0007] The technical solution of the present invention:

[0008] An infrared and visible light image fusion method with enhanced scene guidance and cue representation is provided. The infrared and visible light image fusion method consists of two stages: a scene cue generation network and a cue guidance cross-modal alignment fusion network.

[0009] The specific steps are as follows:

[0010] (1) Scene-provided network generation;

[0011] A learnable contextual cue consists of multiple learnable word vectors and a set of category names, and its formal definition is as follows:

[0012] (1)

[0013] in, This represents multiple learnable word vectors of length n. This represents a set containing k category names;

[0014] First, the infrared image and visible light images Input pre-trained CLIP image encoder Infrared modal features were extracted separately. and visible light mode characteristics At the same time, learnable scenario prompts will be provided. Embedded CLIP text encoder In the process of generating text features ;

[0015] Subsequently, a contextual cue module based on visual perception is introduced to model the interaction between vision and language, thereby enhancing text feature representation using visual contextual information. Specifically, modal features are calculated. Text features Similarity graph between them, where These represent infrared and visible light modal features, respectively; this similarity map is further used for weighted modal features. This generates text features with modal feature embedding. This effectively aggregates key information from the source image. This process can be formally represented as:

[0016] (2)

[0017] in, This represents the cross-attention mechanism. Query projection layer representing text features and Key and value projection layers representing visual features, respectively;

[0018] Next, the text features embedded in the modality features Modal features Matrix multiplication is performed to obtain a score map, thereby capturing the semantic relevance between scene cues and source images.

[0019] Subsequently, they were fused using element-level summation. and This results in a unified text representation. The infrared features embedded in the cue are obtained through the following methods. :

[0020] (3)

[0021] in, Indicates a feature cascade operation;

[0022] Hints at embedded visible light features It's the same operation;

[0023] Finally, the embedded infrared features will be indicated. and hints embedded visible light features Input to the infrared splitter head respectively and visible light splitter This leads to the generation of a predicted infrared segmentation mask. and visible light segmentation mask The training process uses infrared image segmentation loss. Compared with visible light image segmentation loss Optimize learnable scene prompts This allows for the efficient learning of modality-independent scene semantic representations. Infrared image segmentation loss. Compared with visible light image segmentation loss The format is as follows:

[0024] (4)

[0025] (5)

[0026] in, Indicates a separator label. Represents cross-entropy loss, Indicates Dice loss;

[0027] Among them, infrared segmentation head and visible light splitter All are composed of a fully connected structure consisting of multiple parallel MLP layers;

[0028] (2) Prompt guidance for cross-modal alignment and fusion networks;

[0029] Since learnable scene cues are a shared textual semantic prior extracted from source images, they can guide the mapping of infrared and visible light features to a shared embedding space. The cue-guided cross-modal alignment fusion network is powered by an infrared encoder. Visible light encoder Infrared prompt drive feature alignment module Visible light cue driving feature alignment module Cross-modal feature fusion module and fusion image decoder Composition; including, infrared encoder and visible light encoder The encoder consists of 3x3 convolutional layers and average pooling coding blocks, respectively; in the infrared encoder and visible light encoder Subsequently, two parallel infrared cue drive feature alignment modules are connected. Alignment module with visible light cue driving features Infrared prompt drive feature alignment module Alignment module with visible light cue driving features Both employ an attention-based alignment structure, aligning initial infrared modal features and initial visible modal features respectively through query-key-value interactions under the constraints of learned scene cues; a cross-modal feature fusion module... It consists of a fusion structure of convolutional layers and weight map modulation; fusion image decoder It mainly consists of a series of upsampling layers and 3x3 convolutional layers. Details are as follows:

[0030] like Figure 2 As shown, infrared image and visible light images Input to infrared encoder respectively and visible light encoder Extracting initial infrared modal features and initial visible light mode characteristics Frozen CLIP text encoder Receive learned scene prompts To generate text features Subsequently, the initial infrared mode characteristics and initial visible light mode characteristics Respectively with text features Combined, as an infrared cue driving feature alignment module Alignment module with visible light cue driving features Input; initial infrared features and initial visible light characteristics Unified as initial modal features First, initial modal features The query vector is obtained by projecting through a learnable linear layer. Meanwhile, text features Projected as key vectors respectively Sum value vector Subsequently, by querying the vector With key vector The interactive computation yields a text-pixel similarity matrix, used to capture the semantic relevance between language and vision. This text-pixel similarity matrix is ​​then normalized using softmax to obtain attention weights, which are used to weight the vector. To adjust the initial modal features The semantic representation, thereby transforming the initial modality features Mapping to a shared embedding space; finally, reconnecting the initial modality features via residual connections. Feature refinement is performed to obtain the updated modality alignment features. The calculation formula is as follows:

[0031] (6)

[0032] in, This represents modal alignment features in a shared embedding space. It is a scaling factor. , , Through learnable linear projection layers , , get.

[0033] Then, align the modal features. and The images are stitched together, and preliminary fusion features are obtained through three 3×3 convolutional layers. Then, a 1×1 convolutional layer is used to obtain weight maps from the infrared and visible light images, respectively. and Among them, the weighted graph The calculation method is as follows:

[0034] (7)

[0035] in, This represents a convolutional layer consisting of n convolutional kernels of size k×k;

[0036] Weighted graph Calculation method and weight map The calculation method is the same;

[0037] Then, and To each and Channel-wise element-wise multiplication is performed to obtain the enhanced modal features; subsequently, the two enhanced modal features are concatenated along their channel dimensions and further fused using a 3×3 convolutional layer; the fusion process is represented as follows:

[0038] (8)

[0039] Finally, the feature input is fused into the fusion decoder. Generate fused image ;

[0040] During training, a fusion loss is introduced to constrain the fusion model to retain the content and structural information from both infrared and visible light images, as shown below:

[0041] (9)

[0042] in, This represents the absolute value error loss. This represents the structural similarity loss.

[0043] The beneficial effects of this invention are as follows: This invention adaptively learns scene cues from source images through a semantic segmentation task, effectively overcoming the shortcomings of manually annotated text descriptions. Simultaneously, the learned scene cues, as a modality-independent semantic representation, are used to guide cross-modal feature alignment, thereby eliminating feature gaps between different modalities. Based on the above mechanism, the fusion model can effectively preserve key complementary features between source images, thus generating a fused image with high structural fidelity and rich detail information. Attached Figure Description

[0044] Figure 1 Generate a network structure diagram for scene prompts.

[0045] Figure 2 This is a schematic diagram of a cross-modal alignment and fusion network structure to provide guidance.

[0046] Figure 3 This is a schematic diagram of the driver feature alignment module structure.

[0047] Figure 4 The following are visualization results of the fusion features of the prompt-driven feature alignment module, where (a1) and (a2) are infrared images, (b1) and (b2) are visible light images, (c1) and (c2) are visualization results of the fusion features of the prompt-driven fusion module without prompts, and (d1) and (d2) are visualization results of the fusion features of the prompt-driven fusion module with prompts. Detailed Implementation

[0048] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0049] In the first phase, Figure 1 Generate a network structure diagram for scene cues. Infrared image. and visible light images Input pre-trained CLIP image encoder Infrared modal features were extracted separately. and visible light mode characteristics At the same time, learnable scenario prompts will be provided. Embedded CLIP text encoder In the process of generating text features Subsequently, a contextual cue module based on visual perception is introduced to model the interaction between vision and language, thereby obtaining a text representation with enhanced visual semantics, as shown in formulas (2)-(3). Then, the infrared features embedded in the cue are... and hints embedded visible light features Input to the infrared splitter head respectively and visible light splitter And combined with infrared image segmentation loss Compared with visible light image segmentation loss Joint optimization is performed, as shown in formulas (4)-(5). In this way, the learnable scene cues can not only obtain consistent semantic representations across modalities, but also provide a unified semantic prior for subsequent cross-modal feature alignment.

[0050] In the second phase, Figure 2 This is a schematic diagram of a cross-modal alignment and fusion network structure to guide the design. First, an infrared image... and visible light images Input to infrared encoder respectively and visible light encoder Extracting initial infrared modal features and initial visible light mode characteristics Meanwhile, the CLIP text encoder is frozen. Learned scene prompts Convert to text features Then, the initial infrared mode features and initial visible light mode characteristics Text features Enter them separately Figure 3 Infrared cue driving feature alignment module Alignment module with visible light cue driving features This mapping of different modal features to a unified shared embedding space is shown in Equation (6). This design effectively eliminates inconsistencies in cross-modal feature distribution, laying a semantically unified foundation for subsequent feature fusion. Next, infrared modality alignment features... Features aligned with visible light modes Input to cross-modal feature fusion module By fully integrating the complementary information of the two modalities through a weighted aggregation mechanism, a complete fusion feature is obtained, as shown in formulas (7)-(8). Figure 4Visualization of the fused features from the cue-driven feature alignment module is presented. (a1) and (a2) are infrared images, (b1) and (b2) are visible light images, (c1) and (c2) are visualizations of the fused features without the cue-driven fusion module, and (d1) and (d2) are visualizations of the fused features with the cue-driven fusion module. In (c1) and (c2), the fused features generated by the cue-driven feature alignment module exhibit significant texture blurring and loss of structural information (e.g., buildings). In contrast, in (d1) and (d2), the fused features using the cue-driven feature alignment module retain both the thermal radiation information of the infrared image and the fine structural features of the visible light image. Experiments show that learnable scene cues can effectively map cross-modal features with feature gaps to a shared embedding space using the cue-driven feature alignment module, thereby guiding the enhancement of fused representations. Subsequently, the fused features are input into the fusion decoder. A high-quality fused image is reconstructed. Finally, the entire fusion framework employs a method... Loss and The loss function composed of the losses Optimization is shown in formula (9). Wherein, The loss ensures pixel-level consistency. The loss causes the fused image to be similar to the source image in terms of structural and perceptual quality.

Claims

1. A scene-guided prompting representation enhanced infrared and visible image fusion method, characterized in that, The infrared and visible image fusion method is divided into two stages: a scene cue generation network and a cue-guided cross-modal alignment fusion network. (1) Scene cue generation network The learnable scene cue consists of multiple learnable word vectors and a set of category names, which is defined as follows: （1） wherein, represents a plurality of learnable word vectors of length n, represents a set containing k category names; First, infrared images and visible light images are input into a pre-trained CLIP image encoder , respectively, to extract infrared modal features and visible light modal features ; at the same time, learnable scene prompts are embedded into a CLIP text encoder to generate text features ; Subsequently, a context-aware visual perception module is introduced to model the interaction between vision and language: the similarity map between the modal features and the textual features is computed, where denote the infrared modal features and the visible light modal features, respectively; the similarity map is further used to weight the modal features , so as to generate the textual features of the modal feature embeddings , which is formalized as: （2） wherein, denotes a cross-attention mechanism, denotes a query projection layer for text features, and denote key and value projection layers for visual features, respectively; Next, the text features of the modal feature embedding with the modal features Matrix multiplication is performed to obtain a score map; Subsequently, they were fused using element-level summation. and This results in a unified text representation. The infrared features embedded in the cue are obtained through the following methods. : （3） wherein represents a feature concatenation operation; Prompted embedded visual features Same operation also; Finally, the embedded infrared features will be indicated. and hints embedded visible light features Input to the infrared splitter head respectively and visible light splitter This leads to the generation of a predicted infrared segmentation mask. and visible light segmentation mask The training process uses infrared image segmentation loss. Compared with visible light image segmentation loss Optimize learnable scene prompts The format is as follows: （4） （5） wherein, denotes a segmentation label, denotes a cross-entropy loss, denotes a Dice loss; Wherein, the infrared segmentation head and the visible light segmentation head are composed of a fully connected structure composed of multiple parallel MLP layers; (2) Cue-guided cross-modal alignment fusion network The prompt guided cross-modal alignment fusion network is composed of an infrared encoder , a visible light encoder , an infrared prompt driven feature alignment module , a visible light prompt driven feature alignment module , a cross-modal feature fusion module and a fusion image decoder ; wherein the infrared encoder and the visible light encoder are respectively composed of a 3*3 convolution layer and an average pooling encoding block; after the infrared encoder and the visible light encoder , two parallel infrared prompt driven feature alignment modules and visible light prompt driven feature alignment modules are connected; the infrared prompt driven feature alignment module and the visible light prompt driven feature alignment module both adopt an alignment structure based on an attention mechanism, and under the constraint of a learned scene prompt, the initial infrared modal feature and the initial visible modal feature are respectively aligned through query-key-value interaction; the cross-modal feature fusion module adopts a fusion structure composed of a convolution layer and a weight map modulation; the fusion image decoder is mainly composed of a series of up-sampling layers and 3*3 convolution layers.

2. The scene guidance cue representation enhanced infrared and visible image fusion method according to claim 1, characterized in that, The implementation process of the cue-guided cross-modal alignment fusion network is as follows: Infrared image and visible light image are input to an infrared encoder and a visible light encoder extract initial infrared modal features and initial visible light modal features , a frozen CLIP text encoder receives learned scene cues to generate text features ; Subsequently, the initial infrared modality features and the initial visible light modality features are combined with the text features respectively as inputs of the infrared prompt-driven feature alignment module and the visible light prompt-driven feature alignment module ; the initial infrared features and the initial visible light features are collectively denoted as initial modality features ; first, the initial modality features are projected through a learnable linear layer to obtain query vectors ; meanwhile, the text features are projected into key vectors and value vectors respectively; subsequently, a text-pixel similarity matrix is calculated through the interaction between the query vectors and the key vectors ; the text-pixel similarity matrix is normalized through softmax to obtain attention weights, which are used to weight the value vectors to adjust the semantic representation of the initial modality features , so as to map the initial modality features to a shared embedding space; finally, the initial modality features are refined through a residual connection to obtain updated modality alignment features , and the calculation formula is as follows: （6） wherein, denote modal alignment features in the shared embedding space, is a scaling factor, , , are obtained by learnable linear projection layers , , respectively. Then, align the infrared modes with the features. Features aligned with visible light modes The images are stitched together, and preliminary fusion features are obtained through three 3×3 convolutional layers. Then, a 1×1 convolutional layer is used to obtain weight maps from the infrared and visible light images, respectively. and Among them, the weighted graph The calculation method is as follows: （7） wherein, denotes a convolution layer with n convolution kernels of size k x k; weight map is calculated in the same way as the weight map is calculated in the same way as the weight map Then, and respectively, and and element-wise multiplication in the channel dimension, resulting in enhanced modality features; subsequently, the two enhanced modality features are concatenated in the channel dimension and further fused through a 3x3 convolution layer; the fusion process is represented as: （8） Finally, the fused features are input into a fusion decoder to generate a fused image ; During the training process, the fusion loss is introduced to constrain the fusion model to retain the content and structural information from the infrared image and the visible light image, which is represented as: （9） wherein, denotes the absolute value error loss, denotes the structural similarity loss.

Images