A Text-Guided Semantic Awareness-Based Method and System for Infrared and Visible Image Fusion

Abstract

The application discloses an infrared and visible light image fusion method and system based on text-guided semantic perception, and relates to the technical field of image fusion; a text prompt is acquired by using a reference image of semantic segmentation, and features of source images are respectively acquired by using independent encoders with the same structure; a semantic corresponding alignment module is proposed between the encoder and the decoder, rich feature representations related to the text features are acquired, so that the gap between the text and the image modal is narrowed; in the decoder stage, a semantic modulation module is proposed, the interaction between the text and the image is fully realized, and the effective combination of texture details and high-level semantic information is realized in combination with shallow features, so that a fusion image with rich semantics and complete details is finally generated, and the overall quality of the fusion image is significantly improved in terms of semantic consistency and perception quality.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, specifically to a method and system for fusion of infrared and visible light images based on text-guided semantic perception. Background Technology

[0002] In the field of intelligent sensing, such as autonomous driving, intelligent monitoring, and emergency rescue, multimodal image fusion technology is crucial for improving environmental perception capabilities. However, the physical characteristics of a single imaging modality limit the completeness of its information representation. For example, infrared images, by capturing the thermal radiation information of objects, exhibit strong target detection capabilities under conditions of smoke, night, or low light, but their resolution is low and texture details are insufficient. While visible light images possess high resolution and rich color and texture features, conforming to human visual habits, their perception capabilities significantly decrease in low-light or occluded scenes. Therefore, fusing infrared and visible light images, combining their complementary advantages, can enhance scene details while preserving target saliency, thereby improving the visual quality and semantic expressiveness of the fused image.

[0003] Currently, most image fusion methods still rely on low-level feature extraction or simple pixel-level fusion strategies, lacking a deep understanding of the semantic content of images. These methods struggle to achieve high-quality fusion results in complex scenes, easily leading to problems such as blurred target structures and semantic inconsistencies. For example, while traditional multi-scale decomposition-based methods can preserve edge and texture information of multimodal images to some extent, they fail to fully consider high-level semantic relationships, resulting in poor performance of the fusion results in target recognition and scene understanding tasks. Furthermore, although deep learning-based fusion methods can automatically extract features, most are still limited to simple weighting or concatenation of low-level features, failing to effectively utilize semantic information to guide the fusion process and limiting their robustness in complex environments. Summary of the Invention

[0004] The purpose of this invention is to propose a method and system for fusion of infrared and visible light images based on text-guided semantic perception, which can effectively solve the problem of cross-modal semantic alignment, improve the fusion quality in complex scenarios, and has good adaptability and semantic preservation capabilities.

[0005] According to a first aspect of the present disclosure, a method for fusing infrared and visible light images based on text-guided semantic awareness is provided, comprising the following steps:

[0006] Infrared and visible light images are extracted separately using independent image encoders to obtain the corresponding infrared and visible light image features. These two types of features are then combined.

[0007] Key target categories are identified based on semantic segmentation maps of infrared and visible light images, generating structured text prompts; the text prompts are feature-encoded using a frozen visual-language cross-modal model text encoder to obtain text feature vectors with semantic representation capabilities.

[0008] The text features are semantically distilled, retaining key semantic information and filtering redundant content through a self-attention mechanism. Then, they are mapped to the visual feature space through cross-modal projection. The filtered redundant text features and the combined image features are input into the visual encoder. The semantic space region alignment of text features and image features is achieved during the extraction of image visual patch features.

[0009] Semantic modulation modules are applied to the features of the encoding layer and the decoding layer respectively to adjust the features. Then, the modulated features are fused through skip connections to obtain a fused image with semantic consistency and high-quality perception effect.

[0010] In one embodiment, the image encoder includes convolutional layers, dense blocks, and pooling layers. The source image is first processed by convolutional layers to extract shallow features containing rich texture details, and then deep features are extracted through four downsampling operations. In each layer, dense blocks are first used to extract and retain rich feature information, and then pooling layers are used for downsampling to enhance the contextual expressiveness of the features.

[0011] In one embodiment, the text prompt is fed into a frozen CLIP text encoder, which encodes the text prompt into a text feature vector, ensuring semantic consistency and structural integrity of the text features.

[0012] In one embodiment, the self-attention mechanism reduces the interference of redundant information by enhancing the weights of different key target information in the text feature vector; then the text features are processed by two fully connected layers, which reduces information loss while making the vector shape meet the requirements of subsequent calculations.

[0013] In one embodiment, the visual encoder includes a CLIP image encoder and an adapter, wherein the parameters of the CLIP image encoder are kept frozen, and only the parameters of the adapter are updated, so that cross-modal semantic alignment is achieved by interacting with image features in the CLIP image encoder through different text information.

[0014] In one embodiment, the semantic modulation module includes a softmax normalization layer, a multilayer perceptron, residual connections, and a LeakyReLU activation function. First, the text features are subjected to softmax normalization to improve their adaptability to diverse text inputs and reduce information loss. Then, the processed text features are input in parallel into two multilayer perceptrons to generate channel scaling parameters and channel offset parameters, respectively. Dynamic semantic modulation is achieved by performing affine transformations on the fused image features, significantly enhancing the semantic consistency of the features. Finally, through the synergistic effect of residual connections and the LeakyReLU activation function, overfitting is effectively prevented while further enhancing the nonlinear expressive power of the fused image features. This design achieves precise guidance of visual features by textual semantic information, ensuring that the fusion result maintains high semantic consistency while preserving detail.

[0015] In one embodiment, the semantically modulated features are:

[0016]

[0017] Where ⊙ represents the Hadamard product. Indicates fusion features, This represents the features after semantic modulation, where the channel scaling parameter γ bm With channel translation parameter β bm They are respectively:

[0018]

[0019] in, This represents the first multilayer perceptron. f represents the second multilayer perceptron. t Represents text features.

[0020] According to a second aspect of the present disclosure, an infrared and visible light image fusion system based on text-guided semantic awareness is provided, comprising:

[0021] The module combines infrared and visible light images by extracting features from them using independent image encoders. After obtaining the corresponding infrared and visible light image features, these two types of features are combined.

[0022] The text encoding module identifies key target categories based on semantic segmentation maps of infrared and visible light images, and generates structured text prompts; it uses a frozen visual-language cross-modal model text encoder to encode the text prompts to obtain text feature vectors with semantic representation capabilities.

[0023] The semantic alignment module performs semantic distillation on text features, retains key semantic information and filters redundant content through a self-attention mechanism, and then maps them to the visual feature space through cross-modal projection. The filtered redundant text features and the combined image features are input into the visual encoder, and semantic space region alignment between text features and image features is achieved during the extraction of image visual patch features.

[0024] The fusion module applies semantic modulation to the features of the encoding layer and the decoding layer respectively, and then fuses the modulated features through skip connections to obtain a fused image with semantic consistency and high-quality perceptual effect.

[0025] According to a third aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the memory, wherein the processor executes the program to implement the text-guided semantic perception-based infrared and visible light image fusion method.

[0026] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the described text-guided semantic awareness-based infrared and visible light image fusion method.

[0027] The advantages of the above technical solutions adopted in this invention compared with the prior art are as follows:

[0028] 1. This invention employs an encoder with independent parameters but the same structure for different images, and by introducing DenseBlock, extracts and fully preserves contextual features.

[0029] 2. This invention employs text-guided image fusion. Based on the semantic segmentation map corresponding to the image, it identifies key target categories in the image and generates structured text prompts. Semantic correspondence alignment is introduced to reduce the gap between text and image modalities. Compared with existing text generation mechanisms and cross-modal alignment methods, this method further improves the network's utilization of text prompt information and avoids information loss caused by insufficient interaction.

[0030] 3. This invention utilizes a semantic modulation module to further enhance the text's guidance on the fusion process. By introducing Softmax, it enhances the applicability to different text information. By using residual connection to overfit, it effectively improves the semantic consistency between the fusion result and the text prompt.

[0031] 4. This invention introduces skip connections and semantic modulation modules into the decoder. The output features of each layer of the encoder are modulated by the initial text vector, and the output of each layer of the decoder is modulated using the optimized text. The two are combined through skip connections to achieve effective fusion of low-level texture details and high-level semantic features, thereby improving semantic consistency and fusion quality while ensuring the preservation of image details. Attached Figure Description

[0032] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0033] Figure 1 This is a flowchart illustrating the overall image fusion process of the present invention.

[0034] Figure 2 This is a schematic diagram of the semantic alignment process;

[0035] Figure 3 A schematic diagram to guide the semantic modulation process. Detailed Implementation

[0036] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0038] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0039] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of this disclosure. It should be noted that each block in a flowchart or block diagram may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0040] Example 1:

[0041] like Figure 1 As shown, this application provides a text-guided semantic awareness-based infrared and visible light image fusion method. It first uses infrared and visible light source images as input to an encoder to obtain the image feature representation of this method. A semantic segmentation reference image is used as input to a text prompting mechanism to generate target-level text prompts. The text prompts are processed by a text encoder to obtain text vectors. Text features and image features are then processed by an alignment module to obtain rich feature representations. The initial text features and optimized text features are modulated into image features in a decoder, ultimately generating a fused image. Specifically, it includes:

[0042] Step 1: Extract features from the infrared image and the visible light image separately using an independent image encoder. After obtaining the corresponding infrared image features and visible light image features, combine the two types of features.

[0043] Specifically, a shallow feature extraction is first performed through a 3x3 convolutional layer, followed by a DenseBlock to obtain richer feature representations. Then, pooling operations are used to extract feature information at different scales. The entire process includes four DenseBlocks and pooling operations. Through hierarchical feature fusion, the contextual expressive power of the features is gradually improved.

[0044] Step 2: Based on the semantic segmentation maps of infrared and visible light images, key target categories are identified, and structured text prompts are generated to achieve target-level semantic guidance to assist the image fusion process. The text prompts are feature-encoded using a frozen visual-language cross-modal model text encoder to obtain text feature vectors with semantic representation capabilities. This process achieves accurate mapping from visual information to text semantics, providing high-quality semantic priors for subsequent cross-modal alignment.

[0045] Specifically, the generated text prompts have clear semantic correspondences, which can help the model focus on key target areas and enhance the feature expression capabilities during the fusion process.

[0046] Step 3 involves semantic distillation of the text features, using a self-attention mechanism to retain key semantic information and filter redundant content. Then, the text features are mapped to the visual feature space through cross-modal projection. The filtered redundant text features and the combined image features are input into the visual encoder. During the extraction of image visual patch features, the semantic space region alignment between text features and image features is achieved. The aim is to reduce the difference between image and text modalities after the preparation of image and text features is completed.

[0047] like Figure 2 As shown, the text feature vectors are processed by self-attention ACC(·) and fully connected mapping FC(·) to model the semantic dependencies between words and reduce noise in the text prompts.

[0048] f′ t =FC(FC(Att(f t ))) (1)

[0049] Optimized text feature f′ t The image visual patch features extracted by the CLIP visual encoder are fused together, and this process achieves accurate correspondence and matching of cross-modal semantics through an adapter.

[0050] Step 4: Apply semantic modulation modules to adjust the features in the encoding layer and decoding layer respectively, and then fuse the modulated features through skip connections to obtain a fused image with semantic consistency and high-quality perception effect.

[0051] Specifically, in the image reconstruction decoding part, each decoding layer introduces a semantic modulation module to perform text-guided modulation on the decoding features. The main process is to first fuse the output features of each encoding layer, and then use the semantic modulation module to modulate the fused features with the initial text features to generate feature vectors that conform to the text prompts. At the same time, the optimized text features and the features of the decoding layer are combined with the semantic modulation module to obtain modulated features. Finally, the two modulated features are fused through skip connections to finally output the fused image, which ensures semantic consistency and perceptual quality.

[0052] like Figure 3 As shown, the semantic modulation module first normalizes the text features using softmax to adapt to diverse text information and reduce information loss. Then, the generated text features are input into two multilayer perceptrons (MLPs) to generate channel scaling and channel offset parameters. Affine transformation is then used to dynamically modulate the fused image features, thereby enhancing the semantic consistency of the fused features. The expression is as follows:

[0053]

[0054] Next, the module employs residual connections and the LeakyReLU activation function to prevent overfitting and improve the nonlinear expressive power of the fused image.

[0055]

[0056] Where ⊙ represents the Hadamard product. Indicates fusion features, This module represents the features after semantic modulation. Through affine transformation, it achieves the initial integration of text and image features, laying the foundation for ensuring semantic consistency and achieving effective combination in the future.

[0057] Example 2:

[0058] This embodiment provides an infrared and visible light image fusion system based on text-guided semantic awareness, including:

[0059] The module combines infrared and visible light images by extracting features from them using independent image encoders. After obtaining the corresponding infrared and visible light image features, these two types of features are combined.

[0060] The text encoding module identifies key target categories based on semantic segmentation maps of infrared and visible light images, and generates structured text prompts; it uses a frozen visual-language cross-modal model text encoder to encode the text prompts to obtain text feature vectors with semantic representation capabilities.

[0061] The semantic alignment module performs semantic distillation on text features, retains key semantic information and filters redundant content through a self-attention mechanism, and then maps them to the visual feature space through cross-modal projection. The filtered redundant text features and the combined image features are input into the visual encoder, and semantic space region alignment between text features and image features is achieved during the extraction of image visual patch features.

[0062] The fusion module applies semantic modulation to the features of the encoding layer and the decoding layer respectively, and then fuses the modulated features through skip connections to obtain a fused image with semantic consistency and high-quality perceptual effect.

[0063] Example 3:

[0064] An electronic device includes a memory, a processor, and a computer program stored in the memory and running thereon. When the processor executes the program, it implements the aforementioned text-guided semantic awareness-based infrared and visible light image fusion method, comprising:

[0065] Infrared and visible light images are extracted separately using independent image encoders to obtain the corresponding infrared and visible light image features. These two types of features are then combined.

[0066] Key target categories are identified based on semantic segmentation maps of infrared and visible light images, generating structured text prompts; the text prompts are feature-encoded using a frozen visual-language cross-modal model text encoder to obtain text feature vectors with semantic representation capabilities.

[0067] The text features are semantically distilled, retaining key semantic information and filtering redundant content through a self-attention mechanism. Then, they are mapped to the visual feature space through cross-modal projection. The filtered redundant text features and the combined image features are input into the visual encoder. The semantic space region alignment of text features and image features is achieved during the extraction of image visual patch features.

[0068] Semantic modulation modules are applied to the features of the encoding layer and the decoding layer respectively to adjust the features. Then, the modulated features are fused through skip connections to obtain a fused image with semantic consistency and high-quality perception effect.

[0069] Example 4:

[0070] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned text-guided semantic-aware infrared and visible light image fusion method, comprising:

[0071] Infrared and visible light images are extracted separately using independent image encoders to obtain the corresponding infrared and visible light image features. These two types of features are then combined.

[0072] Key target categories are identified based on semantic segmentation maps of infrared and visible light images, generating structured text prompts; the text prompts are feature-encoded using a frozen visual-language cross-modal model text encoder to obtain text feature vectors with semantic representation capabilities.

[0073] The text features are semantically distilled, retaining key semantic information and filtering redundant content through a self-attention mechanism. Then, they are mapped to the visual feature space through cross-modal projection. The filtered redundant text features and the combined image features are input into the visual encoder. The semantic space region alignment of text features and image features is achieved during the extraction of image visual patch features.

[0074] Semantic modulation modules are applied to the features of the encoding layer and the decoding layer respectively to adjust the features. Then, the modulated features are fused through skip connections to obtain a fused image with semantic consistency and high-quality perception effect.

[0075] Those skilled in the art will understand that the modules or steps described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, which can then be stored in a storage device for execution by a computer device. Alternatively, they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. This disclosure is not limited to any particular combination of hardware and software.

[0076] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0077] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A method for infrared and visible image fusion based on text-guided semantic perception, characterized in that, The method comprises the following steps: The infrared image and the visible light image are respectively subjected to feature extraction by independent image encoders to obtain corresponding infrared image features and visible light image features, and then the two features are combined; The key target categories are identified based on the semantic segmentation maps of the infrared image and the visible light image, and a structured text prompt is generated; the text encoder of the frozen visual-linguistic cross-modal model is used to encode the text prompt to obtain a text feature vector with semantic representation capability; The text features are subjected to semantic distillation processing, key semantic information is retained and redundant content is filtered through a self-attention mechanism, and then the text features are mapped to a visual feature space through cross-modal projection; the filtered text features and the combined image features are jointly input into a visual encoder to realize semantic space area alignment of the text features and the image features in the process of extracting image visual patch features; A semantic modulation module is applied to the encoding layer features and the decoding layer features respectively to adjust the features, and then the adjusted features are fused through a skip connection to obtain a fused image with semantic consistency and high-quality perception effect.

2. The method of claim 1, wherein, The image encoder comprises a convolution layer, Dense-Blocks and a pooling layer; a source image is first subjected to the convolution layer to extract shallow features containing rich texture details, and then subjected to four down-sampling operations for deep feature extraction; In each layer, the Dense-Blocks are used to extract and retain rich feature information, and then the pooling layer is used for down-sampling to enhance the context expression capability of the features.

3. The method of claim 1, wherein, The text prompt is input into a frozen CLIP text encoder, the CLIP text encoder encodes the text prompt into a text feature vector, and the semantic consistency and structural integrity of the text features are ensured.

4. The method of claim 1, wherein, The self-attention mechanism enhances the weights of different key target information in the text feature vector and reduces the interference of redundant information; then the text features are processed through two fully connected layers to reduce information loss and make the vector shape meet the requirements of subsequent calculation.

5. The method of claim 1, wherein, The visual encoder comprises a CLIP image encoder and an adapter, wherein the parameters of the CLIP image encoder remain frozen, only the parameters of the adapter are updated, different text information interacts with the image features in the CLIP image encoder to realize cross-modal semantic alignment.

6. The method of claim 1, wherein, The semantic modulation module comprises a softmax normalization layer, multiple perceptrons, a residual connection and a LeakyReLU activation function; the text features are first subjected to softmax normalization processing to improve the adaptability to diversified text inputs and reduce information loss; then the processed text features are input into two multiple perceptrons in parallel to generate channel scaling parameters and channel offset parameters respectively, dynamic semantic modulation is realized by implementing affine transformation on the fused image features to significantly enhance the semantic consistency of the features; finally, through the synergistic effect of the residual connection and the LeakyReLU activation function, overfitting is effectively prevented, and the nonlinear expression capability of the fused image features is further improved.

7. The method according to claim 6, wherein, The features after semantic modulation are: where denotes the Hadamard product, denotes the fused feature, denotes the feature after semantic modulation, where the channel scaling parameter γ bm and the channel translation parameter β bm are respectively: wherein, denotes a first multilayer perceptron, denotes a second multilayer perceptron, f t denotes a text feature.

8. An infrared and visible image fusion system based on text-guided semantic perception, characterized in that, The method comprises the following steps: The combining module extracts features of the infrared image and the visible light image through independent image encoders respectively to obtain corresponding infrared image features and visible light image features, and then combines the two features; The text encoding module identifies key target categories based on semantic segmentation maps of the infrared image and the visible light image, generates structured text prompts, and encodes the text prompts using a text encoder of a frozen visual-linguistic cross-modal model to obtain text feature vectors with semantic representation capabilities; The semantic alignment module performs semantic distillation processing on the text features, retains key semantic information and filters redundant content through a self-attention mechanism, and then maps the text features to a visual feature space through cross-modal projection; the filtered text features and the combined image features are jointly input into a visual encoder to realize semantic spatial region alignment of the text features and the image features in the process of extracting image visual patch features; The fusion module applies a semantic modulation module to adjust the features in the encoding layer features and the decoding layer features, respectively, and then fuses the adjusted features through a skip connection to obtain a fused image with semantic consistency and high-quality perception effect.

9. An electronic device, comprising: The computer program is stored in the memory and executed by the processor, and the processor implements the infrared and visible light image fusion method based on text-guided semantic perception according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the infrared and visible light image fusion method based on text-guided semantic perception according to any one of claims 1-7.

Images