Prompting display analysis infrared and visible light image fusion method

Abstract

The application belongs to the technical field of deep learning, and discloses an infrared and visible light image fusion method for prompt representation analysis. In a prompt extraction stage, paired infrared and visible light images are input into a shared encoder to preliminarily extract a prompt vector. In a prompt analysis stage, two cosine similarity calculations are used to constrain mutual analysis of content semantic prompts and style semantic prompts. In an image reconstruction stage, a reconstruction loss and a cosine similarity loss in the prompt analysis stage are used to constrain two image reconstructors to respectively reconstruct infrared and visible light images by using content and respective style prompts. Through the above stages, pure content prompts are obtained to guide the image fusion process.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning, and relates to an infrared and visible light image fusion method for prompt representation analysis. BACKGROUND

[0002] Prompt learning is a kind of efficient transfer learning method developed in recent years with the rise of large-scale pre-trained models (PMs). Its core idea is to introduce artificially designed or learnable prompts at the input end to guide large models to better adapt and generalize in downstream tasks. Unlike traditional model fine-tuning methods, prompt learning does not need to update a large number of model parameters, but adjusts the input form or trains a small number of prompt parameters to enable pre-trained models to better complete specific tasks, thereby significantly reducing computational and storage overheads.

[0003] Prompt learning was first concerned in the field of natural language processing (NLP). Pre-trained models represented by BERT and GPT learn rich semantic and grammatical knowledge on a large amount of text data, but how to efficiently transfer this knowledge to downstream tasks becomes a key problem.

[0004] With further research, the concept of prompt learning has been introduced into the field of computer vision and developed into visual prompt learning (VPT). Under this framework, researchers embed learnable prompt vectors in network structures such as visual Transformers to enable models to efficiently adapt to different data distributions in image classification, object detection, and semantic segmentation tasks. Practice shows that this method can achieve results close to or even exceeding full fine-tuning with only a small number of updated parameters, making it particularly suitable for scenarios with limited training samples or multiple tasks. At the same time, prompt learning also shows important value in cross-modal tasks. By constructing a unified prompt space, it can effectively promote feature alignment and information interaction between different modalities, thereby making significant progress in tasks such as image-text retrieval, visual question answering, and image generation. In summary, as a new emerging technology with high parameter efficiency and strong transferability, prompt learning is becoming an important development direction in computer vision and multi-modal tasks, providing a new technical path for image processing, feature fusion, and generative modeling.

[0005] Multimodal image fusion is an important research direction in the field of image processing and computer vision, which aims to effectively integrate multi-source image information from different sources and different perception mechanisms to generate a more comprehensive fusion image in visual effect and semantic expression. Compared with single modal image, the fused image can retain the advantages of multiple sensors, enhancing information expression and robustness in complex environments. Taking infrared image and visible light image as an example, the former can reflect the thermal radiation characteristics of objects and has strong target saliency in night, fog or low light environment, but lacks detailed texture and color information; the latter can provide rich structural details and scene content, but its performance decreases in low light or severe interference conditions. By fusing these two types of images, the clarity of infrared targets and the richness of visible light details can be presented in the same image, thus improving the perception and recognition ability of human observation and machine vision system.

[0006] The research of multimodal image fusion has experienced a development process from traditional methods to deep learning methods. Early methods were mostly based on transform domain and filtering theory, typical representatives including methods based on Laplacian pyramid, wavelet transform and sparse representation. These methods usually decompose images into different scale frequency or structural information, and then reconstruct them through certain fusion rules. Although these methods are simple to implement and can preserve the features of source images to some extent, they often lack modeling of high-level semantic information, leading to deficiencies in detail preservation and overall consistency of the fused image. With the development of computer vision, researchers gradually introduced deep learning into image fusion tasks. Convolutional neural network (CNN) has become the mainstream framework of early deep learning fusion methods due to its powerful feature extraction ability. Through end-to-end training, CNN models can automatically learn fusion rules and achieve good results in preserving details and highlighting targets. However, the convolutional structure has limitations in capturing long-distance dependencies and cross-modal relationships, making the fused image still prone to information loss or distortion in complex scenes.

[0007] In recent years, the emergence of new deep generation techniques such as Generative Adversarial Networks (GAN), Transformer, and Diffusion Model has brought new opportunities for multi-modal image fusion. GAN, through the adversarial training mechanism, enables the generator to generate fusion results close to the distribution of real images, showing advantages in improving image clarity and authenticity. However, due to the instability of adversarial training and mode collapse, GAN has certain limitations in practical applications. The fusion method based on Transformer can better capture the global dependency between cross-modal features through its self-attention mechanism, thus achieving more reasonable feature interaction and fusion in complex scenarios. Diffusion Model, as a representative of the emerging generation model in recent years, has achieved remarkable results in image generation and editing tasks with its gradual denoising generation mechanism and more stable training process. Applying Diffusion Model to multi-modal image fusion can not only improve the authenticity and diversity of the fused image, but also maintain the spatial consistency of different modal images during the generation process.

[0008] "CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion", Z. Zhao et al. uses Transformer and CNN to extract low-frequency global information and high-frequency local information, respectively, and fuses and concatenates them as fusion features during fusion; "Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion", X. Yi et al. uses semantic text guidance for image fusion to achieve degradation-aware and interactive image fusion tasks, but it simply concatenates the features of different modalities after attention calculation and then uses text guidance to modulate the fusion features. The above methods directly fuse or concatenate the features of different modal images and then perform the next operation, ignoring the mutual interference between different modal information, resulting in suboptimal results. SUMMARY

[0009] For the infrared and visible light image fusion task, the object of the present application is to obtain high-quality fused images which are convenient for human observation and can be used in the fields of night monitoring, automatic driving, public safety and military reconnaissance. The current infrared and visible light image fusion model extracts the features of infrared and visible light images respectively in the fusion process, and performs fusion operation at the feature level, and then reconstructs the fused features into a fused image. Since the information from different modalities of visible light and infrared is mixed in the fusion process, the final obtained fused image effect is poor. In order to solve this problem, the present application proposes an infrared and visible light image fusion method based on prompt representation analysis, in the prompt extraction stage, the paired infrared and visible light images are input into the shared encoder to preliminarily extract the prompt vector; in the prompt analysis stage, the content semantic prompt and the style semantic prompt are mutually analyzed by two cosine similarity calculations; in the image reconstruction stage, the two image reconstructors respectively reconstruct the infrared and visible light images by using the content and the respective style prompt by using the reconstruction loss and the cosine similarity loss in the prompt analysis stage to constrain. After the above stages, the pure content prompt is obtained to guide the image fusion process.

[0010] The technical scheme of the present application:

[0011] An infrared and visible light image fusion method based on prompt representation analysis, the steps are as follows:

[0012] (1) Prompt extraction stage:

[0013] Given the paired infrared and visible light data set , wherein , respectively represent the infrared image data set and the visible light image data set, and m represents the number of samples in the data set; a shared encoder is used to extract the prompt vector, and the shared encoder is composed of a series of down-sampling modules; the paired infrared image and the visible light image are input into the shared encoder to preliminarily extract the prompt vector, and the infrared prompt vector and the visible light prompt vector are obtained correspondingly;

[0014] (2) Prompt representation analysis stage:

[0015] The content semantic vector and the style semantic vector in the obtained infrared prompt vector and visible light prompt vector are mutually analyzed: the infrared prompt vector and the visible light prompt vector are respectively divided into content prompt vectors and style prompt vectors, and two cosine similarities are used to constrain the relationship between the two groups of content prompt vectors and the two groups of style prompt vectors, so that the two groups of content prompt vectors are close to each other, and the two groups of style prompt vectors are perpendicular to each other, which is expressed by the formula:

[0016] (1)

[0017] (2)

[0018] (3)

[0019] (4)

[0020] wherein, represents the content hint vector of visible light, represents the content hint vector of infrared, represents the style hint vector of visible light, represents the style hint vector of infrared, represents the norm of the hint vector, represents the calculation of the cosine similarity, represents the calculation of the cosine similarity between the content hint vector of visible light and the content hint vector of infrared, represents the calculation of the cosine similarity between the style hint vector of visible light and the style hint vector of infrared, represents the loss of constraining the hint vectors to be close to each other, represents the loss of constraining the hint vectors to be perpendicular to each other;

[0021] (3) Image reconstruction stage:

[0022] After the cosine similarity constraint, the content semantic vector and the style semantic vector of the infrared hint vector and the visible light hint vector are parsed. In order to ensure the integrity of the semantics contained in the infrared hint vector and the visible light hint vector, two groups of content hint vectors and two groups of style hint vectors are spliced to obtain two groups of complete semantic hint vectors. The two groups of complete hint vectors are respectively sent into the visible light feature decoder and the infrared feature decoder to reconstruct the image; the visible light feature decoder and the infrared feature decoder are both composed of a series of up-sampling modules; the reconstruction loss is used to supervise the reconstructed image, which is represented by the formula:

[0023] (5)

[0024] (6)

[0025] wherein, represents the loss of constraining the visible light reconstructed image, represents the loss of constraining the infrared reconstructed image, represents the number of pixel points contained in the image, represents the pixel point of the visible light image, a pixel point representing the visible light image of the reconstructed image, a pixel point representing the infrared image, a pixel point representing the infrared image of the reconstructed image;

[0026] (4) Image fusion stage:

[0027] First, the paired infrared image and visible light image are respectively input into the infrared image encoder IE2 and the visible light image encoder IE1 to extract infrared features and visible light features Then, the similarity vector of the corresponding features in the same area of the paired infrared image and visible light image is calculated; after normalization, the weight of the content semantics contained in different areas is obtained, denoted as ;

[0028] Using the projection matrix, the infrared features and visible light features are converted into visible light query matrix , infrared query matrix , visible light key matrix , and infrared key matrix ; the conversion formula is as follows:

[0029] (7)

[0030] (8)

[0031] (9)

[0032] (10)

[0033] wherein, represents the projection matrix of the visible light query matrix, represents the projection matrix of the visible light key matrix, represents the projection matrix of the infrared query matrix, represents the projection matrix of the infrared key matrix;

[0034] Using the content hint vector obtained in the hint analysis stage and the weight of the content semantics obtained in the image fusion stage, the value matrix is obtained, which is expressed by the formula as follows:

[0035] (11)

[0036] Through the cross-attention mechanism, the value matrix The cross-attention weight is injected, so that the fusion feature without domain-specific style interference is obtained, which is expressed by formula:

[0037] (12)

[0038] (13)

[0039] wherein, represents the cross-attention feature of visible light, represents the cross-attention feature of infrared, represents a Softmax activation function;

[0040] Subsequently, and are spliced into a feature , which is input into a fusion decoder to obtain a fusion image after training. The loss of the training process of the fusion image is an SSIM loss.

[0041] The infrared and visible light image fusion method of the present application can reduce the influence of modal difference in the image fusion process by mutually analyzing the content semantic hint and the style semantic hint, and obtain a high-quality multi-modal fusion image. The goal of the present application is to obtain a fusion image that not only fits human visual intuitive perception, but also can be efficiently utilized by intelligent visual systems. The technology can be widely applied in night monitoring, automatic driving, public safety and military reconnaissance, and provides key technical support for multi-modal information fusion and intelligent perception systems. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a content and style hint representation analysis architecture.

[0043] Figure 2 is an infrared and visible light image fusion architecture guided by content semantic hints. DETAILED DESCRIPTION

[0044] The specific embodiments of the present application will be further described below in combination with the drawings and technical solutions.

[0045] Figure 1 is a content and style hint representation analysis architecture, which first inputs the paired infrared image and visible light image ( and ) into a shared encoder to preliminarily extract a hint vector, and then mutually analyzes the content hint vector and the style hint vector through formulas (1) and (2), and splices the analyzed content hint vector and the style hint vector to input into respective visible light feature decoders and infrared feature decoder to reconstruct infrared image and visible light image and After the first stage training, the shared encoder has the ability to parse the content semantic cues and style semantic cues, and also obtains the content cue vector needed for subsequent .

[0046] Figure 2 is a content semantic cue guided infrared and visible light image fusion architecture. The paired infrared image and visible light image are input into the infrared image encoder IE2 and the visible light image encoder IE1 respectively to extract the infrared feature and visible light feature , and then calculate the similarity vector of the corresponding features in the same region of the paired infrared image and visible light image. After normalization, the weight of the content semantics contained in different regions is obtained, denoted as . Using the content cue vector and the weight of the content semantics , the value matrix can be obtained by formula (11). Using the projection matrix, the infrared feature and the visible light feature are converted into the visible light query matrix , the infrared query matrix , the visible light key matrix , the infrared key matrix , and the value matrix is calculated by cross attention, as shown in formulas (12) and (13), to obtain the fusion feature and without domain-specific style interference. Then they are spliced into a feature and sent to the fusion decoder to obtain the fusion image . The loss of the training process of the fusion image is the SSIM loss.

Claims

1. A method of fusing infrared and visible images representing a scene analysis, characterized in that, The steps are as follows: (1) prompt extraction stage: Given paired infrared and visible light datasets wherein , respectively represent an infrared image dataset and a visible light image dataset, m represents the number of samples in the dataset; a shared encoder is utilized to extract a hint vector, the shared encoder is composed of a series of down-sampling modules; the paired infrared image and visible light image are input into the shared encoder to preliminarily extract a hint vector, and an infrared hint vector and a visible light hint vector are correspondingly obtained; (2) prompt expression analysis stage: The obtained infrared prompt vector and the content semantic vector and the style semantic vector in the visible light prompt vector are mutually analyzed: the infrared prompt vector and the visible light prompt vector are respectively divided into content prompt vectors and style prompt vectors, and two cosine similarities are used to constrain the relationship between the two groups of content prompt vectors and the two groups of style prompt vectors, so that the two groups of content prompt vectors are close to each other, and the two groups of style prompt vectors are perpendicular to each other; (3) image reconstruction stage: The two groups of content prompt vectors and the two groups of style prompt vectors are spliced to obtain two groups of prompt vectors containing complete semantics, and the two groups of complete prompt vectors are respectively sent into a visible light feature decoder and an infrared feature decoder to reconstruct the image; Visible light feature decoder and infrared feature decoder are both composed of a series of up-sampling modules; a reconstruction loss is used to supervise the reconstructed image; (4) image fusion stage: Firstly, the paired infrared image and the visible light image are respectively input into an infrared image encoder IE2 and a visible light image encoder IE1 to extract infrared features and visible light features and visible light features Then, a similarity vector of the corresponding features in the same region of the paired infrared image and the visible light image is calculated. The similarity vector is normalized to obtain the weight of the content semantics contained in different regions, denoted as ; Using a projection matrix, infrared features and visible light characteristics Convert to visible light query matrix Infrared query matrix Visible light bond matrix Infrared key matrix The conversion formula is as follows: （7） （8） （9） （10） wherein, Pviskeyrepresents a projection matrix of the visible light key matrix, Pviskeyrepresents a projection matrix of the visible light key matrix, Pirkeyrepresents a projection matrix of the infrared key matrix, Pirkeyrepresents a projection matrix of the infrared key matrix; The content hint vector obtained in the hint expression analysis stage is used as a hint The weight of the content semantics obtained in the image fusion stage , and a value matrix is obtained , which is expressed by the formula （11） The value matrix is injected into the cross-attention weights through a cross-attention mechanism, so as to obtain fused features without domain-specific style interference, which is expressed by a formula as follows: The value matrix is injected into the cross-attention weights through a cross-attention mechanism, so as to obtain fused features without domain-specific style interference, which is expressed by a formula as follows: （12） （13） wherein, represents cross-attention features for visible light, represents cross-attention features for infrared, represents a Softmax activation function; Subsequently and Spliced into a feature , into the fusion decoder After training, the fusion image is obtained The loss of the training process of the fusion image is the SSIM loss.

2. The method of claim 1, wherein the prompting the representation resolution of the infrared and visible light image fusion is characterized by, Step (2) is expressed by a formula as follows: （1） （2） （3） （4） wherein, represents a content hint vector of visible light, represents a content hint vector of infrared, represents a style hint vector of visible light, represents a style hint vector of infrared, represents a norm of a hint vector, represents a cosine similarity, represents a cosine similarity between a content hint vector of visible light and a content hint vector of infrared, represents a cosine similarity between a style hint vector of visible light and a style hint vector of infrared, represents a loss that constrains hint vectors to be close to each other, represents a loss that constrains hint vectors to be perpendicular to each other.

3. The method of claim 2, wherein the prompting the representation of the resolved infrared and visible light image fusion is performed by a processor. Step (3) is expressed by a formula as follows: （5） （6） wherein, denotes a loss constraining the visible light reconstructed image, denotes a loss constraining the infrared reconstructed image, denotes the number of pixel points contained in the image, denotes a pixel point of the visible light image, denotes a pixel point of the visible light image of the reconstructed image, denotes a pixel point of the infrared image, denotes a pixel point of the infrared image of the reconstructed image.

Images