Unified multi-mode infrared and visible light image fusion and conversion method

Abstract

The invention discloses a unified multi-mode infrared and visible light image fusion and conversion method in the technical field of computer vision and multi-mode image processing. The unified multi-modal infrared and visible light image fusion and conversion method comprises the following steps: constructing a unified encoder based on a UniIVFT architecture, sharing parameters to process IVF and I2V tasks, and introducing comparison loss to align visible light and infrared image embedding in a multi-modal embedding alignment module; and in the multi-modal feature fusion and decoding module, optimization is carried out through reconstruction loss, and the multi-modal features are fused and decoded into an output image. According to the unified multi-modal infrared and visible light image fusion and conversion method, multi-modal features are aligned by sharing an encoder architecture and comparing loss, efficient infrared-visible light image fusion and conversion are achieved, model redundancy is reduced, and cross-task performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and multimodal image processing. Specifically, it relates to a unified multimodal infrared and visible light image fusion and conversion method. Background Art

[0002] Existing infrared-visible image fusion (IVF) and infrared-to-visible image translation (I2V) methods mainly focus on improving performance by increasing the capacity of feature embedding, and mainly have the following problems:

[0003] 1. High computational resource consumption: Existing models rely on complex feature extractors (such as multi-level CNNs or Transformers) to capture modality-specific and modality-shared features. Often, in order to pursue a small performance improvement, the complexity of the model is greatly increased, and the increased overhead does not match the improved performance, often resulting in reduced utility in practical applications, poor real-time performance, and difficulty in being deployed in actual scenarios;

[0004] 2. Model redundancy: IVF and I2V tasks usually use independent models, without fully considering the feasibility and effectiveness of multitasking. The multimodal embeddings and cross-modal interaction information extracted by complex feature encoders can be used for multiple tasks, resulting in the repeated construction of feature encoders and the inability to achieve collaborative optimization between tasks.

[0005] 3. Insufficient feature alignment: Existing methods do not fully align the feature spaces of infrared and visible light modalities, affecting the fusion and conversion effects.

[0006] Representative techniques include methods based on autoencoders (AEs), generative adversarial networks (GANs), and diffusion models. For example, CDDFuse combines CNN and Transformer to extract multimodal features, but the model is complex; I2VGAN improves the conversion quality by introducing perceptual loss, but still requires an independent architecture to handle different tasks. Although these methods have improved in performance, the model complexity and computational overhead have increased significantly, limiting practical applications. Therefore, it is necessary to design a unified multimodal infrared and visible light image fusion and conversion method. Summary of the Invention

[0007] To solve the above problems, the object of the present invention is to provide a unified multimodal infrared and visible light image fusion and conversion method, which solves the technical problems of large computational resource consumption, model redundancy, and insufficient feature alignment in the existing infrared-visible light image fusion (IVF) and infrared-to-visible light image conversion method (I2V). A lightweight and unified multi-task framework (UniIVFT) is provided to align multimodal features through a shared encoder architecture and contrastive loss, realizing efficient infrared-visible light image fusion and conversion, reducing model redundancy, and improving cross-task performance.

[0008] To achieve the above object, the technical solution of the present invention is as follows:

[0009] A unified multimodal infrared and visible light image fusion and conversion method includes the following steps:

[0010] S1. Construct a unified encoder based on the UniIVFT architecture, share parameters to process the infrared-visible light fusion (IVF) and infrared-to-visible light conversion (I2V) tasks, and use specific task tags to control the model output and dynamically switch the task mode;

[0011] S2. In the multimodal embedding alignment module, use independent single-modal encoders to extract the embedding features of infrared and visible light images, and introduce contrastive loss to align the visible light and infrared image embeddings;

[0012] S3. In the multimodal feature fusion and decoding module, optimize through reconstruction loss, fuse the multimodal features, and decode them into output images.

[0013] Furthermore, the UniIVFT architecture is a unified framework for the IVF and I2V tasks, including two components: the multimodal embedding alignment module and the multimodal feature fusion and decoding module; the multimodal embedding alignment module includes a visible light feature encoder and an infrared feature encoder, and the multimodal feature fusion and decoding module includes a feature fusion task encoder, an image fusion task decoder, and an image conversion task decoder.

[0014] Furthermore, the S2 includes the following steps:

[0015] S21. The input visible light image V is encoded into an embedding sequence E by the visible light feature encoder V :{v cls ,v1,…,v N};

[0016] S22. The infrared feature encoder converts the input infrared image I into an embedding sequence E I :{w cls ,w1,…,w N},

[0017] Among them, v cls and w cls are the embeddings of the [CLS] tokens from two single-modal encoders for infrared and visible light;

[0018] S23. Use contrastive loss to optimize the model to align the feature spaces of the two single-modalities. The contrastive loss is calculated as follows:

[0019]

[0020] Among them, s = g v (v cls ) T g w (w cls ) is a similarity function, and g v and g w are two linear layers that map v cls and w cls to low-dimensional representations respectively, and τ is a learnable temperature parameter.

[0021] Furthermore, in the multi-modal embedding alignment module for S23, for each infrared image and visible light image, the contrastive losses of infrared-visible light and visible light-infrared are calculated. Among them, the calculation method of the contrastive loss of infrared-visible light is the same as that of the contrastive loss of visible light-infrared .

[0022] Furthermore, S3 includes the following steps:

[0023] S31. The two tasks of UniIVFT share a task encoder and some parameters with the infrared encoder;

[0024] S32. In the multi-modal feature fusion and decoding module, the visible light image embedding and the infrared image embedding are both input into the task encoder, and the fused features are used to obtain the reconstructed image through the functions of the image fusion task decoder and the image conversion task decoder.

[0025] Furthermore, S31 includes the following steps:

[0026] S311. The infrared feature encoder and the feature fusion encoder share the self-attention SA layer parameters and the feed-forward FF network parameters, and an additional cross-attention CA layer is inserted between the SA layer and the FF layer of each transformation block;

[0027] S312. The feature fusion encoder uses the cross-attention mechanism to learn the fused embedding representation from the infrared image and the visible light image.

[0028] Further, S32 includes the following steps:

[0029] S321. The fused features are input into the image fusion task decoder and the image conversion task decoder according to the specific task tags [FUS] fusion or [TRANS] transformation;

[0030] S322. The above specific task tags can be used as a control mechanism to guide the model to fuse the input and switch between different tasks. During the training process, each task is associated with its respective image decoder, and the decoder is co-trained with the encoder;

[0031] S323. In the image fusion task decoder and the image conversion task decoder, the model is optimized using the reconstruction loss to generate the fused or transformed image.

[0032] Further, S323 includes the following steps:

[0033] S3231. In the fusion task, set the reconstruction loss as the sum of the intensity loss and the texture loss, where the intensity loss is used to constrain the overall visual intensity of the fusion result, while the texture loss constrains the fused image to contain as much texture detail information as possible:

[0034]

[0035] where, represents the Sobel gradient operator;

[0036] S3232. In the conversion task, select the structural similarity index SSIM loss as SSIM provides a method for evaluating image quality that is more in line with human visual perception. It focuses on features such as the brightness, contrast, and structure of the image:

[0037]

[0038] S3233. The loss of the entire model is the sum of the above losses, that is and the sum of.

[0039] Adopting the above technical solutions, the present invention has the following advantages:

[0040] The present invention provides a unified multi-modal infrared and visible light image fusion and conversion method. Based on the multi-task framework UniIVFT architecture, it maintains a lightweight architecture with fewer parameters than the prior art. By sharing an encoder and using contrastive loss to align multi-modal features, it realizes lightweight and high-efficiency infrared-visible light image fusion (IVF) and infrared-to-visible light image conversion (I2V). These two tasks share a unified encoder architecture and use task-specific tags to control the model output, thus effectively reducing the construction and training of redundant models, significantly reducing the number of parameters, and improving the inference speed. The IVF and I2V tasks promote each other through contrastive loss and shared features, and their performance is better than that of independent models, so as to effectively adjust the infrared and visible light image features before multi-modal encoding. This alignment enhances the encoder's ability to capture cross-modal interactions, which is beneficial for the contrastive loss of the IVF and I2V tasks to effectively align multi-modal features, enhancing the model's adaptability to low-quality visible light images and infrared images, and achieving the same or better performance than the prior art in multiple tasks. Brief Description of the Drawings

[0041] Figure 1 It is a schematic diagram of the UniIVFT architecture of the present invention;

[0042] Figure 2 It is a flowchart of the unified multi-modal infrared and visible light image fusion and conversion method of the present invention. Detailed Embodiments

[0043] The technical solutions of the present invention will be specifically described below in conjunction with the drawings of the specification. It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.

[0044] Figure 1 It shows a schematic diagram of the UniIVFT architecture of the present invention; the UniIVFT architecture is a unified framework for the IVF and I2V tasks, and it consists of two main components as Figure 1As shown in the figure, it includes a multi-modal embedding alignment module and a multi-modal feature fusion and decoding module. Among them, the multi-modal embedding alignment module includes a visible light feature encoder and an infrared feature encoder, and the multi-modal feature fusion and decoding module includes a feature fusion task encoder, an image fusion task decoder, and an image conversion task decoder. In the multi-modal embedding alignment module, the single-modal encoder uses contrastive loss to align the visible light and infrared image embeddings. In the multi-modal feature fusion and decoding module, it is optimized through reconstruction loss, and the multi-modal features are fused and decoded into the output image. This model takes a pair of visible light and infrared images {V, I} as input. Assuming the output fused and converted images are and In the multi-modal embedding alignment module, the infrared image and the visible light image are independently processed by the single-modal encoder to extract their embeddings. Before fusion, contrastive loss is adopted between the [CLS] token embeddings of the two modalities to align their feature spaces. In the multi-modal feature fusion and decoding module, the respective features will learn the fused features through a feature fusion encoder, which shares parameters with the infrared encoder. The fused features are input into the corresponding decoder according to the specific task tokens [FUS] (fusion) or [TRANS] (transformation), and the model is optimized using reconstruction loss in the decoder to generate the fused or converted image.

[0045] Figure 2 The figure shows the flowchart of the unified multi-modal infrared and visible light image fusion and conversion method of the present invention. The flowchart of a unified multi-modal infrared and visible light image fusion and conversion method is specifically as Figure 2 shown, including the following steps:

[0046] S1. Construct a unified encoder based on the UniIVFT architecture, share parameters to process the infrared-visible light fusion IVF and infrared-to-visible light conversion I2V tasks, and use specific task tokens to control the model output and dynamically switch the task mode;

[0047] S2. In the multi-modal embedding alignment module, use independent single-modal encoders to extract the embedding features of infrared and visible light images, and introduce contrastive loss to align the visible light and infrared image embeddings;

[0048] UniIVFT introduces contrastive loss to activate the visible light and infrared encoders, so as to align the multi-modal embeddings before fusion. Under special conditions such as low light environments, the quality of some visible light images will significantly decrease. In addition, infrared images lack color and texture information. If the image embeddings obtained by the two single-modal encoders are directly fused in terms of features, these embeddings are not aligned, and the model will not be able to obtain better results. Adding contrastive loss to align the modal features can further improve the model's representation learning performance.

[0049] S2 includes the following specific steps:

[0050] S21. The input visible light image V is encoded by the visible light feature encoder into an embedding sequence E V :{v cls ,v1,…,v N};

[0051] S22. The infrared feature encoder converts the input infrared image I into an embedding sequence E I :{w cls ,w1,…,w N},

[0052] where, v cls and w cls are the embeddings of the [CLS] tokens from the infrared and visible light single-modal encoders;

[0053] S23. Use the contrastive loss to optimize the model to align the two single-modal feature spaces. The contrastive loss is calculated as follows:

[0054]

[0055] where, s = g v (v cls ) T g w (w cls ) is a similarity function, and g v and g w are two linear layers that map v cls and w cls to a low-dimensional (256d) representation respectively, and τ is a learnable temperature parameter.

[0056] In addition, in the multi-modal embedding alignment module, for each infrared image and visible light image, the infrared-visible light and visible light-infrared contrastive losses are calculated. Among them, the calculation method of the infrared-visible light contrastive loss is the same as that of the visible light-infrared contrastive loss .

[0057] S3. In the multi-modal feature fusion and decoding module, it is optimized through the reconstruction loss, and the multi-modal features are fused and decoded into an output image.

[0058] S3 includes the following specific steps:

[0059] S31. To effectively unify multi-task training while building a lightweight model, the two tasks of UniIVFT share a task encoder and some parameters with the infrared encoder;

[0060] S31 includes the following specific steps:

[0061] S311. The infrared feature encoder and the feature fusion encoder share the self-attention SA layer parameters and the feed-forward FF network parameters, and an additional cross-attention CA layer is inserted between the SA layer and the FF layer of each transformation block;

[0062] S312. The feature fusion encoder uses the cross-attention mechanism to learn the fused embedding representation from the infrared image and the visible light image.

[0063] S32. In the multi-modal feature fusion and decoding module, both the visible light image embedding and the infrared image embedding are input into the task encoder, and the fused features are used to obtain the reconstructed image through the functions of the image fusion task decoder and the image transformation task decoder.

[0064] Among them, S32 includes the following specific steps:

[0065] S321. The fused features are input into the image fusion task decoder and the image transformation task decoder according to the specific task tokens [FUS] for fusion or [TRANS] for transformation. In this method, fusion and transformation are achieved through specific task tokens.

[0066] S322. The above specific task tokens can be used as a control mechanism to guide the model to fuse the inputs and switch between different tasks. During the training process, each task is associated with its respective image decoder, and the decoder is co-trained with the encoder;

[0067] S323. In the image fusion task decoder and the image transformation task decoder, the model is optimized using the reconstruction loss to generate the fused or transformed image, and the optimization of both the encoder and the decoder is achieved through the reconstruction loss of specific tasks.

[0068] S323 includes the following steps:

[0069] S3231. In the fusion task, set the reconstruction loss as the sum of the intensity loss and the texture loss, where the intensity loss is used to constrain the overall visual intensity of the fusion result, while the texture loss constrains the fused image to contain as much texture detail information as possible:

[0070]

[0071] Among them, represents the Sobel gradient operator;

[0072] S3232. In the transformation task, select the structural similarity index SSIM loss as SSIM provides a method for evaluating image quality that is more in line with human visual perception. It focuses on features such as the brightness, contrast, and structure of the image:

[0073]

[0074] S3233. These losses above can ensure that the model effectively learns and generates fused or transformed images. The loss of the entire model is the sum of the above losses, that is and the sum of.

[0075] Finally, it should be noted that although the present invention has been described with reference to the current specific embodiments, those of ordinary skill in the art should recognize that the above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Various equivalent changes or substitutions can be made without departing from the spirit of the present invention. Therefore, as long as the changes and variations of the above embodiments are within the scope of the spirit of the present invention, they will fall within the scope of the claims of the present invention.

Claims

1. A unified multimodal infrared and visible light image fusion and conversion method, characterized in that, It includes the following steps: S1. Build a unified encoder based on the UniIVFT architecture, share parameters to process the infrared-visible light fusion IVF and infrared-to-visible light conversion I2V tasks, and use specific task tags to control the model output and dynamically switch the task mode; S2. In the multimodal embedding alignment module, use independent single-modal encoders to extract the embedding features of infrared and visible light images, and introduce a contrastive loss to align the visible light and infrared image embeddings; S3. In the multimodal feature fusion and decoding module, optimize through a reconstruction loss, fuse the multimodal features and decode them into an output image.

2. A unified multimodal infrared and visible light image fusion and conversion method according to claim 1, characterized in that, The UniIVFT architecture is a unified framework for the IVF and I2V tasks, including two components: the multimodal embedding alignment module and the multimodal feature fusion and decoding module; The multimodal embedding alignment module includes a visible light feature encoder and an infrared feature encoder, and the multimodal feature fusion and decoding module includes a feature fusion task encoder, an image fusion task decoder, and an image conversion task decoder.

3. A unified multi-modal infrared and visible light image fusion and conversion method according to claim 2, characterized in that, The S2 includes the following steps: S21. The input visible light image V is encoded by a visible light feature encoder into an embedding sequence E V :{v cls , v1, …, v N}; S22. The infrared feature encoder converts the input infrared image I into an embedding sequence E I :{w cls , w1, …, w N}, where v cls and w cls are the embeddings of the [CLS] tokens from two unimodal encoders for infrared and visible light; S23. Use contrastive loss Optimize the model to align the feature spaces of the two unimodals, and the contrastive loss The calculation formula is as follows: where s = g v (v cls ) T g w (w cls ) is a similarity function, g v and g w are two linear layers that map v cls and w cls to low-dimensional representations, and τ is a learnable temperature parameter.

4. A unified multi-modal infrared and visible light image fusion and conversion method according to claim 3, characterized in that In the multi-modal embedding alignment module in S23, for each infrared image and visible light image, the contrast loss between infrared and visible light and the contrast loss between visible light and infrared need to be calculated. Among them, the contrast loss between infrared and visible light is calculated in the same way as the contrast loss between visible light and infrared is calculated in the same way 5. A unified multimodal infrared and visible light image fusion and conversion method according to claim 4, characterized in that, The S3 includes the following steps: S31. The two tasks of UniIVFT share a task encoder and some parameters with the infrared encoder; S32. In the multimodal feature fusion and decoding module, the visible light image embedding and the infrared image embedding are both input into the task encoder, and the fused features are used to obtain the reconstructed image through the functions of the image fusion task decoder and the image conversion task decoder.

6. A unified multimodal infrared and visible light image fusion and conversion method according to claim 5, characterized in that The S31 includes the following steps: S311. The infrared feature encoder and the feature fusion encoder share the self-attention SA layer parameters and the feed-forward FF network parameters, and an additional cross-attention CA layer is inserted between the SA layer and the FF layer of each transformation block; S312. The feature fusion encoder uses the cross-attention mechanism to learn the fused embedding representation from the infrared image and the visible light image.

7. A unified multimodal infrared and visible light image fusion and conversion method according to claim 5, characterized in that The S32 includes the following steps: S321. The fused features are input into the image fusion task decoder and the image conversion task decoder according to the specific task tags [FUS] for fusion or [TRANS] for conversion; S322. The above specific task tags can be used as a control mechanism to guide the model to fuse the inputs and switch between different tasks. During the training process, each task is associated with its respective image decoder, and the decoder is co-trained with the encoder; S323. In the image fusion task decoder and the image conversion task decoder, use the reconstruction loss to optimize the model to generate the fused or converted image.

8. A unified multi-modal infrared and visible light image fusion and conversion method according to claim 7, characterized in that The S323 includes the following steps: S3231. In the fusion task, set the reconstruction loss as the sum of the intensity loss and the texture loss, where the intensity loss is used to constrain the overall visual intensity of the fusion result, while the texture loss constrains the fused image to contain as much texture detail information as possible: Among them, represents the Sobel gradient operator; S3232. In the conversion task, the structural similarity index SSIM loss is selected as SSIM provides a method for evaluating image quality that is more in line with human visual perception. It focuses on features such as the brightness, contrast, and structure of the image: The loss of the entire model is the sum of the above losses, i.e., and of the sum.

Images