Self-supervised dual-loop infrared and visible image fusion method

Abstract

The application belongs to the technical field of deep learning, and discloses a self-supervised double-cycle infrared and visible light image fusion method, which comprises the following steps: an infrared mode and a visible light mode encoder pre-training stage; a self-supervised double-cycle training stage; and a fused image generation stage.The training framework of the self-supervised double cycle solves the problem that there is no true value in infrared and visible light image fusion and only artificial loss functions can be relied on, without the need to construct fusion labels, manually design gradient loss, structural loss, information entropy loss and other complex constraints, and from the root, the defects of texture smoothing, target distortion, modal information imbalance and the like caused by artificial loss are avoided.The application can be used as an efficient and stable training method to provide high-quality fused data for downstream multi-modal visual tasks.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology and relates to a self-supervised dual-loop infrared and visible light image fusion method. Background Technology

[0002] Currently, the technologies related to this patent include two aspects: the first is infrared and visible light image fusion technology; the second is self-supervised learning and iterative training technology.

[0003] Significant progress has been made in infrared and visible light image fusion technology in recent years, with applications spanning multiple fields such as night vision enhancement, target recognition, privacy protection, and remote sensing. With the continuous evolution of deep learning methods, an increasing number of fusion models have been proposed to improve the detail retention and information richness of the fusion results. Currently, the main image fusion models can be divided into three categories: methods based on traditional signal transformation, fusion methods based on generative adversarial networks (GANs), and end-to-end fusion methods based on encoder-decoder architectures. These methods each have their own characteristics in network structure, feature interaction methods, and loss function design, driving the rapid development of image fusion technology and making it a hot topic in academic and industrial research and applications.

[0004] Among these, encoder-decoder-based fusion frameworks have become mainstream due to their stable training and high-quality output. These frameworks typically use dual encoders to extract infrared and visible light features separately, which are then processed by a fusion module before being output by a decoder as a fused image. To ensure fusion quality, existing methods generally employ explicit fusion losses, including gradient loss, structural similarity loss, information entropy loss, contrast loss, and edge loss. However, explicit losses suffer from drawbacks such as design complexity, hyperparameter sensitivity, and susceptibility to texture smoothing, target blurring, and modal information imbalance. Furthermore, they heavily rely on manual prior knowledge.

[0005] Self-supervised learning utilizes the inherent structure of data to construct supervisory signals, achieving efficient feature learning without manual annotation. It is widely used in tasks such as representation learning, image reconstruction, and cross-modal alignment. The cyclic training mechanism, by alternately updating different network branches, achieves multi-module collaborative optimization, effectively improving training stability and model performance. In this invention, the advantages of self-supervised learning and dual-cycle training are fully utilized, completely abandoning explicit fusion loss and relying solely on source modality encoder features to constrain the fusion features, achieving unsupervised, highly stable infrared and visible light image fusion.

[0006] Traditional infrared and visible light fusion methods have significant limitations: most models rely on explicit fusion loss to drive training, and without loss, fusion quality cannot be guaranteed; the lack of self-supervised constraints using encoder features leads to a disconnect between fused features and source modal features, easily resulting in the loss of infrared thermal information or visible light texture information; single-step training cannot achieve collaborative optimization between the constraint module and the fusion module, resulting in slow model convergence and weak generalization ability. To address these issues, this invention proposes a self-supervised dual-loop infrared and visible light image fusion method. Under conditions without explicit fusion loss, it achieves lossless preservation and high-quality fusion of dual modal information through encoder feature constraints and internal and external dual-loop training. Summary of the Invention

[0007] For infrared and visible light image fusion tasks, the goal of this invention is to achieve high-quality image fusion without explicit fusion loss, thus enabling the invention to serve as an efficient feature enhancement method and provide high-quality data for downstream multimodal vision tasks. Current infrared and visible light fusion models heavily rely on manually designed explicit fusion losses, lack self-supervised constraints on source modal features, and most models employ single-loop training, failing to achieve collaborative optimization between constraint branches and fusion branches. This leads to problems such as texture loss, blurred thermal targets, and modal information imbalance in the fused image. To address this issue, this invention proposes a self-supervised dual-loop infrared and visible light image fusion method. During training, it does not apply any explicit fusion loss to the fused image, but only utilizes the inherent features output by the infrared encoder and visible light encoder to apply bidirectional self-supervised constraints to the fusion features. Self-supervised collaborative optimization is achieved through updating the constraint module in the inner loop and the fusion module in the outer loop, thereby generating a fused image with complete information and rich details. Using the generated high-quality fused image for downstream task training can effectively improve the performance of downstream tasks.

[0008] The technical solution of the present invention: A self-supervised dual-loop infrared and visible light image fusion method, comprising the following steps: (1) Pre-training stage of infrared and visible light mode encoders; Given a paired dataset D=[D] of infrared and visible light modes ir D vis ], D ir =[I ir 1 ,I ir 2, ...,I ir m ]、D VIS =[I vis 1 ,I vis 2 ,...,Ivis m ] represents the infrared modality dataset and the visible light modality dataset, respectively, and m represents the number of paired samples in the dataset; the infrared modality dataset and the visible light modality dataset contain pixel-level paired infrared images and visible light images; ResNet-18 is used as the infrared encoder and the visible light encoder to extract multi-level features of the infrared images and the visible light images, respectively; the structure of the infrared modality encoder and the visible light modality encoder is fixed, and pre-training is completed in a self-supervised manner; 1.1) ResNet-18 was selected as the infrared encoder E ir With visible light encoder E vis These are used to extract infrared modal features and visible light modal features, respectively; Infrared encoder E ir Shallow detail features F of output infrared mode ir 1 Mid-level semantic features F ir 2 Deep global features F ir 3 Visible light encoder E vis Output shallow detail features F of the visible light mode vis 1 Mid-level semantic features F vis 2 Deep global features F vis 3 ; Specifically: [F ir 1 ,F ir 2 ,F ir 3 ]=E ir (I ir ) [F vis 1 ,F vis 2 ,F vis 3 ]=E vis (I vis ) Among them, E ir (I ir ) represents the infrared image I ir Feature encoding results; E vis (I vis ) represents the visible light image I vis The feature encoding results; 1.2) Self-supervised pre-training is adopted, and only the parameters of the infrared encoder and the visible light encoder are optimized. The training objective is to improve the consistency of representation and reconstruction stability of infrared modal features and visible light modal features. After pre-training is completed, all parameters of the infrared encoder and the visible light encoder are frozen. (2) Self-supervised double-cycle training phase; The self-supervised dual-loop training phase employs a fusion network, which includes a constraint module, a fusion module, a reconstruction module, and a decoder. The self-supervised dual-loop training phase uses an alternating inner-outer loop training mechanism, relying solely on infrared modal features and visible light modal features to perform self-supervised constraints on the fusion module. 2.1) Construction of constraint modules; The constraint module is divided into infrared constraint module C. ir With visible light confinement module C vis This is used to map the three-layer fused features of infrared and visible modes back to the feature spaces of infrared and visible modes, thereby restoring the features of infrared and visible modes; Infrared constraint module C ir With visible light confinement module C vis To ensure that the fused features include complete infrared and visible light modal features, specifically: F' ir i =C ir (Φ ir i ) F' vis i =C vis (Φ vis i (i=1,2,3) Where, Φ ir i The infrared fusion feature output by the fusion module; Φ vis i The visible light fusion feature output by the fusion module; F' ir i The infrared modal characteristics reconstructed by the infrared constraint module; F' vis The visible light mode features reconstructed by the visible light constraint module; the constraint target is: F' vis i =F vis i , F' ir i =F ir i Where i = 1, 2, 3; 2.2) Construction of the fusion module; The fusion module structure consists of three layers, comprising bidirectional cross-complementary attention (BCCA), and deep global features F of the infrared modality after feature stitching. ir 3 Deep global features F of visible light modes vis 3 The input to the first-layer fusion module F1 is fused into the first-layer infrared fusion feature Φ. ir 1 Φ and the first layer of visible light fusion feature vis 1 The first layer of infrared fusion feature Φ ir 1 Φ and the first layer of visible light fusion feature vis 1 The second infrared fusion feature Φ is then obtained through the second fusion module F2. ir 2 Second layer visible light fusion feature Φ vis 2 Second-layer infrared fusion feature Φ ir 2 Second layer visible light fusion feature Φ vis 2 The final fused feature Φ is obtained after passing through the third fusion module F3. fuse Specifically: [Φ ir 1 ,Φ vis 1 ]=F1[C(F ir 3 ,F vis 3 )] [Φ ir 2 ,Φ vis 2 ]=F2(Φ ir 1 ,Φ vis 1 ) Φ fuse =F3(Φ ir 2 ,Φ vis 2 ) Where C represents the feature concatenation operation; 2.3) Construction of the reconstruction module and decoder; The reconstruction module includes the infrared reconstruction module R. ir and visible light reconstruction module R vis The final fusion feature Φ fuse After passing through the infrared reconstruction module R ir and visible light reconstruction module Rvis The corresponding infrared reconstruction information θ is obtained. ir and visible light reconstruction information θ vis Infrared reconstruction information θ ir Visible light reconstruction information θ vis and the final fusion feature Φ fuse Then, the image is fed into decoder D to obtain the fused image I. fuse Infrared reconstructed image I ir 'and visible light reconstructed image I vis ', specifically: θ ir =R ir (Φ) fuse ) θ vis =R vis (Φ) fuse ) [I fuse, I ir ',I vis ']=D(Φ fuse ,θ ir ,θ vis ) 2.4) Internal and external dual-cycle training mechanism; Inner loop: Update constraint module; With fixed parameters for the fusion module, infrared encoder, and visible light encoder, only the infrared constraint module C is trained. ir and visible light confinement module C vis The loss function is the feature recovery loss Loss1, which measures the loss from the infrared constraint module C. ir With visible light confinement module C vis The restored infrared modal characteristics and visible light modal characteristics are compared with those of the infrared encoder E. ir With visible light encoder E vis The differences between the extracted infrared modal features and visible light modal features are as follows: Loss1=MSE(F' ir i ,F ir i )+MSE(F' vis i ,F vis i ) The goal of the inner loop is to enable the infrared constraint module and the visible light constraint module to accurately restore the infrared mode features and visible light mode features from their respective fused features; Outer loop: Update the fusion module; The parameters of the infrared constraint module, the visible light constraint module, the infrared encoder, and the visible light encoder are fixed, and only the three-layer fusion module is trained; the loss functions are feature restoration loss Loss1 and reconstruction loss Loss2, and the fusion module is optimized in reverse, specifically as follows: Loss2=MSE(I ir ,I ir ')+MSE(I vis ,I vis ') The outer loop objective is to ensure that the fusion features output by the fusion module can be restored by the infrared constraint module and the visible light constraint module, and to reconstruct images of both modes, ensuring that information of the infrared mode and the visible light mode is not lost. Throughout the training process, the inner and outer loops are executed alternately until the three-layer fusion module converges; (3) Image fusion generation stage; After the fusion network is trained, a pair of pixel-level paired infrared images I are input. ir With visible light image I vis Infrared and visible light modal features are extracted by infrared and visible light encoders, fusion modules generate fused features, and decoders map the features back to the image space, ultimately outputting a lossless, high-fidelity fused image I. fuse .

[0009] The beneficial effects of this invention are as follows: The self-supervised dual-loop training framework of this invention solves the problem in infrared and visible light image fusion where there is no ground truth and one can only rely on a manually defined loss function. It eliminates the need to construct fusion labels and manually design complex constraints such as gradient loss, structural loss, and information entropy loss, thus fundamentally avoiding the defects of texture smoothing, target distortion, and modal information imbalance caused by manual loss. This invention can serve as an efficient and stable training method, providing high-quality fusion data for downstream multimodal vision tasks (object detection, image segmentation, scene understanding, etc.). Attached Figure Description

[0010] Figure 1 It is an infrared and visible light fusion framework.

[0011] Figure 2 It is a self-supervised double-loop training framework. Detailed Implementation

[0012] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0013] like Figure 1 As shown, infrared image I ir and visible light image I vis They passed through infrared encoder E respectively ir With visible light encoder Evis The shallow detail features F of the infrared mode were obtained respectively. ir 1 Mid-level semantic features F ir 2 Deep global features F ir 3 Shallow detail features F of visible light modes vis 1 Mid-level semantic features F vis 2 Deep global features F vis 3 Then the deep global features F are finally extracted. ir 3 and deep global features F vis 3 The components are then fused together in three layers of fusion modules F1, F2, and F3 to obtain fusion characteristics at different stages; the infrared constraint module C is then used for evaluation. ir and visible light confinement module C vis The infrared and visible light modal features are extracted from the three-layer fusion features. The differences between these features and the infrared and visible light modal features in the infrared encoder and visible light encoder are used to implement the first layer of constraints. The final fusion module outputs the final fusion feature Φ. fuse After infrared reconstruction module R ir and visible light reconstruction module R vis The infrared reconstruction information θ is obtained by restoration ir Visible light reconstruction information θ vis Infrared reconstruction information θ ir Visible light reconstruction information θ vis and the final fusion feature Φ fuse Together input decoder generates fused image I fuse Infrared reconstructed image I ir 'and visible light reconstructed image I vis '. Input infrared image I ir and visible light image I vis Infrared reconstructed image I generated by the decoder ir 'and visible light reconstructed image I vis Implement the second layer of constraints to ensure that the final fused features contain the complete infrared image I. ir and visible light image I vis This allows the fusion module to fully integrate all the information of the image, thus enabling the fusion module to fully integrate the features.

[0014] like Figure 2As shown, to achieve precise constraints on the fusion module, a self-supervised double-loop training mechanism is needed. The inner loop updates the two constraint modules. The dataset is divided into a training set and a test set. First, the data in the training set is used to temporarily update the fusion module F'. Then, a set of data from the test set is used to input F' to test the update effect of the two constraint modules on the fusion module. The two constraint modules are updated using Loss1. The inner loop updates the constraint modules to the ideal effect. Then, the outer loop uses Loss1 and Loss2 simultaneously to implement the formal update of the fusion module. The alternating updates of the inner and outer loops ultimately ensure that the entire fusion network achieves the ideal effect.

Claims

1. A self-supervised dual-loop infrared and visible light image fusion method, characterized in that, The steps are as follows: (1) Pre-training stage of infrared and visible light mode encoders; Given a paired dataset D=[D] of infrared and visible light modes ir D vis ], D ir =[I ir 1 ,I ir 2, ...,I ir m ]、D VIS =[I vis 1 ,I vis 2 ,...,I vis m ] represent the infrared modality dataset and the visible light modality dataset, respectively, and m represents the number of paired samples in the dataset; the infrared modality dataset and the visible light modality dataset contain pixel-level paired infrared images and visible light images; ResNet-18 is used as the infrared encoder and the visible light encoder to extract multi-level features of the infrared images and the visible light images, respectively; The structures of the fixed infrared mode encoder and the visible light mode encoder are used to complete the pre-training in a self-supervised manner; 1.1) ResNet-18 was selected as the infrared encoder E ir With visible light encoder E vis These are used to extract infrared modal features and visible light modal features, respectively; Infrared encoder E ir Shallow detail features F of output infrared mode ir 1 Mid-level semantic features F ir 2 Deep global features F ir 3 Visible light encoder E vis Output shallow detail features F of the visible light mode vis 1 Mid-level semantic features F vis 2 Deep global features F vis 3 ; Specifically: [F ir 1 ,F ir 2 ,F ir 3 ]=E ir (I ir ) [F vis 1 ,F vis 2 ,F vis 3 ]=E vis (I vis ) Among them, E ir (I ir ) represents the infrared image I ir Feature encoding results; E vis (I vis ) represents the visible light image I vis The feature encoding results; 1.2) Self-supervised pre-training is adopted, and only the parameters of the infrared encoder and the visible light encoder are optimized. The training objective is to improve the consistency of representation and reconstruction stability of infrared modal features and visible light modal features. After pre-training is completed, all parameters of the infrared encoder and the visible light encoder are frozen. (2) Self-supervised double-cycle training phase; The self-supervised dual-loop training phase employs a fusion network, which includes a constraint module, a fusion module, a reconstruction module, and a decoder. The self-supervised dual-loop training phase uses an alternating inner-outer loop training mechanism, relying solely on infrared modal features and visible light modal features to perform self-supervised constraints on the fusion module. 2.1) Construction of constraint modules; The constraint module is divided into infrared constraint module C. ir With visible light confinement module C vis This is used to map the three-layer fused features of infrared and visible modes back to the feature spaces of infrared and visible modes, thereby restoring the features of infrared and visible modes; Infrared constraint module C ir With visible light confinement module C vis To ensure that the fused features contain complete infrared and visible light modal features, specifically: F' ir i =C ir (F ir i ) F' vis i =C vis (Φ vis i ) (i=1,2,3) Where, Φ ir i The infrared fusion feature output by the fusion module; Φ vis i The visible light fusion feature output by the fusion module; F' ir i The infrared modal characteristics reconstructed by the infrared constraint module; F' vis The visible light mode features reconstructed by the visible light constraint module; the constraint target is: F' vis i =F vis i , F' ir i =F ir i Where i = 1, 2, 3; 2.2) Construction of the fusion module; The fusion module structure consists of three layers, comprising bidirectional cross-complementary attention, and deep global features F of the infrared modality after feature stitching. ir 3 Deep global features F of visible light modes vis 3 The input to the first-layer fusion module F1 is fused into the first-layer infrared fusion feature Φ. ir 1 Φ and the first layer of visible light fusion feature vis 1 The first layer of infrared fusion feature Φ ir 1 Φ and the first layer of visible light fusion feature vis 1 The second infrared fusion feature Φ is then obtained through the second fusion module F2. ir 2 Second layer visible light fusion feature Φ vis 2 Second-layer infrared fusion feature Φ ir 2 Second layer visible light fusion feature Φ vis 2 The final fused feature Φ is obtained after passing through the third fusion module F3. fuse Specifically: [Φ ir 1 ,Φ vis 1 ]=F1[C(F ir 3 ,F vis 3 )] [F ir 2 ,F vis 2 ]=F2(Φ ir 1 ,F vis 1 ) F fuse =F3(Φ ir 2 ,F vis 2 ) Where C represents the feature concatenation operation; 2.3) Construction of the reconstruction module and decoder; The reconstruction module includes the infrared reconstruction module R. ir and visible light reconstruction module R vis The final fusion feature Φ fuse After passing through the infrared reconstruction module R ir and visible light reconstruction module R vis The corresponding infrared reconstruction information θ is obtained. ir and visible light reconstruction information θ vis Infrared reconstruction information θ ir Visible light reconstruction information θ vis and the final fusion feature Φ fuse Then, the image is fed into decoder D to obtain the fused image I. fuse Infrared reconstructed image I ir 'and visible light reconstructed image I vis ', specifically: i ir =R ir (F) fuse ) i vis =R vis (F) fuse ) [I fuse, I ir ',I vis ']=D(Φ fuse ,i ir ,i vis ) 2.4) Internal and external dual-cycle training mechanism; Inner loop: Update constraint module; With fixed parameters for the fusion module, infrared encoder, and visible light encoder, only the infrared constraint module C is trained. ir and visible light confinement module C vis The loss function is the feature recovery loss Loss1, which measures the loss from the infrared constraint module C. ir With visible light confinement module C vis The restored infrared modal characteristics and visible light modal characteristics are compared with those of the infrared encoder E. ir With visible light encoder E vis The differences between the extracted infrared modal features and visible light modal features are as follows: Loss1=MSE(F' ir i ,F ir i )+MSE(F' vis i ,F vis i ) The goal of the inner loop is to enable the infrared constraint module and the visible light constraint module to accurately restore the infrared mode features and visible light mode features from their respective fused features; Outer loop: Update the fusion module; The parameters of the infrared constraint module, the visible light constraint module, the infrared encoder, and the visible light encoder are fixed, and only the three-layer fusion module is trained; the loss functions are feature restoration loss Loss1 and reconstruction loss Loss2, and the fusion module is optimized in reverse, specifically as follows: Loss2=MSE(I ir ,I ir ’)+MSE(I vis ,I vis ’) The outer loop objective is to ensure that the fusion features output by the fusion module can be restored by the infrared constraint module and the visible light constraint module, and to reconstruct images of both modes, ensuring that information of the infrared mode and the visible light mode is not lost. Throughout the training process, the inner and outer loops are executed alternately until the three-layer fusion module converges; (3) Image fusion generation stage; After the fusion network is trained, a pair of pixel-level paired infrared images I are input. ir With visible light image I vis Infrared and visible light modal features are extracted by infrared and visible light encoders, fusion modules generate fused features, and decoders map the features back to the image space, ultimately outputting a lossless, high-fidelity fused image I. fuse .

Images