Global-local domain adaptation method for infrared and visible image fusion

Abstract

The present invention relates to a global-local domain adaptive infrared and visible image fusion method, which belongs to the field of image information processing technology. It includes a global feature adaptive training stage and a local feature adaptive training stage. The present invention transfers intermediate features with domain differences to a shared domain to achieve feature adaptation, and then utilizes adaptive feature decoding to obtain a fusion result containing more information. A feature extraction network and a fusion image reconstruction network are used to design corresponding feature transfer modules and global-local domain adaptation constraints, thereby eliminating the loss of details caused by feature domain differences during direct fusion, the inaccurate artificial division of information by existing domain adaptation methods, and the existence of false details in adversarial methods, thereby improving the fusion performance of the infrared and visible image fusion network.

Description

Technical Field

[0001] The present invention relates to a global-local domain adaptive infrared and visible image fusion method, belonging to the technical field of image information processing. Background Art

[0002] Currently, the technologies related to the present invention include two aspects: the first is the infrared and visible image fusion method; the second is the domain adaptation of cross-modal images.

[0003] Deep learning-based infrared and visible image fusion algorithms can be broadly divided into three categories. The first category, based on loss function design, constrains the details, edges, structure, or content in the fusion result to preserve more information. In "DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion," Zhao et al. proposed a similarity loss function to fuse infrared and visible images by making the backgrounds of the source images similar while the detail feature maps dissimilar. In "CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion," Zhao et al. designed a novel correlation-driven loss to correlate low-frequency features while decorrelating high-frequency features, thereby highlighting detailed texture details. The second category, based on structural design, focuses on mining multi-level semantic information or cross-modal complementary information to improve fusion performance. In "DenseFuse: AFusion Approach to Infrared and Visible Images," Li et al. designed a dense block in which the output of each layer is connected to every other layer to extract multi-level features from the source image for fusion. In the paper "MATR: Multimodal Medical Image Fusion via Multiscale Adaptive Transformer," Tang et al. introduced an adaptive transformer model to model long-range dependencies to enhance the ability to extract complementary semantic information. A third type of approach based on generative adversarial networks (GANs) uses a balance between preserving thermal radiation information in infrared images and preserving texture information in visible images to obtain a fused image that contains more information. In the paper "FusionGAN: A generative adversarial network for infrared and visible image fusion," Ma et al. designed a fusion framework using a GAN, in which the generator aims to produce a fused image that primarily contains infrared intensity and additional visible gradients, while the discriminator aims to force the fused image to contain detail information present in the visible image.In the paper "AttentionFGAN: Infrared and visible image fusion using attention-based generative adversarial networks", Li et al. integrated a multi-scale attention mechanism into the adversarial generative network, enabling the generator to focus on the foreground target information of infrared images and the background detail information of visible light images, while limiting the discriminator to focus on the attention area rather than the entire input image.

[0004] Domain adaptation algorithms for cross-modal images can be roughly divided into two categories. The first category, based on decomposition, divides the source image into different components and processes these components differently during feature extraction or fusion. For example, in "U2fusion: A unified unsupervised image fusion network," Xu et al. estimate the information level by extracting shallow features (such as texture and local shape) and deep features (such as content and spatial structure). The information level is then used to mitigate the information inconsistency between the source images when reconstructing the fused image. The second category, based on adversarial methods, attempts to mitigate the impact of cross-domain feature gaps by preserving infrared and visible light features. In "Ddcgan: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion," Ma et al. propose a dual-discriminator network to distinguish the content and structural differences between the fused image and the two source images, respectively, forcing the fused image to simultaneously preserve the thermal radiation information in the infrared image and the texture details in the visible light image.

[0005] In practical applications, infrared and visible light images are acquired by sensors with different imaging mechanisms, resulting in a cross-domain feature gap between them. Specifically, the same object exhibits different features in different source images: high-contrast thermal radiation information in infrared images and texture information in visible light images. Direct fusion results in the loss of texture details in the fused image. A potential solution to this problem is to perform domain adaptation on cross-modal image features. However, since the information contained in the source images cannot be simply divided into multiple categories according to hand-designed rules, decomposition-based methods cannot achieve good feature adaptation. Meanwhile, adversarial-based methods rely solely on the discriminator to constrain the fusion results, resulting in images with often spurious and blurred details. Summary of the Invention

[0006] To address the domain disparity problem that exists when fusing infrared and visible images, this paper proposes a global-local domain adaptation method for infrared and visible image fusion. This method transfers intermediate features with domain differences to a shared domain to achieve feature adaptation, and then uses adaptive feature decoding to obtain a fusion result that contains more information. This method utilizes a feature extraction network and a fusion image reconstruction network, designing corresponding feature transfer modules and global-local domain adaptation constraints. This eliminates the detail loss caused by feature domain differences during direct fusion, the inaccurate information partitioning caused by existing domain adaptation methods, and the false details present in adversarial methods. This improves the fusion performance of the infrared and visible image fusion network.

[0007] The present invention provides a global-local domain adaptive infrared and visible image fusion method, specifically a method for implementing domain adaptation using a feature transfer module and a global-local domain adaptation constraint. The method is characterized in that it includes the following steps:

[0008] Step 1) Global feature adaptive training phase

[0009] Infrared features extracted from the source image by the feature encoder and optical characteristics There is a domain gap between the two domains, and direct fusion will lead to the loss of texture details. Therefore, a domain-independent shared space is found between the infrared domain and the visible domain, and a feature transfer module T is designed. x Realize domain space transfer; feature transfer module T x The features from different domains are concatenated channel by channel to obtain the channel weights, and then the weights are multiplied by the features of the other modality. Finally, the transferred features are obtained through three convolutional layers with a kernel size of 3x 3;

[0010] Step 2) Local feature adaptive training stage

[0011] Design a mean-variance local constraint based on sliding window.

[0012] Preferably, the feature encoder in step 1) includes four residual blocks, each residual block consists of two convolutional layers with a kernel size of 3x3, and between each residual block, there is a maximum pooling layer with a stride of 2, and a rectified linear unit function is applied.

[0013] Preferably, in order to achieve domain adaptation during the feature transfer process in step 1), a global domain discriminator D is designed G To achieve global domain spatial adaptation; the global domain discriminator consists of a fully connected layer, which inputs features and outputs domain discrimination results; the following loss is used to train the global domain discriminator D G :

[0014]

[0015] Where a and b represent the domain labels of the infrared domain and the visible light domain; and Represents the global domain discriminator for infrared features and optical characteristics Once the domain discriminator can accurately distinguish which domain the input feature comes from, freeze D G Parameters, and encourage global domain adaptation features to be difficult to be distinguished by the domain discriminator to achieve global domain adaptation:

[0016]

[0017] Where c represents the domain label of the adapted domain; when the discriminator cannot distinguish the transferred features, and are collectively transferred to a shared intermediate transfer domain.

[0018] Preferably, the specific steps of step 2) are: using a window of size h×w to obtain local features, for f∈R h× w ×c The local features of , where c is the number of feature channels, and its domain representation Dom is defined as:

[0019] Dom=μ(f)+σ(f) (3)

[0020] Where μ(f) is defined as:

[0021]

[0022] At the same time, σ(f) is defined as:

[0023]

[0024] The local constraint can be expressed as:

[0025]

[0026] in and express and The local features of . So the global to local constraint can be expressed as:

[0027]

[0028] Through this method, the features with domain gaps are first transferred to a shared adaptive domain space through global adaptation, and then local adaptation is further performed in the domain-independent adaptive domain to shorten the distance between features. The feature mismatch caused by the cross-domain feature gap is progressively reduced from global to local, and adaptive features are obtained. and Finally, a decoder D fus The fusion result is obtained from the adaptive features, and the total loss can be expressed as:

[0029]

[0030] Among them L ssim represents the SSIM loss, α, δ, and γ are the tuning parameters for controlling the weights; the decoder D fus It contains 4 deconvolution layers and 4 residual blocks to gradually upsample the feature map to the original size. The kernel size and stride of the deconvolution layer are 2x2 and 2 respectively. Each residual block contains two convolution layers with a kernel size of 3x3.

[0031] Beneficial effects of the present invention: The global-local domain adaptation method proposed in the present invention adopts an adaptive strategy to avoid manually designed categories. At the same time, it does not use discriminators to constrain the final fusion results. Instead, it seeks an intermediate adaptation domain between the visible light domain and the infrared domain, and transfers intermediate features with domain gaps to this adaptive domain to achieve feature adaptation. The present invention first designs a feature transfer module to achieve feature domain transfer, and designs a global domain discriminator to ensure that the extracted cross-domain intermediate features are transferred to the shared adaptive domain. Then, a sliding window-based mean-variance local constraint is proposed to shorten their distance within the adaptive domain, and better feature adaptation is achieved through a progressive adaptive strategy from global to local. In summary, the global-local domain adaptive infrared and visible image fusion method of the present invention effectively alleviates the loss of texture detail information caused by cross-domain feature gaps, and can obtain a fusion result containing more texture details. At the same time, the present invention overcomes some drawbacks of current domain adaptation methods for cross-modal images, such as inaccurate information division in decomposition-based methods and the presence of false details in adversarial-based methods. With the assistance of the proposed global-local domain adaptation, the present invention can generate more realistic, clear, and more informative infrared and visible light image fusion results. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is the overall training flow chart of the network;

[0033] Figure 2 Schematic diagram of the feature encoder structure Figure 1 ;

[0034] Figure 3 Schematic diagram of the feature transfer module structure;

[0035] Figure 4 Schematic diagram of the global domain discriminator structure;

[0036] Figure 5 Schematic diagram of the feature decoder structure Figure 2 . DETAILED DESCRIPTION

[0037] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0038] A global-local domain adaptation infrared and visible image fusion method is provided, specifically a method for implementing domain adaptation of infrared and visible image fusion using a feature transfer module and a global-local domain adaptation constraint, comprising the following steps:

[0039] (1) Global feature adaptive training stage

[0040] Infrared features extracted directly from the source image by the feature encoder and optical characteristics There is a domain gap between the two domains, and direct fusion will lead to the loss of texture details. To this end, the present invention seeks a domain-independent shared space between the infrared domain and the visible domain, and designs a feature transfer module T x Realize domain space transfer. The feature encoder consists of four residual blocks, each of which consists of two convolutional layers with a kernel size of 3x 3. Between each residual block, there is a maximum pooling layer with a stride of 2 and a rectified linear unit function. The feature transfer module T x The features from different domains are concatenated channel by channel to obtain the channel weights, and then the weights are multiplied by the features of the other modality. Finally, the transfer features are obtained through three convolutional layers with a kernel size of 3x 3. The design motivation is to force the features from different domains to be close to the shared intermediate adaptation domain space.

[0041] In order to achieve domain adaptation in the feature transfer process, this paper designs a global domain discriminator D G To achieve global domain spatial adaptation. The global domain discriminator consists of a fully connected layer, which inputs features and outputs domain discrimination results. First, the global domain discriminator is encouraged to distinguish infrared features well. and optical characteristics From which domain, therefore, the following loss is used to train the global domain discriminator D G :

[0042]

[0043] Where a and b represent the domain labels of the infrared domain and the visible light domain. and Represents the global domain discriminator for infrared features and optical characteristics Once the domain discriminator can accurately distinguish which domain the input features come from, freeze D GParameters, and encourage global domain adaptation features to be difficult to be distinguished by the domain discriminator to achieve global domain adaptation:

[0044]

[0045] Where c represents the domain label of the adapted domain. When the discriminator cannot distinguish the transferred features, and are collectively transferred to a shared intermediate transfer domain.

[0046] (2) Local feature adaptive training stage

[0047] In order to further shorten the distance between the features collectively transferred to the adaptation domain in the domain and achieve better feature adaptation, the present invention proposes a mean-variance local constraint based on a sliding window. Specifically, the present invention uses a window of size h×w to obtain local features. For f∈R h×w×c The local features of , where c is the number of feature channels, and its domain representation Dom is defined as:

[0048] Dom=μ(f)+σ(f) (3)

[0049] Where μ(f) is defined as:

[0050]

[0051] At the same time, σ(f) is defined as:

[0052]

[0053] Thus, the local constraint can be expressed as:

[0054]

[0055] in and express and The local features of . So the global to local constraint can be expressed as:

[0056]

[0057] Through this method, the features with domain gaps are first transferred to a shared adaptive domain space through global adaptation, and then local adaptation is further performed in the domain-independent adaptive domain to shorten the distance between features. The feature mismatch caused by the cross-domain feature gap is progressively reduced from global to local, and adaptive features are obtained. and Finally, a decoder D fus The fusion result is obtained from the adaptive features. The total loss can be expressed as:

[0058]

[0059] Among them L ssim Denotes the SSIM loss, α, δ, and γ are the tuning parameters that control the weights. Decoder D fus It contains 4 deconvolution layers and 4 residual blocks to gradually upsample the feature map to the original size. The kernel size and stride of the deconvolution layer are 2x2 and 2 respectively. Each residual block contains two convolution layers with a kernel size of 3x3.

[0060] In this embodiment, Figure 1 The flowchart for the overall network training is as follows: In the first step, the infrared features and visible light features with domain gaps obtained by the feature encoder are transferred to the domain space through a feature transfer module, and a global domain discriminator is used to ensure that the features are transferred to the shared adaptive domain as a whole; In the second step, the features obtained in the first step are locally adapted using the sliding window mean variance local constraint to shorten their distance within the adaptive domain, progressively implementing feature domain adaptation, and using a decoder to restore the fused image from the obtained domain adaptation features. In the first step of global feature adaptation training, Figure 2 The feature encoder extracts infrared features from the input image and optical characteristics Figure 3 The feature transfer module finds an intermediate shared domain between the infrared domain and the visible domain, and transfers the features with differences to this domain. Figure 4 The global domain discriminator is trained for global domain adaptation. It is first trained using formula (1). After the training is completed, that is, when the global domain discriminator can distinguish which domain the feature comes from, its parameters are frozen and the feature is forced to adapt using formula (2). In the second step of local feature adaptation training, the sliding window mean variance local domain constraint is used to shorten the distance between features in the adaptive domain to achieve local domain adaptation, that is, formula (6). Finally, the Figure 5 The feature decoder decodes the obtained domain adaptive features into a fused image. The loss function during training is shown in Formula (8). After global to local domain adaptation, the impact of feature domain differences is effectively alleviated, and a more realistic, clear, and detailed fused image can be obtained.

[0061] During the training process of this infrared and visible image fusion method, infrared features and visible light features with domain differences are transferred to a shared adaptive domain to achieve feature adaptation. During testing, the model can produce fusion results with more texture details. The entire training process is divided into a global feature adaptation training stage and a local feature adaptation training stage. In the global feature adaptation training stage, a global domain discriminator is used to ensure that the extracted cross-domain intermediate features are transferred as a whole to the shared adaptive domain. In the local feature adaptation training stage, a mean-variance local constraint based on a sliding window is used to shorten the distance between features within the adaptive domain.

Claims

1. Global-local domain adaptation infrared and visible image fusion method, characterized by The following steps are involved: Step 1) Global feature adaptive training phase Infrared features extracted from the source image by the feature encoder and optical characteristics There is a domain gap between the two domains, and direct fusion will lead to the loss of texture details. Therefore, a domain-independent shared space is found between the infrared domain and the visible domain, and a feature transfer module T is designed. x Realize domain space transfer; feature transfer module T x The features from different domains are concatenated channel by channel to obtain the channel weights, and then the weights are multiplied by the features of the other modality. Finally, the transferred features are obtained through three convolutional layers with a kernel size of 3x 3; Step 2) Local feature adaptive training stage Design a mean-variance local constraint based on sliding window.

2. The global-local domain adaptive infrared and visible image fusion method according to claim 1, characterized in that The feature encoder in step 1) includes four residual blocks, each of which consists of two convolutional layers with a kernel size of 3x3. Between each residual block, there is a maximum pooling layer with a stride of 2 and a rectified linear unit function is applied.

3. The global-local domain adaptive infrared and visible image fusion method according to claim 2, characterized in that In order to achieve domain adaptation during the feature transfer process described in step 1), a global domain discriminator D is designed G To achieve global domain spatial adaptation; The global domain discriminator consists of a fully connected layer, which inputs features and outputs domain discrimination results; The global domain discriminator D is trained using the following loss G : Where a and b represent the domain labels of the infrared domain and the visible light domain; and They represent the global domain discriminator for infrared features and optical characteristics Once the domain discriminator can accurately distinguish which domain the input feature comes from, freeze D G Parameters, and encourage global domain adaptation features to be difficult to be distinguished by the domain discriminator to achieve global domain adaptation: Where c represents the domain label of the adapted domain; when the discriminator cannot distinguish the transferred features, and are collectively transferred to a shared intermediate transfer domain.

4. The global-local domain adaptive infrared and visible image fusion method according to claim 1, characterized in that The specific steps of step 2) are: use a window of size h×w to obtain local features, for f∈R h×w×d The local features of , where d is the number of feature dimensions, and its domain representation Dom is defined as: Dom=μ(f)+σ(f) (3) Where μ(f) is defined as: At the same time, σ(f) is defined as: The local constraint can be expressed as: in and express and The local features of , so the global to local constraints can be expressed as: Through this method, the features with domain gaps are first transferred to a shared adaptive domain space through global adaptation, and then local adaptation is further performed in the domain-independent adaptive domain to shorten the distance between features. The feature mismatch caused by the cross-domain feature gap is progressively reduced from global to local, and adaptive features are obtained. and Finally, a decoder D fus The fusion result is obtained from the adaptive features, and the total loss can be expressed as: Among them L ssim represents the SSIM loss, α, δ, and γ are tuning parameters that control the weights; Decoder D fus It contains 4 deconvolution layers and 4 residual blocks to gradually upsample the feature maps to the original size. The kernel size and stride of the deconvolution layer are 2x2 and 2 respectively. Each residual block contains two convolution layers with a kernel size of 3x3.

Images