Global-local domain adaptive infrared and visible image fusion method

Abstract

The invention relates to a global-local domain adaptive infrared and visible image fusion method, and belongs to the technical field of image information processing. Comprising a global feature adaptive training stage and a local feature adaptive training stage. Intermediate features with domain differences are transferred to a shared domain to realize feature adaptation, then a fusion result containing more information is obtained by using adaptive feature decoding, a feature extraction network and a fusion image reconstruction network are used, a corresponding feature transfer module and global-local domain adaptation constraints are designed, and a fusion result containing more information is obtained. The problems that details are lost due to feature domain differences during direct fusion, manual information division of an existing domain adaptation method is inaccurate and false details exist in a confrontation method are solved, and the fusion performance of the infrared and visible image fusion network is improved.

Description

Technical Field

[0001] The invention relates to a global-local domain adaptive infrared and visible image fusion method, belonging to the technical field of image information processing. Background Art

[0002] At present, the technologies related to the present invention include two aspects: the first is the infrared and visible image fusion method; the second is the domain adaptation of cross-modal images.

[0003] The infrared and visible image fusion algorithms based on deep learning can be divided into three categories: the first type of methods based on loss function design retain more information by constraining the details, edges, structures or contents in the fusion results. Zhao et al. proposed a similarity loss function in the paper "DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion" to make the background of the source image similar and the detail feature map dissimilar to achieve the fusion of infrared and visible images. Zhao et al. designed a new correlation-driven loss in the paper "CDDFuse: Correlation-Driven Dual-Branch Feature Decompositionfor Multi-Modality Image Fusion" to make low-frequency features related and high-frequency features unrelated, thereby highlighting detailed texture details. The second type of methods based on structural design focus on mining multi-level semantic information or cross-modal complementary information to improve fusion performance. Li et al. designed a dense block in which the output of each layer is connected to each other layer in the paper "DenseFuse: AFusion Approach to Infrared and Visible Images" to obtain multi-level features from the source image for fusion. Tang et al. introduced an adaptive transformer model in the paper "MATR: Multimodal Medical Image Fusion via Multiscale Adaptive Transformer" to model long-distance dependencies to enhance the ability to extract complementary semantic information. The third type of method based on generative adversarial networks obtains a fused image containing more information in the game between retaining thermal radiation information in infrared images and retaining texture information in visible images. Ma et al. designed a fusion framework using a generative adversarial network in the paper "FusionGAN: A generative adversarial network for infrared and visible image fusion", in which the generator aims to generate a fused image that mainly contains infrared intensity and additional visible gradients, while the discriminator aims to force the fused image to contain the detail information present in the visible image.In the paper "AttentionFGAN: Infrared and visible image fusion using attention-based generative adversarial networks", Li et al. integrated the multi-scale attention mechanism into the adversarial generative network, so that the generator focuses on the foreground target information of infrared images and the background detail information of visible light images, while limiting the discriminator to focus on the attention area instead of the entire input image.

[0004] Domain adaptation algorithms for cross-modal images can be roughly divided into two categories: the first category is based on decomposition methods that divide the source image into different parts and process these parts differently during feature extraction or fusion. For example, Xu et al. in the paper "U2fusion: A unified unsupervised image fusion network" estimate the information degree by extracting shallow features (texture, local shape, etc.) and deep features (content, spatial structure, etc.), and then use the information degree to alleviate the problem of inconsistent information contained in the source images when reconstructing the fused image. The second category of adversarial-based methods attempts to mitigate the impact of cross-domain feature gaps in the adversarial preservation of infrared features and visible light features. Ma et al. in the paper "Ddcgan: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion" proposed a dual discriminator network to distinguish the content and structural differences between the fused image and the two source images, respectively, forcing the fused image to simultaneously maintain the thermal radiation information in the infrared image and the texture details in the visible light image.

[0005] In practical applications, since infrared and visible light images are obtained by sensors with different imaging mechanisms, there is a cross-domain feature gap between them. Specifically, the same object exhibits different features in different source images, which are high-contrast thermal radiation information in infrared images and texture information in visible light images. Direct fusion will lead to the loss of texture details in the fused image. To address this problem, a potential solution is to perform domain adaptation on cross-modal image features. However, since the information contained in the source image cannot be simply divided into multiple categories according to manually designed rules, decomposition-based methods cannot achieve good feature adaptation; at the same time, adversarial-based methods rely only on the discriminator to constrain the fusion results, and the final image usually contains false and blurred details. Summary of the invention

[0006] The present invention aims at the domain gap problem existing in the fusion of infrared and visible images, and proposes a global-local domain adaptation infrared and visible image fusion method. The present invention transfers the intermediate features with domain differences to the shared domain to achieve feature adaptation, and then uses adaptive feature decoding to obtain a fusion result containing more information. The present invention uses a feature extraction network and a fusion image reconstruction network, designs corresponding feature transfer modules and global-local domain adaptation constraints, and eliminates the loss of details caused by feature domain differences during direct fusion, the inaccurate artificial division of information in existing domain adaptation methods, and the existence of false details in adversarial methods, thereby improving the fusion performance of the infrared and visible image fusion network.

[0007] A global-local domain adaptive infrared and visible image fusion method of the present invention is specifically a method for implementing domain adaptive infrared and visible image fusion using a feature transfer module and a global-local domain adaptive constraint, and the method is special in that it includes the following steps:

[0008] Step 1) Global feature adaptive training phase

[0009] Infrared features extracted from the source image by the feature encoder and optical characteristics There is a domain gap between the infrared domain and the visible domain, and direct fusion will lead to the loss of texture details. Therefore, a domain-independent shared space is found between the infrared domain and the visible domain, and a feature transfer module T is designed. x Realize domain space transfer; feature transfer module T x The features from different domains are concatenated by channel to obtain the channel weights, and then the weights are multiplied by the features of the other modality. Finally, the transfer features are obtained through three convolutional layers with a kernel size of 3x 3;

[0010] Step 2) Local feature adaptive training stage

[0011] Design a mean-variance local constraint based on sliding windows.

[0012] Preferably, the feature encoder in step 1) comprises four residual blocks, each residual block consists of two convolutional layers with a kernel size of 3x 3, and between each residual block, there is a maximum pooling layer with a stride of 2, and a rectified linear unit function is applied.

[0013] Preferably, in the feature transfer process in step 1), a global domain discriminator D is designed to achieve domain adaptation. G To achieve global domain spatial adaptation; the global domain discriminator consists of a fully connected layer, which inputs features and outputs domain discrimination results; the following loss is used to train the global domain discriminator D G :

[0014]

[0015] Where a and b represent the domain labels of infrared and visible light domains; and They represent the global domain discriminator for infrared features and optical characteristics Once the domain discriminator can accurately distinguish which domain the input feature comes from, freeze D G , and encourages global domain adaptation features to be difficult to be distinguished by the domain discriminator to achieve global domain adaptation:

[0016]

[0017] Where c represents the domain label of the adapted domain; when the discriminator cannot distinguish the transferred features, and are collectively transferred to a shared intermediate transfer domain.

[0018] Preferably, the specific steps of step 2) are: using a window of size h×w to obtain local features, for f∈R h× w ×c The local features of , where c is the number of feature channels, and its domain representation Dom is defined as:

[0019] Dom=μ(f)+σ(f) (3)

[0020] Where μ(f) is defined as:

[0021]

[0022] At the same time, σ(f) is defined as:

[0023]

[0024] The local constraint can be expressed as:

[0025]

[0026] in and express and The local features of . So the global to local constraint can be expressed as:

[0027]

[0028] Through this method, the features with domain gaps are first transferred to a shared adaptive domain space through global adaptation, and then local adaptation is further performed in the domain-independent adaptive domain to shorten the distance between features. The feature mismatch caused by the cross-domain feature gap is progressively reduced from global to local, and the adaptive features are obtained. and Finally, a decoder D fus The fusion result is obtained from the adaptive features, and the total loss can be expressed as:

[0029]

[0030] Where L ssim represents the SSIM loss, α, δ, and γ are the tuning parameters for controlling the weights; the decoder D fus It contains 4 deconvolution layers and 4 residual blocks to gradually upsample the feature maps to the original size. The kernel size and stride of the deconvolution layer are 2x2 and 2 respectively. Each residual block contains two convolution layers with a kernel size of 3x3.

[0031] Beneficial effects of the present invention: The global-local domain adaptation method proposed in the present invention adopts an adaptive strategy to avoid manually designed categories. At the same time, it does not use a discriminator to constrain the final fusion result, but finds an intermediate adaptation domain between the visible light domain and the infrared domain, and transfers the intermediate features with domain gaps to this adaptive domain to achieve feature adaptation. The present invention first designs a feature transfer module to achieve feature domain transfer, and designs a global domain discriminator to ensure that the extracted cross-domain intermediate features are transferred to the shared adaptive domain, and then proposes a mean-variance local constraint based on a sliding window to shorten their distance in the adaptive domain, and achieves better feature adaptation through a progressive adaptive strategy from global to local. In summary, the infrared and visible image fusion method of the global-local domain adaptation of the present invention effectively alleviates the loss of texture detail information caused by the cross-domain feature gap, and can obtain a fusion result containing more texture details. At the same time, the present invention overcomes some drawbacks of current domain adaptation methods for cross-modal images, such as inaccurate information division in decomposition-based methods and the presence of false details in adversarial-based methods. With the assistance of the proposed global-local domain adaptation, the present invention can generate infrared and visible light image fusion results that are more realistic, clear, and contain more information. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is the overall training flow chart of the network;

[0033] Figure 2 Schematic diagram of the feature encoder structure Figure 1 ;

[0034] Figure 3 It is a structural diagram of the feature transfer module;

[0035] Figure 4 Schematic diagram of the global domain discriminator structure;

[0036] Figure 5 Schematic diagram of the feature decoder structure Figure 2 . DETAILED DESCRIPTION

[0037] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0038] A global-local domain adaptation infrared and visible image fusion method, specifically a method for implementing domain adaptation of infrared and visible images using a feature transfer module and a global-local domain adaptation constraint, comprising the following steps:

[0039] (1) Global feature adaptive training stage

[0040] Infrared features extracted directly from the source image by the feature encoder and optical characteristics There is a domain gap between the infrared domain and the visible domain, and direct fusion will lead to the loss of texture details. To this end, the present invention seeks a domain-independent shared space between the infrared domain and the visible domain, and designs a feature transfer module T x Realize domain space transfer. The feature encoder consists of four residual blocks, each of which consists of two convolutional layers with a kernel size of 3x 3. Between each residual block, there is a maximum pooling layer with a stride of 2 and a rectified linear unit function is applied; the feature transfer module T x The features from different domains are concatenated channel by channel to obtain the channel weights, and then the weights are multiplied by the features of another modality. Finally, the transfer features are obtained through three convolutional layers with a kernel size of 3x 3. The design motivation is to force the features from different domains to be close to the shared intermediate adaptation domain space.

[0041] In order to achieve domain adaptation in the feature transfer process, this paper designs a global domain discriminator D G To achieve global domain spatial adaptation. The global domain discriminator consists of a fully connected layer, which inputs features and outputs domain discrimination results. First, the global domain discriminator is encouraged to distinguish infrared features well. and optical characteristics From which domain, therefore, the following loss is used to train the global domain discriminator D G :

[0042]

[0043] Where a and b represent the domain labels of the infrared domain and the visible light domain. and They represent the global domain discriminator for infrared features and optical characteristics Once the domain discriminator can accurately distinguish which domain the input feature comes from, freeze D G, and encourages global domain adaptation features to be difficult to be distinguished by the domain discriminator to achieve global domain adaptation:

[0044]

[0045] Where c represents the domain label of the adapted domain. When the discriminator cannot distinguish the transferred features, and are collectively transferred to a shared intermediate transfer domain.

[0046] (2) Local feature adaptive training stage

[0047] In order to further shorten the distance between the features collectively transferred to the adaptation domain in the domain and achieve better feature adaptation, the present invention proposes a mean-variance local constraint based on a sliding window. Specifically, the present invention uses a window of size h×w to obtain local features. For f∈R h×w×c The local features of , where c is the number of feature channels, and its domain representation Dom is defined as:

[0048] Dom=μ(f)+σ(f) (3)

[0049] Where μ(f) is defined as:

[0050]

[0051] At the same time, σ(f) is defined as:

[0052]

[0053] In this way, the local constraint can be expressed as:

[0054]

[0055] in and express and The local features of . So the global to local constraint can be expressed as:

[0056]

[0057] Through this method, the features with domain gaps are first transferred to a shared adaptive domain space through global adaptation, and then local adaptation is further performed in the domain-independent adaptive domain to shorten the distance between features. The feature mismatch caused by the cross-domain feature gap is progressively reduced from global to local, and the adaptive features are obtained. and Finally, a decoder D fus The fusion result is obtained from the adaptive features. The total loss can be expressed as:

[0058]

[0059] Where L ssim represents the SSIM loss, α, δ, and γ are the tuning parameters that control the weights. Decoder D fus It contains 4 deconvolution layers and 4 residual blocks to gradually upsample the feature maps to the original size. The kernel size and stride of the deconvolution layer are 2x2 and 2 respectively. Each residual block contains two convolution layers with a kernel size of 3x3.

[0060] In this embodiment, Figure 1 The flowchart of the overall network training is as follows: In the first step, the infrared features and visible light features with domain gaps obtained by the feature encoder are transferred to the domain space through a feature transfer module, and the global domain discriminator is used to ensure that the features are transferred to the shared adaptive domain as a whole; In the second step, the features obtained in the first step are locally adapted using the sliding window mean variance local constraint to shorten their distance in the adaptive domain, and feature domain adaptation is progressively implemented, and a decoder is used to restore the fused image from the obtained domain adaptation features. In the first step of global feature adaptation training, Figure 2 The feature encoder extracts infrared features from the input image and optical characteristics Figure 3 The feature transfer module of finds an intermediate shared domain between the infrared domain and the visible domain, and transfers the features with differences to this domain. Figure 4 The global domain discriminator is used for global domain adaptation. It is first trained using formula (1). After the training is completed, that is, when the global domain discriminator can distinguish which domain the feature comes from, its parameters are frozen and the feature is forced to adapt using formula (2). In the second step of the local feature adaptation training phase, the sliding window mean variance local domain constraint is used to shorten the distance between features in the adaptive domain to achieve local domain adaptation, that is, formula (6). Finally, Figure 5 The feature decoder decodes the obtained domain adaptive features into a fused image. The loss function during training is shown in formula (8). After global to local domain adaptation, the impact of feature domain differences is effectively alleviated, and a more realistic and clear fused image containing more texture details can be obtained.

[0061] During the training process of the infrared and visible image fusion method, the infrared features and visible light features with domain differences are transferred to the shared adaptive domain to achieve feature adaptation. The model can produce fusion results containing more texture details during the test process. The entire training process is divided into a global feature adaptive training stage and a local feature adaptive training stage. In the global feature adaptive training stage, a global domain discriminator is used to ensure that the extracted cross-domain intermediate features are transferred as a whole to the shared adaptive domain. In the local feature adaptive training stage, the mean-variance local constraint based on a sliding window is used to shorten the distance of features in the adaptive domain.

Claims

1. Global-local domain adaptation infrared and visible image fusion method, characterized by The following steps are involved: Step 1) Global feature adaptive training phase Infrared features extracted from the source image by the feature encoder and optical characteristics There is a domain gap between the infrared domain and the visible domain, and direct fusion will lead to the loss of texture details. Therefore, a domain-independent shared space is found between the infrared domain and the visible domain, and a feature transfer module T is designed. x Realize domain space transfer; feature transfer module T x The features from different domains are concatenated by channel to obtain the channel weights, and then the weights are multiplied by the features of the other modality. Finally, the transfer features are obtained through three convolutional layers with a kernel size of 3x3; Step 2) Local feature adaptive training stage Design a mean-variance local constraint based on sliding windows.

2. The global-local domain adaptive infrared and visible image fusion method according to claim 1, characterized in that The feature encoder in step 1) includes four residual blocks, each residual block consists of two convolutional layers with a kernel size of 3x3. Between each residual block, there is a maximum pooling layer with a stride of 2 and a rectified linear unit function is applied.

3. The global-local domain adaptive infrared and visible image fusion method according to claim 2, characterized in that In order to achieve domain adaptation in the feature transfer process described in step 1), a global domain discriminator D is designed G To achieve global domain spatial adaptation; The global domain discriminator consists of a fully connected layer, which inputs features and outputs domain discrimination results; The global domain discriminator D is trained using the following loss G : Where a and b represent the domain labels of infrared and visible light domains; and They represent the global domain discriminator for infrared features and optical characteristics Once the domain discriminator can accurately distinguish which domain the input feature comes from, freeze D G , and encourages global domain adaptation features to be difficult to be distinguished by the domain discriminator to achieve global domain adaptation: Where c represents the domain label of the adapted domain; when the discriminator cannot distinguish the transferred features, and are collectively transferred to a shared intermediate transfer domain.

4. The global-local domain adaptive infrared and visible image fusion method according to claim 1, characterized in that The specific steps of step 2) are: use a window of size h×w to obtain local features, for f∈R h×w×c The local features of , where c is the number of feature channels, and its domain representation Dom is defined as: Dim=μ(f)+σ(f) (3) Where μ(f) is defined as: At the same time, σ(f) is defined as: The local constraint can be expressed as: in and express and The local features of , so the global to local constraints can be expressed as: Through this method, the features with domain gaps are first transferred to a shared adaptive domain space through global adaptation, and then local adaptation is further performed in the domain-independent adaptive domain to shorten the distance between features. The feature mismatch caused by the cross-domain feature gap is progressively reduced from global to local, and the adaptive features are obtained. and Finally, a decoder D fus The fusion result is obtained from the adaptive features, and the total loss can be expressed as: Where L ssim represents the SSIM loss, α, δ, and γ are the tuning parameters that control the weights; Decoder D fus It contains 4 deconvolution layers and 4 residual blocks to gradually upsample the feature maps to the original size. The kernel size and stride of the deconvolution layer are 2x2 and 2 respectively. Each residual block contains two convolution layers with a kernel size of 3x3.

Images