Image fusion method based on double-branch feature decoupling auto-encoder

Abstract

The invention discloses an image fusion method based on a double-branch feature decoupling auto-encoder, and belongs to the field of computer vision and image processing. According to the method, for the problems of modal pollution and structural distortion in infrared and visible light image fusion, structural semantic information and high-frequency texture details of a source image are extracted respectively by constructing a content feature coding module and a detail feature coding module, and feature decoupling is achieved. And performing deep fusion on the decoupled features by using an adaptive feature weighting mechanism, and reconstructing a fused image with infrared target saliency and visible light detail definition through a shared decoder. According to the method, an end-to-end two-stage training strategy is adopted for optimization, complex prior or post-processing is not needed, the effects of improving the fused image contrast, edge preservation and target recognition performance are remarkable, and the method is suitable for the fields of weak light monitoring, intelligent perception, unmanned system navigation and the like.

Description

Technical Field

[0001] This invention relates to the field of image information processing technology, and in particular to an image fusion method based on a dual-branch feature decoupling autoencoder. Background Technology

[0002] Image fusion technology aims to integrate complementary information captured by different sensors to generate fused images that are more valuable for subsequent visual tasks. Infrared and visible light image fusion, as an important branch of this technology, can highlight significant thermal radiation targets in infrared images while preserving the rich texture details of visible light images. Therefore, it has wide application value in fields such as autonomous driving, night surveillance, and military reconnaissance.

[0003] Existing image fusion methods can be mainly divided into traditional methods and deep learning-based methods. Traditional methods, such as multi-scale transformation and sparse representation, can achieve certain fusion effects, but they heavily rely on manually designed activity level measures and fusion rules, making it difficult to adaptively extract deep features in complex scenes and easily leading to the loss of important information. Deep learning-based methods, such as convolutional neural networks (CNNs) and generative adversarial networks (GANs), learn fusion mappings in a data-driven manner, resulting in significantly improved performance.

[0004] In recent years, the Transformer architecture has been introduced into the field of computer vision. Chinese patent application publication number "CN118411296A," entitled "A Fusion Method for Infrared and Visible Light Images Based on Convolution and Transformer Collaboration," proposes a dual-branch architecture combining convolutional neural networks and Transformers, promoting the interaction between the two types of features through a complementary feature enhancement module. However, this method directly concatenates infrared and visible light images along the channel dimension as input, failing to fully consider the inherent differences in characteristics between the two modalities, potentially leading to feature confusion. Furthermore, its cascaded fusion strategy is insufficient in terms of the sufficiency of feature interaction, making it difficult to achieve true cross-modal feature complementarity. Simultaneously, this method has limited effectiveness in balancing the contrast information of infrared images with the texture detail information of visible light images, and faces high computational complexity when processing high-resolution images. These factors all restrict further improvement in the quality of the fused image.

[0005] In summary, existing technologies struggle to simultaneously achieve optimal performance in global information modeling, local detail extraction, and cross-modal feature balancing when processing infrared and visible light image fusion. Therefore, there is an urgent need to design a novel image fusion network capable of efficiently establishing correlations between global and local features with low computational overhead, and achieving sufficient extraction and balance of contrast and detail information, thereby generating high-quality fused images. Summary of the Invention

[0006] To address the shortcomings of existing technologies, especially the deficiencies of existing fusion methods in integrating global and local information, extracting and preserving detailed features, and balancing contrast and detail information, this invention provides an image fusion method based on a dual-branch feature decoupling autoencoder.

[0007] To achieve the above objectives, the present invention specifically adopts the following technical solution: An image fusion method based on a dual-branch feature decoupling autoencoder includes the following steps: Step 1, Prepare training data: Select infrared image datasets and visible light image datasets, including the TNO dataset and the MSRS dataset; divide the datasets into training set, validation set and test set; preprocess the images in the training set and randomly crop them into 128×128 pixel image patches; Step 2, Constructing the network model: The network model includes an image input module, a content feature encoding module, a detail feature encoding module, a feature fusion module, and an image decoding module; The image input module receives infrared and visible light images and performs resizing and normalization on the input images to ensure consistent input format and adaptability to the subsequent network structure. The content feature encoding module extracts structural information from the input images to obtain content features including subject shape, regional semantics, and scene distribution characteristics. The detail feature encoding module extracts high-frequency texture and edge features from the input images. The feature fusion module fuses the content and detail features from the infrared and visible light images. The image decoding module performs reverse mapping on the fused features, generating the final fused image through layer-by-layer deconvolution, interpolation reconstruction, and feature channel compression. Step 3, Network Training: A two-stage end-to-end training strategy is adopted. In the first stage, infrared and visible light images are used for input and reconstruction respectively to train the feature decomposition and reconstruction capabilities of the encoder and decoder. The loss function in this stage includes structural similarity loss, feature decomposition loss and gradient loss. In the second stage, infrared and visible light images are used for fusion training. The loss function in this stage includes intensity loss, gradient maximization loss and feature decomposition loss to optimize the fusion effect. Step 4, determine the evaluation indicators: use peak signal-to-noise ratio, structural similarity index, information entropy, mutual information and root mean square error as evaluation indicators to quantitatively evaluate the fusion results and compare them with mainstream fusion methods; Step 5, solidify the network model: After the network parameters are adjusted, fix and save them to obtain the final multimodal image fusion model; Furthermore, in step 2, the content feature encoding module and the detail feature encoding module adopt a dual-branch parallel structure to achieve separate modeling of semantic information and texture information of the input image, thereby avoiding modal interference and feature representation aliasing caused by information coupling, and improving fusion quality and model training stability.

[0008] Furthermore, in step 2, the image decoding module adopts a shared decoding method to ensure that the fused image maintains consistency with the input modality in terms of texture details, structural layout, and brightness hierarchy, and to avoid inconsistent reconstruction problems caused by independent decoding.

[0009] Furthermore, in step 3, the loss function design for the first stage of training is constrained by structural similarity loss, feature decomposition loss, and gradient loss to achieve feature decomposition and accurate reconstruction of the source image by the network. In the second stage of training, intensity loss, gradient maximization loss, and feature decomposition loss are used to constrain the network to achieve a balance and enhancement of infrared targets and visible light details in the fused image. Structural similarity loss is used to evaluate the similarity of the reconstructed image and the original image in terms of structural information, feature decomposition loss is used to guide the network to perform effective modality-specific decomposition of image content, and gradient loss and gradient maximization loss provide constraints for preserving and enhancing the edge texture details of the image, respectively.

[0010] Furthermore, in step 3, the spatial feature extraction module consists of a convolutional nonlinear module, a similarity self-attention module, and a multi-scale spatial feature extraction module, which are used to fully extract spatial features. The convolutional nonlinear module is used to obtain more input information, the similarity self-attention module extracts global spatial features by exploring the relationship between the center pixel and adjacent pixels, and the multi-scale spatial feature extraction module captures receptive fields of different scales from the input data and obtains multi-scale local spatial features obtained by adaptively guiding spatial information with spectral information.

[0011] Compared with existing technologies, this invention provides an image fusion method based on a dual-branch feature decoupling autoencoder, which has the following beneficial effects: This invention provides an infrared-visible image fusion method based on bi-branch decomposition and deep fusion. The method first constructs a content feature encoding module, including a semantic structure extraction submodule and a global context modeling submodule, for extracting scene contours, object region distribution, and global structural information. Then, it constructs a detail feature encoding module, including a high-frequency texture extraction submodule and an edge enhancement submodule, for extracting local texture, detail information, and high-frequency contour changes. Simultaneously, this invention designs a feature fusion module, achieving cross-modal feature deep fusion through channel adaptation, feature alignment, and frequency domain enhancement mechanisms. This method can fully extract complementary information from infrared and visible light images, thereby significantly improving the contrast, detail integrity, and scene readability of the fused image. It has the advantages of clear structure, strong generalization, and high computational efficiency.

[0012] The feature fusion module proposed in this invention employs an adaptive weight allocation mechanism, which generates dynamic fusion weights based on cross-modal feature differences and local saliency information, enabling effective interaction between infrared salient brightness features and visible light texture information. Specifically, the fusion unit dynamically adjusts the modal contribution ratio through a channel attention mechanism and an adaptive matrix calculation method, thereby ensuring consistency and stability of the fusion result in terms of brightness, texture, edge, and contrast. This mechanism enhances the model's adaptability to different scenes, improving fusion quality and robustness.

[0013] This invention introduces a shared multi-scale convolutional structure into the content feature encoding module, combined with a lightweight attention modeling mechanism, which can extract global structural features and regional semantic information, avoiding the loss of structural information caused by relying solely on local convolutions. Simultaneously, through multi-scale contextual receptive fields, the model can adapt to targets of different sizes, background structures, and complex scenes, ensuring structural clarity and semantic fidelity during the reconstruction stage, thereby effectively improving the interpretability and visual quality of the fusion results.

[0014] This invention employs multi-layer sparse convolution and edge enhancement mechanisms in the detail feature encoding module to extract high-frequency detail features by explicitly modeling image gradient changes. Simultaneously, residual paths and skip connections are introduced to enhance feature stability and texture continuity, and reduce texture blurring caused by multi-layer convolution operations. This module can effectively extract visible light texture and contour features while preserving infrared high dynamic range features, thereby achieving detail enhancement, noise suppression, and edge enhancement, improving the quality of the fused image. Attached Figure Description

[0015] Figure 1 This is a flowchart of the image fusion method based on a dual-branch feature decoupling autoencoder according to the present invention; Figure 2 This is a structural diagram of the image fusion method based on a dual-branch feature decoupling autoencoder according to the present invention; Figure 3 This is a structural diagram of the content feature encoding module of the present invention; Figure 4 This is the detailed feature encoding module of the present invention; Figure 5 This is a structural diagram of the feature fusion module for the invention. Detailed Implementation

[0016] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings. The embodiments of the present invention include, but are not limited to, the following examples.

[0017] Example 1 like Figure 1 The flowchart shown is a method for image fusion based on a dual-branch feature decoupling autoencoder proposed in one embodiment of the present invention. The method specifically includes the following steps: Step 1: In this embodiment, the training dataset for the image fusion method uses a publicly available multimodal image alignment dataset, and the original data is preprocessed to meet the network training requirements. The training dataset includes an infrared image set and a visible light image set, both originating from the same scene and maintaining pixel-level alignment to ensure semantic consistency and spatial correspondence in the subsequent fusion learning process.

[0018] In the data processing stage, the original image is first normalized to a fixed resolution.

[0019] Furthermore, to improve the model's generalization ability, various enhancement strategies are employed on the data, including random cropping, horizontal flipping, rotational perturbation, Gaussian noise injection, and brightness / contrast adjustment, to form an enhanced dataset. The image enhancement in this step follows the modal consistency principle, meaning that the infrared and visible light images undergo the same enhancement operations simultaneously to ensure that the cross-modal mapping relationship is not disrupted.

[0020] The enhanced training dataset is divided into training and validation sets in an 8:2 ratio for model training and performance evaluation, and is used as input into subsequent network structures using batch sampling.

[0021] This step enables the construction of a training dataset that includes diverse scene environments, multimodal structural differences, and multi-scale texture features, providing a reliable data foundation for subsequent module feature learning and fusion optimization.

[0022] Step 2, Constructing the network model: The network model includes an image input module, a content feature encoding module, a detail feature encoding module, a feature fusion module, and an image decoding module.

[0023] The encoding stage is used to map the input image to a more representative feature space and perform content feature and detail feature separation modeling, including a content feature encoding module and a detail feature encoding module.

[0024] The content feature encoding module contains four levels of encoding sub-blocks, such as Figure 3 As shown, encoding sub-block one and encoding sub-block two have the same structure, and encoding sub-block three and encoding sub-block four have the same structure, extracting semantic representations step by step in a progressive manner. In encoding sub-block one, the input image channels are sequentially transformed from 3 channels to 64 channels through a 7×7 convolutional layer with a stride of 2, batch normalization, and ReLU activation function. The content features are denoted as follows: and Subsequently, the output features of sub-block one are fed into sub-block two, and through a 5×5 convolutional layer with a stride of 1, batch normalization, and ReLU activation, the feature dimension is further expanded to 128 channels, resulting in... and Next, in the third encoded sub-block, a 3×3 convolutional layer with a stride of 1, batch normalization, and ReLU are applied to expand the channel dimension from 128 to 256, generating... and Finally, in the fourth encoded sub-block, convolution and normalization of the same size are performed to maintain the number of channels at 256 while compressing the spatial resolution, thus obtaining the final content representation features. and Their dimensions are (256, H / 16, W / 16). The convolution operation is denoted as... The content feature extraction process can then be represented as: , In the formula, Indicates the first Layer convolution operation, The number of network layers is determined based on computing resources.

[0025] After encoding, infrared and visible light content features can be obtained separately: , The detail feature encoding module is used to extract high-frequency texture, edge contours, and local detail features, and it shares network weight parameters with both infrared and visible light inputs. This module structure also includes four levels of fine-grained convolutional extraction sub-blocks, such as... Figure 4 As shown, sub-block one and sub-block two have the same structure, and sub-block three and sub-block four have the same structure. In the detail feature sub-block one, a 3×3 convolutional layer with a stride of 1, batch normalization, and ReLU are used, increasing the number of channels from 3 to 48, resulting in... and Subsequently, this output is input to sub-block two, and 5×5 dilated convolution, batch normalization, and ReLU activation are applied to enable the network to capture contextual relationships at a larger scale, expanding the channels to 96 channels, resulting in... and Continuing into sub-block three, the number of channels is expanded to 192 through a 3×3 convolution with a stride of 1, normalization, and activation, resulting in... and Finally, in sub-block four, feature fidelity is enhanced through homostructured convolution and skip connection mechanisms, maintaining 192 channels and outputting the final result. and The detail extraction process can be represented as: , In the formula, It is a smoothing operator used to separate high-frequency content.

[0026] Combining the convolution extraction method, it can be expressed as: , The final results are as follows: , The feature fusion module is used to fuse the above content features and detail features to generate the final fused representation. The module structure is as follows: Figure 5 As shown, it includes an adaptive weight allocation unit, a feature alignment submodule, and a frequency domain enhancement submodule. First, [the following is a description of the module / module]. Inputting the first fusion sub-block separately, and using 1×1 convolution and normalization to compress the channels to a uniform dimension of 256 channels, we obtain the aligned feature set. and Then, proceeding to the second fusion sub-block, a fusion weight matrix is ​​generated using a channel attention mechanism. : , in For the Sigmoid function, This is global average pooling.

[0027] The final fusion feature is represented as: , The image decoding module is used to process the fused features. Layer-by-layer upsampling and deconvolution are performed to restore the feature map to a fused image of the same size as the input.

[0028] The reconstruction process can be represented as: , in, Indicates the first Layer deconvolution or upsampling operation, This represents the number of network decoding layers.

[0029] The output fused image is represented as: , Step 3, Network Training: A two-stage end-to-end training strategy is adopted. The first stage of training uses infrared and visible light images for reconstruction training separately, with a learning rate set to... The training run is performed for 80 epochs. The loss function includes: structural similarity loss. , Feature decomposition loss: , in Represents the correlation coefficient operator. To prevent division by zero for small constants: Gradient loss: , The total loss function is: , Where the weighting coefficient is set to , , .

[0030] The second phase of training involves image fusion, with a learning rate set to [value missing]. The training run is conducted for 40 epochs. The loss function includes: intensity loss. , Maximize gradient loss: , The feature decomposition loss is the same as that in the first stage.

[0031] The total loss function is: , Where the weighting coefficient is set to , , , Both stages use the Adam optimizer, with parameters set to... , The learning rate is halved every 20 cycles, and the batch size is set to 16.

[0032] Step 4, select appropriate evaluation metrics: To comprehensively evaluate the quality of the fused image, the following five core metrics are selected: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Information Entropy, Mutual Information (MI), and Root Mean Square Error (RMSE).

[0033] These metrics can be considered from two dimensions: PSNR, SSIM, and RMSE are mainly used to quantitatively compare the fused result with the reference image, measuring its fidelity and visual similarity; while information entropy and mutual information focus on evaluating the information richness of the fused image itself and the amount of effective information inherited from the source image. Peak Signal-to-Noise Ratio (PSNR): Measures the fidelity of the reconstructed image by calculating the error between the fused image and the reference image. The higher the value, the better the image quality. Structural Similarity Index comprehensively evaluates the visual similarity between images from three dimensions: brightness, contrast, and structure. The closer its value is to 1, the more visually similar the two images are. Information Entropy is used to measure the average information content or complexity of the image itself. The higher the information entropy, the richer the information contained in the image. Mutual Information calculates the statistical dependence between the fused image and the source image, reflecting how much useful information from the source image is retained in the fused result. The higher the mutual information value, the better the fusion effect usually is. Root Mean Square Error quantifies the pixel-level average difference between the fused image and the reference image. The lower the error value, the closer the fused result is to the reference image.

[0034] Table 1 Performance comparison of different models In summary, the proposed image fusion method performs excellently on all evaluation metrics on the test set, effectively improving image quality, preserving key information, and enhancing visual effects. It is suitable for practical application scenarios requiring high-quality image fusion.

[0035] Step 5, Solidify the Network Model: After adjusting the network parameters, fix and save them to obtain the final multimodal image fusion model. Subsequently, to perform an image fusion task, simply input the images to be fused into this model to directly obtain the fused output.

Claims

1. An image fusion method based on a dual-branch feature decoupling autoencoder, characterized in that: The method includes the following steps: Step 1, Prepare training data: Select infrared image datasets and visible light image datasets, including the TNO dataset and the MSRS dataset; divide the datasets into training set, validation set and test set; preprocess the images in the training set and randomly crop them into 128×128 pixel image patches; Step 2, Constructing the network model: The network model includes an image input module, a content feature encoding module, a detail feature encoding module, a feature fusion module, and an image decoding module; Step 3, Network Training: A two-stage end-to-end training strategy is adopted. In the first stage, infrared and visible light images are used for input and reconstruction respectively to train the feature decomposition and reconstruction capabilities of the encoder and decoder. The loss function in this stage includes structural similarity loss, feature decomposition loss and gradient loss. In the second stage, infrared and visible light images are used for fusion training. The loss function in this stage includes intensity loss, gradient maximization loss and feature decomposition loss to optimize the fusion effect. Step 4, determine the evaluation indicators: use peak signal-to-noise ratio, structural similarity index, information entropy, mutual information and root mean square error as evaluation indicators to quantitatively evaluate the fusion results and compare them with mainstream fusion methods; Step 5, solidify the network model: After the network parameters are adjusted, fix and save them to obtain the final multimodal image fusion model.

2. The image fusion method based on a dual-branch feature decoupling autoencoder according to claim 1, characterized in that, The content feature encoding module and the detail feature encoding module adopt a dual-branch parallel structure to process the semantic information and texture information of the input image, respectively. The content feature encoding module is mainly responsible for extracting the high-level semantic information of the image, including the shape of the main body, the distribution of regions and the global structure in the image, while the detail feature encoding module focuses on the local details of the image.

3. The image fusion method based on a dual-branch feature decoupling autoencoder according to claim 1, characterized in that, The feature fusion module includes an adaptive weight allocation unit, a feature alignment submodule, and a frequency domain enhancement submodule.

4. The image fusion method based on a dual-branch feature decoupling autoencoder according to claim 1, characterized in that, The image decoding module adopts a shared decoding method to ensure that the texture details, structural layout, and brightness hierarchy of the final fused image are consistent with the input modal images.

5. The image fusion method based on a dual-branch feature decoupling autoencoder according to claim 4, characterized in that, The images include infrared images and visible light images.

6. The image fusion method based on a dual-branch feature decoupling autoencoder according to claim 4, characterized in that, The shared decoding method ensures that all input features are processed through the same decoding structure, regardless of the modality of the input image.

7. The image fusion method based on a dual-branch feature decoupling autoencoder according to claim 3, characterized in that, The main function of the adaptive weight allocation unit is to dynamically adjust the contributions from different modal information by generating a weight matrix; The adaptive weight allocation unit first performs global average pooling on the input features to capture global information, and then generates adaptive weights through a multilayer perceptron.

Images