Multi-modal image fusion method based on Swinin-MOE

Abstract

The invention belongs to the technical field of image fusion, and particularly relates to a multi-modal image fusion method based on Swi n-MOE, and the method comprises the following steps: 1, preparing training data; a multi-focus data set, a multi-exposure data set, an infrared image data set and a visible light image data set are selected, the data sets are divided into a training set, a verification set and a test set, and original images in the multi-focus data set and the multi-exposure data set and corresponding labels of the original images are preprocessed. According to the invention, a dual-channel adaptive attention fusion architecture and a decoder integrated with a hybrid expert system are innovatively designed, the limitation that an existing fusion model only aims at a single task is broken through, the information collaboration advantage in multi-task fusion is fully excavated, joint optimization of multiple fusion tasks is realized, and the fusion efficiency is improved. And knowledge migration and complementation among different fusion data sets are promoted.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, specifically to a method for multimodal image fusion based on Swin-MOE. Background Technology

[0002] Image fusion technology plays a crucial role in autonomous driving environmental perception systems. By integrating complementary information from multi-source heterogeneous sensors such as infrared sensors and visible light cameras, it significantly improves target detection and semantic segmentation performance in complex scenes. For example, infrared images can effectively capture thermal radiation features, providing reliable environmental information in low-light or adverse weather conditions; while visible light images retain rich texture details and color information. However, current mainstream image fusion methods suffer from three main limitations: First, traditional models typically design dedicated network structures and loss functions for single tasks (such as multi-exposure fusion or multi-focus fusion), making it difficult for the models to generalize to other fusion tasks. Second, existing frameworks lack effective cross-modal feature interaction mechanisms, failing to dynamically coordinate information complementarity between different tasks, and are prone to modality dominance bias and feature conflicts during multi-task collaborative training. Finally, adapting to new fusion tasks often requires retraining the entire model, which not only incurs high computational costs but also leads to suboptimal performance trade-offs between tasks due to fixed parameter sharing mechanisms. These problems severely restrict the performance and deployment efficiency of image fusion technology in practical applications such as autonomous driving.

[0003] The Chinese authorization announcement number is "CN118967479B", and the title is "An Adaptive Feature Extraction Method for Fusion of Infrared and Visible Light Images". It includes: using a single-modal spatial adaptive feature mixing module to perform multi-scale spatial adaptive feature extraction on two modal images; inputting the extracted features into a dual-branch feature extraction module to obtain low-frequency global features and high-frequency local features; then stitching the dual-modal features together and reconstructing the fused image using a dual-modal spatial adaptive feature fusion module.

[0004] The fused images generated by this method suffer from several drawbacks. First, the single network structure struggles to adequately represent the differentiated features of infrared and visible light modes. This is especially true when dealing with complex scenes, where thermal radiation features and texture details can easily interfere with each other. Second, the lack of a dynamic routing mechanism prevents the network from adaptively allocating computational resources to different regional features, resulting in high-value information (such as targets with strong heat) not being processed specifically. Finally, the full-scale computation mode causes significant computational redundancy, and the lack of expert division of labor makes the network decision-making process uninterpretable.

[0005] In summary, designing a novel multimodal image fusion method with a Moe expert mechanism that can improve the network's generalization ability, fully utilize information from different frequencies at multiple scales in the images, balance performance and model complexity, and leverage the differences and commonalities between source images to achieve better fusion results is a pressing issue in this field. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] To address the shortcomings of existing technologies, this invention provides a multimodal image fusion method based on Swin-MOE, which solves the problems mentioned in the background section.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0010] A multimodal image fusion method based on Swin-MOE includes the following steps:

[0011] Step 1: Prepare training data; Select multifocal dataset, multi-exposure dataset, infrared image dataset and visible light image dataset, divide the dataset into training set, validation set and test set, and preprocess the original images and their corresponding labels in multifocal dataset and multi-exposure dataset;

[0012] Step 2: Construct the network model; the network model includes an encoder, a dual-channel adaptive fusion module, and a decoder; the encoder takes two source images as input and decomposes the source images into low-frequency features and high-frequency feature components through continuous downsampling; the dual-channel adaptive fusion module is used to effectively fuse the extracted features; the decoder is used for image reconstruction and image fusion, and the structure of the decoder is completely symmetrical with that of the encoder;

[0013] Step 3: First stage of network training: Supervised pre-training for multiple tasks using labeled data from the multi-exposure and multi-focus datasets to enhance the model's ability to extract complementary features from the multi-exposure and multi-focus tasks;

[0014] Step 4: Second stage of network training: Unsupervised training is performed using unlabeled data from infrared and visible light image datasets to enhance the model's generalization ability while saving data costs; supervised training is performed using labeled data from multi-exposure and multi-focus datasets to enhance the model's ability to preserve source image details.

[0015] Step 5: Determine the evaluation metrics: Select a loss function to evaluate the quality of image fusion, and determine the evaluation metrics to evaluate the network performance;

[0016] Step 6: Solidify the network model: After completing the network model adjustment, fix the network parameters and determine the final multimodal image fusion model. If image fusion tasks are needed in the future, directly input the image to be fused into the network model to obtain the fusion result.

[0017] Furthermore, in step 2, the encoder structure includes an initial convolutional layer and four cascaded hybrid block modules. The convolutional layer is responsible for primary feature extraction, while each hybrid block gradually acquires multi-scale frequency domain representations of the image through multi-level feature extraction and downsampling operations. Each hybrid block consists of convolutional layer one, a Swintransformer block, and convolutional layer two connected by residuals. Correspondingly, the decoder adopts a reverse architecture. The Moe hybrid layer with the Moe expert mechanism recovers image details through feature upsampling and layer-by-layer reconstruction. The Moe hybrid layer consists of convolutional layer one, a Swintransformer block, and convolutional layer two. This symmetrical encoder-decoder design forms a complete Laplacian pyramid processing framework, realizing multi-resolution feature processing from coarse to fine granularity. After obtaining shallow features, the encoder continuously downsamples the features to obtain features of different frequencies.

[0018] Furthermore, in step 2, the dual-channel adaptive fusion module consists of two convolutional layers and a dual-channel attention fusion module. The dual-channel attention fusion module consists of channel self-attention branches and spatial self-attention branches, which can compute the global context information of the spatial domain and channel domain in parallel. The computation process can be represented as follows:

[0019] f(x)=f CA (x)+f SA (x);

[0020] Wherein, the input feature x∈R C×H×W f(x) represents the generated attention map, f CA (x) represents the global channel feature context, f SA (x) represents the global spatial feature context.

[0021] To effectively acquire information features from different image channels, a dual-channel attention fusion module was constructed. This architecture uses the frequency features decomposed by the encoder, such as... and The input is the dual-channel fused feature, and the output is the dual-channel fused feature.

[0022] Channel self-attention in the dual-channel attention fusion module can be represented as:

[0023]

[0024] Where i is the query location index, j enumerates all possible locations, and N = C × H × W represents the total number of locations in the feature; Represents context modeling unrelated to the query, where weights Obtained through global attention pooling; Wk, Wv1, and Wv2 are 1×1 convolutions, LN(·) denotes layer normalization, ρ is the ReLU function, δ is the Sigmoid function, ⊙ cm It is a channel-wise multiplication operator; specifically: (a) through W k (a) Obtain global attention weights using the Softmax function, and use global attention pooling to obtain shared global context features; (b) By W v2 ρ(LN(W v1 (·))) The bottleneck transformation is used to capture the inter-channel dependencies; (c) The normalized channel context features are weighted onto each channel feature;

[0025] Spatial self-attention in the dual-channel attention fusion module can be represented as:

[0026]

[0027] in, Here, g(·) represents the global attention pooling weights, and g(·) represents the global average pooling weights. ⊙ sm For spatial multiplication, λ is the weight coefficient learned from 0; specifically: (a) spatial context features are obtained through global average pooling and the · function; (b) after multiplying by the weight factor λ, the normalized spatial context information is aggregated into the features at each position;

[0028] The overall process of the dual-channel adaptive fusion module is as follows: Given input features First, the global context information of the input features is modeled using residual blocks in a dual-channel attention fusion module, and these residual blocks are summed to generate initial fused features. Then, a soft selection mechanism is used to weight the input features to adaptively learn their relative importance. Specifically, the dual-channel attention fusion module adjusts the key complementary information in the initial fused features to generate normalized attention fusion weights β. The network then fuses the input features based on weights β and 1-β. and The dominant information; this adaptive weighting process enables the model to dynamically learn the relative importance of multimodal features, as expressed below:

[0029]

[0030] in, For the final fusion features, It represents the initial fusion feature, δ is the Sigmoid function, and ⊙ represents element-wise multiplication.

[0031] Furthermore, in step 2, the decoder consists of six Moe mixing layers that continuously upsample and perform residual connections on the image before outputting the fused image. Each Moe mixing layer comprises two 3×3 convolutional layers and a Swin-MOE block. The Swin-MOE includes an LN layer, a SW-Moe module, W-MSA, and SW-MSA. The specific workflow of the Swin-MOE is as follows:

[0032] z l =W-MSA(LN(z) l-1 ))+z l-1 ;

[0033]

[0034] Where, z and These represent the W / SW-MSA and SW-Moe characterizations of the l-th block, respectively.

[0035] The SW-Moe block consists of a gated network, a task sharing adapter, and a fusion layer. First, F1 and F2 are concatenated as the feature representation of the multi-source token pair.

[0036] Φ = L(Cat(F1,F2));

[0037] Where L(·) contains linear and normalization layers for dimensionality reduction, and F1 and F2 represent source images of different modalities, respectively; based on the task to which Φ belongs, a specific task router is selected from the gating network:

[0038]

[0039] In this method, TopK(·) retains the first K (K=2) values, and the rest are set to -∞. g and W noise These are learnable parameters; after weighted summation by the prompt-driven layer, the prompt is obtained:

[0040]

[0041] in, By prompt1 and Composition, with a value range of (0,1); then, after the prompt and multiplication with multi-source features, a source image correlation bias is added:

[0042] h1 = prompt1·f1 + S1;

[0043] h2 = prompt2·f2 + S2;

[0044] Where S1 and S2 represent source image embeddings of different modalities, the final output is:

[0045] f = F(h1 + h2);

[0046] Finally, a fused image is output through a 3×3 convolutional layer;

[0047] All hybrid blocks have the same structure. A hybrid block consists of a convolutional layer, a SwinTransformer block, and convolutional layer residual connections. The kernel size of all convolutional layers is 3×3, the depth of the SwinTransformer block is 2, the window size is 2, and the number of heads for multi-head attention is 6.

[0048] (III) Beneficial Effects

[0049] Compared with existing technologies, this invention provides a multimodal image fusion method based on Swin-MOE, which has the following beneficial effects:

[0050] 1. This invention innovatively constructs a multimodal fusion architecture based on Swin-MOE. Through the collaborative design of a dual-channel adaptive attention fusion network and a hybrid expert decoder, and by making full use of the gradient image information of the input image and the detailed information of the original image, the fusion effect is improved. It not only realizes the complementary fusion of features of multi-source images (infrared / visible light, multi-exposure / multi-focus), but also breaks through the shortcomings of traditional models that are limited to a single task, enabling the model to have the ability to process multimodal image fusion and improving the model's wide applicability.

[0051] 2. The adaptive frequency fusion module proposed in this invention innovatively achieves three-dimensional collaborative optimization of multi-scale features through a spatial-channel dual attention mechanism. By utilizing the multi-channel attention mechanism, it obtains the channel information and spatial information of the source image, thereby improving the feature extraction capability of the network while maintaining their respective characteristics, thus effectively improving the image fusion effect.

[0052] 3. This invention proposes a two-stage training approach. The first stage uses supervised learning to enable the model to master the basic feature representations of image fusion, laying a stable foundation for subsequent training and effectively reducing the volatility of the model in the initial training phase. The second stage introduces an unsupervised learning mechanism, cleverly circumventing the problem of scarce infrared and visible light image annotation data. Simultaneously, through domain adaptation training, the model can autonomously adapt to the differences in data distribution across different imaging modalities (such as infrared and visible light, multifocal and multi-exposure), significantly enhancing the model's cross-modal generalization performance. In particular, the second stage innovatively incorporates an appropriate amount of supervised multifocal and multi-exposure data as a regularization signal, ensuring the stability of unsupervised training while effectively preventing model overfitting. Attached Figure Description

[0053] Figure 1This is a flowchart illustrating the steps of the multimodal image fusion method based on Swin-MOE of the present invention.

[0054] Figure 2 This is a schematic diagram of the overall structure of the network model of the present invention;

[0055] Figure 3 This is a schematic diagram illustrating the structure of the mixing block of the present invention;

[0056] Figure 4 This is a schematic diagram of the structure of the Moe hybrid layer of the present invention;

[0057] Figure 5 This is a schematic diagram of the dual-channel adaptive fusion module of the present invention;

[0058] Figure 6 This is a schematic diagram of the dual-channel attention fusion module of the present invention;

[0059] Figure 7 This is a flowchart illustrating the structure of the SW-Moe of the present invention;

[0060] Figure 8 This is a schematic diagram of the SW-Moe network structure of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Example

[0063] like Figure 1-8 As shown in the figure, an embodiment of the present invention proposes a multimodal image fusion method based on Swin-MOE, which specifically includes the following steps:

[0064] Step 1: Preparing Training Data: This study uses multi-source datasets for model training. For the multifocal image fusion task, RealMFF (710 real samples) and MFI-WHU (190 samples) datasets were selected. To balance the data distribution, 96 samples from each dataset were randomly selected (virtual samples were used for MFI-WHU) for training. For multi-exposure image fusion, the SICE dataset was used, with 192 samples randomly selected from 229 samples, and the image resolution set to 256×256 to optimize computational efficiency. For the visible light and infrared image fusion task, the RoadScene (approximately 1000 pairs) and TNO (approximately 2400 pairs) datasets were used, with a training resolution set to 64×64. All multifocal and multi-exposure data were used for supervised training, while infrared fusion data was used for unsupervised training. This data selection strategy ensures sample balance and effectively avoids the problem of model overfitting to specific datasets.

[0065] Step 2: Construct the network model: such as Figure 2 As shown, the network model mainly consists of an encoder for feature extraction, a dual-channel adaptive fusion module, and a decoder for generating reconstructed images. The encoder takes two source images as input and decomposes them into low-frequency and high-frequency feature components through continuous downsampling. The dual-channel adaptive fusion module is used to effectively fuse the extracted features. The decoder is used for image reconstruction and image fusion, and its structure is completely symmetrical to that of the encoder.

[0066] The encoder structure comprises an initial convolutional layer and four cascaded hybrid block modules. The convolutional layer is responsible for initial feature extraction, while each hybrid block progressively acquires a multi-scale frequency domain representation of the image through multi-level feature extraction and downsampling operations. The hybrid blocks are as follows: Figure 3 As shown, it consists of convolutional layer one, a Swintransformer block, and convolutional layer two connected by residuals. Correspondingly, the decoder adopts an inverse architecture; the Moe hybrid layer with Moe expert mechanism recovers image details through feature upsampling and layer-by-layer reconstruction. The Moe hybrid layer is shown below. Figure 4 As shown, it consists of convolutional layer one, SW-Moe block, and convolutional layer two. This symmetrical encoder-decoder design forms a complete Laplacian pyramid processing framework, enabling multi-resolution feature processing from coarse to fine granular.

[0067] After obtaining shallow features, the encoder continuously downsamples the features to obtain features of different frequencies. The decoder's structure is completely symmetrical to the encoder's;

[0068] Dual-channel adaptive fusion module, such as Figure 5As shown, the dual-channel attention fusion module consists of a channel self-attention branch and a spatial self-attention branch, which can compute the global context information of the spatial domain and the channel domain in parallel. The computation process can be represented as follows:

[0069] f(x)=f CA (x)+f SA (x);

[0070] Wherein, the input feature x∈R C×H×W f(x) represents the generated attention map, f CA (x) represents the global channel feature context, f SA (x) represents the global spatial feature context.

[0071] To effectively acquire information features from different image channels, a dual-channel attention fusion module was constructed, such as... Figure 6 As shown, this architecture uses the frequency features decomposed by the encoder as follows: and The input is the dual-channel fused feature, and the output is the dual-channel fused feature.

[0072] Channel self-attention in the dual-channel attention fusion module can be represented as:

[0073]

[0074] Where i is the query location index, j enumerates all possible locations, and N = C × H × W represents the total number of locations in the feature. Represents context modeling unrelated to the query, where weights Obtained through global attention pooling. Wk, Wv1, and Wv2 are 1×1 convolutions, LN(·) denotes layer normalization, ρ is the ReLU function, δ is the Sigmoid function, and ⊙ cm It is a channel-wise multiplication operator. Specifically: (a) through W k (a) Obtain global attention weights using the Softmax function, and use global attention pooling to obtain shared global context features; (b) By W v2 ρ(LN(W v1 (·))) The bottleneck transformation captures the inter-channel dependencies; (c) The normalized channel context features are weighted onto each channel feature.

[0075] Spatial self-attention in the dual-channel attention fusion module can be represented as:

[0076]

[0077] in, Here, g(·) represents the global attention pooling weights, and g(·) represents the global average pooling weights. ⊙sm For spatial multiplication, λ is the weight coefficient learned starting from 0. Specifically: (a) spatial context features are obtained through global average pooling and the · function; (b) after multiplying by the weight factor λ, the normalized spatial context information is aggregated into the features at each location.

[0078] The overall process of the adaptive fusion module is as follows. Given input features... First, the residual blocks of the dual-channel attention fusion module are used to model the global contextual information of the input features and sum them to generate initial fused features. Then, a soft selection mechanism is used to weight the input features to adaptively learn their relative importance. The specific process is as follows: the dual-channel attention fusion module adjusts the key complementary information in the initial fused features to generate normalized attention fusion weights β. The network then fuses the input features based on weights β and 1-β. and The dominant information. This adaptive weighting process enables the model to dynamically learn the relative importance of multimodal features, as expressed below:

[0079]

[0080] in, For the final fusion features, It represents the initial fusion feature, δ is the Sigmoid function, and ⊙ represents element-wise multiplication.

[0081] The decoder consists of six Moe mixing layers that continuously upsample and perform residual connections on the image before outputting the fused image. Each Moe mixing layer comprises two 3×3 convolutional layers and a Swing-MOE block, where the Swing-MOE is as follows: Figure 7 As shown, it includes the LN layer, SW-Moe module, W-MSA, and SW-MSA. The specific workflow of Swin-MOE is as follows:

[0082] z l =W-MSA(LN(z) l-1 ))+z l-1 ;

[0083]

[0084] Where, z and These represent the W / SW-MSA and SW-Moe characterizations of the l-th block, respectively.

[0085] SW-Moe blocks such as Figure 8 As shown, it consists of a gating network, a task sharing adapter, and a fusion layer. First, F1 and F2 are concatenated as the feature representation of the multi-source token pair:

[0086] Φ = L(Cat(F1,F2));

[0087] L(·) contains a linear layer and a normalization layer for dimensionality reduction, and F1 and F2 represent source images of different modalities, respectively.

[0088] Based on the task to which Φ belongs, select the specific task router from the gated network:

[0089]

[0090] In this method, TopK(·) retains the first K (K=2) values, and the rest are set to -∞. g and W noise These are learnable parameters.

[0091] After weighted summation in the driver layer, the following prompt was obtained:

[0092]

[0093] in, By prompt1 and It consists of a value range of (0,1).

[0094] Then, after the prompt and multiplication with multi-source features, a source image-related bias is added:

[0095] h1 = prompt1·f1 + S1;

[0096] h2 = prompt2·f2 + S2;

[0097] Where S1 and S2 represent source image embeddings of different modalities, the final output is:

[0098] f = F(h1 + h2);

[0099] Finally, a fused image is output through a 3×3 convolutional layer.

[0100] All the mixed blocks have the same compositional structure, such as Figure 3 As shown, the hybrid block consists of convolutional layers, SwinTransformer blocks, and convolutional layer residual connections. All convolutional layers have a kernel size of 3×3, the SwinTransformer block has a depth of 2, a window size of 2, and 6 heads for multi-head attention.

[0101] Step 3: Perform the first stage of training. In the first stage of model training, we first perform joint optimization for two tasks: high-exposure image fusion and multi-focus image fusion. During the joint optimization process, a composite loss function is used for constraints. The composite loss function consists of two parts: the image fusion quality loss is measured using the Structural Similarity Index (SSIM), and the image reconstruction accuracy loss is evaluated using the L1 norm. Specifically, for the high-exposure image fusion task, its loss function can be formally expressed as:

[0102]

[0103] in It is a fusion function. Here, λ1 is the reconstructed source image, SSIM(·) is the SSIM function, and ||·||1 is the L1 norm. The loss function for the multifocal task is as follows:

[0104]

[0105] Step 4: Perform the second stage of network training. In the second stage, we first fix the encoder-decoder parameters trained in the first stage and retain its high-exposure and multi-focus fusion modules. Then, we introduce a second fusion module specifically for infrared and visible light images, employing a semi-supervised training strategy: on the one hand, we use unlabeled infrared-visible light image pairs for feature learning; on the other hand, we combine the supervised knowledge obtained in the first stage for joint optimization. This design balances the contributions of unsupervised and supervised learning, enabling the model to expand its infrared-visible light fusion capabilities while maintaining its original image fusion performance, achieving synergistic improvement in both new and old tasks.

[0106] The total loss function for the second stage of training is a weighted sum of a supervised loss function and an unsupervised loss function:

[0107] L S2 =L super +βL unsuper ;

[0108] Here, β is a positive tradeoff parameter. The loss function used in the first stage is directly adopted as the loss function for supervised training. Unsupervised learning loss functions also include image fusion loss functions and image reconstruction loss functions, as shown below:

[0109]

[0110] Where β1 is a weighting coefficient. The fusion loss mainly consists of three parts: intensity loss function, texture loss function, and structure loss function.

[0111]

[0112] Where I 1u and I 2u It is an unlabeled source image. This represents the corresponding fusion result, where β2 and β3 are weighting coefficients. The intensity loss can be calculated as:

[0113]

[0114] Where MAX(·) is the element-wise maximum operation. The texture loss in the gradient domain can be calculated as:

[0115]

[0116] Where |·| is the absolute value function, and ▽ represents the gradient operator. Simultaneously, the L1 norm reconstruction loss is minimized so that the model can decompose the unlabeled source image into multi-frequency features:

[0117]

[0118] in and It is a reconstructed, unlabeled source image.

[0119] The hyperparameters in the loss function are set as follows:

[0120] λ = 0, λ1 = 0.5, β = 0.1, β1 = 1.25, β2 = 1, β3 = 0.5. The initial learning rate for the first stage is set to 1 × 10^-4, which is halved after 200 training cycles. The initial learning rate for the second stage is set to 3 × 10^-5.

[0121] Step 5: In the image fusion performance evaluation, we used five key indicators for comprehensive quantitative analysis: Peak Signal-to-Noise Ratio (PSNR) measures the pixel-level fidelity between the fused image and the reference image; a higher value indicates less distortion. Structural Similarity Index (SSIM) evaluates visual quality from three dimensions: brightness, contrast, and structure; a value closer to 1 indicates higher similarity. Information Entropy (IE) reflects the richness of information in the image; a higher entropy value represents richer information. Mutual Information (MI) quantifies the ability of the fused image to retain features from the source image; a higher value indicates more complete feature transfer. Root Mean Square Error (RMSE) calculates the average level of pixel-level error; a lower value indicates higher reconstruction accuracy. These five indicators comprehensively evaluate the quality characteristics of the fused image from different perspectives. The definitions of Peak Signal-to-Noise Ratio, Structural Similarity Index, Information Entropy, Mutual Information, and RMSE are as follows:

[0122]

[0123]

[0124] Where, μ x μ y Let x and y represent the mean values ​​of the image, respectively. and Let σ represent the variances of the images x and y, respectively. xy p(x) is the covariance of the graphs x and y, where C1 and C2 are constants; i (x) is the value of each pixel in the image. i The probability distribution is given by n, where n is the total number of different pixel values ​​in the image; p(I(i,j), K(i,j)) is the joint probability distribution, and p(I(i,j)) and p(K(i,j)) are individual probability distributions; N and M are the dimensions of the image, and I(i,j) and K(i,j) are the pixel values ​​of the original image and the fused image, respectively.

[0125] Table 1 Performance comparison of different models

[0126] Model PSNR EN SSIM MI RMSE DenseFuse 30.5 6.2 0.91 1.15 0.45 FusionGan 28.7 5.9 0.89 1.20 0.48 RFN-Nest 29 6.0 0.90 1.18 0.47 U2fusion 27.4 5.7 0.86 1.10 0.50 SwinFusion 31.2 6.3 0.92 1.25 0.43 Ours 30.8 6.6 0.90 1.22 0.38

[0127] Table 1 presents the quantitative evaluation results of the four fusion methods on 20 images in the MSRS dataset. Experimental data show that the proposed method achieves the best values ​​in three metrics: information entropy (EN), structural similarity (SSIM), and root mean square error (RMSE), while ranking second in mutual information (MI) and peak signal-to-noise ratio (PSNR). Combined analysis of subjective visual evaluation and objective data comparison demonstrates that the proposed method significantly outperforms the comparative methods in both image quality perception and quantitative metrics, exhibiting the best overall fusion performance.

[0128] Step 6: Fix the network model: After adjusting the network model, fix the network parameters and determine the final multimodal image fusion model; if image fusion tasks are needed in the future, the images to be fused can be directly input into the network model to obtain the fusion results.

[0129] In the above embodiments, the implementations of convolution, SwingTransformer, activation functions, normalization, normalization exponential functions, matrix multiplication operations, and element-wise multiplication are algorithms well known to those skilled in the art, and the specific processes and methods can be found in relevant textbooks or technical documents.

[0130] Furthermore, the present invention can be implemented as a system, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0131] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for multimodal image fusion based on Swin-MOE, characterized in that: Includes the following steps: Step 1: Prepare training data; Select multifocal dataset, multi-exposure dataset, infrared image dataset and visible light image dataset, divide the dataset into training set, validation set and test set, and preprocess the original images and their corresponding labels in multifocal dataset and multi-exposure dataset; Step 2: Construct the network model; the network model includes an encoder, a dual-channel adaptive fusion module, and a decoder; the encoder takes two source images as input and decomposes the source images into low-frequency features and high-frequency feature components through continuous downsampling; the dual-channel adaptive fusion module is used to effectively fuse the extracted features. The decoder is used for image reconstruction and image fusion, and its structure is completely symmetrical to that of the encoder. Step 3: First stage of network training: Supervised pre-training for multiple tasks using labeled data from the multi-exposure and multi-focus datasets to enhance the model's ability to extract complementary features from the multi-exposure and multi-focus tasks; Step 4: Second stage of network training: Unsupervised training is performed using unlabeled data from infrared and visible light image datasets to enhance the model's generalization ability while saving data costs; supervised training is performed using labeled data from multi-exposure and multi-focus datasets to enhance the model's ability to preserve source image details. Step 5: Determine the evaluation metrics: Select a loss function to evaluate the quality of image fusion, and determine the evaluation metrics to evaluate the network performance; Step 6: Solidify the network model: After completing the network model adjustment, fix the network parameters and determine the final multimodal image fusion model. If image fusion tasks are needed in the future, directly input the image to be fused into the network model to obtain the fusion result.

2. The method for multimodal image fusion based on Swin-MOE according to claim 1, characterized in that: In step 2, the encoder structure includes an initial convolutional layer and four cascaded hybrid block modules. The convolutional layer is responsible for primary feature extraction, while each hybrid block gradually acquires multi-scale frequency domain representations of the image through multi-level feature extraction and downsampling operations. The hybrid block consists of convolutional layer 1, a Swintransformer block, and convolutional layer 2 connected by residuals. Correspondingly, the decoder adopts an inverse architecture. The Moe hybrid layer with Moe expert mechanism recovers image details by upsampling features and reconstructing them layer by layer. The Moe hybrid layer consists of convolutional layer 1, a Swintransformer block, and convolutional layer 2. This symmetrical encoder-decoder design forms a complete Laplacian pyramid processing framework, realizing multi-resolution feature processing from coarse to fine granular. After obtaining shallow features, the encoder continuously downsamples the features to obtain features of different frequencies.

3. The method for multimodal image fusion based on Swin-MOE according to claim 2, characterized in that: In step 2, the dual-channel adaptive fusion module consists of two convolutional layers and a dual-channel attention fusion module. The dual-channel attention fusion module consists of channel self-attention branches and spatial self-attention branches, which can compute global context information in the spatial and channel domains in parallel. The computation process can be represented as follows: f(x)=f CA (x)+f SA (x); Wherein, the input feature x∈R C×H×W f(x) represents the generated attention map, f CA (x) represents the global channel feature context, f SA (x) represents the global spatial feature context; To effectively acquire information features from different image channels, a dual-channel attention fusion module was constructed. This architecture uses the frequency features decomposed by the encoder, such as... and The input is the dual-channel fused feature, and the output is the dual-channel fused feature. Channel self-attention in the dual-channel attention fusion module can be represented as: Where i is the query location index, j enumerates all possible locations, and N = C × H × W represents the total number of locations in the feature; Represents context modeling unrelated to the query, where weights Obtained through global attention pooling; Wk, Wv1, and Wv2 are 1×1 convolutions, LN(·) denotes layer normalization, ρ is the ReLU function, δ is the Sigmoid function, ⊙ cm It is a channel-wise multiplication operator; specifically: (a) through W k The global attention weights are obtained using the Softmax function, and the shared global context features are obtained using global attention pooling. (b) By W v2 ρ(LN(W v1 (·))) The bottleneck transformation is used to capture the inter-channel dependencies; (c) The normalized channel context features are weighted onto each channel feature; Spatial self-attention in the dual-channel attention fusion module can be represented as: in, For global attention pooling weights, g(·) represents global average pooling: ⊙ sm For spatial multiplication, λ is the weight coefficient learned from 0; specifically: (a) spatial context features are obtained through global average pooling and the · function; (b) after multiplying by the weight factor λ, the normalized spatial context information is aggregated into the features at each position; The overall process of the dual-channel adaptive fusion module is as follows: Given input features First, the global context information of the input features is modeled using residual blocks in a dual-channel attention fusion module, and these residual blocks are summed to generate initial fused features. Then, a soft selection mechanism is used to weight the input features to adaptively learn their relative importance. Specifically, the dual-channel attention fusion module adjusts the key complementary information in the initial fused features to generate normalized attention fusion weights β. The network then fuses the input features based on weights β and 1-β. and The dominant information; this adaptive weighting process enables the model to dynamically learn the relative importance of multimodal features, as expressed below: in, For the final fusion features, It represents the initial fusion feature, δ is the Sigmoid function, and ⊙ represents element-wise multiplication.

4. The method for multimodal image fusion based on Swin-MOE according to claim 2, characterized in that: In step 2, the decoder consists of six Moe mixing layers that continuously upsample and perform residual connections on the image before outputting the fused image. Each Moe mixing layer comprises two 3×3 convolutional layers and a Swin-MOE block. The Swin-MOE includes an LN layer, a SW-Moe module, W-MSA, and SW-MSA. The specific workflow of the Swin-MOE is as follows: z l =W-MSA(LN(z l-1 ))+z l-1 ; Where, z and These represent the W / SW-MSA and SW-Moe characterizations of the l-th block, respectively. The SW-Moe block consists of a gated network, a task sharing adapter, and a fusion layer. First, F1 and F2 are concatenated as the feature representation of the multi-source token pair. Φ = L(Cat(F1,F2)); Where L(·) contains linear and normalization layers for dimensionality reduction, and F1 and F2 represent source images of different modalities, respectively; based on the task to which Φ belongs, a specific task router is selected from the gating network: In this method, TopK(·) retains the first K (K=2) values, and the rest are set to -∞. g and W noise These are learnable parameters; after weighted summation by the prompt-driven layer, the prompt is obtained: in, By prompt1 and Composition, with a value range of (0,1); then, after the prompt and multiplication with multi-source features, a source image correlation bias is added: h1 = prompt1·f1 + S1; h2 = prompt2·f2 + S2; Where S1 and S2 represent source image embeddings of different modalities, the final output is: f = F(h1 + h2); Finally, a fused image is output through a 3×3 convolutional layer; All hybrid blocks have the same structure. A hybrid block consists of a convolutional layer, a SwinTransformer block, and convolutional layer residual connections. The kernel size of all convolutional layers is 3×3, the depth of the SwinTransformer block is 2, the window size is 2, and the number of heads for multi-head attention is 6.

Images