A multi-modal image fusion method based on SwinTransformer

Abstract

The application discloses a kind of based on Swi nTransformer's multi-modal image fusion method, including the following steps: preparation training data set, data set is divided, and is divided into training set, verification set and test set and is preprocessed;Network model including two encoders, a fusion module and a decoder is constructed;The network model is trained in first stage using prepared high exposure and multi-focus training data set, until reaching preset threshold;The network model is trained again using infrared and visible light data set fine-tuning, obtains final model;The parameter of final model is obtained and solidified, saves model.The application designs new neural network framework, solves the existing image fusion model mainly concentrates on solving single task fusion problem, ignores the potential possibility of information complementation in multi-task fusion, the model not only promotes the collaborative training of multiple fusion tasks, but also learns effective information complementation between the data sets of different fusion tasks.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to a multimodal image fusion method based on SwinTransformer. Background Technology

[0002] Image fusion plays a crucial role in enhancing the quality and accuracy of semantic segmentation, which is particularly critical for autonomous driving systems. By fusing information from multiple imaging sensors or modalities (such as infrared and visible light images), image fusion enriches the data, thereby improving the perception capabilities of autonomous vehicles. However, existing image fusion methods primarily focus on solving single-task fusion problems, neglecting the potential for information complementarity in multi-task fusion. Therefore, how to perform multimodal image fusion and explore information complementarity in multi-task fusion remains a challenge.

[0003] The Chinese patent announcement number is "CN117173525B", entitled "A General Multimodal Image Fusion Method and Apparatus". This method first uses convolution to extract shallow information, then extracts multi-scale features through an encoder constructed from i+1 sequentially connected Transformer modules, each followed by a wavelet downsampling module. The decoder consists of i sequentially connected upsampling modules, each followed by a Transformer module, used to generate a high-quality fused image. While this method uses wavelet downsampling to obtain multi-scale image features, this operation can cause image distortion in certain frequency ranges, especially high-frequency components, and can lead to loss of detail information in low-frequency regions. To improve the network's extraction of global information, Transformer modules are used, which increases computational overhead and resource requirements, and weakens the ability to perceive local structural information. Furthermore, simply stitching two images together before feeding them into the model results in the loss of spatial relationships, insufficient feature fusion, and data redundancy.

[0004] In summary, designing a novel multimodal image fusion method that can improve the generalization ability of the network, fully utilize the information of different frequencies at multiple scales in the images, balance performance and model complexity, and achieve better fusion by leveraging the differences and commonalities between source images is a problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The technical solution of this invention to solve the above-mentioned technical problems is to provide a multimodal image fusion method based on SwinTransformer, comprising the following steps:

[0006] Step 1, Prepare training data: Select multifocal dataset, multi-exposure dataset, infrared image dataset and visible light image dataset, divide the dataset into training set, validation set and test set, and preprocess the original images and their corresponding labels in multifocal dataset and multi-exposure dataset.

[0007] Step 2, construct the network model: including an image receiving module for receiving image data, a shallow feature extraction module for extracting coarse shallow information of the image, a global feature extraction module for extracting fine information with global information, an adaptive frequency fusion module for fusing the difference information and common information of the image, and an image reconstruction module for reconstructing the fused image; the global feature extraction module includes two Laplacian pyramid encoders with the same structure, and the image reconstruction module includes a decoder with an inverse Laplacian pyramid structure;

[0008] Step 3, First stage of network training: Supervised pre-training for multiple tasks using labeled data from the multi-exposure and multi-focus datasets to enhance the model's ability to extract complementary features from the multi-exposure and multi-focus tasks;

[0009] Step 4, Second stage of network training: Unsupervised training is performed using unlabeled data from infrared and visible light image datasets to enhance the model's generalization ability while saving data costs; supervised training is performed using labeled data from multi-exposure and multi-focus datasets to enhance the model's ability to preserve source image details.

[0010] Step 5, determine the evaluation metrics: select a loss function to evaluate the quality of image fusion, and determine the evaluation metrics to evaluate the network performance.

[0011] Furthermore, the multimodal image fusion method based on SwinTransformer also includes the following steps: Step 6, fix the network model: After completing the network model adjustment, fix the network parameters and determine the final multimodal image fusion model; if image fusion tasks are needed in the future, the image to be fused can be directly input into the network model to obtain the fusion result.

[0012] Further, in step 2, after receiving two types of image data, the image input module sends them to the shallow feature extraction module to perform a 3×3 convolution operation on the input image to obtain local information and shallow features of the image; the shallow information is then sent to the global feature extraction module; the global feature extraction module includes two Laplacian pyramid encoders with the same structure, which continuously downsample and perform Laplacian difference on the image, decompose it into feature maps of different frequencies, and then send them to the adaptive frequency fusion module; the adaptive frequency fusion module consists of an image embedding module, a difference information fusion module, two common information fusion modules, and a refining module, which fuses images of the same frequency, and then sends the fused images to the image reconstruction module for reconstruction; the image reconstruction module includes a decoder with an inverse Laplacian pyramid structure, which continuously upsamples and performs residual connection on the image before outputting the fused image.

[0013] Furthermore, in step 2, the encoder and decoder form a Laplace pyramid structure; the encoder obtains the lowest-scale features. Then, upsampling is performed, and residuals are calculated with features of the same scale to generate high-frequency features. The decoder's structure is completely symmetrical to the encoder's;

[0014] The adaptive frequency fusion module includes an image embedding module, a difference information fusion module, two common information fusion modules, and a refinement module. The image embedding module segments the image into patches, performs linear projection, and then introduces position encoding, outputting a sequence containing patch embedding vectors as input to the subsequent model. The embedding process can be represented as:

[0015]

[0016] A cross-attention architecture is introduced to construct a differential information fusion module, using the frequency features decomposed by the encoder, such as... and As input, output differential information features; and The feature segments are divided into n local feature fragments as shown below:

[0017]

[0018] in and And s = h × w;

[0019] Using a linear layer to transform feature fragments into query Q, key K, and value V, the linear projection can be represented as:

[0020] Q i =LinearQ(Q i ),K i =LinearK(K) i),V i =LinearV(V i );

[0021] Where i = 1, ..., n, and Linear(·) represents the linear projection operation shared between different segments;

[0022] The similarity matrix between query Q and key K is calculated using a dot product attention layer, and then multiplied by the value V to infer the relevance between Q and V; expressed as:

[0023]

[0024] Where, d k It is a scaling factor that can alleviate the problem of the softmax function converging to the region of minimum gradient as the dot product increases;

[0025] The difference between Q and V is obtained by removing common information; represented as:

[0026] DV = Linear(V - CMV);

[0027] Inject the difference information into Q; represented as:

[0028]

[0029] Where LN(·) represents the normalization layer, MLP(·) represents the multilayer perceptron, and F dm It is the output of the difference information fusion module;

[0030] Two identical information fusion modules are introduced, and the output fragment of the difference information fusion module is used to provide Q, while The fragment provides K and V, therefore F dm and The shared information between them can be represented as:

[0031]

[0032] Then F dm and Public information between CM and F dm , represented as:

[0033]

[0034] Among them, F cm This represents the output of the first public information fusion module;

[0035] F cm and Public information between them is injected into the fusion features to further enrich the fusion features;

[0036] The fused features are input into the refining module for further feature fusion. The refining module consists of convolutional layer 1, a mixing block, and convolutional layer 2, which are connected by residuals and then passed through Solftmax to output the fused result.

[0037] Compared with the prior art, this application has the following beneficial effects:

[0038] 1. This invention designs a novel image fusion framework with a dual-decoder Laplacian pyramid structure in the image processing module. It makes full use of the gradient image information of the input image and the detailed information of the original image to improve the fusion effect, and utilizes the potential information complementarity between different fusion tasks, so that the model can have the ability to process multimodal image fusion and improve the model's wide applicability.

[0039] 2. This invention designs an adaptive frequency fusion module, which includes a difference information fusion module and a common information fusion module in the image fusion module. By using a cross-attention mechanism, the difference features and common features of the two images are obtained. While maintaining their respective characteristics, the feature extraction capability of the network is improved, thereby effectively improving the image fusion effect.

[0040] 3. This invention proposes a two-stage training method. The first stage of supervised training helps the model learn some basic characteristics of image fusion, providing a good foundation for the second stage of training and reducing model instability in the initial stage. The second stage uses unsupervised training, which can effectively solve the problem of obtaining high-quality labels for infrared and visible light image data. In addition, infrared and visible light images and multi-focus and multi-exposure data have different distributions. Unsupervised training can help the model adapt to new modalities and characteristics, improving the model's generalization ability. A portion of supervised multi-focus and multi-exposure data is incorporated into the unsupervised training in the second stage as a positive supervision signal to stabilize the unsupervised training process and avoid overfitting. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the steps of the multimodal image fusion method based on SwinTransformer described in this invention.

[0043] Figure 2This is a schematic diagram of the overall structure of the network model described in this invention;

[0044] Figure 3 This is a schematic diagram of the adaptive frequency fusion module described in this invention;

[0045] Figure 4 This is a schematic diagram illustrating the structure of the difference information fusion module described in this invention;

[0046] Figure 5 This is a schematic diagram illustrating the structure of the public information fusion module described in this invention;

[0047] Figure 6 This is a schematic diagram of the refining module described in this invention;

[0048] Figure 7 This is a schematic diagram of the structure of the hybrid block described in this invention. Detailed Implementation

[0049] This invention proposes a multimodal image fusion method based on SwinTransformer, which aims to...

[0050] The multimodal image fusion method based on SwinTransformer proposed in this invention will be described below in specific embodiments:

[0051] Example 1:

[0052] A multimodal image fusion method based on SwingTransformer, such as Figure 1 As shown, it includes the following steps:

[0053] Step 1, Prepare training data: Select multifocal dataset, multi-exposure dataset, and infrared and visible light image dataset. Divide the dataset into training set, validation set and test set. Preprocess the original images and their corresponding labels in the multifocal and multi-exposure training sets.

[0054] Step 2, construct the network model: including an image receiving module for receiving image data, a shallow feature extraction module for extracting coarse shallow information of the image, a global feature extraction module for extracting fine information with global information, an adaptive frequency fusion module for fusing the difference information and common information of the image, and an image reconstruction module for reconstructing the fused image; the global feature extraction module includes two Laplacian pyramid encoders with the same structure, and the image reconstruction module includes a decoder with an inverse Laplacian pyramid structure;

[0055] Specifically, after receiving two types of image data, the image input module sends them to the shallow feature extraction module to perform a 3×3 convolution operation on the input image to obtain local information and shallow features of the image. The shallow information is then sent to the global feature extraction module. The global feature extraction module includes two Laplacian pyramid encoders with the same structure, which continuously downsample and perform Laplacian difference on the image to decompose feature maps of different frequencies, and then send them to the adaptive frequency fusion module. The adaptive frequency fusion module consists of an image embedding module, a difference information fusion module, two common information fusion modules, and a refining module. It fuses images of the same frequency, and the fused images are then sent to the image reconstruction module for reconstruction. The image reconstruction module includes a decoder with an inverse Laplacian pyramid structure, which continuously upsamples and performs residual connections on the image before outputting the fused image.

[0056] Step 3, First stage of network training: Supervised pre-training for multiple tasks using labeled data from the multi-exposure and multi-focus datasets to enhance the model's ability to extract complementary features from the multi-exposure and multi-focus tasks;

[0057] Specifically, supervised pre-training for multiple tasks is performed using labeled data from multi-exposure and multi-focus tasks to enhance the model's ability to extract complementary features from these tasks. Furthermore, labeled data helps the model converge better at this stage. A composite loss function for image fusion and image reconstruction is selected. For image fusion, the Structural Similarity Index (SSIM) loss is chosen, and the L1 norm is used as the reconstruction loss. The total loss is calculated as a weighted combination of the above losses. Network training begins, minimizing the fusion result loss function value until the number of training iterations reaches an initially set threshold or the loss function value reaches a preset range. At this point, the network model is considered to be trained successfully, and the network model parameters are saved.

[0058] Step 4, Second stage of network training: Unsupervised training is performed using unlabeled data from infrared and visible light image datasets to enhance the model's generalization ability while saving data costs; supervised training is performed using labeled data from multi-exposure and multi-focus datasets to enhance the model's ability to preserve source image details.

[0059] Specifically, semi-supervised fine-tuning is performed using labeled and unlabeled data. Visible and infrared image data (unlabeled) are used to train the infrared and visible image fusion. Then, the first-stage multi-task supervised training is still utilized to enhance unsupervised learning. Unsupervised training improves the model's generalization ability in tasks with scarce labeled data, such as infrared and visible light fusion. Simultaneously, supervised training ensures the model's ability to retain detailed information, thus aiding learning on unlabeled data. The total loss function in the second stage is a weighted sum of a supervised loss function and an unsupervised loss function. The supervised loss function directly uses the loss function from the first stage. The unsupervised loss function includes both image fusion and image reconstruction loss functions. The fusion loss comprises intensity, texture, and outcome loss functions, which are weighted together. The reconstruction loss function uses the L1 norm.

[0060] Step 5, determine the evaluation metrics: select a loss function to evaluate the quality of image fusion, and determine the evaluation metrics to evaluate the network performance.

[0061] Step 6, solidify the network model: After completing the network model adjustment, fix the network parameters and determine the final multimodal image fusion model; if image fusion tasks are needed in the future, the images to be fused can be directly input into the network model to obtain the fusion results.

[0062] Furthermore, in step 2, the encoder and decoder form a Laplace pyramid structure; the encoder obtains the lowest-scale features. Then, upsampling is performed, and residuals are calculated with features of the same scale to generate high-frequency features. The decoder's structure is completely symmetrical to the encoder's.

[0063] The adaptive frequency fusion module includes an image embedding module, a difference information fusion module, two common information fusion modules, and a refinement module. The image embedding module segments the image into patches, performs linear projection, and then introduces position encoding, outputting a sequence containing patch embedding vectors as input to the subsequent model. The embedding process can be represented as:

[0064]

[0065] A cross-attention architecture is introduced to construct a differential information fusion module, using the frequency features decomposed by the encoder, such as... and As input, output differential information features; and The feature segments are divided into n local feature fragments as shown below:

[0066]

[0067] in and And s = h × w;

[0068] Using a linear layer to transform feature fragments into query Q, key K, and value V, the linear projection can be represented as:

[0069] Q i =LinearQ(Q i ),K i =LinearK(K) i ),V i =LinearV(V i );

[0070] Where i = 1, ..., n, and Linear(·) represents the linear projection operation shared between different segments;

[0071] The similarity matrix between query Q and key K is calculated using a dot product attention layer, and then multiplied by the value V to infer the relevance between Q and V; expressed as:

[0072]

[0073] Where, d k It is a scaling factor that can alleviate the problem of the softmax function converging to the region of minimum gradient as the dot product increases;

[0074] The difference between Q and V is obtained by removing common information; represented as:

[0075] DV = Linear(V - CMV);

[0076] Inject the difference information into Q; represented as:

[0077]

[0078] Where LN(·) represents the normalization layer, MLP(·) represents the multilayer perceptron, and F dm It is the output of the difference information fusion module;

[0079] Two identical information fusion modules are introduced, and the output fragment of the difference information fusion module is used to provide Q, while The fragment provides K and V, therefore F dm and The shared information between them can be represented as:

[0080]

[0081] Then F dm and Public information between CM and Fdm , represented as:

[0082]

[0083] Among them, F cm This represents the output of the first public information fusion module;

[0084] F cm and Public information between them is injected into the fusion features to further enrich the fusion features;

[0085] The fused features are input into the refining module for further feature fusion. The refining module consists of convolutional layer 1, a mixing block, and convolutional layer 2, which are connected by residuals and then passed through Solftmax to output the fused result.

[0086] Example 2:

[0087] A multimodal image fusion method based on SwinTransformer specifically includes the following steps:

[0088] Step 1, Prepare Training Data: The multifocal datasets selected are RealMFF and MFI-WHU. The MFI-WHU dataset contains 190 samples, from which 96 virtual samples are randomly selected for training. The RealMFF dataset contains 710 real samples. Since the sample distributions in the two datasets are very different, 96 samples are randomly selected from the RealMFF dataset for training. This ensures that the number and distribution of samples in both datasets are roughly equivalent, preventing the model trained using these samples from overfitting to a specific dataset. The multi-exposure dataset is the SICE dataset, which contains 229 samples. 192 samples are randomly selected for training. To save computational resources, the image resolution for the first stage of training is limited to 256×256. These datasets are used for supervised training. The datasets used for the visible light and infrared image fusion task are the RoadScene and TNO datasets. The RoadScene dataset contains nearly 1,000 pairs of infrared and visible light image data, and the TNO dataset contains about 2,400 pairs of infrared and visible light image data. Similarly, in order to save computational resources, the image resolution for the second stage of training is limited to 64×64. These data are used for unsupervised training.

[0089] Step 2, construct the network model: such as Figure 2As shown, the network model mainly includes two encoders, an adaptive frequency fusion module, and a shared decoder. The two encoders take two source images as input and decompose the source images into low-frequency features and high-frequency feature components. The adaptive frequency fusion module is used to effectively fuse the extracted features. The shared decoder is used for image reconstruction and image fusion. The structure of the decoder is completely symmetrical with that of the encoder.

[0090] The encoder consists of convolutional layer 1, blending block 1, blending block 2, blending block 3, and blending block 4. The convolutional layers are used to extract shallow features, and the blending blocks repeatedly extract features from the source image and downsample them to obtain frequency features at different levels. The two encoders have completely identical structures. Conversely, the decoder extracts features layer by layer from the encoder's features and upsamples them to reconstruct the image. The encoder and decoder form a Laplacian pyramid structure.

[0091] The encoder obtains features at the lowest scale. Then, upsampling is performed, and residuals are calculated with features of the same scale to generate high-frequency features. The decoder's structure is completely symmetrical to the encoder's;

[0092] Adaptive frequency fusion module, such as Figure 3 As shown, the adaptive frequency fusion module consists of an image embedding module, a difference information fusion module, two common information fusion modules, and a refining module. The image embedding module segments the image into patches, performs linear projection, and then introduces position encoding, outputting a sequence containing patch embedding vectors as input to the subsequent model. The embedding process can be represented as:

[0093]

[0094] To effectively capture the differential features between images, a cross-attention architecture was introduced, and a differential information fusion module was constructed, such as... Figure 4 As shown, this architecture uses the frequency features decomposed by the encoder as... and The input is used as input, and the output is the difference information feature. Specifically, it will... and The feature segments are divided into n local feature fragments as shown below:

[0095] in and And s = h × w.

[0096] Next, a linear layer is used to transform these feature fragments into a query Q, a key K, and a value V. The linear projection can be represented as:

[0097] Q i=LinearQ(Q i ),K i =LinearK(K) i ),V i =LinearV(V i )

[0098] Where i = 1, ..., n, and Linear(·) represents the linear projection operation shared between different segments.

[0099] To extract common information from the features of two images and consider long-distance feature relationships, a dot-product attention layer is used to compute the similarity matrix between query Q and key K, which is then multiplied by the value V to infer the relevance between Q and V. This process can be represented as:

[0100]

[0101] Where, d k It is a scaling factor that can alleviate the problem that the softmax function converges to the region of minimum gradient when the dot product increases.

[0102] Then, the differences between Q and V are obtained by removing common information. This process can be represented as:

[0103] DV = Linear(V - CMV);

[0104] In order to Obtain complementary information and inject the difference information into Q, specifically as follows:

[0105]

[0106] Where LN(·) represents the normalization layer, MLP(·) represents the multilayer perceptron, and F dm It is the output of the difference information fusion module.

[0107] Simply extracting the difference information from the feature maps can easily lead to the loss of background details. Therefore, a fusion module for two identical information maps is introduced, such as... Figure 5 As shown. Specifically, the output fragment of the difference information fusion module is used to provide Q, while The fragment provides K and V, therefore F dm and The shared information between them can be represented as:

[0108]

[0109] Then F dm and Public information between CM and F dm This process can be represented as:

[0110]

[0111] Among them, F cm This represents the output of the first public information fusion module.

[0112] Then F also needs to be cm and Public information between them is injected into the fusion features to further enrich the fusion features. The process is the same as described above.

[0113] Finally, the fused features are input into the refining module for further feature fusion. The refining module is as follows: Figure 6 As shown, the fusion result is output after passing through Solftmax and then through convolutional layer 1, the mixing block, and convolutional layer 2 via residual connections.

[0114] All the mixed blocks have the same compositional structure, such as Figure 7 As shown, the hybrid block consists of convolutional layers, SwinTransformer blocks, and convolutional layer residual connections. All convolutional layers have a kernel size of 3×3, the SwinTransformer block has a depth of 2, a window size of 2, and 6 heads for multi-head attention.

[0115] Step 3: Perform the first stage of training, which consists of high-exposure and multi-focus fusion tasks. For these two tasks, the loss function includes image fusion loss and image reconstruction loss function. The Structural Similarity Index (SSIM) is used as the image fusion loss function, and the L1 norm is used as the reconstruction loss function. The loss function for the overexposure task can be summarized as follows:

[0116]

[0117] in It is a fusion function. Here, λ1 is the reconstructed source image, SSIM(·) is the SSIM function, and P·P1 is the L1 norm. The loss function for the multifocal task is as follows:

[0118]

[0119] Step 4: Perform the second stage of training. After the first stage of training is completed, fix the parameters of the encoder and decoder, and save the image fusion module of the first stage to handle high exposure and multi-focus tasks. Train the second set of image fusion modules. In this stage, use unlabeled infrared and visible light images to train the model on the dataset in an unsupervised manner. However, it is not enough to rely solely on unlabeled data from the infrared and visible light images for unsupervised training. It is also necessary to combine the supervised learning of the first stage and use weight coefficients to balance the weights of multiple tasks in this training stage.

[0120] The total loss function for the second stage of training is a weighted sum of a supervised loss function and an unsupervised loss function:

[0121] L S2 =L super +βL unsuper ;

[0122] Here, β is a positive tradeoff parameter. The loss function used in the first stage is directly adopted as the loss function for supervised training. Unsupervised learning loss functions also include image fusion loss functions and image reconstruction loss functions, as follows:

[0123] L unsuper =L fuse +β1L recon ;

[0124] Where β1 is a weighting coefficient. The fusion loss mainly consists of three parts: intensity loss function, texture loss function, and structure loss function.

[0125]

[0126] Among them I 1u and I 2u It is an unlabeled source image. This represents the corresponding fusion result, where β2 and β3 are weighting coefficients. The intensity loss can be calculated as:

[0127]

[0128] Where MAX(·) is the element-wise maximum operation. The texture loss in the gradient domain can be calculated as:

[0129]

[0130] Where |·| is the absolute value function, and ▽ represents the gradient operator. Simultaneously, the L1 norm reconstruction loss is minimized so that the model can decompose the unlabeled source image into multi-frequency features:

[0131]

[0132] in and It is a reconstructed, unlabeled source image.

[0133] The hyperparameters in the loss function are set to λ = 0, λ1 = 0.5, β = 0.1, β1 = 1.25, β2 = 1, and β3 = 0.5, respectively. The initial learning rate for the first stage is set to 1 × 10^-4 and then halved after 200 training cycles. The initial learning rate for the second stage is set to 3 × 10^-5.

[0134] Step 5: Select appropriate evaluation metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSI), Information Entropy, Mutual Information, and Root Mean Square Error (RMSE). PSNR calculates the difference between the fused image and the original reference image; a higher PSNR indicates better image quality. The SSI considers the similarity of brightness, contrast, and structural information, reflecting the visual quality of the image; a value closer to 1 indicates greater similarity. Information Entropy measures the complexity of the image; a higher information entropy means more information in the image. Mutual Information calculates the correlation between the fused image and the source image, reflecting how much useful information from the original image is contained in the fused image; a higher mutual information indicates better fusion. RMSE measures the average error between the original image and the fused image; a lower value is better. The definitions of PSNR, SSI, Information Entropy, Mutual Information, and RMSE are as follows:

[0135]

[0136] Where, μ x μ y Let x and y represent the mean values ​​of the image, respectively. and Let σ represent the variances of the images x and y, respectively. xy p(x) is the covariance of the graphs x and y, where C1 and C2 are constants; i ) is the value x of each pixel in the image. i The probability distribution is given by n, where n is the total number of different pixel values ​​in the image; p(I(i,j),K(i,j)) is the joint probability distribution, and p(I(i,j)) and p(K(i,j)) are individual probability distributions; N and M are the dimensions of the image, and I(i,j) and K(i,j) are the pixel values ​​of the original image and the fused image, respectively.

[0137] Table 1 Performance comparison of different models

[0138]

[0139] Table I presents the objective evaluation of 20 pairs of images from the MSRS dataset using four different fusion methods. Our proposed method performs best on EN, SSIM, and RMSE metrics, and second best on MI and PSNR metrics. Combining subjective and objective analyses, we can conclude that our proposed method outperforms other methods in both visual perception and objective metrics, achieving the best fusion performance.

[0140] Step 6, solidify the network model: After completing the network model adjustment, fix the network parameters and determine the final multimodal image fusion model; if image fusion tasks are needed in the future, the images to be fused can be directly input into the network model to obtain the fusion results.

[0141] In the above embodiments, the implementations of convolution, SwingTransformer, activation functions, normalization, normalization exponential functions, matrix multiplication operations, and element-wise multiplication are algorithms well known to those skilled in the art, and the specific processes and methods can be found in relevant textbooks or technical documents.

[0142] Furthermore, the present invention can be implemented as a system, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0143] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A SwinTransformer-based multi-modal image fusion method, characterized in that, The method comprises the following steps: Step 1, preparing training data: selecting a multi-focus dataset, a multi-exposure dataset, an infrared image dataset and a visible light image dataset, dividing the datasets into a training set, a validation set and a test set, and preprocessing the original images in the multi-focus dataset and the multi-exposure dataset and their corresponding labels; Step 2, constructing a network model: the network model comprises an image receiving module for receiving image data, a shallow feature extraction module for extracting rough shallow information of an image, a global feature extraction module for extracting fine information with global information, an adaptive frequency fusion module for fusing the difference information and the common information of an image, and an image reconstruction module for reconstructing the fused image; the global feature extraction module comprises two Laplacian pyramid encoders with the same structure, and the image reconstruction module comprises a decoder with an inverse Laplacian pyramid structure; Step 3, first-stage network training: using the labeled data in the multi-exposure dataset and the multi-focus dataset for multi-task supervised pre-training to enhance the ability of the model to extract complementary features from the multi-exposure and multi-focus tasks; Step 4, second-stage network training: using the unlabeled data in the infrared and visible light image dataset for unsupervised training to enhance the generalization ability of the model on the basis of saving data cost; using the labeled data of the multi-exposure dataset and the multi-focus dataset for supervised training to enhance the ability of the model to retain the details of the source image; Step 5, determining an evaluation index: selecting a loss function to evaluate the quality of image fusion and determining an evaluation index to evaluate the performance of the network; In step 2, after the image input module receives the two kinds of image data, the input image is subjected to a 3x3 convolution operation in the shallow feature extraction module to obtain the local information and the shallow features of the image; the shallow information is sent to the global feature extraction module; The global feature extraction module comprises two Laplacian pyramid encoders with the same structure, which continuously perform down-sampling and Laplacian difference on the image; after the feature maps with different frequencies are decomposed, they are sent to the adaptive frequency fusion module; The adaptive frequency fusion module comprises an image embedding module, a difference information fusion module, two common information fusion modules and a refining module, which fuse the images with the same frequency; after fusion, the fused image is output by the image reconstruction module, which comprises a decoder with an inverse Laplacian pyramid structure, which continuously performs up-sampling and residual connection on the image; In step 2, the encoder and the decoder form a Laplacian pyramid structure; the encoder obtains the lowest scale feature After upsampling, the residual is calculated with the feature of the same scale, and the high-frequency feature is generated 、 、 The structure of the decoder is completely symmetrical with the encoder; The adaptive frequency fusion module comprises an image embedding module, a difference information fusion module, two common information fusion modules and a refining module; the image embedding module divides the image into patches, introduces position encoding after linear projection, and outputs a sequence containing patch embedding vectors as the input of the subsequent model; the embedding process can be represented as: ； wherein is a sequence comprising patch embedding vectors, denotes the process of segmenting an image into patches, introducing positional encoding after linear projection; The cross-attention architecture is introduced, a difference information fusion module is constructed, and the frequency features decomposed by the encoder are taken as and inputs, and difference information features are output. will be described below. and The division into n local feature segments is shown below. ； wherein and and ; Converting feature segments to queries using linear layers , keys and values , linear projections can be expressed as: ； wherein , denotes a linear projection operation shared between different segments; The dot product attention layer is used to compute the query and key similarity matrix, then the value is multiplied by the inferred relevance between the query and ; denoted as: ； wherein, is a scaling factor that can alleviate the softmax function converging to the minimum region of gradient when the dot product increases; by removing the common information acquisition and the difference information between them; represented as: ； injecting the difference information into is represented as: ； wherein, represents a normalization layer, represents a multi-layer perceptron, is an output of the difference information fusion module; Introducing two identical information fusion modules, and using the output fragment of the difference information fusion module to provide... ,and The fragment provides and ,therefore and The shared information between them can be represented as: ； The common information between and is then added to , expressed as: ； wherein, represents the output of the first common information fusion module; injecting common information between and into the fused features for further enriched fused features; The fused features are input into the refining module for further feature fusion; the refining module comprises a convolution layer one, a mixing block, a convolution layer two, which are connected by a residual connection, and outputs the fusion result after softmax.

2. The SwinTransformer-based multi-modal image fusion method according to claim 1, characterized in that, The method further comprises the following steps: Step 6, solidify the network model: after completing the network model adjustment, fix the network parameters, and determine the final multi-modal image fusion model; If subsequent image fusion tasks are needed, the image to be fused is directly input into the network model to obtain the fusion result.

Images