Multi-modal image fusion method

Abstract

The invention provides a multi-modal image fusion method, which belongs to the technical field of image processing and comprises three steps of feature extraction, feature fusion and image reconstruction. The method comprises the following steps of: 1, feature extraction: performing multi-level feature extraction on an image through a convolutional neural network and a self-attention mechanism, and capturing local details and global context information of the image so as to obtain rich feature representation; and step 2, feature fusion. Information transmission is carried out between features of different modes through a cross-mode self-distillation technology, and information interaction between the modes is enhanced. And step 3, image reconstruction. And the fused feature map is converted back to an image through a decoder assembly. Through the image reconstruction process, a high-quality fused image can be obtained. And an encoder-decoder network with excellent performance is finally obtained and can be used for a downstream multi-modal image fusion task.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and machine learning, and in particular to the technical field of image processing, and specifically to a multimodal image fusion method. Background Art

[0002] Traditional fusion methods mainly include fusion methods based on multi-scale transformation and fusion methods based on sparse representation. Fusion methods based on multi-scale transformation, such as wavelet transform and non-subsampled contourlet transform, can capture the features of the source image at different scales. However, in the process of image decomposition and reconstruction, this method easily causes the loss of detail texture and structure of the source image, resulting in the failure to fully preserve the detail features of the fused image (for example, Zhang et al. proposed an image fusion method based on multi-scale transformation in IEEE Transactions on Image Processing). Fusion methods based on sparse representation, such as dictionary learning and sparse coding, can effectively avoid problems such as information loss, but when processing images of large size or complex structure, the fused image often encounters defects such as artifacts, distortion, and distortion, which limits the applicability of this method to a certain extent (for example, Wang et al. proposed an image fusion method based on sparse representation in Pattern Recognition).

[0003] In recent years, with the rapid development of deep learning technology, researchers have proposed many image fusion methods based on deep learning in order to break through the limitations of traditional image fusion methods. Liu et al. used the weight map obtained by convolutional neural network (CNN) to combine the pixel activity data of multiple source images in "Medical Image Analysis" to achieve medical image fusion based on CNN, but there are problems such as insufficient feature extraction and loss of fused image information. Jian et al. proposed the SEDRFuse network in "Neurocomputing". The network effectively reduces the loss of shallow texture details of the source image by introducing residual blocks in the encoder and using jump connections between the encoder and decoder, avoiding artifacts in the fused image. However, due to its focus on shallow features, the fused image loses image information integrity. Zhao et al. proposed in IEEE Transactions on Cybernetics that the low-frequency features of the two modalities are related, while the high-frequency features are unrelated. They extracted low-frequency features at a shallow level by sharing an encoder, used a reversible neural network to prevent the loss of high-frequency features, and combined the Transformer's ability to extract global features with the advantages of CNN's local feature extraction and computational efficiency to improve the network's feature extraction capability. However, this structure only considers shared features at a shallow level, and is insufficient for cross-modal feature extraction. In addition, the Transformer has a large amount of computation, which affects the computational efficiency of the overall network. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a multimodal image fusion method based on the mutual guidance of convolutional neural network (CNN) and self-attention mechanism, which fuses images from different sources and promotes the execution of downstream tasks.

[0005] The technical solution of the present invention is:

[0006] A multimodal image fusion method mainly includes three steps: feature extraction, feature fusion and image reconstruction.

[0007] Step 1: Feature extraction: Multi-level feature extraction is performed on the image through convolutional neural network (CNN) and self-attention mechanism to capture the local details and global context information of the image, thereby obtaining rich feature representation.

[0008] Step 2: Feature fusion. Through cross-modal self-distillation technology, information is transferred between features of different modalities to enhance information interaction between modalities, ensure the preservation and enhancement of the correlation between modalities during the feature extraction process, and thus achieve feature information fusion between different modalities.

[0009] Step 3: Image reconstruction. The fused feature map is converted back to an image through the decoder component. The decoder is a special neural network that receives the fused feature map as input and gradually restores the spatial details of the image through a series of complex calculations. The final output is a final result that retains the original image content and integrates the new modality information. Through this image reconstruction process, a high-quality fused image can be obtained. Finally, an encoder-decoder network with excellent performance is obtained, which can be used for downstream multimodal image fusion tasks.

[0010] Furthermore,

[0011] The convolution kernel slides on the image and extracts local information from the image through pixel-by-pixel convolution operations;

[0012] Convolutional neural networks gradually extract higher-level abstract features through multi-layer convolution and pooling operations to form a hierarchical feature representation.

[0013] The self-attention mechanism dynamically focuses on key areas in the image by calculating the importance weights of the features at each position, thereby enhancing the feature representation of these areas.

[0014] The formula for image feature extraction using convolutional neural network is as follows:

[0015] F i =CNN(I i )i=1,2,...,L

[0016] The formula for image feature extraction using the self-attention mechanism is as follows:

[0017] Q i =W Q F i

[0018] K i =W K F i

[0019] V i =W V F i

[0020]

[0021] F′ i =Fi +A(F i )

[0022] First, the query vector Q_i is obtained by transforming the input feature F_i through the query weight matrix W_Q; then, the key vector K_i is obtained by transforming the input feature F_i through the key weight matrix W_K; then, the value vector V_i is obtained by transforming the input feature F_i through the value weight matrix W_V; the attention weight vector A(F_i) is obtained by dividing the dot product of the query vector Q_i and the key vector K_i by the square root of the key vector dimension d_k through the softmax function, and then multiplying it by the value vector V_i; finally, the output feature vector F'_i is obtained by adding the original input feature F_i to the attention weight vector A(F_i).

[0023] Going further,

[0024] The feature extraction loss uses the L2 norm to quantify the difference between the two modal features.

[0025] The feature extraction loss is defined as follows:

[0026] L FEL is measured by calculating the difference between images from different sources in the feature space. A C,n , F B C,n represents the output features of images A and B obtained by the nth layer CNN feature extractor, F A B,n , F B B,n Represents the output features of images A and B obtained by the n-th layer self-attention feature extractor. ||·|| 2 2 represents the square of the Euclidean distance.

[0027] The total feature extraction loss L_FEL is measured by calculating the difference between images from different sources in the feature space;

[0028] Specifically, this loss consists of two parts: the difference between the features extracted by the CNN branch of image A and image B; the difference between the features extracted by the self-attention branch of image A and image B;

[0029] For each feature channel n, the square of the L2 distance between the features extracted by the CNN branch of image A and image B, as well as the square of the L2 distance between the features extracted by the self-attention branch, are calculated; these squared distances are then summed and divided by the total number of features N to obtain the total feature extraction loss.

[0030] Going further,

[0031] Mutual information loss is introduced as the optimization objective, and the definition of information complementarity loss is as follows:

[0032]

[0033] T(F_A, F_B) represents the mutual information between the features of image A and image B, which is used to measure the amount of information shared between two random variables.

[0034] Where T(F A ,F B ) represents the mutual information between different features. The formula for calculating mutual information is as follows:

[0035]

[0036] Where H(F) represents the information entropy of feature F, which is a measure of the uncertainty of a random variable, that is, the average amount of information. For a discrete random variable F, its information entropy H(F) can be calculated by the following formula: H(F) = -sum(p_i*log_2(p_i))

[0037] Where p_i represents the probability of the random variable F taking the i-th possible value. Here sum represents the sum of all possible values, and log_2 represents the logarithm with base 2. For the joint information entropy H(F_A, F_B) of two random variables F_A and F_B, a similar formula can be used to calculate H(F_A, F_B) = -sum(sum(p_ij*log_2(p_ij))) where p_ij represents the joint probability of F_A and F_B taking a specific value combination (i, j) at the same time.

[0038] Going further,

[0039] The decoder receives the fused feature map as input, and gradually restores the spatial details of the image through calculation, and finally outputs the final result that retains the original image content and integrates the new modality information;

[0040] Through this image reconstruction process, a fused image is obtained, and an encoder-decoder network with excellent performance is obtained. The network contains a total of 4 layers of dual-branch encoders. The first, second and fourth layers are each composed of 3 InceptionNeXt blocks and 3 BiFormer blocks, and the third layer is composed of 9 InceptionNeXt blocks and 9 BiFormer blocks stacked together; the eigenvalues ​​of image A and image B obtained by each layer of dual-branch encoder are: H / 2×W / 2×C1, H / 4×W / 4×C2, H / 8×W / 8×C3 and H / 16×W / 16×C4, where Cj (j=1,2,3,4) represents the number of channels of each layer of eigenvalues.

[0041] Going further,

[0042] L1 loss and structural similarity loss are used as training targets. L1 loss measures the difference between the fused image and the source image by calculating the absolute difference at each pixel. The definition of L1 loss is as follows:

[0043]

[0044] L1(I_f,I_s) represents the sum of the absolute differences between the fused image I_f and the source image I_s at each pixel. Here I_f refers to the fused output image, and I_s refers to one of the original input source images.

[0045] The structural similarity loss measures the difference between the source image and the fused image by comparing their structural similarity; the structural similarity loss is defined as follows:

[0046]

[0047] SSIM loss is a measure used to compare the structural similarity between the source image I_s and the fused image I_f. It takes into account the similarity in brightness, contrast and structure. fmax μ If Respectively represent the mean of fmax and If, σ fmax,If Represents the variance between fmax and If.

[0048] The relationship between the two losses is adjusted by the coefficient λ as follows:

[0049] L=L1+λL SSIM .

[0050] The overall loss is the weighted sum of L1 loss and SSIM loss, where λ is a weight coefficient used to adjust the relative importance between the two losses.

[0051] The beneficial effects of the present invention are

[0052] By combining the advantages of convolutional neural networks (CNN) and self-attention mechanisms, this method can more comprehensively extract local and global features of images, enhance information interaction between modalities, improve the quality of fused images, and enhance the generalization ability of the model. The improved U-Net encoding structure and attention gate mechanism enable the network to focus on key features more effectively and reduce the interference of unimportant features, thereby improving the information integrity and clarity of the fused image. In addition, cross-modal self-distillation technology promotes knowledge transfer between modalities during feature extraction, further improving the performance of the model. Experiments have shown that this method has significantly improved the image fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of the overall training framework of the present invention;

[0054] Figure 2 It is the overall training flow chart of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0056] The present invention proposes a multimodal image fusion method based on mutual guidance of convolutional neural network and self-attention mechanism. The method is based on U-Net as the basic network structure, and combines the convolution block with Inception structure and Biformer block in the encoder as a feature extraction module to capture local features and global features, and enhance the richness and representation ability of features. In addition, the present invention introduces cross-modal information complementation technology to promote knowledge transfer between modalities in the feature extraction process, improve the mutual influence of different modal features, and thus realize feature information interaction between different modalities.

[0057] Image fusion technology is a technology that combines multiple images obtained from different sensors or the same sensor at different times and under different conditions into one image. This technology can effectively improve the information content, clarity and contrast of the image, thus playing an important role in many fields such as medical diagnosis, remote sensing monitoring, military reconnaissance, and machine vision.

[0058] Feature extraction: This method uses Convolutional Neural Network (CNN) and self-attention mechanism to extract features at each level. Convolutional Neural Network (CNN) can effectively capture local features in images through convolution operations. The convolution kernel slides on the image, and through pixel-by-pixel convolution operations, local information such as edges and textures in the image can be extracted. Local features are very important for image recognition and classification tasks because these features usually contain key information of the image. Through multi-layer convolution and pooling operations, CNN can gradually extract higher-level abstract features to form a hierarchical feature representation. The self-attention mechanism can capture global information in the image. By calculating the importance weights of the features at each position, the self-attention mechanism can dynamically focus on the key areas in the image, thereby enhancing the feature representation of these areas. Global information is critical for understanding the overall structure and contextual relationships of the image. The self-attention mechanism can effectively capture long-distance dependencies in the image and improve the semantic understanding and reasoning capabilities of the model.

[0059] The formula for image feature extraction using convolutional neural network is as follows:

[0060] F i =CNN(I i )i=1,2,...,L (4.1)

[0061] The formula for image feature extraction using the self-attention mechanism is as follows:

[0062] Q i =W Q F i

[0063] K i =W K F i

[0064] V i =W V F i

[0065]

[0066] F′ i =F i +A(F i )

[0067] First, the query vector Q_i is obtained by transforming the input feature F_i through the query weight matrix W_Q. Next, the key vector K_i is obtained by transforming the input feature F_i through the key weight matrix W_K. Then, the value vector V_i is obtained by transforming the input feature F_i through the value weight matrix W_V. The attention weight vector A(F_i) is obtained by dividing the dot product of the query vector Q_i and the key vector K_i by the square root of the key vector dimension d_k through the softmax function, and then multiplying it by the value vector V_i. Finally, the output feature vector F'_i is obtained by adding the original input feature F_i to the attention weight vector A(F_i). These steps together form the core of the self-attention mechanism, which enables the model to focus on relevant information at different positions in the input sequence, thereby improving the model's expressiveness.

[0068] Feature extraction loss: In order to strengthen the encoder's focus on identifying similarities and consistency between modalities, this method uses the L2 norm to quantify the difference between the features of the two modalities. In this way, the model can better understand and integrate information from different modalities to improve the overall learning effect and performance. The feature extraction loss is defined as follows:

[0069]

[0070] The total feature extraction loss (L_FEL) is measured by calculating the difference between images from different sources in the feature space. Specifically, this loss consists of two parts: the difference between the features extracted by image A and image B in the CNN branch; the difference between the features extracted by image A and image B in the self-attention branch. For each feature channel n, we calculate the square of the L2 distance between the features extracted by image A and image B in the CNN branch, and the square of the L2 distance between the features extracted by the self-attention branch. The squares of these distances are then summed and divided by the total number of features N to obtain the total feature extraction loss. By minimizing this loss, the model can better capture and integrate information from different modalities, thereby improving the quality of the fused image.

[0071] Information complementarity loss: Information complementarity loss is an important indicator to ensure that the correlation between modalities is preserved and enhanced during the feature extraction process. In order to maximize the mutual information between the feature extraction results of different modalities, this method introduces mutual information loss as an optimization objective. In this way, the model can encourage information sharing between modalities and promote information exchange between modalities, thereby ensuring the preservation and enhancement of the correlation between modalities during the feature extraction process. The definition of information complementarity loss is as follows:

[0072]

[0073] Where T(F A ,F B ) represents the mutual information between different features. Specifically, the formula for calculating the mutual information is as follows:

[0074]

[0075] Where H(F) represents the information entropy of feature F.

[0076] Improved U-Net encoding structure: This method improves the encoder part of the U-Net network and proposes a dual-branch encoder. The encoder combines the advantages of convolutional neural network (CNN) and self-attention mechanism, and can more effectively capture and retain the local and global features of the image, thereby improving the quality of the fused image. The network contains a total of 4 layers of dual-branch encoders. The first, second and fourth layers each consist of 3 InceptionNeXt blocks and 3 BiFormer blocks, and the third layer is composed of 9 InceptionNeXt blocks and 9 BiFormer blocks stacked. The eigenvalues ​​of image A and image B obtained by each layer of dual-branch encoder are: H / 2×W / 2×C1, H / 4×W / 4×C2, H / 8×W / 8×C3 and H / 16×W / 16×C4, where Cj (j=1,2,3,4) represents the number of channels of the eigenvalues ​​of each layer.

[0077] Overall loss: In order to ensure that the encoder-decoder network can generate high-quality fused images during training, this method uses L1 loss and structural similarity index loss (SSIMLoss) as training targets. These two loss functions measure the difference between the fused image and the source image from different perspectives, thereby improving the quality of the fused image.

[0078] The L1 loss measures the difference between the fused image and the source image by calculating the absolute difference at each pixel. The L1 loss is defined as follows:

[0079]

[0080] The structural similarity loss measures the difference between the source image and the fused image by comparing their structural similarity. The definition of the structural similarity loss is as follows:

[0081]

[0082] The relationship between the two losses is adjusted by the coefficient λ as follows:

[0083] L=L1+λL SSIM

[0084] The above description is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A multimodal image fusion method, characterized in that: There are three steps involved: Step 1: Feature extraction: Use convolutional neural networks and self-attention mechanisms to perform multi-level feature extraction on the image, capture local details and global context information of the image, and obtain feature representation; Step 2: Feature fusion: Information is transferred between features of different modalities through cross-modal self-distillation technology, which enhances the information interaction between modalities and realizes the fusion of feature information between different modalities. Step 3: Image reconstruction, converting the fused feature map back to an image through the decoder component.

2. The method according to claim 1, characterized in that The convolution kernel slides on the image and extracts local information from the image through pixel-by-pixel convolution operations; Convolutional neural networks gradually extract higher-level abstract features through multi-layer convolution and pooling operations to form a hierarchical feature representation.

3. The method according to claim 1 or 2, characterized in that: The self-attention mechanism dynamically focuses on key areas in the image by calculating the importance weights of the features at each position, thereby enhancing the feature representation of these areas.

4. The method according to claim 3, characterized in that The formula for image feature extraction using convolutional neural network is as follows: F i =CNN(I i )i=1,2,...,L The formula for image feature extraction using the self-attention mechanism is as follows: Q i =W Q F i K i =W K F i V i =W V F i F′ i =F i +A(F i ) First, the query vector Q_i is obtained by transforming the input feature F_i through the query weight matrix W_Q; then, the key vector K_i is obtained by transforming the input feature F_i through the key weight matrix W_K; then, the value vector V_i is obtained by transforming the input feature F_i through the value weight matrix W_V; the attention weight vector A(F_i) is obtained by dividing the dot product of the query vector Q_i and the key vector K_i by the square root of the key vector dimension d_k through the softmax function, and then multiplying it by the value vector V_i; finally, the output feature vector F'_i is obtained by adding the original input feature F_i to the attention weight vector A(F_i).

5. The method according to claim 4, characterized in that The L2 norm is used to quantify the difference between the two modal features.

6. The method according to claim 5, characterized in that The feature extraction loss is defined as follows: Among them, the total feature extraction loss L_FEL is measured by calculating the difference between images from different sources in the feature space; F A C,n , F B C,n represents the output features of images A and B obtained by the nth layer CNN feature extractor, F A B,n , F B B,n represents the output features of images A and B obtained by the n-th layer self-attention feature extractor, ||·|| 2 2 represents the square of the Euclidean distance; Specifically, this loss consists of two parts: the difference between the features extracted by the CNN branch of image A and image B; the difference between the features extracted by the self-attention branch of image A and image B; For each feature channel n, the square of the L2 distance between the features extracted by the CNN branch of image A and image B, as well as the square of the L2 distance between the features extracted by the self-attention branch, are calculated; these squared distances are then summed and divided by the total number of features N to obtain the total feature extraction loss.

7. The method according to claim 5, characterized in that Mutual information loss is introduced as the optimization objective, and the definition of information complementarity loss is as follows: T(F_A, F_B) represents the mutual information between the features of image A and image B, which is used to measure the amount of information shared between two random variables; where T(F_A, F_B) A ,F B ) represents the mutual information between different features. The formula for calculating mutual information is as follows: Where H(F) represents the information entropy of feature F, which is a measure of the uncertainty of a random variable, that is, the average amount of information. For a discrete random variable F, its information entropy H(F) can be calculated by the following formula: H(F) = -sum(p_i*log_2(p_i)) Where p_i represents the probability of the random variable F taking the i-th possible value. Here sum represents the sum of all possible values, and log_2 represents the logarithm with base 2; for the joint information entropy H(F_A, F_B) of two random variables F_A and F_B, a similar formula can be used to calculate H(F_A, F_B) = -sum(sum(p_ij*log_2(p_ij))) where p_ij represents the joint probability of F_A and F_B taking a specific value combination (i, j) at the same time.

8. The method according to claim 7, characterized in that The decoder receives the fused feature map as input, and gradually restores the spatial details of the image through calculation, and finally outputs the final result that retains the original image content and integrates the new modality information; Through this image reconstruction process, a fused image is obtained, and an encoder-decoder network with excellent performance is obtained.

9. The method according to claim 8, characterized in that The network contains a total of 4 layers of dual-branch encoders. The 1st, 2nd and 4th layers each consist of 3 InceptionNeXt blocks and 3 BiFormer blocks, and the 3rd layer is composed of 9 InceptionNeXt blocks and 9 BiFormer blocks stacked together. The eigenvalues ​​of image A and image B obtained after each layer of dual-branch encoder are H / 2×W / 2×C1, H / 4×W / 4×C2, H / 8×W / 8×C3 and H / 16×W / 16×C4, respectively, where Cj (j=1,2,3,4) represents the number of channels of the eigenvalues ​​of each layer.

10. The method according to claim 8, characterized in that L1 loss and structural similarity loss are used as training targets. L1 loss measures the difference between the fused image and the source image by calculating the absolute difference at each pixel. The definition of L1 loss is as follows: L1(I_f,I_s) represents the sum of the absolute differences between the fused image I_f and the source image I_s at each pixel; The structural similarity loss measures the difference between the source image and the fused image by comparing their structural similarity; the structural similarity loss is defined as follows: μ fmax μ If Respectively represent the mean of fmax and If, σ fmax,If represents the variance between fmax and If; The relationship between the two losses is adjusted by the coefficient λ as follows: L=L1+λL SSIM The overall loss is the weighted sum of the L1 loss and the structural similarity loss, where λ is a weight coefficient used to adjust the relative importance between the two losses.

Images