Infrared and visible light image fusion method based on swinTransformer
By using a swinTransformer-based image fusion method, combined with convolutional neural networks and correlation coefficient loss function, the shortcomings of feature extraction and modal information interaction in infrared and visible light image fusion are solved, achieving high-quality image fusion and improving computational efficiency.
Patent Information
- Application Number
- CN202410688338.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-30
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-05-30
AI Technical Summary
Existing infrared and visible light image fusion methods have shortcomings in feature extraction and modal information interaction, making it difficult to simultaneously preserve the texture details and global features of the image, and they also have high computational complexity.
An image fusion method based on swinTransformer is adopted, including an image input module, a coarse feature extraction module, an attention enhancement module, a lossless feature transfer module, a modal interaction module, and a feature reconstruction module. Combined with a convolutional neural network, feature enhancement and lossless transfer are achieved, while fine-grained extraction and modal interaction are performed using swinBlock. A correlation coefficient loss function is added to optimize the fusion effect.
It improves the quality of fused images, enhances texture detail preservation and global feature fusion, reduces computational complexity, and generates higher-quality fused images.
Smart Images

Figure CN118570076B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multimodal image fusion, and in particular to a method for fusing infrared and visible light images based on swinTransformer. Background Art
[0002] Multimodal image fusion is an important image processing method, which aims to use the image information of different modalities to generate a fused image containing richer information based on the complementarity and similarity between the information. The fusion of infrared and visible light images belongs to multimodal image fusion. Infrared and visible light images can complement each other because the information contained in these two modal images can complement each other. Specifically, infrared images can reflect the thermal radiation information of objects under harsh and adverse conditions, and can effectively highlight significant targets, but lack detailed descriptions of scenes, such as texture details, ambient lighting, etc. Visible light images can well represent rich texture detail information and lighting information and are more in line with human visual perception, but are easily affected by ambient light. The image after fusing these two complementary modalities can avoid the disadvantages of light sensitivity and low resolution and noise of infrared images.
[0003] With the application of various types of sensors in imaging systems, multimodal image fusion has become a hot research topic. There is information complementarity and redundancy between infrared and visible light images from the same scene. Through image fusion technology, more comprehensive information of the scene can be obtained, making subsequent visual tasks more effective. At present, image fusion methods at home and abroad are mainly divided into traditional methods and deep learning-based methods. Although traditional methods can meet the needs of specific scenes in most cases, fusion strategies still need to be manually selected. For complex scenes, manually set fusion strategies are not enough to meet the needs. Methods based on deep learning include convolutional neural networks, adversarial generative networks, autoencoders and Transformer methods. Transformer's excellent performance in visual tasks has made it shine in the field of image fusion. The image generated by the fusion of infrared and visible light images contains more comprehensive information about the scene, so it has a wide range of applications in the fields of autonomous driving and drones.
[0004] At present, the image fusion method based on CNN implicitly learns image features through the neural network structure and loss function constraints to achieve feature fusion and image reconstruction process, and can pay more attention to local detail information of the image. However, the image fusion method based on CNN is difficult to control and cannot fully extract the feature information of the image. When performing modal information interaction, it often cannot learn the difference information well. The image fusion architecture based on transformer can effectively pay attention to the global overall information of the image, but after long-term time series modeling, the learning of features will ignore more detailed information. In addition, the attention mechanism of transformer will increase the calculation time and reasoning complexity. Therefore, the above two types of image fusion methods have their own defects. As a result, there will be some deficiencies in image fusion, which will affect subsequent visual tasks. Summary of the invention
[0005] The purpose of the present invention is to provide a method for fusing infrared and visible light images based on swinTransformer, which is used for fusing infrared and visible light images, thereby improving the quality of fused images.
[0006] To achieve the above object, the present invention provides a method for fusion of infrared and visible light images based on swinTransformer, comprising the following steps:
[0007] S1. Construct an image fusion system, including an image input module, a feature coarse extraction module, an attention enhancement module, a feature fine extraction module, a feature lossless transfer module, a modal interaction module and a feature reconstruction module. The image input module receives two image data sources at the same time.
[0008] S2, inputting the infrared image and the visible light image into the image fusion system through the image input module;
[0009] S3, input the images input in S2 into the feature rough extraction module respectively, and further extract features through the attention enhancement module and the feature lossless transfer module;
[0010] S4, input the features obtained in S3 into the feature fine extraction module, perform fine-grained feature extraction on the image features, and then input them into the modal interaction module;
[0011] S5, output the features of different modalities obtained in S4 to the feature reconstruction module after passing through the modality interaction module containing cross attention;
[0012] S6: Use convolution to remap the features obtained in S5 back to the corresponding image space to complete the output of the fused image.
[0013] Preferably, in S2, images corresponding to different modalities are prepared before inputting the images, and the input images are registered images.
[0014] Preferably, the specific steps of S3 are as follows:
[0015] S31, performing a 3*3 convolution operation on the input image to obtain the coarse-grained features of the image and the corresponding local information, avoiding the loss of information transmission caused by too deep convolution;
[0016] S32, sending the coarse-grained features of the image to the attention enhancement module, which includes two parts: a spatial attention enhancement module and a channel attention enhancement module. The spatial attention enhancement module obtains the corresponding spatial weight by learning the spatial position of the image. The channel attention enhancement module uses maximum pooling and average pooling for simultaneous processing, and shares information through the same MLP. The spatial attention sequentially uses maximum pooling and average pooling in the channel dimension, and obtains enhanced features through convolution and activation functions.
[0017] S33, the enhanced features are received by the reversible neural network of the lossless transfer module, and the features are equally divided into two parts according to the channels. In the reversible neural network, a bottleneck residual block is added, and the features after the channel averaging are associated with the information between the channels through the bottleneck residual block. After the feature addition and multiplication operations, the separated channel features are spliced together;
[0018] S34: The bottleneck residual block first performs a 1*1 convolution, uses the BN and RELU6 functions to activate the features, puts them into a 3*3 DW convolution, and activates them by RELU6 after BN. The final features are transformed by a 1*1 convolution, and the feature processing is completed after BN.
[0019] Preferably, the specific steps of S4 are as follows:
[0020] S41, the feature size is H×W×C, swinBlock (the main module of swinTransformer) will divide the features into sliding windows, and the M*M size window divides the features into The features are mapped to Q, K, V vectors for processing; {Q, K, V} = {XW Q ,XW K ,XW V}
[0021] S42, after LN normalization operation, the window multi-head self-attention processes the features, adds them to the features received by the LN layer, merges them, processes them using LN and MLP, and then adds them to the results of the previous addition operation to complete the processing of a swinBlock; the attention operation is:
[0022]
[0023] Among them, K T represents the transpose of K, B represents the bias, and d k Represents feature dimension;
[0024] S43. Repeat step S42 to achieve fine-grained feature extraction.
[0025] Preferably, the specific steps of S5 are as follows:
[0026] S51, inputting fine-grained modal features of different sources into a fusion module of a modal interaction module;
[0027] S52, obtaining Q, K, and V vectors from different sources, and exchanging K and V vectors between modes;
[0028] S53, the recombined QKV vector is input into the modal interaction module to complete the complementation of modal features, and after LN, the Q vector of the current modality is added to the interactive features;
[0029] The features obtained in S54 and S53 are added to the features processed sequentially by MLP and LN.
[0030] Preferably, the specific steps of S6 are as follows:
[0031] S61, input the features of different modalities into the convolution at the same time to obtain fused features;
[0032] S62, use 4 swinBlocks to reconstruct the fused features;
[0033] S63. Use convolution to map the features back to the original image space to complete the output of the fused image.
[0034] Preferably, the correlation coefficient is added to the loss function of the image fusion system to guide the method to generate a better fused image. The specific steps are as follows:
[0035] (1) Calculate the mean of the input infrared image and visible light image respectively Fusion Image I f The mean μ of
[0036] (2) Calculate the correlation coefficient according to the formula as follows
[0037]
[0038] Among them, M and N represent the size of the feature, I represents the image to be compared, and I f represents the fused image, represents the mean of the image to be compared, μ represents the mean of the fused image, i,j represents the position of the pixel of the image, and I(i,j) represents the pixel value of the image;
[0039] (3) Subtract the correlation coefficient of each image from 1, sum the calculation results of the two modes, and then add the average to the overall loss function to constrain the learning process.
[0040] A computer storage medium stores computer execution commands, and the execution commands execute the steps of a method for fusing infrared and visible light images based on swinTransformer.
[0041] Therefore, the present invention adopts the above-mentioned method of infrared and visible light image fusion based on swinTransformer, which has the following beneficial effects:
[0042] (1) Through feature coarse extraction and feature enhancement module, the image features are enhanced. The feature lossless transmission module transmits the features losslessly to the next module and further enhances the image features, retaining the image details. After fine-grained extraction composed of swinBlock, the modal interaction module completes the interaction of modal information, and the feature reconstruction module reconstructs the fused image to obtain the fused image.
[0043] (2) Combining the advantages of convolutional neural networks and swinTransformer networks, the two methods complement each other. By introducing the attention enhancement module and feature lossless transmission module, the fusion effect of the fused image on texture detail retention and global features is improved;
[0044] (3) By adding the correlation coefficient loss to the total loss function, the fused image is guided to retain more source image information, thereby improving the image fusion quality.
[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 is a flow chart of an embodiment of the present invention;
[0047] Figure 2 is a flow chart of network training according to an embodiment of the present invention;
[0048] Figure 3 Schematic diagram of the results of the embodiment of the present invention. DETAILED DESCRIPTION
[0049] Example
[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0051] like Figure 1-3 As shown, a method for fusing infrared and visible light images based on swinTransformer includes the following steps:
[0052] S1. Construct an image fusion system, including an image input module, a feature coarse extraction module, an attention enhancement module, a feature fine extraction module, a feature lossless transfer module, a modality interaction module and a feature reconstruction module;
[0053] Image input module, used to read the infrared and visible light test data sets to be input;
[0054] The feature rough extraction module uses a convolutional network to extract image features with detailed features;
[0055] Attention enhancement module uses the characteristics of attention to enhance the features of the image. The attention enhancement module includes channel attention enhancement and spatial attention enhancement. By enhancing the space and channel, it effectively highlights the effective information.
[0056] The feature lossless transmission module is implemented through a reversible neural network structure. The reversibility of the network is used to achieve lossless transmission of features and further enhance the features.
[0057] The feature extraction module uses the swinTransformer structure, which can not only perform global modeling of the image, but also effectively reduce the amount of calculation and running time;
[0058] The modality interaction module obtains information of the modality and other modalities by exchanging the Q, K, and V vectors of the attention mapping, completing complementary learning of information and fusion of modality information;
[0059] The feature reconstruction module reconstructs the received features according to the features of modal fusion and completes the transformation from image features to images.
[0060] S2, the image input module inputs the registered infrared image and visible light image of the same scene;
[0061] S3, formatting the registered infrared image and visible light image of the same scene. If the selected image channel number is 1, the trained image is a grayscale image. If the selected image channel number is 3, all images are read as RGB color images.
[0062] S4. The obtained images are respectively input into the feature rough extraction module, and the features are further extracted through the attention enhancement module and the feature lossless transfer module; the details are as follows:
[0063] S41, inputting the image into a feature coarse extraction module for convolution operation to obtain coarse-grained features of the image and corresponding local information, so as to avoid information transmission loss caused by too deep convolution;
[0064] S42, sending the coarse-grained features of the image to the attention enhancement module, which includes two parts: a spatial attention enhancement module and a channel attention enhancement module. The spatial attention enhancement module obtains the corresponding spatial weight by learning the spatial position of the image. The channel attention enhancement module uses maximum pooling and average pooling for simultaneous processing, and shares information through the same MLP. The spatial attention sequentially uses maximum pooling and average pooling in the channel dimension, and obtains enhanced features through convolution and activation functions.
[0065] S43, the enhanced features are received by the reversible neural network of the lossless transfer module, and the features are equally divided into two parts according to the channels. In the reversible neural network, a bottleneck residual block is added, and the features after the channel averaging are associated with the information between the channels through the bottleneck residual block. After the feature addition and multiplication operations, the separated channel features are spliced together;
[0066] S44: The bottleneck residual block first performs a 1*1 convolution, uses the BN and RELU6 function to activate the feature, puts it into the 3*3 DW convolution, and activates it by RELU6 after BN. The final feature is transformed by a 1*1 convolution, and the feature processing is completed after BN. Table 1 shows the ablation experiment results of the attention enhancement module and the feature lossless transfer module on MSRS.
[0067] Table 1 Ablation experiment results
[0068] method PSNR <![CDATA[Q AB / F ]]> SD AG SF w / o attention enhancement 62.3201 0.6389 41.3276 3.1876 9.3246 w / o lossless transmission 64.2430 0.6622 43.2010 3.5659 11.1238 Attention Enhancement & Lossless Transmission 64.5860 0.6931 42.3475 3.5496 11.1231
[0069] S5, input the features obtained in S4 into the feature fine extraction module, perform fine-grained feature extraction on the image features, and then input them into the modal interaction module; the details are as follows:
[0070] S51, the feature size is H×W×C, swinBlock will divide the feature into sliding windows, and the M*M size window will divide the feature into The features are mapped to Q, K, V vectors for processing; {Q, K, V} = {XW Q ,XW K ,XW V}
[0071] S52, after LN normalization operation, the window multi-head self-attention processes the features, adds them to the features received by the LN layer, merges them, processes them using LN and MLP, and then adds them to the results of the previous addition operation to complete the processing of a swinBlock; the attention operation is:
[0072]
[0073] Among them, K T represents the transpose of K, B represents the bias, and d k Represents feature dimension;
[0074] S53, repeat step S52 to achieve fine-grained feature extraction.
[0075] S6: Output the features of different modalities obtained in S5 to the feature reconstruction module after passing through the modality interaction module containing cross attention; the details are as follows:
[0076] S61, inputting fine-grained modal features of different sources into a fusion module of a modal interaction module;
[0077] S62, obtaining Q, K, and V vectors from different sources, and exchanging K and V vectors between modes;
[0078] S63, the recombined QKV vector is input into the modal interaction module to complete the complementation of modal features, and after LN, the Q vector of the current modality is added to the interactive features;
[0079] The features obtained in S64 and S63 are added to the features processed sequentially by MLP and LN.
[0080] S7, after obtaining the interactive features of S6, after image reconstruction, use convolution to remap the features back to the corresponding image space to complete the output of the fused image;
[0081] S71, input the features of different modalities into the convolution at the same time to obtain fused features;
[0082] S72, use 4 swinBlocks to reconstruct the fused features;
[0083] S73. Use convolution to map the features back to the original image space to complete the output of the fused image.
[0084] Finally, the correlation coefficient is added to the loss function of the image fusion system to guide this method to generate a better fused image. The specific steps are as follows:
[0085] 1. Calculate the mean of the input infrared image and visible light image respectively Fusion Image I f The mean μ of
[0086] 2. Calculate the correlation coefficient according to the formula as follows
[0087]
[0088] Among them, M and N represent the size of the feature, I represents the image to be compared, and I f represents the fused image, represents the mean of the image to be compared, μ represents the mean of the fused image, i,j represents the position of the pixel of the image, and I(i,j) represents the pixel value of the image;
[0089] 3. Subtract the correlation coefficient of each image from 1, sum the calculation results of the two modalities, and then add the average to the overall loss function to constrain the learning process. Table 2 shows the results of the comparative experiments on MSRS.
[0090] Table 2 Comparative experimental results
[0091] DenseFusion NestFuse RFNFuse SeAFuse YDTR SwinFuse DATFuse Ours PSNR 61.8966 59.7805 61.4467 64.3714 60.9680 64.3201 59.5663 65.5860 <![CDATA[Q AB / F ]]> 0.5158 0.5665 0.4951 0.6702 0.5120 0.6578 0.6287 0.6931 SD 23.4722 40.6445 29.0775 41.8581 25.0453 42.3386 36.4756 42.3475 AG 2.0425 3.2386 2.1074 3.6693 2.2285 3.3891 3.5690 3.5496 SF 6.0159 10.1225 6.1634 11.1231 7.4225 10.7410 10.9268 11.1231
[0092] A computer storage medium stores computer execution commands, and the execution commands execute the steps of a method for fusing infrared and visible light images based on swinTransformer.
[0093] Therefore, the present invention adopts the above-mentioned method of infrared and visible light image fusion based on swinTransformer, enhances and transmits the effective information of infrared and visible light images through feature enhancement and lossless transmission of information, and interacts the enhanced information of different modalities through a multi-head attention network to achieve information complementarity and enhancement of common features, which helps to improve the quality of the fused image of infrared and visible light images.
[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.
Claims
1. A method for fusion of infrared and visible light images based on swinTransformer, characterized in that: The following steps are involved: S1. Construct an image fusion system, including an image input module, a feature coarse extraction module, an attention enhancement module, a feature fine extraction module, a feature lossless transfer module, a modal interaction module and a feature reconstruction module. The image input module receives two image data sources at the same time. S2, inputting the infrared image and the visible light image into the image fusion system through the image input module; S3, input the images input in S2 into the feature rough extraction module respectively, and further extract features through the attention enhancement module and the feature lossless transfer module; S4, input the features obtained in S3 into the feature fine extraction module, perform fine-grained feature extraction on the image features, and then input them into the modal interaction module; S5, output the features of different modalities obtained in S4 to the feature reconstruction module after passing through the modality interaction module containing cross attention; S6, use convolution to remap the features obtained in S5 back to the corresponding image space to complete the output of the fused image; The specific steps of S3 are as follows: S31, perform input image The convolution operation is used to obtain the coarse-grained features of the image and the corresponding local information; S32, sending the coarse-grained features of the image to the attention enhancement module, which includes two parts: a spatial attention enhancement module and a channel attention enhancement module. The spatial attention enhancement module obtains the corresponding spatial weight by learning the spatial position of the image. The channel attention enhancement module uses maximum pooling and average pooling for simultaneous processing, and shares information through the same MLP. The spatial attention sequentially uses maximum pooling and average pooling in the channel dimension, and obtains enhanced features through convolution and activation functions. S33, the enhanced features are received by the reversible neural network of the lossless transfer module, and the features are equally divided into two parts according to the channels. In the reversible neural network, a bottleneck residual block is added, and the features after the channel averaging are associated with the information between the channels through the bottleneck residual block. After the feature addition and multiplication operations, the separated channel features are spliced together; S34: Bottleneck residual block first Convolution, use BN and RELU6 function to activate the features, and put them into The DW convolution is activated by RELU6 after BN, and the final feature is The convolution is transformed, and the feature processing is completed after BN; The specific steps of S4 are as follows: S41, characteristic size is , swinBlock will divide the features into sliding windows, The window of size divides the feature into , the features are mapped to Q, K, and V vectors for processing; S42, after LN performs normalization operation, the window multi-head self-attention processes the features, adds them to the features received by the LN layer, and then uses LN and MLP to process the features, and then adds them to the results of the previous addition operation to complete the processing of a swinBlock; S43. Repeat step S42 to extract features in a fine-grained manner.
2. The method for fusion of infrared and visible light images based on swinTransformer according to claim 1, characterized in that: In S2, images corresponding to different modalities are prepared before inputting the image, and the input image is a registered image.
3. The method for fusion of infrared and visible light images based on swinTransformer according to claim 1, characterized in that: The specific steps of S5 are as follows: S51, inputting fine-grained modal features of different sources into a fusion module of a modal interaction module; S52, obtaining Q, K, and V vectors from different sources, and exchanging K and V vectors between modes; S53, the recombined QKV vector is input into the modal interaction module to complete the complementation of modal features, and after LN, the Q vector of the current modality is added to the interactive features; The features obtained in S54 and S53 are added to the features processed sequentially by MLP and LN.
4. The method for fusion of infrared and visible light images based on swinTransformer according to claim 1, characterized in that: The specific steps of S6 are as follows: S61, input the features of different modalities into the convolution at the same time to obtain fused features; S62, use 4 swinBlocks to reconstruct the fused features; S63. Use convolution to map the features back to the original image space to complete the output of the fused image.
5. The method for fusion of infrared and visible light images based on swinTransformer according to claim 1, characterized in that: Add the correlation coefficient to the loss function of the image fusion system. The specific steps are as follows: (1) Calculate the mean of the input infrared image and visible light image respectively , fused image The mean ; (2) Calculate the correlation coefficient according to the formula as follows ; in, , Both represent the size of the feature, represents the image to be compared, represents the fused image, represents the mean of the image to be compared, represents the mean of the fused image, represents the position of the pixel in the image, Represents the pixel value of the image; (3) Subtract the correlation coefficient of each image from 1, sum the calculation results of the two modes, and then add the average to the overall loss function to constrain the learning process.
6. A computer storage medium, characterized in that: The computer storage medium stores computer execution commands, and the execution commands execute the steps of the method for fusion of infrared and visible light images based on swinTransformer as described in any one of claims 1 to 5.
