Infrared and visible light image fusion method based on self-attention and convolutional neural network

Abstract

The present invention discloses an infrared visible light image fusion method based on self-attention and convolutional neural network, comprising: constructing an image fusion network; the image fusion network comprises: a convolutional neural network module, a visual self-attention module and a loss calculation module; training the image fusion network; inputting the infrared image and the visible light image into the convolutional neural network module respectively to obtain a feature map; inputting the feature map into the self-attention module to obtain the long-range correlation of the feature map; using the long-range correlation information of the feature map, inputting the feature map into the image reconstruction unit to obtain an initial fused image; using the loss calculation module to perform loss calculation on the initial fused image, and finally obtaining a trained image fusion network; using the trained image fusion network to perform image fusion to obtain a fused image. The present invention has the beneficial effects of being able to generate a fused image with rich information and ideal effect, and the fused image contains more detail information and saliency information.

Description

Technical Field

[0001] The present invention relates to the field of image fusion, and in particular to an infrared and visible light image fusion method based on self-attention and convolutional neural network. Background Art

[0002] Infrared image is the image obtained by infrared camera in infrared band, which is usually grayscale image; visible image is the image obtained by camera in visible light band visible to human eye, which is usually color image. Infrared image can still work normally in bad weather, such as at night and in extreme weather with low visibility; while visible light image can provide image texture and details suitable for human vision, but generally can only work in good environment. Therefore, we extract the saliency information of infrared image and the detail information of visible light image, and concentrate the useful information of two images into one image. The final fusion result is an infrared image with prominent target and rich details, that is, the fusion result combines the advantages of two types of images.

[0003] Infrared and visible light image fusion technology is actually an image enhancement technology, and its ultimate goal is to combine images acquired by different sensors to generate images with large amounts of information, which is convenient for subsequent processing or helps decision-making. The results obtained by the infrared and visible light image fusion technology in the present invention can be used to improve the performance of human visual perception, target detection and target recognition. These two types of images can provide scene information with complementary properties from different aspects, and they exist in all objects. Image fusion mainly involves three main steps: activity level measurement, image transformation and fusion rule design. Existing fusion methods usually use the same transformation or representation for different types of source images in the fusion process. However, this method is not suitable for infrared and visible light images, because the thermal radiation area in the infrared image and the detail content in the visible light image are two completely different manifestations. In addition, the fusion rules in the existing methods are mostly designed manually, and the fusion process is becoming more and more complicated, with problems of difficulty in implementation and high computational cost. This traditional method is very time-consuming and labor-intensive to model the extracted features and design fusion rules, so its performance is greatly limited.

[0004] In the past decade, deep learning methods have gradually surpassed traditional methods. Compared with traditional infrared and visible light image fusion methods, deep learning-based image fusion methods have two advantages: (1) Using a large number of known samples as training sets, more specific feature information can be extracted from the two types of images respectively. (2) The end-to-end framework based on deep learning can automatically extract key information and reconstruct the features of the source image without the need for post-processing, greatly reducing the complexity of the program. Although existing deep learning methods have made some progress in infrared and visible light image fusion, there are still some problems that need to be improved. First, the deep learning methods currently used are basically convolutional neural network methods. Although convolutional neural networks are very efficient, this method interrupts the correlation of images and cannot extract long-term related information in the image well, which has a certain impact on subsequent image fusion. Second, the loss function in the existing network framework is mostly to calculate the loss function of the fused image and the original image. This pixel-level loss function will overemphasize that the fused image and the original image are exactly the same, but it is still hoped that the fused image and the original image are similar in structure and salient information. Summary of the invention

[0005] In order to overcome the above problems, the present application proposes an infrared visible light image fusion network based on visual self-attention and convolutional neural network. The proposed network consists of three modules: convolutional neural network module (CNN-Module), visual self-attention module (ViT-Module) and feature loss module (Feature Loss). The convolutional neural network module is mainly used to preprocess two different types of images and preliminarily extract significant information and detail information in the image. However, if only this module is used, it will cause the above-mentioned problem of insufficient dependence. The information extracted from an image has no correlation, which has a great impact on the subsequent results. Therefore, the present application not only uses the traditional convolutional neural network module, but also connects a visual self-attention module in series behind this module to preserve and extract the constant correlation in the original image, which makes up for this defect of the convolutional neural network, so that the long-term correlation of the image can be preserved, and the obtained fusion result is more natural, and finally the ideal fusion result is obtained. The feature loss module is mainly used to calculate the feature loss of the fused image. The loss function in this network includes basic pixel loss, such as SSIM loss and MSE loss, etc., but only these loss functions will have the above-mentioned problems. Therefore, this network adds a feature loss module, which calculates the corresponding loss by extracting the feature maps of the fused image and the original image. The loss of different layers can be used to calculate the detail loss and structure loss of the image, making up for the defects of the pixel loss function.

[0006] Specifically, the present invention provides an infrared and visible light image fusion method based on self-attention and convolutional neural network, comprising the following steps:

[0007] S1: construct an image fusion network; the image fusion network includes: a convolutional neural network module, a visual self-attention module and a loss calculation module;

[0008] S2: training the image fusion network; inputting the infrared image and the visible light image into the convolutional neural network module respectively to obtain a feature map; inputting the feature map into the self-attention module to obtain a long-range correlation of the feature map; using the long-range correlation information of the feature map, inputting the feature map into the image reconstruction unit to obtain an initial fused image; during the training process, using the loss calculation module to perform loss calculation on the initial fused image, and finally obtaining a trained image fusion network;

[0009] S3: Use the trained image fusion network to perform image fusion to obtain a fused image.

[0010] Furthermore, the convolutional neural network module adopts a dual-branch structure, including: a densely connected network and a fast downsampling network.

[0011] Furthermore, the densely connected network specifically refers to: the input of all layers in the network is connected to the output of all previous layers, as shown in the following formula:

[0012] X p =F p ([X0, X1, X2, ..., X p-1 ])

[0013] Among them, F p is the nonlinear operation of the pth layer, [X0, X1, X2, ..., X p-1 ] is the concatenation of all output layers.

[0014] Furthermore, the specific process of obtaining the feature map in step S2 is:

[0015] S21: performing registration processing on the infrared image and the visible light image to obtain an input image of a preset size;

[0016] S22: Pass the input image through a convolution layer to obtain an initial feature map;

[0017] S23: inputting the initial feature map into the dense connection network and the fast downsampling network respectively to obtain the detail features and structural features of the image respectively;

[0018] S24: Connect the detail features and the structural features to obtain a feature map.

[0019] Furthermore, the convolutional neural network module adopts the MISH function as an activation function.

[0020] Furthermore, the visual self-attention module includes: a channel visual self-attention unit and a spatial visual self-attention unit.

[0021] Furthermore, the loss calculation module includes: a pixel loss function and a feature loss function.

[0022] Furthermore, the pixel loss function is specifically:

[0023] L=αL MSE +βL SSIM +γL gradient

[0024] Where α, β and y are the constant coefficients of the three loss function components respectively; L MSE It is used to calculate the mean square error between the infrared image and the initial fused image; Lgradient is the gradient loss, which is used to obtain the high-frequency gradient information of the initial fused image; L SSIM is the structural loss, which is used to calculate the structural loss between the fused image and the infrared image.

[0025] The feature loss function processes the visible light image and the initial fusion image respectively through the pre-trained VGG19 network, and calculates the loss in different feature layers of the VGG19 network, as shown in the following formula:

[0026]

[0027] Among them, MAE(X, Y) represents the feature loss function, N is the total number of training set samples; X n is the nth visible light image in the training set; Y n is the nth initial fused image in the training set.

[0028] The beneficial effect provided by the present invention is that it is possible to generate a fused image with rich information and ideal effect, and the fused image contains more detail information and saliency information. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic flow chart of the method of the present invention;

[0030] Figure 2 It is the structural diagram of the image fusion network;

[0031] Figure 3 It is a diagram of the module structure of a convolutional neural network;

[0032] Figure 4 This is the structure diagram of the visual self-attention module;

[0033] Figure 5 This is a schematic diagram of the main framework of VisionTransformer;

[0034] Figure 6 It is the processing flow chart of the spatial visual self-attention unit;

[0035] Figure 7 It is the processing flow chart of the channel visual self-attention unit;

[0036] Figure 8 This is the first set of experimental comparison results;

[0037] Fig. 9 This is the comparison result of the second set of experiments;

[0038] Fig.10 It is the result of EN quantitative experiment;

[0039] Fig.11 It is the result of MI quantitative experiment;

[0040] Fig.12 It is the SSIM quantitative experimental result. DETAILED DESCRIPTION

[0041] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0042] Please refer to Figure 1 , Figure 1 It is a schematic flow chart of the method of the present invention;

[0043] The infrared and visible light image fusion method based on self-attention and convolutional neural network includes the following steps:

[0044] The proposed network consists of three modules: a convolutional neural network module (CNN-Module), a visual self-attention module (ViT-Module) and a feature loss module (Feature Loss). The convolutional neural network module is mainly used to preprocess two different types of images and preliminarily extract significant information and detail information in the image. However, if only this module is used, the above-mentioned problem of insufficient dependence will be caused. The information extracted from an image has no correlation, which has a great impact on subsequent results. Therefore, this application not only uses the traditional convolutional neural network module, but also connects a visual self-attention module in series behind this module to preserve and extract the constant correlation in the original image, which makes up for this defect of the convolutional neural network, so that the long-term correlation of the image can be preserved, and the obtained fusion result is more natural, and finally an ideal fusion result is obtained. The feature loss module is mainly used to calculate the feature loss of the fused image. The loss function in this network includes basic pixel losses, such as SSIM loss and MSE loss, etc., but only these loss functions will have the above-mentioned problems. Therefore, this network adds a feature loss module, which calculates the corresponding loss by extracting the feature maps of the fused image and the original image. The loss of different layers can be used to calculate the detail loss and structure loss of the image, making up for the defects of the pixel loss function.

[0045] Specifically, the present invention provides an infrared and visible light image fusion method based on self-attention and convolutional neural network, comprising the following steps:

[0046] S1: construct an image fusion network; the image fusion network includes: a convolutional neural network module, a visual self-attention module and a loss calculation module;

[0047] Please refer to Figure 2 , Figure 2 It is the structural diagram of the image fusion network;

[0048] It should be noted that in this application, the role of the convolutional neural network module (CNN-Module) is to preprocess the input image, extract the detail information and structural information of the input image, and then input it into the visual self-attention model for further processing. Infrared images and visible light images contain different information. What this application needs to do is to extract different information from different images. Through the convolutional neural network module, structural information and detailed texture information can be effectively obtained from the original image. The size of the feature map generated by the dual-branch structure of the convolutional neural network is consistent with the number of feature layers, which is convenient for further operation in the next module.

[0049] The function of the visual self-attention module (ViT-Module) is to further process the feature map sent by the convolutional neural network module. The information contained in the feature map is extracted by the convolutional neural network. After the above analysis, this type of feature map has no long-term dependency, so in order to circumvent this problem, this application uses the visual self-attention module for further processing. The feature map sent in is processed from the channel perspective and the spatial perspective respectively to extract the long-term correlation information in the original image to the greatest extent. The obtained feature map is then sent to the image reconstruction module, and the initial fused image can be obtained.

[0050] The role of the loss calculation module is to make up for the shortcomings of pixel loss. Pixel loss seeks to make the target image and the original image completely identical, but the result of the fusion task cannot be 100% identical. It needs to be similar to the original image in details and structure. Therefore, feature loss can avoid the consequences of pixel loss.

[0051] S2: training the image fusion network; inputting the infrared image and the visible light image into the convolutional neural network module respectively to obtain a feature map; inputting the feature map into the self-attention module to obtain a long-range correlation of the feature map; using the long-range correlation information of the feature map, inputting the feature map into the image reconstruction unit to obtain an initial fused image; during the training process, using the loss calculation module to perform loss calculation on the initial fused image, and finally obtaining a trained image fusion network;

[0052] It should be noted that the convolutional neural network module adopts a dual-branch structure, including: a densely connected network and a fast downsampling network.

[0053] Please refer to Figure 3 ,like Figure 3 As shown, the convolutional neural network module CNN-Module proposed in this application uses a dual-branch convolutional neural network for image preprocessing and image feature extraction, which is effective for high-quality image restoration tasks based on unsupervised learning.

[0054] The dual-branch convolutional neural module has a detail branch and a structure branch. Before being input into the visual self-attention module for processing, all images to be processed need to be convolved to obtain the initial feature map. The initial feature map is then sent to the detail branch and the structure branch respectively.

[0055] Among them, the upper branch is a densely connected network, which is conducive to making full use of shallow features and avoiding gradient disappearance during training. It is mainly used to extract detailed information in the image; the lower branch is a fast downsampling network, which reduces the size of the feature map while increasing the number of feature maps, and is mainly used to extract structural information in the image. The two convolutional network branches complement each other. While doing image preprocessing, they also extract different information from the two images. The feature maps of the two branches are the same size and the same number of channels. After being connected, it is convenient to send to the next step of processing.

[0056] The densely connected network specifically means that the input of all layers in the network is connected to the output of all previous layers, as shown in the following formula:

[0057] X p =F p ([X0, X1, X2, ..., X p-1 ])

[0058] Among them, F p is the nonlinear operation of the pth layer, [X0, X1, X2, ..., X p-1 ] is the concatenation of all output layers.

[0059] It should be noted that the function of the detail branch is to extract as much texture information as possible from the input image. Therefore, this application modifies the detail branch above the network into a dense block structure (i.e., a densely connected network) and designs the number of layers to 4. Figure 3 As shown in the detail branch, the input of all layers is connected to the output of all previous layers.

[0060] That is, the output of the pth layer is X p =F p ([X0, X1, X2, ..., X p-1 ]). In this formula, F x is the nonlinear operation of the xth layer. Nonlinear operations include norm layer operations, convolution layer operations, and activation layer operations. Among them, [X0, X1, X2, ..., X p-1 ] is the concatenation of all output layers. The reason for designing such a densely connected network is that such a design allows the detail branch to extract the low-level information of the image, so this application allows this branch to specifically extract the low-level information of the image.

[0061] On the other hand, for the input image, not only detail information is needed, but also the structural information of the image needs to be extracted using the structural branch. In order to obtain the global information of the image, this application designs a fast downsampling network structure, also called the structural branch. The specific structure is as follows Figure 3As shown in the lower branch, the input feature map is quickly downsampled, and while doing this, the number of feature channels is increased, with a total of three layers. Finally, the feature map is upsampled to the original size to facilitate the connection with the features of the detail branch. After the original image passes through the parallel network, two sets of feature maps with different characteristics can be obtained, and then the two sets of feature maps are connected and sent to the next module for further processing.

[0062] The specific process of obtaining the feature map in step S2 is:

[0063] S21: performing registration processing on the infrared image and the visible light image to obtain an input image of a preset size;

[0064] S22: Pass the input image through a convolution layer to obtain an initial feature map;

[0065] S23: inputting the initial feature map into the dense connection network and the fast downsampling network respectively to obtain the detail features and structural features of the image respectively;

[0066] S24: Connect the detail features and the structural features to obtain a feature map.

[0067] As an embodiment, during the training phase, the infrared image and the visible light image are first registered, and then their size is set to 256×256. The advantage of presetting the image size is that it is convenient for the subsequent operation of the ViT-Module. Then, the input image is sent to the initial convolution layer, and 16 feature maps are finally obtained. Then, these feature maps are copied separately and put into the detail branch and the structure branch. In the detail branch, four convolution layers are included, which are operated with dense connections. The final feature map size and the input map size remain unchanged, and the number of feature map channels in the middle are 8, 16, 24 and 32 respectively; in the structure branch, we also first use four convolution layers, and the step size is designed to be 2 at this time. The number of feature map channels in the middle is 32, 16 and 8 respectively. Finally, through upsampling, the number of channels is changed to 32, so that the number of channels is consistent with the detail branch of the former, so as to achieve the purpose of fast downsampling. In the last layer of the structure branch, the bilinear structure of image processing is finally used to upsample the feature map to expand its size by 8 times.

[0068] The convolutional neural network module uses the MISH function as an activation function.

[0069] It should be noted that in the field of image processing, ReLU is often used as an activation function. However, the negative half axis of the ordinary ReLU function is 0, that is, negative activation is not used directly, and half of the information is lost. This ReLU function that discards information is more suitable for image classification and segmentation tasks, but it is not suitable for image reconstruction.

[0070] Therefore, this application uses the MISH function similar to the LeakyReLU activation function to make up for this defect, and its negative half axis retains the negative activation. The MISH function used in this network has the characteristic of suppressing errors. Compared with the LeakyReLU activation function, its curve is smoother and more suitable for image reconstruction. The formula is as follows:

[0071] Mish(x) = x·tanh(ln(1+e x ))

[0072] In the above formula, x is the pixel point in the feature map. After being processed by the MISH function, the activation information of the negative half axis is retained.

[0073] Please refer to Figure 4 , Figure 4 It is a structural diagram of a visual self-attention module; the visual self-attention module includes: a channel visual self-attention unit and a spatial visual self-attention unit.

[0074] It should be noted that the feature map output from the previous convolutional neural network module has a size of 256×256 and 64 layers.

[0075] This set of feature maps is input into the visual self-attention module, and the channel visual self-attention (ChannelTransformer) operation and the spatial visual self-attention (SpatialTransformer) operation are performed respectively. The 64-layer VisionTransformer weight map (Map) output by this operation is then dot-multiplied with the original feature map, and the calculated results are input into the image reconstruction module to finally obtain a fused image.

[0076] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the main framework of VisionTransformer;

[0077] Please refer to Figure 6 and Figure 7 , Figure 6 It is the processing flow chart of the spatial visual self-attention unit; Figure 7 It is the processing flow chart of the channel visual self-attention unit;

[0078] It should be noted that Figure 5The existing Vision Transformer is applied to the classification task and uses a classification symbol (CLS Token) to output the final image classification result. However, this task is more inclined to generate a fused image, so this application directly abandons the addition of the CLS Token operation to avoid unnecessary calculations. In addition, this application not only uses the traditional spatial visual self-attention (Spatial Transformer) operation, but also adds the channel visual self-attention (Channel Transformer) operation to further extract multi-layer information in the image.

[0079] refer to Figure 6 ,Spatial Transformer performs self-attention operation on the feature map of each layer, and the result represents the mutual relationship on the feature map, extracting the long-range correlation of each feature map;

[0080] refer to Figure 7 , Channel Transformer performs self-attention operation on all feature maps, which represents the relationship between all feature maps. With the addition of Channel Transformer, this application can pay attention to the relationship between feature maps and ensure that long related information is fully extracted.

[0081] The loss calculation module includes: a pixel loss function and a feature loss function.

[0082] It should be noted that the loss function of the image fusion network mainly includes two parts: one is the conventional pixel-level loss, and the other is the feature map loss.

[0083] The overall pixel loss function is:

[0084] L=αL MsE +βL sSIM +γL gradient

[0085] Among them, α, β and γ are constant coefficients of the three loss function components, which are used to control their sizes and are determined according to specific experiments. The MSE loss function formula is as follows:

[0086]

[0087] The MSE loss function is used to calculate the mean square error and the square error between the input and output. The loss function MSE can obtain the loss of detail information of the input image and the reconstructed image. However, if only the MSE is calculated, the output image will be too smooth and the high-frequency gradient information will be lost. Therefore, this application adds a gradient loss function:

[0088] Lgradient =MSE(Gradient(I re ), Gradient(I in ))

[0089] Where Gradient(x) is the gradient map obtained by sharpening the image using the Laplacian operator. The Laplacian operator performs a mathematical convolution operation. SSIM calculates the structural loss, and the formula is as follows:

[0090]

[0091] The above is the conventional pixel loss, while the feature loss is calculated in different feature layers by processing the original image and the fused image separately through the pre-trained VGG19 network, which can avoid the problems caused by pixel loss. The loss function is the absolute value loss function, and the formula is as follows:

[0092]

[0093] S3: Use the trained image fusion network to perform image fusion to obtain a fused image.

[0094] As an example, the method of the present application is qualitatively compared with the existing advanced infrared visible light fusion method, and the results are as follows: Figure 8 and Fig. 9 As shown; Figure 8 This is the first set of experimental comparison results; Fig. 9 This is the comparison result of the second set of experiments.

[0095] Depend on Figure 8 and Fig. 9 It can be seen that the proposed method retains more detail information and has more salient information than other fusion methods, that is, the effect on the picture is: more detail information is retained and the target is more salient. In addition, this application selects some quantitative indicators EN, MI and SSIM, and the results are as follows Figure 10-12 As shown; Fig.10 It is the result of EN quantitative experiment; Fig.11 It is the result of MI quantitative experiment; Fig.12 It is the SSIM quantitative experimental result;

[0096] In the quantitative results, except for EN which is slightly inferior to the first place, the other two indicators are relatively ideal, which shows that this method has advantages in preserving structural information and brightness information.

[0097] The above experimental results show that the network framework in the present invention is reasonably designed. By introducing the VisionTransformer model, the shortcomings of the traditional convolutional neural network are avoided, and finally a relatively ideal fusion result is obtained.

[0098] The beneficial effect of the present invention is that it can generate a fused image with rich information and ideal effect, and the fused image contains more detail information and saliency information.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An infrared and visible light image fusion method based on self-attention and convolutional neural network, characterized by: S1: construct an image fusion network; the image fusion network includes: a convolutional neural network module, a visual self-attention module and a loss calculation module; S2: training the image fusion network; inputting the infrared image and the visible light image into the convolutional neural network module respectively to obtain a feature map; inputting the feature map into the self-attention module to obtain a long-range correlation of the feature map; using the long-range correlation information of the feature map, inputting the feature map into the image reconstruction unit to obtain an initial fused image; during the training process, using the loss calculation module to perform loss calculation on the initial fused image, and finally obtaining a trained image fusion network; The loss calculation module includes: a pixel loss function and a feature loss function; The pixel loss function is specifically: L=αL MSE +βL SSIM +γL gradient Where α, β and γ are the constant coefficients of the three loss function components respectively; L MSE Used to calculate the mean square error between the infrared image and the initial fusion image; L gradient is the gradient loss, which is used to obtain the high-frequency gradient information of the initial fused image; L SSIM is the structural loss, which is used to calculate the structural loss between the initial fusion image and the infrared image; The feature loss function processes the visible light image and the initial fusion image respectively through the pre-trained VGG19 network, and calculates the loss in different feature layers of the VGG19 network, as shown in the following formula: Among them, MAE(X,Y) represents the feature loss function, N is the total number of training set samples; X n is the nth visible light image in the training set; Y n is the nth initial fusion image in the training set; S3: Use the trained image fusion network to perform image fusion to obtain a fused image.

2. The infrared-visible light image fusion method based on self-attention and convolutional neural network as claimed in claim 1, characterized in that: The convolutional neural network module adopts a dual-branch structure, including: a densely connected network and a fast downsampling network.

3. The infrared-visible light image fusion method based on self-attention and convolutional neural network as claimed in claim 1, characterized in that: The densely connected network specifically means that the input of all layers in the network is connected to the output of all previous layers, as shown in the following formula: X p =F p ([X0,X1,X2,…,X p-1 ]) Among them, F p is the nonlinear operation of the pth layer, [X0,X1,X2,...,X p-1 ] is the concatenation of all output layers.

4. The infrared-visible light image fusion method based on self-attention and convolutional neural network as claimed in claim 3, characterized in that: The specific process of obtaining the feature map in step S2 is: S21: performing registration processing on the infrared image and the visible light image to obtain an input image of a preset size; S22: Pass the input image through a convolution layer to obtain an initial feature map; S23: inputting the initial feature map into the dense connection network and the fast downsampling network respectively to obtain the detail features and structural features of the image respectively; S24: Connect the detail features and the structural features to obtain a feature map.

5. The infrared-visible light image fusion method based on self-attention and convolutional neural network as claimed in claim 1, characterized in that: The convolutional neural network module uses the MISH function as an activation function.

6. The infrared-visible light image fusion method based on self-attention and convolutional neural network as claimed in claim 1, characterized in that: The visual self-attention module includes: a channel visual self-attention unit and a spatial visual self-attention unit.

Images