An infrared and visible light image fusion method incorporating layer features

Abstract

An infrared and visible light image fusion method incorporating layer features is proposed. This method uses a convolutional neural network to obtain the image depth features of both infrared and visible light images, which are then fed into fully connected layers to obtain weights. Layer features of the two images are then extracted. Using the weights of the two source images, their respective grayscale pixel values, and their respective layer features, fusion is performed at each layer in a multi-scale space. The fused images from each layer in the multi-scale space are then reconstructed to obtain the fused image. This method improves the contrast between the target and background while maintaining target sharpness, solves the problem of blurring in background detail areas, and enhances the image fusion effect, resulting in higher quality fused images than existing CNN algorithms. Furthermore, it does not introduce additional training parameters, does not increase the training complexity of deep learning methods, and can be used in image processing devices with high resolution requirements.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a method for enhancing image fusion effect by integrating layer features. BACKGROUND

[0002] In the fields of electrical fault detection and intelligent transportation, infrared and visible light image fusion technology is a basic but very important problem. Infrared images rely on the temperature difference imaging of targets and backgrounds, fault points and non-fault points, and people and objects. Infrared images are not easily affected by lighting conditions and can achieve all-day monitoring, but the background information of the images is single. Visible light images rely on light reflection imaging, and the images have rich background information. Infrared and visible light image fusion methods are used to extract important visual information features from infrared and visible light images and integrate them to obtain a fused image. In order to make the background detail area of the fused image richer and the target area more prominent, feature extraction of infrared and visible light images is the main problem considered in image fusion methods.

[0003] Existing infrared and visible light image fusion methods mainly include two types of methods: traditional methods and deep learning-based methods. Traditional methods usually decompose images into several parts in the frequency domain, then use infrared and visible light image pixel value features to obtain the weight values of infrared and visible light images, and set different fusion rules for different parts. Deep learning-based methods use deep learning networks to extract deep features of infrared and visible light images, and use the deep features to obtain the weight values of the two images, and use the pixel information of the source images for fusion. With the success of deep learning in image processing, deep learning-based image fusion methods, especially CNN-based methods, have attracted increasing attention and have become one of the most popular methods.

[0004] However, deep learning-based methods believe that deep features are the most suitable for human visual observation requirements, so in CNN-based methods, deep features extracted by convolutional networks are used for weight calculation, and pixel values of source images are used for feature fusion. This method is currently widely accepted, but using only deep features of images for image fusion is not the best way to represent images, because the extraction of deep features of images is generally for the purpose of target detection, and the deep features mainly represent features centered on targets, while losing the pixel values of the background detail area. SUMMARY

[0005] In view of the fact that the existing CNN method only uses deep features of images for infrared and visible light fusion, which may result in loss of background information, the present application provides a fusion method that integrates layer features to clearly display background information. This method provides a layer feature extraction method for representing image features, and integrates layer features to construct an infrared and visible light image fusion method based on a CNN fusion framework.

[0006] To achieve the above-mentioned purposes, the present application provides an infrared and visible light image fusion method with layer features, comprising the following steps:

[0007] 1. The CNN fusion framework takes infrared image and visible light image as two source images, respectively, and each of the two source images is subjected to three-layer convolution operation and linear correction unit ReLU to obtain the deep features of the source images, and then the deep features of the two source images are respectively input into the full connection layer to obtain the weight values;

[0008] characterized in that it further comprises the following steps:

[0009] 2. The first layer convolution operation and linear correction unit ReLU are used to realize layer feature extraction of the two source images, respectively;

[0010] 3. The weight values of the two source images, the respective gray values and the respective layer features of the two source images are used to perform fusion in each layer space of the multi-scale space;

[0011] 4. The fused images in each layer space of the multi-scale space are subjected to image reconstruction to obtain the fused image.

[0012] Preferably, the output method of the weight values comprises the following two steps:

[0013] 1.1. The infrared and visible light images are respectively taken as source images for color space conversion, and the color source images are converted from RGB color space to gray space to obtain respective gray images, and the formula is as follows:

[0014]

[0015] wherein I is the gray value of the source image at the (x, y) position, R, G and B are the red, green and blue brightness values of the pixels in the color image;

[0016] 1.2. The gray images are input into the CNN fusion framework to output the weight values.

[0017] Preferably, the specific method of inputting the gray images into the CNN fusion framework to output the weight values is as follows:

[0018] 1.2.1. The gray images are subjected to the first layer convolution operation and linear correction unit ReLU to obtain the Conv1 layer feature map of the source image, and the number of convolution kernels is 64 and the size is 3*3;

[0019] 1.2.2. The Conv1 layer feature map is input into the second layer convolution operation and linear correction unit ReLU to obtain the Conv2 layer feature map of the source image, and the number of convolution kernels is 128 and the size is 3*3;

[0020] 1.2.3, input the Conv2 layer feature map into a max pooling layer to obtain a feature map Conv2_pool of the source image;

[0021] 1.2.4, input the Conv2_pool into a third layer convolution operation and a linear correction unit ReLU to obtain a Conv3 layer feature map of the source image, i.e. a deep feature of each of the infrared and visible light source images;

[0022] 1.2.5, fully connect the deep features of the infrared and visible light source images obtained in 1.2.4 respectively to obtain a weight of the infrared and visible light images.

[0023] Preferably, the method for extracting the layer features of the two source images comprises the following steps:

[0024] 2.1, split the Conv1 layer feature map of the two source images along the third dimension to obtain 64 two-dimensional feature images;

[0025] 2.2, the layer features LFA and LFB of the two source images are calculated as follows:

[0026]

[0027]

[0028] wherein LFA is the layer feature of the infrared image, and LFB is the layer feature of the visible light image; is a two-dimensional Conv1 layer feature image of the infrared image, is a two-dimensional Conv1 layer feature image of the visible light image, and L is a constant 64.

[0029] Preferably, the fusion process comprises the following steps:

[0030] 3.1, normalize the weight of the two source images, the respective gray value, and the respective layer feature of the two source images;

[0031] 3.2, decompose the normalized gray value and layer feature of the two source images using a Laplacian pyramid respectively, and decompose the normalized weight using a Gaussian pyramid;

[0032] 3.3, in each layer of the obtained decomposition, use the gray value and the layer feature to represent the feature of the source image together, and the specific calculation is as follows:

[0033] FIR total = a*LFA + b*I IR

[0034] FVI total = a*LFB + b*I VI

[0035] where a and b are weight factors of layer feature and source image, a = 0.2, b = 0.8; I IR and I VI are the gray values of infrared and visible light images; LFAand LFBare the layer features of infrared and visible light images, respectively; FIR total and FVI total are the feature values of infrared and visible light images, respectively;

[0036] 3.4, using the formula for comparing sizes in applied mathematics, the feature values of infrared and visible light images are compared in terms of significance to obtain a similarity value, as follows

[0037]

[0038] where s is a support window centered at pixel (x, y) with a size of 3*3; M is the similarity value of the feature values of infrared and visible light images, FIR total and FVI total are the feature values of infrared and visible light images, respectively;

[0039] The pixels of infrared and visible light images are divided into similar regions and dissimilar regions using the similarity value, M≥t is a similar region, otherwise, it is a dissimilar region, where t = 0.6.

[0040] 3.5, in the similar region, a fusion method based on weight value is used to obtain a fused image, specifically as follows

[0041] FU(x, y) = w(x, y) · FIR total (x, y) + (1 - w(x, y)) · FVI total (x, y)

[0042] where FU(x, y) is the fused image, FIR total and FVI total are the feature values of infrared and visible light images, respectively, and w is the weight value.

[0043] In the dissimilar region, a maximum value fusion method is used to obtain a fused image, specifically as follows

[0044] FU(x, y) = max(FIR total (x, y), FVI total (x, y))

[0045] where FU(x, y) is the fused image, FIR total and FVI total are the feature values of infrared and visible light images, respectively.

[0046] The present application has the characteristics and benefits that: compared with the prior art, the present application extracts the layer features in the deep learning CNN network, and adds the layer features to the image fusion framework to generate a fusion image, while maintaining the target clarity, improving the contrast of the target and the background, solving the problem of blurred background detail area, enhancing the image fusion effect, and obtaining a higher quality fusion image than the existing CNN algorithm. Moreover, no additional training parameters are introduced, and the complexity of training in the deep learning method is not increased, and the method can be used in image processing equipment with high clarity requirements. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 Flow chart of the deep learning CNN network-based infrared and visible light image fusion method with layer features;

[0048] Figure 2 Comparison result diagram of layer features and deep feature images;

[0049] Figure 3 Comparison result diagram of fusion images under the scene of "person + background wall";

[0050] Figure 4 Comparison result diagram of fusion images under the scene of "person + car + background wall";

[0051] Figure 5 Comparison result diagram of fusion images under the scene of "bicycle + roadside";

[0052] Figure 6 Comparison result diagram of fusion images under the scene of "person + roadside + leaves";

[0053] Figures 3-6 In the figure, (a) is an infrared image, (b) is a visible light image, (c) represents the result of the existing CNN fusion framework, and (d) represents the result of the present application. DETAILED DESCRIPTION

[0054] The specific implementation process of the present application will be described in detail below with reference to the accompanying drawings.

[0055] Reference Figure 1 The specific implementation process of the present application is as follows:

[0056] 1. In the existing CNN fusion framework, the deep features of the infrared image and the visible light image are used to calculate the weight.

[0057] The existing CNN fusion framework takes the infrared image and the visible light image as the source images, respectively, and obtains the deep features of the source images through three layers of convolution operation and linear correction unit ReLU. Then the deep features of the two source images enter the full connection layer to obtain the weight, which includes the following steps:

[0058] (1) Respectively, the infrared and visible light images are taken as source images for color space conversion, and the color source images are converted from the RGB color space to the gray space to obtain gray images, and the formula is as follows:

[0059]

[0060] Wherein, I is the gray value of the source image at the (x, y) position, R, G, B are the red, green and blue brightness values of the pixels in the color image.

[0061] (2) The obtained gray image of the source image is subjected to the first layer of convolution operation and the linear correction unit ReLU to obtain the Conv1 layer feature map of the source image. The parameter settings are as follows: the number of convolution kernels is 64, and the size is 3*3;

[0062] (3) The Conv1 layer feature map is input into the second layer of convolution operation and the linear correction unit ReLU to obtain the Conv2 layer feature map of the source image. The parameter settings are as follows: the number of convolution kernels is 128, and the size is 3*3;

[0063] (3) The Conv2 layer feature map is input into the maximum pooling layer to obtain the feature map Conv2_pool of the source image;

[0064] (4) The feature map Conv2_pool is input into the third layer of convolution operation and the linear correction unit ReLU to obtain the Conv3 layer feature map of the source image. The Conv3 layer feature map is the deep feature of the source image;

[0065] (5) The deep features of the source images obtained in 1.1.2.4 from the infrared and visible light source images are fully connected to obtain the weight.

[0066] 2, the first layer of convolution operation and the linear correction unit ReLU are used to realize the layer feature extraction of the source image, and the specific steps are as follows:

[0067] Because the number of convolution kernels is 64, the obtained Conv1 layer feature is a three-dimensional matrix, and the Conv1 layer feature map is split along the third dimension to obtain 64 two-dimensional feature images;

[0068] The layer feature of the infrared and visible light images is calculated as follows:

[0069]

[0070]

[0071] Wherein, LFA and LFB are the layer features of the infrared and visible light images. is the Conv1 layer two-dimensional feature image of the infrared image, is a two-dimensional feature image of the Conv1 layer of the visible light image, and L is a constant of 64.

[0072] In the infrared and visible light image fusion method, effective image feature extraction is the key, and its effect directly affects the quality of the subsequent fusion image. The layer feature extraction of the present application splits along the third dimension by using the Conv1 layer features of the infrared image and the visible light image, and adds the absolute values of the 64 two-dimensional feature images to obtain the low-level content information such as contour, edge, color, texture, corner and shape, while retaining the image contrast between the target and the background, more details of the background area are retained, laying a solid foundation for the final image fusion result.

[0073] 3. The infrared and visible light image fusion is carried out in each layer space of the multi-scale space by using the weight values of the two source images, the respective gray values and the respective layer features of the two source images, and the specific fusion process is as follows:

[0074] (1) The gray values of the infrared and visible light images, the weight values of the two source images and the layer features of the two source images are normalized;

[0075] (2) The normalized source images and layer features are decomposed by using the Laplace pyramid, and the normalized weight values are decomposed by using the Gaussian pyramid;

[0076] (3) In each layer of the obtained decomposition, the gray value and the layer feature of the source image are used to represent the features of the source image together, and the specific calculation is as follows:

[0077] FIR total =a*LFA+b*I IR

[0078] FVI total =a*LFB+b*I VI

[0079] Wherein, a and b are weight factors of the layer feature and the source image, a = 0.2, b = 0.8; I IR and I VI are the pixel gray values of the infrared and visible light images; LFA and LFB are the layer features of the infrared and visible light images, FIR total and FVI total are the feature representations of the infrared and visible light images;

[0080] (4) The popular size comparison formula in mathematics is applied to compare the saliency of the infrared and visible light image features, and the similarity value is obtained, as follows

[0081]

[0082] Where s is a 3x3 support window centered at pixel (x,y); M is the similarity value between infrared and visible light image features. total and FVI total This represents the feature representation of the infrared and visible light images obtained in step 3.3;

[0083] (5) Based on the similarity value, the pixels of the infrared and visible light images are divided into similar and dissimilar regions. M ≥ t is a similar region, otherwise it is a dissimilar region; where t = 0.6.

[0084] (6) In similar regions, a weight-based fusion method is used to obtain a fused image, as shown in the following formula.

[0085] FU(x,y)=w(x,y)·FIR total (x,y)+(1-w(x,y))·FVI total (x,y)

[0086] Where FU(x,y) is the fused image, and FIR... total and FVI total , respectively, are the feature representations of infrared and visible light images, and w is the weight obtained in step 1;

[0087] In dissimilar regions, a maximum value-based fusion method is used to obtain the fused image, as shown in the following formula:

[0088] FU(x,y)=max(FIR total (x,y),FVI total (x,y))

[0089] Where FU(x,y) is the fused image, and FIR... total and FVI total These represent the features of infrared and visible light images, respectively.

[0090] This invention normalizes the weights, layer features, and grayscale values ​​of two source images. The normalized source images and layer features are then decomposed using a Laplacian pyramid, and the normalized weights are decomposed using a Gaussian pyramid. In each layer of the resulting decomposition, the source image features are represented using weighted source image layer features and grayscale values, resulting in a more comprehensive representation of image pixel features. Then, based on the feature comparison results, the images are partitioned. In similar regions, a weight-based fusion method is used to obtain a fused image; in dissimilar regions, a maximum value-based fusion method is used, resulting in a final fused image that exhibits better fusion performance.

[0091] 4. Reconstruct the fused images from each layer of the multi-scale space to obtain the fused image.

[0092] Test Example

[0093] The feasibility of the infrared and visible image fusion method incorporating layer features proposed in the present application is verified below through specific tests. The present application method and the existing CNN framework fusion method are compared and analyzed under the classic evaluation indexes: information entropy (EN), standard deviation (SD), spatial frequency (SF), average gradient (AG), and sum of correlation differences (SCD) (the test sample is from the LLVIP public dataset). The greater the value of the five evaluation indexes, the higher the quality of the fused image.

[0094] 1. Working conditions:

[0095] The test platform of the present application is implemented on a Windows 10 computer with a 2.80GHz Intel Core i7-7700HQ CPU, a GeForce GTX1050Ti, and 16GB of memory. The programming language is MATLAB.

[0096] 2. Test content and result analysis:

[0097] As shown in Figure 2 , (a) and (b) are infrared images, Figure 2 (c) and (d) are layer features of the infrared images, Figure 2 (e) and (f) are depth features of the infrared images. Figure 2 (g) and (h) are visible light images, Figure 2 (i) and (j) are layer features of the visible light images, Figure 2 (k) and (l) are depth features of the visible light images. As can be seen from the observation rectangular area, the depth features mainly capture the high-order features of the object, but some detailed features are lost. Figure 2 As shown in Figure 2 (c) and (d) and Figure 2 (e) and (f), the thermal radiation information extracted in the layer features is more abundant than in the depth features. Figure 2 In Figure 2 (i) and (j) and (k) and (l), the contour of the pedestrian crossing is prominent and the image brightness is higher in the layer features, while in the depth features, the pedestrian crossing is mostly missing.

[0098] The visual quality comparison results of the present application and the CNN algorithm under different scenes are shown in Figures 3 to 6 . The objective evaluation of the four groups of infrared and visible light images is shown in Table 1.

[0099] As can be seen from Figures 3 to 6 , under different scenes, the visual quality of the fused image of the present application algorithm performs better than the CNN algorithm. As shown in Figure 3(c) The text on the wall is blurred and low in brightness, and the low contrast of the image background results in missing information of the manhole cover on the ground. Figure 5 (c) In the lower left corner, the contrast between the curb and the ground is low, resulting in poor visual effect of the crosswalk. Figure 5 (d) In (d), the edge profile of the crosswalk, the electric pole, the tree branch and the ground texture details can be clearly seen.

[0100] As can be seen from Table 1, all objective evaluation index values of the algorithm of the application are obviously better than those of the CNN algorithm. According to the definition of the objective evaluation index, the algorithm of the application has better performance in image contrast between the target and the background, object definition and comprehensive content representation of the fused image.

[0101] Table 1 Comparison of average values of objective indexes

[0102]

[0103] The above test results show that the fused image of the algorithm of the application is better than the CNN algorithm in visual quality and objective evaluation. The infrared and visible light image fusion method of the application which integrates layer features is suitable for fusing infrared and visible light images from different scenes, and the fused image can fully extract thermal radiation information of the infrared image and texture detail information of the visible light image, thereby improving the quality of the fused image and solving the problem of losing background information by only using deep features.

Claims

1. An infrared and visible light image fusion method integrating layer features, comprising the following steps: (1) a CNN fusion framework takes infrared and visible light images as two source images, each of which is subjected to three layers of convolution operation and linear correction unit ReLU to obtain the deep features of the source images, and then the deep features of the two source images are respectively input into a fully connected layer to obtain weights; (2) the first layer of convolution operation and linear correction unit ReLU are used to realize layer feature extraction of the two source images respectively; (3) the weights of the two source images, the pixel values of the respective gray images, and the respective layer features of the two source images are used to perform fusion in each layer of the multi-scale space; (4) the fused images in each layer of the multi-scale space are subjected to image reconstruction to obtain the fused image; the method for obtaining the weights comprises: (1.1) color space conversion is performed on the infrared and visible light images as source images to convert the color source images from RGB color space to gray space to obtain respective gray images, and the formula is as follows: wherein I is the gray value of the source image at the (x, y) position, R, G, and B are the red, green, and blue brightness values of the pixels in the color image; (1.2) The gray image of the obtained source image is subjected to a first layer convolution operation and a linear correction unit ReLU to obtain a Conv1 layer feature map of the source image, the number of convolution kernels is 64, and the size is 3 3; (1.3) The Conv1 layer feature map is input to a second layer convolution operation and a linear correction unit ReLU to obtain a Conv2 layer feature map of the source image, the number of convolution kernels is 128, and the size is 3 3; (1.4) the Conv2 feature map is input into a maximum pooling layer to obtain the feature map Conv2_pool of the source image; (1.5) the feature map Conv2_pool is input into the third layer of convolution operation and linear correction unit ReLU to obtain the Conv3 layer feature map of the source image, which is the deep feature of the source image; (1.6) the deep features of the infrared and visible light two source images are fully connected to obtain the weights of the infrared and visible light images; the layer feature extraction method comprises: (2.1) since the number of convolution kernels is 64, the obtained Conv1 layer feature is a three-dimensional matrix, and the Conv1 layer feature map is split along the third dimension to obtain 64 two-dimensional feature images; (2.2) the layer features of the infrared and visible light images are calculated as follows: Wherein, LFA is a layer feature of the infrared image, LFB is a layer feature of the visible light image; is a two-dimensional feature image of the Conv1 layer of the infrared image, is a two-dimensional feature image of the Conv1 layer of the visible light image, and L is a constant 64. the feature fusion method comprises: (3.1) the gray values of the infrared and visible light images, the weights of the two source images, and the layer features of the two source images are normalized; (3.2) the normalized source images and layer features are decomposed using a Laplacian pyramid, and the normalized weights are decomposed using a Gaussian pyramid; (3.3) in each layer of the obtained decomposition, the gray value and the layer feature of the source image are used to represent the features of the source image together, and the specific calculation is as follows: where a and b are weight factors of the layer feature and source image, a = 0.2, b = 0.8; and are the gray values of the infrared and visible light images; LFA and LFB are the layer features of the infrared and visible light images, respectively; and are the feature values of the infrared and visible light images, respectively; (3.4) the features of the infrared and visible light images are compared in terms of saliency to obtain a similarity value, as follows: Wherein, s is a support window with a size of 3*3 centered at the pixel (x, y); M is a similarity value of the infrared and visible light image features, and are the feature values of the infrared and visible light images, respectively. (3.5) according to the similarity value, the pixels of the infrared and visible light images are divided into a similar region and a dissimilar region, M≥t is the similar region, and otherwise it is the dissimilar region; in the formula, t=0.6; (3.6) in the similar region, a weight-based fusion method is used to obtain the fused image, and the specific formula is as follows: wherein FU(x, y) is a fusion image, and are eigenvalues of the infrared and visible light images, respectively, and w is a weight. in the dissimilar region, a maximum value-based fusion method is used to obtain the fused image, and the specific formula is as follows: where FU(x, y) is the fused image, and are the feature values of the infrared and visible light images, respectively.

Images