Visible and infrared image fusion method based on three-branch autoencoder network

Abstract

The present application belongs to the technical field of computer image processing, and particularly relates to a visible light and infrared image fusion method based on a three-branch auto-encoder network, wherein the infrared image has clear contour structure, the visible light image has high resolution and clear texture, and the fusion of the two forms an image with the advantages of both. The fusion process in the present application is based on a unique auto-encoder network, and the network is characterized by having three branches, i.e. a texture branch, a content branch and a base branch. In the encoder, important features of the infrared image and the visible light image are extracted through the three branches, then two groups of features are fused to obtain fusion features, the decoder decodes the fusion features to obtain the fused image, and the fused image has the advantages of both the infrared image and the visible light image, has richer image information and higher contrast, and has better visual effect.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer image processing, and particularly relates to a visible light and infrared image fusion method based on a three-branch autoencoder network. BACKGROUND

[0002] Image fusion is mainly used for synthesizing complementary information of multiple source images. The same scene is identified by generating a fused image. The source images can be obtained from different types of imaging sensors, and images obtained by different sensors have different characteristics. The fused image has the advantages and characteristics of all source images, and is easier for human to view or machine to identify. In recent years, with the development of image applications such as digital photography, remote sensing images, video monitoring and medical images, the field of image fusion in image processing has become increasingly hot. Fusing infrared and visible light images is an important research direction. Infrared images can distinguish targets and backgrounds according to radiation differences, and can work well in both day and night. Compared with the global structural information of infrared images, visible light images provide high-resolution and clear texture details. Fusing these two kinds of images can combine the advantages of thermal radiation information in infrared images and detailed texture information in visible light images. High-quality fused images have many practical applications.

[0003] As a popular technology in the field of image processing, image fusion has various effects and can make ordinary images have more information and better quality. However, many current image fusion technologies are based on traditional fusion methods, which ignore the processing of image features. SUMMARY

[0004] In order to solve the above technical problems, the application provides an excellent infrared image and visible light image fusion method, which well processes image features, and the specific technical scheme is as follows:

[0005] A visible light and infrared image fusion method based on a three-branch autoencoder network, comprising the following steps:

[0006] Step 1: obtaining infrared and visible light source images from a set data set, the obtained infrared source images and visible light source images correspond to each other;

[0007] Step 2: preprocessing the obtained infrared source images and visible light source images to obtain a training set trainDatas;

[0008] Step 3: An infrared image and visible light image fusion model framework based on a three-branch autoencoder network is established, and the infrared and visible light image fusion model framework comprises an encoder A, a fusioner B and a decoder C, wherein the encoder A is used for extracting features of the infrared image and the visible light image, the fusioner B is used for fusing the features of the infrared image and the visible light image, and the decoder C is used for decoding the fused infrared image and visible light image features to obtain a fused image;

[0009] Step 4: According to the set parameters, a loss function is obtained; the loss function is set to ensure that the fused image retains more effective information of the source image as much as possible;

[0010] Step 5: According to the trainDatas training set and the loss function, the infrared and visible light image fusion model framework is trained to obtain an infrared visible light fusion model;

[0011] Step 6: The infrared image and the visible light image to be fused are input into the infrared visible light model obtained in step 5 to obtain a fused image, and the step is ended.

[0012] Further improvement of the application, the pre-processing process of the infrared source image and the visible light source image in step 2 includes: first, a plurality of infrared source images and visible light source images are selected, and then the source images are cut into source image blocks of a set size; during the source image cutting process, the cutting positions of the corresponding infrared source image and the visible light source image are consistent.

[0013] Further improvement of the application, the encoder in step 3 comprises three branches for extracting features, which are a texture branch, a content branch and a base branch, and in the encoder, important features of the infrared image and the visible light image are extracted through the three branches.

[0014] Further improvement of the application, the first branch of the three branches in step 4 is a texture branch, and a four-layer dense connection structure is used in the texture branch. The input of each layer of convolution is cascaded with the output of all previous layers. x x [O0, O1, O2, …, O x-1 ]. L x represents various operations of the xth layer, and these operations are often nonlinear, such as CNN and normalization. O0, O1, O2, …, O x-1 represents the output of all layers. The dense connection operation used allows the texture branch to obtain surface texture features and edge features of the source image. The image size is unchanged in the processing process.

[0015] ​Further improvement of the present application, the second branch of the three branches in step 4, the content branch, is used to extract the main content information of the image, and a fast down-sampling network structure is used to obtain the features. The features are down-sampled three times, reducing the size while increasing the number of channels. Finally, the content features are up-sampled to the size of the source image.

[0016] Further improvement of the present application, the third branch of the three branches in step 4, the base branch, extracts the image base information and features using several corresponding convolution operations, ensuring the integrity and minimum distortion of the image. The image after convolution is the size of the source image.

[0017] Further improvement of the present application, the features extracted by the three branches in step 4 are cascaded.

[0018] During testing, a set of corresponding infrared images and visible light images are input into the encoder, and through the three branches, a set of features is obtained. The set of features is put into the fusion layer for fusion, and the obtained fusion features are decoded by the decoder to obtain the final image.

[0019] Further, in step 3, the encoder A extracts the image features using a 3x3 convolution kernel to perform the first convolution operation on the image to obtain a set of features with 32 channels, and then inputs the set of features into the three branches. In the texture branch, the features are convolved four times, and then a dense connection operation is performed. The feature size remains unchanged, and the number of channels increases from 16 to 32, 48, and 64. In the content branch, 4 convolutions with a step size of 2 are performed to down-sample the features. The feature size is reduced to 64, 32, and 16. The number of channels first increases to 64 and 128, and then decreases to 64. At this time, the number of channels in the texture branch and the content branch is the same. At the end of the content branch, the features are bilinearly interpolated to up-sample the image by 8 times. In the base branch, the image is first convolved once to change the number of image channels from 32 to 64, and then three convolutions are performed to keep the final number of channels at 64, and the image size remains at 128x128.

[0020] Further, in step 3, the fusion device B uses addition strategy and channel strategy for fusion. The addition strategy directly adds the two sets of features and then averages them, and the formula is as follows:

[0021] F f x,y=(F r x,y+F v x,y) / 2

[0022] Where F r represents infrared features, F v represents visible light features, and F fRepresent the fusion features. Use (x, y) to represent the position of the corresponding pixel points between features. Channel strategy, first F v and F r (F∈R H × W × C Global average pooling is a v , a r , a∈R 1 × 1 × C . Then calculate the corresponding weight vector b v , b r , and F v and F r are multiplied by b v , b r to get enhanced feature channels and Finally, add and corresponding pixels one by one to get the final F f .

[0023] The specific steps are as follows:

[0024]

[0025]

[0026] Further, characterized in that the step 3 decoder C, in the decoder, using a 3x3 convolution kernel, the initial channel number is the sum of the branch channel number 192, after several times of convolution to decode, the feature channel number becomes one, and the final fused image is output

[0027] Further, characterized in that the activation function uses LeakyReLU activation function, the formula is as follows:

[0028] LeakyReLU(x)=max(0,x)+leak*min(0,x)

[0029] Further, characterized in that the loss function in step 5, the loss function designs three kinds of loss formula as follows:

[0030] L=L pix +αL color +βL profile

[0031] Where, α, β are weight parameters used to adjust the loss between each other.

[0032] L pix Loss function is used to calculate the error between the source image and the fused image in pixels, Iin I is the input source image out I is the fused image. The formula is as follows:

[0033] L pix = MSE(I out , I in )

[0034]

[0035] Since the visible light image and the infrared image are different in brightness and color, the L color color loss function is used to calculate the color histogram error of the source image and the fused image. Histogram(x) is the color histogram of x. Since the gray level of color is 0-255, the number of histogram vertical bars is set to 255 in this paper, and the histogram calculates the histogram color error of the fused image and the source image. The histogram results of the input image and the output image are subtracted, and then the two-norm is calculated and divided by 255 as the color loss. The formula is as follows:

[0036]

[0037] L profile is the contour loss function, which calculates the similarity of the structural contour of two images, which is used to calculate the similarity between the input image and the output image, and the formula is as follows:

[0038] L profile = 1-SSIM(I out , I in ).

[0039] The beneficial effects of the present application are as follows: the present application fuses the infrared image with the visible light image, and the fused image has the advantages of the infrared image and the visible light image, including the clear contour and structure of the infrared image and the rich and delicate main information of the visible light image. From the intuitive naked eye, the fused image has more information than the infrared image and the visible light image before fusion, and the evaluation index of the experimental data can also prove that the fused image has high quality. The fusion technology proposed in the present application can be applied to image processing, monitoring identification and other related research and application. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is the flowchart of the present application.

[0041] Figure 2 is the autoencoder network structure diagram in the present application.

[0042] Figure 3 is the structure diagram of the fused image using the autoencoder network in the present application.

[0043] Figure 4 For the addition strategy fusion process chart in the application.

[0044] Figure 5 For the channel number strategy fusion process chart in the application.

[0045] Figure 6 For the image fusion effect chart in the application.

[0046] Specific implementation

[0047] In order to deepen the understanding of the application, the application will be further described in detail below in combination with the drawings and examples, which are only used to explain the application and do not constitute a limitation on the protection scope of the application.

[0048] Embodiment: as Figure 1 shown, a visible light and infrared image fusion method based on a three-branch autoencoder network, comprising the following steps:

[0049] Step 1: obtain the source images of infrared and visible light from the set data set, and the obtained infrared source images and visible light source images correspond to each other;

[0050] Step 2: pre-process the obtained infrared source images and visible light source images to obtain the training set trainDatas;

[0051] Step 3: establish an infrared image and visible light image fusion model framework based on a three-branch autoencoder network, the infrared and visible light image fusion model framework comprising an encoder A, a fusioner B and a decoder C, wherein the encoder A is used to extract the features of the infrared image and the visible light image, the fusioner B is used to fuse the features of the infrared image and the visible light image, and the decoder C is used to decode the fused infrared image and visible light image features to obtain the fused image;

[0052] Step 4: obtain the loss function according to the set parameters; by setting the loss function, the fused image can retain more effective information of the source image as much as possible;

[0053] Step 5: train the infrared and visible light image fusion model framework according to the trainDatas training set and the loss function to obtain an infrared visible light fusion model;

[0054] Step 6: input the infrared image and visible light image to be fused into the infrared visible light model obtained in step 5 to obtain a fused image, and end the step.

[0055] The unique three-branch autoencoder network proposed in this embodiment can extract different features of the source image through three branches respectively, and the extracted features are used for image fusion. I rrepresents the infrared image, using I v represents the visible light image, using I f represents the fused image.

[0056] As shown in Figure 2 , the encoder network has three branches, which are texture branch, content branch and base branch. In the encoder, in order to make the operation more representative and more intuitive, the size of the image and the number of channels are roughly represented by the size of the cube. The larger the front area of the cube represents the larger the image size, and the thicker the cube represents the more image channels. The infrared image I r and the visible light image I v will be adjusted to a fixed size. During training, a certain number of I r and I v are input. First, I r or I v is convolved to obtain a set of features, and then the obtained features are simultaneously passed through the three branches.

[0057] The first branch of the network, the texture branch, is used to extract the edge information of the source image. In the texture branch, the embodiment uses a four-layer dense connection structure. If Figure 1 the first branch of the encoder is shown, the input of each layer of convolution is cascaded with the output of all previous layers, for example, the output O x of the xth layer is O x ([O0, O1, O2, …, O x-1 ]). L x represents various operations of the xth layer, which are often nonlinear, such as CNN and normalization. O0, O1, O2, …, O x-1 represents the output of all layers, and the use of this dense connection operation enables the texture branch to obtain the surface texture features and edge features of the source image, and the processing process does not change the image size.

[0058] The second branch of the network is the content branch, which is used to extract the main content information of the image. In the present embodiment, a fast down-sampling network structure is designed to obtain features, which are down-sampled three times to reduce the size while increasing the number of channels, and then the content features are up-sampled to the size of the source image.

[0059] The third branch is the base branch, which uses several corresponding convolution operations to extract the basic information and features of the image, ensuring the integrity and minimum distortion of the image. The convolved image is the size of the source image, and finally the features extracted by the three branches are cascaded. During testing, a corresponding set of I r and I vInto the encoder, through three branches, get a set of features, put this set of features into the fusion layer fusion, get the fusion features through the decoder to get the final image. As Figure 3 shown.

[0060] This embodiment uses two fusion methods, including addition strategy and channel strategy. As Figure 4 shown. Figure 5

[0061] The addition strategy directly adds two sets of features and then averages them. The method is simple but effective. The formula is as follows:

[0062] F f (x,y=(F r (x,y+F v (x,y) / 2

[0063] Where F r represents infrared features, F v represents visible light features, and F f represents fusion features. (x, y) is used to represent the position of corresponding pixel points between features. Since infrared images and visible light images have their own characteristics, such as visible light images having more abundant main content information, while infrared images having more obvious contour and edge information, a channel strategy is designed as Figure 4 shown. F v and F r (F∈R H×W×C ) are globally averaged to a v , a r , a∈R 1×1×C , then the corresponding weight vector b v , b r is calculated, and F v and F r are multiplied by b v and b r respectively to obtain enhanced feature channels and Finally, F f is obtained by adding corresponding pixels of and one by one. The specific steps are as follows:

[0064]

[0065]

[0066]

[0067]

[0068] ​First, the image is preprocessed to adjust the image size to 128x128, and then the first convolution operation is performed on the image to obtain a set of features with 32 channels. Then, the set of features is input into three branches. In the texture branch, the features are convolved four times, and then a dense connection operation is performed to keep the feature size unchanged and the number of channels accumulated from 16 to 32, 48, and 64. In the content branch, 4 convolutions with a step size of 2 are performed to downsample the features, and the feature size is reduced to 64, 32, and 16, and the number of channels is first increased to 64 and 128, and then reduced to 64. At this time, the number of channels in the texture branch and the content branch is the same. At the end of the content branch, the bilinear interpolation of the features is performed to upsample the image by 8 times. In the base branch, the image is first convolved once to change the number of image channels from 32 to 64, and then three convolutions are performed to keep the final number of channels at 64, and the image size is always maintained at 128x128. Finally, in the decoder, the starting channel number is the sum of the channel numbers of the three branches, which is 192, and several convolutions are performed to decode the features to change the number of feature channels to one, and the final fused image is output. In all the above steps, a 3x3 convolution kernel is used. The network parameters are shown in Table 1.

[0069] ReLU activation function is the most commonly used activation function in neural networks, but it will lose negative activation. This paper uses LeakyReLU activation function to solve this problem. The formula is as follows:

[0070] LeakyReLU(x)=max (0,x)+leak*min(0,x)

[0071] Table 1 Network parameters

[0072]

[0073] In this embodiment, three kinds of loss are designed, pixel loss, color loss and contour loss. The formula is as follows:

[0074] L=L pix +αL color +βL profile

[0075] Where, α, β are weight parameters used to adjust the weight between each loss.

[0076] L pix Loss function is used to calculate the error between source image and fused image in pixels, I in is the input source image, I out is the fused image. The formula is as follows:

[0077] L pix =MSE(I out , I in )

[0078]

[0079] Since visible light image and infrared image are different in brightness and color. Use L color Color loss function calculates the color histogram error of source image and fused image. Histogram(x) is the color histogram of x. Since the gray level of color is 0-255, the number of histogram vertical bars is set to 255, and the histogram calculates the histogram color error of the fused image and the source image. The histogram results of the input image and the output image are subtracted and then the two norms are calculated and divided by 255 as the color loss. The formula is as follows:

[0080]

[0081] L profile is the contour loss function, the contour loss function calculates the similarity of two images in structural contour, here is used to calculate the similarity between the input image and the output image, the formula is as follows:

[0082] L profile = 1-SSIM(I out , I in )

[0083] NVIDIA A100 graphics card is used for experiment. In the loss function, set α = 0.001, set β = 1000, make the loss of each part as close as possible. In the training stage, use COCO train2017 as the training set, which contains thousands of pictures, use 5120 pictures in it as the training set. Use 10 pairs of images in TNO dataset as test set. First, convert the color images in the training set to grayscale images, and then adjust their size to 128x128. Train 64 images each time, train 100 iterations, and set the learning rate to 1e-4. In the test process, the source image needs to be input to the network, and then the fused image can be output.

[0084] A "Pedestrian with Umbrella" image in the TNO dataset is used for testing. The main content of the image is clear and distinct, and the umbrella and pavilion are well overlapped, which can test the image fusion quality. The subjective evaluation and objective evaluation are combined to evaluate the quality of the fused image. The fusion method of the present application is compared with some classic or popular fusion methods. These methods include Laplacian pyramid (LP), gradient transfer and total variation minimization (GTF), curvelet wave transform (CVT), cross bilateral filter fusion method (CBF), low-pass ratio pyramid (RP), wavelet transform (Wavelet), multi-resolution singular value decomposition (MSVD), deep fusion (Deepfuse), double discriminator generative adversarial network (DDcGAN), perceptual generative adversarial network (FusionGan), dense fusion (Densefuse) and residual network fusion (ResNetFusion). The subjective evaluation refers to the feeling of the naked eye observation. For example, the brightness, color, contrast, clarity and the like of the image. As shown in Figure 6 The addition strategy of the embodiment distinguishes the content of the umbrella and the pavilion to a great extent, the posture of the pedestrian is relatively clear, and the texture of the middle part of the branch is also relatively clear, but there is some noise. The channel strategy in this paper performs well on the whole image, including the distinction between the umbrella and the pavilion, and the branch content is also relatively clear and obvious, and the noise and artifacts of the whole image are also less.

[0085] Of course, the subjective evaluation varies from person to person and cannot be used as a stable and reliable evaluation standard. Seven objective evaluation indicators are introduced in this paper, which are SCD, Qabf, SSIM, SF, EI, EN and MI. SCD represents the sum of the correlation of the image difference, Qabf represents the fusion quality, SSIM represents the structural similarity between two images, SF represents the mutation degree in the image, EI represents the contrast intensity of adjacent pixels, EN represents the information contained in the image, and MI represents the correlation of two images. As shown in Table 2, the addition strategy performs best in the SSIM index, meaning that the addition strategy maximally preserves the structural similarity of the infrared and visible light images. The addition strategy performs second in the SCD index, indicating that the fused image has a smaller gap with the source image. The channel strategy achieves the first result in Qabf, SF and EI, and SF and EI perform outstandingly. The channel strategy also achieves the second result in EN and MI, which means that the fused image of the channel strategy has strong adjacent pixel contrast, contains a large amount of information, has great correlation with the source image, and has good fusion quality in many aspects.

[0086] Table 2 Evaluation indexes of various fusion methods

[0087]

[0088] The present application proposes a novel three-branch autoencoder network for image fusion. It is mainly used for infrared image and visible light image fusion at present, and the fused image has better overall quality. The network includes an encoder, a fusion layer and a decoder. The encoder in this paper has three branches, which realizes different feature extraction. The addition strategy and channel strategy are used for fusion in the fusion layer, and finally the decoder outputs the fused image.

[0089] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, and these changes and improvements fall within the scope of the claimed present application.

Claims

1. A method for fusion of visible light and infrared images based on a three-branch autoencoder network, characterized in that, Includes the following steps: Step 1: Obtain source images of infrared and visible light from the set dataset. The obtained infrared source images correspond to the visible light source images. Step 2: Preprocess the acquired infrared source image and visible light source image to obtain the training set trainDatas; Step 3: Establish an infrared and visible light image fusion model framework based on a three-branch autoencoder network. The infrared and visible light image fusion model framework includes encoder A, fusion unit B, and decoder C. Encoder A is used to extract features from infrared and visible light images, fusion unit B is used to fuse features from infrared and visible light images, and decoder C is used to decode the features from the fused infrared and visible light images to obtain the fused image. Step 4: Obtain the loss function based on the set parameters. By setting the loss function, ensure that the fused image retains as much effective information as possible from the source images. Step 5: Train the infrared and visible light image fusion model framework based on the trainDatas training set and loss function to obtain the infrared and visible light fusion model; Step 6: Input the infrared image and visible light image to be fused into the infrared and visible light model obtained in Step 5 to obtain the fused image, and end the step; The encoder A in step 3 includes three branches for extracting features from infrared and visible light images: texture branch, content branch, and basic branch. The texture branch uses a four-layer dense connection structure, where the input of each convolutional layer is concatenated with the outputs of all preceding layers, and the output of the x-th layer is O. x =L x ([O0,O1,O2,…,O x-1 ]), L x Represents the various operations at level x, O0, O1, O2, ..., O x-1 This represents the output of all cascaded layers; The content branch is used to extract the main content information of the image. It uses a fast downsampling network structure to obtain features, performs downsampling on the features three times to reduce the size while increasing the number of channels, and finally upsamples the content features to the size of the source image. The basic branch extracts basic information and features of the image through several corresponding convolution operations, ensuring the integrity of the image and minimizing distortion. The image after convolution is the same size as the source image. The features extracted from the three branches of encoder A are concatenated. During testing, a set of corresponding infrared and visible light images are input into the encoder. Through the three branches, a set of features is obtained. This set of features is then put into the fusion layer for fusion. The resulting fused features are then decoded by the decoder to obtain the final image.

2. The visible light and infrared image fusion method based on a three-branch autoencoder network according to claim 1, characterized in that, The preprocessing of the infrared source image and the visible light source image in step 2 includes: first, selecting a set number of infrared source images and visible light source image pairs, and then cropping the source images into source image blocks of a set size; during the source image cropping process, the corresponding infrared source images and visible light source images are cropped at the same position.

3. The visible light and infrared image fusion method based on a three-branch autoencoder network according to claim 2, characterized in that, In step 3, encoder A extracts image features using a 3x3 convolution kernel. The image undergoes its first convolution operation to obtain a set of features with 32 channels. This set of features is then input into three branches. In the texture branch, the features are convolved four times, followed by dense connection operations, keeping the feature size constant while increasing the number of channels from 16 to 32, 48, and 64. In the content branch, four convolutions with a stride of 2 are performed for downsampling, reducing the feature size to 64, 32, and 16. The number of channels first increases to 64 and 128, then decreases back to 64. At the end of the content branch, bilinear interpolation of the features is performed to upsample the image by 8 times. In the base branch, the image is first convolved once, changing the number of channels from 32 to 64, and then three more convolutions are performed, ensuring the final number of channels remains 64. The image size is consistently maintained at 128×128.

4. The visible light and infrared image fusion method based on a three-branch autoencoder network according to claim 3, characterized in that, In step 3, in fusion unit B, the fusion method uses an addition strategy and a channel strategy. The addition strategy directly adds the two sets of features and then averages them, as shown in the following formula: F f (x,y)=(F r (x,y)+F v (x,y)) / 2 Where F r Represents infrared characteristics, F v F represents the characteristics of visible light. f Representing the fused features, (x, y) represents the corresponding pixel positions between features. The channel strategy first... v and F r F∈R H×W×C Global average pooling is a v a r , a∈R 1×1×C Then calculate the corresponding weight vector b. v b r Then F v and F r and b respectively v b r Enhanced feature channels through multiplication and Finally and The corresponding pixels are added one by one to get the final F. f The specific steps and formulas are as follows:

5. The visible light and infrared image fusion method based on a three-branch autoencoder network according to claim 4, characterized in that, In step 3, the decoder C uses a 3x3 convolution kernel. The initial number of channels is the sum of the three branch channels, which is 192. After several convolutions, the feature channel number becomes one, and the final fused image is output.

6. The visible light and infrared image fusion method based on a three-branch autoencoder network according to claim 5, characterized in that, The activation function used is LeakyReLU, as shown in the following formula: LeakyReLU(x)=max(0,x)+leak*min(0,x).

7. The visible light and infrared image fusion method based on a three-branch autoencoder network according to claim 6, characterized in that, The loss function in step 5 has three possible formulas, as follows: L=L pix +αL color +βL profile Where α and β are weighting parameters used to adjust the relationships between the various losses; L pix The loss function is used to calculate the pixel-level error between the source image and the fused image, I in For the input source image, I out The formula for the fused image is as follows: L pix =MSE(I out ,I in ) Use L color The color loss function calculates the color histogram error between the source image and the fused image. Histogram(x) is the color histogram of x. The color loss is calculated by subtracting the histogram results of the input and output images, taking the L2 norm, and dividing by 255. The formula is as follows: L profile This is the contour loss function, which calculates the similarity between two images in terms of structural contours. Here, it is used to calculate the similarity between the input and output images, and the formula is as follows: L profile =1-SSIM(I out ,I in )。

Images