An Infrared Dual-Band Image Fusion Method Based on Feature Heat Maps

Abstract

The present invention relates to an infrared dual-band image fusion method based on a feature heat map. Through a deep learning detection network and an image fusion network, the neural network algorithm is used for classification and perception characteristics to obtain a feature heat map and an image fusion weight matrix, thereby obtaining an infrared dual-band image fusion method based on a feature heat map, improving the target information of the fused image under the complex and changeable background of infrared detection, and suppressing the background environment. By using the feature heat map, this method changes the characteristic that the original traditional image fusion method can only perform global fusion during fusion, enabling the fusion algorithm to fuse specified regions in the image according to different needs while suppressing the information in the remaining regions, thus realizing image fusion.

Description

Technical Field

[0001] The present invention belongs to the field of image fusion in digital image processing, and relates to an infrared dual-band image fusion method based on a feature heat map. Background Art

[0002] With the increasing complexity of the modern battlefield environment, the long-distance detection requirements of infrared detection devices such as infrared imaging seekers and infrared search and tracking systems are also growing, no longer limited to a single environment and climatic background. However, due to the complex and changeable long-distance detection environment and the harsh meteorological environment, the information volume detected by single-wave infrared can no longer meet the requirements for detecting and identifying targets, resulting in the inability to determine target information. Therefore, the current development mainly relies on dual-wave / multi-band infrared cooperative detection to improve the acquisition of target information, and at the same time, the infrared dual-wave image fusion technology is developed.

[0003] Currently, most methods for infrared dual-band image fusion are carried out through full-image fusion and other methods. Pixel-level or feature-level fusion methods all adopt the global fusion method, enhancing the background information while enhancing the target information, and it is impossible to simultaneously enhance the target and suppress the background information. This leads to background interference during the detection and identification process, and problems such as false detection, missed detection, and misidentification are likely to occur.

[0004] Therefore, based on an in-depth study of existing infrared dual-band image fusion methods, an infrared dual-band image fusion method based on a feature heat map is proposed to highlight the target information in the fused image, suppress the background environment, and improve the autonomy and scene perception adjustment ability of the fusion process. Summary of the Invention

[0005] Technical Problems to be Solved

[0006] In order to avoid the deficiencies of the prior art, the present invention proposes an infrared dual-band image fusion method based on a feature heat map to achieve target enhancement and background suppression in image fusion.

[0007] Technical Solution

[0008] An infrared dual-band image fusion method based on a feature heat map, characterized in that the steps are as follows:

[0009] Step 1: Register the infrared medium-wave and long-wave image data, retain the overlapping part of the image background, crop the two images to the same size, and construct a fused image dataset Data = {(I m1 , I l1 ), (I m2 , I l2 ), … (I mn , I ln )}, where I m represents the medium-wave image, Il Represents the long-wave image;

[0010] Step 2: Divide the dataset into two groups, Data train and Data test , where the Data train image is used as the training sample, and Data test is used as the test sample. Label the Data train data to obtain the labeled data as the input for network training;

[0011] Data label = Label(Data train ) = {(I1, x1, y1, width1, hight1),…,(I n , x n , y n , width n , hight n )}

[0012] Step 3: Construct an image fusion network. The input of the image fusion network is the medium-wave and long-wave images in the training set of Step 2, and the output is the fusion weight matrix of the pixel-level image fusion method:

[0013] ω = Fuse(I m , I L )

[0014] where ω is the fusion weight matrix and Fuse is a deep learning network;

[0015] The input of the detection network is the labeled data, and the output is the position information of the target;

[0016] Step 4: Feed the training data in Step 2 into the image fusion network for training, and feed the labeled data into the detection network for training to obtain the trained model;

[0017] Step 5: Combine the pre-trained models obtained in Step 4. First, the input of the first frame image of the model is the infrared medium-wave and long-wave image data. Obtain the fusion weight through the image fusion model, and fuse the medium-wave and long-wave images according to the fusion weight to obtain the fused image. Second, use the fused image as the input of the detection network. After passing through the detection network, obtain the feature map before the output layer of the detection network, and fuse the feature information according to the feature map weight to obtain the feature heat map. Finally, starting from the second frame, multiply the feature heat map obtained from the previous frame by the fusion weight obtained from the current frame image fusion network to obtain the enhanced fusion weight of the target, and then combine it with the current frame medium-wave and long-wave images to form the fused image;

[0018] Step 6: According to the model obtained in Step 5, group and input the medium-wave and long-wave images in the test set described in Step 2 of the images into the model to obtain a fused image.

[0019] Beneficial effects

[0020] An infrared dual-band image fusion method based on a feature heat map proposed by the present invention, through a deep learning detection network and an image fusion network, uses the classification and perception characteristics of the neural network algorithm to obtain a feature heat map and an image fusion weight matrix, and obtains an infrared dual-band image fusion method based on a feature heat map, which improves the target information of the fused image under a complex and changeable background of infrared detection and suppresses the background environment. By using the feature heat map, this method changes the characteristic that the original traditional image fusion method can only perform global fusion during fusion, enabling the fusion algorithm to fuse specified regions in the image according to different needs while suppressing the information of the remaining regions to achieve image fusion.

[0021] The advantages and beneficial effects of the present invention are mainly reflected in: based on the existing image fusion method, using the perception ability of the deep learning algorithm, realizing the target enhancement of the image fusion algorithm, suppressing the fusion weight in the image of the environment, enhancing the target information, and not affecting the information of the original image at the same time, and finally achieving the purpose of the target information in the fused image. Description of the drawings

[0022] Figure 1 : Image fusion model structure based on a feature heat map;

[0023] Figure 2 : Fusion flow chart based on a feature heat map;

[0024] Figure 3 : Fusion method flow chart based on a feature heat map;

[0025] Figure 4 : Fusion result based on a feature heat map

[0026] a, long-wave sky background; b, medium-wave sky background; c, fusion result; d, long-wave sky background; e, medium-wave sky background; f, fusion result. Specific implementation manners

[0027] Now, the present invention will be further described in combination with embodiments and drawings:

[0028] The present invention provides the following technical solutions: establish an infrared dual-band image fusion model based on a feature heat map, and the model design method includes the following steps, which mainly include three parts: the first part is to preprocess the data set; the second part is the construction and training of the deep learning model; the third part is the construction of the image fusion model and the testing of the fusion model:

[0029] The first part includes two steps:

[0030] Step 1: Obtain infrared medium-wave and long-wave image data, preprocess the images, and construct a fused image dataset Data = {(I m1 , I l1 ), (I m2 , I l2 ), … (I mn , I ln )}, where I m represents the medium-wave image and I l represents the long-wave image.

[0031] Step 2: Divide the dataset described in Step 1 into two groups, Data train and Data test . Use the Data train images as training samples and Data test as test samples. Label the Data train data to obtain the labeled data as the input for network training.

[0032] Data label = Label(Data train ) = {(I1, x1, y1, width1, hight1), …, (I n , x n , y n , width n , hight n )} (1)

[0033] The second part includes two steps:

[0034] Step 3: Construct a deep learning detection network and an image fusion network. The input of the image fusion network is the medium-wave and long-wave images in the training set described in Step 2, and the output is the fusion weight matrix of the pixel-level image fusion method. The input of the detection network is the labeled data, and the output is the position information of the target.

[0035] Step 4: Use the detection and fusion models in Step 3 as the basic model, input the training data and labeled data in Step 2, train the basic model, and obtain the trained basic model for inference.

[0036] The third part includes two steps:

[0037] Step 5: Combine the pre-trained models obtained in Step 4. First, the input of the first frame image of the model is a dual-band infrared image. The fusion weight is obtained through the image fusion model. The medium-wave and long-wave images are fused according to the fusion weight to obtain a fused image. Second, the fused image is used as the input of the detection network. After passing through the detection network, a feature map is obtained before the output layer of the detection network. The feature information is fused according to the feature map weight to obtain a feature heat map. Finally, starting from the second frame, the feature heat map obtained from the previous frame is multiplied by the fusion weight obtained from the current frame image fusion network to obtain a fusion weight after target enhancement, and then combined with the current frame medium-wave and long-wave images to form a fused image.

[0038] Step 6: According to the model obtained in Step 5, group and input the medium-wave and long-wave images in the test set described in Step 2 of the image into the model to obtain fused images.

[0039] The specific implementation method of the present invention will be further described in conjunction with the accompanying drawings:

[0040] The specific steps of the method of the present invention are as follows, divided into three parts: The first part is to preprocess the data set; the second part is the construction and training of the deep learning model; the third part is the construction of the image fusion model and the testing of the fusion model:

[0041] The first part includes two steps:

[0042] Step 1: Obtain infrared medium-wave and long-wave image data, preprocess the images, and construct a fused image data set Data = {(I m1 , I l1 ), (I m2 , I l2 ), … (I mn , I ln )}, where I m represents the medium-wave image and I l represents the long-wave image.

[0043] Step 2: Divide the data set described in Step 1 into two groups, Data train and Data test . The images in Data train are used as training samples, and Data test is used as a test sample. The data in Data train is labeled, and the labeled data is used as the input for network training.

[0044] Data label = Label(Data train ) = {(I1, x1, y1, width1, hight1), …, (I n , x n , y n,width n ,hight n )} (2)

[0045] The second part includes two steps:

[0046] Step 3: Construct a deep learning detection network and an image fusion network. The input of the image fusion network is the medium-wave and long-wave images in the training set described in Step 2, and the output is the fusion weight matrix of the pixel-level image fusion method. The input of the detection network is the labeled data, and the output is the position information of the target. The detection network selects YOLOV5, and the image fusion network selects DeepFuse.

[0047] ω = DeepFuse(I m ,I L ) (3)

[0048] x, y, width, hight = Yolov5(I) (4)

[0049] Step 4: Take the detection and fusion models in Step 3 as the basic model, send the training data and labeled data in Step 2 into it, train the basic model, and obtain the trained basic model for inference.

[0050] model deepfuse = DeepFuse train (Data train (I m ,I l )) (5)

[0051] model yolo = Yolov5 train (Data label (I1,…,I n )) (6)

[0052] The loss functions selected for the image fusion model are structural similarity and mean squared error, and the loss function selected for the detection network is the default loss function of the YOLOV5 network.

[0053]

[0054] The third part includes two steps:

[0055] Step 5: Combine the pre-trained models obtained in Step 4. First, the input of the first frame image of the model is the dual-band infrared image. Through the image fusion model, the fusion weight is obtained, and the medium-wave and long-wave images are fused according to the fusion weight to obtain the fused image.

[0056] ω test = model deepfuse (Datatest (I m ,I l )) (8)

[0057]

[0058] Secondly, take the fused image as the input of YOLOv5. The three-channel outputs of YOLOv5 are used as feature maps. According to the sum of the weights of the feature components in each feature map as the feature heat map of the current feature map, scale the three feature maps to the same size and select the feature with the largest eigenvalue to obtain the feature heat map, as Figure 1 shown.

[0059] feature(f1,f2,f3) = model yolo (I fuse ) (10)

[0060]

[0061] h = max(resize(h1,h2,h3)) (12)

[0062] where f is the feature and h is the feature heat map.

[0063] Finally, starting from the second frame, multiply the feature heat map obtained from the previous frame by the fusion weight obtained from the image fusion network of the current frame to obtain the fusion weight after target enhancement, and then combine it with the medium-wave and long-wave images in the current frame to form a fused image, as Figure 2 shown.

[0064]

[0065] Step 6: According to the model obtained in Step 5, group and input the medium-wave and long-wave images in the test set described in Step 2 of the images into the model to obtain a fused image, as Figure 3 .

[0066] Example implementation effect

[0067] The experimental results are as Figure 4 shown. Among them, there are two groups of infrared small and weak target images under the air background, and the fusion results represent the feature heat map image fusion results based on the above image fusion mode.

Claims

1. An infrared dual-band image fusion method based on feature heat maps, characterized in that The steps are as follows: Step 1: Register the infrared medium-wave and long-wave image data, retain the overlapping part of the image background, crop the two images to the same size, and construct a fused image dataset Data = {(I m1 , I l1 ), (I m2 , I l2 ),...(I mn , I ln )}, where I m1 ,..., I mn represent n frames of medium-wave images, and I l1 ,..., I ln represent n frames of long-wave images; Step 2: Divide the dataset into two groups, Data train and Data test ; use the images in Data train as the training set, and Data test as the test set. Label the Data train data to obtain the labeled data as the input for network training; Data label = Label(Data train ) = {(I1, x1, y1, width1, hight1),...,(I n , x n , y n , width n , hight n )} Step 3: Construct an image fusion network. The input of the image fusion network is the medium-wave and long-wave images in the training set of Step 2, and the output is the fusion weight matrix of the pixel-level image fusion method: ω = Fuse(I m , I L ) where ω is the fusion weight matrix and Fuse is the deep learning network; The input of the detection network is the labeled data, and the output is the position information of the target; Step 4: The training set in Step 2 is fed into the image fusion network for training, and the labeled data is fed into the detection network for training to obtain the trained model; Step 5: Combine the pre-trained models obtained in Step 4. First, the input of the first frame of the model is the medium-wave and long-wave image data of the infrared. The fusion weight is obtained through the image fusion model. The medium-wave and long-wave images are fused according to the fusion weight to obtain the fused image. Secondly, the fused image is used as the input of the detection network. After passing through the detection network, a feature map is obtained before the output layer of the detection network. The feature information is fused according to the feature map weight to obtain the feature heat map. Finally, starting from the second frame, the feature heat map obtained from the previous frame is multiplied by the fusion weight of the current frame image to obtain the enhanced fusion weight of the target, and then combined with the medium-wave and long-wave images of the current frame to form the fused image; Step 6: According to the model obtained in Step 5, the medium-wave and long-wave images in the test set described in Step 2 are grouped and input into the model to obtain the fused images.

Images