A pixel-level fusion method for infrared dual-band images based on intelligent parameter learning

Abstract

This invention relates to a pixel-level fusion method for infrared dual-band images using intelligent parameter learning. By constructing an unsupervised deep learning network through an encoder-decoder architecture, the method leverages the classification and perceptual characteristics of a neural network algorithm to derive an adaptive fusion weight matrix. Combined with a traditional pixel-level image fusion algorithm, this method addresses the problem of intelligently selecting fusion weights under the complex and ever-changing backgrounds of air combat environments. By utilizing an unsupervised deep neural network, this method overcomes the traditional requirement for manually setting fusion parameters under different environments and backgrounds. This method now autonomously generates the required fusion parameters based on the environmental context, independent of manual selection, to achieve image fusion.

Description

Technical Field

[0001] The present invention belongs to the field of image fusion in digital image processing and relates to a parameter intelligent learning method for pixel-level fusion of infrared dual-band images. Background Art

[0002] Infrared imaging missiles and infrared search and tracking systems primarily target various combat aircraft, cruise missiles, and other targets during long-range detection. However, due to the complex and ever-changing battlefield environment, modern detection equipment is no longer limited to inland or coastal areas. The long-range detection environment and background are even more challenging due to the addition of complex weather conditions to combat operations, which have changed from a single continental climate. Therefore, infrared dual-band image fusion systems can provide more abundant raw data and new target feature information.

[0003] Currently, most methods for pixel-level fusion of infrared dual-band images use fixed parameter fusion. For example, in traditional weighted fusion algorithms, image fusion is achieved by manually selecting weight ratios, but it is impossible to autonomously select fusion weights based on the input scene image type.

[0004] Therefore, based on the in-depth study of traditional infrared dual-band image fusion methods, a parameter intelligent learning infrared dual-band image pixel-level fusion method is proposed to improve the environmental adaptability of the pixel-level image fusion method, highlight the target information in the fused image, and improve the autonomy of the fusion process and the optimality of the fusion result. Summary of the Invention

[0005] Technical problems to be solved

[0006] In order to avoid the shortcomings of the existing technology, the present invention proposes a parameter intelligent learning infrared dual-band image pixel-level fusion method to achieve adaptive selection of pixel-level image fusion weights.

[0007] Technical Solution

[0008] A parameter intelligent learning infrared dual-band image pixel-level fusion method is characterized by the following steps:

[0009] Step 1: Register the infrared medium-wave and long-wave image data with the same content, retain the parts that are completely consistent after registration, and crop the rest to obtain the processed medium-wave and long-wave fusion image dataset Data = {(I m1 ,I l1 ),(I m2 ,I l2 ),...(I mn ,I ln )}, where I m Represents the medium wave image, I l Represents a long-wave image;

[0010] Step 2: Divide the dataset into two groups train and Data test ,Data train Images are used as training samples, Data test As a test sample;

[0011] Step 3: Build a deep learning network. The input of the deep learning network is the medium-wave and long-wave images of the training set in step 2. The output is the fusion weight matrix of the pixel-level image fusion method, which has the same size as the image. The fusion weight generation formula is as follows:

[0012] α=FuseNet(I m ,I L )

[0013] Where α is the fusion weight matrix, and FuseNet is the deep learning network;

[0014] Step 4: Use the pixel-level image fusion weights from step 3 as the fusion weight matrix of the traditional pixel-level image fusion method. Add the Hadamard product of the fusion weight matrix and the medium-wave image to the Hadamard product of the difference between the identity matrix and the fusion weight matrix and the long-wave image to obtain the fused image, completing the fusion model construction.

[0015] Fusion formula:

[0016] Among them I fuse is the fused image;

[0017] Step 5: Train the fusion model with the training data set in step 2 to obtain the deep learning image fusion weight model:

[0018] model fuse =FuseNet train (Data train )

[0019] Step 6: Group the medium-wave and long-wave images in the test set of step 2 and input them into the model trained in step 5, and obtain the fused image according to the fusion image fusion formula.

[0020] The deep learning network can be unsupervised or supervised.

[0021] Beneficial effects

[0022] This paper proposes a parameter-intelligent learning method for pixel-level fusion of infrared dual-band images. This method uses an encoder-decoder structure to construct an unsupervised deep learning network. The method utilizes the classification and perception characteristics of a neural network algorithm to derive an adaptive fusion weight matrix. Combined with a traditional pixel-level image fusion algorithm, this method addresses the problem of intelligently selecting fusion weights in complex and changing air combat environments. By utilizing an unsupervised deep neural network, this method overcomes the traditional pixel-level image fusion method's requirement for manually setting fusion parameters under different environments and backgrounds. This method now autonomously generates the required fusion parameters based on the environmental context, independent of manual selection, to achieve image fusion.

[0023] The advantages and beneficial effects of the present invention are mainly reflected in: on the basis of the existing traditional pixel-level image fusion method, the strong classification and perception capabilities of the deep learning algorithm are utilized to realize the intelligent fusion parameters of the pixel-level image fusion algorithm, and the fusion weight can be independently determined according to different environments to realize environmental adaptive fusion. At the same time, it improves the fusion effect of the traditional method, increases the interpretability of deep learning, and ultimately achieves the purpose of highlighting the target information in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 : Pixel-level fusion model structure of infrared dual-band images;

[0025] Figure 2 : Pixel-level fusion flow chart of infrared dual-band images;

[0026] Figure 3 :Flowchart of infrared dual-band image pixel-level fusion method;

[0027] Figure 4 : Pixel-level fusion results of infrared dual-band images

[0028] a, long-wave sky background; b, medium-wave sky background; c, fusion weight; d, medium-wave-long wave; e, long-wave-medium wave; f, long-wave ground background; g, medium-wave ground background; h, fusion weight; i, medium-wave-long wave; j, long-wave-medium wave; k, long-wave sea background; l, medium-wave sea background; m, fusion weight; n, medium-wave-long wave; o, long-wave-medium wave. DETAILED DESCRIPTION

[0029] The present invention will now be further described with reference to the embodiments and accompanying drawings:

[0030] The present invention provides the following technical solution: establishing a parameter intelligent learning infrared dual-band image pixel-level fusion model. The model design method includes the following steps, which mainly include three parts: the first part is preprocessing the data set; the second part is model construction; and the third part is training and testing the fusion model:

[0031] The first part consists of two steps:

[0032] Step 1: Obtain infrared medium-wave and long-wave image data, perform pre-processing operations such as registration and cropping on the medium-wave and long-wave data, and obtain the processed medium-wave and long-wave fusion image dataset Data = {(I m1 ,I l1 ),(I m2 ,I l2 ),…(I mn ,I ln )}, where I m Represents the medium wave image, I l Represents long-wave image

[0033] Step 2: Divide the data set described in step 1 into two groups train and Data test ,Data train Images are used as training samples, Data test As a test sample.

[0034] The second part consists of two steps:

[0035] Step 3: Based on the encoder-decoder structure, an unsupervised deep learning network is constructed. Its input is the medium-wave and long-wave images in the training set described in step 2, and its output is the fusion weight matrix of the pixel-level image fusion method, which has the same size as the image. The fusion weight generation formula is as follows:

[0036] α=FuseNet(I m ,I L ) (1)

[0037] Where α is the fusion weight matrix and FuseNet is the deep learning network.

[0038] Step 4: The pixel-level image fusion weights described in step 3 are used as the fusion weight matrix of the traditional pixel-level image fusion method. The Hadamard product of the fusion weight matrix and the medium-wave image is added to the Hadamard product of the difference between the unit matrix and the fusion weight matrix and the long-wave image to obtain the fused image, completing the fusion model construction. The specific model is as follows: Figure 1 shown.

[0039] The specific fusion formula is as follows:

[0040]

[0041] Among them I fuse is the fused image.

[0042] The third part consists of two steps:

[0043] Step 5: According to the image fusion model described in step 4, the training data set described in step 2 is used for training to obtain a deep learning image fusion weight model.

[0044] Step 6: Based on the model obtained in step 5, the medium-wave and long-wave images in the test set described in step 2 are grouped and input into the model to obtain fusion weights, and the fused image is obtained according to the fusion formula. Specific embodiment:

[0046] The specific implementation method of the present invention is further described with reference to the accompanying drawings:

[0047] The specific steps of the method of the present invention are as follows, which are divided into three parts: the first part is preprocessing the data set; the second part is model construction; and the third part is training and testing the fusion model:

[0048] The first part consists of two steps:

[0049] Step 1: Obtain infrared medium-wave and long-wave image data. The medium-wave and long-wave data are images taken at the same time and angle. The medium-wave and long-wave data are registered to eliminate the influence of spatial displacement. At the same time, according to the registration result, the two images are cropped to the same size and the areas with different scenes in the two images are eliminated. The processed medium-wave and long-wave fusion image dataset Data = {(I m1 ,I l1 ),(I m2 ,I l2 ),...(I mn ,I ln )}, two data images are grouped together, where I m Represents the medium wave image, I l Represents long-wave image

[0050] Step 2: Divide the data set described in step 1 into two groups train and Data test ,Data train Images are used as training samples, Data test As a test sample.

[0051] The second part consists of two steps:

[0052] In step three, based on the encoder-decoder architecture, the DeepFuse network was selected as the core network to construct an unsupervised deep learning network. The encoder consists of a two-layer convolutional neural network with a 3×3 kernel size. This twin architecture shares network parameters, and both images pass through the same encoder. The encoder primarily serves as a feature extractor, converting images into high-dimensional features.

[0053] Fm ,F l =Encoder(Data train (I m ,I l )) (3)

[0054] Among them F m ,F l They are high-dimensional features after encoding medium-wave and long-wave images.

[0055] After obtaining two high-dimensional features, the Concatenate operation is used to fuse the two high-dimensional features into one high-dimensional feature as the input of the decoder.

[0056] F=concat(F m ,F l ) (4)

[0057] Where F is the fused feature

[0058] The decoder consists of a three-layer convolutional neural network. Its essence is to perform deconvolution decoding on the fused high-dimensional features to obtain the required pixel fusion weight matrix. The decoder convolution kernel size is 3×3.

[0059] α=Decoder(F) (5)

[0060] Where α is the fusion weight matrix.

[0061] The input of step 3 is the medium-wave and long-wave images in the training set described in step 2, and the output is the fusion weight matrix of the pixel-level image fusion method, which has the same size as the image. The fusion weight generation formula is as follows:

[0062] α=FuseNet(I m ,I L ) (6)

[0063] Among them, FuseNet is a deep learning network.

[0064] Step 4: Use the pixel-level image fusion weights described in step 3 as the fusion weight matrix of the traditional pixel-level image fusion method, add the Hadamard product of the fusion weight matrix and the medium-wave image, and add the difference between the unit matrix and the fusion weight matrix and the Hadamard product of the long-wave image to obtain the fused image.

[0065] The specific fusion formula is as follows:

[0066]

[0067] Among them I fuse is the fused image, and E is the unit matrix.

[0068] After obtaining the fused image, it is necessary to calculate the structural similarity and mean square error between the fused image and the original input image, construct the Loss function for deep learning training, and improve the training negative feedback mechanism to complete the fusion model construction. The specific model is as follows: Figure 1 shown.

[0069]

[0070] Among them, SSIM(*) is the formula for calculating structural similarity, and MSE(*) is the formula for calculating mean square error

[0071] The third part consists of two steps:

[0072] Step 5: According to the image fusion model described in step 4, the training data set described in step 2 is used for training to obtain a deep learning image fusion weight model.

[0073] model fuse =FuseNet train (Data train ) (9)

[0074] Step 6: Based on the model obtained in step 5, the medium-wave and long-wave images in the test set described in step 2 are grouped and input into the model to obtain the fusion weights, and the fusion image is obtained according to the fusion image fusion formula. The fusion process is as follows: Figure 2 shown.

[0075] α test =model fuse (Data test (I m ,I l )) (10)

[0076]

[0077] The main process of the method of the present invention is as follows Figure 3 As shown, based on the characteristics of infrared dual-band images, the advantages of traditional pixel-level fusion methods, and the classification perception capabilities of deep learning, this paper proposes a parameter-intelligent learning method for infrared dual-band image pixel-level fusion. This method fully utilizes the infrared dual-band image information and uses an encoder-decoder structure to construct a fusion parameter adaptive network. This improves the network's perception range, intelligently perceives background environmental changes, and can derive different fusion parameters for different environments in real time. This improves the fusion algorithm's environmental perception capabilities, increases interpretability, and highlights target information in the environment. The algorithm is simple, highly operational, and has broad applicability.

[0078] Example implementation effect

[0079] The present invention is tested based on real-life shooting of the sky, sea and ground environments. The test results are as follows: Figure 4 The fusion weight is the visualization of the real value multiplied by 255. Medium-wave-long-wave refers to the image input order of medium-wave image first and long-wave image second, while long-wave-medium-wave refers to the image input order of long-wave image first and medium-wave image second.

Claims

1. A parameter intelligent learning infrared dual-band image pixel-level fusion method, characterized by Here are the steps: Step 1: Register the infrared medium-wave and long-wave image data with the same content, retain the parts that are completely consistent after registration, and crop the rest to obtain the processed medium-wave and long-wave fusion image dataset Data = {(I m1 ,I l1 ),(I m2 ,I l2 ),…(I mn ,I ln )}, where I m Represents the medium wave image, I l Represents a long-wave image; Step 2: Divide the dataset into two groups train and Data test ,Data train Images are used as training samples, Data test As a test sample; Step 3: Build a deep learning network. The input of the deep learning network is the medium-wave and long-wave images of the training set in step 2. The output is the fusion weight matrix of the pixel-level image fusion method, which has the same size as the image. The fusion weight generation formula is as follows: α=FuseNet(I m ,I L ) Where α is the fusion weight matrix, and FuseNet is the deep learning network; Step 4: Use the pixel-level image fusion weights from step 3 as the fusion weight matrix of the traditional pixel-level image fusion method. Add the Hadamard product of the fusion weight matrix and the medium-wave image to the Hadamard product of the difference between the identity matrix and the fusion weight matrix and the long-wave image to obtain the fused image, completing the fusion model construction. Fusion formula: Among them I fuse is the fused image; Step 5: Train the fusion model with the training data set in step 2 to obtain the deep learning image fusion weight model: model fuse =FuseNet train (Data train ) Step 6: Group the medium-wave and long-wave images in the test set of step 2 and input them into the model trained in step 5, and obtain the fused image according to the fusion image fusion formula.

2. The infrared dual-band image pixel-level fusion method with intelligent parameter learning according to claim 1 is characterized by: The deep learning network can be unsupervised or supervised.

Images