An Infrared Dual-Band Image Feature-Level Fusion Method with Intelligent Parameter Learning

Abstract

The present invention relates to an infrared dual-band image feature-level fusion method with intelligent parameter learning. By combining deep learning methods, an unsupervised deep learning network is constructed. The feature adaptive fusion parameter matrix is obtained by using the classification and perception characteristics of the neural network algorithm. Combining with the traditional feature-level image fusion algorithm, an infrared dual-band image feature-level fusion method with intelligent parameter learning is obtained, which solves the problem of intelligent selection of fusion parameters in the complex and changeable background in the air combat environment. By using the deep neural network, this method changes the problem that the original traditional feature-level image fusion method needs to manually set fusion parameters in different environments, enabling the algorithm to generate the parameters required for fusion autonomously according to the environmental background without relying on manual selection, and realizing image fusion.

Description

Technical Field

[0001] The present invention belongs to the field of image fusion in digital image processing, and relates to an infrared dual-band image feature-level fusion method with intelligent parameter learning. Background Art

[0002] Background scenes such as bright clouds in the sky, "fish-scale light" and "bright bands" on the sea surface, scorching deserts, high-temperature adjacent walls, and complex towns cause problems such as "weak" and "small" targets or "dark targets" with background brightness higher than target brightness in long-distance detection by infrared imaging missiles, infrared search and tracking systems, etc. To solve such target detection problems, by studying composite detection systems, such as infrared dual-band imaging detection, new unique and different machine vision and other-dimensional physical features are obtained.

[0003] Currently, most methods for infrared dual-band image feature-level fusion adopt a fixed-parameter fusion method. For example, in traditional feature fusion algorithms and wavelet transform image fusion algorithms, the high-frequency and low-frequency feature fusion parameters are fixed values, and image fusion is achieved by artificially selecting parameter ratios. The imaging effect is strongly related to human subjective selection, and it is impossible to autonomously select fusion parameters according to the input scene image type.

[0004] Therefore, based on in-depth research on traditional infrared dual-band image fusion methods, an infrared dual-band image feature-level fusion method with intelligent parameter learning is proposed to improve the environmental adaptability of feature-level image fusion methods, highlight the information of targets in the fused image, and enhance the autonomy of the fusion process and the optimality of the fusion result. Summary of the Invention

[0005] Technical Problems to be Solved

[0006] In order to avoid the deficiencies of the prior art, the present invention proposes an infrared dual-band image feature-level fusion method with intelligent parameter learning to achieve adaptive selection of feature-level image fusion parameters.

[0007] Technical Solution

[0008] An infrared dual-band image feature-level fusion method with intelligent parameter learning, characterized by the following steps:

[0009] Step 1: Register the infrared medium-wave and long-wave image data, retain the overlapping part of the image background, crop the two images to the same size, and then perform a scaling operation on the images to uniformly scale the image size to 640×640, obtaining the image dataset Image ={(IR m1 ,IR l1 ),(IR m2 ,IR l2 ),...(IR mn ,IRln )}, Two data images form a group, where IR m represents the medium-wave image, and IR l represents the long-wave image;

[0010] Step 2: The data set is divided into two groups, Image train and Image test , Image train images are used as training samples, and Image test is used as a test sample;

[0011] Step 3: Build a deep learning network. The input image size of the network is 640×640, and the output is the fusion parameter matrix of the feature-level image fusion method, whose size is the same as the number of features used by the selected traditional feature fusion algorithm. The essence of the network is a classification network;

[0012] The formula for generating the fusion parameters is as follows:

[0013] A(a1, a2,..., a n ) = FuseClassification(IR m , IR l )

[0014] where A is the fusion parameter matrix and the number of parameters is n;

[0015] Step 4: Use the traditional feature-level fusion method to extract the features of the medium-wave and long-wave images in the training set of Step 2, and convert the images into traditional features:

[0016] Feature m (F m1 , F m2 ,..., F mn ) = FE(IR m )

[0017] Feature l (F l1 , F l2 ,..., F ln ) = FE(IR l )

[0018] where: FE(*) is the traditional feature extraction method;

[0019] Step 5: Use the feature-level image fusion parameters in Step 3 as the fusion parameter matrix of the traditional features in Step 4. The Hadamard product of the fusion parameter matrix A and the feature Feature m plus the Hadamard product of I - A and the feature Feature l is used as the fusion feature. The specific fusion formula is as follows:

[0020] Feature fuse (F fuse1 ,F fuse2 ,...F fusen ) = A·Feature m +(I - A)·Feature l

[0021] Where: Feature fuse is the fused feature;

[0022] Step 6: Use the traditional feature extraction algorithm to perform inverse transformation on the fused feature in Step 5 to obtain the fused image, and thus complete the construction of the fusion model. The specific model:

[0023] IR fuse = FE -1 (Feature fuse )

[0024] Step 7: Input the training data set into the fusion model for training to obtain the deep learning image fusion parameter model;

[0025] Step 8: Group and input the medium-wave and long-wave images in the test set of the image in Step 2 into the model to obtain the fusion parameters, and obtain the fused image according to the fused image fusion formula.

[0026] Beneficial effects

[0027] An infrared dual-band image feature-level fusion method with intelligent parameter learning proposed by the present invention combines the deep learning method on the basis of the traditional method to construct an unsupervised deep learning network. By using the classification and perception characteristics of the neural network algorithm, the feature adaptive fusion parameter matrix is obtained, and combined with the traditional feature-level image fusion algorithm, an infrared dual-band image feature-level fusion method with intelligent parameter learning is obtained, which solves the problem of intelligent selection of fusion parameters under complex and changeable backgrounds in the air combat environment. By using the deep neural network, this method changes the problem that the original traditional feature-level image fusion method needs to manually set fusion parameters in different environments, enabling the algorithm to generate the fusion parameters required according to the environmental background without relying on manual selection, and realizing image fusion.

[0028] The advantages and beneficial effects of the present invention are mainly reflected in: on the basis of the existing traditional feature-level image fusion method, by using the strong classification and perception ability of the deep learning algorithm, the fusion parameters of the feature-level image fusion algorithm are realized to be intelligent, and the fusion parameters can be determined autonomously according to different environments, realizing environment adaptive fusion. At the same time, the fusion effect of the traditional feature fusion method is improved, and the interpretability of deep learning is increased. Description of the drawings

[0029] Figure 1: Infrared dual-band image feature-level fusion model structure;

[0030] Figure 2 : Flow chart of infrared dual-band image feature-level fusion;

[0031] Figure 3 : Flow chart of infrared dual-band image feature-level fusion method;

[0032] Figure 4 : Experimental results of infrared dual-band image feature-level fusion

[0033] a. Long-wave sky background; b. Medium-wave sky background; c. Adaptive feature fusion; d. Long-wave ground background; e. Medium-wave ground background; f. Adaptive feature fusion; g. Long-wave sea background; h. Medium-wave sea background; i. Adaptive feature fusion. Detailed implementation mode

[0034] The present invention will be further described in combination with embodiments and drawings:

[0035] The present invention provides the following technical solution: establish an infrared dual-band image feature-level fusion model with intelligent parameter learning. The model design method includes the following steps, which mainly include three parts: the first part is the screening and data preprocessing of the data set; the second part is the construction of the intelligent parameter model; the third part is the training and testing of the intelligent parameter model:

[0036] The first part includes two steps:

[0037] Step 1, obtain infrared dual-wave image data. The infrared dual-wave image data are images of the real-shot sky, ground, and sea. Since the infrared acquisition instruments are two different instruments, there are partial offsets in the shooting results. Therefore, it is necessary to register the infrared dual-wave image data to reduce the influence of shooting. At the same time, according to the registered results, retain the overlapping part of the image background, crop the two images into the same size, and then perform a scaling operation on the images to uniformly scale the image size to 640×640. The obtained image data set Image = {(IR m1 , IR l1 ), (IR m2 , IR l2 ),...(IR mn , IR ln )}, and the data images are in groups of two, where IR m represents the medium-wave image, and IR l represents the long-wave image.

[0038] Step 2, divide the data set described in Step 1 into two groups Image train and Image test , Imagetrain as a training sample, Image test as a test sample.

[0039] The second part includes four steps:

[0040] Step 3: Construct an unsupervised deep learning network. The input image size of the network is 640×640, and the output is the fusion parameter matrix of the feature-level image fusion method, whose size is the same as the number of features used in the selected traditional feature fusion algorithm. The essence of the network is a classification network. The medium-wave and long-wave images in the training set described in Step 2 are sent into the network to obtain the output fusion parameters. The fusion parameter generation formula is as follows:

[0041] A(a1,a2,...,a n ) = FuseClassification(IR m ,IR l ) (1)

[0042] where A is the fusion parameter matrix, and the number of parameters is n.

[0043] Step 4: According to the selected traditional feature-level fusion method, perform feature extraction on the medium-wave and long-wave images in the training set described in Step 2, and convert the images into traditional features.

[0044] Feature m (F m1 ,F m2 ,...,F mn ) = FE(IR m ) (2)

[0045] Feature l (F l1 ,F l2 ,...,F ln ) = FE(IR l ) (3)

[0046] where FE(*) is the traditional feature extraction method.

[0047] Step 5: Use the feature-level image fusion parameters described in Step 3 as the fusion parameter matrix of the traditional features described in Step 4. The Hadamard product of the fusion parameter matrix A and the feature Feature m plus the Hadamard product of I - A and the feature Feature l is used as the fusion feature. The specific fusion formula is as follows:

[0048] Feature fuse (F fuse1 ,F fuse2 ,...F fusen) = A·Feature m +(I - A)·Feature l (4)

[0049] where Feature fuse is the fused feature.

[0050] Step six, according to the fused feature described in step five, use the inverse transformation of the traditional feature extraction algorithm to reconstruct the image and obtain the fused image, thus completing the construction of the fusion model. The specific model is as Figure 1 shown.

[0051] IR fuse = FE -1 (Feature fuse ) (5)

[0052] The third part includes two steps:

[0053] Step seven, according to the image fusion model described in step six, train through the training dataset described in step two to obtain a deep learning image fusion parameter model.

[0054] Step eight, according to the model obtained in step seven, group and input the medium-wave and long-wave images in the test set described in step two into the model to obtain fusion parameters, and obtain the fused image according to the fused image fusion formula. Specific embodiments:

[0056] The specific steps of the method of the present invention are as follows, divided into three parts: The first part is the screening and data preprocessing of the dataset; the second part is the construction of the parameter intelligent model; the third part is the training and testing of the parameter intelligent model:

[0057] The first part includes two steps:

[0058] Step one, obtain infrared dual-wave image data. The infrared dual-wave image data is the image of the real sky, ground and sea surface. Since the infrared acquisition instruments are two different instruments, there is partial offset in the shooting results. Therefore, it is necessary to register the infrared dual-wave image data to reduce the impact of shooting. At the same time, according to the registered results, retain the overlapping part of the image background, crop the two images into the same size, and then perform a scaling operation on the images to uniformly scale the image size to 640×640. The obtained image dataset Image = {(IR m1 , IR l1 ), (IR m2 , IR l2 ),...(IR mn , IR ln )}, with two data images in a group, where IR m represents the medium-wave image, IRl Indicates the long-wave image.

[0059] Step 2: Divide the dataset described in Step 1 into two groups, Image train and Image test . Image train images are used as training samples, and Image test is used as a test sample.

[0060] The second part includes four steps:

[0061] Step 3: Construct an unsupervised deep learning network. The input image size of the network is 640×640, and the output is the fusion parameter matrix of the feature-level image fusion method, whose size is the same as the number of features used by the selected traditional feature fusion algorithm. The network is essentially a classification network. The main modules in the network consist of a convolutional layer, a BN layer, and a max pooling layer, forming a feature extraction module. The convolutional kernel size is 3×3, and the max pooling kernel is 2×2.

[0062] The network consists of three parts. The first part is the feature extraction part, which consists of two feature extraction modules, mainly extracting the features of the input image. The network parameters of the feature extraction part are shared, that is, the mid-wave and long-wave images share a feature extraction part.

[0063] F m , F l = FeaExa(image train (IR m , IR l )) (6)

[0064] where F m , F l are the high-dimensional features of the mid-wave and long-wave images abstracted by the network respectively.

[0065] The second part is the feature merging part, which merges the high-level features obtained in the first part into a combined feature. The Concatenate operation is used to fuse two high-dimensional features into a high-dimensional combined feature.

[0066] F = concat(F m , F l ) (7)

[0067] where F is the fused feature

[0068] The third part is the fusion parameter generation part, which mainly generates fusion parameters based on features. The third part consists of three feature extraction modules and a fully connected module. The output of the fully connected layer is the fusion parameter matrix.

[0069] A(a1, a2,..., a n) = FuseClassification(IR m , IR l ) (8)

[0070] Where A is the fusion parameter matrix, and the number of parameters is n.

[0071] Step 4: The selected traditional feature fusion method in this project is the Haar wavelet transform. The Haar wavelet transform will output one high-frequency feature and three low-frequency features. According to the selected traditional feature-level fusion method, perform wavelet transform on the medium-wave and long-wave images in the training set described in Step 2, and convert the images into traditional features.

[0072] Feature m (F m1 , F m2 , F m3 , F m4 ) = DWT(IR m ) (9)

[0073] Feature l (F l1 , F l2 , F l3 , F l4 ) = DWT(IR l ) (10)

[0074] Where DWT(*) is the wavelet transform feature extraction method.

[0075] Step 5: Use the feature-level image fusion parameters described in Step 3 as the fusion parameter matrix for the wavelet transform features described in Step 4. The inner product of the fusion parameter matrix A and the feature Feature m plus the inner product of I - A and the feature Feature l is used as the fusion feature. The specific fusion formula is as follows:

[0076] Feature fuse (F fuse1 , F fuse2 , F fuse3 , F fuse4 ) = A · Feature m + (I - A) · Feature l (11)

[0077] Where Feature fuse is the feature after fusing the long-wave image feature and the medium-wave image feature.

[0078] Step 6: Based on the fused features described in Step 5, use the inverse wavelet transform to reconstruct the image and obtain the fused image, thus completing the construction of the fusion model. The specific model is as shown in Figure 2 as follows.

[0079] IR fuse = DWT -1 (Feature fuse ) (12)

[0080] The third part includes two steps:

[0081] Step 7: According to the image fusion model described in Step 4, train it with the training dataset described in Step 2 to obtain a deep learning image fusion parameter model.

[0082] model fuse = FuseClassification train (image train ) (13)

[0083] Step 8: According to the model obtained in Step 5, group the medium-wave and long-wave images in the test set described in Step 2 and input them into the model to obtain the fusion parameters, and obtain the fused image according to the fused image fusion formula. The fusion process is as shown in Figure 3 as follows.

[0084] A test = model fuse (image test (IR m , IR l )) (14)

[0085]

[0086]

[0087] After obtaining the fused image, it is necessary to calculate the structural similarity between the fused image and the original input image, construct the Loss function for deep learning training, and improve the training negative feedback mechanism. Thus, the construction of the fusion model is completed.

[0088] Loss = 0.5 * SSIM(I fuse , I m ) + 0.5 * SSIM(I fuse , I l ) (17)

[0089] where SSIM(*) is the structural similarity calculation formula.

[0090] The main process of the method of the present invention is as shown in Figure 3As shown, according to the characteristics of infrared dual-band images, the advantages of traditional feature-level fusion methods, and the classification perception ability based on deep learning, the present invention provides a feature-level fusion method for infrared dual-band images with intelligent parameter learning. This method can obtain different fusion parameters in real time for different environments to make more full use of the information in infrared dual-band images, improve the environmental perception ability of the fusion algorithm, and increase interpretability. The algorithm of the present invention is simple, highly operable, and has wide applicability.

[0091] Example implementation effect

[0092] The present invention is tested based on real-shot sky, sea surface, and ground environments, and the test results are as Figure 4 shown.

Claims

1. An infrared dual-band image feature-level fusion method with intelligent parameter learning, characterized in that The steps are as follows: Step 1: Register the infrared medium-wave and long-wave image data, retain the overlapping part of the image background, crop the two images into the same size, and then perform a scaling operation on the images to uniformly scale the image size to 640×640, obtaining the image dataset Image={(IR m1 ,IR l1 ),(IR m2 ,IR l2 ),...(IR mn ,IR ln )}, with two data images in a group, where IR m represents the medium-wave image and IR l represents the long-wave image; Step 2: The dataset is divided into two groups, Image train and Image test , where the Image train images serve as training samples and the Image test serves as a test sample; Step 3: Construct a deep learning network. The input image size of the network is 640×640, and the output is the fusion parameter matrix of the feature-level image fusion method, the size of which is the same as the number of features used by the selected traditional feature fusion algorithm. The essence of the network is a classification network; The formula for generating the fusion parameters is as follows: A(a1,a2,...,a n ) = FuseClassification(IR m , IR l ) where A is the fusion parameter matrix, and the number of parameters is n; Step 4: Use the traditional feature-level fusion method to extract features from the medium-wave and long-wave images in the training set in Step 2, and convert the images into traditional features: Feature m (F m1 ,F m2 ,...,F mn ) = FE(IR m ) Feature l (F l1 ,F l2 ,...,F ln ) = FE(IR l ) where: FE(*) is the traditional feature extraction method; Step 5: Use the feature-level image fusion parameters in Step 3 as the fusion parameter matrix for the traditional features in Step 4. The Hadamard product of the fusion parameter matrix A and the feature Feature m plus the Hadamard product of I - A and the feature Feature l is used as the fused feature. The specific fusion formula is as follows: Feature fuse (F fuse1 ,F fuse2 ,...F fusen ) = AFeature m +(I - A)Feature l Wherein: Feature fuse is the fused feature; Step 6: Use the traditional feature extraction algorithm to perform inverse transformation on the fused features in Step 5 to obtain the fused image. Thus, the construction of the fusion model is completed. The specific model: IR fuse = FE -1 (Feature fuse ) Step 7: Input the training data set into the fusion model for training to obtain the deep learning image fusion parameter model; Step 8: Input the medium-wave and long-wave images in the test set of Step 2 into the model in groups to obtain the fusion parameters, and obtain the fused image according to the fused image fusion formula.

Images