Infrared image and visible light image fusion method based on improved GAN network

Abstract

The application relates to an infrared image and visible light image fusion method based on an improved GAN network, which comprises the following steps: firstly, decomposing the infrared image and the visible light image into basic layer images and detail layer images through a guided filter; then, enhancing the basic layer image of the visible light image through a histogram mapping mode, and fusing the basic layer image by using an improved LT algorithm; secondly, extracting feature information of source images by combining a generator and an encoder in the improved GAN algorithm, and guiding the fusion of the detail layer images of the infrared and visible light images in a manner of adversarial learning; and finally, setting relevant parameters for the basic layer fusion image and the detail layer fusion image, and obtaining the final fusion image by adopting a weighted average method. The application effectively avoids the defects that a traditional fusion method excessively depends on complex fusion rules, retains global structure features and local textures of the image, and makes the image have a natural visual effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an infrared image and visible light image fusion method based on an improved GAN network. BACKGROUND

[0002] Infrared and visible light image fusion is a core technology in multi-sensor image fusion. Infrared image refers to the thermal radiation image of an object, which can produce clear foreground contour information under the influence of environmental factors such as light, and the disadvantage is that the resolution is low. Visible light image has the advantages of high resolution and rich target texture details, and the defect is that it is easily affected by environmental factors such as fog, light, etc. The fusion of the two types of images can effectively concentrate the advantages and make up for the disadvantages of each other to produce a clear and large amount of information image, so the technology has a wide application in target detection, object recognition and military detection fields.

[0003] Image fusion is an information technology that fuses the details, contours and other characteristics of different images into a new image. The fused image of infrared and visible light not only has clear texture details of visible light image, but also has anti-interference characteristics of infrared image. Image fusion methods are roughly divided into three types. One is the traditional method using transform domain, mainly including Laplacian pyramid fusion method and non-subsampled contourlet transform (NSCT). This kind of algorithm analyzes the image through filter according to the characteristics of the image, so as to realize fusion, and the shortcoming is that it depends too much on complex fusion rules. The second is the fusion method based on spatial domain, mainly including principal component analysis algorithm, which reduces the image information by dimensionality reduction of the data set, and the shortcoming is that the operation amount is large. The third is the fusion method based on deep learning, which mainly combines convolutional neural network and image processing to better extract the depth features of the image and prepare for the subsequent fusion stage. Since 2018, GAN network has been gradually applied to the field of image fusion. This method is roughly divided into two stages: in the network stage, the images to be fused are used as a training set to train the network, which contains a generator and a discriminator to extract features and guide the generation of images, and the images are fused through the trained model. The GAN algorithm solves the problem of complex fusion rules in traditional algorithms, but the random network initialization parameters make the results of the adversarial generation stage uncertain, and the extracted image features have a fuzzy effect, resulting in noise and artifacts in the fused image. SUMMARY

[0004] To solve the problems in the prior art, the present application provides an infrared image and visible light image fusion method based on an improved GAN network, which can increase the detail texture information of the fused image of infrared image and visible light image, effectively reduce the artifacts and noise, and retain high contrast.

[0005] To achieve the above object, the present application adopts the following technical scheme:

[0006] An infrared image and visible light image fusion method based on an improved GAN network, comprising the following steps:

[0007] (1) Using a guided filter to respectively solve the basic layer image of the infrared image I1 and the basic layer image of the visible light image I2 and respectively calculate the corresponding detail layer image and the detail layer image

[0008] (2) Using a histogram mapping algorithm to enhance the basic layer image to obtain the enhanced basic layer image Then, the basic layer image and the enhanced basic layer image are fused by using an improved LT algorithm to obtain the basic layer fusion image F b , and the detail layer image and the detail layer image are used as a training set to train the fusion model of the improved GAN network to obtain the detail layer fusion image F d , wherein the improved GAN network comprises a generator, an encoder, a D z iscriminator and a target discriminator, the input detail layer image is mapped to a low-dimensional feature vector z through convolution operation by the encoder, the low-dimensional feature vector z and a label vector l are input into the generator, the low-dimensional image is mapped into a high-dimensional image in the generator using two deconvolution methods, the label vector l is connected to the low-dimensional feature vector z, the new vector [z, l] is fed back to the generator and outputs the detail layer fusion image F d , the detail layer image is used as a target image, a D z iscriminator is constructed between the detail layer image and the detail layer image , the D z iscriminator is used to force the distribution of the generated low-dimensional feature vector z to gradually approach the prior, a target discriminator is constructed between the detail layer image and the detail layer fusion image F d , the target discriminator is used to perform adversarial learning between the detail layer fusion image F d and the detail layer image ;

[0009] (3) The basic layer fusion image F b and the detail layer fusion image F dThe corresponding pixel points are weighted and added to obtain a final fusion image.

[0010] The infrared image and visible light image fusion method based on the improved GAN network provided in the application is a hierarchical image fusion method based on an improved generative adversarial network (GAN) and Laplace transformation (LT). The method first decomposes the infrared image and visible light image into basic layer images and detail layer images through a guide filter; then, the basic layer image of the visible light image is enhanced by histogram mapping to improve the contour effect of the fused image, and the improved LT algorithm is used to fuse the basic layer image; secondly, the generator and encoder in the improved GAN algorithm are combined to extract the feature information of the source image, and the detail layer images of the infrared and visible light images are fused in a guided manner through adversarial learning, so that the fused image has more detailed information; finally, related parameters are set for the basic layer fusion image and the detail layer fusion image, and a weighted average method is used to obtain the final fusion image. The application effectively avoids the disadvantages of traditional fusion methods that excessively rely on complex fusion rules, retains the global structural features and local textures of the image, and makes the image have a natural visual effect. The application has the following beneficial effects:

[0011] 1) The end-to-end characteristics of the GAN network are used to avoid the excessive reliance of traditional registration methods on manual design of weight distribution and complex fusion rules, so that the algorithm performance is improved;

[0012] 2) The basic layer image of the low-light visible light image is enhanced by histogram mapping, and then fused with the basic layer image of the infrared image through the improved LT algorithm, so that the fused image has good contrast and overall appearance;

[0013] 3) The target image and the generated image are discriminated according to the double discriminators and the encoder, the network parameters are constrained by the low-rank prior decomposition to guide the generated image, so that the anti-interference ability is improved, and the stability and continuity are good;

[0014] 4) The generator combines the deep migration module to strengthen the extraction of image detail information, has strong fault tolerance, can better prevent information loss, and reduces noise and artifacts in the fused image. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 The principle diagram of the infrared image and visible light image fusion method based on the improved GAN network of the application;

[0016] Figure 2 The schematic diagram of the histogram mapping algorithm;

[0017] Figure 3 For the base layer image and enhance the base layer image Fusion diagram;

[0018] Figure 4 To improve the structure diagram of GAN networks;

[0019] Figure 5 This is a diagram of the densely connected network structure of the deep feature transfer module;

[0020] Figure 6 This is a schematic diagram for image reconstruction by fusion. Detailed Implementation

[0021] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and preferred embodiments.

[0022] This invention provides a method for fusing infrared and visible light images based on an improved GAN network, such as... Figure 1 As shown, the method includes the following steps:

[0023] (1) Image decomposition based on guided filtering

[0024] This invention relates to a novel method for fusing visible light and infrared images, the principle of which is as follows: Figure 1 As shown. First, the basic layer images of the infrared image I1 are solved using a guided filter. and the basic layer image of visible light image I2 The corresponding detail layer images were calculated respectively. and detail layer images Compared to traditional methods, the guided filter multi-scale decomposition analysis approach can separate overlapping features in space. Assume I... k (k=1,2) are the input images, then I1 and I2 are the infrared image and the visible light image, respectively. For each input image I k The corresponding basic layer image can be obtained by solving equation (1). Formula (1) is as follows:

[0025]

[0026] Among them, g x = [-1, 1] denotes the horizontal gradient operator, g y = [-1, 1] T Let λ represent the vertical gradient operator, and let λ represent the scaling factor. In the above equation, the parameter λ is set to 5.

[0027] Detail layer image It is obtained by subtracting the base layer image portion from the source image, and the calculation formula is as follows:

[0028]

[0029] Wherein the base layer image contains large-scale contour information, and the detail layer image contains small-scale texture information and edge information.

[0030] (2) Base layer image fusion and detail layer image fusion

[0031] In the base layer image fusion, first, the base layer image of the visible light image is divided into m x n blocks, and the local entropy of each block is calculated. The local entropy of each block is calculated. Then, the improved LT algorithm-based fusion of the enhanced base layer image and the base layer image is performed. And The histogram mapping algorithm is to divide the image S into m x n sub-intervals using local minimum value, S = [s1, s2, …, s m×n ], and the base image is enhanced and reconstructed by sub-interval histogram, T = [t1, t2, …, t m×n ], as shown in Figure 2 The local entropy is used to control the mapping range of the sub-histogram, which avoids the generation of too many histogram peaks in the traditional method, wherein the local entropy is used to measure the texture richness of the image block, and if the local entropy is smaller, it means that the texture is less, and the remapping range is larger. The mapping range controlled by the local entropy is:

[0032]

[0033] Wherein, γ j ∈[0, 0.8] is used for normalization, is the cumulative distribution function of the sub-histogram interval [m j , m j+1 ], wherein p() represents the probability density, and u represents the sub-histogram interval.

[0034] When the histogram interval has less texture, R j will be allocated to more mapping intervals. Therefore, the enhanced base layer image obtained after the final enhancement processing is as follows:

[0035]

[0036] Wherein, R t is the local entropy of each block j.

[0037] The improved LT algorithm (i.e. the fusion rule of Laplacian pyramid transformation) is used to fuse the enhanced base layer image and the base layer image The fusion process includes the following steps:

[0038] As shown in Figure 3 , the enhanced base layer image and the base layer image are respectively decomposed into and , where l∈(1, 2, 3, 4). Wherein the m×n matrix of the size of the region centered on each pixel in the image calculates the regional average gradient, and the top layer image and are fused, and the formula is as follows:

[0039]

[0040] Wherein, ΔI x and ΔI y are the first-order difference of the pixel point f(x, y) in the x-axis and y-axis, respectively, and G1(i, j) and G2(i, j) represent the regional average gradient of each pixel point of the top layer, which can reflect the definition of the image, and the top layer image fusion can be expressed as:

[0041]

[0042] The bottom three layers LT fusion need to calculate the energy of each layer of the infrared and visible light image The calculation formula is as follows:

[0043]

[0044] Wherein, p, q are both 1, and w is a 3×3 matrix, then when 0<l<4, the fusion result of the LT image pyramid of the l layer is as follows:

[0045]

[0046] Based on the improved LT, the base layer infrared image and the visible light image fusion algorithm is used to enhance the low light visible light image base layer through histogram mapping, and the base layer of the infrared image is fused through the improved LT algorithm, so that the fused image has good contrast and overall appearance.

[0047] In the process of detail layer image fusion, the detail layer image of the infrared image and the detail layer image of the visible light image are used as the training set to train the improved GAN network, and the image fusion process is optimized and guided through the way of adversarial learning, and the detail layer fusion image F d is obtained. Wherein the improved GAN network includes a generator, an encoder and two discriminators, as shown in Figure 4As shown, the encoder consists of five convolutional layers and one batch normalization (BN) layer, and the generator consists of three densely connected modules and three deconvolutional modules. z The discriminator and target discriminator each consist of a convolutional layer, a batch normalization (BN) layer, a Leaky ReLU layer, and a fully connected layer, respectively. The improved GAN network in this invention differs from traditional generative adversarial networks (GANs) in two ways: first, it adds a discriminator and an encoder to provide a priori optimization model to guide the generation of the fused image; second, it incorporates a deep feature transfer module into the generator network, enabling the network to better extract the depth features of the source image and achieve the best fusion effect. Based on the improved GAN's detail layer infrared and visible light image fusion algorithm, compared to traditional GANs, it adds an encoder and a discriminator, constrains network parameters through low-rank prior decomposition, guides the generated fused image to be closer to the target image, and designs a loss function to ensure the stability of the network model.

[0048] The encoder aims to reduce the dimensionality of the source image while providing a priori initialization constraint model. This involves transforming the detail layer image... The input x is fed into the newly added encoder E. Encoder E maps the input x to a low-dimensional feature vector z through a convolution operation, concatenates it with a label vector o, and constrains the network parameters through low-rank prior decomposition. Where:

[0049] E(x) = z∈R n (9)

[0050] Among them, R n Representing an n-dimensional space, the output retains the texture features of the input x.

[0051] The generator aims to extract more detailed information from the source image and generate a fused image with rich texture. A deep feature transfer module is introduced to achieve feature transfer and dimensionality upscaling of the low-dimensional feature vector z, such as... Figure 3 As shown. The deep feature transfer module uses dense connections to extract features of real semantic information. Assuming the network has L layers, x0 is the network input, x... l x is the output of the l-th layer in the network. l-1 H is the output of the (l-1)th layer. l () represents the nonlinear transformation acting on the l-th layer. The densely connected network structure of the deep feature transfer module is as follows: Figure 5As shown, Input in the figure represents the input of the image, BN-ReLU-Conv represents the regularized layer, the activation layer and the convolution layer connected in turn, and the relationship is shown in formula (10). The deep feature migration module integrates the features of each stage in a dense connection manner, and adds the output features of the previous layer to each input end, so as to expand the dimension. In the generator, 2 deconvolutions are used to map the low-dimensional image into a high-dimensional image, and the label vector l is connected to the low-dimensional feature vector z. The new vector [z, l] is fed back to the generator, and the output detail layer fusion image F is output d For formula (11):

[0052] x l =H l ([x0,x1,...,x l-1 ]) (10)

[0053] F d =G(z,l)=G(E(x),l) (11)

[0054] In combination with the generator network design of the deep feature migration module, the features of each stage are integrated in a skip connection manner, the output features of the previous layer are added to each input end, and the low-dimensional image is mapped into a high-dimensional image by combining deconvolution, so that the fusion image has more rich detail information, and the artifacts and noise are effectively reduced.

[0055] The purpose of the discriminator is to distinguish between the generated image and the target image, and to migrate information to the fusion process of the generator. The detail layer image with richer texture features is taken as the target image, and the discriminators D are constructed between the detail layer image and the detail layer image z , and the target discriminator is constructed between the detail layer image and the detail layer fusion image F d . Among them, the discriminator D z is used to force the distribution of the generated low-dimensional feature vector z to gradually approach the prior. The target discriminator is used to perform adversarial learning between the detail layer fusion image F d and the detail layer image , so that the detail layer fusion image F d is more real. Still referring to Figure 4 , the discriminators D z and the target discriminator both contain three convolution modules, a normalization layer, an activation function layer and a fully connected layer, wherein the normalization layer uses the BN layer to normalize the data, the activation function layer uses Leaky Relu as the activation function, and the fully connected layer is used for prediction.

[0056] The training loss function of the encoder E and the generator G in the application is as follows:

[0057]

[0058] where L() denotes the norm, x is the input, i.e., the detail layer image G() and E() represent the generator output and the encoder output, respectively, and l is the label vector.

[0059] D z Loss function of the discriminator and the loss function L of the target discriminator D D are as follows, respectively:

[0060]

[0061]

[0062] where b and c represent the real labels of the base layer image and the detail layer image , respectively, and their values range from 0.4 to 0.6 and from 0.7 to 1, respectively; d represents the real label of the fused detail layer, and its value ranges from 0 to 0.3; N represents the number of fused images; and represent the classification results of and F d , respectively.

[0063] (3) Fused image reconstruction

[0064] Through the above steps (1) and (2), the base layer fused image F b and the detail layer fused image F d can be obtained, and the final fused image is obtained using the base layer fused image F b and the detail layer fused image F d . The corresponding pixel points in the base layer fused image F b and the detail layer fused image F d are weighted and added, as shown in equation (14), to obtain the final fused image, as shown in Figure 6

[0065] F(x,y) = aF b (x,y) + bF d (x,y) (14)

[0066] where (x, y) represents the pixel point coordinates of the fused image, a is the base layer fusion parameter, and b is the detail layer fusion parameter. In this patent, a = 0.6 and b = 0.4.

[0067] ​Any combination of the technical features in the above-described embodiments can be made, and for the sake of brevity, not all possible combinations are described, however, as long as the combination of the technical features does not exist in contradiction, it shall be considered within the scope of the present disclosure.

[0068] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it shall not be understood as a limitation on the patent scope of the present application. It shall be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these shall be within the protection scope of the present application. Therefore, the protection scope of the present application patent shall be subject to the appended claims.

Claims

1. An infrared image and visible light image fusion method based on an improved GAN network, characterized in that, The method comprises the following steps: (1) The basic layer images of infrared image I1 are obtained by using the guided filter. and the basic layer image of visible light image I2 The corresponding detail layer images were calculated respectively. and detail layer images (2) using a histogram mapping algorithm to enhance the base layer image to obtain an enhanced base layer image Then the enhanced base layer image and the base layer image are fused by improving the LT algorithm to obtain a base layer fused image F b , and the detail layer image and the detail layer image are used as a training set to train the improved GAN network to obtain a detail layer fused image F d , wherein the improved GAN network includes a generator, an encoder, a D z discriminator and a target discriminator, the encoder maps the input detail layer image to a low-dimensional feature vector z through a convolution operation, the low-dimensional feature vector z and a label vector l are input into the generator, the low-dimensional image is mapped into a high-dimensional image in the generator using two deconvolution methods, the label vector l is connected to the low-dimensional feature vector z, the new vector [z, l] is fed back to the generator and outputs the detail layer fused image F d , the detail layer image is used as a target image, a D z discriminator is constructed between the detail layer image and the detail layer image , the D z discriminator is used to force the distribution of the generated low-dimensional feature vector z to gradually approach the prior, a target discriminator is constructed between the detail layer image and the detail layer fused image F d , the target discriminator is used to perform adversarial learning between the detail layer fused image F d and the detail layer image , and the D z discriminator and the target discriminator each include three convolution modules, a BatchNorm normalization layer, a Leaky Relu activation function layer and a fully connected layer; the improved GAN network introduces a deep feature migration module, the deep feature migration module integrates the features of each stage in a skip connection manner, adds the output features of the previous layer to each input end, and combines deconvolution to map the low-dimensional image into a high-dimensional image. Utilizing a histogram mapping algorithm on the base layer image When enhancing the base layer image, the local entropy is used to control the mapping range of the sub-histogram, and the mapping range controlled by the local entropy is: where γ j ∈ [0, 0.8] is used for normalization, is the cumulative distribution function of the sub-histogram interval [m j , m j+1 ], where p() denotes the probability density and u denotes the sub-histogram interval. The enhanced base layer image obtained after the enhancement process The calculation formula is as follows: wherein R t is the local entropy for each tile j; By improving the LT algorithm to enhance the base layer image and the base layer image , the L-layer pyramid of the enhanced base layer image and the base layer image is respectively decomposed into and where l∈(1, 2, 3, 4); the region average gradient is calculated by an m×n matrix of the region size centered on each pixel in the image, and the top layer image and are fused, and the formula is as follows: where ΔI x and ΔI y are the first-order difference of pixel f(x, y) in x-axis and y-axis, respectively, and G1(i, j) and G2(i, j) represent the area average gradient of each pixel of the top layer, respectively, and the top layer image fusion can be represented as: The bottom three layers LT fusion needs to calculate the area energy of each layer of infrared and visible light images The calculation formula is as follows: Wherein, p, q are both 1, w is a 3*3 matrix, and when 0 The training loss function of the encoder (E) and the generator (G) is as follows: where L() denotes a norm, x is the input, i.e., the detail layer image G() and E() represent the generator output and the encoder output, respectively, and l is the label vector. D z Loss function of the discriminator and the loss function L of the target discriminator D D are as follows, respectively: where b and c represent the real labels of the detail layer images and the detail layer images , whose values range from 0.4 to 0.6 and from 0.7 to 1, respectively; d represents the real label of the fused detail layer, whose value ranges from 0 to 0.3; N represents the number of fused images; and represent the classification results of the detail layer images the detail layer images and the detail layer fused image F d , respectively. (3) the base layer fusion image F b and the detail layer fusion image F d corresponding pixel points are weighted and added to obtain the final fusion image.

Images