A multi-scale generative adversarial fusion network approach for infrared and visible light images

Abstract

The present invention discloses a method for a multi-scale generative adversarial fusion network for infrared and visible light images. A plurality of infrared and visible light image pairs are selected from a standard training set, and then the image pairs are input into an edge-preserving filter to obtain a base layer and a detail layer; the base layer is input into a gradient filter to obtain a gradient map and a new base layer, and the gradient map and the original detail layer are added as a new detail layer. The discriminator network parameters are obtained by calculation, and then training is performed, and the output obtained at last is the final fused image. The image obtained by fusion retains the target information and texture information of the source image to the maximum extent, improves the quality of the fused image, and provides a more convenient prerequisite for subsequent target detection and recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of image decomposition and image fusion in digital image processing, and in particular relates to a method for a multi-scale generative adversarial fusion network for infrared and visible light images. Background Art

[0002] Image fusion is a branch of information fusion and belongs to cross-field research, involving sensor imaging, image preprocessing, computer vision and artificial intelligence. With the rapid development of multi-type imaging sensors, the problem of limited image target information provided by a single sensor has been effectively solved. For the same scene, by fusing two or more source images from the same or different imaging sensors, a fused image with rich information and high clarity can be obtained. Visible light sensors use the reflected light from objects to form images, and the images obtained have the characteristics of high resolution and rich details. However, under poor lighting conditions, the images obtained are not very clear. When infrared sensors are used for imaging, they use the thermal radiation information of the target to form images, and have strong penetration. At the same time, they can solve the problem that the imaging effect of visible light sensors is poor when there is insufficient light or objects are blocked. It can still detect the target under poor lighting conditions, but the image details and contrast information are insufficient. Infrared and visible light image fusion technology can complement the advantages of the two types of images, ensuring that the final fused image is full of thermal radiation information, contrast information and detail information, so as to better understand the image target information and ultimately realize the system's all-weather operation. In recent years, important progress has been made in the technology of multi-scale image fusion. Generally, the infrared and visible light image fusion scheme based on multi-scale transformation includes three steps. First, each source image is decomposed into a series of multi-scale representations, and then the multi-scale representations of the source images are fused according to the given fusion rules, and finally the fused images are subjected to the corresponding multi-scale inverse transformation. At the same time, with the rapid development of deep learning, unsupervised deep learning has been expanded in the field of fusion and achieved certain results. Although this type of method is more suitable for multi-source image fusion without reference images, it puts forward higher requirements for the design of network structure and loss function. Therefore, the fusion method based on unsupervised generative adversarial network has gradually attracted the attention of researchers. Summary of the invention

[0003] The purpose of the present invention is to provide a method for a multi-scale generative adversarial fusion network for infrared and visible light images, by performing multi-scale decomposition on the source image and then inputting the corresponding components after decomposition into the generative adversarial network to obtain the final fused image. The fused image retains the target information and texture information of the source image to the maximum extent, improves the quality of the fused image, and provides a more convenient prerequisite for subsequent target detection and recognition.

[0004] The technical solution adopted by the present invention is a method for a multi-scale generative adversarial fusion network for infrared and visible light images, which is specifically implemented according to the following steps:

[0005] Step 1: Select several infrared and visible light image pairs from the standard training set, and then input the image pairs into the edge-preserving filter to obtain the base layer and the detail layer;

[0006] Step 2: Input the base layer obtained in step 1 into the gradient filter to obtain a gradient map and a new base layer. The gradient map and the detail layer in step 1 are added together to form a new detail layer.

[0007] Step 3: Input the base layer and detail layer obtained in step 2 into the generator network G. After passing through the generator network G, the fused image of the corresponding source image pair is obtained, and the generator loss function L is calculated. G , update the parameters of the generator network G to obtain the final generator network parameters, input the source image and the fused image into the discriminator network D for classification, and calculate the discriminator loss function L D , update the discriminator network parameters to obtain the final discriminator network parameters;

[0008] Step 4: Start training the network and determine whether the iteration is finished, that is, whether the current number of iterations has reached the set number of iterations. The network parameters obtained when the number of iterations reaches the set number of iterations are used as the final network parameters and the network parameters are saved.

[0009] Step 5: Load the generator network parameters obtained in step 4 into the generator network in the test network, perform multi-scale decomposition on the test infrared and visible light source images, that is, the filtering operations in steps 1 and 2, and then concatenate the corresponding base layer and detail layer obtained by the decomposition as the input of the test network. The output obtained is the final fused image.

[0010] The present invention is also characterized in that:

[0011] The filtering formula in step 1 is as follows:

[0012]

[0013] in:

[0014]

[0015] In formula (1), I q is the input image, is the filtered image, q is I q A pixel point of , s is a set of pixels of q, p is a pixel in the field of q, is a partial input image patch, yes The surrounding image blocks, is the spatial filter kernel, is the distance filter kernel. Both spatial kernel and distance kernel are usually expressed in Gaussian form.

[0016]

[0017] In formula (3), I d0 is the detail layer obtained by bilateral filtering, I b0 To obtain the base layer.

[0018] In step 2

[0019] The gradient values ​​of all pixels in the image are calculated using the following formula:

[0020]

[0021] Then define a threshold Gmax, if the pixel gradient value is greater than the threshold, set it to white, otherwise it is black, so as to obtain the gradient map I G ;

[0022] The base layer I obtained in step 1 b0 Perform gradient filtering to obtain the gradient map I G , and then the original base layer I b0 Subtracting the gradient map gives the new base layer I b , gradient map I G Same as original detail layer I d0 Add together to get the new detail layer I d .

[0023] The generator network structure in step 3 consists of a two-stream network and a convolutional neural network connected to it. The upper and lower networks in the two-stream network have the same structure, both of which are six-layer convolutional neural networks. The first four layers have the same structure, which is a 3×3 convolutional layer, a batch normalization layer, and an activation layer. The activation function of the activation layer is Leaky Relu; the last two layers have the same structure, which is composed of a 5×5 convolutional layer, a batch normalization layer, and an activation layer. The activation function of the activation layer is Leaky Relu. The network structure connected to the two-stream network consists of a 1×1 convolutional layer and an activation layer. The activation function of the activation layer is tanh. The output of this layer of convolutional neural network is the final fused image.

[0024] The generator loss function L in step 3 G for:

[0025] L G =λL content +L Gen , (6)

[0026] Among them, L contentis the content loss after comparing the generator input and output, L Gen is the adversarial loss between the generator and the discriminator, and λ is a constant;

[0027]

[0028] Where H and W are the image height and width of the generator input, respectively, and ||·|| 2 To calculate the second norm, I f is the output of the generator, that is, the fused image, I b is the base layer of the input generator, I d is the detail layer of the input generator, is the gradient operator, ξ is a constant;

[0029] L Gen =E[log(1-D V (G(I b ,I d )))]+E[log(1-D I (G(I b ,I d )))](8)

[0030] D V (G(I b ,I d )) represents the discriminant value of the discriminator with infrared image or fusion image as input, G(I b ,I d ) represents the fused image generated by the generator, D I (G(I b ,I d )) represents the discriminant value of the discriminator with the visible light image or the fused image as input.

[0031] Calculate the generator loss function L G At the same time, SGD is used to update the network parameters to achieve the purpose of optimization and obtain the network parameters of the generator.

[0032] The two discriminator networks D in step 3 I and D V They have the same network structure, which consists of a five-layer convolutional neural network. The first four layers have the same structure, which is a 3×3 convolutional layer, a batch normalization layer, and an activation layer. The activation function of the activation layer is Leaky Relu. The last layer is a fully connected layer, which outputs the classification result of the input, thereby predicting whether the output is a fused image or a source image.

[0033] The discriminator loss function with the input of infrared image and fusion image is:

[0034]

[0035] Among them, D I (I I ) represents the discriminant value of the discriminator with infrared image as input, D I (G(I b ,I d )) represents the discriminant value of the discriminator with the fused image as input;

[0036] The discriminator loss function with visible light image and fused image as input is:

[0037]

[0038] Among them, D V (I V ) represents the discriminant value of the discriminator with visible light image as input, D V (G(I b ,I d )) represents the discriminant value of the discriminator with the fused image as input;

[0039] Set a threshold for the discriminator output. When the discriminator output value is greater than the preset threshold, continue to update the network parameters until it is less than the preset threshold. In this process, it must pass through the discriminator D I and D V After that, calculate the corresponding discriminator loss function and The optimization method for updating the network parameters is SGD to finally obtain the network parameters of the discriminator.

[0040] The beneficial effect of the present invention is that a method for a multi-scale generative adversarial fusion network for infrared and visible light images combines multi-scale decomposition and a generative adversarial network, optimizes the source image, and applies a neural network with good fusion effect to the fusion process. A method for a multi-scale generative adversarial fusion network for infrared and visible light images, firstly obtains the base layer and detail layer of the image through an edge filter retainer and a gradient filter, so that the image components obtained retain the information we need to the maximum extent, and then uses the two branch networks of the generator network in the generative adversarial network to fuse the base layer (structural information) and detail layer (detail information) after multi-scale decomposition respectively, and finally adds the generated base layer image and detail layer image to obtain the final fused image, and at the same time, the two discriminator structures in the generative adversarial network classify and discriminate the two source images and the fused image. The image obtained by the fusion of the present invention retains the target information and texture information of the source image to the maximum extent, improves the quality of the fused image, and thus provides more convenient conditions for subsequent target detection and recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1It is an overall flow chart of the method of the multi-scale generative adversarial fusion network for infrared and visible light images of the present invention;

[0042] Figure 2 It is the base layer and detail layer image after bilateral filtering of the source image of the present invention;

[0043] Figure 3 The base layer after bilateral filtering of the present invention is subjected to gradient filtering to obtain a new base layer and detail layer map;

[0044] Figure 4 It is a network structure diagram of the generator in the generative adversarial network of the present invention;

[0045] Figure 5 It is a network structure diagram of the discriminator in the generative adversarial network of the present invention. DETAILED DESCRIPTION

[0046] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] The present invention is a method for a multi-scale generative adversarial fusion network for infrared and visible light images. First, the source image is decomposed by an edge filter holder and a gradient filter to obtain a base layer and a detail layer of the image. The base layer and the detail layer are then input into the generator network of the generative adversarial network for fusion. The fused image and the two source images are then input into the discriminator for discrimination to optimize the network parameters and obtain the final fused image, thereby achieving image fusion. The overall network structure of the algorithm is shown in FIG. Figure 1 As shown in the figure, the infrared and visible light image fusion process based on the multi-scale decomposition generative adversarial network is mainly divided into the following three stages:

[0048] 1) Multi-scale decomposition of source image

[0049] The multi-scale decomposition of the source image is mainly divided into three steps. First, the image is input into the edge-preserving filter (bilateral filtering) to obtain the base layer and detail layer. The base layer and detail layer are as follows: Figure 2 As shown, the base layer is then filtered through a gradient filter to obtain a gradient map and a new base layer. Finally, the gradient map and the detail layer are added together as a new detail layer to obtain a new base layer and a new detail layer as shown in Figure 3 As shown. The principles of bilateral filtering and gradient filtering are as follows:

[0050] Bilateral filtering is an edge-preserving filter that can achieve the effect of preserving edges, reducing noise and smoothing. Like other filtering principles, bilateral filtering also uses a weighted average method, using the weighted average of the brightness values ​​of surrounding pixels to represent the intensity of a pixel. The weighted average used is based on Gaussian distribution. Most importantly, the weight of bilateral filtering not only considers the Euclidean distance of pixels (such as ordinary Gaussian low-pass filtering, which only considers the influence of position on the center pixel), but also considers the radiation difference in the pixel range domain (such as the similarity between the pixel in the convolution kernel and the center pixel, color intensity, depth distance, etc.), and considers these two weights at the same time when calculating the center pixel. The filtering formula is as follows:

[0051]

[0052]

[0053] In formula (1), I q is the input image, The filtered image

[0054] Step 2: Input the base layer obtained in step 1 into the gradient filter to obtain a gradient map and a new base layer. The gradient map and the detail layer in step 1 are added together to form a new detail layer. The principle of gradient filtering is as follows:

[0055] Gradient is simply to find the derivative. There are three different filters: Sobel, Scharr and Laplacian. Sobel and Scharr are actually to find the first or second order derivative. Scharr is an optimization of Sobel. Laplacian is to find the second order derivative. The Sobel filter is used here to allow high frequencies to pass through and block low frequencies, making the edges more obvious to achieve the purpose of enhancing the image. The specific principles are as follows:

[0056] The Sobel operator is a discrete difference operator that is used to calculate the approximate grayscale value of the image brightness function. Using this operator at any point in the image will generate the corresponding grayscale vector or its normal vector.

[0057]

[0058] The operator contains two sets of 3x3 matrices, one for horizontal and one for vertical. By performing a plane convolution with the image, the approximate values ​​of the horizontal and vertical brightness differences can be obtained. If A represents the original image, Gx and Gy represent the grayscale values ​​of the image after horizontal and vertical edge detection, respectively, the formula is as follows:

[0059] G X =G x *A and G Y =G y *A (4)

[0060] 2) Generate adversarial network parameter acquisition

[0061] Obtaining the parameters of the discriminator network: The base layer and detail layer obtained in 1) are concatenated in pairs in the dimension of the image channel as the input of the generator. The network structure diagram of the generator is as follows: Figure 4 As shown, it consists of a two-stream network followed by a convolution block. The upper and lower networks of the two-stream network are the same, consisting of a six-layer convolutional neural network. The first four layers have the same structure, and the network structure is a 3×3 convolution layer, a batch normalization layer, and an activation layer (the activation function is Leaky Relu); the last two layers have the same structure, consisting of a 5×5 convolution layer, a batch normalization layer, and an activation layer (the activation function is Leaky Relu). The network structure followed by the two-stream network consists of a 1×1 convolution layer and an activation layer (the activation function is tanh). The output of this layer is the final fused image. Then the respective fusion results (the fusion images of the base layer and the detail layer) are spliced ​​to obtain the final fused image. In this process, after passing through the generator G, the generator loss function L is calculated. G At the same time, SGD (stochastic gradient descent) is used to update the network parameters to achieve the purpose of optimization and obtain the network parameters of the generator.

[0062] Discriminator network parameter acquisition: Because the source images are a pair, two discriminators are used, one to obtain the probability P that the fused image is an infrared image I , and the other obtains the probability P that the fused image is a visible light image V The two discriminator network structures are exactly the same, such as Figure 5 As shown in the figure, it consists of five layers of convolutional neural network. The first four layers have the same structure, which is a 3×3 convolution layer, a batch normalization layer, and an activation layer (the activation function is Leaky Relu); the last layer is a fully connected layer, and the output is the classification result of the input. The input source image and the fused image get two probabilities. When the probability is greater than the preset threshold, the network parameters are updated until the probability is less than the preset threshold. In this process, the discriminator D I and D V After that, calculate the corresponding discriminator loss function and The optimization method for updating the network parameters is SGD (stochastic gradient descent), and finally the network parameters of the discriminator are obtained.

[0063] The loss function includes the generator loss function L G And the loss function of the two discriminators and The design is as follows:

[0064] The purpose of the generator loss function is to preserve more information of the source image. It consists of two parts: content loss and adversarial loss:

[0065] L G =λL content +L Gen , (5)

[0066] Among them, L content is the content loss after comparing the generator input and output, L Gen is the adversarial loss between the generator and the discriminator, and λ is a constant;

[0067]

[0068] where H and W are the image height and width of the generator input, respectively, ||·|| 2 To calculate the second norm, I f is the output of the generator, that is, the fused image, I b is the base layer of the input generator, I d is the detail layer of the input generator, is the gradient operator, ξ is a constant;

[0069] L Gen =E[log(1-D V (G(I b ,I d )))]+E[log(1-D I (G(I b ,I d )))] (7)

[0070] D V (G(I b ,I d )) represents the discriminant value of the discriminator with infrared image or fusion image as input, G(I b ,I d ) represents the fused image generated by the generator, D I (G(I b ,I d )) represents the discriminant value of the discriminator with the visible light image or the fused image as input.

[0071] Using two discriminators is effective in reducing the information loss of the fusion result, and its function is also to allow the generator to save more source image information; the definition is as follows:

[0072]

[0073]

[0074] Among them, D V (IV ) represents the discriminant value of the discriminator with visible light image as input, D V (G(I b ,I d )) represents the discriminant value of the discriminator with the fused image as input, D I (I I ) represents the discriminant value of the discriminator with infrared image as input, D I (G(I b ,I d )) represents the discriminant value of the discriminator with the fused image as input.

[0075] 3) Fusion test network

[0076] The generator network parameters obtained in 2) are input into the generator network. The test image is first decomposed into multiple scales, and then the corresponding base layer and detail layer obtained by the decomposition are spliced ​​together and input into the generator network. In this way, the output of the generator is the final fused image.

[0077] The method of the multi-scale generative adversarial fusion network for infrared and visible light images of the present invention is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0078] Combination Figure 2 ,Step 1, select several infrared and visible light image pairs from the standard training set, and then input the image pairs into the edge-preserving filter (bilateral filtering) to obtain the base layer and the detail layer;

[0079] The filtering formula in step 1 is as follows:

[0080]

[0081] in:

[0082]

[0083] In formula (1), I q is the input image, is the filtered image, q is I q A pixel point of , s is a set of pixels of q, p is a pixel in the field of q, is a partial input image patch, yes The surrounding image blocks, is the spatial filter kernel, is the distance filter kernel. Both spatial kernel and distance kernel are usually expressed in Gaussian form.

[0084]

[0085] In formula (3), Id0 is the detail layer obtained by bilateral filtering, I b0 To obtain the base layer.

[0086] Combination Figure 3 , step 2, input the base layer obtained in step 1 to the gradient filter to obtain a gradient map and a new base layer, and add the gradient map and the detail layer of step 1 as a new detail layer;

[0087] In step 2

[0088] The principle of gradient filtering is as follows:

[0089] Gradient filtering is to simply derive the image, including three different filters: Sobel, Scharr and Laplacian; Sobel and Scharr are for first-order derivatives, Scharr is an optimization of Sobel, and Laplacian is for second-order derivatives. The Sobel filter is used here to allow high-frequency information to pass through, block low-frequency information, make the edge more obvious, and achieve the purpose of enhancing the image.

[0090] The specific principles are as follows:

[0091] The Sobel operator is a discrete difference operator that is used to calculate the approximate grayscale value of the image brightness function. Using this operator at any point in the image will generate the corresponding grayscale vector or its normal vector.

[0092]

[0093] The operator contains two sets of 3x3 matrices, one for horizontal and one for vertical. By performing a plane convolution with the image, the approximate values ​​of the horizontal and vertical brightness differences can be obtained. If A represents the original image, Gx and Gy represent the grayscale values ​​of the image after horizontal and vertical edge detection, respectively, the formula is as follows:

[0094] G X =G x *A and G y =G y *A (4)

[0095] The gradient values ​​of all pixels in the image are calculated using the following formula:

[0096]

[0097] Then define a threshold Gmax (here defined as 150), if the pixel gradient value is greater than the threshold value, it is set to white, otherwise it is black, so as to obtain the gradient map I G ;

[0098] The base layer I obtained in step 1b0 Perform gradient filtering to obtain the gradient map I G , and then the original base layer I b0 Subtracting the gradient map gives the new base layer I b , gradient map I G Same as original detail layer I d0 Add together to get the new detail layer I d .

[0099] Combination Figure 4 , Figure 5 , Step 3, input the base layer and detail layer obtained in step 2 into the generator network G, obtain the fused image of the corresponding source image pair after passing through the generator network G, and calculate the generator loss function L G , update the parameters of the generator network G to obtain the final generator network parameters, input the source image and the fused image into the discriminator network D for classification, and calculate the discriminator loss function L D , update the discriminator network parameters to obtain the final discriminator network parameters;

[0100] The generator network structure in step 3 consists of a two-stream network and a convolutional neural network connected to it. The upper and lower networks in the two-stream network have the same structure, both of which are six-layer convolutional neural networks. The first four layers have the same structure, which is a 3×3 convolutional layer, a batch normalization layer, and an activation layer. The activation function of the activation layer is Leaky Relu; the last two layers have the same structure, which is composed of a 5×5 convolutional layer, a batch normalization layer, and an activation layer. The activation function of the activation layer is Leaky Relu. The network structure connected to the two-stream network consists of a 1×1 convolutional layer and an activation layer. The activation function of the activation layer is tanh. The output of this layer of convolutional neural network is the final fused image.

[0101] The generator loss function L in step 3 G for:

[0102] L G =λL content +L Gen , (6)

[0103] Among them, L content is the content loss after comparing the generator input and output, L Gen is the adversarial loss between the generator and the discriminator, and λ is a constant;

[0104]

[0105] Where H and W are the image height and width of the generator input, respectively, and ||·|| 2 To calculate the second norm, I f is the output of the generator, that is, the fused image, Ib is the base layer of the input generator, I d is the detail layer of the input generator, is the gradient operator, ξ is a constant;

[0106] L Gen = E[log(1-D V (G(I b ,I d )))]+E[log(1-D I (G(I b ,I d )))] (8)

[0107] D V (G(I b ,I d )) represents the discriminant value of the discriminator with infrared image or fusion image as input, G(I b ,I d ) represents the fused image generated by the generator, D I (G(I b ,I d )) represents the discriminant value of the discriminator with the visible light image or the fused image as input.

[0108] Calculate the generator loss function L G At the same time, SGD (stochastic gradient descent) is used to update the network parameters to achieve the purpose of optimization and obtain the network parameters of the generator.

[0109] The two discriminator networks D in step 3 I and D V They have the same network structure, both of which are composed of five layers of convolutional neural networks. The first four layers have the same structure, which are 3×3 convolutional layers, batch normalization layers, and activation layers. The activation function of the activation layer is Leaky Relu; the last layer is a fully connected layer, which outputs the classification result of the input, thereby predicting whether the output is a fused image or a source image (there are two possible source images here, infrared and visible light images. Which one is specifically referred to depends on whether the discriminator is D I Or D V , D I Refers to infrared image, D V refers to visible light images);

[0110] The discriminator loss function with the input of infrared image and fusion image is:

[0111]

[0112] Among them, D I (I I ) represents the discriminant value of the discriminator with infrared image as input, DI (G(I b ,I d )) represents the discriminant value of the discriminator with the fused image as input;

[0113] The discriminator loss function with visible light image and fused image as input is:

[0114]

[0115] Among them, D V (I V ) represents the discriminant value of the discriminator with visible light image as input, D V (G(I b ,I d )) represents the discriminant value of the discriminator with the fused image as input;

[0116] Set a threshold for the discriminator output. When the discriminator output value is greater than the preset threshold, continue to update the network parameters until it is less than the preset threshold. In this process, it must pass through the discriminator D I and D V After that, calculate the corresponding discriminator loss function and The optimization method for updating the network parameters is SGD (stochastic gradient descent), and finally the network parameters of the discriminator are obtained.

[0117] Step 4: Start training the network and determine whether the iteration is finished, that is, whether the current number of iterations has reached the set number of iterations. The network parameters obtained when the number of iterations reaches the set number of iterations are used as the final network parameters and the network parameters are saved.

[0118] Specifically, the generator network parameters obtained in step 4 are loaded into the generator network in the test network, and the test infrared and visible light source images are decomposed at multiple scales, that is, the filtering operations in steps 1 and 2. Then, the corresponding base layer and detail layer obtained by the decomposition are spliced ​​together as the input of the test network, and the output obtained is the final fused image.

[0119] Step 5: Load the generator network parameters obtained in step 4 into the generator network in the test network, perform multi-scale decomposition on the test infrared and visible light source images, that is, the filtering operations in steps 1 and 2, and then concatenate the corresponding base layer and detail layer obtained by the decomposition as the input of the test network. The output obtained is the final fused image.

Claims

1. A method for multi-scale generative adversarial fusion network for infrared and visible light images, characterized in that: Follow the steps below to implement it: Step 1: Select several infrared and visible light image pairs from the standard training set, and then input the image pairs into the edge-preserving filter to obtain the base layer and the detail layer; The filtering formula in step 1 is as follows: in: In formula (1), I q is the input image, is the filtered image, q is I q A pixel point of , s is a set of pixels of q, p is a pixel in the field of q, is a partial input image patch, yes The surrounding image blocks, is the spatial filter kernel, is the distance filter kernel. Both spatial kernel and distance kernel are usually expressed in Gaussian form. In formula (3), I d0 is the detail layer obtained by bilateral filtering, I b0 is the obtained base layer; Step 2: Input the base layer obtained in step 1 to the gradient filter to obtain a gradient map and a new base layer. The gradient map and the detail layer in step 1 are added together to form a new detail layer. Step 3: Input the base layer and detail layer obtained in step 2 into the generator network G. After passing through the generator network G, the fused image of the corresponding source image pair is obtained, and the generator loss function L is calculated. G , update the parameters of the generator network G to obtain the final generator network parameters, input the source image and the fused image into the discriminator network D for classification, and calculate the discriminator loss function L D , update the discriminator network parameters to obtain the final discriminator network parameters; The generator network structure in step 3 is composed of a two-stream network and a convolutional neural network connected thereto. The upper and lower networks in the two-stream network have the same structure, both of which are six-layer convolutional neural networks. The first four layers have the same structure, and the network structure is a 3×3 convolutional layer, a batch normalization layer, and an activation layer. The activation function of the activation layer is Leaky Relu; the last two layers have the same structure, and are composed of a 5×5 convolutional layer, a batch normalization layer, and an activation layer. The activation function of the activation layer is Leaky Relu. The network structure connected to the two-stream network is composed of a 1×1 convolutional layer and an activation layer. The activation function of the activation layer is tanh. The output of this layer of convolutional neural network is the final fused image. The generator loss function L in step 3 G for: THE G =λL content +L Gen ,(6) Among them, L content is the content loss after comparing the generator input and output, L Gen is the adversarial loss between the generator and the discriminator, and λ is a constant; Among them, H and W are the image height and width of the generator input, ||·||2 is the calculation of the bi-norm, and I f is the output of the generator, that is, the fused image, I b is the base layer of the input generator, I d is the detail layer of the input generator, is the gradient operator, ξ is a constant; L Gen =E[log(1-D V (G(I b ,I d )))]+E[log(1-D I (G(I b ,I d )))](8) G(I b ,I d ) represents the fused image generated by the generator, D V (G(I b ,I d )) represents the fused image G(I b ,I d ) is the input discriminator D V The discriminant value, D I (G(I b ,I d )) represents the fused image G(I b ,I d ) is the input discriminator D I The discriminant value of Calculate the generator loss function L G , and use SGD to update the network parameters to achieve the purpose of optimization and obtain the network parameters of the generator; The two discriminator networks D in step 3 I and D V They have the same network structure, which consists of a five-layer convolutional neural network. The first four layers have the same structure, which is a 3×3 convolutional layer, a batch normalization layer, and an activation layer. The activation function of the activation layer is Leaky Relu. The last layer is a fully connected layer, which outputs the classification result of the input, thereby predicting whether the output is a fused image or a source image. The discriminator loss function with the input of infrared image and fusion image is: L DI =E[-logD I (I I )]+E[-log(1-D I (G(I b ,I d )))](9) Among them, D I (I I ) represents the discriminant value of the discriminator with infrared image as input, D I (G(I b ,I d )) represents the fused image G(I b ,I d ) as the input of the discriminator D I The discriminant value of The discriminator loss function with visible light image and fused image as input is: L DV =E[-logD V (I V )]+E[-log(1-D V (G(I b ,I d )))](10) Among them, D V (I V ) represents the discriminant value of the discriminator with visible light image as input, D V (G(I b ,I d )) represents the fused image G(I b ,I d ) as the input of the discriminator D V The discriminant value of Set a threshold for the discriminator output. When the discriminator output value is greater than the preset threshold, continue to update the network parameters until it is less than the preset threshold. In this process, it must pass through the discriminator D I and D V After that, calculate the corresponding discriminator loss function L DI and L Dv , the optimization method for updating the network parameters is stochastic gradient descent SGD, and finally the network parameters of the discriminator are obtained; Step 4: Start training the network and determine whether the iteration is finished, that is, whether the current number of iterations has reached the set number of iterations. The network parameters obtained when the number of iterations reaches the set number of iterations are used as the final network parameters and the network parameters are saved. Step 5: Load the generator network parameters obtained in step 4 into the generator network in the test network, perform multi-scale decomposition on the test infrared and visible light source images, that is, the filtering operations in steps 1 and 2, and then concatenate the corresponding base layer and detail layer obtained by the decomposition as the input of the test network. The output obtained is the final fused image.

2. The method of multi-scale generative adversarial fusion network for infrared and visible light images according to claim 1, characterized in that: In step 2 The gradient values ​​of all pixels in the image are calculated using the following formula: Then define a threshold Gmax, if the pixel gradient value is greater than the threshold, set it to white, otherwise it is black, so as to obtain the gradient map I G ; The base layer I obtained in step 1 b0 Perform gradient filtering to obtain the gradient map I G , and then the original base layer I b0 Subtracting the gradient map gives the new base layer I b , gradient map I G Same as original detail layer I d0 Add together to get the new detail layer I d .

Images