A visible light and infrared image fusion method based on a spatial gradient guided network

Abstract

The application provides a visible light and infrared image fusion method based on a spatial gradient guided network, which converts an image fusion problem into extraction and reconstruction of gradient and saliency information, and a new loss function is designed for a feature extraction part. The loss function is composed of a gradient term and a saliency term. For the gradient term, a spatial gradient module is introduced, and the optimization target of the gradient distribution is determined according to the richness of the pixel scale texture information of the visible light image. For the saliency term, the intensity information proportion of the salient target is changed by adjusting the weight of the loss term of the infrared light image. For the feature reconstruction part, a light loss function is designed, and a spatial guidance module is added in the fusion process to reduce the loss of information and retain the image light intensity. The visible light and infrared image fusion method realizes the function of accurate scene brightness distribution of the fused image and improves the salient information of the heat source target.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image processing, information fusion, deep learning and machine vision, and relates to a visible light and infrared image fusion method based on a spatial gradient guided network, which is suitable for the fusion of visible light images and infrared images in the fields of military, remote sensing and target detection systems. BACKGROUND

[0002] Linfeng Tang published a paper STDFusionNet: An infrared and visible image fusion network based on salient target detection in IEEE Transactions on Instrumentation and Measurement in August 2021, which greatly improves the training length of image data, but does not accurately estimate the brightness distribution and scene understanding.

[0003] In addition, Yang Y published a paper Infrared and Visible Image Fusion Using Visual Saliency Sparse Representation and Detail Injection Model in IEEE Transactions on Instrumentation and Measurement in July 2021, which well suppresses image noise, but excessively relies on manual design of fusion rules in the feature extraction process, and the extracted features lack diversity, have low contrast and blurred texture, and are prone to artifacts.

[0004] Therefore, a visible light and infrared image fusion method based on a spatial gradient guided network is proposed to solve the problems of excessive reliance on manual design of fusion rules in the feature extraction process, lack of diversity of extracted features, low contrast, blurred texture and easy occurrence of artifacts, and better understanding of the brightness distribution of the fused image scene. SUMMARY

[0005] A visible light and infrared image fusion method based on a spatial gradient guided network converts the image fusion problem into the extraction and reconstruction of gradient and saliency information.

[0006] For the feature extraction part, a loss function is designed, which is composed of a gradient term and a saliency term; for the gradient term, a spatial gradient module is introduced, and the optimization target of the gradient distribution is determined according to the richness of the pixel scale texture information of the visible light image.

[0007] For the saliency item, the intensity information proportion of the salient target is changed by adjusting the weight of the loss term of the infrared light image;

[0008] For the feature reconstruction part, a light loss function is designed, and a spatial guidance module is added in the fusion process to reduce information loss and preserve image light intensity. The spatial guidance module first uses a Gaussian low-pass filter to reduce the influence of noise, selects a smaller weight in the edge area, and guides the gradient distribution of the fused image to be close to the original pixel with greater gradient saliency, ensuring the consistency of the edge area of the visible light image and the infrared light image, while reducing the distortion problem during infrared light image fusion;

[0009] For the saliency information, since different fusion tasks require different saliency information, the weight proportion of each saliency loss term needs to be adjusted to select effective and prominent information for preservation during the fusion process;

[0010] Gradient and saliency information is extracted and reconstructed using gradient and saliency strategies;

[0011] The visible light and infrared image fusion method based on the spatial gradient guidance network can solve the problem of inaccurate scene brightness distribution in the fused image, realize the function of accurate scene brightness distribution in the fused image, optimize the detail contour and texture information of the fused image, and improve the salient information of the heat source target;

[0012] A visible light and infrared image fusion method based on a spatial gradient guidance network, the steps in use are:

[0013] Step (1): Feature extraction of infrared image, first train the first layer of infrared image, the convolution kernel size of the first layer is 1x1, then use four weight convolution layers to fully extract the depth features of the infrared image, after the output of the second layer, the third layer and the fourth layer of the infrared image, there is a saliency item module behind it to reduce the loss of salient target information in feature extraction, the saliency item module can extract more detailed information of the convolution layer, except the first layer, the kernel size of all convolution layers is 3x3, and all layers of feature extraction use ReLU as the activation function;

[0014] Saliency item loss function is:

[0015]

[0016] Where: x and y represent the elements of the xth row and yth column of the spatial module mapping or gradient mapping, H is the height of the infrared image, W is the width of the infrared image, is the background loss function, is the salient target loss function, Loss function of fusion image, alpha is weight ratio, n is pixel value of image;

[0017] Step (2): Visible light image feature extraction, first train the first layer of visible light image, the convolution kernel size of the first layer is 1*1, then use four weight convolution layers to fully extract the depth features of the visible light image, after the output of the second layer, third layer and fourth layer of the visible light image, there is a gradient item module to reduce the loss of detail texture information in feature extraction, the gradient item module can extract the texture information of convolution layer, the kernel size of all convolution layers is 3*3 except the first layer, and all layers of feature extraction use ReLU as activation function;

[0018] Gradient item loss function Is:

[0019]

[0020] Wherein: I 1(x,y) represents the original image, I f(x,y) represents the fusion image, S 1(x,y) represents the gradient map generated by the spatial gradient module according to the gradient level of the original image, represents the gradient operator;

[0021] The gradient operator uses scharr gradient operator, in order to avoid the influence of a large amount of noise on gradient judgment, the spatial gradient module first carries out bilateral filtering on the original image, then uses Scharr operator to find gradient mapping, and generates spatial module mapping on pixel scale according to the size of gradient, the generation process of spatial gradient module is:

[0022]

[0023]

[0024]

[0025]

[0026] In the formula, w(x,y,k,l) is the weighting coefficient, which is determined by the domain kernel and the value domain kernel; d(x,y,k,l) is the domain kernel; r(x,y,k,l) is the value domain kernel, the size of the value domain kernel depends on the difference between the gray value of the convolved pixel and the gray value of the field pixel; k and l are the coordinates of the field pixel; ||·|| represents norm; σ d And σ rrespectively represent the domain kernel and the value kernel Gaussian variance; the greater the variance, the smaller the weight difference; exp() is the exponential function; f(x, y) is the gray value corresponding to (x, y); f(k, l) is the gray value corresponding to (k, l);

[0027] Since the original visible light image adopts a bilateral filter function, pixel points with large gradient values are selected, which can retain a large amount of texture detail information;

[0028] The spatial gradient module is used to act on the gradient loss function, so that the fused image retains rich texture in the pixel scale in an optimized manner; step (3): in the feature reconstruction part, the features extracted from the infrared image and the visible light image are spliced as the input of image fusion; image reconstruction includes five convolutional layers, and the outputs of the first layer, the second layer and the third layer are followed by a spatial guidance module, which extracts the brightness distribution information in the visible light image, improves the illumination brightness of the fused image, and reduces the loss of information in the fusion process again, fully integrates the detail texture information and the salient target information, and generates the finally fused image; except that the fifth layer has a kernel of 1x1, the sizes of the remaining layers are all 3x3;

[0029] The fifth layer of the feature reconstruction part uses a Tanh activation function, and all the other layers use ReLU as the activation function;

[0030] Step (4): the illumination brightness information of the fused image is estimated according to the pixel of the visible light image to estimate the brightness distribution of the scene, the input is the visible light image, and the output is the brightness probability, which is composed of five convolutional layers, a global pooling layer and two fully connected layers; all convolutional layers use ReLU as the activation function, the global pooling layer integrates the illumination information, and two fully connected layers are used to calculate the brightness probability;

[0031] The spatial guidance module fully extracts the brightness distribution information while reducing the training time of the fused image;

[0032] The illumination loss function of the spatial guidance module is:

[0033]

[0034] Wherein: P ir and P vi are the brightness weights of the infrared image and the visible light image respectively; and are the brightness losses of the infrared image and the visible light image respectively;

[0035] Since the brightness loss refers to the difference between the fused image and the original image at the pixel level, the brightness losses of the infrared light and the visible light image are respectively:

[0036] ​

[0037]

[0038] wherein: I ir(x,y) represents an infrared image, I vi(x,y) represents a visible light image,;

[0039] Step (5): fusing the light intensity information with the spliced infrared image and visible light image to obtain a final fused image; the loss function of the fused image is:

[0040]

[0041] wherein: is a gradient term loss function; is a saliency term loss function; beta is used to balance the saliency term and the gradient term coefficient; is a light loss function in the fusion process.

[0042] The present application has the following advantages and effects relative to the prior art:

[0043] (1) The existing feature extraction network and feature reconstruction network use the same loss function and cannot pay attention to the details of two different picture attributes. In contrast, the present application is composed of a feature extraction network and a feature reconstruction network, and different loss functions are proposed, which greatly retain the feature information of the original image that needs to be extracted.

[0044] (2) The existing gradient term loss function that may produce negative numbers may cause image blurring. However, the present application proposes a gradient term loss function that is all positive numbers for feature extraction of visible light images, enhances the retention ability of edge information and detailed texture, and reduces the problems of fusion distortion and blurring.

[0045] (3) The negative values produced by the existing saliency term loss function of image feature extraction will cause image distortion problems. A non-negative saliency term loss function is proposed for feature extraction of infrared light images, highlighting the information features of heat source targets.

[0046] (4) Most of the existing loss functions do not consider the distribution problem of light intensity, which will cause the problem of being unable to distinguish the scene, and the loss function considering the light intensity does not consider the problem of brightness overlap. The present application proposes an estimation of brightness distribution based on the extraction of details of infrared light and visible light, which can clearly distinguish the brightness and color problems of the scene. A new light intensity loss function is proposed in the fusion process, which greatly retains the brightness distribution problem of the scene in the visible light image, while reducing the brightness overlap problem of the infrared heat source target. Attached Figure Description

[0047] Figure 1 This is a diagram of the image fusion network framework of the method of the present invention.

[0048] Figure 2 This is a flowchart of the image fusion process of the present invention.

[0049] Figure 3 This is a structural diagram of the spatial gradient module of the method of the present invention.

[0050] Figure 4 This is a schematic diagram of the spatial gradient module of the method of the present invention. Detailed Implementation

[0051] The present invention proposes a visible light and infrared image fusion method based on a spatial gradient guided network, which is described in detail below with reference to the accompanying drawings:

[0052] Figure 1is the image fusion network framework diagram of the method. First, the infrared image is subjected to feature extraction, the first layer of the infrared image is first trained, the convolution kernel size of the first layer is 1*1, then four weight convolution layers are used to fully extract the depth features of the infrared image, wherein there is a significant item module behind the output of the second layer, the third layer and the fourth layer of the infrared image, so as to reduce the loss of significant target information in feature extraction, the significant item module can effectively extract more detailed information of the convolution layer, except for the first layer, the kernel size of all convolution layers is 3*3, and all layers of feature extraction are used as the activation function. Then, the visible light image is subjected to feature extraction, the first layer of the visible light image is first trained, the convolution kernel size of the first layer is 1*1, then four weight convolution layers are used to fully extract the depth features of the visible light image, wherein there is a gradient item module behind the output of the second layer, the third layer and the fourth layer of the visible light image, so as to reduce the loss of detailed texture information in feature extraction, the gradient item module can effectively extract more texture information of the convolution layer, except for the first layer, the kernel size of all convolution layers is 3*3, and all layers of feature extraction are used as the activation function. Furthermore, the visible light and infrared images after feature extraction are subjected to feature reconstruction, in the feature reconstruction part, the features extracted from the infrared image and the visible light image are spliced as the input of image fusion, the image reconstruction includes five convolution layers, wherein the output of the first layer, the second layer and the third layer is followed by a spatial guidance module, which extracts the brightness distribution information in the visible light image, improves the illumination brightness of the fused image, and reduces the information loss in the fusion process again, can fully integrate the detailed texture information and the significant target information, and generate the finally fused image. Except that the fifth layer has a kernel of 1*1, the size of the remaining layers is 3*3. The fifth layer of the feature reconstruction part uses the Tanh activation function, and all the remaining layers use the ReLU activation function. Finally, the illumination brightness information is fused with the spliced infrared image and visible light image to obtain the final fused image.

[0053] Figure 2 is the image fusion flow chart of the method. First, the visible light image and the infrared image are subjected to feature extraction respectively, and the corresponding loss function is designed. Then, the brightness distribution information of the visible light image is extracted, the corresponding illumination brightness loss function is designed, and then the features are spliced. Finally, the fusion image of the visible light image and the infrared image is obtained.

[0054] Figure 3is a spatial gradient module structure diagram of the method of the present application. First, the visible light image is subjected to local feature extraction through five convolution layers, and through convolution operation, feature extraction is carried out at different positions of the original image to generate a plurality of convolution feature maps. Then, the extracted feature maps are subjected to a global pooling layer to down-sample the convolution feature maps, reduce the dimension and calculation amount of the data, and improve the robustness of the model. Then, a fully connected layer is further passed through to map the convolution feature maps to a classification result, and finally the illumination intensity feature information of the visible light image is obtained.

[0055] Figure 4 is a principle diagram of the spatial gradient module of the method of the present application. First, the original image is subjected to bilateral filtering, then the Scharr operator is used to find the gradient mapping, and the spatial module mapping is generated on the pixel scale according to the size of the gradient.

[0056] The above description is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1.A method for visible and infrared image fusion based on a spatial gradient guided network, characterized in that, The visible light and infrared image fusion method converts the image fusion problem into extraction and reconstruction of gradient and saliency information; For the feature extraction part, a loss function is designed, which is composed of a gradient term and a saliency term; for the gradient term, a spatial gradient module is introduced, and the optimization target of the gradient distribution is determined according to the richness of the pixel scale texture information of the visible light image; For the saliency term, the intensity information proportion of the salient target is changed by adjusting the weight of the loss term of the infrared light image; For the feature reconstruction part, a lighting loss function is designed, and a spatial guidance module is added in the fusion process to reduce information loss and preserve image lighting intensity; the spatial guidance module first uses a Gaussian low-pass filter to reduce the influence of noise, selects a smaller weight in the edge area, and guides the gradient distribution of the fused image to be closer to the original pixel with greater gradient saliency, ensuring the consistency of the edge area of the visible light image and the infrared light image, while reducing the distortion problem of the infrared light image during fusion; For the saliency information, different fusion tasks require different saliency information, so the weight proportion of each saliency loss term needs to be adjusted to select effective and prominent information during the fusion process; The gradient term and the saliency term are used to extract and reconstruct the gradient and saliency information; The visible light and infrared image fusion method based on the spatial gradient guided network can solve the problem of inaccurate scene brightness distribution in the fused image, realize the function of accurate scene brightness distribution in the fused image, optimize the detail contour and texture information of the fused image, and improve the salient information of the heat source target; A visible light and infrared image fusion method based on a spatial gradient guided network, the steps in use are: Step (1): feature extraction of the infrared image, first train the first layer of the infrared image, the convolution kernel size of the first layer is 1x1, then use four weight convolution layers to fully extract the depth features of the infrared image, after the output of the second layer, the third layer and the fourth layer of the infrared image, there is a saliency term module to reduce the loss of salient target information in feature extraction, the saliency term module can extract more detailed information of the convolution layer, except for the first layer, the kernel size of all convolution layers is 3x3, and all layers of feature extraction use ReLU as the activation function; Significant term loss function is: wherein: x and y represent the element of the xth row and yth column of the spatial module mapping or gradient mapping, H is the height of the infrared image, and W is the width of the infrared image, is a background loss function, is a salient target loss function, is a loss function of the fused image, a is a weight ratio, and n is a pixel value of the image. Step (2): feature extraction of the visible light image, first train the first layer of the visible light image, the convolution kernel size of the first layer is 1x1, then use four weight convolution layers to fully extract the depth features of the visible light image, after the output of the second layer, the third layer and the fourth layer of the visible light image, there is a gradient term module to reduce the loss of detail texture information in feature extraction, the gradient term module can extract texture information of the convolution layer, except for the first layer, the kernel size of all convolution layers is 3x3, and all layers of feature extraction use ReLU as the activation function; Gradient term loss function is: wherein: I 1(x,y) denotes the original image, I f(x,y) denotes the fused image, S 1(x,y) denotes a gradient map generated by the spatial gradient module according to the gradient level of the original image, denotes a gradient operator; The gradient operator adopts the Scharr gradient operator. In order to avoid the influence of a large amount of noise on the gradient judgment, the spatial gradient module first performs bilateral filtering on the original image, then finds the gradient mapping by using the Scharr operator, and generates the spatial module mapping on the pixel scale according to the size of the gradient. The generation process of the spatial gradient module is: where w(x, y, k, l) is a weighting coefficient, which is determined by the domain kernel and the range kernel; d(x, y, k, l) is the domain kernel; r(x, y, k, l) is the range kernel, the size of which depends on the difference between the gray value of the pixel to be convolved and the gray value of the field pixel; k and l are the coordinate points of the field pixel; ||·|| represents the norm; σ d and σ r respectively represent the Gaussian variance of the domain kernel and the range kernel; the greater the variance, the smaller the weight difference; exp() is the exponential function; f(x, y) is the gray value corresponding to (x, y); f(k, l) is the gray value corresponding to (k, l); Since the original visible light image adopts the bilateral filtering function, the pixel points with larger gradient values can retain a large amount of texture detail information; The spatial gradient module is used for the gradient loss function, so that the fused image can retain rich texture on the pixel scale in an optimized manner; step (3): in the feature reconstruction part, the features extracted from the infrared image and the visible light image are spliced as the input of image fusion; the image reconstruction includes five convolutional layers, and the outputs of the first layer, the second layer and the third layer are followed by a spatial guidance module, which extracts the brightness distribution information in the visible light image, improves the illumination brightness of the fused image, and reduces the loss of information in the fusion process again, fully integrates the detail texture information and the significant target information, and generates the finally fused image; except that the fifth layer has a kernel of 1x1, the sizes of the kernels of the remaining layers are all 3x3; The fifth layer of the feature reconstruction part uses the Tanh activation function, and all the remaining layers use the ReLU as the activation function; Step (4): the illumination brightness information of the fused image is estimated according to the brightness distribution of the scene based on the pixels of the visible light image. The input is the visible light image, and the output is the brightness probability, which is composed of five convolutional layers, a global pooling layer and two fully connected layers; all the convolutional layers use the ReLU as the activation function, the global pooling layer integrates the illumination information, and the two fully connected layers calculate the brightness probability; The spatial guidance module fully extracts the brightness distribution information while reducing the training time of the fused image; Space guidance module's light loss function is: wherein: P ir and P vi are the luminance weights of the infrared image and the visible light image, respectively; and are the luminance losses of the infrared image and the visible light image, respectively; Since the brightness loss refers to the pixel-level measurement of the difference between the fused image and the original image, the brightness losses of the infrared light and the visible light image are respectively: wherein: I ir(x,y) represents an infrared image, I vi(x,y) represents a visible light image; Step (5): the light illumination information is fused with the spliced infrared image and the visible light image to obtain a final fusion image; a loss function of the fusion image is is: wherein: is the gradient term loss function; is the saliency term loss function; β is used to balance the saliency term and the gradient term coefficient; is the illumination loss function in the fusion process.

Images