Visible light and infrared image fusion enhancement method and system in low light environment

Abstract

The present invention discloses a visible light and infrared image fusion enhancement method and system in a low-light environment. The method of the present invention comprises fusing a visible light image and an infrared image in a low-light environment to obtain a fused image after fusing texture information of the visible light image and target heat information in the infrared image; enhancing the fused image to remove the influence of light intensity; sharpening the enhanced fused image to compensate for the image contour, enhance the image edge and grayscale jump part, improve the image clarity and contrast, and finally obtain a fused output image. The present invention can realize the complementary information fusion of visible light images and infrared images, improve the image information expression capability in a low-light environment, weaken the negative influence of adverse factors such as low light, and greatly improve the final target detection capability of the fused image.

Description

Technical Field

[0001] The present invention relates to a visible light and infrared image fusion technology in image processing, and in particular to a visible light and infrared image fusion enhancement method and system in a low-light environment. Background Art

[0002] Low light environment refers to the situation that the video signal amplitude output by the camera is lower than a certain percentage value (e.g. 50%) of the maximum amplitude under the specific scene brightness value and camera lens parameters, making the picture dim and lacking in layers. Generally speaking, visible light images are the most important means of observing the environment. In low light environments, the role of visible light images is severely limited. For example, dim images and lack of layers lead to target loss and inability to obtain environmental information. Visible light sensors can capture reflected light information and reflect rich texture and structural information, but are very sensitive to the environment, such as light or occlusion. Infrared images can reflect the temperature and emissivity of an object itself, highlight the intrinsic characteristics of the object, and are not affected by light. The dual light fusion of visible light and infrared images can fuse the detailed texture of visible light with the prominent target information of infrared images. However, under low light conditions, the images collected by the sensor will introduce noise of different types and degrees. The phenomenon of excessively wide dynamic range of infrared images is very obvious, making the dark area black due to insufficient exposure, resulting in poor visual effects of the fused image and loss of target detail information.

[0003] So far, many scholars have conducted relevant research on image fusion and enhancement methods. In terms of image fusion, deep learning-based methods mainly include those based on autoencoders, convolutional neural networks, and generative adversarial networks. Wang et al. proposed the use of discrete wavelet transform (DWT) + convolutional neural network (CNN) for multi-focal image fusion. DWT decomposes the image into low-frequency subbands and high-frequency subbands. For different frequency subbands, CNN generates different decision graphs for fusion. Ma et al. proposed FusionGAN based on generative adversarial network. Through adversarial learning between the generator and the discriminator, the thermal radiation information of infrared images and the visible light texture information are selected and retained. However, fused images with excellent image quality are not necessarily suitable for high-level visual tasks. Therefore, task-driven methods have emerged. In 2021, Tang et al. proposed the fusion method of SeAFusion, which uses the semantic loss back propagation of the segmentation network to guide the training of the fusion network, and uses the adaptive training strategy of combining low-level and high-level features to maintain the balance of low-level and high-level visual tasks. At present, there are few methods that directly combine the target detection task with the image fusion algorithm, and the statistical evaluation index of the fused image is single. In fact, the evaluation of image fusion also needs the support of the target task and is driven by the visual task rather than being disconnected from the visual task. In terms of image enhancement, it mainly includes histogram equalization image enhancement algorithm, wavelet transform image enhancement algorithm and image enhancement algorithm based on Retinex theory. The standard histogram equalization algorithm maps the original image into an enhanced image with an approximately uniform grayscale distribution through the cumulative distribution function, which has good real-time performance but uneven brightness; the wavelet transform image enhancement algorithm can effectively highlight the details in the image by enhancing the wavelet coefficients, but at the same time amplifies the noise; the Retinex image enhancement algorithm removes the influence of the illumination component in the original image and solves the reflection component that reflects the essential color of the object, but the enhancement effect is poor in areas with large brightness differences. Summary of the invention

[0004] The technical problem to be solved by the present invention is as follows: In view of the above-mentioned problems in the prior art, a method and system for fusion and enhancement of visible light and infrared images in a low-light environment are provided. The present invention can realize the complementary information fusion of visible light images and infrared images, improve the image information expression capability in a low-light environment, weaken the negative influence of adverse factors such as low light, and greatly enhance the final target detection capability of the fused image.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A visible light and infrared image fusion enhancement method in a low light environment, comprising:

[0007] S1, fusing the visible light image and the infrared image in a low light environment to obtain a fused image that combines texture information of the visible light image and target heat information in the infrared image;

[0008] S2, enhances the fused image to remove the influence of light intensity;

[0009] S3, sharpening the enhanced fused image to compensate for the image contour, enhance the image edge and grayscale jump part, improve the image clarity and contrast, and finally obtain the fused output image.

[0010] Optionally, the function expression for enhancing the fused image in step S2 is:

[0011] r(x,y)=logS(x,y)-log[F(x,y)*S(x,y)]

[0012] In the above formula, r(x, y) is the enhanced fusion image, S(x, y) is the received reflected light image, * is the convolution symbol, F(x, y) is the center surround function, (x, y) is the pixel coordinates, and the calculation function expression of the center surround function is:

[0013]

[0014] In the above formula, λ is the wavelength of light, and c represents the Gaussian surround scale.

[0015] Optionally, step S3 includes: for the enhanced fused image, comparing the grayscale of the central pixel with the average grayscale of other pixels in the neighborhood, when the grayscale of the central pixel is lower than the average grayscale of other pixels, adding the template convolution result of the enhanced fused image to the original pixel to further reduce the grayscale of the central pixel; when the grayscale of the central pixel is higher than the average grayscale of other pixels, adding the template convolution result of the enhanced fused image to the original pixel to further increase the grayscale of the central pixel.

[0016] Optionally, the fusion in step S1 refers to inputting the visible light image and the infrared image in the low-light environment into a pre-trained fusion network to obtain a fused image, the fusion network includes two feature extraction branches and a decoder for feature aggregation and image reconstruction, and the visible light image and the infrared image in the low-light environment are respectively input into a feature extraction branch, wherein: the feature extraction branch includes a common convolution layer and at least one level of gradient residual dense block, the common convolution layer is used to perform a convolution operation with a convolution kernel size of 3×3, a batch normalization operation, and an activation using a LeakyReLU activation function on the input image features to extract shallow features; the gradient residual dense block is used to extract deep features; the decoder includes at least one level of decoding module and an output layer, the decoding module is used to perform a convolution operation with a convolution kernel size of 3×3, a batch normalization operation, and an activation using a LeakyReLU activation function on the input image features for decoding and restoration, and the output layer is used to perform a convolution operation with a convolution kernel size of 1×1, a batch normalization operation, and an activation using a Tanh activation function on the input image features to obtain a fused image.

[0017] Optionally, the gradient residual dense block includes a dense stream branch and a residual stream branch, the dense stream branch includes two convolutional layers with a kernel size of 3×3 and a common convolutional layer with a kernel size of 1×1 using dense connections, the dense connection means that the input features of the gradient residual dense block are respectively used as the inputs of the two convolutional layers and the common convolutional layer, and the outputs of the two convolutional layers with a kernel size of 3×3 are respectively used as the inputs of the common convolutional layer, the residual stream branch includes a gradient operator and a standard convolutional layer with a kernel size of 1×1, the input features of the gradient residual dense block are calculated by the gradient operator to obtain the gradient amplitude, the gradient amplitude is eliminated by the standard convolutional layer with a kernel size of 1×1, and then the output features of the gradient residual dense block are obtained by summing with the output of the dense stream branch.

[0018] Optionally, before step S1, the fusion network is further trained, and the loss function used in training the fusion network is a joint loss, and the function expression of the joint loss is:

[0019] L joint =L con +βL sem

[0020] In the above formula, L joint is the joint loss, L con is the spatial loss of the fusion network, β is the semantic loss weight, L sem is the semantic loss of the segmentation network; the segmentation network is a semantic segmentation model, which is used to perform semantic segmentation according to the fused image output by the fusion network to obtain the semantic segmentation result of the fused image.

[0021] Optionally, the segmentation network includes a spatial attention branch, a channel attention branch and a pooling fusion unit, wherein the spatial attention branch is used to extract spatial attention features from the fused image, the channel attention branch is used to extract channel attention features from the fused image, and the pooling fusion unit is used to average pool the channel attention features, then add them to the spatial attention features at the pixel level, and finally obtain the semantic segmentation result of the fused image through the convolution normalization activation layer. The spatial attention branch is used to first pass the fused image through the convolution normalization activation layer and then through the maximum pooling and average pooling operations respectively, and then sum the features obtained by the maximum pooling and average pooling operations, and then pass the gated channel attention module GCT and the convolution normalization activation layer to obtain the features and add them to the original fused image as the output features of the spatial attention branch; the processing of the fused image by the channel attention branch includes: adding the features obtained by passing the fused image through the global average pooling, the gated channel attention module GCT, and the convolution normalization activation layer in sequence to the original fused image as the output features of the channel attention branch.

[0022] Optionally, the training of the fusion network includes performing M iterative training, and each iterative training includes first completing p rounds of training on the fusion network to update the network parameter Nf of the fusion network, and updating the semantic loss weight β according to the following formula:

[0023] β=γ×(m-1)

[0024] In the above formula, γ is a constant that balances semantic loss and content loss, and m is the number of iterations; then q rounds of training of the segmentation network are completed to update the network parameters Ns of the fusion network.

[0025] In addition, the present invention also provides a visible light and infrared image fusion enhancement system in a low-light environment, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the steps of the visible light and infrared image fusion enhancement method in a low-light environment.

[0026] In addition, the present invention also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is used to be programmed or configured by a microprocessor to execute the steps of the visible light and infrared image fusion enhancement method in a low-light environment.

[0027] Compared with the prior art, the present invention mainly has the following advantages:

[0028] 1. The present invention fuses visible light images and infrared images in low-light environments to obtain a fused image that combines texture information from visible light images and target thermal information from infrared images, thereby achieving complementary information fusion between visible light images and infrared images, improving the image information expression capability in low-light environments, weakening the negative impact of adverse factors (such as low light) on images, and greatly improving the final target detection capability of the fused image.

[0029] 2. The present invention includes enhancing the fused image to remove the influence of light intensity, eliminating the problems of large image dynamic range and target information loss in low light, further weakening the negative impact of undesirable factors (low light, etc.) on the image, and greatly improving the final target detection capability of the fused image.

[0030] 3. The present invention sharpens the enhanced fused image to compensate for the image contour, enhance the image edge and the grayscale jump part, improve the image clarity and contrast, improve the image expression ability, further weaken the negative impact of adverse factors (low light, etc.) on the image, and greatly improve the final target detection ability of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0032] Figure 2 It is a schematic diagram of the principle of Retinex enhancement in an embodiment of the present invention.

[0033] Figure 3 4 is a comparison diagram of the sharpening effect in the embodiment of the present invention.

[0034] Figure 4 Schematic diagram of the network structure of the fusion network in an embodiment of the present invention.

[0035] Figure 5 Schematic diagram of the network structure of the gradient residual dense block in an embodiment of the present invention.

[0036] Figure 6 Schematic diagram of the loss function used when training the fusion network in an embodiment of the present invention.

[0037] Figure 7 Schematic diagram of the network structure of a segmented network in an embodiment of the present invention.

[0038] Figure 8 Schematic diagram of the two branch network structures of the segmented network in an embodiment of the present invention.

[0039] Fig. 9 Schematic diagram of the network structure of the gated channel attention module GCT in an embodiment of the present invention.

[0040] Fig.10 The figure shows the experimental effect of the method of the embodiment of the present invention.

[0041] Fig.11 Schematic diagram of target detection results of the method according to an embodiment of the present invention.

[0042] Fig.12 The figure is a schematic diagram of an experiment comparing the fusion method according to an embodiment of the present invention and the existing fusion method.

[0043] Fig.13 It is a comparison chart of evaluation index curves of the fusion method according to the embodiment of the present invention and the existing fusion method.

[0044] Fig.14 Schematic diagram of target detection results before and after enhancement by the fusion method according to an embodiment of the present invention.

[0045] Fig.15 It is a comparison diagram of the evaluation index curves before and after enhancement of the fusion method in the embodiment of the present invention. DETAILED DESCRIPTION

[0046] like Figure 1 The visible light and infrared image fusion enhancement method in a low light environment of this embodiment includes:

[0047] S1, fusing the visible light image and the infrared image in a low light environment to obtain a fused image that combines texture information of the visible light image and target heat information in the infrared image;

[0048] S2, enhances the fused image to remove the influence of light intensity;

[0049] S3, sharpening the enhanced fused image to compensate for the image contour, enhance the image edge and grayscale jump part, improve the image clarity and contrast, and finally obtain the fused output image.

[0050] In this embodiment, step S2 enhances the fused image to remove the influence of light intensity, specifically by adding an algorithm for removing the illumination of the fused image based on the existing Retinex theory to eliminate the adverse effects of low light. The Retinex theory was proposed by Land and McCann in the 1960s. Its basic theory is: the real world is colorless, and the visually perceived color is the result of the interaction between light and matter. The color of an object is determined by the object's ability to reflect long-wave (red), medium-wave (green) and short-wave (blue) light, rather than by the absolute value of the intensity of the reflected light. The color of an object is not affected by the non-uniformity of light and is consistent, that is, the Retinex theory is based on color consistency (color constancy). In order to restore the true appearance of the image, it is necessary to remove the influence of light intensity.

[0051] The basic model of Retinex theory is as follows Figure 2 As shown, the observed image S is obtained by the reflection of the incident light L by the object surface. The reflectivity R is determined by the object itself and does not change with the incident light L. Then:

[0052] S(x,y)=R(x,y)·L(x,y)

[0053] In the above formula, L(x,y) represents the incident light image, which directly determines the dynamic range that the pixels in the image can reach, R(x,y) represents the reflection property image of the object, that is, the intrinsic property of the image, and S(x,y) represents the reflected light image received by the human eye. The Retinex theory eliminates the influence of L(x,y) in the original image, thereby retaining the essential reflection property image of the object as much as possible. For the convenience of calculation, logarithmic calculation is used, and then:

[0054] logS=log(R·L)

[0055] logS=logR+logL

[0056] s=r+l

[0057] In the above formula, s represents the logarithmic value of the observed image, r represents the logarithmic value of the original reflection component of the image, and l represents the logarithmic value of the incident illumination component of the image.

[0058] The core of the Retinex method is to estimate the illumination L, estimate the L component from the image S, and remove the L component to obtain the original reflection component R, that is:

[0059] l=f(s)

[0060] r = sf(s)

[0061] In the above formula, f(s) represents the logarithmic value of the incident illumination component of the image.

[0062] The Retinex method plays a great role in restoring and enhancing low-light images. Its calculation principle is:

[0063]

[0064] Based on the above calculation principle, it can be deduced that:

[0065] r(x,y)=logS(x,y)-log[F(x,y)*S(x,y)]

[0066] In the above formula, r(x, y) is the enhanced fusion image, S(x, y) is the received reflected light image, * is the convolution symbol, F(x, y) is the center surround function, (x, y) is the pixel coordinates, and the calculation function expression of the center surround function is:

[0067]

[0068] In the above formula, λ is the wavelength of light, and c represents the Gaussian surround scale. The function expression for enhancing the fused image in step S2 is:

[0069] r(x,y)=logS(x,y)-log[F(x,y)*S(x,y)]

[0070] In step S2, the change of illumination in the image is estimated based on the above formula and removed, and finally only the reflection property of the object in the image is retained, thereby achieving the purpose of enhancement.

[0071] After removing the influence of illumination, this embodiment further adopts image sharpening in step S3 to compensate for the contour of the image, enhance the edge of the image and the grayscale jump part, and improve the image clarity and contrast. Thereby improving the detail expression of the image, facilitating the computer to process the image, and promoting the improvement of the target detection effect. Specifically, step S3 of this embodiment includes: for the enhanced fused image, comparing the grayscale of the central pixel with the average grayscale of other pixels in the neighborhood, when the grayscale of the central pixel is lower than the average grayscale of other pixels, adding the template convolution result of the enhanced fused image to the original pixel to further reduce the grayscale of the central pixel; when the grayscale of the central pixel is higher than the average grayscale of other pixels, adding the template convolution result of the enhanced fused image to the original pixel to further increase the grayscale of the central pixel. The comparison before and after image sharpening is as follows. Figure 3 As shown (it has been grayed out due to publication requirements).

[0072] Image fusion can be performed by pixel addition fusion and convolutional neural network. As an optional implementation, the fusion in step S1 of this embodiment refers to inputting the visible light image and infrared image in a low-light environment into a pre-trained fusion network to obtain a fused image, wherein the fusion network is a convolutional neural network. After the network parameters are determined through training, the visible light image and infrared image in a low-light environment are input to obtain a fused image.

[0073] The basic structure of the fusion network is to extract features first and then reconstruct the image. Figure 4 As shown, the fusion network in this embodiment includes two feature extraction branches and a decoder for feature aggregation and image reconstruction. The visible light image and the infrared image in the low-light environment are respectively input into a feature extraction branch, wherein:

[0074] The feature extraction branch includes a common convolution layer and at least one level of gradient residual dense block (represented as RDG module in the figure, specifically 2 levels in the figure, with 16-channel features as input, 32-channel features obtained after processing by the first-level RDG module, and 48-channel features as output). The common convolution layer is used to perform a convolution operation with a convolution kernel size of 3×3 (represented as 3*3conv in the figure), a batch normalization operation (represented as BN in the figure), and a LeakyReLU activation function (represented as ReLU in the figure) on the input image features for activation to extract shallow features; the gradient residual dense block (RDG module) is used to extract deep features;

[0075] The decoder includes at least one level of decoding module ( Figure 3 Specifically, there are 3 levels, namely 48-channel features, 32-channel features, 16-channel features) and an output layer (represented as ConvBnTanh in the figure). The decoding module is used to perform a convolution operation with a convolution kernel size of 3×3, a batch normalization operation, and an activation with a LeakyReLU activation function on the input image features for decoding and restoration. The output layer is used to perform a convolution operation with a convolution kernel size of 1×1, a batch normalization operation, and an activation with a Tanh activation function on the input image features to obtain a fused image.

[0076] like Figure 5 As shown, the gradient residual dense block (RDG module) includes a dense flow branch and a residual flow branch. The dense flow branch includes two convolutional layers with a kernel size of 3×3 and a common convolutional layer with a kernel size of 1×1 using dense connections. The dense connection means that the input features of the gradient residual dense block are respectively used as the inputs of the two convolutional layers and the common convolutional layer, and the outputs of the two convolutional layers with a kernel size of 3×3 are respectively used as the inputs of the common convolutional layer. The residual flow branch includes a gradient operator and a standard convolutional layer with a kernel size of 1×1. The input features of the gradient residual dense block are calculated by the gradient operator to obtain the gradient amplitude. The gradient amplitude is eliminated by the standard convolutional layer with a kernel size of 1×1, and then summed with the output of the dense flow branch to obtain the output features of the gradient residual dense block. The gradient residual dense block (RDG module) uses dense connections to make full use of the features extracted by various convolutional layers, uses dense flow branches and residual flow branches, and the residual flow branch integrates gradient operations, which reduces the amount of calculation. The features fused by the gradient residual dense block (RDG module) enter the decoder structure for feature aggregation and image reconstruction. The decoder structure also consists of 3×3 common convolutional layers and activation layers. During the fusion process, the padding is set to the same, the stride is set to 1, the image features are not downsampled, and the size of the fused image is consistent with the source image, so very little information is lost.

[0077] like Figure 6As shown, before step S1, the fusion network is also trained, and the loss function used in training the fusion network is the joint loss, and the function expression of the joint loss is:

[0078] L joint =L con +βL sem

[0079] In the above formula, L joint is the joint loss, L con is the spatial loss of the fusion network, β is the semantic loss weight, L sem is the semantic loss of the segmentation network; the segmentation network is a semantic segmentation model, which is used to perform semantic segmentation according to the fused image output by the fusion network to obtain the semantic segmentation result of the fused image.

[0080] In this embodiment, the spatial loss L of the fusion network con The calculation function expression is:

[0081]

[0082] In the above formula, H and W are the height and width of the fused image respectively, I f represents the fused image intensity, I ir Indicates the infrared image intensity, I vi represents the intensity of the visible light image, ‖·‖1 is the L1-norm, and max is the maximum value.

[0083] In this embodiment, the semantic loss L of the segmentation network sem The calculation function expression is:

[0084]

[0085] In the above formula, H and W are the height and width of the fused image, C is the channel of the fused image, and L is so represents the segmentation label one-hot vector, I s Represents the semantic segmentation result output by the segmentation network, and the dimension satisfies:

[0086] L so ∈R H×W×C , I s ∈R H×W×C .

[0087] As an optional implementation, the segmentation network can use a real-time semantic segmentation network with a lightweight bilateral attention decoder to construct semantic loss. The input of the semantic segmentation network is the preliminary fusion image, and the semantic segmentation result of the fusion image is fed back to the fusion network to guide the update of the fusion network parameters, forming an iterative update, and finally training a fusion model with a smaller loss. Figure 7As shown in the figure, the segmentation network (lightweight bilateral attention decoder) includes a spatial attention branch, a channel attention branch and a pooling fusion unit. The spatial attention branch is used to extract spatial attention features from the fused image, the channel attention branch is used to extract channel attention features from the fused image, and the pooling fusion unit is used to average the channel attention features and then add them to the spatial attention features at the pixel level. Finally, the semantic segmentation result of the fused image is obtained through the convolution normalization activation layer. The segmentation network (lightweight bilateral attention decoder) fuses feature maps of different levels through two steps: feature extraction and information fusion: In the first step of feature extraction, the preliminary fused image is input into the bilateral attention branch, which includes a channel attention branch and a spatial attention branch. The high-level feature map extracted by the channel attention branch can capture more accurate semantic information, and the low-level feature map extracted by the spatial attention branch can capture more accurate spatial information. In the second step of feature fusion, this paper uses a new pooling fusion block to fuse the extracted high-level and low-level feature maps. This fusion block can make full use of the high-level and low-level feature maps, give full play to the advantages of both, and obtain high-quality fusion results.

[0088] The bilateral attention branch can improve the ability to capture semantic information and spatial information. As a further improvement to the above segmentation network (lightweight bilateral attention decoder), this embodiment improves the attention mechanism in the segmentation network and introduces the gated channel attention module GCT to generate competition between different neurons (or channels) in the neural network. This transformation not only improves the network reasoning speed, but also improves the generalization ability of deep convolutional networks in visual recognition tasks and data sets. Figure 8 As shown in , the spatial attention branch is used to pass the fused image through the convolution normalization activation layer (used to reduce the number of feature channels) and then through the maximum pooling and average pooling operations respectively. Then, the features obtained by the maximum pooling and average pooling operations are summed (so that the spatial detail features of the image can be fully extracted) and the features obtained by the gated channel attention module GCT (enhancing the representation ability of the convolution layer) and the convolution normalization activation layer (conv+BN+Sigmoid layer, compressing the channel into one) are added to the original fused image as the output feature (spatial attention mask) of the spatial attention branch; as shown in Figure 8 As shown in the figure, the processing of the fused image by the channel attention branch includes: adding the features obtained by passing the fused image through global average pooling (GAP), gated channel attention module GCT (enhancing the representation ability of the convolutional layer), and convolution normalization activation layer (conv+BN+Sigmoid layer, compressing the channel into one) in sequence to the original fused image as the output feature (channel attention mask) of the channel attention branch.

[0089] The gated channel attention module GCT can achieve efficient and accurate context information modeling and enhance the representation ability of the convolutional network by modeling channel relationships. Fig. 9 As shown in Figure 1, the gated channel attention module GCT consists of a global context embedding module, a channel normalization module, and a gated adaptation module, where:

[0090] The global context embedding module aggregates the global context information in each channel to avoid local ambiguity caused by a small receptive field. This module can utilize global context information so that the model can pay more attention to the global while focusing on the local. Assuming the embedding weight is α, the function expression of the global context embedding module is:

[0091]

[0092] In the above formula, s c is the feature output by the global context embedding module, α c represents the embedding weight value, x c represents the activation feature of each original channel; the subscript c represents the channel; ε represents a small constant used to avoid calculation problems at zero points; H and W are the image height and width respectively, and x c i,j Represents the eigenvalue of the i-th row and j-th column of the image, and uses the Lp norm to reduce the dimension and output to prevent the model from overfitting. Embedding weight value α c Trainable, so that the gated channel attention module GCT can learn the unique situation of the channel.

[0093] The channel normalization module is used to create a competitive relationship between neurons (or channels) with lightweight computing resources and stable training performance. Channel normalization is performed by embedding information and trainable parameters, and the L2 norm is used to operate across channels, that is, channel normalization (normalization). Let s = [s1, ..., sC], s represents the feature output by the global context embedding module, s1~sC represent the weighted feature values ​​of each channel respectively, and the function expression of channel normalization is:

[0094]

[0095] In the above formula, For the characteristics c The result of channel normalization, C represents the number of channels, s c is the feature output by the global context embedding module, and ε is the small constant mentioned above.

[0096] In the gated adaptive module, each original channel x c The scale of will be adjusted through its corresponding gate, training weight γ c and the bias parameter β c, the gate channel can be effectively activated. When the gating weight of a channel γ c When activated positively, the gated channel attention module GCT promotes the channel to compete with other channels. When the gated weight is negatively activated, GCT encourages the channel to cooperate with other channels. The function expression of the gated adaptive module is:

[0097]

[0098] In the above formula, is the original channel x c After the output of the gated adaptive module, tanh represents the tanh activation function. For the characteristics c The result of channel normalization, γ c is the weight, β c is the bias parameter.

[0099] The gated channel attention module GCT is similar to a batch normalization layer and works best when inserted before a convolutional layer.

[0100] See also Figure 6 , the input infrared image samples and visible light image samples are iteratively updated through the fusion network and the segmentation network. The training process is divided into the fusion stage and the segmentation stage for iterative training. First, the joint loss composed of spatial information loss and semantic loss is calculated according to the fusion network results. The joint loss is used to guide the p-step cyclic update of the fusion network parameters. The fused image generated after the update is passed through the segmentation network to obtain a new semantic loss, and the segmentation network parameters are cyclically updated for q steps. The new semantic loss is fed back to the joint loss, so that the fusion network and the segmentation network are iterated for a set number of times as a whole, and finally a trained fusion network model is obtained. Among them, p and q are natural numbers. The larger the value, the higher the accuracy, but it will also lead to a decrease in training efficiency, so it can be specified as needed.

[0101] It should be noted that the semantic loss weight β can be a static parameter. However, in order to achieve dynamics and calculate the joint loss composed of spatial information loss and semantic loss, the semantic loss weight β is used as a hyperparameter in this embodiment, so that it gradually increases with the combined low-level and high-level adaptive training strategies. Specifically, when training the fusion network in this embodiment, it includes performing M iterative training, and each iterative training includes first completing p rounds of training on the fusion network to update the network parameters Nf of the fusion network, and updating the semantic loss weight β according to the following formula:

[0102] β=γ×(m-1)

[0103] In the above formula, γ is a constant that balances semantic loss and content loss, and m is the number of iterations; then q rounds of training of the segmentation network are completed to update the network parameters Ns of the fusion network.

[0104] In many scenes with harsh conditions, lighting has a huge impact on images. In low light, a single visible light image cannot convey reliable information and may even lead to the loss of target information. At this time, the advantages of the fusion of visible light and infrared images will be revealed. For a given pair of visible light and infrared images, which will be collectively referred to as dual-light images in this article, it is crucial to maximize the use of their information and improve the information expression ability of the fused image. Traditional methods evaluate the fusion performance with statistical indicators and are not linked to upstream tasks. The method in this embodiment takes the target detection task as the ultimate goal and studies the image fusion method driven by the target detection task. Semantic loss is used to guide the fusion of dual-light images. Through the visual enhancement module, the information expression level of the image is improved, and the negative impact of adverse factors (low light, etc.) on the image is weakened. As a result, the final target detection capability of the fused image is greatly improved.

[0105] To validate the method of this embodiment, we conducted extensive qualitative and quantitative experiments on the MSRS dataset. The MSRS dataset is a road scene dataset, in which the training set contains 1083 pairs of infrared and visible light images, and the test set consists of 361 pairs of images, including daytime and nighttime situations. We mainly select nighttime images for testing and verification. The dataset provides semantic labels for objects such as cars, people, bicycles, curves, car stops, guardrails, color tones, and backgrounds. All images are normalized to [0,1] before being input into the network. This paper trains a semantic-aware fusion network on the MSRS dataset. According to the joint low-level and high-level adaptive training strategy, the fusion network and the segmentation network are iteratively trained. All parameters in our joint adaptive training strategy are set as follows: M = 4, p = 1500, q = 10,000 and γ = 1. At the same time, our fusion model is optimized under the guidance of the joint loss. In addition, we use mini-batch stochastic gradient descent with a batch size of 16, a momentum of 0.9, and a weight decay of 0.0005 to optimize the segmentation network. The initial learning rate is set to 0.01, and the exponential decay method is used to control the learning rate of each iteration. The power is set to 0.9. This method is implemented in the PyTorch deep learning framework. The machine uses NVIDIA GeForce GTX 1080Ti for the experiment. Then we use the visual perception enhancement module proposed in this paper to enhance the results of the trained model, use the Retinex method, set the number of iterations to 4, and then use the Laplacian operator center point to take 5 for sharpening.

[0106] 1. Qualitative analysis

[0107] In this embodiment, a semantically aware fusion network is used to combine the spatial information and semantic information of visible light and infrared images in low-light scenes, and visual enhancement is performed at the same time to obtain a considerable fusion enhancement result. The main steps are as follows: Fig.10 European style, the effect is shown as Fig.11 See Fig.11 ,The infrared image and visible light image are first processed by a task-driven image fusion network and then enhanced. The final result highlights the pedestrian target in the dark, removes the influence of illumination, better restores the image color, and greatly improves the image information expression.

[0108] This paper conducted a test on the MSRS dataset, selecting the fused and enhanced images as follows Fig.12 , where (a) represents an infrared image, (b) represents a visible light image, (c) represents a fused image obtained by the existing NSCT-SR method, (d) represents a fused image obtained by the existing LPfuse method, (e) represents a fused image obtained by the existing Densefuse method, (f) represents a fused image obtained by the existing Curvelet method, (g) represents a fused image obtained by the existing IFCNN method, and (h) represents a fused image obtained by the method of this embodiment. Fig.12 It can be seen that after the infrared images and visible light images under low-light conditions are enhanced, the amount of image information is greatly improved, the number of detectable target types in the image increases, and the clarity is greatly improved, laying a good foundation for target detection.

[0109] 2. Quantitative Analysis

[0110] 1. Comparative experiment.

[0111] The fusion results of the proposed method are compared with other methods in terms of traditional indicators (information entropy EN, spatial frequency SF, average gradient AG), target detection accuracy (mAP) and the final overall evaluation index. The statistical numerical evaluation index is based on the properties of the fused image itself, mainly evaluating the amount of information it contains and the clarity of the image. SF measures the spatial frequency information contained in the fused image. The average gradient AG can reflect the texture information of the image. The three can indicate the richness of the information carried by the image. The higher the value, the richer the information. Fig.12 The fusion effect of this method and the other five methods is shown in Table 1. The experimental comparison results of different fusion methods are shown in Table 1.

[0112] Table 1: Comparison of evaluation indicators of different methods.

[0113] P R mAP@0.5 mAP@0.5:0.95 Q Densefuse 0.831 0.679 0.706 0.428 2.1832 IFCNN 0.786 0.694 0.635 0.352 2.7208 Lpfuse 0.64 0.756 0.642 0.38 2.7976 Nsctsr 0.72 0.66 0.655 0.383 2.8041 Curvelet 0.726 0.674 0.666 0.384 2.7418 Methods 0.786 0.795 0.75 0.433 2.7816

[0114] In the table, Densefuse, IFCNN, Lpfuse, Nsctsr and Curvelet are all well-known existing methods, P represents detection accuracy, R represents recall rate, mAP@0.5 represents the ratio of target detection accuracy of 0.5, mAP@0.5:0.95 represents the ratio of target detection accuracy of 0.5-0.9, single image statistic index Qp=1 / 3(EN+SF+AG), EN represents information entropy, SF represents spatial frequency, and AG represents average gradient. Fig.12 As can be seen from Table 1, the image fused by the proposed method has the best mAP value, and the traditional index mean Qp ranks third. The Qp of the fused image after enhancement can reach 7.8.

[0115] As for the final evaluation index, currently very few of them combine the achievement of task objectives into the evaluation index. This paper establishes a new evaluation index model and selects two indicators for combination. The first is a quantitative evaluation of the information attributes of the fused image itself, mainly including information entropy EN, average gradient AG, etc. The second is an evaluation based on the contribution of the fused image to the low-light target detection task effect.

[0116] The overall evaluation index for the fused image is:

[0117] η=α1·Qp+α2·mAP

[0118] Since this paper focuses on the target detection task, the weight α1 is set to 0.2, the weight α2 is set to 0.8, and the test results of different fusion methods and the final evaluation index η of the fusion enhanced image in this paper on the MSRS dataset are as follows: Fig.13 As shown in the figure, the curve ours is the overall evaluation result obtained by the fusion method in this paper. Compared with the other five methods, the final evaluation index of this paper can reach the highest. It can be seen that the improved fusion method proposed in this paper can better overcome the adverse effects under low light conditions, enhance the information richness of the fused image, and also improve the target detection ability.

[0119] 2. Ablation experiment

[0120] The effectiveness of the enhancement module is evaluated using an ablation experiment. The visual enhancement module strengthens the image's ability to express information and eliminates the negative effects of low light on the occlusion and weakening of image targets. In order to evaluate the improvement of the visual enhancement module on the detection task performance, the enhancement module is not added first, and the target detection network YOLOv5 is used to detect the fused image, and the number of correctly detected target types and confidence levels are recorded. Subsequently, the enhancement module is added, and the number of correctly detected target types and confidence levels are also recorded, and the fusion evaluation index is calculated. The two are compared to highlight the advantages of the enhancement module.

[0121] Three groups of dual-light fusion image results under low-light conditions were randomly selected. Fig.14 (Grayscaled due to publication restrictions) The first row is the fused image before enhancement, and the second row is the image after enhancement. Comparing the evaluation indicators and effects before adding the visual enhancement module: It can be seen from the ablation experiment results that in the first group of images, the left side is before enhancement, and the right side is after enhancement. After enhancement, the number of correct target detections is 5 more than before, and the total confidence is increased by 2.03. The statistical indicators of single images are: Qp1=2.8353, Qp2=8.3033, where Qp1 is the statistical indicator of a single image in the first row, and Qp2 is the statistical indicator of a single image in the second row. In the second group of images, the number of correct target detections after enhancement is 1 more than before enhancement, and the total confidence is increased by 0.8. The statistical indicators of single images are: Qp1=3.2537, Qp2=8.6267. According to the evaluation model proposed in this article, taking the first group as an example, the evaluation indicators before and after fusion image enhancement are calculated as follows: Fig.15 As shown in the figure, the left curve "fused" is the average index before adding the enhancement module, and the right curve "fusedandenchanced" is the average index twice as high as before after adding the enhancement module. Fig.15 ,It can be seen that the enhancement module can greatly improve the image expression ability, and can effectively improve the types and accuracy of target detection.

[0122] In summary, the visible light and infrared image fusion enhancement method under low light environment of this embodiment is inspired by the existing SeAFusion fusion method. Aiming at the problems of uneven brightness, color distortion, serious information loss and low adaptability to advanced visual tasks of fused images under low light, this paper proposes a task-driven dual-light image fusion and enhancement method under low light conditions. First, based on the existing fusion image evaluation indicators, the effective evaluation indicators of dual-light fusion images are further defined. The ultimate goal of the fusion image is to obtain better target detection performance, including detecting as many target types as possible in the image and higher confidence. This paper includes whether this goal can be achieved well as one of the important indicators for evaluating fusion images. Then, based on the semantic perception fusion network (SeAFusion), the attention mechanism is improved to improve the generalization ability of the model. Secondly, in view of the problem of weak information expression ability of fused images under low light conditions, a visual enhancement module is proposed to maintain high image fidelity, compress the dynamic range of the image, achieve multi-scale adaptation, and achieve texture detail enhancement, eliminating the visual disadvantages caused by low light. Finally, an image evaluation index model including theoretical analysis and task requirements is established. The visible light and infrared image fusion enhancement method in low light environment of this embodiment takes the target detection task as the ultimate purpose of the fused image, fuses the rich texture information of the visible light image and the target heat information of the infrared image through the semantic perception fusion network, and improves the image expression ability (image clarity and information volume) through the visual enhancement module, so as to maximize the information utilization of the infrared and visible light image pairs provided. The fused enhanced image proposed in this paper can be applied to situations such as harsh conditions, lack of light, and low image resolution, and is of great significance to traffic road monitoring, battlefield environment monitoring, and field environment survey.

[0123] In addition, this embodiment also provides a visible light and infrared image fusion enhancement system in a low-light environment, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the steps of the aforementioned visible light and infrared image fusion enhancement method in a low-light environment.

[0124] In addition, this embodiment also provides a computer-readable storage medium, in which a computer program is stored. The computer program is used to be programmed or configured by a microprocessor to execute the steps of the aforementioned visible light and infrared image fusion enhancement method in a low-light environment.

[0125] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0126] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A visible light and infrared image fusion enhancement method in low light environment, Features include: S1, fusing the visible light image and the infrared image in a low light environment to obtain a fused image that combines texture information of the visible light image and target heat information in the infrared image; S2, enhances the fused image to remove the influence of light intensity; S3, sharpening the enhanced fused image to compensate for the image contour, enhance the image edge and grayscale jump part, improve the image clarity and contrast, and finally obtain the fused output image; Before step S1, the fusion network is also trained, and the loss function used in training the fusion network is the joint loss, and the function expression of the joint loss is: In the above formula, L joint For joint losses, L con is the spatial loss of the fusion network, β is the semantic loss weight, L sem is the semantic loss of the segmentation network; the segmentation network is a semantic segmentation model, which is used to perform semantic segmentation according to the fused image output by the fusion network to obtain the semantic segmentation result of the fused image; The segmentation network includes a spatial attention branch, a channel attention branch and a pooling fusion unit, wherein the spatial attention branch is used to extract spatial attention features from the fused image, the channel attention branch is used to extract channel attention features from the fused image, and the pooling fusion unit is used to average pool the channel attention features, then add them to the spatial attention features at the pixel level, and finally obtain the semantic segmentation result of the fused image through the convolution normalization activation layer; the spatial attention branch is used to first pass the fused image through the convolution normalization activation layer and then through the maximum pooling and average pooling operations respectively, and then sum the features obtained by the maximum pooling and average pooling operations, and then sequentially pass through the gated channel attention module GCT, the convolution normalization activation layer to obtain the features and the original fused image to obtain the output features of the spatial attention branch; The processing of the fused image by the channel attention branch includes: adding the features obtained by sequentially passing the fused image through global average pooling, gated channel attention module GCT, and convolution normalization activation layer to the original fused image as the output features of the channel attention branch.

2. The visible light and infrared image fusion enhancement method in a low light environment according to claim 1, It is characterized in that The function expression for enhancing the fused image in step S2 is: In the above formula, r ( x , y ) is the enhanced fusion image, S ( x , y ) is the received reflected light image, * is the convolution symbol, F ( x , y ) is the center surround function, ( x , y ) is the pixel coordinate, where the calculation function expression of the center surround function is: In the above formula, λ is the wavelength of light, c Expressed as a Gaussian surround scale.

3. The visible light and infrared image fusion enhancement method in a low light environment according to claim 1, It is characterized in that Step S3 includes: for the enhanced fused image, comparing the grayscale of the central pixel with the average grayscale of other pixels in the neighborhood, when the grayscale of the central pixel is lower than the average grayscale of other pixels, adding the template convolution result of the enhanced fused image to the original pixel to further reduce the grayscale of the central pixel; when the grayscale of the central pixel is higher than the average grayscale of other pixels, adding the template convolution result of the enhanced fused image to the original pixel to further increase the grayscale of the central pixel.

4. The visible light and infrared image fusion enhancement method in a low light environment according to claim 1, It is characterized in that The fusion in step S1 refers to inputting the visible light image and the infrared image in the low-light environment into a pre-trained fusion network to obtain a fused image, wherein the fusion network includes two feature extraction branches and a decoder for feature aggregation and image reconstruction, and the visible light image and the infrared image in the low-light environment are respectively input into a feature extraction branch, wherein: the feature extraction branch includes a common convolution layer and at least one level of gradient residual dense block, the common convolution layer is used to perform a convolution operation with a convolution kernel size of 3×3, a batch normalization operation, and an activation using a LeakyReLU activation function on the input image features to extract shallow features; the gradient residual dense block is used to extract deep features; the decoder includes at least one level of decoding module and an output layer, the decoding module is used to perform a convolution operation with a convolution kernel size of 3×3, a batch normalization operation, and an activation using a LeakyReLU activation function on the input image features for decoding and restoration, and the output layer is used to perform a convolution operation with a convolution kernel size of 1×1, a batch normalization operation, and an activation using a Tanh activation function on the input image features to obtain a fused image.

5. The visible light and infrared image fusion enhancement method in a low light environment according to claim 4, It is characterized in that The gradient residual dense block includes a dense flow branch and a residual flow branch. The dense flow branch includes two convolutional layers with a kernel size of 3×3 and a common convolutional layer with a kernel size of 1×1 using dense connections. The dense connection means that the input features of the gradient residual dense block are respectively used as the inputs of the two convolutional layers and the common convolutional layer, and the outputs of the two convolutional layers with a kernel size of 3×3 are respectively used as the inputs of the common convolutional layer. The residual flow branch includes a gradient operator and a standard convolutional layer with a kernel size of 1×1. The input features of the gradient residual dense block are calculated by the gradient operator to obtain the gradient amplitude. The gradient amplitude is eliminated by the standard convolutional layer with a kernel size of 1×1, and then summed with the output of the dense flow branch to obtain the output features of the gradient residual dense block.

6. The visible light and infrared image fusion enhancement method in a low light environment according to claim 1, It is characterized in that The training of the fusion network includes performing M iterations of training, and each iteration of training includes first completing p rounds of training on the fusion network to update the network parameters Nf of the fusion network, and updating the semantic loss weight according to the following formula: β : In the above formula, γ To balance the constant between semantic loss and content loss, m is the number of iterations; then q rounds of training for the segmentation network are completed to update the network parameters Ns of the fusion network.

7. A visible light and infrared image fusion enhancement system in a low light environment, comprising a microprocessor and a memory connected to each other, It is characterized in that The microprocessor is programmed or configured to execute the steps of the visible light and infrared image fusion enhancement method in a low-light environment as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored therein, It is characterized in that The computer program is used to be programmed or configured by a microprocessor to execute the steps of the visible light and infrared image fusion enhancement method in a low-light environment as described in any one of claims 1 to 6.

Images