OpenCL-based low-light scene infrared and visible light image rapid fusion method

Abstract

The invention relates to a low-light scene infrared and visible light image rapid fusion method based on OpenCL, and belongs to the technical field of computer vision. According to the method, a low-light enhancement task of a visible light image and a fusion task of infrared and visible light images are subjected to combined modeling. Based on a brightness component of a color space, low light enhancement and gradient transfer are jointly optimized, and simple and effective enhancement fusion of an infrared image and a visible light image in a low-illumination night scene is realized. The invention provides a complementary logarithmic gamma correction function, the ELGC realizes low light enhancement, the SLGC realizes high light suppression, and the ELGC and the SLGC interact with each other, so that the enhancement of general brightness in the daytime can be realized.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method for rapid fusion of infrared and visible light images in low-light scenes based on OpenCL. Background Technology

[0002] The fusion of infrared and visible light images can be applied to military operations, object detection, and autonomous driving. While many mature fusion algorithms exist, designing a simple and effective fusion algorithm for infrared and visible light images to adapt to low-light nighttime scenarios remains a challenge.

[0003] Deep learning-based image fusion techniques can adapt well to multimodal data, such as autoencoders (AEs), convolutional neural networks (CNNs), and generative adversarial networks (GANs), effectively improving the quality of fused images. However, these techniques typically require large amounts of training data and incur high training costs, and are often accompanied by problems such as insufficient model generalization ability.

[0004] Traditional multimodal image fusion methods include multi-scale decomposition fusion, sparse representation methods, and subspace-based methods. Among these, multi-scale decomposition methods are computationally fast and easy to implement. However, they are weak in extracting image edge information and are prone to redundancy and artifacts at image edges. This is especially evident when fusing visible light glare targets and their corresponding infrared targets in low-light night scenes. Sparse representation-based image fusion techniques address the problem of insufficient feature information in multi-scale transformations, but overly comprehensive dictionary representation capabilities and complex fusion rules limit the fusion quality. These limitations become more pronounced when fusing strong light source targets such as traffic lights and traffic signals in nighttime environments. Subspace-based image fusion techniques are weak in extracting nonlinear features, and in low-light nighttime environments, visible light images are noisy and sparse in prominent features, which may lead to feature extraction failure.

[0005] Gradient Transfer Fusion (GTF) uses image fusion as an optimization method, possessing powerful feature extraction capabilities and avoiding the need to design complex fusion rules. The fusion result preserves both the radiance of the infrared image and the gradient information of the visible light image. However, due to the low brightness and blurred texture details of the visible light image in night scenes, the fusion result obtained by GTF is similar to that of the infrared image, losing the rich color information of the visible light image and affecting the quality of the fused image. Summary of the Invention

[0006] (a) Technical problems to be solved The technical problem to be solved by the present invention is to provide a method for enhancing and fusing infrared and visible light images in low-light environments at night, in order to solve the problem of insufficient cooperation between traditional visible light and infrared fusion (VIF) methods and low-light enhancement techniques in night scenes.

[0007] (II) Technical Solution To address the aforementioned technical problems, this invention provides a method for rapid fusion of infrared and visible light images in low-light scenes based on OpenCL. This method, implemented using OpenCL and designed specifically for the luminance channel, firstly enhances the source image (visible light image) using a complementary log-gamma correction function and BPDHE sharpening method. This enhances low-brightness areas in the night scene while suppressing high-brightness areas, thereby increasing the brightness of the visible light image. Secondly, the luminance component of the enhanced visible light image is fused with the infrared image using a gradient transfer fusion (GTF) algorithm, ensuring that the pixel intensity distribution of the fused image matches that of the infrared image, ultimately yielding the fused image.

[0008] (III) Beneficial Effects ① This invention jointly models the low-light enhancement task of visible light images and the fusion task of infrared and visible light images. Based on the luminance component of the color space, it jointly optimizes low-light enhancement and gradient transfer, achieving a simple and effective enhancement and fusion of infrared and visible light images in low-light night scenes.

[0009] ② This invention proposes a complementary log-gamma correction function, in which ELGC achieves low-light enhancement and SLGC achieves high-light suppression. The two interact to achieve general brightness enhancement during the day.

[0010] ③ To address the issue of image detail loss after high brightness enhancement, this method introduces an improved Brightness-Preserving Dynamic Histogram Equalization (BPDHE) sharpening technique. The fusion result maintains high scene brightness while suppressing glare and enhances the contrast of nighttime visible images.

[0011] ④ In order to improve the performance of the algorithm on edge computing devices, this invention uses the concept of OpenCL single instruction multithreading to accelerate the deployment of the algorithm. It makes full use of the heterogeneous characteristics of edge computing devices and achieves collaborative optimization of the CPU-side algorithm flow control and the GPU-side core algorithm, which effectively improves the computation speed of the algorithm. Attached Figure Description

[0012] Figure 1The results are qualitative experimental results, where (a) is vis, (b) is ir, (c) is LP, (d) is RP, (e) is DTCW, (f) is CVT, (g) is MSVD, (h) is GTF, (i) is DIVFusion, (j) is FusionGAN, and (k) is LLE-GTF. Figure 2 Quantify experimental results for multiple indicators; Figure 3 This is a flowchart of the LLE-GTF algorithm of the present invention. Detailed Implementation

[0013] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0014] This invention proposes a fast image fusion method for infrared and visible light (LLE-GTF) based on gradient transfer and low-light enhancement co-optimization. LLE-GTF is an improved strategy of GTF. Experiments show that the fusion results achieved by the method of this invention in low-light nighttime scenes not only suppress glare and approach the clarity of daytime scenes, but also better match human visual sensitivity.

[0015] The present invention proposes a fusion algorithm called LLE-GTF, which can solve the problem of insufficient cooperation between traditional visible light and infrared fusion (VIF) methods and low light enhancement technology in nighttime scenes.

[0016] (1) Technical problems to be solved ① At night, visible light images are affected by light attenuation. The task of fusing infrared and visible light images can be decomposed into two sub-problems: visible light image enhancement and infrared-visible light image fusion. However, existing non-deep learning works often treat these two problems as independent issues. Therefore, how to jointly model the enhancement and fusion tasks and generate high-quality color fusion results has become the key to infrared-visible light image fusion in low-light conditions at night.

[0017] ② Gamma correction is an image correction technique developed to address the non-linear output characteristics of early CRT monitors. Its core principle lies in non-linear transformation, meaning the relationship between output brightness and input brightness follows a power-law formula. Among them, gamma value Approximately Gamma correction is used to compensate for the non-linear response of the display and ensure that the image can be accurately displayed on the picture tube. Gamma correction is a simple and effective method for enhancing low light, but it not only enhances the brightness of dark areas in the image, but also further exposes the bright areas, resulting in image distortion.

[0018] ③ Brightness enhancement will reduce the contrast of the image. In order to balance brightness and contrast, the commonly used method is histogram equalization (HE) operation. HE will cause image distortion and loss of detail. Therefore, it is necessary to solve the dynamic balance between brightness enhancement and contrast preservation.

[0019] ④ Existing image enhancement and image fusion technologies run fast on high-performance terminals, but when applied to edge computing devices, the computing speed is limited by computing power and algorithm complexity.

[0020] (2) The technical solution mainly includes the following contents: ① This invention jointly models the low-light enhancement task of visible light images and the fusion task of infrared and visible light images. Based on the luminance component of the color space, it jointly optimizes low-light enhancement and gradient transfer, achieving a simple and effective enhancement and fusion of infrared and visible light images in low-light night scenes.

[0021] ②This invention proposes a complementary log-gamma correction function, in which ELGC achieves low-light enhancement and SLGC achieves high-light suppression. The two interact to achieve general brightness enhancement during the day.

[0022] ③ To address the issue of image detail loss after high brightness enhancement, this method introduces an improved Brightness-Preserving Dynamic Histogram Equalization (BPDHE) sharpening technique. The fusion result maintains high scene brightness while suppressing glare and enhances the contrast of nighttime visible images.

[0023] ④ In order to improve the performance of the algorithm on edge computing devices, this invention uses the concept of OpenCL single instruction multithreading to accelerate the deployment of the algorithm. It makes full use of the heterogeneous characteristics of edge computing devices and achieves collaborative optimization of the CPU-side algorithm flow control and the GPU-side core algorithm, which effectively improves the computation speed of the algorithm.

[0024] (3) Detailed description of the technical solution 1. Algorithm Framework Design The LLE-GTF algorithm of this invention ( Figure 3 Based on OpenCL, this design primarily focuses on the luminance channel (V-channel). First, it enhances the source image (visible light image) using complementary log-gamma correction and BPDHE sharpening techniques. The aim is to enhance low-brightness areas in night scenes while suppressing high-brightness areas, thereby improving shadow detail and overall brightness. Then, it fuses the luminance component of the enhanced visible light image with the infrared image using a gradient transfer fusion (GTF) algorithm. This ensures that the pixel intensity distribution of the fused image matches that of the infrared image, preserving the advantages of the infrared image in night scenes while utilizing the detail information from the visible light image to improve image sharpness.

[0025] 2. Complementary Gamma Logarithmic Function For the luminance component V in the HSV color space of visible light images, a low-light enhancement log-gamma correction function ELGC is proposed to optimize image enhancement under low-light conditions. This function ELGC significantly improves the visual quality of images in low-light environments by adjusting the luminance component V while maintaining color balance. The output of the ELGC function is as follows: in, Represents the input luminance component V, parameter Based on experience points, according to experience points Set it to 20. To balance overexposure in high-brightness regions, a high-brightness suppression log-gamma function SLGC, complementary to ELGC, is proposed, which is related to ELGC... Centrally symmetric, SLGC and ELGC form a complementary log-gamma correction function. ELGC enhances the image by increasing the brightness of low-brightness areas, while SLGC effectively preserves image details and prevents visual distortion by suppressing overexposure in originally high-brightness areas. The output of the SLGC function is as follows: To achieve global adaptive balance, adaptive weighting parameters are introduced. Specifically, two weighting parameters are defined. and The output of the complementary log-gamma correction function is : 3. Improved BPDHE The BPDHE method optimizes the specular distortion and detail loss that the HE method may introduce, but it cannot guarantee the visual realism of the image when dealing with nighttime noise. The following is a method for enhancing the source image (i.e., the visible light image) using BPDHE sharpening technology: Use the original luminance component V Standard Gaussian kernel smoothing is used to obtain : By adjusting the parameters Enhance the original luminance component V to obtain .

[0026] The contrast enhancement component is obtained by the difference between the two. : Will and By combining them with equal weights, the enhanced luminance components are obtained. .

[0027] 4. LLE-GTF By optimizing the model, the GTF method not only preserves the thermal radiation information in the infrared image but also effectively incorporates the rich information from the visible light image. To ensure the fusion result has rich color and image detail, the enhanced luminance component and the infrared image are used as inputs, resulting in a new mathematical modeling expression: in It is the empirical error of the intensity distribution, using Paradigm representation, This represents the column vector of the fused image, and the value to be determined is... This represents the column vector of the infrared image.

[0028] in, It is the empirical error of the gradient distribution, denoted by the q-norm; Represents the gradient of the column vector of the fused image; Venhanced represents the gradient of the brightness component after weak light enhancement. The calculation of the fusion result can be expressed as optimizing the following objective function: The parameter λ controls the amount of gradient information transferred from the visible light image to the fusion result, and the size of the paradigm is set to p = 1 and q = 1.

[0029] Solve The method is as follows: set up The objective function can be written as: i represents the pixel index, and m and n are the length and width of the image, respectively. for The value of the i-th pixel, Let I be the value of the i-th pixel. for The value of the i-th pixel, for The value of the horizontal component of the gradient at the i-th pixel. for The vertical component of the gradient at the i-th pixel is used to find the optimal solution y using the objective function, and the fusion result is obtained through calculation. The fusion result is used as a new luminance component and then merged into the RGB space to obtain the final fused image.

[0030] 5. OpenCL Heterogeneous Acceleration The color space conversion of the image is calculated using a lookup table. A lookup table mapping RGB to HSV channels is used, and this table is stored in the GPU's global memory, enabling rapid mapping and separation of color spaces.

[0031] In the process of accelerating low-light enhancement using complementary log-gamma functions, a method was created... Each GPU thread corresponds to one pixel unit. Each thread independently performs logarithmic enhancement, contrast enhancement, and weight combination.

[0032] During the gradient transfer optimization process, the calculation results at each location are stored in local memory, moved to global memory after several iterations, and then moved to the CPU.

[0033] The experiments comprehensively evaluated the performance of LLE-GTF using the publicly available LLVIP dataset. Qualitative, quantitative, and ablation experiments were conducted, and the results of LLE-GTF were compared and analyzed with those of six traditional image fusion methods and two deep learning-based methods. These methods include LP, RP, DTCWT, CVT, MSVD, GTF, as well as DIVFusion based on convolutional neural networks (CNN) and FusionGAN based on generative adversarial networks (GAN).

[0034] Experimental results show that the qualitative experimental results are as follows: Figure 1 As shown, the fused images obtained by (c)LP, (d)RP, (e)DTCWT, (g)MSVD, and (i)FusionGAN are relatively blurry and lack clear details and textures. In contrast, (i)DIVFusion achieves high contrast and high brightness enhancement, but the fusion result shows color distortion. The LLE-GTF method significantly suppresses glare in the fusion result, such as... Figure 1 Glare from electric bicycles. The LLE-GTF method enhances shadow details, such as... Figure 1 The results show that even in dark environments, bright scenes resembling daytime can be obtained with rich texture details.

[0035] To comprehensively evaluate the performance of LLE-GTF, we randomly selected 50 sets of images from the LLVIP dataset and a self-made dataset for quantitative experiments. The results are shown below. Figure 2See Table 1. Higher scores on the metrics used in the experiment indicate better fusion performance. The experimental results include two aspects: first, the average value of each method on each metric; and second, the standard deviation of the results obtained by each method.

[0036] Table 1. Quantitative experimental results (optimal results are indicated in bold, and suboptimal results are indicated by underline). The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An OpenCL-based fast fusion method of infrared and visible light images in low-light scenes, characterized in that, The method is realized based on OpenCL, and is designed for the luminance channel, wherein first, a complementary logarithmic gamma correction function and a BPDHE sharpening method are used to enhance the source image, i.e., the visible light image, to enhance the low-brightness area in the night scene while suppressing the high-brightness area, and to improve the brightness of the visible light image; second, the brightness component of the enhanced visible light image is fused with the infrared image through a gradient transfer fusion GTF algorithm, so that the pixel intensity distribution of the fused image is consistent with the pixel intensity distribution of the infrared image, and finally a fused image is obtained.

2. The method of claim 1, wherein, In the method, a weak light enhancement logarithmic gamma correction function ELGC is designed for the luminance component V of the visible light image in the HSV color space; the function ELGC adjusts the luminance component V to improve the visual quality of the image in a weak light environment while maintaining color balance; and the output of the function ELGC is as follows: wherein, represents the input luminance component V, the parameters are empirical values.

3. The method of claim 2, wherein, is set to 20.

4. The method of claim 2, wherein, In the method, a high-brightness suppression logarithmic gamma function SLGC complementary to the ELGC is designed, and the SLGC and the ELGC are complementary with respect to The center is symmetrical, and the SLGC and the ELGC constitute complementary logarithmic gamma correction functions; the output of the function SLGC is as follows: Define two weighting parameters and The output of the complementary log-gamma correction function is then : 。 5. The method of claim 4, wherein, The ELGC enhances the image by enhancing the brightness of the low-brightness area, and the SLGC prevents visual distortion by suppressing the overexposure of the originally high-brightness area.

6. The method of claim 4, wherein, In the method, the BPDHE sharpening method is used to enhance the visible light image as follows: The luminance component V of the visible light image is used Gaussian kernel smoothing processing, obtaining : By adjusting parameters Enhancing the luminance component V to obtain : By differencing the two, a contrast-enhanced component is obtained : combining with equal weight, resulting in an enhanced luminance component with combining with equal weight, resulting in an enhanced luminance component .

7. The method of claim 6, wherein, 。 8. The method of claim 6, wherein, The method of fusing the brightness component of the enhanced visible light image with the infrared image through the gradient transfer fusion GTF algorithm is specifically as follows: enhanced luminance component and the infrared image as input, a mathematical modeling expression is obtained: wherein, is the empirical error of the intensity distribution, with is expressed in the paradigm, denotes the fused image column vector, for the values to be sought, denotes the infrared image column vector; wherein, is the empirical error of the gradient distribution, expressed in the q-norm; denotes the gradient of the fused image column vector; Venhanceddenotes the gradient of the luminance component after light enhancement; it is calculated by optimizing the following objective function : where the parameter λ is used to control the amount of gradient information transferred from the visible light image to the fusion result The size of the norm is set as: p = 1, q = 1.

9. The method of claim 8, wherein, Solving The method is as follows: Let The objective function is written as: i represents pixel index, m, n are length, width of image, is the value of the i-th pixel, is the value of the i-th pixel, is the value of the i-th pixel, is the horizontal component of the gradient of the i-th pixel, is the vertical component of the gradient of the i-th pixel, the optimal solution y is found by using the objective function, and the fusion result is obtained by calculation .

10. The method of claim 9, wherein, The fusion result As a new luminance component, then merged into the RGB space, get the final fusion image.

Images