Contrast-preserving near-infrared and high-noise visible light image fusion method

Abstract

The present invention belongs to the field of image processing technology, specifically a method for fusing near-infrared and high-noise visible light images while maintaining contrast. The present invention includes: performing pre-denoising on the noisy visible light image before image fusion; in the image fusion process, first generating a low-frequency image with a contrast similar to NIR and an appearance similar to VIS, and denoising it again as the base layer after fusion; then extracting high-frequency components from VIS and NIR respectively; generating a fusion weight map based on local contrast, adaptively weighting and fusion the high-frequency components of VIS and NIR as the fused detail layer; finally, adding the detail layer to the base layer, converting it to the color space required for display, and obtaining the final image fusion result. The present invention only uses pixels in the local neighborhood for processing each pixel, without obtaining global information or iterative solution, which is conducive to hardware implementation and real-time processing, and the generated image content is accurate and rich in details.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for fusing near-infrared and high-noise visible light images while maintaining contrast. Background Art

[0002] Visible light images (VIS) and near-infrared images (NIR) each have their own strengths and weaknesses in image quality. Fusion of the two can complement each other's information and enhance the visible light image, improving its signal-to-noise ratio and clarity. Existing VIS and NIR image fusion algorithms typically first decompose the image (into a base layer + detail layer, or using a pyramid decomposition), apply different fusion rules to components of different frequencies, and finally reconstruct the image from the fused components.

[0003] Usually, iterative calculations are used to solve optimization problems using numerical methods to decompose and fuse images. Although this method has good results, it is computationally intensive and not conducive to real-time processing.

[0004] Existing methods often focus on the fusion of noise-free images. However, in real-world low-light conditions, visible light images are often noisy. Most existing methods lack pre-processing for denoising, and the fusion process fails to consider the impact of noise, resulting in poor fused image quality. Summary of the Invention

[0005] The purpose of the present invention is to provide a near-infrared image fusion method with large noise visible light and small computational complexity and good fused image quality while maintaining contrast.

[0006] The present invention provides a method for fusing near-infrared and high-noise visible light images while preserving contrast, and the specific steps are as follows:

[0007] (1) Pre-denoising: pre-denoising the noisy visible light image before image fusion;

[0008] (2) Fusion base layer: In the image fusion process, a low-frequency image with a contrast similar to NIR and an appearance similar to VIS is first generated and denoised again as the fused base layer;

[0009] (3) Fusion detail layer; then extract high-frequency components from VIS and NIR respectively; generate a fusion weight map based on local contrast, and adaptively weight the high-frequency components of VIS and NIR to form the fused detail layer;

[0010] (4) Final fusion: add the detail layer to the base layer and convert it to the color space required for display to obtain the final image fusion result.

[0011] Further:

[0012] In step (1), the pre-denoising process of the noisy visible light image is performed by using bilateral filtering for denoising, wherein the size of the filter kernel, the variance of the Gaussian filter kernel in the spatial domain and the range are configurable;

[0013] The visible light images before and after denoising are converted from RGB format to YUV format, and the separated brightness components are and

[0014] In step (2), the generation process of the base layer is as follows:

[0015] For each pixel in the image, do the following:

[0016] (a) Take the pixel value; suppose the pixel currently being processed is the i-th pixel in the image, then take the position of the i-th pixel as the center, and calculate the pixel value in the NIR image I nir and the denoised VIS brightness component Take a 9*9 neighborhood; arrange the pixel values ​​contained in the neighborhood into the following matrix form:

[0017]

[0018]

[0019] (b) Calculate the average contrast between VIS and NIR and fill it into a two-element vector

[0020]

[0021] Among them, avg means taking the average, R i I i Indicates taking a 9*9 neighborhood around the i-th pixel;

[0022] (c) Obtain the mapping parameters; First, generate an 81*81 diagonal matrix W, where each element on the diagonal corresponds to each position in the 9*9 window, and the element value is Where d is the square of the Euclidean distance from a certain position in the window to the center of the window; σ is a constant that is determined according to the actual situation, for example, it is set to 1.7. When σ is set to a small value, the image clarity and contrast are better, but brightness distortion may occur; otherwise, the contrast is slightly reduced, but the brightness is more realistic.

[0023] The calculated mapping parameters are:

[0024]

[0025] α i Contains two elements α i，1 and α i，2vector, μ c is a constant, determined according to the actual situation, for example, the value is 7500; μ c The smaller the value, the more the result tends to maintain the average contrast between VIS and NIR; μ c The larger the value, the more the result tends to maintain the appearance and brightness characteristics of VIS;

[0026] (d) Using the parameters obtained in (c), map the NIR pixel values:

[0027]

[0028] (e) Denoising; the base layer I generated in (d) d Perform a joint bilateral filtering. Spatial filtering uses a Gaussian filter kernel with configurable size and variance. The reference image for range filtering uses the result in (d) and the NIR, and the range also uses a Gaussian filter kernel with configurable variance.

[0029] The generation process of the detail layer in step (3) is as follows:

[0030] (a) Extracting high-frequency components: For the near-infrared image and the visible light image before denoising, low-pass filtering is performed using a 19*19 Gaussian filter kernel. The high-frequency components of each image are obtained by subtracting the filtered result from the filtered image.

[0031] (b) Calculate the local contrast of the near-infrared image and the denoised visible light image. For the pixel at position i, take the 5*5 neighborhood N(i) around it and calculate its local contrast as follows:

[0032]

[0033] Among them, max and min represent the maximum and minimum values ​​in the 5*5 neighborhood N(i) respectively; Indicates the calculation of gradient;

[0034] (c) Calculate the fusion weight; Based on the local contrast value obtained in process (3)(b), calculate the weights of the near-infrared image detail layer and the visible light image detail layer at position i, which are:

[0035]

[0036]

[0037] (d) According to the weights obtained in process (c), the high-frequency components of the visible light image and the near-infrared image are weighted and summed to obtain the fused detail layer.

[0038] The final fusion process in step (4) is as follows: the detail layer obtained in step (3) is added to the base layer obtained in step (2) to obtain the fused luminance component; the newly obtained luminance component and the original chrominance component constitute the fusion result in YUV format; according to display requirements, it can be converted back to RGB domain.

[0039] Features and advantages of the method of the present invention:

[0040] The present invention only needs to use pixels in a local neighborhood to process each pixel, without obtaining global information or iterative solution, which is conducive to hardware implementation and real-time processing.

[0041] The present invention adds different degrees of denoising processing before and during image fusion, which can better process visible light images with high noise levels under low light conditions and improve their signal-to-noise ratio.

[0042] The present invention combines the base layer and detail layer extraction algorithms to generate images with accurate content and rich details, achieving a good balance between preserving details and suppressing noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Flowchart illustrating the method of the present invention.

[0044] Figure 2 is a set of processing examples, including from left to right the visible light image (VIS), the near infrared image (NIR), and the output of this algorithm.

[0045] Figure 3 for Figure 2 The intermediate results generated during the calculation process of the example shown are: the VIS after pre-denoising, the base layer, the base layer after denoising, and the detail layer.

[0046] The numbers in the figure are: BF is bilateral filtering, CPM is contrast-preserving mapping, JBF is joint bilateral filtering, LC is calculating local contrast, and HPF is high-pass filtering. DETAILED DESCRIPTION

[0047] The details and results of the present invention are described in detail below with reference to examples, and the advantages of the present method are explained.

[0048] The algorithm flow chart of the present invention is as follows: Figure 1 As shown; a set of processing examples such as Figure 2 As shown in the figure, from left to right are the visible light image (VIS), the near infrared image (NIR) and the output of this algorithm; the intermediate results of the algorithm are shown in the figure. Figure 3 As shown, from left to right are the VIS after pre-denoising, the base layer, the base layer after denoising, and the detail layer.

[0049] Step 1: Pre-denoising

[0050] A bilateral filtering algorithm is used to denoise the input visible light image, mitigating the effects of noise on image decomposition. Bilateral filtering requires setting three parameters: the filter kernel size, the spatial Gaussian kernel variance, and the range Gaussian kernel variance. Larger filter kernels, smaller spatial Gaussian kernel variances, and smaller range Gaussian kernel variances result in more pronounced denoising but poorer edge preservation. Conversely, smaller values ​​result in poorer denoising but better edge preservation. In this example, a 9*9 filter kernel is used, the range Gaussian kernel variance is set to 50, and the spatial Gaussian kernel variance is set to 0 (i.e., a 9*9 mean filter kernel is used in the spatial domain).

[0051] The visible light images before and after denoising are converted from RGB format to YUV format, and the separated brightness components are and

[0052] Step 2: Generate the base layer

[0053] The base layer is generated using the denoised VIS brightness component and the NIR image. For each pixel in the image, the following processing is performed:

[0054] (a) Get pixel value. Assume that the pixel currently being processed is the i-th pixel in the image. Then, with the position of the i-th pixel as the center, the NIR image I nir and the denoised VIS brightness component Take a 9*9 neighborhood; arrange the pixel values ​​contained in the neighborhood into the following matrix form:

[0055]

[0056]

[0057] (b) Calculate the average contrast between VIS and NIR and fill it into a two-element vector

[0058]

[0059] Among them, avg means taking the average, R i I i Indicates taking a 9*9 neighborhood around the i-th pixel;

[0060] (c) Obtain the mapping parameters. First, generate an 81*81 diagonal matrix W, where each element on the diagonal corresponds to each position in the 9*9 window, and the element value is Where d is the square of the Euclidean distance from a position in the window to the window center; σ is a constant determined based on actual conditions. A smaller σ value results in better image clarity and contrast, but may cause brightness distortion. Conversely, a larger value slightly reduces contrast but produces more realistic brightness. After increasing σ to a certain level, further increases no longer significantly alter the image quality. Based on experiments, a setting of σ of 1.7 in this example achieves a good balance between contrast and visual realism.

[0061] To obtain the mapping parameter α i , the following objective function can be minimized:

[0062]

[0063] Among them, the first term is the appearance fidelity term, the second term is the contrast fidelity term, and the mapping parameter α i is the optimization variable to be solved. This optimization problem has an analytical solution:

[0064]

[0065] α i Contains two elements α i，1 and α i，2 vector, μ c is a constant and is determined according to actual conditions. c The smaller the value, the more the result tends to maintain the average contrast between VIS and NIR; μ c The larger the value, the more the result tends to maintain the appearance and brightness characteristics of VIS. c The value is 7500.

[0066] (d) Using the parameters obtained in (c), map the NIR pixel values:

[0067]

[0068] (e) Denoising. The base layer I generated in (d) d Perform a joint bilateral filtering. Spatial filtering uses a Gaussian filter kernel, and the size and variance of the filter kernel are configurable; the reference image of the domain filtering uses NIR and the result generated in (d), and the domain filtering also uses a Gaussian filter kernel with configurable variance. The larger the filter kernel, the smaller the variance of the spatial Gaussian kernel, and the smaller the variance of the domain Gaussian kernel, the more obvious the denoising effect, but the worse the edge preservation effect; conversely, the worse the denoising effect, but the better the edge preservation effect. In this example, the variance of the spatial domain filter kernel is configured as 0 (that is, the spatial domain uses a 9*9 mean filter kernel), and the variance of the domain Gaussian kernel is set to 50. That is: in this example, for a pixel p in the 9*9 neighborhood of pixel i, its weighting coefficient is:

[0069]

[0070] Here, g is a Gaussian weighted kernel with a variance of 50.

[0071] Step 3: Generate detail layer:

[0072] (a) Extracting high-frequency components. For the near-infrared image and the visible light image before denoising, low-pass filtering is performed using a 19*19 Gaussian filter kernel. The high-frequency components of each image are obtained by subtracting the filtered result from the image before filtering.

[0073] (b) Calculate the local contrast of the near-infrared image and the denoised visible light image. For the pixel at position i, take the 5*5 neighborhood N(i) around it and calculate its local contrast as follows:

[0074]

[0075] Among them, max and min represent the maximum and minimum values ​​in the 5*5 neighborhood N(i) respectively; In this example, the gradient is calculated by subtracting the values ​​of the pixels to the left and right of the current pixel and taking their absolute value. Then, the values ​​of the pixels above and below the current pixel are subtracted and their absolute value is taken. Finally, the two absolute values ​​are added together to form the gradient value.

[0076] (c) Calculate the fusion weight. Based on the local contrast value obtained in process (3)(b), calculate the weights of the near-infrared image detail layer and the visible light image detail layer at position i, which are:

[0077]

[0078]

[0079] (d) According to the weights obtained in process (c), the high-frequency components of the visible light image and the near-infrared image are weighted and summed to obtain the fused detail layer.

[0080] The final fusion process is as follows: the detail layer obtained in step (3) is added to the base layer obtained in step (2) to obtain the fused luminance component; the newly obtained luminance component and the original chrominance components form the fusion result in YUV format. It can be converted back to RGB domain according to display needs.

[0081] The specific results of this example are as follows Figure 2As shown in Figure 2, the VIS image contains color information but is noisy and lacks some image detail. The NIR image contains only the luminance channel but is less noisy. Furthermore, each image has some details that are superior to the other. After fusion using this algorithm, the resulting image combines useful information from both the VIS and NIR images (including the color and overall appearance of the VIS image, as well as their contrast and image detail) with less noise, resulting in a superior signal-to-noise ratio compared to both source images.

Claims

1. A method for fusing near-infrared and high-noise visible light images while preserving contrast, characterized in that: The specific steps are: (1) Pre-denoising: Before image fusion, the noisy visible light image is pre-denoised, specifically using bilateral filtering for denoising; and the visible light images before and after denoising are converted from RGB format to YUV format, and the separated brightness components are respectively and (2) Fusion base layer: In the image fusion process, a low-frequency image with a contrast similar to NIR and an appearance similar to VIS is first generated, and then denoised again to serve as the fused base layer. The specific process is as follows: For each pixel in the image, do the following: (a) Take the pixel value; suppose the pixel currently being processed is the i-th pixel in the image, then take the position of the i-th pixel as the center, and calculate the pixel value in the NIR image I nir and the denoised VIS brightness component Take a 9*9 neighborhood; arrange the pixel values ​​contained in the neighborhood into the following matrix form: (b) Calculate the average contrast between VIS and NIR and fill it into a two-element vector Among them, avg means taking the average, R i I i Indicates taking a 9*9 neighborhood around the i-th pixel; (c) Obtaining the mapping parameters: First, generate an 81*81 diagonal matrix W, where each element on the diagonal corresponds to each position in the 9*9 window, and the element value is Where d is the square of the Euclidean distance from a position in the window to the center of the window, and σ is a constant; the mapping parameters are obtained: α i Contains two elements α i，1 and α i，2 vector, μ c is a constant; (d) Using the parameters obtained in (c), map the NIR pixel values: (e) Denoising; the base layer I generated in (d) d Perform a joint bilateral filtering. Spatial filtering uses a Gaussian filter kernel with configurable size and variance. Range filtering uses the NIR and the result generated in (d) as reference images, and range filtering also uses a Gaussian filter kernel with configurable variance. (3) Fusion detail layer; then extract high-frequency components from VIS and NIR respectively; generate a fusion weight map based on local contrast, and adaptively weight the high-frequency components of VIS and NIR to form the fused detail layer; (4) Final fusion: add the detail layer to the base layer and convert it to the color space required for display to obtain the final image fusion result.

2. The method for fusion of near-infrared and high-noise visible light images with preserved contrast according to claim 1, characterized in that: The generation process of the detail layer in step (3) is as follows: (a) Extract high-frequency components; for near-infrared image I nir And the visible light image before denoising Use 19*19 Gaussian filter kernel to perform low-pass filtering respectively, and subtract the image before filtering from the result after filtering to obtain the respective high-frequency components; (b) For near-infrared image I nir And the denoised visible light image Calculate the local contrast separately; for the pixel at position i, take the 5*5 neighborhood N(i) around it and calculate its local contrast as follows: Among them, max and min represent the maximum and minimum values ​​in the 5*5 neighborhood N(i) respectively; Indicates the calculation of gradient; (c) Calculate the fusion weight; Based on the local contrast value obtained in process (3)(b), calculate the weights of the near-infrared image detail layer and the visible light image detail layer at position i, which are: (d) According to the weights obtained in process (c), the high-frequency components of the visible light image and the near-infrared image are weighted and summed to obtain the fused detail layer.

3. The method for fusion of near-infrared and high-noise visible light images with preserved contrast according to claim 2, characterized in that: The final fusion process in step (4) is as follows: the detail layer obtained in step (3) is added to the base layer obtained in step (2) to obtain the fused luminance component; the newly obtained luminance component and the original chrominance component constitute the fusion result in YUV format; Convert it back to RGB domain as required for display.

Images