Visible light and near-infrared image fusion method based on illumination difference and reflection characteristics

Abstract

The invention provides a visible light near-infrared image fusion method based on illumination difference and reflection characteristics, which is used for night video monitoring so as to provide a high-quality, clear and rich-detail image fusion effect. According to the method, firstly, the overall brightness of visible light and near-infrared images is calculated, and self-adaptive alignment is carried out according to the difference of the overall brightness of visible light and near-infrared images; then, based on a visible light and near-infrared imaging model, visible light obtains a reflection image through Retinex, and near-infrared light obtains a reflection image through estimation according to the attenuation rule of brightness along with the square of the distance; and obtaining a near-infrared effective brightness weight and a near-infrared high-reflectivity weight according to the aligned image pair and the reflection image pair, and further obtaining a brightness weight map needing to be fused through brightness and reflection decomposition. And a visible light and near-infrared image pair fused with texture information is obtained through three-layer texture fusion. And finally, performing region weight fusion on the texture fusion result according to the brightness weight map to obtain a final fusion result.

Description

Technical Field

[0001] The invention relates to a visible light and near infrared image fusion method based on illumination difference and reflection characteristics, and belongs to the field of computer vision. Background Art

[0002] Video surveillance is a technology that relies on cameras to capture video and image information and automatically processes and intelligently analyzes it. It is widely used in many fields such as security, traffic management, and industrial monitoring. Through all-weather, multi-angle real-time monitoring, video surveillance technology can significantly reduce labor costs and improve monitoring efficiency. With the development of technologies such as artificial intelligence, the application scenarios of video surveillance are gradually expanding and showing new application prospects in all walks of life. At present, most intelligent video surveillance systems rely on visible light images for recognition and monitoring. Visible light images can well reflect scene details under good lighting conditions, but the effect decreases significantly when the lighting changes. Especially at night and in low-light environments, the image quality captured by visible light cameras is poor, the outline and details of objects are difficult to identify, and even after image signal processing (ISP) optimization, there may be information missing. This shortcoming of visible light imaging puts higher requirements on video surveillance systems, especially in low-light environments, and other spectral means are needed to improve image quality and enhance monitoring performance. Near-infrared imaging technology does not rely on the intensity of visible light, so it is suitable for working at night and in low-light environments. Near-infrared cameras can effectively fill in the light in dark environments, presenting clear target details and texture information, which may be lost in traditional visible light images. By fusing near-infrared images with visible light images, video surveillance systems can obtain richer information. In actual outdoor surveillance applications, fusion algorithms can improve the accuracy and adaptability of video surveillance, make up for the shortcomings of existing single imaging systems in complex environments, and provide strong technical support for all-weather monitoring.

[0003] However, for nighttime video surveillance in actual scenarios, the existing visible light and near-infrared image fusion algorithms cannot meet the needs, and there are mainly the following problems:

[0004] 1) Fusion without considering the difference between visible light and near-infrared imaging leads to blurred texture: Visible light imaging depends on ambient light sources and image signal processing (ISP) is used to improve image quality, but its effect depends on the current lighting conditions. When the light is insufficient, the visible light image will become blurred. Near-infrared imaging mainly relies on the camera's fill light to provide light, so the information quality is limited by the fill light distance. When the target exceeds the fill light range, the image quality drops significantly. If this difference between visible light and near-infrared imaging is not fully considered during the image fusion process, the result will often contain blurred visible light information or near-infrared information with insufficient fill light, resulting in unclear details and textures in the fused image.

[0005] 2) Failure to consider the regional characteristics of visible and near-infrared images simultaneously leads to color distortion: Objects in the scene have different reflection and absorption characteristics for visible and near-infrared spectra. For example, some objects may have low reflectivity in visible light and high reflectivity in near-infrared light, and vice versa. If these differences in reflectivity characteristics are not fully considered during the image fusion process, the visible light spectrum information in some areas of the fusion result may be distorted, causing color deviation. Summary of the invention

[0006] In view of the above problems existing in the existing visible light and near-infrared image fusion algorithms in night video surveillance scenes, the present invention proposes a visible light and near-infrared image fusion algorithm based on illumination difference and reflection characteristics, which includes: firstly, by calculating the overall brightness of different channels of visible light and near-infrared images, the overall brightness alignment guided by near-infrared is realized, and the effective brightness weight of near-infrared is obtained from the aligned visible light and near-infrared images, and then the reflection map is extracted according to different imaging models of visible light and near-infrared to obtain the high reflection weight of near-infrared, and then the brightness weight map is obtained by brightness and reflection decomposition, and finally the texture fusion unrelated to the reflection characteristics and the regional fusion with spectral characteristics maintained are realized. Specifically including:

[0007] In view of the color cast problem in the fusion result, this paper proposes a method for aligning the overall brightness of visible light and near-infrared with brightness consistency constraints, which actively senses the ambient light and realizes the adaptive alignment of the brightness of visible light guided by near-infrared. The overall brightness of the three channels of visible light and near-infrared is sensed. For scenes with higher overall brightness of visible light, the original brightness is maintained. Otherwise, the three channels of visible light are mapped to the overall brightness value of near-infrared to realize scene-adaptive brightness alignment.

[0008] In order to achieve texture fusion and color preservation in local areas, the present invention proposes a regional complementary fusion based on spectral response characteristics, which effectively fuses texture information while maintaining regional spectral characteristics. For texture information, selective fusion is performed based on texture richness; for regional brightness information, reflectance maps are estimated based on different imaging models for visible light and near infrared, and brightness weight maps are obtained to maintain visible light brightness in areas with large differences in reflectance characteristics and fuse near infrared brightness in other areas.

[0009] According to one aspect of the present invention, a visible light and near infrared image fusion method based on illumination difference and reflection characteristics is provided, which is characterized by comprising the following steps:

[0010] A) By calculating the overall brightness of the three visible light channels and the near-infrared image, the overall brightness of the three visible light channels is mapped to the overall brightness of the near-infrared image to achieve overall brightness alignment;

[0011] B) Using different imaging models for visible light and near infrared, a visible light reflection map is obtained by Retinex, and then a near infrared reflection map is obtained by estimating the fill light intensity;

[0012] C) extracting near-infrared effective brightness weight from the aligned visible light and near-infrared images, and then obtaining near-infrared high reflection weight from the reflection map of visible light in the near-infrared image. Decomposing the near-infrared effective brightness weight and the near-infrared high reflection weight by brightness and reflection, thereby obtaining the brightness weight map to be fused;

[0013] D) The aligned visible light and near-infrared images are subjected to texture fusion of three-layer texture extraction to obtain the initial fusion result, and then the initial fusion result is subjected to regional weight fusion according to the brightness weight map to obtain the final fusion result.

[0014] in:

[0015] The overall brightness alignment described in step A) comprises the following steps:

[0016] A1) Apply adaptive local tone mapping ALTM to the original visible light image to generate the corresponding gain map,

[0017] A2) Calculate the average brightness difference between the visible light image and the near infrared image to obtain an adjustment coefficient, and use the adjustment coefficient to correct the overall brightness of the visible light so that the overall brightness of the visible light and near infrared images are aligned. The adjustment coefficient is calculated as follows:

[0018]

[0019] where k {r,g,b} Indicates the adjustment coefficient of the three-channel graph, I {r,g,b} is the three-channel image of the visible light image, NIR is the near infrared image, L e represents the gain map obtained by ALTM, N represents the total number of pixels of the image for which the summation is performed, i represents the number of the current pixel,

[0020] A3) Apply the adjustment coefficient to the gain map to brighten the visible light image, thereby achieving brightness alignment. The brightening method is as follows:

[0021]

[0022] Where I′ {r,g,b} represents the visible light image after overall brightness alignment,

[0023] Step B) comprises the following steps:

[0024] B1) For the case of visible light ambient illumination, the brightness of the visible light is provided by the ambient illumination, and its imaging model is:

[0025] I′{r,g,b} =L·R {r,g,b}

[0026] Where L represents the visible light environment illumination, R {r,g,b} Represents the three-channel reflectance map of visible light,

[0027] For near-infrared fill light illumination, the near-infrared brightness is provided by the fill light, and its imaging model is:

[0028] NIR=L n ·T n ·R n

[0029] L n Indicates the overall near-infrared fill light, T n Indicates the transmittance of near-infrared light, R n represents the near-infrared reflectance map,

[0030] B2) The maximum value of the three channels of the visible light image is used as the estimated result of the visible light illumination, and then the visible light reflection map is obtained by Retinex.

[0031] The overall fill light intensity of the near infrared image is the average of the brightest 1% pixels in the near infrared image. The overall fill light intensity is used as the reference value of the camera's closest position, and the relative distance between other pixels and the camera is estimated by the brightness ratio:

[0032]

[0033] Where d is the relative distance between each pixel in the image and the camera, 0 is the assumed distance between the reference point and the camera, and d 0 Set to GuidF is the guided filter: Guided filtering is applied to NIR, and the transmittance T is obtained based on the square decay law of light intensity with distance. n for:

[0034]

[0035] Then we get the near infrared reflection map:

[0036]

[0037] Step C) comprises the following steps:

[0038] C1) From the aligned visible light and near infrared images, the near infrared effective brightness weight is obtained according to the brightness ratio of the near infrared. That is, the weight of the near infrared region with higher brightness than the visible light is higher, and the weight of the invalid region of the near infrared due to insufficient fill light is smaller. The brightness ratio of the near infrared is determined as follows:

[0039]

[0040] Where W i is the effective brightness weight of near infrared,

[0041] From the reflection graphs of visible light and near infrared, according to the proportion of near infrared reflection graph, we can get the high reflection weight of near infrared compared to visible light:

[0042]

[0043] Where W r It is the high reflection weight of near infrared.

[0044] C2) The effective brightness weight of near infrared includes the difference between brightness and reflection information, and the high reflection weight of near infrared only includes the difference in reflection information. By decomposing brightness and reflection, the brightness weight to be fused can be obtained to generate a brightness fusion weight map. The generation method is as follows:

[0045]

[0046] The distribution map of W is the brightness weight map for fusion.

[0047] Step D) comprises the following steps:

[0048] D1) A three-layer guided filtering method is used, namely high-frequency layer, medium-frequency layer and low-frequency layer, to extract the texture information of visible light and near-infrared images, and introduce a texture enhancement coefficient in each layer fusion to enhance the texture performance. The texture information of each layer fusion is calculated by the following formula:

[0049]

[0050] in is the visible light image of the previous layer, is the near infrared image of the previous layer, GuidF represents the guided filtering operation, Express Layer applies guided filtering, λ i is the texture enhancement coefficient of the i-th layer, and the fusion result of texture information is:

[0051]

[0052] Where D i is the texture information of the i-th layer, F {r,g,b} is the visible light texture fusion result, F n is the near infrared texture fusion result,

[0053] D2) Based on the generated brightness weight map, the brightness difference areas are fused, and the fusion formula is:

[0054] F=F {r,g,b} (1-W)+F n ·W

[0055] Where F is the final fusion result map, and W is the aforementioned brightness weight map. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 (a)- Figure 1 (d) are the visible light images and near-infrared images processed by ISP at night outdoors and indoors.

[0057] Figure 2 (a) and Figure 2 (b) is the imaging model of nighttime visible light images and near-infrared light images.

[0058] Figure 3 It is a flowchart of the brightness area weight map generation.

[0059] Figure 4 It is the flow chart of the fusion algorithm. DETAILED DESCRIPTION

[0060] A flowchart of a method for fusing visible light and near infrared images based on illumination difference and reflection characteristics according to an embodiment of the present invention is as follows: Figure 3 and Figure 4 As shown, it includes the following steps:

[0061] A) By calculating the overall brightness of the three visible light channels and the near-infrared image, the overall brightness of the three visible light channels is mapped to the overall brightness of the near-infrared image to achieve overall brightness alignment;

[0062] B) Using different imaging models for visible light and near infrared, a visible light reflection map is obtained by Retinex, and then a near infrared reflection map is obtained by estimating the fill light intensity;

[0063] C) extracting near-infrared effective brightness weight from the aligned visible light and near-infrared images, and then obtaining near-infrared high reflection weight from the reflection map of visible light in the near-infrared image. Decomposing the near-infrared effective brightness weight and the near-infrared high reflection weight by brightness and reflection, thereby obtaining the brightness weight map to be fused;

[0064] D) The texture fusion of the three-layer texture extraction of the visible light and near-infrared images after alignment is performed to obtain the initial fusion result, and then the initial fusion result is subjected to regional weight fusion according to the brightness weight map to obtain the final fusion result.

[0065] A) Visible light and near-infrared brightness align

[0066] The present invention is based on automatic perception of the brightness of visible light and near infrared, and adaptively enhances the brightness of the original visible light image, thereby aligning the overall brightness of visible light and near infrared. The depth of field and ambient lighting differences in different scenes lead to different brightness distributions of visible light and near infrared, such as Figure 1 (a) and Figure 1 (b) shows the visible light and near-infrared images respectively. As shown in the outdoor near-infrared, the overall brightness of the visible light is higher due to the lack of distant fill light. Figure 1 (c) and Figure 1 In the visible light and near-infrared images of (d), the overall brightness of the near-infrared in the room is higher. By calculating the overall brightness of the visible light and the near-infrared, it can be determined whether the near-infrared needs to be used to guide the visible light for brightening. When the overall brightness of the visible light is higher, the original brightness is maintained, and when the visible light brightness is lower, the overall brightness of the visible light is aligned to be consistent with the near-infrared.

[0067] In scenarios where brightness alignment is required, adaptive local tone mapping (ALTM) is first applied to the original visible light image to generate the corresponding gain map. Then, the adjustment coefficient is obtained by calculating the average brightness difference between the visible light image and the near-infrared image to correct the overall brightness of the visible light. The adjustment coefficient is calculated as follows:

[0068]

[0069] where k {r,g,b} Indicates the adjustment coefficient of the three-channel graph, I {r,g,b} is the three-channel image of the visible light image, NIR is the near infrared image, L e represents the gain map obtained by ALTM, N represents the number of image pixels, and i represents the position of each pixel. Finally, the adjustment coefficient is applied to the gain map to brighten the visible light image. The overall brightness of the three channels after brightening can be kept consistent with the overall brightness of the near-infrared, thereby achieving brightness alignment. The brightening method is as follows:

[0070]

[0071] Among them I { ' r,g,b} Represents the visible light image after brightness mapping.

[0072] B) Reflection map generation

[0073] Figure 2 (a) and Figure 2(b) The visible light and near-infrared imaging models respectively show that visible light and near-infrared images have different brightness sources and lighting directions. Therefore, it is necessary to construct corresponding reflection models for different lighting and reflection characteristics. Specifically, the brightness of visible light images is mainly affected by ambient light. Based on the Retinex reflection model, the visible light reflectivity after illumination restoration can be extracted. Near-infrared images mainly rely on the near-infrared fill light of the camera, and its fill light effect decays with the distance between the object and the camera. Different imaging models can be obtained from different imaging processes:

[0074] I′ {r,g,b} =L·R {r,g,b}

[0075] NIR=L n ·T n ·R n

[0076] Where L represents the visible light environment illumination, R {r,g,b} Represents the three-channel reflection map of visible light, L n Indicates the overall near-infrared fill light, T n Indicates the transmittance of near-infrared light, R n Represents the near-infrared reflectance map.

[0077] The visible light illumination is estimated by the maximum value of the three channels, and then the visible light reflection map is obtained. The overall fill light intensity of the near infrared image is the average of the brightest 1% pixels in the near infrared image. The overall fill light intensity is used as a reference value for the camera's closest position, and the relative distance between other pixels and the camera is estimated by the brightness ratio:

[0078]

[0079] Where d is the relative distance between each pixel in the image and the camera, 0 Assume that the distance from the reference point is d 0 Set to GuidF is the guided filter. Therefore, the transmittance can be estimated based on the square decay law of light intensity with distance, and the transmittance T n for:

[0080]

[0081] Then we can get the near infrared reflectance map:

[0082]

[0083] C) Brightness weight map generation

[0084] The difference between the original image pairs reflects the changes in illumination and reflection characteristics, while the reflectance map only represents the reflection characteristics of the object. The near-infrared effective brightness weight and near-infrared high reflection weight can be obtained:

[0085]

[0086] In order to maintain the spectral characteristics of the object, the area with greater reflection difference should maintain the reflection characteristics of the original visible light, and the area with brightness difference should use brighter and clearer near-infrared images to reduce the noise of the visible light image. The near-infrared effective brightness weight contains brightness information, but it is also affected by reflection information. The brightness weight map can be generated by the near-infrared effective brightness weight and the near-infrared high reflection weight. The generation method is as follows:

[0087]

[0088] Where W is the brightness weight map for regional fusion.

[0089] C) Fusion image generation

[0090] Texture information can be directly extracted from the image itself, and fusion can be achieved by comparing the texture intensity. To this end, the present invention adopts a three-layer guided filtering method, namely a high-frequency layer, a medium-frequency layer, and a low-frequency layer, to extract the texture information of visible light and near-infrared images, and introduces a texture enhancement coefficient in each layer of fusion to enhance the texture performance. The texture information of each layer of fusion is calculated by the following formula:

[0091]

[0092] in is the visible light image of the previous layer, is the near infrared image of the previous layer, GuidF represents the guided filtering operation, λ i is the texture enhancement coefficient of the i-th layer, and the texture enhancement coefficients of the high-frequency layer, medium-frequency layer, and low-frequency layer are set to 1.2, 1.5, and 1.8 respectively. This paper not only fuses texture information into visible light images, but also fuses it into near-infrared images to ensure that texture information will not be lost in the subsequent regional reflection characteristic fusion process. The fusion result of texture information is:

[0093]

[0094] Where D i is the texture information of the i-th layer, F {r,g,b} is the visible light texture fusion result, F n This is the near-infrared texture fusion result.

[0095] Based on the texture information fusion of visible light and near-infrared images, a visible light and near-infrared fused image with rich texture details can be obtained. However, in the fusion result, the structural information of near-infrared light is clearer, while the visible light image still has certain noise. Therefore, based on the generated brightness weight map, the brightness difference area can be effectively fused while maintaining the spectral characteristics of the object, thereby significantly improving the signal-to-noise ratio of the image. The specific fusion formula is:

[0096] F=F {r,g,b} (1-W)+F n ·W

[0097] Where F is the final fusion result map, and W is the aforementioned brightness weight map.

[0098] Advantages and / or beneficial effects of the present invention include:

[0099] 1) The present invention fully considers the different imaging characteristics of visible light and near-infrared light, and selectively fuses spectrally independent texture information based on texture intensity, thereby avoiding the influence of the blurring effect of the fusion result caused by the insufficient near-infrared fill light area.

[0100] 2) For spectrally related brightness information, the present invention utilizes the difference in reflective characteristics between visible light and near-infrared, retains the original spectral characteristics of visible light in areas with significant differences in reflective characteristics, and achieves effective fusion of near-infrared brightness in other brightness areas.

[0101] 3) The method of the present invention not only significantly improves the regional structure clarity of the fused image, but also effectively avoids the color distortion problem caused by differences in reflection characteristics.

Claims

1. A visible light and near infrared image fusion method based on illumination difference and reflection characteristics, characterized in that The following steps are involved: A) By calculating the overall brightness of the three visible light channels and the near-infrared image, the overall brightness of the three visible light channels is mapped to the overall brightness of the near-infrared image to achieve overall brightness alignment; B) Using different imaging models for visible light and near infrared, a visible light reflection map is obtained by Retinex, and then a near infrared reflection map is obtained by estimating the fill light intensity; C) extracting near-infrared effective brightness weight from the aligned visible light and near-infrared images, and then obtaining near-infrared high reflection weight from the reflection map of visible light in the near-infrared image. Decomposing the near-infrared effective brightness weight and the near-infrared high reflection weight by brightness and reflection, thereby obtaining the brightness weight map to be fused; D) The aligned visible light and near-infrared images are subjected to texture fusion of three-layer texture extraction to obtain the initial fusion result, and then the initial fusion result is subjected to regional weight fusion according to the brightness weight map to obtain the final fusion result. in: The overall brightness alignment described in step A) comprises the following steps: A1) Apply adaptive local tone mapping ALTM to the original visible light image to generate the corresponding gain map, A2) Calculate the average brightness difference between the visible light image and the near infrared image to obtain an adjustment coefficient, and use the adjustment coefficient to correct the overall brightness of the visible light so that the overall brightness of the visible light and near infrared images are aligned. The adjustment coefficient is calculated as follows: where k {r,g,b} Indicates the adjustment coefficient of the three-channel graph, I {r,g,b} is the three-channel image of the visible light image, NIR is the near infrared image, L e represents the gain map obtained by ALTM, N represents the total number of pixels of the image for which the summation is performed, i represents the number of the current pixel, A3) Apply the adjustment coefficient to the gain map to brighten the visible light image, thereby achieving brightness alignment. The brightening method is as follows: Where I′ {r,g,b} represents the visible light image after overall brightness alignment, Step B) comprises the following steps: B1) For the case of visible light ambient illumination, the brightness of the visible light is provided by the ambient illumination, and its imaging model is: I′ {r,g,b} =L·R {r,g,b} Where L represents the visible light environment illumination, R {r,g,b} Represents the three-channel reflectance map of visible light, For near-infrared fill light illumination, the near-infrared brightness is provided by the fill light, and its imaging model is: NIR=L n ·T n ·R n L n Indicates the overall near-infrared fill light, T n Indicates the transmittance of near-infrared light, R n represents the near-infrared reflectance map, B2) The maximum value of the three channels of the visible light image is used as the estimated result of the visible light illumination, and then the visible light reflection map is obtained by Retinex. The overall fill light intensity of the near infrared image is the average of the brightest 1% pixels in the near infrared image. The overall fill light intensity is used as the reference value of the camera's closest position, and the relative distance between other pixels and the camera is estimated by the brightness ratio: Where d is the relative distance between each pixel in the image and the camera, and d0 is the assumed distance between the reference point and the camera. Set d0 to GuidF is the guided filter: Guided filtering is applied to NIR, and the transmittance T is obtained based on the square decay law of light intensity with distance. n for: Then we get the near infrared reflection map: Step C) comprises the following steps: C1) From the aligned visible light and near infrared images, the near infrared effective brightness weight is obtained according to the brightness ratio of the near infrared. That is, the weight of the near infrared region with higher brightness than the visible light is higher, and the weight of the invalid region of the near infrared due to insufficient fill light is smaller. The brightness ratio of the near infrared is determined as follows: Where W i is the effective brightness weight of near infrared, From the reflection graphs of visible light and near infrared, according to the proportion of near infrared reflection graph, we can get the high reflection weight of near infrared compared to visible light: Where W r It is the high reflection weight of near infrared. C2) The effective brightness weight of near infrared includes the difference between brightness and reflection information, and the high reflection weight of near infrared only includes the difference in reflection information. By decomposing brightness and reflection, the brightness weight to be fused can be obtained to generate a brightness fusion weight map. The generation method is as follows: The distribution map of W is the brightness weight map for fusion. Step D) comprises the following steps: D1) A three-layer guided filtering method is used, namely high-frequency layer, medium-frequency layer and low-frequency layer, to extract the texture information of visible light and near-infrared images, and introduce a texture enhancement coefficient in each layer fusion to enhance the texture performance. The texture information of each layer fusion is calculated by the following formula: in is the visible light image of the previous layer, is the near infrared image of the previous layer, GuidF represents the guided filtering operation, Express Layer applies guided filtering, λ i is the texture enhancement coefficient of the i-th layer, and the fusion result of texture information is: Where D i is the texture information of the i-th layer, F {r,g,b} is the visible light texture fusion result, F n is the near infrared texture fusion result, D2) Based on the generated brightness weight map, the brightness difference areas are fused, and the fusion formula is: F=F {r,g,b} ·(1-W)+F n ·W Where F is the final fusion result map, and W is the aforementioned brightness weight map.

2. The visible light and near infrared image fusion method based on illumination difference and reflection characteristics according to claim 1 is characterized in that: The texture enhancement coefficients of the high-frequency layer, medium-frequency layer, and low-frequency layer are set to 1.2, 1.5, and 1.8, respectively.

Images