A low-light video clarity processing method for highway scenes guided by vehicle headlight area

Abstract

The present invention discloses a method for processing low-light video clarity in highway scenes guided by vehicle headlight regions, comprising: extracting vehicle headlight positions in low-light input video frames and accumulating background texture information of the area in front of the vehicle headlights to obtain a background image (S1); weightedly fusing the input video frame and the background image to obtain a background information fused image (S2); extracting the direct light region in the fused image and calculating the ambient light illumination after masking the fused image (S3); and performing brightness mapping on the pixel values of the fused image based on the ambient light illumination (S4). Compared to traditional low-light clarity methods, the present invention fully considers the additional information introduced by the vehicle headlight region in highway scenes and its impact on ambient light estimation. It can effectively enhance the texture of dark areas in low-light video frames and suppress the glare effect of direct light. The brightness distribution is more consistent with human visual quality, and the method can be applied as an image preprocessing plug-in to highway video surveillance systems.

Description

Technical Field

[0001] The invention relates to a method for clear processing of low-illumination video of a highway scene guided by a vehicle light area. Background Art

[0002] For cost and safety reasons, highways lack streetlights. This results in severely insufficient ambient illumination at night, resulting in extremely weak reflected signals reaching surveillance camera sensors. This severely degrades image quality and reduces image visibility, causing significant inconvenience for surveillance managers. To ensure the proper operation of highway video surveillance systems in low-light conditions at night, researchers are working to employ image or video sharpening methods to mitigate low-light image degradation and enhance image clarity.

[0003] These image sharpening methods are primarily categorized as image enhancement and image restoration. Image enhancement methods include gamma correction, adaptive histogram equalization, and homomorphic filtering. These methods directly enhance the image's dynamic range, color richness, and edge sharpness, but they can easily over-enhance the image, which can be detrimental to human visual perception. Image restoration methods model specific imaging degradation mechanisms and use the degradation model to estimate characteristic parameters, thereby suppressing specific degradation factors and achieving a more natural sharpening effect. Low-light degradation modeling is based on retinal cortex theory, which posits that the pixel value of each point in a camera image is determined by the object's reflectivity and the incident ambient light. Therefore, by separating the illumination and reflectivity components of the image and adjusting the brightness of the illumination component, low-light video sharpening can be achieved.

[0004] In highway scenarios, the presence of passing vehicles, and the headlights of these vehicles acting as direct light sources, can significantly impact low-light restoration processing. These include: after brightness adjustment, the original direct light area will be diffused to varying degrees, resulting in a stronger glare effect; and the estimated light component in the area surrounding the headlights is too large, preventing the brightness of the area surrounding the headlights from being effectively improved, resulting in a darker image. However, in extreme scenarios where ambient light is extremely weak, some areas of the scene have virtually no information, and the visible light information from a single frame alone is insufficient to support the sharpening process. At this point, the headlights, acting as active light sources, can provide some scene information, such as lane markings, for subsequent sharpening processing. Summary of the Invention

[0005] In this context, the present invention uses the headlight area as a guide and proposes a method for clearing low-light videos in highway video surveillance scenarios. The introduction of the headlight prior makes the background texture of the low-light video clearing processing result richer and the brightness distribution more in line with the visual effect of the human eye, which is of great significance for highway video surveillance systems.

[0006] According to one aspect of the present invention, a method for processing low-light highway video clarity guided by vehicle headlight area is provided, which is characterized by comprising:

[0007] A) For a low-light video frame I, the vehicle headlight region position mask 1 is obtained by applying brightness, chroma, and saturation thresholds. The pixel values at positions where mask 1 is set in the current frame I are then weighted and accumulated into a background image back. After t frames of video, the background image back contains texture information at different locations in the scene.

[0008] B) performing pixel-level weighted fusion of the input low-light video frame I and the accumulated background image back, where the weights are calculated based on the pixel values at corresponding positions in the low-light video frame I, to obtain an image F containing the fused background information;

[0009] C) Obtain the direct light area position mask2 in the fused image F through the brightness threshold, and mask F according to mask2 to obtain F ′ , get F ′ The three color channels F in the neighborhood of each pixel r ′ (x),F g ′ (x),F b ′ The maximum value of (x) is used as the ambient light intensity at that point, and the weighted sum is performed with mask2(x) as the weight and 1 to obtain the rough ambient light estimation result U ′ , with F ′ is the reference image pair U ′ Perform guided filtering to obtain the final ambient lighting estimation image U;

[0010] D) According to the ambient light U, the pixel values of the fused image F are mapped to brighten the pixel values in the darker areas to obtain the final cleared image J. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 The present invention is a flowchart of a method for clear processing of low-light highway videos guided by vehicle headlight areas according to an embodiment of the present invention.

[0012] Figure 2(a) to Figure 2(d)Schematic diagram and rendering of background modeling using the headlight area as a guide according to an embodiment of the present invention; wherein, FIG2(a) shows the result of brightness stretching of a darker area in a low-light scene on a highway, where there is basically no information; FIG2(b) shows the background texture information in front of the headlights when a vehicle passes by in the same area, which is utilized by the method of the present invention; FIG2(c) shows the position of the headlight-illuminated area extracted by the method of the present invention based on the brightness, hue and saturation thresholds; and FIG2(d) is the cumulative background image obtained through multiple frames of continuous video.

[0013] Figures 3(a) and 3(b) are the ambient light map estimation results before and after improvement using the method according to the present invention; among them, Figure 3(a) does not use the method according to the present invention to process the direct light area, so in the illumination estimation result, the original direct light area spreads to the surrounding areas to a large extent, and these surrounding areas cannot effectively improve the brightness during the subsequent brightness mapping processing. Figure 3(b) is the ambient light estimation result of the method of the present invention, which can effectively suppress the influence of direct light.

[0014] Figure 4(a) to Figure 4(f) The experimental results of the vehicle headlight area guided low-light image sharpening method according to the present invention and the existing typical low-light image sharpening method on the test image are compared. FIG4( a ) shows an example of a low-light image in a highway scene. Figure 4(b) to Figure 4(e) 4( f ) shows an example of a low-light sharpened image obtained by the MF, DRBN, Zero-RDC, and LLFlow methods, respectively. FIG. 4( f ) shows an example of a sharpened image obtained by the method of the present invention. DETAILED DESCRIPTION

[0015] According to one aspect of the present invention, a method for sharpening low-light video in highway scenarios guided by vehicle headlight areas is provided. This method achieves sharpening of low-light video in highway scenarios, and the resulting sharpened image has the advantages of clear texture and significantly enhanced brightness.

[0016] To achieve clarity of low-light video, the present invention adopts the following technical solutions:

[0017] According to one aspect of the present invention, a method for processing low-light highway video clarity guided by vehicle headlight area is provided, which is characterized by comprising:

[0018] A) For a low-light video frame I, the vehicle headlight region position mask 1 is obtained by applying brightness, chroma, and saturation thresholds. The pixel values at positions where mask 1 is set in the current frame I are then weighted and accumulated into a background image back. After t frames of video, the background image back contains texture information at different locations in the scene.

[0019] B) performing pixel-level weighted fusion of the input low-light video frame I and the accumulated background image back, where the weights are calculated based on the pixel values at corresponding positions in the low-light video frame I, to obtain an image F containing the fused background information;

[0020] C) Obtain the direct light area position mask2 in the fused image F through the brightness threshold, and mask F according to mask2 to obtain F ′ , get F ′ The three color channels F in the neighborhood of each pixel r ′ (x),F g ′ (x),F b ′ The maximum value of (x) is used as the ambient light intensity at that point, and the weighted sum is performed with mask2(x) as the weight and 1 to obtain the rough ambient light estimation result U ′ , with F ′ is the reference image pair U ′ Perform guided filtering to obtain the final ambient lighting estimation image U;

[0021] D) According to the ambient light U, the pixel values of the fused image F are mapped to brighten the pixel values in the darker areas to obtain the final cleared image J.

[0022] According to a further embodiment of the present invention, in the above step A), the brightness, hue and saturation thresholds are used to obtain the illuminated position mask1 in front of the vehicle headlight:

[0023] mask1(x)=thre(Y(x))·thre(H(x))·thre(S(x)) (1)

[0024] Where x is the position index, mask1 is the extracted position information, 1 indicates that the point belongs to the illuminated area in front of the headlight, 0 indicates that it does not belong, thre indicates the thresholding operation, Y, H, and S represent the brightness, hue, and saturation component values of the original image, respectively. During initialization, the threshold range of Y is [10, 35], the threshold range of H is [65, 115], and the threshold range of S is [0, 0.5].

[0025] Then, the background image back is updated in a weighted manner using the pixel values where mask1 is 1 in the current frame I:

[0026]

[0027] Among them, back represents the accumulated background texture image, is the weight of the background update, which is determined by the brightness value of the background image at that point, and σ is the variance, which is set to 55 at initialization.

[0028] According to a further embodiment of the present invention, in the above step B), the input low-light video frame I is fused with information in the background image back in a weighted manner to obtain a fused image F:

[0029]

[0030] Among them, I is the low-light video frame, α is a hyperparameter controlling the texture supplement strength, which is set to 0.5 at initialization. is the background information fusion weight, indicating that the area with smaller brightness in the original video frame needs more background information to supplement it. σ is the variance, which is set to 6 during initialization.

[0031] According to a further embodiment of the present invention, in the above step C), the position mask2 of the direct light area in the fused image F is obtained by using the brightness threshold, and F is masked according to mask2 to obtain F ′ :

[0032]

[0033] Among them, mask2 is the extracted direct light area mask, and its value range is [0,1]. The closer it is to 1, the more likely the point is a direct light area. β and γ are parameters that control the shape of the curve and indirectly control the brightness threshold of the direct light area. They are set to 2 and 150 during initialization.

[0034] Next, get F ′ The maximum value of the three color channels in the neighborhood of each pixel is taken as the ambient light intensity of the point, and the weighted sum is performed with mask2(x) as the weight to obtain a rough ambient light estimation result U ′ , with F ′ is the reference image pair U ′ Perform guided filtering to obtain the final ambient lighting estimation image U:

[0035]

[0036] in, represents the maximum value of the three channels of the point, Ω is the square neighborhood centered at the point, which is initialized to the minimum side length of the input video frame * 0.02, and guide represents the guided filtering operation.

[0037] According to a further embodiment of the present invention, in the above step D), brightness mapping is performed on the pixel value of each point in the fused image F according to the ambient light U to obtain a sharpened image J:

[0038] J(x)=F(x)·log(U(x)) (6)

[0039] Wherein, log represents a log transformation function, and J is the final clarified result outputted by the present invention.

[0040] Therefore, the present invention proposes a method for sharpening low-light videos in highway scenarios guided by headlight areas. This sharpening method fully considers the additional background texture information provided by the headlights and their impact on ambient light estimation: during the sharpening process, the background texture of the discrete area in front of the headlights is modeled to supplement the background texture of the current frame; and before ambient light estimation, the area directly illuminated by the headlights is masked to prevent the glare effect from spreading throughout the scene. Therefore, the method of the present invention is suitable for application scenarios such as highways where headlights serve as active light sources. It can equalize the brightness distribution of low-light, degraded videos, significantly improving visual quality.

[0041] The low-light clarity processing method guided by vehicle lamp area provided by the present invention constitutes a substantial and significant improvement to the existing low-light clarity processing method.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1) Avoids the spread of glare effects, and the brightness distribution near direct light is more natural;

[0044] 2) The available information in the dark area is increased, making the edge texture of the area clearer;

[0045] 3) The sharpening results have better visual quality overall.

[0046] The following describes the embodiments of the present invention in detail.

[0047] In the present invention, the principles described are as follows:

[0048] The traditional image sharpening model is

[0049] I(x)=L(x)·R(x) (7)

[0050] Where x represents the pixel position index, I is the input image, and L and R reflect the illumination and reflectance components, respectively. For low-light images with insufficient ambient lighting, the illumination component is calculated and pixel values are remapped based on the illumination conditions at each point to balance the brightness distribution and achieve clarity.

[0051] However, in highway scenarios, due to the lack of streetlights, the ambient light is extremely low at night, and the visible light information in the weak signal areas of the scene is largely lost. Furthermore, the presence of direct light from vehicle headlights can cause deviations in the estimated illumination components in areas surrounding the direct light. Therefore, we have improved the traditional image generation model for low-light highway scenarios at night:

[0052] I(x)=U(x)·R(x)-T(x)+δ (8)

[0053] The present invention considers that under extremely low illumination conditions, the original texture information in the scene's weakest ambient light areas is largely lost, represented by the variable T. Furthermore, the presence of vehicle headlights causes the estimated illumination component to diffuse, represented by the variable U. The first term in the formula represents the scene's reflected signal, the second term represents the edge texture information lost in the extremely low ambient light areas, and the third term represents the noise introduced during the imaging phase of the device due to insufficient signal strength.

[0054] In the present invention, based on the model in formula (8), the method for processing low-light video clarity at night on a highway guided by the vehicle light area is as follows: Figure 1 The following parts are included (Note: Figure 1 (The original image shown in the figure is in color): First, extract the texture information of the local area illuminated in front of the headlights in each frame of the video, and accumulate this information in the time domain to obtain a background image. Second, use the background image to supplement the texture information of the area with extremely weak illumination in the current frame image to obtain a fused image. Third, extract the direct light area in the fused image and mask it, and then predict the scene ambient lighting component. Fourth, remap the pixel values of the fused image according to the ambient lighting component to obtain the final sharpening processing result. The following is a detailed description of the sharpening steps of the present invention in the form of accompanying drawings.

[0055] 1. Accumulate background information

[0056] At night, in highway scenes, due to the lack of streetlights, the ambient light in the scene is extremely weak, and visible light information is basically lost. As shown in Figure 2(a), the pixel value distribution in the red frame area is almost entirely zero, and it is no longer possible to restore effective information through sharpening processing. However, there are passing vehicles on the highway, and their headlights act as active light sources, illuminating the local area in front of the headlights and providing additional background texture information for this area, as shown in Figure 2(b). As time goes by, the continuous movement of vehicles also provides fragmented background texture information throughout the scene. The present invention uses this prior information of the headlights to accumulate the fragmented background texture information into a background image, corresponding to the T component in the formula.

[0057] The present invention first extracts the area illuminated in front of the headlights by constraining brightness, chroma, and saturation, and obtains the location information of the area:

[0058] mask1(x)=thre(Y(x))·thre(H(x))·thre(S(x)) (9)

[0059] Where x is the position index, mask1 is the extracted position information, 1 indicates that the point belongs to the illuminated area ahead of the headlights, 0 indicates that it does not, thre represents the thresholding operation, Y, H, and S represent the brightness, hue, and saturation component values of the original image, respectively. During initialization, the threshold range for Y is [10, 35], the threshold range for H is [65, 115], and the threshold range for S is [0, 0.5]. Figure 2(c) shows the position of the illuminated area ahead of the headlights extracted by the present invention.

[0060] Next, the texture information of the area in front of the headlights extracted from each frame of the continuous video is accumulated into the background image to obtain the T component in the formula. The background information is updated as follows:

[0061]

[0062] Where, back represents the accumulated background texture image, mask1 is the illuminated area in front of the headlight obtained in formula (9), is the weight of background update, which is determined by the brightness value of the background image at that point, and σ is the variance, which is set to 55 during initialization. It can be seen that the accumulated background image obtained by the present invention in Figure 2(d) contains the texture information of the background area.

[0063] 2. Integrate background information

[0064] For the current video frame, the visible light information in the extremely weak ambient light area of the scene has been basically lost and cannot be restored by image sharpening processing. In this invention, the texture information contained in the accumulated background image is used to supplement the texture of the weak ambient light area of the current frame to obtain the intermediate result F. The process can be expressed as:

[0065]

[0066] Among them, I is the low-light video frame, α is a hyperparameter controlling the texture supplement strength, which is set to 0.5 at initialization. is the background information fusion weight, indicating that the area with smaller brightness in the original video frame needs more background information to supplement it. σ is the variance, which is set to 6 during initialization.

[0067] 3. Predicting ambient lighting

[0068] In order to prevent direct light from affecting the scene ambient lighting estimation process, it is necessary to remove the direct light area before performing ambient lighting prediction on the video frame F after background texture addition. The specific method is to use the brightness threshold to extract the direct light area and set the pixel values in these areas to 0:

[0069]

[0070] Among them, mask2 is the extracted direct light area mask, and its value range is [0,1]. The closer it is to 1, the more likely the point is a direct light area. β and γ control the shape of the curve and indirectly control the brightness threshold of the direct light area. They are set to 2 and 150 during initialization. The smooth weight curve makes the brightness estimation value more consistent with the visual effect of the human eye.

[0071] Next, the maximum value in the neighborhood of each pixel in the preprocessed image is calculated as the preliminary ambient light estimation value of the point, and the light value of the direct light area is set to 1. Then, the original image is combined with the guided filtering to obtain the final light estimation result U:

[0072]

[0073] in, represents the maximum value of the three channels at that point, Ω is the square neighborhood centered at that point, initialized to the minimum side length of the input video frame * 0.02, and guide represents the guided filtering operation. As can be seen, the ambient lighting feature map estimated by our method in Figure 3(b) is more effective than the original estimate in Figure 3(a) in suppressing the spread of glare from direct light.

[0074] 4. Mapping ambient lighting

[0075] According to the estimated ambient light component U, the pixel values of the fused image F are brightness mapped. The log function brightens the areas with darker pixel values with greater intensity, making the scene brightness distribution tend to be balanced:

[0076] J(x)=F(x)·log(U(x)) (14)

[0077] Wherein, log represents a log transformation function, and J is the final clarified result outputted by the present invention.

[0078] FIG4 shows the comparison of the experimental results of the present invention's sharpening method on the test image after fusing background texture information and ambient light mapping with the existing low-light sharpening method (Note: Figure 4(a) to Figure 4(f) All the original images are in color. Figure 4(a) shows a low-light video frame in a highway scene. Figure 4(b)-Figure 4(e) Figure 4(f) shows the processing results of existing low-light sharpening methods, including MF, DRBN, Zero-RDC, and LLFlow, respectively. Figure 4(f) shows the sharpening results of the method of the present invention. It can be seen that the sharpened image obtained by the present invention has richer background textures and less glare, demonstrating that the present invention has better processing effects on low-light videos in highway scenes.

[0079] It should be noted that the above disclosure is only a specific implementation example of the present invention. According to the technical ideas provided by the present invention, any changes that can be conceived by those skilled in the art should fall within the scope of protection of the present invention.

Claims

1. A method for processing low-light video clarity in highway scenes guided by vehicle headlight area, characterized in that The steps include: A) For a low-light video frame I, the luminance, chroma, and saturation thresholds are used to obtain the illuminated position mask1 in front of the headlights. The pixel values at positions where mask1 is 1 in the current frame I are accumulated in a weighted manner into the background image back. After t frames of video, the background image back contains texture information at different locations in the scene. B) performing pixel-level weighted fusion of the input low-light video frame I and the accumulated background image back, where the weights are calculated based on the pixel values at corresponding positions in the low-light video frame I, to obtain an image F containing the fused background information; C) Obtain the direct light area position mask2 in the fused image F through the brightness threshold, and mask F according to mask2 to obtain F', and obtain the three color channels F' in the neighborhood of each pixel in F' r (x),F′ g (x),F′ b The maximum value of (x) is taken as the ambient light intensity of the point, and the weighted sum is performed with mask2(x) and 1 to obtain the rough ambient light estimation result U′. The guided filtering of U′ is performed with F′ as the reference image to obtain the final ambient light estimation image U; D) Mapping the pixel values of the fused image F according to the ambient light U, brightening the pixel values in the darker areas, and obtaining the final sharpened image J; in: The step A) includes: obtaining the illuminated position mask1 in front of the vehicle headlight using brightness, hue and saturation thresholds: mask1(x)=thre(Y(x))·thre(H(x))·thre(S(x)) (1) Among them, x is the position index, mask1 is the extracted position information, 1 indicates that the point belongs to the illuminated area in front of the headlight, 0 indicates that it does not belong to the illuminated area in front of the headlight, thre indicates the threshold operation, Y, H and S represent the brightness, hue and saturation component values of the original image respectively. At the time of initialization, the threshold range of Y is [10,35], the threshold range of H is [65,115], and the threshold range of S is [0,0.5]. Then, the background image back is updated in a weighted manner using the pixel values where mask1 is 1 in the current frame I: Among them, back represents the background image, is the weight of background update, which is determined by the brightness value of the background image at that point, and σ is the variance, which is set to 55 at initialization. The step B) includes: fusing the information in the background image back with the input low-light video frame I in a weighted manner to obtain a fused image F: Among them, I is the low-light video frame, α is a hyperparameter controlling the texture supplement strength, which is set to 0.5 at initialization. is the background information fusion weight, indicating that the area with smaller brightness in the original video frame needs more background information to supplement it. σ is the variance, which is set to 6 during initialization. The step C) includes: obtaining the direct light area position mask2 in the fused image F by using a brightness threshold, and masking F according to mask2 to obtain F′: Among them, mask2 is the extracted direct light area mask, and its value range is [0,1]. The closer to 1, the more likely the point is a direct light area. β and γ control the shape of the curve and indirectly control the brightness threshold of the direct light area. They are set to 2 and 150 during initialization. Next, obtain the maximum value of the three color channels in the neighborhood of each pixel in F′ as the ambient light intensity of the point, and perform weighted summation with mask2(x) as the weight to obtain a rough ambient light estimation result U′. Use F′ as the reference image to perform guided filtering on U′ to obtain the final ambient light estimation image U: in, Indicates the maximum value of the three channels of the point, Ω is the square neighborhood centered at the point, which is initialized to the minimum side length of the input video frame * 0.02, and guide represents the guided filtering operation. The step D) includes: performing brightness mapping on the pixel value of each point in the fused image F according to the ambient light U to obtain a sharpened image J: J(x)=F(x)·log(U(x)) (6) Wherein, log represents the log transformation function, and J is the final output of the low-illumination video sharpening processing method for highway scenes guided by the vehicle headlight area.

2. The method for processing low-light video clarity of highway scenes guided by vehicle headlight area according to claim 1, characterized in that: In steps A), B), and C), the additional information introduced by the headlight area is used to perform background modeling to supplement the dark area background texture of the current video frame to be processed. Then, the direct light area of the headlight is masked when estimating the ambient light illumination, effectively suppressing the spread of the glare effect and clarifying the image with a balanced brightness distribution and better visual effects.

Images