A method, device, electronic device and storage medium for adaptive enhancement of low-light-level night vision color images

Abstract

The present invention discloses a method, device, electronic device, and storage medium for adaptive enhancement of low-light-level night vision color images. The method utilizes a multi-level decomposition method to decompose night vision visible light images to extract high-frequency and low-frequency information. For the low-frequency information, an adaptive enhancement network is constructed to extract background texture features of the low-frequency image. For the high-frequency information, a color mapping method is used to convert the RGB image into an HSV image, which is then normalized to obtain enhanced high-frequency image detail information. The processed high-frequency and low-frequency images are reconstructed to form an enhanced night vision visible light image. The above method overcomes the problem that existing technical solutions are not suitable for enhancing night vision dark scene images. It can effectively improve the brightness of blind spots and extreme environments generated during nighttime autonomous driving, improve image quality, and thus enhance the safety level of nighttime autonomous driving.

Description

Technical Field

[0001] The present invention relates to the field of vehicle automatic driving technology, and in particular to a method, device, electronic device and storage medium for adaptively enhancing low-light-level night vision color images. Background Art

[0002] In recent years, accidents involving autonomous vehicles at night have become frequent, posing significant risks to personal safety. The primary causes of these accidents are, on the one hand, a failure to maintain a safe distance; on the other hand, blind spots and extreme weather conditions during nighttime driving prevent timely vehicle detection. Consequently, autonomous vehicles at night can become dangerous "executioners." Therefore, for vehicles driving at night in fog, rain, and blind spots, the dim light sources present cannot effectively identify scene features, necessitating the use of image processing algorithms to enhance additional scene information.

[0003] For night vision image enhancement, the existing solution is to determine the critical grayscale value of halation in the night vision visible light image, invert the night vision halation image, estimate the initial transmittance of the inverted image, and then construct an adaptive transmittance function based on the image's initial transmittance and the critical grayscale value of halation. Finally, the adaptive transmittance function is used to adaptively enhance images with different halation levels. However, this method amplifies the noise and halation area in the image, resulting in a lack of effective information enhancement, making it unsuitable for enhancing images with dark backgrounds at night. Therefore, for the bright and dark areas in visible light images, a method is needed to adaptively enhance the image without oversaturation. Summary of the Invention

[0004] The purpose of the present invention is to provide a method, device, electronic device and storage medium for adaptively enhancing low-light-level night vision color images. This method overcomes the problem that existing technical solutions are not suitable for enhancing night vision dark scene images. It can effectively improve the brightness of blind spots and extreme environments generated during nighttime autonomous driving, improve image quality, and thus improve the safety level of nighttime autonomous driving.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for adaptively enhancing a low-light-level night vision color image, the method comprising:

[0007] Step 1: Decompose the night vision visible light image using a multi-level decomposition method to extract high-frequency information and low-frequency information; wherein the high-frequency information contains image details and texture features, corresponding to components with high grayscale values ​​in the image; the low-frequency information contains image contour features, corresponding to background components in the image;

[0008] Step 2: For low-frequency information, an adaptive enhancement network is constructed to extract background texture features of low-frequency images;

[0009] Step 3: For high-frequency information, use the color mapping method to convert the RGB image into an HSV image, and then normalize it to obtain enhanced high-frequency image detail information;

[0010] Step 4: reconstruct the processed high-frequency image and low-frequency image to form an enhanced night vision visible light image.

[0011] A low-light-level night vision color image adaptive enhancement device, the device comprising:

[0012] An image decomposition unit is used to decompose the night vision visible light image using a multi-level decomposition method to extract high-frequency information and low-frequency information; wherein the high-frequency information contains image details and texture features, corresponding to components with high grayscale values ​​in the image; the low-frequency information contains image contour features, corresponding to background components in the image;

[0013] A low-frequency information processing unit is used to construct an adaptive enhancement network to extract background texture features of low-frequency images;

[0014] A high-frequency information processing unit is used to convert the RGB image into an HSV image using a color mapping method, and then normalize it to obtain enhanced high-frequency image detail information;

[0015] The image reconstruction unit is used to reconstruct the processed high-frequency image and low-frequency image to form an enhanced night vision visible light image.

[0016] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method described in an embodiment of the present invention.

[0017] A computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the method described in an embodiment of the present invention.

[0018] It can be seen from the technical solution provided by the above-mentioned present invention that the above-mentioned method overcomes the problem that the existing technical solution is not suitable for enhancing night vision dark scene images, can effectively improve the brightness of blind spots and extreme environments generated during nighttime autonomous driving, improve image quality, and thus improve the safety level of nighttime autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic flow chart of a method for adaptively enhancing a color image with low-light-level night vision provided by an embodiment of the present invention;

[0021] Figure 2 Schematic diagram of the model framework for adaptive enhancement of night vision visible light images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments, and do not constitute a limitation of the present invention. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0023] like Figure 1 FIG2 is a flow chart of a method for adaptively enhancing a color image in low-light-level night vision according to an embodiment of the present invention. The method includes:

[0024] Step 1: Decompose the night vision visible light image using a multi-level decomposition method to extract high-frequency information and low-frequency information;

[0025] The high-frequency information includes the details and texture features of the image, which correspond to the components with high grayscale values ​​in the image; the low-frequency information includes the contour features of the image, which correspond to the background components in the image;

[0026] In this step, the night vision visible light image P is input, and the output image q is obtained by filtering the guide image I. On the local window w, there is a local linear relationship between the guide image I and the output image q:

[0027]

[0028] i is the pixel value of image I; w k is a local window centered at pixel k; a k and b k is the local window w k The linear coefficient within can be estimated by introducing a minimization function;

[0029] At the same time, on window w, the output image q and the input image P have the following relationship:

[0030]

[0031] This linear relationship ensures that in each local window w k If there is an edge in the guided graph I, the output image q will keep the edge unchanged; at the same time, the output image q should be as identical as possible to the input image P to reduce the information loss caused by filtering. The least squares expression of the algorithm is:

[0032]

[0033] This is the problem of finding the optimal value. A regularization parameter ε is introduced to prevent a k Too large, and the loss function is obtained:

[0034]

[0035] Use the least squares method to solve the minimum value, and use the derivative of the minimum value to be 0. The solution process is as follows:

[0036] w| is the number of pixels in window w, and it can be derived that:

[0037]

[0038] Right now:

[0039]

[0040] u k and σ k are the local windows w in image I k The mean and variance of is the local window w k The mean value of all pixels in ;

[0041] For edge preservation, when I = p, that is, when the input image P and the guide image I are the same image, the algorithm becomes an edge preservation filter, and the solution of the equation is expressed as follows:

[0042]

[0043] Among them, ε is equivalent to the threshold that defines the smooth area and the edge area, which is used as the guide for the edge-preserving smoothing filter of the image;

[0044] After filtering, the image is decomposed into low-frequency and high-frequency parts. Because discrete wavelet transform can greatly eliminate the correlation between different extracted features by selecting appropriate filters, the high-frequency wavelet coefficients in the image contain overall details. Therefore, the filtered image is decomposed using discrete wavelet transform to obtain high-frequency and low-frequency images. The discrete wavelet transform decomposition process is as follows:

[0045] Given a separable two-dimensional image signal f(x,y), the low-frequency component obtained by the scaling and conversion function is:

[0046]

[0047] Among them, 0≤i≤255, 0≤j≤255, W A(m,n) is the low-frequency component (also called the approximation coefficient); h(i,j) is the low-pass filter coefficient; m and n are the coordinates of the low-frequency component; this formula represents the convolution operation of the original image signal with the low-pass filter, and the low-frequency component is obtained by downsampling;

[0048] The high frequency components obtained are:

[0049] Horizontal high-frequency component:

[0050]

[0051] Among them, W H(m,n) is the horizontal high-frequency component; g_H(i,j) is the horizontal high-pass filter coefficient;

[0052] Vertical high-frequency component:

[0053]

[0054] Among them, W V(m,n) is the vertical high-frequency component; g_V(i,j) is the vertical high-pass filter coefficient;

[0055] Diagonal high-frequency components:

[0056]

[0057] Among them, W D(m,n) is the diagonal high-frequency component; g_D(i,j) is the diagonal high-pass filter coefficient.

[0058] Step 2: For low-frequency information, an adaptive enhancement network is constructed to extract background texture features of low-frequency images;

[0059] In this step, for low-frequency images, since the human eye is most sensitive to green, second most sensitive to red, and least sensitive to blue, different weights are used to obtain a reasonable adaptive grayscale image. The specific values ​​are:

[0060]

[0061] I R Represents the R (red) channel of the image; I G Represents the G (green) channel of the image; I B Represents the B (blue) channel of the image; (x, y) represents the pixel value of the image; W (x,y) Represents the low-frequency component of the image; Indicates the brightness value of the image output;

[0062] Solve for the average value of the image input brightness, expressed as:

[0063]

[0064] Represents the average value of the input brightness logarithm; N represents the total number of pixels; δ is a minimum value, which is used to avoid numerical overflow when performing log calculations on pure black pixels, a very common problem in image processing;

[0065]

[0066] L g (x,y) represents the logarithmic ratio of the output; L W (x, y) represents the brightness value of the input image; L wmax Represents the maximum value of the input image brightness value pair;

[0067]

[0068] L min (x,y) represents the minimum grayscale value of the input image; L max (x,y) represents the maximum grayscale value of the input image; L(x,y) represents the result of image adaptive output.

[0069] Step 3: For high-frequency information, use the color mapping method to convert the RGB image into an HSV image, and then normalize it to obtain enhanced high-frequency image detail information;

[0070] In this step, for high-frequency images, the filtered image is converted from RGB to HSV image using color mapping, specifically:

[0071]

[0072] R, G, B represent the high-frequency image components W ( ' x,y) The red, green and blue channels of the image are as follows; H, S and V represent the high-frequency image components W after conversion. ( 'x,y) Hue, saturation, and brightness; max(R,G,B) is the maximum value of each channel in the (R,G,B) three channels; min(R,G,B) is the minimum value of each channel in the (R,G,B) three channels;

[0073] Normalize the converted HSV image and perform the high-frequency image component W ( ' x,y) After normalization, we get K:

[0074]

[0075] K is the normalized value of the image after color mapping.

[0076] Step 4: reconstruct the processed high-frequency image and low-frequency image to form an enhanced night vision visible light image.

[0077] It should be noted that the contents not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art.

[0078] Based on the above method, an embodiment of the present invention further provides a device for adaptively enhancing a low-light-level night vision color image, the device comprising:

[0079] An image decomposition unit is used to decompose the night vision visible light image using a multi-level decomposition method to extract high-frequency information and low-frequency information; wherein the high-frequency information contains image details and texture features, corresponding to components with high grayscale values ​​in the image; the low-frequency information contains image contour features, corresponding to background components in the image;

[0080] A low-frequency information processing unit is used to construct an adaptive enhancement network to extract background texture features of low-frequency images;

[0081] A high-frequency information processing unit is used to convert the RGB image into an HSV image using a color mapping method, and then normalize it to obtain enhanced high-frequency image detail information;

[0082] The image reconstruction unit is used to reconstruct the processed high-frequency image and low-frequency image to form an enhanced night vision visible light image.

[0083] The specific implementation means of each unit in the above device can be found in the method embodiment.

[0084] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method described in the embodiment of the present invention.

[0085] An embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the method described in the embodiment of the present invention.

[0086] like Figure 2 Figure 2 shows a schematic diagram of the model framework for adaptive enhancement of night vision visible light images according to an embodiment of the present invention. To further verify the effectiveness of the method described herein, the following comparative example evaluates the night vision visible light image enhancement results of the present invention. Due to changes in image color and structure and the lack of a full reference image for comparison, this example uses the no-reference evaluation criteria LOE (lightness order error) and NIQE (natural image quality evaluator) to evaluate the enhanced image. It should be noted that these metrics can only reflect certain aspects of image quality and are not completely consistent with the evaluation results given by the human visual system.

[0087] The LOE evaluation metric is the sequential brightness difference of an image, which is used to judge the illumination change of an image by evaluating the sequential change of the brightness of the image in the neighborhood. The NIQE evaluation metric is a non-reference image evaluation metric used to evaluate the quality of natural images. It extracts image features such as structure, contrast, and color distribution and uses a machine learning algorithm to train a model to predict the image quality score. Table 1 below compares the LOE and NIQE evaluation metrics of five enhancement algorithms in the prior art and the method described in the embodiment of the present invention:

[0088] Table 1

[0089]

[0090] From the results in Table 1, it can be seen that compared with the other five algorithms, the method of the present invention has lower LOE and NIQE indicators, which shows that the use of the adaptive enhancement method has a better effect on enhancing night vision images.

[0091] In summary, the enhancement experiments and result analysis of different night vision scene images show that the adaptive enhancement method of the present invention can significantly improve the brightness and clarity of night vision images, and the evaluation indicators show that compared with different algorithms, the image index data after enhancement using the algorithm of the present invention is relatively good, which fully verifies the universality of the adaptive enhancement algorithm proposed in the present invention in processing different images, thereby providing accurate detection for target identification under nighttime security monitoring, providing obstacle avoidance for dark targets in the blind spots of nighttime automatic driving systems, and providing a safe environment with rich background information and prominent target features for nighttime automatic driving under the fusion system.

[0092] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. A method for adaptively enhancing color images for low-light-level night vision, characterized in that: The method comprises: Step 1: Decompose the night vision visible light image using a multi-level decomposition method to extract high-frequency information and low-frequency information; wherein the high-frequency information contains image details and texture features, corresponding to components with high grayscale values ​​in the image; the low-frequency information contains image contour features, corresponding to background components in the image; The process of step 1 is specifically as follows: Input night vision visible light image P, filter it through guide image I, and get output image q. On the local window w, there is a local linear relationship between guide image I and output image q: q i =a k I i +b k , i is the pixel value of image I; w k is a local window centered at pixel k; a k and b k is the local window w k The linear coefficient within is estimated by introducing a minimization function; At the same time, on window w, the output image q and the input image P have the following relationship: q i =p i -n i , n i is the noise value at pixel i; This linear relationship ensures that in each local window w k If there is an edge in the guided graph I, the output image q will keep the edge unchanged; at the same time, the output image q should be as identical as possible to the input image P to reduce the information loss caused by filtering. The least squares expression of the algorithm is: This is the problem of finding the optimal value. A regularization parameter ε is introduced to prevent a k Too large, and the loss function is obtained: Use the least squares method to solve the minimum value, and use the derivative of the minimum value to be 0. The solution process is as follows: |w| is the number of pixels in window w, and we can deduce that: Right now: u k and σ k are the local windows w in image I k The mean and variance of is the local window w k The mean value of all pixels in ; For edge preservation, when I = p, that is, when the input image P and the guide image I are the same image, the solution of the equation is expressed as follows: Among them, ε is equivalent to the threshold that defines the smooth area and the edge area, which is used as the guide for the edge-preserving smoothing filter of the image; The filtered image is decomposed using discrete wavelet transform to obtain high-frequency image and low-frequency image. The discrete wavelet transform decomposition process is as follows: Given a separable two-dimensional image signal f(x,y), the low-frequency component obtained by the scaling and conversion function is: Among them, 0≤i≤255, 0≤j≤255, W A(m,n) is the low-frequency component; h(i,j) is the low-pass filter coefficient; m and n are the coordinates of the low-frequency component; Formula (9) represents the convolution operation of the original image signal with the low-pass filter, and the low-frequency component is obtained by downsampling; The high frequency components obtained are: Horizontal high-frequency component: Among them, W H(m,n) is the horizontal high-frequency component; g_H(i,j) is the horizontal high-pass filter coefficient; Vertical high-frequency component: Among them, W V(m,n) is the vertical high-frequency component; g_V(i,j) is the vertical high-pass filter coefficient; Diagonal high-frequency components: Among them, W D(m,n) is the diagonal high-frequency component; g_D(i,j) is the diagonal high-pass filter coefficient; Step 2: For low-frequency information, an adaptive enhancement network is constructed to extract background texture features of low-frequency images; Step 3: For high-frequency information, use the color mapping method to convert the RGB image into an HSV image, and then normalize it to obtain enhanced high-frequency image detail information; Step 4: reconstruct the processed high-frequency image and low-frequency image to form an enhanced night vision visible light image.

2. The method for adaptively enhancing low-light-level night vision color images according to claim 1, characterized in that: In step 2, for low-frequency images, different weights are used to obtain reasonable adaptive grayscale images. The specific values ​​are: I R Represents the R (red) channel of the image; I G Represents the G (green) channel of the image; I B Represents the B (blue) channel of the image; (x, y) represents the pixel value of the image; W (x,y) Represents the low-frequency component of the image; Indicates the brightness value of the image output; Solve for the average value of the image input brightness, expressed as: Represents the average value of the input brightness logarithm; N represents the total number of pixels; δ is the minimum value, which is used to avoid numerical overflow when performing log calculation on pure black pixels; L g (x,y) represents the logarithmic ratio of the output; L W (x, y) represents the brightness value of the input image; L wmax Represents the maximum value of the input image brightness value pair; L min (x,y) represents the minimum grayscale value of the input image; L max (x,y) represents the maximum grayscale value of the input image; L(x,y) represents the result of image adaptive output.

3. The method for adaptively enhancing low-light-level night vision color images according to claim 1, characterized in that: In step 3, for the high-frequency image, the filtered image is converted from RGB to HSV image using color mapping, specifically: Among them, R, G, B represent the high-frequency image components W′ (x,y) The red, green and blue channels of the image are as follows; H, S and V represent the high-frequency image components W′ after conversion. (x,y) Hue, saturation, and brightness; max(R,G,B) is the maximum value of each channel in the (R,G,B) three channels; min(R,G,B) is the minimum value of each channel in the (R,G,B) three channels; Normalize the converted HSV image and perform the high-frequency image component W′ (x,y) After normalization, we get K: K is the normalized value of the image after color mapping.

4. A low-light-level night vision color image adaptive enhancement device, characterized in that: The device comprises: An image decomposition unit is used to decompose the night vision visible light image using a multi-level decomposition method to extract high-frequency information and low-frequency information; wherein the high-frequency information contains image details and texture features, corresponding to components with high grayscale values ​​in the image; the low-frequency information contains image contour features, corresponding to background components in the image; The specific processing process of the image decomposition unit is: Input night vision visible light image P, filter it through guide image I, and get output image q. On the local window w, there is a local linear relationship between guide image I and output image q: q i =a k I i +b k , i is the pixel value of image I; w k is a local window centered at pixel k; a k and b k is the local window w k The linear coefficient within is estimated by introducing a minimization function; At the same time, on window w, the output image q and the input image P have the following relationship: q i =p i -n i , n i is the noise value at pixel i; This linear relationship ensures that in each local window w k If there is an edge in the guided graph I, the output image q will keep the edge unchanged; at the same time, the output image q should be as identical as possible to the input image P to reduce the information loss caused by filtering. The least squares expression of the algorithm is: This is the problem of finding the optimal value. A regularization parameter ε is introduced to prevent a k Too large, and the loss function is obtained: Use the least squares method to solve the minimum value, and use the derivative of the minimum value to be 0. The solution process is as follows: |w| is the number of pixels in window w, and we can deduce that: Right now: u k and σ k are the local windows w in image I k The mean and variance of is the local window w k The mean value of all pixels in ; For edge preservation, when I = p, that is, when the input image P and the guide image I are the same image, the solution of the equation is expressed as follows: Among them, ε is equivalent to the threshold that defines the smooth area and the edge area, which is used as the guide for the edge-preserving smoothing filter of the image; The filtered image is decomposed using discrete wavelet transform to obtain high-frequency image and low-frequency image. The discrete wavelet transform decomposition process is as follows: Given a separable two-dimensional image signal f(x,y), the low-frequency component obtained by the scaling and conversion function is: Among them, 0≤i≤255, 0≤j≤255, W A(m,n) is the low-frequency component; h(i,j) is the low-pass filter coefficient; m and n are the coordinates of the low-frequency component; Formula (9) represents the convolution operation of the original image signal with the low-pass filter, and the low-frequency component is obtained by downsampling; The high frequency components obtained are: Horizontal high-frequency component: Among them, W H(m,n) is the horizontal high-frequency component; g_H(i,j) is the horizontal high-pass filter coefficient; Vertical high-frequency component: Among them, W V(m,n) is the vertical high-frequency component; g_V(i,j) is the vertical high-pass filter coefficient; Diagonal high-frequency components: Among them, W D(m,n) is the diagonal high-frequency component; g_D(i,j) is the diagonal high-pass filter coefficient; A low-frequency information processing unit is used to construct an adaptive enhancement network to extract background texture features of low-frequency images; A high-frequency information processing unit is used to convert the RGB image into an HSV image using a color mapping method, and then normalize it to obtain enhanced high-frequency image detail information; The image reconstruction unit is used to reconstruct the processed high-frequency image and low-frequency image to form an enhanced night vision visible light image.

5. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 3.

6. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 3.

Images