Low-light night vision color image adaptive enhancement method and device, electronic equipment and storage medium

Abstract

The invention discloses a low-light night vision color image adaptive enhancement method and device, electronic equipment and a storage medium, and the method comprises the steps: decomposing a night vision visible light image through employing a multi-level decomposition method, and extracting high-frequency information and low-frequency information; for low-frequency information, constructing an adaptive enhancement network to extract background texture features of a low-frequency image; for the high-frequency information, converting an RGB image into an HSV image by using a color mapping method, and carrying out normalization processing on the image to obtain enhanced high-frequency image detail information; and reconstructing the processed high-frequency image and low-frequency image to form an enhanced night vision visible light image. According to the method, the problem that the existing technical scheme is not suitable for enhancing the night vision dark scene image is solved, the brightness of a blind area generated during automatic driving at night and the brightness of an extreme environment can be effectively improved, the image quality is improved, and the safety level of automatic driving at night is improved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle automatic driving technology, and in particular to a method, device, electronic device and storage medium for adaptively enhancing a low-light-level night vision color image. Background Art

[0002] In recent years, self-driving vehicles have often been involved in accidents while driving at night, causing great harm to personal safety. The main reasons for these accidents are that on the one hand, the vehicle does not have a sufficient safety distance; on the other hand, the blind spots and extreme weather during night driving cause the vehicle to be unable to perceive in time. As a result, self-driving cars at night have become dangerous "executioners". Therefore, for vehicles driving at night, in fog or rain and night vision blind spots, the weak light source that appears cannot effectively identify scene features, and image processing algorithms are needed to enhance additional information features in the scene.

[0003] For night vision image enhancement processing, the existing technology is to determine the critical grayscale value of the halo of the night vision visible light image, invert the night vision halo image, estimate the initial transmittance of the inverted image, and then construct an adaptive transmittance function based on the initial transmittance of the image and the critical grayscale value of the halo. Finally, the images with different halo degrees are adaptively enhanced according to the adaptive transmittance function. However, this method amplifies the noise and halo area in the image, resulting in a weak effect of improving effective information, and is not suitable for enhancing images with dark backgrounds at night. Therefore, for the bright and dark areas in the visible light image, it is necessary to propose a method that allows the image to be adaptively enhanced without oversaturation. Summary of the invention

[0004] The purpose of the present invention is to provide a method, device, electronic device and storage medium for adaptively enhancing low-light-level night vision color images. The method overcomes the problem that the prior art solutions are not suitable for enhancing night vision dark scene images, and can effectively improve the brightness of blind spots and extreme environments generated during nighttime autonomous driving, improve image quality, and thus improve the safety level of nighttime autonomous driving.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A method for adaptively enhancing a low-light-level night vision color image, the method comprising:

[0007] Step 1: Decompose the night vision visible light image using a multi-level decomposition method to extract high-frequency information and low-frequency information; wherein the high-frequency information includes the details and texture features of the image, corresponding to the components with high grayscale values ​​in the image; the low-frequency information includes the contour features of the image, corresponding to the background components in the image;

[0008] Step 2: For low-frequency information, an adaptive enhancement network is constructed to extract background texture features of low-frequency images;

[0009] Step 3: For high-frequency information, use the color mapping method to convert the RGB image into an HSV image, and then normalize it to obtain enhanced high-frequency image detail information;

[0010] Step 4: reconstruct the processed high-frequency image and low-frequency image to form an enhanced night vision visible light image.

[0011] A low-light-level night vision color image adaptive enhancement device, the device comprising:

[0012] An image decomposition unit is used to decompose the night vision visible light image using a multi-level decomposition method to extract high-frequency information and low-frequency information; wherein the high-frequency information includes image details and texture features, corresponding to components with high grayscale values ​​in the image; the low-frequency information includes image contour features, corresponding to background components in the image;

[0013] A low-frequency information processing unit is used to construct an adaptive enhancement network to extract background texture features of low-frequency images;

[0014] A high-frequency information processing unit is used to convert the RGB image into the HSV image by using a color mapping method, and then normalize the image to obtain enhanced high-frequency image detail information;

[0015] The image reconstruction unit is used to reconstruct the processed high-frequency image and low-frequency image to form an enhanced night vision visible light image.

[0016] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method described in an embodiment of the present invention.

[0017] A computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the method described in an embodiment of the present invention.

[0018] It can be seen from the technical solution provided by the above-mentioned present invention that the above-mentioned method overcomes the problem that the existing technical solution is not suitable for enhancing night vision dark scene images, can effectively improve the brightness of blind spots and extreme environments generated during nighttime autonomous driving, improve image quality, and thus improve the safety level of nighttime autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0020] Figure 1 A schematic diagram of a process flow of a method for adaptively enhancing a low-light-level night vision color image provided by an embodiment of the present invention;

[0021] Figure 2 Schematic diagram of the model framework for adaptive enhancement of night vision visible light images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] like Figure 1 The figure is a schematic flow chart of a method for adaptively enhancing a low-light-level night vision color image provided by an embodiment of the present invention, the method comprising:

[0024] Step 1: Decompose the night vision visible light image using a multi-level decomposition method to extract high-frequency information and low-frequency information;

[0025] The high-frequency information includes the details and texture features of the image, which correspond to the components with high grayscale values ​​in the image; the low-frequency information includes the contour features of the image, which correspond to the background components in the image;

[0026] In this step, the night vision visible light image P is input, and the output image q is obtained by filtering the guide image I. On the local window w, there is a local linear relationship between the guide image I and the output image q:

[0027]

[0028] i is the pixel value of image I; w k is a local window centered at pixel k; a k and b k is the local window w k The linear coefficient within can be estimated by introducing a minimization function;

[0029] At the same time, on window w, the output image q and the input image P have the following relationship:

[0030]

[0031] This linear relationship ensures that in each local window w k There is an edge in the guided graph I, and the output image q will keep the edge unchanged; at the same time, the output image q should be as similar as possible to the input image P to reduce the information loss caused by filtering. The least squares expression of the algorithm is:

[0032]

[0033] This is a problem of finding the optimal value. A regularization parameter ε is introduced to prevent a k Too large, and the loss function is:

[0034]

[0035] Use the least squares method to solve the minimum value, and use the derivative of the minimum value to be 0. The solution process is as follows:

[0036] w| is the number of pixels in window w, and it can be derived that:

[0037]

[0038] Right now:

[0039]

[0040] u k With σ k are the local windows w in image I k The mean and variance of is the local window w k The mean value of all pixels in ;

[0041] For edge preservation, when I = p, that is, when the input image P and the guide image I are the same image, the algorithm becomes an edge preservation filter, and the solution of the equation is expressed as follows:

[0042]

[0043] Among them, ε is equivalent to the threshold that defines the smooth area and the edge area, which is used as the guide for edge-preserving smoothing filtering of the image;

[0044] After filtering, the image is decomposed into low-frequency and high-frequency parts. Because discrete wavelet transform can greatly eliminate the correlation between different extracted features by selecting appropriate filters, the high-frequency wavelet coefficients in the image contain overall details. Therefore, the filtered image is decomposed using discrete wavelet transform to obtain high-frequency and low-frequency images. The discrete wavelet transform decomposition process is:

[0045] Given a separable two-dimensional image signal f(x,y), the low-frequency component obtained by the scaling and transformation function is:

[0046]

[0047] Among them, 0≤i≤255, 0≤j≤255, W A(m,n) is the low-frequency component (also called the approximate coefficient); h(i,j) is the low-pass filter coefficient; m and n are the coordinates of the low-frequency component; this formula means that the original image signal is convolved with the low-pass filter and the low-frequency component is obtained by downsampling;

[0048] The high frequency components obtained are:

[0049] Horizontal high frequency component:

[0050]

[0051] Among them, W H(m,n) is the horizontal high-frequency component; g_H(i,j) is the horizontal high-pass filter coefficient;

[0052] Vertical high frequency component:

[0053]

[0054] Among them, W V(m,n) is the vertical high-frequency component; g_V(i,j) is the vertical high-pass filter coefficient;

[0055] Diagonal high frequency components:

[0056]

[0057] Among them, W D(m,n) is the diagonal high frequency component; g_D(i,j) is the diagonal high pass filter coefficient.

[0058] Step 2: For low-frequency information, an adaptive enhancement network is constructed to extract background texture features of low-frequency images;

[0059] In this step, for low-frequency images, since the human eye is most sensitive to green, second to red, and least sensitive to blue, different weights are used to obtain a reasonable adaptive grayscale image. The specific values ​​are:

[0060]

[0061] I R Represents the R (red) channel of the image; I G Represents the G (green) channel of the image; I B represents the B (blue) channel of the image; (x, y) represents the pixel value of the image; W (x,y) Represents the low-frequency component of the image; Indicates the brightness value of the image output;

[0062] Solve for the average value of the image input brightness, expressed as:

[0063]

[0064] Represents the average value of the input brightness logarithm; N represents the total number of pixels; δ is a minimum value, which is used to avoid numerical overflow when performing log calculation on pure black pixels. This problem is very common in image processing;

[0065]

[0066] L g (x, y) represents the logarithmic ratio of the output; L W (x, y) represents the brightness value of the input image; L wmax Represents the maximum value of the input image brightness value pair;

[0067]

[0068] L min (x, y) represents the minimum grayscale value of the input image; L max (x, y) represents the maximum grayscale value of the input image; L(x, y) represents the result of image adaptive output.

[0069] Step 3: For high-frequency information, use the color mapping method to convert the RGB image into an HSV image, and then normalize it to obtain enhanced high-frequency image detail information;

[0070] In this step, for high-frequency images, the filtered image is converted from RGB to HSV image using color mapping, specifically:

[0071]

[0072] R, G, B represent the high-frequency image components W ( ' x,y) The red, green and blue channels of the image are as follows; H, S and V represent the high-frequency image components W after conversion. ( 'x,y) Hue, saturation, and brightness; max(R,G,B) is the maximum value of each of the three channels (R,G,B); min(R,G,B) is the minimum value of each of the three channels (R,G,B);

[0073] The converted HSV image is normalized and the high-frequency image component W is ( ' x,y) After normalization, we get K:

[0074]

[0075] K is the normalized value of the image after color mapping.

[0076] Step 4: reconstruct the processed high-frequency image and low-frequency image to form an enhanced night vision visible light image.

[0077] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field.

[0078] Based on the above method, an embodiment of the present invention further provides a low-light-level night vision color image adaptive enhancement device, the device comprising:

[0079] An image decomposition unit is used to decompose the night vision visible light image using a multi-level decomposition method to extract high-frequency information and low-frequency information; wherein the high-frequency information includes image details and texture features, corresponding to components with high grayscale values ​​in the image; the low-frequency information includes image contour features, corresponding to background components in the image;

[0080] A low-frequency information processing unit is used to construct an adaptive enhancement network to extract background texture features of low-frequency images;

[0081] A high-frequency information processing unit is used to convert the RGB image into the HSV image by using a color mapping method, and then normalize the image to obtain enhanced high-frequency image detail information;

[0082] The image reconstruction unit is used to reconstruct the processed high-frequency image and low-frequency image to form an enhanced night vision visible light image.

[0083] The specific implementation means of each unit in the above device can be found in the method embodiment.

[0084] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method described in the embodiment of the present invention.

[0085] An embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the method described in the embodiment of the present invention.

[0086] like Figure 2 The figure shows a schematic diagram of the model framework of the adaptive enhancement of night vision visible light images according to an embodiment of the present invention. In order to further verify the effectiveness of the method according to the present invention, the night vision visible light image enhancement results of the present invention are evaluated with a specific comparative example. Since the color and structure of the image change and there is no full reference image for comparison, this example uses the non-reference evaluation standard LOE (lightness order error) and NIQE (Natural image quality evaluator) indicators to evaluate the enhanced image. It should be noted that these indicators can only reflect certain aspects of image quality and are not completely consistent with the evaluation results given by the human visual system.

[0087] The LOE evaluation index is the brightness order difference of the image, which judges the illumination change of the image by evaluating the sequential change process of the brightness of the image in the neighborhood; the NIQE evaluation index is a reference-free image evaluation index used to evaluate the quality of natural images. It extracts image features such as structure, contrast, color distribution, etc., and uses a machine learning algorithm to train the model to predict the quality score of the image. The following Table 1 compares the five enhancement algorithms in the prior art with the method described in the embodiment of the present invention in terms of LOE and NIQE evaluation indicators:

[0088] Table 1

[0089]

[0090] From the results in Table 1, it can be seen that compared with the other five algorithms, the method of the present invention has relatively low LOE and NIQE indicators, which shows that the use of the adaptive enhancement method has a better effect on the enhancement of night vision images.

[0091] In summary, the enhancement experiments and result analysis on different night vision scene images show that the adaptive enhancement method of the present invention can significantly improve the brightness and clarity of night vision images, and the evaluation indicators show that compared with different algorithms, the image index data after enhancement using the algorithm of the present invention is relatively good, which fully verifies the universality of the adaptive enhancement algorithm proposed in the present invention in processing different images, which is beneficial to provide accurate detection for target identification under nighttime security monitoring, to provide obstacle avoidance for dark targets in the blind spots of nighttime automatic driving systems, and to provide a safe environment with rich background information and prominent target features for nighttime automatic driving under a fusion system.

[0092] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A method for adaptively enhancing a low-light-level night vision color image, characterized in that: The method comprises: Step 1: Decompose the night vision visible light image using a multi-level decomposition method to extract high-frequency information and low-frequency information; wherein the high-frequency information includes the details and texture features of the image, corresponding to the components with high grayscale values ​​in the image; the low-frequency information includes the contour features of the image, corresponding to the background components in the image; Step 2: For low-frequency information, an adaptive enhancement network is constructed to extract background texture features of low-frequency images; Step 3: For high-frequency information, use the color mapping method to convert the RGB image into an HSV image, and then normalize it to obtain enhanced high-frequency image detail information; Step 4: reconstruct the processed high-frequency image and low-frequency image to form an enhanced night vision visible light image.

2. The method for adaptively enhancing low-light-level night vision color images according to claim 1, characterized in that: The process of step 1 is specifically as follows: Input night vision visible light image P, filter through guide image I, and get output image q. On local window w, there is a local linear relationship between guide image I and output image q: i is the pixel value of image I; w k is a local window centered at pixel k; a k and b k is the local window w k The linear coefficients within are estimated by introducing a minimization function; At the same time, on window w, the output image q and the input image P have the following relationship: This linear relationship ensures that in each local window w k There is an edge in the guided graph I, and the output image q will keep the edge unchanged; at the same time, the output image q should be as similar as possible to the input image P to reduce the information loss caused by filtering. The least squares expression of the algorithm is: This is a problem of finding the optimal value. A regularization parameter ε is introduced to prevent a k Too large, and the loss function is: Use the least squares method to solve the minimum value, and use the derivative of the minimum value to be 0. The solution process is as follows: w| is the number of pixels in window w, and it can be derived that: Right now: u k With σ k are the local windows w in image I k The mean and variance of is the local window w k The mean value of all pixels in ; For edge preservation, when I = p, that is, when the input image P and the guide image I are the same image, the solution of the equation is expressed as follows: Among them, ε is equivalent to the threshold that defines the smooth area and the edge area, which is used as the guide for edge-preserving smoothing filtering of the image; The filtered image is decomposed using discrete wavelet transform to obtain high-frequency image and low-frequency image. The discrete wavelet transform decomposition process is: Given a separable two-dimensional image signal f(x,y), the low-frequency component obtained by the scaling and transformation function is: Among them, 0≤i≤255, 0≤j≤255, W A(m,n) is the low-frequency component; h(i,j) is the low-pass filter coefficient; m and n are the coordinates of the low-frequency component; Formula (9) represents the convolution operation of the original image signal with the low-pass filter, and the low-frequency component is obtained by downsampling; The high frequency components obtained are: Horizontal high frequency component: Among them, W H(m,n) is the horizontal high-frequency component; g_H(i,j) is the horizontal high-pass filter coefficient; Vertical high frequency component: Among them, W V(m,n) is the vertical high-frequency component; g_V(i,j) is the vertical high-pass filter coefficient; Diagonal high frequency components: Among them, W D(m,n) is the diagonal high frequency component; g_D(i,j) is the diagonal high pass filter coefficient.

3. The method for adaptively enhancing low-light-level night vision color images according to claim 1, characterized in that: In step 2, different weights are used to obtain reasonable adaptive grayscale images for low-frequency images. The specific values ​​are: I R Represents the R (red) channel of the image; I G Represents the G (green) channel of the image; I B represents the B (blue) channel of the image; (x, y) represents the pixel value of the image; W (x,y) Represents the low-frequency component of the image; Indicates the brightness value of the image output; Solve for the average value of the image input brightness, expressed as: Represents the average value of the input brightness logarithm; N represents the total number of pixels; δ is the minimum value, which is used to avoid numerical overflow when performing log calculation on pure black pixels; L g (x, y) represents the logarithmic ratio of the output; L W (x, y) represents the brightness value of the input image; L wmax Represents the maximum value of the input image brightness value pair; L min (x, y) represents the minimum grayscale value of the input image; L max (x, y) represents the maximum grayscale value of the input image; L(x, y) represents the result of image adaptive output.

4. The method for adaptively enhancing low-light-level night vision color images according to claim 1, characterized in that: In step 3, for the high-frequency image, the filtered image is converted from RGB to HSV image using color mapping, specifically: Among them, R, G, B represent the high-frequency image components W ( ' x,y) The red, green and blue channels of the image are as follows; H, S and V represent the high-frequency image components W after conversion. ( ' x,y) Hue, saturation, and brightness; max(R,G,B) is the maximum value of each of the three channels (R,G,B); min(R,G,B) is the minimum value of each of the three channels (R,G,B); The converted HSV image is normalized and the high-frequency image component W is ( ' x,y) After normalization, we get K: K is the normalized value of the image after color mapping.

5. A low-light-level night vision color image adaptive enhancement device, characterized in that: The device comprises: An image decomposition unit is used to decompose the night vision visible light image using a multi-level decomposition method to extract high-frequency information and low-frequency information; wherein the high-frequency information includes image details and texture features, corresponding to components with high grayscale values ​​in the image; the low-frequency information includes image contour features, corresponding to background components in the image; A low-frequency information processing unit is used to construct an adaptive enhancement network to extract background texture features of low-frequency images; A high-frequency information processing unit is used to convert the RGB image into the HSV image by using a color mapping method, and then normalize the image to obtain enhanced high-frequency image detail information; The image reconstruction unit is used to reconstruct the processed high-frequency image and low-frequency image to form an enhanced night vision visible light image.

6. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 4.

7. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 4.

Images