Unsupervised low-light enhancement method for road scene

Abstract

The invention relates to an unsupervised low-light enhancement method for a road scene, and provides the unsupervised low-light enhancement method for the road scene, aiming at improving the perception capability of an automatic driving system in a low-light environment, and particularly ensuring the safe operation of a pilotless automobile at night or under a complex illumination condition. The invention provides a joint enhancement strategy based on RGB (Red, Green, Blue), LAB (Laboratory) and HSI (Hue, Saturation and Intensity) color spaces through multi-domain collaborative feature extraction and enhancement in order to overcome the defects of an existing method between an enhancement effect and real-time performance. The method does not depend on a large amount of annotation data, is suitable for practical application scenes, and maintains good color consistency while improving image brightness and details. And through combination of the lightweight convolutional network and physical prior, the requirement of real-time processing is ensured. Finally, the enhanced image is suitable for a sensing module of an automatic driving system, the sensing precision and safety in a low-light environment are effectively improved, and the method has wide practical application value.

Description

Technical Field

[0001] The invention belongs to the field of image enhancement, and mainly relates to an unsupervised low-light enhancement method for road scenes. Background Art

[0002] With the rapid development of driverless technology, the application of autonomous driving systems in road scenes has gradually matured. In order to ensure the safe operation of driverless cars in complex environments, especially in low-light or night driving scenarios, image enhancement technology plays a vital role. Low-light image enhancement aims to improve the quality of images collected in low-light environments, thereby providing clear visual information for the autonomous driving system to ensure its judgment and decision-making capabilities during driving.

[0003] In the field of low-light image enhancement, early methods mostly rely on traditional image processing techniques, such as histogram equalization, Retinex algorithm, contrast stretching, etc. These methods try to improve the image details by enhancing the visual features of the image such as brightness and contrast, so as to make the image more visible. Histogram equalization adjusts the brightness distribution of the image so that the details of the bright and dark parts of the image are fully presented; while the Retinex algorithm attempts to simulate the visual processing mechanism of the human eye and remove the influence of uneven lighting in the image to improve the image details and color performance. However, these traditional methods often have some limitations when facing complex environmental changes and changing lighting conditions. Histogram equalization may cause loss of image details or color distortion, while the Retinex algorithm may introduce noise or artifacts, especially in scenes with strong light and dark areas, and cannot achieve the ideal enhancement effect.

[0004] With the rapid development of deep learning technology, low-light enhancement methods based on convolutional neural networks and generative adversarial networks have gradually become a research hotspot. Deep learning methods can learn the enhancement rules of low-light images from a large amount of data by training deep models, thereby significantly improving the visibility, brightness and detail recovery ability of images. Unlike traditional algorithms, deep learning methods can adaptively adjust the brightness and contrast in images, automatically mine important features in images, and avoid errors introduced by artificially set parameters. The GAN model can generate more realistic enhanced images, effectively remove noise and enhance details through the game between the generator and the discriminator. However, although deep learning methods have shown significant advantages in low-light enhancement, existing deep learning-based solutions still face some challenges. First, these methods usually require a large amount of labeled data for training, and the cost of obtaining labeled data is high, especially in low-light environments. The lack of labeled data limits the generalization ability of the model. Second, deep learning models take a long time to train and require large computing resources, which makes it difficult to deploy them efficiently in application scenarios with high real-time requirements.

[0005] In recent years, unsupervised learning methods have gradually become a research hotspot in the field of low-light enhancement. Unlike supervised learning, unsupervised learning methods do not rely on labeled data, but extract features from a large amount of unlabeled data and learn the rules of image enhancement through self-supervision or data comparison learning. The advantage of this method is that it can greatly reduce the cost of data annotation, and at the same time, it can automatically learn and optimize the image enhancement process without human intervention. In the field of unmanned driving, low-light image enhancement especially needs to consider the issues of real-time and accuracy, and unsupervised learning methods can perform effective image enhancement when data is scarce, especially suitable for scenarios with unlabeled data. Summary of the invention

[0006] In view of the lack of paired data sets in current road scenes and the shortcomings of existing methods in balancing enhancement effect and real-time performance, the present invention proposes an unsupervised low-light enhancement method for road scenes, aiming to improve the driving safety and environmental perception ability of vehicles in low-light conditions.

[0007] The technical solution of the present invention is as follows:

[0008] (1) Preprocess the relevant datasets of open source low-light road scenes, set the image size to a uniform size, and perform normalization to improve the stability of the model.

[0009] (2) Based on the preprocessed dataset, the RGB images are converted to LAB and HSI domains, and the original low-light images are processed using a physical algorithm.

[0010] (3) The enhanced feature maps of the image are extracted through residual convolutional neural networks in the RGB, HSI and LAB domains respectively.

[0011] (4) In the RGB, HSI and LAB domains, the three images are enhanced by extracting the enhanced feature maps.

[0012] (5) The enhanced HIS and LAB domain images are converted into RGB images, and three-channel fusion is performed with the enhanced RGB images. Combined with the previous prior images, the enhanced images are output.

[0013] Furthermore, the step (1) adjusts the low-light image to a uniform size through cropping and scaling operations, and normalizes the values ​​of the three channels R, G and B to 0 to 1, so as to meet the input requirements of the neural network model and improve the consistency of the data during the training process.

[0014] Furthermore, in step (2), by converting the low-light image into RGB, LAB and HSI color spaces respectively, the characteristics of each color space are fully utilized to mine the potential information of the image.

[0015] At the same time, combined with the physical-based prior method, the brightness and contrast of the image are further optimized through brightness gain and color balance. The overall brightness is improved by using the brightness gain factor, and the mean and standard deviation of each channel are calculated for normalization to enhance the contrast between channels, thereby improving the color imbalance problem under low-light conditions. After normalization, appropriate contrast stretching adjustment is performed to ensure the visual naturalness and clarity of the output image. In addition, in order to suppress the noise in the low-light image, a Gaussian filter is used for denoising. By constructing a two-dimensional Gaussian kernel and performing a convolution operation, high-frequency noise is effectively removed while retaining edge details. Thus, the physical method is used to perform prior processing on the low-light image to generate a prior image of normal light.

[0016] Furthermore, in the step (3), in the three color spaces of RGB, LAB and HSI, an enhanced feature map is extracted through a residual network structure, and the network design aims to fully tap the multi-level semantic and detail information to improve the enhancement effect of low-light images. The network adopts a modular residual connection structure, and each module contains multiple convolutional layers for extracting local features of the image. At the same time, through cross-layer residual connections, high-level semantic information can be effectively integrated with underlying detail information to avoid the reduction of enhancement effect due to information loss. Each convolution module in the network gradually enhances the input features, and through a multi-level information feedback mechanism across modules, local features are gradually integrated into global features with stronger representation capabilities. In this way, feature extraction networks in different color domains can operate independently and do not interfere with each other during the enhancement process, providing high-quality multi-domain feature expression for subsequent fusion and image enhancement steps.

[0017] Furthermore, in step (4), the brightness and contrast of the images in the RGB, LAB and HSI color domains are optimized pixel by pixel by using a recursive enhancement curve function. The enhancement process follows the following recursive formula:

[0018] X n =X n-1 +μ*X n-1 (1-X n-1 )

[0019] Among them, X n represents the image after n-th iteration enhancement, X n-1is the enhancement result of the previous iteration, and μ is the enhanced feature map extracted by the residual network in step (3). Through this recursive method, each enhancement operation will further improve the image quality based on the optimization of the previous stage, making the brightness gradually improved and the details richer.

[0020] However, for the AB channel in LAB and the H channel in HIS, enhancement by the above formula may cause color deviation, so it is necessary to control its reasonable range. The enhancement curve for the AB channel has the following formula:

[0021]

[0022] The enhancement formula for the H channel is as follows:

[0023]

[0024] Furthermore, in step (5), the enhanced HSI domain and LAB domain images are first converted back to the RGB color space respectively, and the enhanced images of the three domains are weighted fused through a multi-branch fusion strategy. The fusion process adopts a weighted method based on weight adaptive optimization, which not only combines the enhancement advantages of each color domain, but also introduces prior information from the original low-light image, further ensuring that the output image has excellent brightness, contrast and color fidelity. In addition, through the multi-domain feature fusion method, the complementary characteristics of different color domains are fully utilized, so that the final enhanced image is more natural in visual effect, while meeting the dual needs of brightness enhancement and detail fidelity in practical applications.

[0025] The advantages of the present invention are:

[0026] (1) It relies on multi-domain collaborative feature extraction and enhancement, getting rid of the dependence on a large amount of labeled data and is suitable for practical application scenarios.

[0027] (2) Through the comprehensive use of LAB, HSI and RGB color spaces, the enhanced image can maintain good color consistency and visual effects while improving brightness.

[0028] (3) Utilize lightweight convolutional networks and physical prior guidance to ensure that real-time processing requirements are met while improving image enhancement effects.

[0029] (4) Focusing on road scenes, the enhanced images are suitable for the perception module of the autonomous driving system, improving the vehicle's ability to adapt to low-light environments and have broad practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The overall framework of the network

[0031] Figure 2Network structure for extracting enhanced features

[0032] Figure 3 Enhancement curves for enhancement processing

[0033] Figure 4 Enhancement curve of AB channel

[0034] Figure 5 Enhancement curve of H channel

[0035] Figure 6 The overall process of physical priors DETAILED DESCRIPTION

[0036] In order to make the technical solution of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings.

[0037] An unsupervised low-light enhancement method for road scenes includes the following steps:

[0038] (1) Preprocess the relevant datasets of open source low-light road scenes, set the image size to a uniform size, and perform normalization to improve the stability of the model.

[0039] (2) Based on the preprocessed dataset, the RGB images are converted to LAB and HSI domains, and the original low-light images are processed using a physical algorithm.

[0040] (3) The enhanced feature maps of the image are extracted through residual convolutional neural networks in the RGB, HSI and LAB domains respectively.

[0041] (4) In the RGB, HSI and LAB domains, the three images are enhanced by extracting the enhanced feature maps.

[0042] (5) The enhanced HIS and LAB domain images are converted into RGB images, and three-channel fusion is performed with the enhanced RGB images. Combined with the previous prior images, the enhanced images are output.

[0043] Furthermore, in step (1), the relevant dataset of the open source low-light road scene is preprocessed, and the specific implementation method is as follows:

[0044] 1) The original image is cropped to a uniform size of 512 × 512 pixels and scaled by bilinear interpolation.

[0045] 2) Normalize the R, G, and B channel pixel values ​​and map them to the range of 0 to 1.

[0046] Furthermore, in step (2), the RGB image is converted to the LAB and HSI domains, and the original low-light image is processed by a physical algorithm. The specific implementation method is as follows:

[0047] 1) Perform gamma correction on each channel value of the RGB image to correct the nonlinear response of the display device. The formula is as follows:

[0048]

[0049] Among them, t represents the pixel value of R, G, and B, and then the RGB image is converted from the RGB color space to the XYZ color space using the following matrix multiplication formula.

[0050]

[0051] Next, normalize XYZ, and then use the following formula to perform nonlinear transformation on the three channels of XYZ:

[0052]

[0053] XYZ is converted into f through nonlinear transformation X 、f Y 、f Z , and then convert to LAB using the following formula:

[0054]

[0055] 2) Calculate HSI from the normalized RGB using the following formula:

[0056]

[0057]

[0058]

[0059]

[0060] 3) Physical prior for low-light RGB images. First, the brightness of the image is improved overall by multiplying the pixel values ​​of the input image by a fixed gain factor. This can increase the overall brightness of the image and make it look clearer. However, in order to ensure the legality of the pixel values, all pixel values ​​will be limited to the range of 0 to 1 to prevent overflow or illegal values. This process is similar to using a linear amplifier to enhance the brightness of the image without losing its original dynamic range.

[0061] Next, in order to improve the color contrast of the image, each color channel is standardized separately, and the mean and standard deviation of each channel are calculated to represent the average brightness of the current color channel and the distribution range of pixel values. Then, the mean is subtracted from each pixel value and then divided by the standard deviation, which can remove the problem of illumination deviation or uneven distribution in the image. In order to further adjust the contrast and range of the image, the standardized result is linearly transformed so that the range of pixel values ​​is mapped to between 0 and 1. This not only enhances the contrast of the image, but also improves the balance between different color channels, making the image appear more natural and layered. The specific formula is as follows:

[0062]

[0063] where μ c is the mean of channel c, σ c is the standard deviation of channel c, ∈ is often set to a minimum value to prevent the denominator from being zero.

[0064] The enhanced image often contains some noise. In order to further improve the image quality, it is necessary to perform Gaussian denoising on the enhanced image. The image is convolved with a weight kernel based on Gaussian distribution, and each pixel value is replaced with the weighted average of its surrounding pixels. The size and standard deviation of the Gaussian kernel determine the strength of the denoising. A small standard deviation can retain more details, and a large standard deviation can remove noise more strongly. The Gaussian filter smoothes each pixel in the area around it to make the image smoother without obvious distortion. This can effectively reduce the small noise commonly seen in low-light images while maintaining the edges and main features of the image. The specific formula is as follows:

[0065]

[0066] Among them, G ′ (i, j) is a normalized two-dimensional Gaussian kernel expressed as follows:

[0067]

[0068] Where k is the Gaussian kernel radius and σ is the standard deviation of the Gaussian distribution.

[0069] Furthermore, in step (3), a residual convolutional neural network is designed to extract enhanced feature maps from the three color domains of RGB, LAB and HSI. The network adopts a modular residual structure, and each module contains 7 layers of convolution, which is used to extract local features of the image and effectively fuse high-level semantic information with underlying detail features through cross-layer residual connections, thereby avoiding the degradation of enhancement effect caused by information loss. Each convolution module gradually enhances the input features through a multi-level information feedback mechanism, and finally generates global features with high representation capabilities. These features are extracted from the three color domains of RGB, LAB and HSI respectively and run independently without interfering with each other, providing high-quality multi-domain feature expression for subsequent image enhancement and fusion.

[0070] Furthermore, in step (4), the brightness and contrast of the images in the RGB, LAB and HSI color domains are optimized pixel by pixel by recursive enhancement curve functions. The enhancement operation follows the recursive formula:

[0071] X n =X n-1 +μ*X n-1 (1-X n-1 )

[0072] This formula can ensure that each pixel value of the enhanced image is within the normalized range [0,1] to avoid information loss caused by overflow truncation. At the same time, the curve is monotonic and as simple and differentiable as possible during the gradient backpropagation process. Through multiple iterations, each enhancement operation further improves the image quality based on the previous optimization, so that the brightness gradually increases and the details are richer. After performing the enhancement operation in each color domain, an independently optimized enhanced image is obtained. However, for the AB channel in LAB and the H channel in HIS, the enhancement by the above formula may cause color deviation, so it is necessary to control its reasonable range. The enhancement curve for the AB channel has the following formula:

[0073]

[0074] The enhancement formula for the H channel is as follows:

[0075]

[0076] Furthermore, step (5) converts the enhanced HIS and LAB images into RGB images, and performs three-channel fusion with the enhanced RGB image, and outputs the enhanced image in combination with the previous prior image. The specific implementation method is as follows:

[0077] 1) Convert LAB image to RGB image. First, convert LAB to XYZ. The formula is as follows:

[0078]

[0079] Then a nonlinear change is performed, and the formula is as follows:

[0080]

[0081] Then convert XYZ to RGB, the formula is as follows:

[0082]

[0083] Then perform the gamma transformation:

[0084]

[0085] The converted RGB image can be obtained.

[0086] 2) Convert the HSI image to an RGB image, when 0≤H<120°:

[0087]

[0088] When 120°≤H<240°: H=H-120°

[0089]

[0090] When 240°≤H<360°: H=H-240°

[0091]

[0092] 3) The converted LAB and HIS images are fused with RGB, and the mean fusion method is used to add the three images and divide them by 3 to obtain the enhanced image. Then the loss function is calculated. First, the spatial consistency loss is calculated to enhance the spatial consistency of the image by retaining the difference between the adjacent areas between the input image and the enhanced image, according to the following formula:

[0093]

[0094] Where K is the number of local regions, and Ω(i) is the four neighboring regions (upper, lower, left, and right) centered on region i. We denote Y and i as the average intensity values ​​of the local regions in the enhanced and input images, respectively. We empirically set the size of the local region to 4×4. This loss is stable considering the size of other regions. To suppress underexposed / overexposed regions, we design an exposure control loss L exp To control the exposure level. The exposure control loss measures the distance between the average intensity value of the local area and the good exposure level E, and its formula can be expressed as follows:

[0095]

[0096] Where M represents the number of non-overlapping local regions of size 16×16, and Y is the average intensity value of the local region in the enhanced image. Based on the gray-world color constant assumption, that is, the color of each sensor channel is averaged gray over the entire image, a color constant loss is used to correct the potential color deviation in the enhanced image, and the relationship between the three adjustment channels is established.

[0097]

[0098] In order to maintain the monotonic relationship between adjacent pixels, we add an illumination smoothness loss L in each curve parameter map. tvA Defined as:

[0099]

[0100] Where N is the number of iterations, and Represents the horizontal and vertical gradient operations respectively. The mean square error loss is calculated by inputting the previous prior image and the enhanced image, which can be expressed as the following formula:

[0101]

[0102] Calculate the total loss function:

[0103] L totall =ω0L spa +ω1L exp +ω2L col +ω3L tvA +ω4L mse

[0104] The model is trained by forward deducing the total loss function.

Claims

1. An unsupervised low-light enhancement method for road scenes, characterized by comprising the following steps: (1) Preprocess the relevant datasets of open source low-light road scenes, set the image size to a uniform size, and perform normalization to improve the stability of the model. (2) Based on the preprocessed dataset, the RGB images are converted to LAB and HSI domains, and the original low-light images are processed using a physical algorithm. (3) The enhanced feature maps of the image are extracted through residual convolutional neural networks in the RGB, HSI and LAB domains respectively. (4) In the RGB, HSI and LAB domains, the three images are enhanced using the extracted enhanced feature maps. (5) The enhanced HIS and LAB domain images are converted into RGB images, and three-channel fusion is performed with the enhanced RGB images. Combined with the previous prior images, the enhanced images are output.

2. According to claim 1, in step (2), by converting the low-light image into RGB, LAB and HSI color spaces respectively, the characteristics of each color space are fully utilized to mine the potential information of the image. At the same time, combined with the physical-based prior method, the brightness and contrast of the image are further optimized through brightness gain and color balance. The overall brightness is improved by using the brightness gain factor, and the mean and standard deviation of each channel are calculated for normalization to enhance the contrast between channels, thereby improving the color imbalance problem under low-light conditions. After normalization, appropriate contrast stretching adjustment is performed to ensure the visual naturalness and clarity of the output image. In addition, in order to suppress the noise in the low-light image, a Gaussian filter is used for denoising. By constructing a two-dimensional Gaussian kernel and performing a convolution operation, high-frequency noise is effectively removed while retaining edge details. Thus, the physical method is used to perform prior processing on the low-light image to generate a prior image of normal light.

3. According to claim 1, in step (3), in the three color spaces of RGB, LAB and HSI, the enhanced feature map is extracted by the residual network structure, and the network design aims to fully tap the multi-level semantic and detail information to improve the enhancement effect of low-light images. The network adopts a modular residual connection structure, and each module contains multiple convolutional layers for extracting local features of the image. At the same time, through cross-layer residual connections, high-level semantic information can be effectively integrated with underlying detail information to avoid the reduction of enhancement effect due to information loss. Each convolution module in the network gradually enhances the input features, and through a multi-level information feedback mechanism across modules, the local features are gradually integrated into global features with stronger representation capabilities. In this way, feature extraction networks in different color domains can run independently and do not interfere with each other during the enhancement process, providing high-quality multi-domain feature expression for subsequent fusion and image enhancement steps.

4. According to claim 1, in step (4), the brightness and contrast of the images in the RGB, LAB and HSI color domains are optimized pixel by pixel by using a recursive enhancement curve function. The enhancement process follows the following recursive formula: X n =X n-1 +μ*X n-1 (1-X n-1 ) in, X n represents the image after n-th iteration enhancement, X n-1 is the enhancement result of the previous iteration, and μ is the enhanced feature map extracted by the residual network in step (3). Through this recursive method, each enhancement operation will further improve the image quality based on the optimization of the previous stage, making the brightness gradually improved and the details richer. However, for the AB channel in LAB and the H channel in HIS, enhancement by the above formula may cause color deviation, so it is necessary to control its reasonable range. The enhancement curve for the AB channel has the following formula: The enhancement formula for the H channel is as follows:

5. According to claim 1, in step (5), the enhanced HSI domain and LAB domain images are converted back to RGB color space respectively, and the enhanced images of the three domains are weighted fused through a multi-branch fusion strategy. The fusion process adopts a weighted method based on weight adaptive optimization, which not only combines the enhancement advantages of each color domain, but also introduces prior information from the original low-light image, further ensuring that the output image has excellent brightness, contrast and color fidelity. In addition, through the multi-domain feature fusion method, the complementary characteristics of different color domains are fully utilized, so that the final enhanced image is more natural in visual effect, while meeting the dual needs of brightness enhancement and detail fidelity in practical applications.

Images