A low-light image enhancement method of spatial adaptive supervised learning

Abstract

The application discloses a low-light image enhancement method based on spatial adaptive supervised learning, and the target image is input into a trained low-light enhancement network model for low-light enhancement; the low-light enhancement network model comprises a local branch, a global branch and an adaptive feature fusion module; the local branch obtains local features of the target image; the global branch obtains global features of the target image; and the adaptive feature fusion module fuses the obtained local features and global features to complete low-light image enhancement.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a spatially adaptive supervised learning method for low-light image enhancement. Background Technology

[0002] Image processing is now a crucial area within deep learning. Low-light image enhancement plays a vital role in low-level visual tasks. Images acquired under low-light conditions often suffer from multiple degradations, including poor visual quality, low limitations, and high ISO noise, challenging human visual perception and downstream visual tasks. To mitigate these degradations, numerous algorithms have been proposed, covering histogram analysis, Retinex algorithms, and deep learning, to improve the performance of intelligent systems in various applications, such as autonomous driving and security monitoring. Therefore, low-light image enhancement is highly sought after.

[0003] Low-light image enhancement aims to improve the perceptual quality of images acquired in low-light environments. Although there are many good methods for low-light image enhancement, existing methods have obvious drawbacks. This is because (1) most methods only consider the influence of brightness when processing low-light images without interpreting noise; (2) some methods process illumination and remove noise in space at the same time, which ultimately leads to unsuitable matching; (3) some methods are unaware of noise and can only process spatially independent noise; (4) some model structures are too cumbersome and have too low efficiency in processing low-light images.

[0004] Traditional low-light image enhancement tends to use histogram equalization and gamma correction, which produce unwanted artifacts in the enhanced real-world images. Recently, Li et al.

[12] proposed a robust Retinex model that incorporates new lighting conditions and reflectance canonical terms, which improves the performance of low-light image enhancement under strong noise compared to the traditional Retinex model. However, retinex-based methods often distort colors locally when enhancing complex real-world images.

[0005] Different regions in a low-light image possess varying brightness characteristics, noise levels, and visibility. Areas with extremely low brightness often contain very high noise, while other regions exhibit acceptable visibility and contrast. For an image, denoising planar regions is generally easier than denoising textured regions because textured regions must preserve edges and details well. Therefore, our focus is on addressing the low-light image enhancement problem for these two regions, particularly considering their distinct characteristics. Summary of the Invention

[0006] The purpose of this invention is to address the technical deficiencies in existing technologies by providing a spatially adaptive supervised learning method for low-light image enhancement.

[0007] The technical solution adopted to achieve the purpose of this invention is:

[0008] A spatially adaptive supervised learning method for low-light image enhancement involves inputting the target image into a trained low-light enhancement network model for low-light enhancement.

[0009] The low-light enhancement network model includes a feature extraction module, a region segmentation module, a local branch module, a global branch module, and an adaptive feature fusion module;

[0010] The feature extraction module performs feature extraction on the input image to obtain input features; the input features are then input to the local branch module and the global branch module respectively.

[0011] The region segmentation module divides the input image into regions, which are then divided into smooth regions and texture regions.

[0012] The local branch module performs secondary feature extraction on the input features to obtain local features of the smooth region;

[0013] The global branch module performs secondary feature extraction on the input features to obtain the global features of the texture region;

[0014] The adaptive feature fusion module fuses the obtained local and global features to complete the low-light image enhancement.

[0015] In the above technical solution, the region division refers to using a hyperparameter α to divide the smooth region and texture region of the low-light image.

[0016] In the above technical solution, image patches at each spatial location (i, j) are first extracted densely, then the standard deviation σ is calculated, and finally a piecewise function is used to convert σ into α.

[0017] The standard deviation σ is expressed as follows:

[0018]

[0019] std(·) represents the standard deviation function, and is measured by averaging over a channel patch;

[0020] n represents the size of the local window.

[0021] α is represented as follows:

[0022]

[0023] S(·) represents the Sigmoid function; l and u are set to default values.

[0024] In the above technical solution, the local branch module includes five convolutional residual blocks, and each convolutional residual block includes two convolutions and one partial convolution.

[0025] In the above technical solution, the global branch module includes an encoder, a decoder, and a parameterless attention module;

[0026] The encoder and decoder each contain three convolutional layers and two upsampling layers;

[0027] The parameterless attention module includes a normalization layer, multi-head self-attention, a multilayer perceptron, and 12 converter modules.

[0028] In the above technical solution, the training method of the low-light enhancement network model includes the following steps:

[0029] Step 1: Obtain training samples; the training samples include low-light images and normal-light images of the samples;

[0030] Step 2: Preprocess the obtained training samples;

[0031] Step 3: Input the preprocessed training sample data into the low-light enhancement network model to perform low-light image enhancement and obtain the output image;

[0032] Step 4: Calculate the reconstruction loss and update the parameters of the low-light enhancement network model using backpropagation to complete one model training cycle;

[0033] Continue model training by repeating steps 1-4 until the reconstruction loss is acceptable, thus completing model training.

[0034] In the above technical solution, the training samples adopt the SID and SDSD training sets.

[0035] In the above technical solution, the preprocessing includes image pairing, image cropping, and data augmentation.

[0036] In the above technical solution, the reconstruction loss includes Charbonnier loss and perceptual loss.

[0037] In the above technical solution, the overall loss function of the reconstruction loss is expressed as follows:

[0038] L = L r *α+Lr*(1-α)+λ*L vgg ;

[0039] Among them, L rThis is the Charbonnier loss; L vgg For perceptual loss; λ and α are both hyperparameters;

[0040] Charbonnier loss r It is expressed as follows:

[0041]

[0042] In the formula, I' represents the output image. For the real image, ξ was set to 10 in all experiments. -3 ;

[0043] Perceived loss L vgg For I' and The L1 distance between them is expressed as follows:

[0044]

[0045] In the formula, Φ() is the operation for extracting features from the VGG network.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] 1. This invention provides a spatially adaptive supervised learning-based low-light image enhancement method that improves image clarity by enhancing the visibility of low-light images. A hyperparameter α is used to divide the low-light image into smooth and textured regions. For planar regions, an improved convolution is used to acquire local information; for textured regions, a transformer structure is used to introduce a parameter-free attention mechanism, which enhances the low-light image by collecting global information. Extensive experiments demonstrate that our model has good performance. In particular, when the enhanced image is applied to face recognition, its performance is superior to previous methods.

[0048] 2. The spatial adaptive supervised learning low-light image enhancement method provided by this invention employs a model with high sensitivity and high computational efficiency.

[0049] 3. The spatially adaptive supervised learning-based low-light image enhancement method provided in this invention can generate enhanced images that more closely resemble real images. Furthermore, our output exhibits fewer visual artifacts and locally distorts colors in regions with complex textures. These results also demonstrate the effectiveness of the image enhancement, suppressing noise while enhancing details. Attached Figure Description

[0050] Figure 1 The diagram shows the flowchart and overall network structure of the spatial adaptive supervised learning method for low-light image enhancement.

[0051] Figure 2 The diagram shows the structure of a partial convolutional PLB (left) and residual block (right) in a local branch structure.

[0052] Figure 3 The diagram shows the structure of the parameterless attention (SIM) module in the global branch structure.

[0053] Figure 4 The flowchart shown is for the training method;

[0054] Figure 5 The image shows a comparison of low-light enhancement algorithms. Detailed Implementation

[0055] The present invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0056] Example 1

[0057] A spatially adaptive supervised learning low-light enhancement network model includes a feature extraction module, a region partitioning module, a local branch module, a global branch module, and an adaptive feature fusion module;

[0058] The feature extraction module performs feature extraction on the input image to obtain input features; the input features are then input to the local branch module and the global branch module respectively.

[0059] The region segmentation module divides the input image into regions, which are then divided into smooth regions and texture regions.

[0060] The local branch module performs secondary feature extraction on the input features to obtain local features of the smooth region;

[0061] The global branch module performs secondary feature extraction on the input features to obtain the global features of the texture region;

[0062] The adaptive feature fusion module fuses the obtained local and global features to complete the low-light image enhancement.

[0063] Specifically

[0064] The feature extraction module includes two 3x3 convolutions.

[0065] The region division refers to using the hyperparameter α to divide the smooth region and texture region of the low-light image.

[0066] First, image patches are extracted densely for each spatial location (i, j), then the standard deviation σ is calculated, and finally a piecewise function is used to convert σ to α.

[0067] The standard deviation σ is expressed as follows:

[0068]

[0069] std(·) represents the standard deviation function, and is measured by averaging over a channel patch;

[0070] n represents the size of the local window.

[0071] α is represented as follows:

[0072]

[0073] S(·) represents the Sigmoid function; based on experience, we set l to 1 and u to 5.

[0074] The local branch contains five improved convolutional residual blocks, each containing two convolutions and one partial convolution, the structure of which is as follows: Figure 2 As shown, partial convolution treats valid elements and missing pixels differently. The convolutional layer only performs convolution and normalization operations on valid pixels that meet the conditions, and then automatically updates the mask according to the mask update rules until all values ​​in the mask are 1, thereby improving computational efficiency and enhancing the model's sensitivity.

[0075] The global branch contains an encoder, a decoder, and a parameterless attention module. Each encoder and decoder contains three convolutional layers and two upsampling layers. The parameterless attention module includes channel attention and spatial attention. During the feature extraction stage, in each cross-stage residual attention, features are progressively separated through channel separation and ultimately fused across stages, while assigning different weights to different channel and spatial features. The parameterless attention module is as follows: Figure 3 As shown, it includes a normalization layer, multi-head self-attention (MHSA), and a multilayer perceptron (MLP), as well as 12 transformer modules. The SIM module, located at the back of the network structure, can infer the 3D attention weights of the multilayer feature maps, improving the representational power of many convolutional networks.

[0076] Example 2

[0077] This embodiment describes the training method based on embodiment 1.

[0078] The training method for the low-light enhancement network model includes the following steps: Figure 4 As shown,

[0079] Step 1: Obtain training samples; the training samples include low-light images and normal-light images of the samples;

[0080] This embodiment uses the SID and SDSD training sets as training samples for training. SID consists of images captured using Sony cameras, including low-light and normal-light images; SDSD contains an indoor image dataset and an outdoor image dataset, both providing pairs of low-light and normal-light image data.

[0081] Step 2: Preprocess the obtained training sample data;

[0082] The preprocessing includes image pairing, image cropping, and data augmentation.

[0083] The image pairing refers to synthesizing corresponding image pairs by combining low-light image data and normal-light image data of the samples.

[0084] Image cropping refers to cropping the training samples to a size of 600*400. The purpose of image cropping is to prevent overfitting during network training and improve the network's robustness. The size can be adjusted according to the actual situation during the specific operation.

[0085] The data augmentation refers to normalizing the pixel values ​​of the training samples to the range [-1, 1] to prevent excessive differences in pixel values ​​among the input training samples from causing difficulties in network convergence. Simultaneously, the cropped training samples are augmented through random horizontal flipping and 90° rotation to increase sample richness.

[0086] Taking into account the respective characteristics of planar regions and textured regions in low-light images, the preprocessing also includes region segmentation;

[0087] Step 3: Input the preprocessed training sample data into the low-light enhancement network model to perform low-light image enhancement and obtain the output image;

[0088] Step 4: Calculate the reconstruction loss and update the parameters of the low-light enhancement network model using backpropagation to complete one model training cycle;

[0089] Continue model training by repeating steps 1-4 until the reconstruction loss is acceptable, thus completing model training.

[0090] The overall loss function for the reconstruction loss is expressed as follows:

[0091] L = L r *α+L r *(1-α)+λ*L vgg (1);

[0092] Among them, L r This is the Charbonnier loss; L vgg For perceptual loss; λ and α are both hyperparameters;

[0093] Charbonnier loss r It is expressed as follows:

[0094]

[0095] In the formula, I' represents the output image. For the real image, ξ was set to 10 in all experiments. -3 ;

[0096] Perceived loss L vgg For I' and The L1 distance between them is expressed as follows:

[0097]

[0098] In the formula, Φ() is the operation for extracting features from the VGG network.

[0099] During training, the low-light enhancement network was trained on a 2080Ti GPU with an image batch size of 32. The optimizer's momentum against Adam was 0.9.

[0100] Example 3

[0101] A spatially adaptive supervised learning method for low-light image enhancement involves inputting the target image into a pre-trained low-light enhancement network model for low-light enhancement.

[0102] like Figure 1 and Figure 4 As shown, the input image is first subjected to a 3*3 convolution to obtain input features. Then, a hyperparameter α is used to divide the smooth and texture regions of the low-light image. The texture region of the target image will obtain global features through a global branch, while the smooth region will obtain local features through a local branch. Finally, an adaptive feature fusion module is applied to fuse the obtained local and global features to complete the low-light image enhancement.

[0103] like Figure 5 The images shown are examples of image enhancement effects achieved by this invention and other methods for low-light and backlit images. Experimental results demonstrate that, despite these being original images with significant noise and weak lighting, the method presented in this application can still generate enhanced images that more closely resemble the real images. Furthermore, the output of this application exhibits fewer visual artifacts and less local color distortion in areas with complex textures. These results also demonstrate the effectiveness of image enhancement, suppressing noise while enhancing details.

[0104] This application compares our framework with 23 state-of-the-art low-light image enhancement methods, including Dong, LIME, MF, SRIE, BIMEF, DRD, RRM, SID, DeepUPE, KIND, DeepLPF, FIDE, LPNet, MIR-Net, RF, 3DLUT, A3DLUT, Band, EG, and Sparse. Furthermore, we compare our framework with two state-of-the-art low-level task transformer architectures (IPT and Uformer).

[0105] We use Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index.

[0106]

[0107] The SSIM (Significant Score Simulation) is used for evaluation. A higher SSIM usually means that the final result has more high-frequency details and structure. Table 1 shows the qualitative comparison results of this invention with other methods on the LOL dataset.

[0108] Table 1 shows the comparison results between our method and the state-of-the-art methods. Our method outperforms all baseline methods.

[0109] The above description is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A spatially adaptive supervised learning method for low-light image enhancement, characterized in that: The target image is input into a trained low-light enhancement network model for low-light enhancement. The low-light enhancement network model includes a feature extraction module, a region segmentation module, a local branch module, a global branch module, and an adaptive feature fusion module; The feature extraction module performs feature extraction on the input image to obtain input features; the input features are then input to the local branch module and the global branch module respectively. The region segmentation module divides the input image into regions, which are then divided into smooth regions and texture regions. The local branch module performs secondary feature extraction on the input features to obtain local features of the smooth region; The global branch module performs secondary feature extraction on the input features to obtain the global features of the texture region; The adaptive feature fusion module fuses the obtained local and global features to complete the low-light image enhancement.

2. The low-light image enhancement method based on spatial adaptive supervised learning as described in claim 1, characterized in that: The region division refers to using the hyperparameter α to divide the smooth region and texture region of the low-light image.

3. The low-light image enhancement method based on spatial adaptive supervised learning as described in claim 2, characterized in that: First, image patches are extracted densely for each spatial location (i, j), then the standard deviation σ is calculated, and finally a piecewise function is used to convert σ into α. The standard deviation σ is expressed as follows: std(·) represents the standard deviation function, and is measured by averaging over a channel patch; n represents the size of the local window; α is represented as follows: S(·) represents the Sigmoid function; l and u are set to default values.

4. The low-light image enhancement method based on spatial adaptive supervised learning as described in claim 1, characterized in that: The local branch module contains five convolutional residual blocks, each of which contains two convolutions and one partial convolution.

5. The low-light image enhancement method based on spatial adaptive supervised learning as described in claim 1, characterized in that: The global branch module includes an encoder, a decoder, and a parameterless attention module; The encoder and decoder each contain three convolutional layers and two upsampling layers; The parameterless attention module includes a normalization layer, multi-head self-attention, a multilayer perceptron, and 12 converter modules.

6. The low-light image enhancement method based on spatial adaptive supervised learning as described in claim 1, characterized in that: The training method for the low-light enhancement network model includes the following steps: Step 1: Obtain training samples; the training samples include low-light images and normal-light images of the samples; Step 2: Preprocess the obtained training samples; Step 3: Input the preprocessed training sample data into the low-light enhancement network model to perform low-light image enhancement and obtain the output image; Step 4: Calculate the reconstruction loss and update the parameters of the low-light enhancement network model using backpropagation to complete one model training cycle; Continue model training by repeating steps 1-4 until the reconstruction loss is acceptable, thus completing model training.

7. The low-light image enhancement method based on spatial adaptive supervised learning as described in claim 6, characterized in that: The training samples used were the SID and SDSD training sets.

8. The low-light image enhancement method based on spatial adaptive supervised learning as described in claim 6, characterized in that: The preprocessing includes image pairing, image cropping, and data augmentation.

9. The low-light image enhancement method based on spatial adaptive supervised learning as described in claim 6, characterized in that: The reconstruction loss includes Charbonnier loss and perceptual loss.

10. The low-light image enhancement method based on spatial adaptive supervised learning as described in claim 9, characterized in that: The overall loss function for the reconstruction loss is expressed as follows: L=L r *a+L r *(1-a)+λ*L vgg ; Among them, L r This is the Charbonnier loss; L vgg For perceptual loss; λ and α are both hyperparameters; Charbonnier loss r It is expressed as follows: In the formula, I' represents the output image. For the real image, ξ was set to 10 in all experiments. -3 ; Perceived loss L vgg For I' and The L1 distance between them is expressed as follows: In the formula, Φ() is the operation for extracting features from the VGG network.

Images