A local scale perception based low-light image enhancement method
By constructing the SA-LLNet network and utilizing a scale-aware converter module and a residual upsampling network, the problem of CNNs' difficulty in capturing multi-scale and global information is solved, achieving adaptive low-light image enhancement and improving image quality and detail clarity.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUNAN UNIV
- Filing Date
- 2024-04-25
- Publication Date
- 2026-06-05
AI Technical Summary
Existing CNN-based low-light image enhancement methods struggle to capture multi-scale feature information and global information, leading to problems such as insufficient enhancement or overexposure in local areas.
We construct a scale-aware low-light enhancement network, SA-LLNet, and extract multi-scale features and global information by designing a scale-aware converter module. Combined with a residual upsampling network, we generate high-quality images.
Achieve adaptive low-light image enhancement, avoiding local overexposure or underexposure, and improving image quality and detail clarity.
Smart Images
Figure CN118429241B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an image enhancement method, and more particularly to a low-light image enhancement method with local scale awareness. Background Technology
[0002] Low-light image enhancement is an important and challenging task in computer vision. Images captured in low light or insufficient light conditions often suffer from problems such as low brightness, missing image information, and excessive noise, severely impacting image quality. Low-light image enhancement aims to improve the lighting, color, and contrast of such images to make them appear more natural and clear. This task has been widely used in practical applications such as security monitoring, product inspection, human-computer interaction, medical applications, and object detection. In recent years, with the rapid development of convolutional neural networks (CNNs), CNNs have been extensively applied to computer vision tasks, and CNN-based low-light enhancement methods have been extensively studied. Existing techniques have achieved low-light enhancement by constructing a stacked sparse denoising autoencoder (LLNet), demonstrating for the first time that deep learning is suitable for low-light enhancement tasks. In 2020, a method was proposed to introduce multi-scale features into multiple branch structures and fuse features from multiple branches and stages to achieve low-light image enhancement. In 2021, a generative adversarial network with a self-attention mechanism was designed and trained in an unpaired GAN manner. Although this method outperforms a series of existing GAN-based low-light enhancement methods, it always produces some unknown artifacts during the enhancement process.
[0003] The aforementioned CNN-based methods have achieved significant results. However, these methods are limited by the size of the convolutional kernel, making it difficult to capture multi-scale feature information and global information. The drawbacks are twofold: First, from a local perspective, the illumination distribution of low-light images is complex and diverse, with localized areas of extremely low illumination or large color differences. Therefore, convolutional operations of a single size are insufficient to fully learn the feature information of the low-light distribution. Second, from a global perspective, the illumination of low-light images exhibits a certain global style, such as being generally very dark or generally overexposed. Learning global information helps the model guide local illumination enhancement from a global perspective. By capturing both global and local information, the model possesses adaptive enhancement capabilities. Limited by the size of the receptive field of the convolutional kernel, CNNs struggle to learn global contextual information, leading to insufficient enhancement in slightly darker local areas or overexposure in slightly brighter local areas. Summary of the Invention
[0004] The purpose of this invention is to provide a low-light image enhancement method with local scale awareness, thereby overcoming the shortcomings of the prior art.
[0005] The technical solution of this invention is to construct a scale-aware low-light enhancement network, namely Low-Light ImageEnhancement via Scale Aware Network, abbreviated as SA-LLNet. By improving the model's multi-scale feature representation ability and learning the global vision, it fully learns rich local and global information, thereby achieving adaptive low-light image enhancement. The input of this network is a low-light image, and the output is its corresponding enhanced image.
[0006] The core of this invention is the design of a scale-aware Transformer module for extracting multi-scale features and rich global information. It first obtains features at different scales through three sampling operations, then calculates the local contextual dependencies of the corresponding scale features, and finally fuses these local contextual dependencies and processes them through a fully connected layer to obtain the features at the current stage. Based on this module, this invention constructs a scale-aware feature extraction network to extract feature information at different levels. Simultaneously, a scale-aware context learning network is designed to further learn global image information based on features with a large receptive field and rich semantics, thereby obtaining features that include image scale and context. Finally, these features are processed through a residual upsampling network to gradually generate high-quality images with reasonable illumination and clear details.
[0007] The present invention specifically includes the following steps:
[0008] Step 1 involves constructing the SA-LLNet network, which stands for Low-Light Image Enhancement via Scale Aware Network. This network adaptively enhances low-light images, avoiding local overexposure and underexposure. The network consists of a scale-aware feature extraction network, a scale-aware context extraction network, and a residual upsampling network. SA-LLNet can be represented as follows:
[0009] f s =H SAE (I ll )
[0010] f c =H SAC (f s )
[0011] I en =H De (f c )
[0012] Among them, f s The extracted features, H SAE(·) represents a scale-aware feature extraction network, which mainly consists of a scale-aware Transformer module and a sampling operation; f c Representing features at different scales, extracting multi-scale information from the context, H SAC (·) represents a scale-aware context extraction network, which consists of a scale-aware Transformer module and residual connections; I en H represents the upsampled image. De (·) represents a residual upsampling network, which mainly consists of residual modules and upsampling operations;
[0013] Step 2: To enable SA-LLNet to extract multi-scale data and rich context, a novel scale-aware transformer (SA-Trans) module is designed, mainly consisting of an image patch embedding module, a multi-scale attention learning module, and a fully connected layer. The input features are first processed through image patch embedding serialization, then fused with location embeddings before being fed into the multi-scale attention learning module. This module uses downsampling to obtain input features of multiple sizes, learns local scale dependencies, and then fuses them to obtain features containing rich scale information and local context. These features are then fed into a fully connected layer to integrate the local contextual relationships. Finally, the serialized vector is reshaped into a two-dimensional feature map and then upsampled to obtain the final output.
[0014] S2.1 Image Patch Embedding: This method divides the input features into several p*p patches and converts them into a serialized vector. First, the input features (C×H×W) are fed into a convolution kernel and a convolution operation H with a stride of p. conv get The characteristics are then reshaped (H) Resh and normalization operation H Norm Obtain the serialized vector F FE It can be represented as:
[0015]
[0016] S2.2 Multi-scale Attention Learning Module: The multi-scale attention learning module is the core component of the scale-aware transformer (Transformer) module. It is used to capture the contextual relationships within features at different scales. First, downsampling is performed using convolution operations with different kernel sizes (3, 5, 7) to obtain input features of three sizes. Then, attention maps are calculated separately and applied to the input features of the corresponding sizes to obtain features containing local contextual information. Finally, the features containing different local contextual information are fused to obtain rich contextual information. This part can be represented as:
[0017] f i =Conv i (x), i = 3, 5, 7
[0018]
[0019]
[0020] Where x is the input feature; Conv i (·) represents a convolution operation with kernel size i; H Norm (·) indicates the normalization layer; H MHSA (·) represents multi-head attention; w1, w2, and w3 are the key vector, query vector, and value vector obtained by linear transformation, respectively.
[0021] S2.3 Fully Connected Layer: After obtaining rich local contextual information through the multi-scale attention learning module, we feed it into the fully connected layer to obtain long-distance dependencies between local elements. This part can be represented as:
[0022]
[0023]
[0024] Among them, H FC (·) indicates a fully connected layer. This is done by adding pixels one by one.
[0025] Step 3: Construct a scale-aware feature extraction network and a scale-aware context extraction network based on SA-Trans. These mainly consist of SA-Trans and residual connections. The core part can be represented as follows:
[0026]
[0027] Where x is the input feature, H SAT (·) represents SA-Trans.
[0028] Step 4: The output of the scale-aware context learning network is used as the input of the upsampling sub-network; finally, a 1×1 convolution is used to map the number of channels to the output channels to obtain the enhanced image.
[0029] To evaluate the image quality of low-light image enhancement, this invention considers image loss in terms of global information, detail information, and generated content. It defines a multinomial loss function combining reconstruction loss, perceptual loss function, and content generation loss function to compensate for insufficient image quality evaluation criteria and guide the network to generate high-quality images. rec For reconstruction loss, L per For perceived loss, where w rec =1,w per =0.006;
[0030] L ACE-VAE =w rec L rec +w per L per
[0031] Reconstruction loss is used to ensure that SA-LLNet generates low-light enhanced images with complete structural content. Specifically, MSE is used to calculate the error between the SA-LLNet(.) output and the normal illumination image; as shown in the following formula, I gt For images with normal illumination, SA-LLNet(.) represents the SA-LLNet network;
[0032]
[0033] While smoothing noise in images, MSE also smooths out detailed texture features, leading to blurry images and a tendency to produce artifacts. Therefore, perceptual loss is introduced to guide the network to retain more texture information and improve the quality of the enhanced image, as shown in the following formula:
[0034]
[0035] Among them, w ij ,h ij ,c ij The dimensions of each feature map in the SA-LLNet network. μ represents the i-th convolutional module of the j-th convolutional group in SA-LLNet. x ,μ y The average pixel value. Let σ be the variance. xy Let C1 and C2 be the covariances, and C1 and C2 be constants.
[0036] The beneficial effects of this invention are that the model can improve the multi-scale feature representation capability and learn the global perspective, fully learn rich local and global information, thereby achieving adaptive low-light image enhancement. While ensuring image quality, it improves color saturation and diversity, enhances the detail information of low-light images, and makes the low-light image enhancement effect more natural and clearer. Attached Figure Description
[0037] Figure 1 It is a scale-aware feature extraction network.
[0038] Figure 2 This is a scale-aware converter module. Detailed Implementation
[0039] The following is in conjunction with the appendix Figures 1 to 2 The specific embodiments of the present invention are further described below. A scale-aware low-light enhancement network, namely Low-Light Image Enhancement via Scale Aware Network (SA-LLNet), is constructed. By improving the model's multi-scale feature representation ability and learning the global perspective, it fully learns rich local and global information, thereby achieving adaptive low-light image enhancement. The input of this network is a low-light image, and the output is its corresponding enhanced image.
[0040] The core of this invention is the design of a scale-aware Transformer module for extracting multi-scale features and rich global information. It first obtains features at different scales through three sampling operations, then calculates the local contextual dependencies of the corresponding scale features, and finally fuses these local contextual dependencies and processes them through a fully connected layer to obtain the features at the current stage. Based on this module, this invention constructs a scale-aware feature extraction network to extract feature information at different levels. Simultaneously, a scale-aware context learning network is designed to further learn global image information based on features with a large receptive field and rich semantics, thereby obtaining features that include image scale and context. Finally, these features are processed through a residual upsampling network to gradually generate high-quality images with reasonable illumination and clear details.
[0041] The present invention specifically includes the following steps:
[0042] Step 1 constructs the SA-LLNet network, which adaptively enhances low-light images, avoiding local overexposure and underexposure. This network consists of a scale-aware feature extraction network, a scale-aware context extraction network, and a residual upsampling network. SA-LLNet can be represented as follows:
[0043] f s =HSAE (I ll )
[0044] f c =H SAC (f s )
[0045] I en =H De (f c )
[0046] Among them, f s The extracted features, H SAE (·) represents a scale-aware feature extraction network, which mainly consists of a scale-aware Transformer module and a sampling operation; f c Representing features at different scales, extracting multi-scale information from the context, H SAC (·) represents a scale-aware context extraction network, which consists of a scale-aware Transformer module and residual connections; I en H represents the upsampled image. De (·) represents a residual upsampling network, which mainly consists of residual modules and upsampling operations;
[0047] Step 2: To enable SA-LLNet to extract multi-scale data and rich context, a novel scale-aware transformer (SA-Trans) module is designed, mainly consisting of an image patch embedding module, a multi-scale attention learning module, and a fully connected layer. The input features are first processed through image patch embedding serialization, then fused with location embeddings before being fed into the multi-scale attention learning module. This module uses downsampling to obtain input features of multiple sizes, learns local scale dependencies, and then fuses them to obtain features containing rich scale information and local context. These features are then fed into a fully connected layer to integrate the local contextual relationships. Finally, the serialized vector is reshaped into a two-dimensional feature map and then upsampled to obtain the final output.
[0048] S2.1 Image Patch Embedding: This method divides the input features into several p*p patches and converts them into a serialized vector. First, the input features (C×H×W) are fed into a convolution kernel and a convolution operation H with a stride of p. conv get The characteristics are then reshaped (H) Resh and normalization operation H Norm Obtain the serialized vector F FE It can be represented as:
[0049]
[0050] S2.2 Multi-scale Attention Learning Module: The multi-scale attention learning module is the core component of the scale-aware transformer (Transformer) module. It is used to capture the contextual relationships within features at different scales. First, downsampling is performed using convolution operations with different kernel sizes (3, 5, 7) to obtain input features of three sizes. Then, attention maps are calculated separately and applied to the input features of the corresponding sizes to obtain features containing local contextual information. Finally, the features containing different local contextual information are fused to obtain rich contextual information. This part can be represented as:
[0051] f i =Conv i (x), i = 3, 5, 7
[0052]
[0053]
[0054] Where x is the input feature; Conv i (·) represents a convolution operation with kernel size i; H Norm (·) indicates the normalization layer; H MHSA (·) represents multi-head attention; w1, w2, and w3 are the key vector, query vector, and value vector obtained by linear transformation, respectively.
[0055] S2.3 Fully Connected Layer: After obtaining rich local contextual information through the multi-scale attention learning module, we feed it into the fully connected layer to obtain long-distance dependencies between local elements. This part can be represented as:
[0056]
[0057]
[0058] Among them, H FC (·) indicates a fully connected layer. This is done by adding pixels one by one.
[0059] Step 3: Construct a scale-aware feature extraction network and a scale-aware context extraction network based on SA-Trans. These mainly consist of SA-Trans and residual connections. The core part can be represented as follows:
[0060]
[0061] Where x is the input feature, H SAT (·) represents SA-Trans.
[0062] Step 4: The output of the scale-aware context learning network is used as the input of the upsampling sub-network; finally, a 1×1 convolution is used to map the number of channels to the output channels to obtain the enhanced image.
[0063] To evaluate the image quality of low-light image enhancement, this invention considers image loss in terms of global information, detail information, and generated content. It defines a multinomial loss function combining reconstruction loss, perceptual loss function, and content generation loss function to compensate for insufficient image quality evaluation criteria and guide the network to generate high-quality images. rec For reconstruction loss, L per For perceived loss, where w rec =1,w per =0.006;
[0064] L ACE-VAE =w rec L rec +w per L per
[0065] Reconstruction loss is used to ensure that SA-LLNet generates low-light enhanced images with complete structural content. Specifically, MSE is used to calculate the error between the SA-LLNet(.) output and the normal illumination image; as shown in the following formula, I gt For images with normal illumination, SA-LLNet(.) represents the SA-LLNet network;
[0066]
[0067] While smoothing noise in images, MSE also smooths out detailed texture features, leading to blurry images and a tendency to produce artifacts. Therefore, perceptual loss is introduced to guide the network to retain more texture information and improve the quality of the enhanced image, as shown in the following formula:
[0068]
[0069] Among them, w ij ,h ij ,c ij The dimensions of each feature map in the SA-LLNet network. μ represents the i-th convolutional module of the j-th convolutional group in SA-LLNet. x ,μ y The average pixel value. Let σ be the variance. xy Let C1 and C2 be the covariances, and C1 and C2 be constants.
Claims
1. A method for enhancing low-light images with local scale awareness, characterized in that: A scale-aware low-light enhancement network, abbreviated as SA-LLNet, is constructed. By improving the model's multi-scale feature representation ability and learning global vision, it fully learns rich local and global information, thereby achieving adaptive low-light image enhancement. The input of this network is a low-light image, and the output is its corresponding enhanced image. To extract multi-scale features, a scale-aware converter module, abbreviated as SA-Trans, was designed to extract multi-scale features and rich global information. First, features at different scales are obtained through three sampling operations. Then, the local contextual dependencies of the corresponding scale features are calculated. Finally, these local contextual dependencies are fused and processed by a fully connected layer to obtain the features of the current stage. Based on the converter module, a scale-aware feature extraction network was constructed, and feature information at different levels was extracted using the converter module. Meanwhile, a scale-aware context learning network was designed to further learn global information of the image on features with a large receptive field and rich semantics, so as to obtain features that include image scale and context. Specifically, the following steps are included: Step 1: Construct the SA-LLNet network, which adaptively enhances low-light images and consists of a scale-aware feature extraction network, a scale-aware context extraction network, and a residual upsampling network. SA-LLNet is represented as follows: ； in, These are the extracted features. This represents a scale-aware feature extraction network, which mainly consists of a scale-aware transformer module, namely Transformer, and sampling operations; It represents features at different scales and extracts multi-scale information from the context. This represents a scale-aware context extraction network, which consists of a scale-aware converter module and residual connections; This represents a residual upsampling network, which mainly consists of residual modules and upsampling operations. Step 2: In order to enable SA-LLNet to extract multi-scale data and rich contextual information, a new scale-aware module is designed, which mainly consists of an image patch embedding module, a multi-scale attention learning module, and a fully connected operation. The input features are first processed by image patch embedding serialization, then fused with location embedding and fed into the multi-scale attention learning module. The multi-scale attention learning module uses downsampling to obtain input features of multiple sizes, learns the local scale dependencies respectively, and then fuses them to obtain features containing rich scale information and local context. Subsequently, it is fed into a fully connected layer to integrate the local contextual relationships. Finally, the serialized vector is transformed into a two-dimensional feature map through deformation operation and fed into an upsampling operation to obtain the final output. S2.1 Image Patch Embedding: Used to divide the input features into several p*p patches and convert them into serialized vectors; first, the input features... A convolution operation with a kernel and stride of p is performed. get The characteristics, and then after deformation and normalization operation Obtain the serialized vector , is represented as: ； S2.2 Multi-scale attention learning module: The multi-scale attention learning module is the core component of scale awareness, used to capture the contextual relationships within features at different scales; First, downsampling is performed using convolution operations with different kernel sizes: 3, 5, and 7, resulting in input features of three sizes. Then, attention maps are calculated for each size and applied to the corresponding input features to obtain features containing local contextual information. Finally, features containing different local contextual information are fused to obtain rich contextual information. This part is represented as: ； in, This represents the image features extracted by convolution. For input features; Indicates the kernel size as Convolution operations; express Image features after head attention processing; Indicates the normalization layer; For the attention of multiple parties; The key vector, query vector, and value vector are obtained by linear transformation, respectively. S2.3 Fully Connected Layer: After obtaining rich local contextual information through the multi-scale attention learning module, this information is fed into the fully connected layer to obtain long-distance dependencies between local elements. This part is represented as follows: ； in, Indicates a fully connected layer. This is done by adding pixels one by one; Step 3: Construct a scale-aware feature extraction network and a scale-aware context extraction network based on SA-Trans. These mainly consist of SA-Trans and residual connections, with the core part represented as follows: ； in, For input features, For SA-Trans; Step 4: The output of the scale-aware context learning network is used as the input to the upsampling sub-network; finally, the output is used... Convolution maps the number of channels to the output channels, resulting in an enhanced image.
2. The low-light image enhancement method with local scale awareness according to claim 1, characterized in that: To evaluate the image quality of low-light image enhancement, the image loss is considered from the aspects of global information, detail information and generated content. A multinomial loss function for reconstruction loss, perceptual loss function and content generation is defined to make up for the lack of evaluation criteria for generated image quality and guide the network to generate high-quality images. ； To reconstruct the loss, To perceive loss, among which ; Reconstruction loss is used to ensure that SA-LLNet generates low-light enhanced images with complete structural content. Specifically, MSE is used to calculate the error between the SA-LLNet output and the normal illumination image, as shown in the following formula. For images with normal illumination, SA-LLNet(.) represents the SA-LLNet network; ； While smoothing noise in images, MSE also smooths out detailed texture features, leading to blurry images and a tendency to produce artifacts. Therefore, perceptual loss is introduced to guide the network to retain more texture information and improve the quality of the enhanced image, as shown in the following formula: ； in, The dimensions of each feature map in the SA-LLNet network. This represents the i-th convolutional module in the j-th convolutional group of SA-LLNet. The average pixel value. For variance, For covariance, It is a constant.