Low-illumination image enhancement method based on image quality prior guidance

Abstract

The invention discloses a low-illumination image enhancement method based on image quality prior guidance, which belongs to an image processing technology, and comprises the following steps: firstly, extracting image quality features through a pre-trained image quality evaluation model, then obtaining targeted features required by image enhancement through correlation calculation of the image quality features and image features obtained by an encoder, and finally, guiding the network to pay attention to more important areas in combination with a self-attention mechanism, and decoding and outputting an enhanced image through a decoder. According to the method, image quality feature information can be fully utilized, the network is guided to intensify the image in a targeted manner, the designed quality priori selective guide module is plug-and-play, the existing algorithm can be improved, and further enhancement is realized. The method can be applied to urban road automatic driving at night and underground space dim unmanned navigation systems to assist downstream visual tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer image processing, and in particular relates to a low-light image enhancement method based on image quality prior guidance. Background Art

[0002] Low-light image enhancement is a critical task in computer vision, as poor lighting often results in images with noticeable noise, reduced contrast, and distorted colors. These issues can severely impact the performance of downstream tasks such as object detection, recognition, and segmentation, which are essential in applications such as surveillance, autonomous navigation, and medical diagnosis.

[0003] In "Guided Super-Resolution Restoration of Single Image Based on Image Quality Evaluation Network," Chen et al. used an IQA model to evaluate the quality of super-resolution images and fed the evaluation results back to the network as part of the loss function. This feedback mechanism enables the network to continuously optimize during training to produce higher-quality super-resolution images. Wagner Barros also used an IQA model in "Single-shot Underwater Image Restoration: A Visual Quality-Aware Method Based on Light Propagation Model" to evaluate the quality of restored underwater images and fed the evaluation results back to the optimization algorithm. In "An Image Quality-Aware Approach with Adaptive Scattering Coefficients for Single Image Dehazing," Song et al. designed a series of quality-aware metrics to evaluate the quality of dehazed images. These metrics include texture information, chromaticity shift, and haze density. Existing methods primarily use image quality assessment in the optimization phase, rather than fully utilizing quality features during model training or enhancement. As a result, image quality improvements rely heavily on traditional pixel-level or feature-level losses and lack targeted optimization for specific quality issues, such as noise, blur, or insufficient contrast. Summary of the Invention

[0004] The present invention aims to provide a low-light image enhancement method based on image quality prior guidance. Current mainstream low-light image enhancement methods improve image quality by learning end-to-end mappings. However, this process does not fully consider the role of image quality assessment, thus limiting their potential for improving image quality. The present invention provides a low-light image enhancement method based on image quality prior guidance, which can fully utilize image quality prior information to achieve targeted enhancement.

[0005] The technical solution to achieve the purpose of the present invention is: a low-light image enhancement method based on image quality prior guidance, comprising the following steps:

[0006] Step 1. The LOLv1 low-light image dataset includes 500 pairs of original low-light images and original normal-light image groups. The low-light image dataset is divided into a training set and a test set in proportion. All original images in the training set are normalized and the pixel size is unified to H×W. The corresponding low-light images and normal-light image groups are obtained, where H is the image height and W is the image width. Go to step 2.

[0007] Step 2: Use the quality-aware encoder of the pre-trained no-reference image quality assessment model REIQA to build an image quality feature extraction network IQA, and send the low-light images in the training set to IQA to extract the quality features f q , go to step 3.

[0008] Step 3: Build a low-light image enhancement network:

[0009] The low-light image enhancement network includes an encoder module, a decoder module, and a quality prior selective guidance module. The quality prior selective guidance module includes a self-attention submodule and a feedforward network submodule. The feedforward network submodule adopts a two-branch parallel structure, and both branches use a dilated depthwise convolution operation for processing.

[0010] The encoder and decoder modules serve as the encoding and decoding networks for feature extraction. The quality prior selective guidance module is used to select quality features that are helpful for low-light image enhancement, guiding the network to focus on more important areas and features, and then proceeding to step 4.

[0011] Step 4: Use the training set to train the low-light image enhancement network to obtain a trained low-light image enhancement model, as follows:

[0012] Step 4-1: Input the low-light image in the training set into the encoder module of the low-light image enhancement network constructed in step 3 to obtain the latent feature f of the low-light image. l .

[0013] Step 4-2: The latent features f of the low-light image l and the quality feature f of the low-light image extracted in step 2 q Input the quality prior selective guidance module to obtain the enhanced features f of low-light images e .

[0014] Step 4-3: Enhance the low-light image features f e Input to the decoder module and output the enhanced image I En .

[0015] Step 4-4: Construct a loss function and train the low-light image enhancement network based on the normal-light images in the training set to obtain a trained low-light image enhancement model.

[0016] Step 5: Input the low-light images in the test dataset into the trained low-light image enhancement model to obtain the corresponding enhanced images.

[0017] The low-light image enhancement method based on image quality prior guidance can be applied in autonomous driving on urban roads at night and unmanned navigation systems in dark underground spaces to assist downstream visual tasks.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] (1) Compared with existing low-light image enhancement methods, the low-light image enhancement method based on image quality prior guidance described in the present invention introduces an image quality assessment model and realizes quality-guided low-light image enhancement from two perspectives: the feature learning process and loss design of the image enhancement network.

[0020] (2) In order to make full use of the image quality prior information, the present invention designs a screening and guidance module for image quality features, which can effectively guide the attention mechanism to focus on important areas and features in the image, thereby improving the enhancement effect.

[0021] (3) Quality features are used in loss design to constrain the similarity between the enhanced image and the normal illumination image, thereby improving the subjective visual quality of the enhanced image. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a low-light image enhancement method based on image quality prior guidance.

[0023] Figure 2 Visualization results for the LOLv1 dataset. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below:

[0025] This paper introduces an image quality assessment model into the low-light image enhancement process to capture image quality features. This model guides low-light image enhancement from two perspectives: the low-light image feature learning process and model optimization. Specifically, a quality prior selective guidance module is designed to leverage image quality features to direct the enhancement network to focus on areas requiring enhancement. Furthermore, a quality-perceived loss is designed to guide network optimization from a quality-perceived perspective, ensuring that the enhanced image is more consistent with human visual perception.

[0026] Combine Figure 1 , a low-light image enhancement method based on image quality prior guidance, the steps are as follows:

[0027] Step 1. The LOLv1 low-light image dataset includes 500 pairs of original low-light images and original normal-light image groups. The low-light image dataset is divided into training and test sets in proportion. All original images in the training set are normalized and the pixel size is unified to H×W, corresponding to the low-light image and normal-light image groups, where H is the image height and W is the image width.

[0028] Furthermore, the training set uses 485 pairs of low-light images and normal-light images, and the remaining 15 pairs are used as the test set.

[0029] Go to step 2.

[0030] Step 2: Use the quality-aware encoder of the pre-trained no-reference image quality assessment model REIQA to build an image quality feature extraction network IQA, and send the low-light images in the training set to IQA to extract the quality features f q , go to step 3.

[0031] Step 3: Build a low-light image enhancement network:

[0032] The low-light image enhancement network includes an encoder module, a decoder module, and a quality prior selective guidance module. The quality prior selective guidance module includes a self-attention submodule and a feedforward network submodule. The feedforward network submodule adopts a two-branch parallel structure, and both branches use dilated depthwise convolution operations for processing.

[0033] The encoder module and decoder module serve as the encoding and decoding network for feature extraction. The quality prior selective guidance module is used to screen quality features that have a guiding role in low-light image enhancement, guiding the network to focus on more important areas and features, and then proceed to step 4.

[0034] Step 4: Use the training set to train the low-light image enhancement network to obtain a trained low-light image enhancement model, as follows:

[0035] Step 4-1: Input the low-light image in the training set into the encoder module of the low-light image enhancement network constructed in step 3 to obtain the latent feature f of the low-light image. l .

[0036] Step 4-2: The latent features f of the low-light image l and the quality feature f of the low-light image extracted in step 2 q Input the quality prior selective guidance module to obtain the enhanced features f of low-light images e .

[0037] Step 4-2-1: The low-light image latent feature f obtained in step 4-1 is l and the quality characteristics of low-light images f q Input to the self-attention submodule of the quality prior selective guidance module, f l After layer normalization, the atrous depth convolution operation is used to obtain the query vector Q, key vector K, and value vector V, which are expressed as follows:

[0038]

[0039] Where LN(·) is the layer normalization operation, Represents a dilated depthwise convolution operation with a convolution kernel of m and a dilation rate of d1.

[0040] Step 4-2-2, calculate the quality feature f of the low-light image q Correlation with query vector Q:

[0041] S=Cor(Q,f q )

[0042] Where S represents the correlation score of two vectors, and Cor(·) represents the correlation calculation of two vectors using cosine similarity.

[0043] The relevance score can reflect the quality characteristics f q The correlation relationship between the query vector Q and the quality feature f is filtered out through the correlation score. q The k channel features f that are most relevant to the query vector Q qtopk :

[0044] f qtopk =Topk(S,f q )

[0045] In the formula, Topk(·) represents the quality feature f according to the correlation score S. q Select the features f of the k channels with the highest correlation scores qtopk Filter function.

[0046] The most relevant quality feature f qtopk Cascaded with the query vector Q to obtain the enhanced query vector Q en , and use it to guide the calculation of self-attention, so that the network pays more attention to areas that are more relevant to visual quality:

[0047] Q en =Conat(Q,f qtopk ),

[0048] Attention(Q en ,K,V)=V⊙Softmax(KQ en )

[0049] f o =Attention(Q en ,K,V)+f l

[0050] Where COncat(·) represents the cascade operation, f o represents the output feature of the self-attention submodule, and ⊙ represents the element-wise multiplication operation.

[0051] Step 4-2-3, the output feature f of the self-attention submodule o The feedforward network submodule of the input quality prior selective guidance module uses GELU as a nonlinear activation function, and then multiplies the outputs of the two branches as a gating mechanism to regulate the information flow. The result of the multiplication of the two branches and the output feature f of the self-attention submodule are o Add together to form a residual structure, promote information flow transmission, and obtain the enhanced feature f of low-light images e :

[0052]

[0053] f e =GELU(f1)⊙f2+f o

[0054] Where, represents a dilated depthwise convolution operation with a convolution kernel of n and a dilation rate of d2. f1 and f2 represent the output results of the two branches of the feedforward network submodule, respectively. GELU(·) represents the nonlinear activation function used.

[0055] Step 4-3: Enhance the low-light image features f e Input to the decoder module and output the enhanced image I En ;

[0056] Step 4-4: Construct a loss function and train the low-light image enhancement network based on the normal-light images in the training set to obtain a trained low-light image enhancement model.

[0057] The loss function includes the reconstruction loss L recon and quality-aware loss L Q , as follows:

[0058] L recon =||I En -I Gt ||1

[0059] Where, I En is the enhanced image obtained in step 4-3, I Gt For I En The corresponding normal lighting image.

[0060] The enhanced image I En and normal lighting image I Gt Input the image quality feature extraction network IQA constructed in step 2 respectively, extract the corresponding quality features, and construct the L1 loss between them:

[0061] L Q =||IQA(I En )-IQA(I Gt )||1

[0062] Overall loss I total as follows:

[0063] I total =L recon +λL Q

[0064] Where λ is a hyperparameter for balanced training.

[0065] Step 5: Input the low-light images in the test dataset into the trained low-light image enhancement model to obtain the corresponding enhanced images.

[0066] Example 1

[0067] The present invention provides a low-light image enhancement method based on image quality prior guidance, comprising the following steps:

[0068] Step 1: The League of Legends v1 low-light image dataset consists of 500 pairs of original low-light images and original normal-light images. The dataset is divided into training and test sets proportionally. The training set uses 485 pairs of low-light and normal-light images, and the remaining 15 pairs serve as the test set. All original images in the training set are normalized to a uniform pixel size of 256×256. This yields the corresponding low-light and normal-light image sets, and the process proceeds to Step 2.

[0069] Step 2: Use the quality-aware encoder of the pre-trained no-reference image quality assessment model REIQA to build an image quality feature extraction network IQA, and send the low-light images in the training set to IQA to extract the quality features f q , go to step 3.

[0070] Step 3: Build a low-light image enhancement network:

[0071] The low-light image enhancement network includes an encoder module, a decoder module, and a quality prior selective guidance module. The quality prior selective guidance module includes a self-attention submodule and a feedforward network submodule. The feedforward network submodule adopts a two-branch parallel structure, and both branches use dilated depthwise convolution operations for processing.

[0072] The encoder module and decoder module serve as the encoding and decoding network for feature extraction. The quality prior selective guidance module is used to screen quality features that have a guiding role in low-light image enhancement, guiding the network to focus on more important areas and features, and then proceed to step 4.

[0073] Step 4: Use the training set to train the low-light image enhancement network to obtain a trained low-light image enhancement model, as follows:

[0074] Step 4-1: Input the low-light image in the training set into the encoder module of the low-light image enhancement network constructed in step 3 to obtain the latent feature f of the low-light image. l .

[0075] Step 4-2: The latent features f of the low-light image l and the quality feature f of the low-light image extracted in step 2 q Input the quality prior selective guidance module to obtain the enhanced features f of low-light images e .

[0076] Step 4-2-1: The low-light image latent feature f obtained in step 4-1 is l and the quality characteristics of low-light images f q Input to the self-attention submodule of the quality prior selective guidance module, f lAfter layer normalization, the atrous depth convolution operation is used to obtain the query vector Q, key vector K, and value vector V, which are expressed as follows:

[0077]

[0078] Where LN(·) is the layer normalization operation, Represents a dilated depthwise convolution operation with a convolution kernel of m and a dilation rate of d1.

[0079] Step 4-2-2, calculate the quality feature f of the low-light image q Correlation with query vector Q:

[0080] S=Cor(Q,f q )

[0081] Where S represents the correlation score of two vectors, and Cor(·) represents the correlation calculation of two vectors using cosine similarity.

[0082] The relevance score can reflect the quality characteristics f q The correlation relationship between the query vector Q and the quality feature f is filtered out through the correlation score. q The k channel features f that are most relevant to the query vector Q qtopk :

[0083] f qtopk =Topk(S,f q )

[0084] In the formula, Topk(·) represents the quality feature f according to the correlation score S. q Select the features f of the k channels with the highest correlation scores qtopk Filter function.

[0085] The most relevant quality feature f qtopk Cascaded with the query vector Q to obtain the enhanced query vector Q en , and use it to guide the calculation of self-attention, so that the network pays more attention to areas that are more relevant to visual quality:

[0086] Q en =Concat(Q,f qtopk ),

[0087] Attention(Q en ,K,V)=V⊙Softmax(KQ en )

[0088] f o =Attention(Q en ,K,V)+f l

[0089] Where Concat(·) represents the cascade operation, f o represents the output feature of the self-attention submodule, and ⊙ represents the element-wise multiplication operation.

[0090] Step 4-2-3, the output feature f of the self-attention submodule o The feedforward network submodule of the input quality prior selective guidance module uses GELU as a nonlinear activation function, and then multiplies the outputs of the two branches as a gating mechanism to regulate the information flow. The result of the multiplication of the two branches and the output feature f of the self-attention submodule are o Add together to form a residual structure, promote information flow transmission, and obtain the enhanced feature f of low-light images e :

[0091]

[0092] f e =GELU(f1)⊙f2+f o

[0093] Where, represents a dilated depthwise convolution operation with a convolution kernel of n and a dilation rate of d2. f1 and f2 represent the output results of the two branches of the feedforward network submodule, respectively. GELU(·) represents the nonlinear activation function used.

[0094] Step 4-3: Enhance the low-light image features f e Input to the decoder module and output the enhanced image I En .

[0095] Step 4-4: Construct a loss function and train the low-light image enhancement network based on the normal-light images in the training set to obtain a trained low-light image enhancement model.

[0096] The loss function includes the reconstruction loss L recon and quality-aware loss L Q , as follows:

[0097] L recon =||I En -I Gt ||1

[0098] Where, I En is the enhanced image obtained in step 4-3, I Gt For I En The corresponding normal lighting image.

[0099] The enhanced image I En and normal lighting image IGt Input the image quality feature extraction network IQA constructed in step 2 respectively, extract the corresponding quality features, and construct the L1 loss between them:

[0100] L Q =||IQA(I En )-IQA(I Gt )||1

[0101] Overall loss I total as follows:

[0102] I total =L recon +λL Q

[0103] Where λ is the hyperparameter for balanced training and is set to 0.5.

[0104] Step 5: Input the low-light images in the test dataset into the trained low-light image enhancement model to obtain the corresponding enhanced images.

[0105] The present invention conducts relevant experiments on a network built with Python programming language and Pytorch deep learning framework on a computer configured with i7 CPU and NVIDIA3080 GPU. During the training process, the batch size is set to 2, the optimizer uses AdamW optimizer with weight decay of 0.01, and the learning rate adjustment method adopts cosine annealing learning rate adjustment, with an initial learning rate of 0.00004. Multiple batches are trained on the training set to obtain the low-light image enhancement method based on image quality feature prior guidance described in the present invention. The visualization experimental results are shown in Figure 2. Figure 2 shown.

[0106] This paper designs a plug-and-play quality prior selective guidance module and a quality-aware loss function. To demonstrate the superior performance of this paper, a recently developed low-light image enhancement method based on a codec network structure was selected as a baseline comparison model and compared with this paper. The comparative experimental results are shown in Table 1.

[0107] Table 1 Comparative experimental results of different methods on the LOLv1 dataset

[0108]

[0109] From the quantitative experimental results in Table 1, it can be seen that after the introduction of the present invention, the PSNR of SANet increased by 2.18 and the SSIM increased by 0.032; after the introduction of the present invention, the PSNR of LLFlow-L increased by 2.3. In particular, FSNet is a general image enhancement network. After the introduction of the present invention, the SSIM increased by 0.04, and the enhancement effect was significantly improved. Figure 2,After introducing this method on the basis of the baseline method, the ,enhanced image details are clearer and more natural, thus more ,conducive to downstream tasks. ,In summary, both qualitative and quantitative experiments demonstrate the ,effectiveness of this method.

Claims

1. A low-light image enhancement method based on image quality prior guidance, characterized in that: Here are the steps: Step 1: The LOLv1 low-light image dataset includes 500 pairs of original low-light images and original normal-light image groups. The low-light image dataset is divided into a training set and a test set proportionally. All original images in the training set are normalized to the same pixel size of H×W, where H is the image height and W is the image width. The corresponding low-light image and normal-light image groups are obtained, and then the process goes to step 2. Step 2: Use the quality-aware encoder of the pre-trained no-reference image quality assessment model REIQA to build an image quality feature extraction network IQA, and send the low-light images in the training set to IQA to extract the quality features f q , go to step 3; Step 3: Build a low-light image enhancement network: The low-light image enhancement network includes an encoder module, a decoder module, and a quality prior selective guidance module; The encoder and decoder modules serve as the encoding and decoding networks for feature extraction. The quality prior selective guidance module is used to select quality features that have a guiding effect on low-light image enhancement, guiding the network to focus on more important areas and features, and then proceeding to step 4. Step 4: Use the training set to train the low-light image enhancement network to obtain a trained low-light image enhancement model, as follows: Step 4-1: Input the low-light image in the training set into the encoder module of the low-light image enhancement network constructed in step 3 to obtain the latent feature f of the low-light image. l ; Step 4-2: The latent features f of the low-light image l and the quality feature f of the low-light image extracted in step 2 q Input the quality prior selective guidance module to obtain the enhanced features f of low-light images e ; Step 4-3: Enhance the low-light image features f e Input to the decoder module and output the enhanced image I En ; Step 4-4: Construct a loss function and train the low-light image enhancement network based on the normal-light images in the training set to obtain a trained low-light image enhancement model. Step 5: Input the low-light images in the test dataset into the trained low-light image enhancement model to obtain the corresponding enhanced images.

2. The low-light image enhancement method based on image quality prior guidance according to claim 1, characterized in that: The training set consists of 485 pairs of low-light images and normal-light images, and the remaining 15 pairs are used as the test set.

3. The low-light image enhancement method based on image quality prior guidance according to claim 1, characterized in that: In step 3, the quality prior selective guidance module includes a self-attention submodule and a feedforward network submodule.

4. The low-light image enhancement method based on image quality prior guidance according to claim 3, characterized in that: The feedforward network submodule adopts a two-branch parallel structure, and both branches are processed using hole depth convolution operations.

5. The low-light image enhancement method based on image quality prior guidance according to claim 4, characterized in that: In step 4-2, the latent feature f of the low-light image is l and the quality feature f of the low-light image extracted in step 2 q Input the quality prior selective guidance module to obtain the enhanced features f of low-light images e , as follows: Step 4-2-1: The low-light image latent feature f obtained in step 4-1 is l and the quality characteristics of low-light images f q Input to the self-attention submodule of the quality prior selective guidance module, f l After layer normalization, the atrous depth convolution operation is used to obtain the query vector Q, key vector K, and value vector V, which are expressed as follows: Where LN(·) is the layer normalization operation, Represents a dilated depthwise convolution operation with a convolution kernel of m and a dilation rate of d1; Step 4-2-2, calculate the quality feature f of the low-light image q Correlation with query vector Q: S=Cor(Q,f q ) Where S represents the correlation score of two vectors, and Cor(·) represents the correlation calculation of two vectors using cosine similarity; The relevance score can reflect the quality characteristics f q The correlation relationship between the query vector Q and the quality feature f is filtered out through the correlation score. q The k channel features f that are most relevant to the query vector Q qtopk : f qtopk =Topk(S,f q ) In the formula, Topk(·) represents the quality feature f according to the correlation score S. q Select the features f of the k channels with the highest correlation scores qtopk The filter function; The most relevant quality feature f qtopk Cascaded with the query vector Q to obtain the enhanced query vector Q en , and use it to guide the calculation of self-attention, so that the network pays more attention to areas that are more relevant to visual quality: Q en =Concat(Q,f qtopk ), Attention(Q en ,K,V)=V⊙Softmax(KQ en ) f o =Attention(Q en ,K,V)+f l Where Concat(·) represents the cascade operation, f o represents the output feature of the self-attention submodule, and ⊙ represents the element-wise multiplication operation; Step 4-2-3, the output feature f of the self-attention submodule o The feedforward network submodule of the input quality prior selective guidance module uses GELU as a nonlinear activation function, and then multiplies the outputs of the two branches as a gating mechanism to regulate the information flow. The result of the multiplication of the two branches and the output feature f of the self-attention submodule are o Add together to form a residual structure, promote information flow transmission, and obtain the enhanced feature f of low-light images e : f e =GEL(f1)⊙f2+f o Where, represents a dilated depthwise convolution operation with a convolution kernel of n and a dilation rate of d2. f1 and f2 represent the output results of the two branches of the feedforward network submodule, respectively. GELU(·) represents the nonlinear activation function used.

6. The low-light image enhancement method based on image quality prior guidance according to claim 5, characterized in that: In step 4-4, a loss function is constructed and the low-light image enhancement network is trained with the normal-light images in the training set to obtain a trained low-light image enhancement model, as follows: The loss function includes the reconstruction loss L recon and quality-aware loss L Q , as follows: L recon =||I En -I Gt ||1 Where, I En is the enhanced image obtained in step 4-3, I Gt For I En The corresponding normal lighting image; The enhanced image I En and normal lighting image I Gt Input the image quality feature extraction network IQA constructed in step 2 respectively, extract the corresponding quality features, and construct the L1 loss between them: L Q =||IQA(I En )-IQA(I Gt )||1 Overall loss I total as follows: THE total =L recon +λL Q Where λ is a hyperparameter for balanced training.

Images